Case study
Confirmation bias and how funders might avoid it
29 April 2021
The Data Champions programme brings funders together to collaborate and learn how to grow a data culture in their organisations. In this blog, programme facilitator Dirk Slater shares insights from the group discussion on February 10, 2021 about grantmakers privilege and how to overcome confirmation bias.
“Many of the processes in which we collect data could be disempowering or empowering depending on the applicant.” – 360Giving Data Champion
Privilege and power in the funder-grantee relationship
The Data Champions noted that, as grantmakers, they have a privilege in deciding what data they request from grantees, how it is collected and how it will be used. They recognised that collecting data can both disempower and empower applicants, and can also give power to them as grantmakers. For example, the group pointed out that an inaccessible application form that uses technical terms could present challenges for some grantees and disempower them, possibly excluding them from the funding opportunity.
Often grantmakers are interested in collecting data that demonstrates the impact of the funds given. While grantees may also be interested in this, their priorities may differ – focusing on sustaining their activities instead. As a result, grantees might give greater priority to the data funders request rather than their own interests. As grantmakers, the Data Champions saw how they may be defining a dataset that is helpful to them without considering its usefulness to grantees, and recognised the opportunity to collect data that is useful to both their work and that of the grantees.
“Biases can come from ourselves but also be inherited within our systems.” – 360Giving Data Champion
How to avoid confirmation bias
According to Cambridge dictionary confirmation bias is, ‘the fact that people are more likely to accept or notice information if it appears to support what they already believe or expect’. In the context of funders working with data, it is about being aware of pre-existing ideas and beliefs that may influence the way data is understood, analysed and collected. The Data Champions recognised the need to consider confirmation bias in all areas of work, from data collection to analysis and organisational culture.
We asked the Data Champions, ‘How can you avoid confirmation bias?’. Below are the ideas that they came up with:
Self-awareness
- Accept that confirmation bias is an issue and be proactive in addressing it. Recognise that as funders, we are in a position of privilege.
- Present both sides of the argument rather than just your usual side.
- Don’t answer the question before you have asked it
- Can we actively avoid confirmation bias, or is it a case of mitigating it?
- Do we know what unbiased data looks like?
Data collection
- Avoid bringing hypotheses to data collection
- Sample as widely as possible (randomly)
- Collect the data yourself – i.e. based on your sources, rather than asking applicants to self-serve and supply you with all the answers up front
- Review the questions you’re asking applicants & grant holders. Potential for applicants and grantholders to co-produce applications together.
- Don’t ask leading questions
- Anonymise application forms so decisions are more likely to be based on need
- Ask applicants to determine their own outcomes
- Ask: “What is the data not telling us? What is missing? Why?”
Data Culture
- Establish a culture that encourages challenging confirmation bias.
- Consult with people you are collecting the data from
- Use the ‘Five Whys’ technique – ask ‘why?’ five times
- Don’t answer the question before you have asked it
- Include the people/organisations that will be impacted by the result of your data collection, in the design of it
- Recognise that you are a biased person (unconscious!) and get other people’s eyes on things
- Have a diverse pool of decision-makers
- Question who is involved in the decision-making process
Data analysis
- Be careful not to confuse correlation with cause and effect
- Find data that goes against the grain – what does not correlate? How was this data collected?
- Compare your data with external data sources
- Don’t have expect to prove a point through the data that you’ve ‘got a hunch on’
- Collect more than one data set for comparison.
- Have an outside person look at the data you’re collecting before putting it out
- Test, test, test and test again
“Slowing down the process to look for bias and encouraging others to voice their opinions shouldn’t be underestimated.” – Hannah Howard, OVO Foundation and 360Giving Data Champion
Resources: learn more about bias
Our Data Champions shared resources to learn more about data and biases, and how to reduce them:
- Weapons of Math Destruction. This book is about the societal impact of algorithms, written by Cathy O’Neil.
- Data Feminism, book – by Catherine D’Ignazio and Lauren F. Klein, shares ‘a new way of thinking about data science and data ethics that is informed by the ideas of intersectional feminism.’
- We All Count Tools – Practical tools to help increase data equity from We All Count.
- How Confirmation Bias Works, article – from Verywellmind defining confirmation bias, providing examples and looking at its impact.
- Cognitive Bias Codex, graphic that categorises cognitive biases into four areas: what should we remember, too much information, not enough meaning and need to act fast.
- Determine The Root Cause: 5 Whys, article explaining how asking ‘why?’ five times can be a technique to get to the root cause of a problem. (isixsigma.com)
- Leading with data: Tackling Bias, webinar from Data & Marketing Association about the role of how technology, people and leadership can drive data biases and why this is detrimental to you and your business.
- Scientific Method, Wikipedia – A starting point for understanding scientific methods.
Look out for our next blog: data for leaders
If you found this blog helpful, you may enjoy our previous blog about responsible data, ‘key questions to ensure responsible data practice for funders’. In our next blog we will share insights from our Data Champions on getting leaders to use data to encourage growing a data culture.
If you have found this blog useful or have any feedback, we’d love to know! We also welcome ideas for blogs and other content from our community, to help enable better use of data for funding organisations. Drop us an email at comms@threesixtygiving.org.