When is an impact measurement good enough?
The determination of the scope and thoroughness of an impact study involves one important question when is it good enough? Organisations regularly ignore this question and are keen to ‘measure everything as comprehensively as possible’. The result: unnecessarily complex impact measurements that take a lot of time and often do not lead to useful insights.
It depends on several factors if impact measurements will be good enough. The most important ones are the following:
- What is the purpose of the impact measurement?
Is the main goal that you want to learn and be able to make adjustments to your project or program?
If so, then don't turn it into an exact science. Determine for yourself when you have enough ‘sense’ of the impact made and what does and does not work.
Do you have to report to a funder or client?
Talk to them about their expectations. When is it good enough for them?
Do you want to prove your impact and make big claims about it?
In that case, robust and more scientific research will be needed. Three interviews may sometimes be enough to learn from, but are probably not enough to convince others and definitely will not offer solid proof.
- What is the scope of the initiative that is being researched?
The scope of an impact measurement must be proportional to what you are researching. It would be a waste to develop a large-scale impact study for a two-month project with fifteen participants. In that case, keep it small, practical and simple. On the other hand, in a large and multi-annual programme, it is a missed opportunity if there is no continuous impact monitoring appropriate. There is a lot to learn, after all.
- What knowledge, resources and capacity are available?
Be realistic about what is feasible and how high you can and want to set the bar. Organisations with years of experience with impact measurement and with capacity to carry this out, obviously may set the bar a lot higher. If your organisation is new to this, consider it to be a process of growth and accept that you start small. Of course, always ensure that you are transparent about what the results of an impact measurement do and do not indicate.
Collecting data for impact measurement
Collecting data is an indispensable part of every impact measurement. Often, you collect this data from the people or organisations you want to impact. This step, in particular, may feel like a considerable undertaking. And that's true: for large impact studies, data collection takes (a lot of) time. Yet, here too, the principle applies: don't make it unnecessarily large or complex.
Additionally, start with the data you already have. This is often more than you initially think, for example: records of programme participation, feedback forms, online reviews or even informal conversations with participants in a project or the visitors of your activity. In short: in the course of your activities, you have often already collected a lot of data. So, make good use of this data and turn it into useful impact information! Here’s how:
- Give structure to the data
Data in itself says very little. You still have to structure it to prevent it from remaining 'just snippets' of information. For example, consider informal conversations you have with participants in a project. These provide interesting insights but have little value if you do not record the results properly.
So, structuring data means in this respect: recording it in a way that allows you to do something with it. You apply structure directly if you collect the data in a structured way. For example, schedule a conversation with three participants after each meeting, instead of having ad hoc conversations.
- Give meaning to the data
Structured data becomes truly valuable when you give it meaning. This is human work and cannot be done by an analysis program (or ChatGPT). It’s all about asking: what does this actually tell us, and what do we think about it? Compare the collected data with the goals you set at the beginning of a program.
What conclusions can you draw? What effects have been realised and which have not yet been achieved? Is this surprising or did we expect it? And what can we then do with these insights? What are you going to do differently? What follow-up questions do you have? Giving meaning to the data means above all: asking (yourself) many questions and reflecting critically.
- Ensure transparency about the impact measurement
Finally, and perhaps this is self-evident, the importance of transparency about the quality and nature of the impact measurement. Share how you collected and analysed the data as well as how it resulted in certain conclusions. Also, share the limitations of the research. This openness builds trust and provides an opportunity for feedback and dialogue. This also means that you should make things appear larger or more important than they are. Three interviews may be very valuable, yet they do not provide the basis for large claims about the realised impact.
Getting started with your impact study
- Start small
You don't have to measure everything at once. Start with one project, programme or activity and then expand. Think of it as a process of growth.
- Use existing data
Most likely, you are measuring more than you think. Think carefully about what data you (unconsciously) already collect without referring to this as ‘impact data’. Use this data as a starting point.
- Learn from others
Consider the way in which similar organisations or initiatives measure their impact and learn from their experiences. A huge amount is already happening. It's a shame to reinvent the wheel.
- Get Started
Don't fall into the trap of 'wantingto think it all through perfectly first'. Practical impact measurement means above all: getting started, learning from the things that work and that do not work and taking it from there.
Would you like more information about impact measurement?
Please contact one of our experts. They are here to help.
Contact us
