Accounting for Reality in Impact Measurement

I’ve been reading John Lewis Gaddis’ ‘Landscape of History’ recently, which is a great guide for anyone curious about how historians navigate various tensions as they research, study and write history.

In the book, Gaddis explores how the practice of history places itself amongst the natural sciences (physics, biology etc.) and the social sciences (economics, geography etc). One chapter in particular offers clues to the social sector, in how we think about the messiness that reality brings to impact measurement.

Gaddis asserts that there are two ways of understanding causation. The first is a reductionist view, that assumes one can break reality down almost like a mathematical equation to isolate the effects that a variable has on the outcome. In this way, you can also rank the causes in a hierarchy.

The ecological perspective on the other hand, says that all causal variables have antecedents, one thing causes another, causes another, and this ‘ecology’ of variables creates an outcome, which is itself a cause in a future outcome or many. 

What struck me most was Gaddis’ observation that while the natural sciences have moved on from a reductionist view (he shares the examples of the study of quantum physics, and epigenetics) to a more ecological view, the social sciences seem to still be stuck trying to prove legitimacy through a focus on ‘hard’ numbers, and independent variables alone. 

One line in particular that stood out to me from the book was: "The reason, I argued, is that too many social scientists, in their efforts to specify independent variables, have lost sight of a basic requirement of theory, which is to account for reality."

This is what we’ve been arguing for the last 6+ years:

Accounting for reality’ can’t happen with numbers alone; it has to bring in the views of communities, frontline workers, collaborators in all its messiness, and as it evolves while your program runs.. And then well beyond.

The beauty is to do it through stories, interviews, impressions, meeting transcripts and so much more! And when you bring these together with various numbers, you can then begin to form an ‘ecological view’ of your program, and grasp the landscape of the impact your program has had, and will be having.  

What’s truly heartening is some of the conversations I’ve had over the past year. I see more and more groups working on landscape-level interventions in truly collaborative, and multidisciplinary collectives.

As their work gets off the ground, they’re now grappling with how to understand what’s happening, what’s working, what are the variables, without forcing it into frameworks that flatten complexity. And they’re willing to take more risks to build an ecological view of their programs. 

These small shifts that I’m seeing give me hope that we might just make the right moves yet, step away from just fancy statistics and towards approaches that in the words of Gaddis, ‘account for reality’.