My esteemed colleague Jeff Kutash told me last week that he’d seen a study that says that the good news is that these days low income children have more “screen time” (think bridging the digital divide) than their more affluent peers. The bad news is that these days low income children have more “screen time” than their more affluent peers. Specifically, research shows that in general, kids of all groups are spending too much time in front of screens, and that low income kids now spend more time than their more affluent peers. Yes, the good news is the bad news in this case. This got me thinking about the danger of outputs or the trouble with wrongly defining the problem. And about microcredit. And foreclosures. Curious? Read on…
Measuring results is hard. And even harder when evaluating “impact” – aka “change”. So we often settle for proxies that are more pragmatic. Things we can count. However, more and more I see this as a dangerous method in the long-run.
Take the aforementioned digital divide. If we articulate the problem only as “bridging the digital divide” – aka ensuring “access to information technology” we’ve done ourselves a favor as access is somewhat easy to count. And according to Jeff, “To be clear, for a long time, there was a problem with the digital divide with low income households and students of color unable to participate in the digital age, and the problem was ensuring access. So at that time, measuring access was actually okay. But of course, once you ensure access, you have to ensure that use is of quality.”
In other words, as you get closer to the finish line, you better move the goal post again. Because if we stick to the original measures of success around access or usage (screen time) as a proxy for success, we’ve gone down a dangerous path. As you can surely imagine, more screen time is not necessarily good! It hampers creativity, interpersonal interaction, self-directed exploration, etc. The problem should have been articulated differently: “low income children aren’t able to experience the personal and academic benefits that can come from access to information technology.” Then the measurement would not have been number of children reached, but the actual personal and academic benefits. These benefits could include improved educational achievement, ability to lead healthier lives, increased economic opportunity, and participation in their communities Harder to measure of course, but avoids the trap of declaring success just by posting high usage numbers.
Microcredit is a similar story. The problem was wrongly articulated as “not enough low income households can access loans.” Voila – the measure of success becomes number of loans or volume of loans dispersed. Easy to count and easy to celebrate as a win. We all know what happened in this case. Instead, the problem should have been defined as “not enough entrepreneurs can access capital to help grow their businesses or farms”. In this case, the measure of success would have been growth in asset base and cashflow of borrowers over time. And loans would only have been given to those who could put them to productive use. Again, by settling for a proxy (number of loans disbursed) impact was in many places quite negative.
Similarly, the US mortgage crisis comes to mind. The problem was unfortunately defined as “not enough low income households own their own home” rather than “not enough low income households are building a long-term asset base through home ownership”. Subtle difference in articulation, but major implication on what counts as success.
In all of these examples, focusing on increasing outputs of the (wrongly defined) problem actually ended up having a negative effect on impact!
As you think about your own theories of change, ask yourself if you’ve correctly defined the ultimate problem you’re trying to solve and if any of the indicators you’re tracking along the way may in the end be counterproductive.
