This blog post is the fourth in a series focusing on a study of early learning success markers (indicators), ages 0-8, supported by the W.K. Kellogg Foundation. The full report, Markers that Matter: Success Indicators in Early Learning and Education was released on July 15, 2013. In May, FSG hosted a webinar where we shared some highlights of our work to research, synthesize and vet a set of early learning indicators. In this blog series, we are attempting to answer several questions posed by webinar participants. You can see the webinar presentation, including the indicators and emerging themes, in the slides posted here.  During the webinar, Kyle Snow of the National Association for the Education of Young Children noted the value of FSG’s research in approaching the identification of indicators based on what one wants to know versus what is readily measured.  The below follow-up pursues this idea – emphasizing the utility of indicators over the ease of measurement in identifying potential indicators – in further detail.   

Webinar participants asked: It is an interesting idea to start with what you want to know, not what you can [readily] measure. How does this lead to the successful use of indicators?

The value of any measure is the degree to which it provides useful information. Sometimes the “use” we have is within a range of descriptive purposes – simply to document and monitor over time, or across groups, the incidence of something that we think is important to know about. From this perspective, we take any measures that we can get, which we can use to paint increasingly complex pictures, and also through which we may uncover some interesting parts of the picture. Other times, our intended use is to inform decision-making. Here the focus is on gathering information that can lead to some action – so the target should be something that we think is important, and also something that we can impact. When we start our thinking with “what can we measure,” information is not being defined by utility but ease or availability. In some cases these two may lead to the same place. For example, measuring infant mortality is both reasonably well-established and leads to important information about trends in prenatal and neonatal health services. In contrast, when we start from the perspective of “what I want to know is [this],” we usually have some specific sense of what we want to know and how we would use that information, rather than focusing on what is easily measured.  In other words, the utility, or use, is driving the measurement choice.

When our indicators are driven by our ability to articulate the what (even if not the how), we build in the likelihood of increased use once we can find the means of measurement. This is important, I think, because it will sustain us through the challenge of measurement. We may find an easy way of capturing the information. For example, the indicator of infant mortality has been developed over many years, and is generally agreed-upon in definition, and is not too challenging to gather (comparatively speaking). While we certainly want to use such an indicator as a basic descriptor, for example, identifying disparities in infant health, it is also useful as a means of guiding action (for example, to target prenatal and neonatal services to higher-risk groups). But more likely, we will recognize that we have to invest resources (time, energy, money) to adequately capture (i.e., measure) what we really want to know. The greater the investment that is required to get to what we really want to know, the more critical we are of 1) the short-term goal  to identify or develop an appropriate measure for a desired indicator, and 2) the long-term goal of what a measure of that indicator may be able to do for us. This tension between the challenge of developing an adequate measure and the potential value of having it is ultimately going to lead us to abandon the measurement, or to come out of the process placing a very high value on the resulting measurement. At a macro-level, we have seen this with the human genome project – the goal of mapping the genetic code for humans was extraordinary in resource demands, but the value seen in doing so was compelling. Likewise, recent commitments to map the brain will entail a large investment, but again, the expectation is that a better understanding of how the brain works will lead to a host of advances in physical and mental health.

Identifying what you want to know is a critical step in defining indicators that will ultimately be useful.  While the complexity of measurement varies widely, specifying how an indicator can serve a desired purpose from the outset will make any investment in measuring that indicator more fruitful.

To learn more, download Markers that Matter: Success Indicators in Early Learning and Education

Kyle Snow, Ph.D., is the Director of the Center for Applied Research at the National Association for the Education of Young Children.