In defence of the dollar – why impact measurement needs a common denominator

Variations on the adage ‘what you don’t measure doesn’t matter’, or ‘you can’t improve what you don’t measure’ are often held up as self-evident justifications for one form or another of program or project evaluation. But what do we really mean by measurement? And how does it actually help us to ascribe meaning or value to something?

Measurement is generally understood to be the quantification of the characteristics of some event or object, which can then be used to compare it against other subjects. You could also measure the change in one subject over time. In the case of measuring the impact of a public project, our subjects are people, and the characteristic we are interested in changing is their well-being. 

However, measuring well-being (or life satisfaction, happiness, capability, or utility) has an obvious problem. Even if it were well-defined (which it is not), it is a unitless abstract totality of a person’s subjective experiences, positive and negative. People are typically able to answer on a scale of 1-10 how satisfied they are with their life, as in life-satisfaction studies, but these answers tend to be anchored to moments in time and their contextual circumstances, and therefore not directly comparable say between two groups or time periods. 

So an objective measure of total well-being seems impossible, and the task of program evaluation is lost. 

But not so fast.

The study of welfare economics has come leaps and bounds in addressing this very issue in both theory and practice. 

The welfare economist’s approach begins from one simple stylised fact: that people will prefer more of something that increases their well-being compared to something that increases it less, has no net effect, or reduces it. By way of a simple example, if a person would is indifferent to eating one ice-cream compared to two chocolate bars (say because it is hot today), then we can say that the relative amount of happiness that they get from an ice-cream is twice that of the chocolate bar, even if we still don’t know what the amount of happiness is per se. We need not choose ice-cream as our measure of relative well-being – it could be loaves of bread, pears, or hours of leisure time. 

So far, so good, we can continue this procedure with all possible goods or services that a person can receive, and measure the changes in happiness they get from each in terms of ice-creams. A bowl of soup may be worth three ice-creams, a new pair of jeans may be worth 100 ice-creams, a car worth 10,000 ice-creams. 

But how can we compare between people? 

100 ice-creams may be welcome to one person, but a lactose intolerant person may have their well-being actively diminished by even one dairy-based dessert. Instead of denominating happiness with ice-creams, we can denominate it with the amount of money it takes to buy those ice-creams, or any other good. Person A (who loves ice-cream) and person B (who is lactose intolerant) could both be given $100, and the relative number of ice-creams and all other goods that they spend that money on will tell us the relative amount of well-being that they gain from each. 

What about so-called non-market goods – those intangible experiences that we cannot easily express our preferences for with a crisp $100 note? 

Things like a love of nature, cultural experiences, or living a connected or healthy life? Surely these things are out of reach of monetary measures. 

But not so.

Just as we can infer relative value with how people spend their money directly, we can also infer relative value by observing how or with whom people spend their time, choose where to live, or their choice to consume things that indirectly affects their exposure to any non-market amenity. Even the way we take risks tells us something about how much we value our life and health compared to the things we risk them for. Much research has been undertaken to infer how we can use this notion of relative well-being to bring many things back to some notion of a dollar-denominated value. 

From all this we can see that from the welfare economist’s perspective, economic valuation isn’t really about ‘putting a dollar value’ on anything and everything, it is about determining the relative well-being that people gain (or lose) from different goods, services, or subjective experiences, and denominating those in common units to give them a relative weight. 

To most fair-minded people, we should prefer those bundles of goods or experiences that give the greatest weighted sum of changes in well-being. That our chosen weights are denoted in dollar terms is inconsequential, but convenient for being able to compare across people, groups, and time. 

That said, there are still non-market goods and contexts for which common denomination in monetary terms is still difficult. For example, many Indigenous communities around the world do not have a culture of exchange (to say nothing of money as its medium) through which easily observable measures of relative value can be obtained. That is not to say that relative well-being as an idea does not exist for them, Aboriginal people are often able to articulate their preferences for one thing over another as well as anybody else, only that our instruments of measurement are not yet up to the task. 

What then, for the role of qualitative or purely descriptive measurements of value? Like the likert-scale of subjective well-being, qualitative insights may be informative, but ultimately unitless measures of total well-being or changes in it. 

Without weights, the choice of which social program or project to pursue becomes a matter for the decision-maker, who must use their own subjective weights to compare one set of subjective experiences against all the others. In the best-case scenario where we have inclusive participatory decision-making, qualitative analysis may enable informed voting to take place, which provides all the weights we need to make a decision. However, participatory democratic processes are the rare exception, not the rule – and costly in terms of people’s time, attention and energy. Furthermore, voting weights the voices of the majority over minorities who may benefit from a program being funded many times more than the rest of the community, and thus may miss out on opportunities no matter how well articulated the benefits may be. The welfare economist’s approach to project evaluation is to ensure that not only is every stakeholder counted (not just those who show up), but their votes in favour or against a proposal are weighted (in dollar terms) with respect to the actual change in well-being that they may experience. 

Importantly, economic analysis enables us to weigh the opportunity costs of a project. That is, even if the time, money, and effort of doing something may deliver some overall benefits – if those same inputs could generate more outputs and benefits elsewhere, then we should prefer the latter. As decision-makers and their advisors, it is hard to imagine a better or more principled criteria predicated on a purely descriptive analysis of either option. 

That is not to say that qualitative analysis is superfluous to program evaluation. Indeed, it is often crucial for several reasons:

• Providing evidence that theorised impacts painted in the program logic model actually occurred. 
• Informing people of the broad outcomes of some project so that they can accurately express their preferences for or against it. 
• Uncovering unintended consequences of a project that are unexpected.
• Covering the gaps left by any empirical or economic analysis, such as helping to guide the most appropriate choice of dollarised values from a list of options or methods. 

The final point in the above list may be one of the most understated but important aspects of the link between qualitative analysis and economic evaluation. By way of example, the NSW Movement and Place Framework provides a range of tools which help to qualitatively describe the overall effect of a change in the urban transport fabric on people’s lives. Undertaking this process is key to being able to select the appropriate economic measures of impact from the large list of social and environmental costs given in the Transport for NSW Economic Parameter Values database. In much the same way, quantitative research into people’s relative preferences for different types of social or environmental outcomes can only help to guide qualitative research towards the themes most likely to yield valuable insights. 

Discussions of whether or not one type of impact analysis – economic, social, or environmental – deserve precedence over the other are ill-founded and miss the fact that all three are necessary to inform good decision making and each other. 

That said, economic analysis carries obvious advantages, being a robust way to weigh the impacts between different physical, financial, and subjective intangible effects for individual people, and then weight the balance of those effects between people. That dollars are only a convenient, but arbitrary, yardstick of relative preferences and therefore well-being is aside to the actual aims of economic analysis. Albeit one often poorly explained by economists, and therefore, commonly misunderstood by non-economists through no fault of their own. 

Blog post source – Sparrowly Group

Written by Morgan Thorn, Sparrowly Group