How do you measure the intangible?

Adi Forman, Charmaine Rodrigues, Leah Henderson

The challenge for L&D to demonstrate all of the impact that they have through their initiatives is a “permeating conversation” for Adi Forman, Professor of Practice at Hult International Business School. Within that, there’s business pressure to show evidence of impact – and to show it quickly.  

“The organization wants demonstrable return on investment and that's a very immediate need,” says Forman. “But these are slow, percolating capacities, capabilities, and mindset shifts that are not an on/off switch.” Impact may become visible weeks, months, or years after an intervention. 

Those outside of L&D may not realize this complexity. Unlike the binary world that functions like Finance operate in, impact of learning comes with a lot of nuances. Not only can it be slow to materialize, but it can manifest in unexpected, unintended ways.  

And then of course, there are intangible aspects. “Impact at a human level will translate itself into many things that are intangible, like compassion or hope,” says Forman. “These are part of the story. How do you make that tangible?” 

Charmaine Rodrigues is Global Head of Learning, Leadership and Talent Development at Grundfos, a global organization specializing in pump technology. For L&D leaders like Rodrigues, she says, “being able to validate what you say with data sends a powerful message.”  

Many leaders are simply more comfortable with clear data because it’s easier for them to know what to do with it. “Saying, ‘5 out of 10 people have used LinkedIn learning,’ or, ‘The programs show an NPS of 8.5’ is much more understandable than a conversation on why a learning culture is essential to business growth.” 

Grundfos runs a survey asking people how they feel about the company as a learning organization, which is then measured against a global benchmark. While based on perception, the data provides indicators such as how supportive leadership is. “That has lent a level of richness to the dialogue that takes us beyond event-based measures,” says Rodrigues.  

It allows her team to show indicators of impact. “If the learning program is tagged to retention and mobility, what has been the retention and mobility of that group? Over time, we’ll be able to show if it’s higher – while staying mindful that the program is not the only contributor.” 

This is important, because as Forman explains, “there is a danger of the misconception that data demonstrates causality.” The use of surveys and data visualization can perpetuate the idea that it’s a perfect science. “We're trying to get a picture by pixelating, but we don't know that it's a whole picture. It opens up a conversation about the impact. It doesn't tell you the impact.” 

For example, a 360 evaluation allows you to capture behavioral shifts. “If these behaviors are linked to corporate values, by default we must be shifting ROI. Then, you can link it to other KPIs like revenue,” explains Forman. “It’s helpful – but it's still indirect and there are many other components that influence revenue.” 

As Rodrigues says, “we don't necessarily have to put effort into isolating the impact, as long as we're able to indicate it has been one of the factors that has led to improvement – as opposed to saying, ‘the learning has had X percentage of impact on trade’.” The key is viewing the data points as indicators.  

Forman says that impact conversations need to begin with clearer definitions around different levels of impact. This was discussed in our webinar Let’s Be Honest About Impact.

1. Individual impact: Has the individual changed behaviors, skills, capabilities or mindset?  

2. Group impact: Have they shifted how they are with other groups or the impact they have on other groups?  

3. Organizational impact: Has the business been impacted?   

4. Societal impact: Has there been an impact on the wider environment? 

With this scope in mind, Forman says she would be dubious of any off-the-shelf impact measurement tool claiming to have all the answers. “How can it? Impact is beyond the boundaries of the system.”  

And, she points out, there are stories to tell around that. “Stories are so important as part of any impact evaluation.” And not just from participants themselves – it could be managers, clients, or stakeholders. 

And there's another angle: “If you really want to understand the intangible, evaluate the other key component in the room.” She’s talking about the facilitators. “We see change in participants, we witness them and their experience, so, we hold data.” Data that may not get picked up on – heat moments that are pivotal in people’s learning. 

Forman calls for more experimentation in measuring intangibles. By way of example, she described an exercise from a program that was supporting a cultural transformation. Participants were asked to draw pictures of how they were experiencing change in that moment. Several months later, the exercise was repeated at a different organizational level to discover how experiences of change had evolved. “It demonstrates how transformation might be happening and how the culture is beginning to shift.” 

Given its complexity, “there is a significant amount of effort from the research perspective to make the connection from learning to behavior change to performance, and then measure it over time to make a valid judgement,” says Rodrigues. “For L&D leaders, it's about being thoughtful around the critical few learning interventions where we go the whole nine yards.”

There’s also no getting around the fact that traditional impact measurement can be onerous on the participants. Forman asks, is it always necessary? “When the focus is more at the individual level than on the business, do you need to do an impact evaluation? Perhaps not beyond the standard evaluation.”

She also thinks we need to pay more attention to the individual's interests. Impact assessments tend to be ‘add-ons’ at the end, which participants complete only to never see them again. “But imagine that five or six peers from a cohort are brought back together for a 2-hour focus group that asks them to tell stories about what they’ve experimented with, or to explore certain concepts with the benefit of time and experience.”

That dialogic process with peers opens up opportunities for reflection. “Suddenly, that conversation is a learning process. And you, as a researcher, take the data. It's not a quick approach, but it gives something back to the participants and supports the continuous process of learning.”

If all the complexity could be stripped back, Forman shares what her ideal approach would be: “I would ask them the simplest of questions. If you look back now, six months or one year down the line, what do you remember? Maybe it’s a conversation, or how you felt, or a model. What do you remember, and why is that interesting to you now? Two questions – as simple as that.”

“Ultimately,” she concludes, “it's a puzzle that is incredibly complex and may not have one solution. And that, I think, makes it a very exciting conversation with the right partners.”