How should journalists define, measure and discuss impact? It’s a question I’ve spent much of the past several years thinking about, reading about and — most recently — writing about.

My first attempt to answer the question came in a journal article I co-authored, “What Can Nonprofit Journalists Actually do for Democracy?,” that examines how one prestigious nonprofit newsroom conceptualizes impact.

From the Journalism Studies abstract:

At a time when news organizations are struggling to define and promote their own relevance, some argue that journalists should make explicit their efforts to contribute to democracy as well as the results of those efforts. This study focuses on one prominent nonprofit news organization, the International Consortium of Investigative Journalists (ICIJ), that openly discusses the impact of its work in a way that may point to the beginnings of a new journalistic theory of democracy. We conduct a discourse analysis evaluating ICIJ’s own language about three of its high-profile investigative reports into global tax evasion, and discover four ways in which the organization makes reference to the impact of those projects. Ultimately, we argue that this focus on impact—encouraged, at least in part, by the organization’s foundation funders—is leading ICIJ to measure its democratic role in a way that sets its behavior apart from traditional journalistic entities, presenting an opportunity for scholars to discuss journalism’s evolving role in democracy.

Less than two months after publishing the study, ICIJ became a media darling after coordinating the massive Panama Papers investigation. Around that time, I wrote a piece in MediaShift about how ICIJ discusses impact and how it helps us understand the response to its latest series of reports.

In doing research for this study, I became interested in all the potential ways to measure impact other than the typical audience metrics such as page views and social sharing. Two that stuck out to me as incredibly important but rarely referenced: audience awareness and understanding.

That became the topic for another MediaShift article,  “Audience Awareness and Comprehension: The Seldom-Measured Metrics.” The piece, part of a section on the website called MetricShift, includes interviews with ProPublica, Center for Investigative Reporting, WNYC, Solutions Journalism Network, Engaging News Project, The Bill & Melinda Gates Foundation and other news outlets/foundations thinking about impact. From the article:

As more news organizations seek to explain complex topics in engaging ways online, there’s a growing need for data on the effectiveness of their efforts. In a white paper on measuring the impact of nonprofit journalism, ProPublica president Richard Tofel wrote that “Explanatory journalism seeks primarily to elucidate.… The impact of explanatory journalism will be determined by measuring how much readers’ awareness or understanding has increased.”

I returned to researching this topic with a study that examines how local news journalists define, measure, and discuss impact. The study, “Selecting metrics, reflecting norms,” is part of Digital Journalism‘s special edition on “Measurable journalism.”

From the abstract:

To thrive in the digital age, local news outlets must inform audiences and funders of civic-oriented journalism’s impact in their communities. There is little scholarly research on how journalists conceptualize and discuss impact. Literature commonly focuses on nonprofit news outlets with a national audience and an investigative bent – an important but small part of the news ecosystem. Through in-depth interviews, this study examines how journalists (n=20) from varied types of local news organizations (n=13) in one U.S. city with a diverse news environment define, measure, and discuss their work’s impact. Results show that journalists considered impact as central to their mission, had no qualms about discussing impact with colleagues, but were often hesitant to discuss it outside the newsroom. They most often measured audience traffic and social media traction but considered harder-to-track metrics such as audience awareness, public deliberation, and public policy as better indicators of impact. Findings shed light on how journalists view their mission, assess their civic contributions, and perceive profession norms. Ways to expand the definition of measurable journalism are explored.

This study considered a range of potential impact metrics (listed and defined above). I wrote about my findings — and the findings of other studies in this special journal edition on metrics — in an article in MediaShift.

My most recent study on impact examines how journalists affiliated with the Solutions Journalism Network assess the potential (and actual) impact of their work. The study, “No quick fix: How journalists assess the impact and define and boundaries of solutions journalism,” was presented at the AEJMC conference in Washington D.C.

From the abstract:

The U.S.-based Solutions Journalism Network (SJN) defines its mission as supporting and connecting journalists interested in “rigorous reporting on responses to social problems.” One problem facing journalists and researchers is the lack of a shared framework for discussing the impact of solutions journalism. This mixed-methods study addresses how journalists affiliated with SJN assess solutions journalism’s impact on individuals, communities, and institutions. A discourse analysis examines how impact is referenced in SJN’s official materials and comments from organization representatives. A survey of SJN’s members (n=50) examines how journalists evaluate and share their work’s impact. Findings shed light on how proponents and practitioners of solutions journalism view its objectives, measure its effects, and define its boundaries. Implications for the practice of solutions journalism and scholarship on journalism’s impact are discussed.  

The study was later published in Journalism Studies.