ELI Primary Menu

Skip to main content

Measuring What is Meaningful: INECE Convenes Discussion Series on Performance Measurement for Environmental Compliance and Enforcement

Thursday, July 13, 2017
Emmett McKinney

Emmett McKinney

Research Associate

In 2009, the New York Times Magazine published a story about the professional basketball player Shane Battier, dubbed the “No Stats All Star.” In terms of the sport’s most often cited statistics—points, rebounds, assists, steals—Battier was no standout. Nonetheless, each successive team that Battier joined won significantly more games the following season.

This is the ultimate goal any basketball team—to win games. To many analysts, this is what makes Shane Battier’s dual tendencies to produce middle-of-the-pack statistics and help his team win so perplexing.  However, as observed by Daryl Morey, the general manager of the Houston Rockets,  “For most of its history basketball has measured not so much what is important as what is easy to measure […] and these measurements have warped perceptions of the game.”

This question—how best to measure the impact of a particular action or actor—is fundamental not only to managing professional basketball teams, but also to environmental compliance and enforcement. Agencies charged with implementing environmental laws must constantly assess whether their compliance and enforcement actions (such as facility inspections) are producing the desired outcomes (improved environmental quality).

ELI is proud to host the Secretariat for the International Network for Environmental Compliance and Enforcement.

As Secretariat of the International Network for Environmental Compliance and Enforcement, ELI hosted a series of five webinar-based discussions from January to March, 2017 to address the topic of performance measurement for environmental compliance and enforcement.

The goals of the series, organized in partnership with Environment and Climate Change Canada and the U.S. Environmental Protection Agency, were to share and identify best practices regarding performance measurement, and to highlight examples of how innovative legal approaches and advanced technologies can facilitate these assessments. A white paper summarizing key insights from the discussions as well as examples of innovative approaches to performance measurement, was published on July 5.

The first session, “What to Measure?”, focused on fundamental questions related to performance measurement: what are environmental agencies' fundamental objectives, and which indicators should be used to measure progress towards these goals?

In the words of Michael Enns of Environment and Climate Change Canada, “At best, any indicator tells an incomplete story. At worst, it is misleading.” Many agencies track their inspections and investigations of regulated parties, the behavior uncovered by those actions, and the resulting enforcement measures and penalties. However, such data does not always reveal the full impact of agencies’ compliance and enforcement efforts – for example, the extent to which a penalty handed down to one regulated entity may deter future violations by another.

Likewise, using a given indicator in isolation may create perverse incentives for law enforcement officials. Gauging success strictly by the number of inspections carried out, for example, may incentivize officials to focus on facilities that are convenient to inspect while overlooking more serious offenders that take more time to investigate and prosecute. This could lead an agency to inspect every dry cleaner within a 10 mile-radius, while overlooking a major industrial facility located further afield.

The second session, “How to Measure,” highlighted methods for conducting performance measurement. Professor Jay Shimshack of the University of Virginia discussed best practices in using technology for measurement and reporting, as well as statistical and scientific approaches to data analysis. Randomized control trials and analyses of existing administrative data, for example, can help officials compare the impacts of different enforcement strategies.

The third session explored how qualitative and quantitative data can be integrated throughout the performance measurement process to help officials understand not just the frequency or severity of violations, but also the underlying causes, public perceptions, and political contexts that impact environmental compliance. Interviews and survey data can be especially helpful when quantitative techniques are cost-prohibitive, produce political resistance, or produce unexplainable results.

Water Advanced Monitoring – Real-Time Monitoring with Telemetry: Cyanobacteria in the Charles River, MA (U.S. EPA)

In the fourth session, Ed Messina and Jon Silberman of the U.S. EPA highlighted how advanced technologies, such as continuous emissions data monitoring, fenceline monitoring, remote sensing, and infrared cameras can help regulators better assess outcomes in the field. This more precise data clarifies the sources of environmental harms and allows officials to identify clear, actionable opportunities for remediation. Messina and Silberman also discussed approaches for gauging the impact of new technologies, for example by sharing real-time compliance and enforcement data on platforms like U.S. EPA’s Enforcement and Compliance History Online tool.

Altogether, performance measurement is about telling a story of impact through data. Used in isolation, however, conventional indicators only tell part of the story about the impact of compliance and enforcement actions. Defining the problem clearly— and understanding the context for compliance and enforcement actions— is critical to painting a fuller picture.

New technologies and legal approaches are creating opportunities for agencies to improve their performance measurement through continuous experimenting, learning, and adapting. Ultimately, this helps environmental agencies achieve the most important outcome – a cleaner, safer environment.

ELI is proud to serve as the Secretariat for the International Network for Environmental Compliance and Enforcement. More information and resources from the Performance Measurement Discussion Series are available at the INECE website, here