I spent the last month of nights and weekends modeling how the Bipartisan American Affordability and Jobs Act (BAAJA) might affect United States energy systems. BAAJA is a draft bill that would reshape federal oversight of infrastructure, especially energy infrastructure, in profound and uncertain ways. BAAJA’s sponsors in the Senate released the first public draft on September 30. A Senate cloture vote is scheduled for November 9, just 40 days later.
I’m used to scientific publishing in peer-reviewed journals, so releasing a white paper that had not undergone formal peer review felt uncomfortable. However, in 16 years of scientific publishing, I’ve never seen peer review take less than 40 days. There just wasn’t time before the Senate vote.
I tried to hold myself to the highest standards of transparency anyway. I did my best to clearly explain the model structure and input data, to enumerate and discuss my assumptions, to highlight sources of uncertainty and analyze how they influence model outputs. I made my code open source so other modelers can reproduce or falsify my results. I strove to create a parsimonious model, one that captures the core dynamics without too many equations or parameters, and to clearly structure and comment the code. I hope this helps people audit the model and, if they want to, adjust the model inputs or structure and see how outputs respond. I disclosed my funding sources (none) and relevant financial entanglements (none).
This transparency has exposed me to criticism from BAAJA proponents, but also to constructive feedback offered in good faith. Through calls, emails, and social media, a variety of scientists, energy professionals, journalists, activists, and enthusiasts have influenced how I think and write about BAAJA. I’ve done my best to respond openly to their feedback and update the white paper as needed, maintaining a public version archive. This process resembles a kind of open, crowd-sourced peer review, minus the journal editor acting as arbitrator. The experience has been interesting, to put it mildly.
I was surprised to learn that other groups who modeled BAAJA did not hold themselves to the same standards of transparency. Since September 30, four think tanks that I know of have released BAAJA modeling studies [1, 2, 3, 4]. With all of them, I found it difficult or impossible to answer even basic questions, such as “Did the modelers assume it is impossible for BAAJA to increase electricity demand, for example, from data centers?” As far as I know, none of the think tanks have made their code open source. I hope they will.
If modeling is going to influence policymakers’ votes on something as complex and potentially influential as BAAJA, then the models had better be trustworthy.
The stakes, after all, are not low. I estimate BAAJA’s potential annual impacts, in the energy sector alone, in the tens of billions of dollars of revenue, tens of thousands of jobs, hundreds of millions of tonnes of greenhouse gas emissions, and a thousand or more lost lives attributable to air pollution from fossil fuels.
Transparency is not a sufficient condition for trustworthiness—transparent models can still be wrong—but I believe it is a necessary condition.