The Role of Statistics in Health Policymaking

Statistics play a vital role in health policymaking by providing the evidence needed to make informed decisions that impact public. Through the collection, analysis, and interpretation of data, statistics help policymakers to identify health trends, evaluate the effectiveness of interventions, and allocate resources appropriately. Reliable data and statistics support the development of policies that address urgent health concerns, improve healthcare quality, and enhance disease prevention efforts. By transforming complex health information into actionable insights, statistics ensure that policies are grounded in scientific evidence rather than assumptions. Ultimately, statistics contribute to building healthier communities and stronger healthcare systems. To gain further perspectives, we are thrilled to interview Dr. Elizabeth (Liz) Stuart from Johns Hopkins University.

Robert: Could you start off by telling us a little bit about yourself and your day-to-day work at Hopkins? What do you enjoy most about your job?

Liz: I’m a Professor and Chair of the Department of Biostatistics at Hopkins, with joint appointments in Mental Health and Health Policy and Management. As indicated by that combination, most of my work is in statistical methods for mental health and substance use research, and a lot of policy evaluation across a variety of areas, including gun violence, abortion, and substance use. I am also the evidence lead for the Bloomberg American Health Initiative, which has a focus on improving the health of individuals in the US through close connection with public health practice, and a co-founder of the Center for Mental Health Addiction Policy with Colleen Barry. Of course, now in my Chair role I spend a good chunk of my time on administrative work, so my overall time is maybe 40% research (combination of methodological work and collaborative applications), 40% administration, and 20% teaching. I would say what I love is the sense of mission at the School – it's a place full of smart and collegial people looking to do good science to make a difference in the world. That ethos leads to a wonderful work environment, where people work hard and expect high quality work but also treat each other with respect and kindness.

Robert: What opportunities does Johns Hopkins offer for students or others who are interested in health policy?

Liz: It's a great place for health policy and (bio)statistics! Our Department of Health Policy and Management has over 100 faculty and Biostatistics has about 45, with interesting work across a whole range of methodological and applied areas. There are also a number of interesting health policy relevant centers, including the Center for Mental Health and Addiction Policy, the Bloomberg American Health Initiative, and the Center for Gun Violence Solutions. Hopkins also recently built a new building near Capitol Hill in Washington, DC, the Bloomberg Center (in what used to be the Newseum building), with great opportunities for engagement with government, policymakers, and other relevant groups. Being close to DC is also generally useful in terms of connections to federal staff and policymakers, and with many convenings and meetings happening there, whether at NIH, the National Academies, FDA, or elsewhere. There is also just a huge amount of research going on in related areas and so many interesting studies and methodological questions that haven't been answered. It's hard to imagine a better place for work at the intersection of health policy and (bio)statistics than Hopkins!

Robert: How would you describe the role of statistics in shaping and evaluating health policies?

Liz: There are lots of ways statistics can be used in health policy. Some of my first work at Mathematica (before I moved to Hopkins) was microsimulation modeling to understand the coverage and financial implications of different policy choices on things like SNAP and other eligibility-based programs. In other cases, statisticians might be helping design and analyze surveys to understand how policies are being implemented, or public sentiment regarding policies. They also might be helping design and analyze randomized trials of new innovations or non-experimental studies of existing programs (e.g., the Centers for Medicare & Medicaid Services run some demonstration projects that have complex statistical needs), while some support research studies that directly inform policy.

Robert: From your experience, can you give me an example where statistical analyses had a significant influence on a health policy decision?

Liz: I think it’s often hard to see a direct link between a given study and policy decisions, and especially given how hard policy evaluation is, in some ways I would hope that it's a body of evidence that would inform policy rather than one individual study. (And as we know, there are also many other factors that go into policymaking!). The most explicit example that I have is a case where a specific gun violence study I worked on (led by my post-doc at the time, Kara Rudolph, and done in close collaboration with my colleague Daniel Webster who is an expert in gun violence) was quoted by President Obama in an executive order about gun control; we had found that Connecticut implementing a permit-to-purchase law reduced firearm homicides by 40%, and that was quoted as part of the background and motivation for the proposed policies! However, given broader challenges with federal firearm regulations, I'm not sure much actually came of it, but we were at least directly informing the conversation.

Robert: What are some effective strategies to communicate complex statistical results to non-expert stakeholders, such as policymakers and the general public?

Liz: Great question! I think about this a lot. I think one piece is transparency of methods and results, and potentially of using methods that we trust and that are easy to explain rather than using the newest, fanciest approach (assuming we trust the findings using the more basic approach). This may be controversial to say, but sometimes what is valued by the academic statistical community is not what is needed for clear communication with decisionmakers or the general public, and I would love to see more attention to things like transparency of methods and clear diagnostics. The other note I would say on this is that I also lean very heavily on my substantive expert collaborators. I am not an expert in abortion, or gun violence, or opioids; I have learned about these topics from deep and long-term collaborations, but I view them as the people with the broader lens on the content area and really the leads in terms of communication of the big picture findings and implications.

Robert: From your perspective, what are some of the most common and reliable sources of health data for policymakers today?

Liz: This is a tough one! Normally, I would say sources like the large federal data sources, whether it's Medicare or Medicaid claims, CDC surveillance data, or rigorous and nationally representative surveys such as the National Health Interview Survey. However, with budget challenges and other threats to these programs, it's hard to know whether we’ll be able to continue using these data sources reliably. There are other useful surveys, such as those run by Pew or NORC, but it is useful to have the federal surveys as true population benchmarks, and also to ensure access over the long-term. Other key data sources are private insurance claims (sometimes aggregated by various groups, although that data can be expensive to access), electronic health records (including from networks like OHDSI and PCORnet), or sources such as the AllOfUs resource. Each of them has its strengths and limitations, and one note I would say is that it is important to deeply understand any dataset you use – where it came from, what the variables mean, what the challenges are, etc. I have worked on many examples where naively using the data could have led to inappropriate conclusions, but working with people who really knew the data, we were able to make sense of it.

Robert: What ethical considerations arise when using data and statistical techniques to inform health policy decisions?

Liz: There are a lot of considerations here, including aspects such as attention to groups that may be underrepresented, or not represented at all, in data sources. In fact, a chunk of my methodological work has been on methods to understand and improve the representativeness (or lack thereof) of individuals in randomized trials. But here, I’ll focus mostly on transparency and not letting the finding drive the design and analysis. One thing I always appreciated at Mathematica, and now at Hopkins, was the underlying desire for true research – not wanting the results to come out one way or another but wanting to discover the "truth" (as much as we can in any given scientific study). And that's how I knew I wanted to be a statistician – that in a sense I care more about the methods used to answer a question than the answer itself. I think we need to have that type of approach – recognizing our implicit or underlying biases, but also trying to counter them, which can include setting up studies in ways that help minimize the potential for getting the answer we might expect (or hope). This is captured well in a quote from Donald Rubin that I really like (Rubin, 2001): “If we could try hundreds of designs and for each see the resultant answer, we could […] choose the design that generated the answer we wanted! The lack of availability of outcome data when designing experiments is a tremendous stimulus for ‘honesty’ in experiments, and can be in well-designed observational studies as well.” This fits into broader themes of pre-specification of analysis plans and associated topics. That said, and perhaps for another time, but I will also note that pre-specification of analysis plans is often more complex in non-experimental studies than in experiments, and that sometimes it is important to look at the data somewhat to understand which analyses are feasible or appropriate. I think this general topic needs more attention but for now, I will highlight a recent paper by Dylan Small, where he talks through some of these points.

Robert: How is the increasing surge of big data, machine learning, and artificial intelligence impacting the use of statistics in health policy today?

Liz: Another good question! There is clearly a lot of potential use of machine learning and AI in health policy statistics. For example, we are using large language models right now to pull out more information from the clinical notes in the Hopkins electronic health record system, allowing us to have better information on the symptoms of people with depression. This will then allow us to more easily combine information from randomized trials, which typically use symptom scores as outcomes, with data from the electronic health record system, which otherwise normally just has mental health outcomes such as services received. I have also used machine learning methods such as generalized boosted models to estimate propensity scores, as they are a particularly useful and flexible approach. The thing that worries me is that I think it will be easy for people to simply "throw" a big machine learning or AI model at a dataset and then have results come out. We will still need researchers who can think about things such as the quality of the data, who is in (and not in) the data, how are variables measured, what is the uncertainty in the result, etc. I really encourage all of us to find ways to contribute to the use of AI in research and the evaluation of AI tools in practice, where I think we have a lot to contribute!

Robert: What improvements would you like to see in the way that statistics are integrated into the health policymaking process?

Liz: This is tough to tackle, especially now when it feels like there is a lack of prioritization of data and evidence, but I am trying to stay true to my own values and trust that solid data and evidence does matter, and will make a difference. Figuring out what does – and doesn't – work is an incredibly important task. And I'll also pivot to a slightly different question and answer, which is that I hope that we can continue to foster truly interdisciplinary work, where statisticians, and statistics, are highly valued and seen as core members of research teams, working together with substantive experts. Some of the best studies I have worked on are mixed methods studies that have combined qualitative and quantitative work, and I think that type of work will continue to be extremely important (including sometimes, sadly, to help explain null findings). Here's one paper where we talked about that type of design, if you're curious. We as statisticians should continue to be engaged in these conversations and learn how to talk and work with people from a variety of fields. We have a lot to offer, with our focus on data, sampling, uncertainty assessment, and much more. Let's all work together to help make sure we keep our seat at the table in health policy discussions!