by Lixia Yao, PhD
At the DIA 2026 Annual Meeting, I sat through hundreds of slides. Most conveyed useful information. A few introduced new ideas. But one slide, tucked inside an FDA session titled "Integrating Artificial Intelligence and Real-World Evidence: FDA's Framework for Advancing Medical Product Development," stayed with me long after the conference ended.
It was not a particularly complicated slide. It did not present a novel methodology, a groundbreaking AI application, or a new regulatory pathway. Instead, it listed a series of considerations that should be evaluated when using real-world data (RWD) for regulatory decision-making: time periods, geographic regions, diagnoses, prognostic factors, treatments, follow-up periods, intercurrent events, outcomes, missing data, and others.
At first glance, it looked like another checklist. Yet the more I reflected on it, the more I realized that the slide was making a much deeper point about a question I have spent much of my career thinking about: What does it actually mean for RWD to be "high quality"? And what makes real-world evidence regulatory-grade?
For years, the biomedical informatics community has described data quality through familiar dimensions such as completeness, conformance, and plausibility. These concepts remain important and should not be dismissed. They provide a useful framework for evaluating whether data have been collected, standardized, and curated appropriately. At the same time, they share an implicit assumption: that quality is an intrinsic property of a dataset that can be assessed largely in isolation.
The FDA framework suggests a different way of thinking.
A dataset is not inherently high quality or low quality. Rather, its quality depends on whether it is capable of supporting answering a particular question. In the context of regulatory-grade real-world evidence, the central challenge is often not describing a population but comparing populations (groups). We are frequently trying to understand how patients observed in routine clinical practice relate to patients enrolled in a clinical trial, or how one treatment strategy compares with another outside the randomized setting. The quality of the data therefore depends not only on the accuracy of individual data elements, but also on whether the populations being compared are sufficiently similar to support a credible causal inference.
Viewed through this lens, the FDA's list of considerations takes on a different meaning. Each item represents a specific way in which comparability between populations can break down.
Patients treated during different time periods may appear similar on paper, yet receive care under fundamentally different standards of practice. New therapies become available, treatment guidelines evolve, diagnostic technologies improve, and supportive care advances. As a result, a patient treated in 2018 may not be directly comparable to a patient treated in 2025, even if both carry the same diagnosis.
Geographic variation introduces another layer of complexity. Where patients receive care influences who gets diagnosed, who receives treatment, how outcomes are measured, and whether patients remain observable within the healthcare system. Differences across countries, health systems, provider networks, and referral patterns can introduce important sources of confounding.
Diagnosis itself is often a surprisingly imperfect basis for comparison. Patients who share the same diagnosis code may differ substantially in disease severity, functional status, molecular characteristics, and other prognostic factors. Many of the variables that clinicians rely on intuitively are not routinely captured in administrative claims or electronic health record data. A valid diagnosis in a real-world study often requires far more than an ICD-10 or SNOMED code. When key prognostic factors are measured differently—or not measured at all—comparability becomes difficult to establish, regardless of how sophisticated the downstream statistical analysis may be.
The same principle applies to treatment exposure. In real-world practice, treatment is rarely a simple binary variable. Dose intensity, treatment duration, adherence, sequencing, concomitant therapies, and physician decision-making all influence outcomes. Two patients who appear to have received the same treatment may, in reality, have experienced very different therapeutic journeys.
Perhaps nowhere is the issue of comparability more consequential than in the definition of index dates and follow-up periods. One of the most common threats to validity in real-world studies is the misalignment of time zero. A poorly specified index date can introduce immortal-time bias and generate an apparent treatment benefit that exists only as an artifact of study design. In these situations, no amount of statistical adjustment can rescue the analysis because the underlying comparison was flawed from the outset.
The same concern applies to outcomes and missing data. Outcomes measured differently across data sources cannot simply be assumed to represent the same clinical construct. Likewise, when missingness occurs disproportionately across comparison groups, the resulting bias can materially alter study conclusions. These challenges are not merely data management issues; they strike at the heart of causal inference.
What I find particularly valuable about the FDA framework is that it highlights a misconception that continues to surface across the RWD ecosystem. Many organizations still approach data quality primarily as a data-engineering problem. Their efforts focus on data cleaning, standardization, de-duplication, quality-control procedures, and reducing missing values. These activities are necessary and often resource-intensive. Yet they address only part of the challenge.
Comparability cannot be engineered into existence after the fact. It is fundamentally a study-design problem. No amount of cleaning can recover a prognostic factor that was never collected. No statistical model can fully compensate for an index date that was defined incorrectly. No artificial intelligence algorithm can reconstruct information that simply does not exist within the underlying data source. This reality helps explain why, after applying inclusion and exclusion criteria designed to support a valid comparison, patient counts often decline dramatically.
This perspective also aligns closely with how regulators evaluate evidence. The FDA has consistently emphasized the concepts of relevance and reliability. Relevance concerns whether the data contain the information necessary to answer the scientific question, while reliability concerns whether the data are accurate and dependable enough for decision-making. The considerations presented in the DIA session can be viewed as a practical decomposition of these broader principles into the specific domains where real-world comparisons most frequently fail. More importantly, they reflect a regulatory focus on supporting credible causal inference rather than merely identifying statistical associations.
This was the most valuable lesson I took away from DIA 2026. Instead of asking whether a dataset is high quality, we should ask whether it is sufficiently comparable to answer the question at hand. The first question invites a generic assessment of the data itself. The second forces us to think carefully about study design, patient populations, sources of bias, and the decision that the evidence is ultimately intended to support.
Asking the right question is inherently more demanding, and it has direct implications for how organizations invest in data infrastructure, develop evidence-generation strategies, and assign accountability. Most biopharmaceutical companies recognize the strategic value of real-world data and real-world evidence. Far fewer invest systematically in the foundational infrastructure required to generate regulatory-grade evidence at scale. Ownership is often fragmented across clinical development, medical affairs, HEOR, data science, and IT, creating a situation in which everyone contributes but no one truly owns the problem.
The FDA slide concluded with a simple idea: stronger comparability leads to stronger evidence, which leads to better decisions and ultimately better patient outcomes. That progression captures the real purpose of our work. We do not generate real-world evidence simply to produce a hazard ratio, a p-value, or a publication. We generate evidence so that regulators, payers, clinicians, and patients can make consequential decisions with confidence. In that sense, the most important question is not whether a dataset is high quality, but whether it is comparable enough to support the decision we are asking it to inform.
Other Posts You Might Like
