Keeping it Real-World: The importance of RWD and RWE for AIaMD.

RWD and RWE are crucial to the proper assessment and development of AI medical devices

There are many acronyms we hear often in the healthcare regulatory space. RWD (Real-World data) and RWE (Real-World evidence) are at the top of the list, and they deserve to be there. They are hugely valuable when it comes to the proper development and post-market follow-up of AI as a medical device (AIaMD).

What are they?

RWD and RWE are closely related terms, sometimes used interchangeably.

RWD (Real-World data) - refers to the clinical data generated in real-world settings, reflecting the natural evolution of patients undergoing medical procedures and treatments within the healthcare system.

There are two main approaches to collecting this data:

• Primary data collection: This involves gathering new data specifically for a study or analysis. It includes, for example, patient data collection by clinical research personnel from the Electronic Health Records (EHR) of a healthcare provider, or surveys to collect Patient-Reported Outcomes (PRO). It is usually more tailored when it comes to answering a research question, but often more resource intensive.

• Secondary data collection: This uses existing data sources like patient registries or claims databases, and can be an efficient way of leveraging existing information.

RWE (Real-World evidence) - refers to the evidence generated using RWD by applying statistical and analytical methods to provide meaningful insights for clinical and regulatory decision-making. RWE bridges the gap between raw data and actionable knowledge.

RWE vs clinical trials: What are the differences?

Clinical trials

Clinical trials are widely regarded as the gold standard for the clinical evaluation of medical interventions, including drugs and medical devices. These trials are prospective studies where patients are recruited and monitored in a controlled environment, often with the inclusion of a control group.

This controlled setting allows researchers to establish a clear cause-and-effect relationship between the intervention and patient outcomes. The rigour of clinical trials means they minimise biases, ensure accurate data collection, and provide systematic follow-up of patients, which collectively contribute to the generation of strong, reliable evidence.

They are costly to implement, and may take years to yield final results. Their constrained environment and selective patient criteria often mean that the results may not fully represent real-world practices and patient populations, which is where RWE studies come into play.

RWE

In contrast to the controlled environment of clinical trials, RWE studies offer a valuable perspective by utilising the data gathered from routine clinical practice. RWE reflects the actual experiences of diverse patient populations across various healthcare settings, making it more representative of real-world conditions.

This broader applicability allows RWE to provide insights into the long-term safety, effectiveness, and performance of medical interventions, including AI-powered devices, in everyday use.

Although RWE can be influenced by the biases and missing data inherent to observational settings, its ability to capture real-life outcomes and its growing acceptance by regulatory bodies make it a very valuable tool for post-market surveillance, and the continuous improvement of medical devices.

Ongoing RWE can ensure devices work safely and fairly

Integration of RWE into the clinical evaluation framework of an AI medical device can be transformative when it comes to fulfilling regulatory requirements for continuous clinical evaluation, as well as addressing challenges specific to AI device deployment.

As part of their Post-Market Clinical Follow up (PMCF), manufacturers of CE-marked medical devices (as per EU MDR) “shall proactively collect and evaluate clinical data from the use in or on humans of a device which bears CE marking [...] with the aim of confirming the safety and performance throughout the expected lifetime of the device”.

RWE studies can play a crucial role in providing ongoing, real-world insights into how a device performs across a diverse patient population and in various healthcare settings to ensure it is safe, effective, and fair.

Fairness here is described as ensuring that an individual’s characteristics, including race, gender, age, or other factors/attributes, do not prevent or affect their ability to receive high-quality healthcare from an AI medical device. This is an increasingly important consideration for the regulation of AIaMD, and is notably one of the five key principles for the regulatory use of AI defined by the UK’s Medicines and Healthcare products Regulatory Agency.

RWE can power the development of AI systems

RWE can also be instrumental in detecting and managing dataset shift and model drift — two critical issues in the continuous development of AI systems.

Dataset shift

Dataset shift occurs when the statistical properties of the input data change over time and are different from the data used for training, potentially leading to decreased performance of the AI model.

An example of this could be the deployment of a computer vision model that diagnoses skin diseases in ethnically diverse areas if the model has been trained only on patients with fair skin. This difference in the input data distribution can lead to reduced accuracy or bias in the model's predictions or classifications, as the model may not be as effective in recognising or interpreting the features present in the more diverse population.

RWE studies can help detect the performance issues and collect patient data that can be used to retrain the model.

Model drift

Similarly, model drift occurs when the relationships between the input features and the predicted outcomes evolve due to changes in clinical practices, treatment protocols, or patient behaviours.

For instance, consider an AI model developed to predict the likelihood of a patient being readmitted to hospital within 30 days after discharge. Over time, new treatment protocols and guidelines are introduced, such as improved post-discharge follow-up procedures or updated medication protocols. These changes effectively reduce readmission rates, but the model, which was trained on data from before these changes, is not aware of the new interventions. As a result, the model's predictions become less accurate, as it continues to operate based on outdated relationships between patient data and readmission likelihood. To maintain accuracy, it is crucial to continuously monitor the model's performance and update it to reflect current practices.

By leveraging RWE, manufacturers can continuously monitor the model's performance in the real world, detect any drift, and update the model accordingly to ensure it remains aligned with current practices and maintains its accuracy.

Dynamic benefits

Whilst clinical trials are a well-established and high-quality way to perform the clinical evaluation of a medical device, RWE also provides clear benefits, particularly when it comes to the dynamic world of AIaMD.

While RWE presents real dangers of bias, it can also offer crucial insights into the actual experiences of diverse patient populations in real-world conditions - a rich source of data that can be used to quickly iterate and improve medical devices and the outcomes they offer patients. It is particularly valuable when it comes to post-market surveillance and continuous improvement, and should be considered an integral part of the development and regulation of AI medical devices.