Are Predetermined Change Control Plans on the road to Global Harmonization?

In October 2021, FDA and MHRA (United Kingdom’s Medicines and Healthcare products Regulatory Agency) jointly developed 10 guiding principles for the development of Good Machine Learning Practice (GMLP) with the goal of promoting “safe, effective, and high-quality medical devices” that are based on Artificial Intelligence/Machine Learning (AI/ML) technologies.  Guiding Principle 10 focused on monitoring the performance of the models and managing re-training risks.  It stated:

Deployed models have the capability to be monitored in “real world” use with a focus on maintained or improved safety and performance.  Additionally, when models are periodically or continually trained after deployment, there are appropriate controls in place to manage risks of overfitting, unintended bias, or degradation of the model (for example, dataset drift) that may impact the safety and performance of the model as it is used by the Human-AI team.

Since that time, FDA issued a draft guidance for predetermined change control plans (PCCPs) for Artificial Intelligence/Machine Learning (AI/ML) software functions.  See our prior blog post on the topic here.  FDA announced that FDA, Health Canada, and MHRA are jointly publishing guiding principles for PCCPs for AI/ML devices to help stakeholders when developing solutions for these countries. Five guiding principles were identified for PCCPs and relate to being focused, risk-based, evidence-based, transparent, and taking into consideration total product lifecycle management.

Focused and Bounded

This guiding principle recommends that the PCCP describe a specific planned change that is consistent with the claimed intended use and intended purpose of the Machine Learning Medical Device (MLMD).  The plan should identify the methods for verifying and validating the change.  If the change does not meet specified performance criteria, it will not be implemented under the PCCP.

Risk-based

This guiding principle ensures risk management principles are used to evaluate the individual and cumulative changes over the life of the device.

Evidence-based

This guiding principle discusses the generation of evidence for the change.  Data should demonstrate the benefits of the change outweigh the risks and any risk identified are appropriately controlled and mitigated to ensure the device remains safety and effective.  The evidence used to measure the device’s performance should be scientifically and clinically justified, consistent with the level of risk for the proposed change.

Transparency

This guiding principle calls for manufacturers to be transparent with users regarding the device performance before and after the implementation of the change.  Considerations include transparency regarding the data used to develop the change, comprehensive testing of the change, characterizing the performance of the device before and after the change, and plans in place for ongoing monitoring of device performance and communication of any unexpected changes in performance.

Total Product Lifecycle (TPLC)

This guiding principle reminds manufacturers that PCCPs and MLMD changes should be integrated into the lifecycle management of the device and be part of their existing quality system processes including but not limited to risk management and post market monitoring.

FDA considers these guiding principles as complimentary to their recent efforts around PCCPs including their proposed draft guidance on PCCPs. However, FDA’s draft guidance included more requirements around data management practices, re-training practices, and update procedures that are not included in these guiding principles.  Perhaps, that is why the Agency believes these guiding principles, although developed specifically for AI/ML devices, could apply to other devices when developing PCCPs.

A recent review of 510(k) summaries for AI/ML devices show very few have taken advantage of including a PCCPs as companies struggle to apply the recommendations in the draft guidance in a practical manner within their already established design change processes.  For those that have referenced a PCCPs, the 510(k) summaries do include a list of the specific modifications however the explanation of the PCCP itself is vague and high level, simply stating that the modifications will be controlled and implemented in a manner that assures the device is safety and effective, the modification will be analyzed against acceptance criteria which will demonstrate substantial equivalence, and labeling will be provided to the end users to inform them of the changes and characterize the performance. The 510(k) summaries also include details on the planned modification protocols, including an impact assessment to address requirements for data management.  It will be interesting to see how these principals are combined with FDA’s draft guidance in practice and whether this will accelerate the use of PCCPs or increase confusion for industry.