Shhh! It’s a Secret! FDA is Not Providing Key Details in the EUA Templates

We have previously posted about the heartache and anxiety that is the EUA process (here).  Companies are waiting months for feedback from FDA and are frequently given comically short timelines for response.

Another layer in this onion of poor management is the growing gap between the requirements and expectations that FDA has released publicly to gain authorization and the actual study requirements to support an EUA.

CDRH, helpfully, has created a series of detailed EUA templates for the interactive review process.  Throughout 2020, FDA periodically issued new or revised EUA templates as the pandemic progressed.

Has this been a perfect system? No.  There are still many cases where changes to FDA requirements were applied before new templates were issued and companies with EUAs under review took these changes in the teeth, delaying their authorization while new data was collected.

Sadly, the pace of versioning existing templates has either slowed or stopped for IVD products.  What we are left with now are hard and fast requirements that are barriers to an EUA authorization, that are not disclosed for public consumption.  This gap can leave companies blindsided when their reviewer emails them requesting new data.  In these communications FDA will claim that these requirements have been in place for months, but they are nowhere to be found in any publicly available EUA template or announcement.

Of particular importance:  there are several surprise requirements are all linked to a single issue for PCR and Antigen tests.  This requirement is to ensure that the clinical agreement study is populated with a sufficient number ‘low positive’ samples.

This single concern touches upon many design considerations for a clinical agreement study.

Here are key points to consider:

1. What does it mean for a sample to be ‘low positive’ in the clinical agreement study?

FDA does not have a definitions section in the templates, but the agency does provide some insight as to what ‘low positive’ means in the Molecular Diagnostic Template for Commercial Manufacturers (updated July 28, 2020) when describing the details of a clinical agreement study to support Point-of-Care (Page 34).

You should conduct testing with samples prepared in clinical matrix with SARS-CoV-2 viral load near the LoD of your assay. The testing should be performed by inexperienced users at the clinical sites. The test samples should consist of 10 low positives (<2x LoD) and 10 negatives (matrix) per site.

In reading the above, a reasonable person might assume that when FDA is discussing LoD in this context they are referring to the LoD of the candidate device, as directly stated in the first sentence meaning that the description of ‘low positive’ is <2x LoD of the candidate device.  We have found, however, that a study designed in this fashion would not be considered adequate to support an EUA.  If you are attempting to validate a PCR device for CLIA Labs or Point-of-Care, the above statement is all FDA provides in your template.

To try and get more context for ‘low positive’ one would have to search the templates that were drafted for other indications or device technologies.  For instance, FDA has a slightly different description of low positive in the Template for Manufacturers of Molecular and Antigen Diagnostic COVID-19 Tests for Non-Laboratory Use (July 29, 2020). Low positive is only referred to in this template when describing the Flex Studies, but it largely consistent with the one found in the manufacturers template:  “Flex studies should be conducted by testing a negative sample and a low positive sample (at 1.5x – 2x LoD) for each condition being evaluated.”

FDA, again, has a somewhat different description of low positive in the Molecular Diagnostic Template for Laboratories (updated July 28, 2020) when describing the clinical data required for respiratory panels that implies the description of ‘low positive’ is based on the LoD of the candidate device:

The pre-selection of archived positive samples should represent a range of viral load or Ct values including low positive samples near the assay cut-off.

Finally, FDA has a fourth description of low positive that can be found in the Antigen Template for Test Developers (October 26, 2020)

[L]ow positives (i.e., RT-PCR Ct counts >30)

While it may not seem appropriate to scour an EUA template for an antigen test to gain insight into FDA’s thinking for a PCR test, you will see later in this post that this type of exercise can be informative.

If you are attempting to validate a PCR assay you may be confused as to which of these descriptions of low positive is most appropriate for your test.  The answer, none of them.

Rather, FDA is providing the true or current requirements for low positives during conversations with individual sponsors.  Recently, in the context of PCR assays, FDA has been defining ‘low positives’ as specimens having Ct results obtained with the comparator that are ≤3Ct of the mean Ct for the LoD of the comparator. This description is different than the all of the versions found in the EUA templates.  This is an important difference, because it could mean that your study does not have the correct distribution of sample titers if you did not know about this informally imposed requirement.

2.  Ct values for all positive specimens in the Clinical Agreement study are required

In the Molecular Diagnostic Template for Commercial Manufacturers (updated July 28, 2020) FDA outline the expectations for the positive samples on page 15:

The use of samples previously tested positive by another EUA RT-PCR assay may be acceptable without additional comparator testing. You should indicate the source of the samples, provide results for each tested sample, indicate specimen type, and initial test date.

If you follow the recommendations provided in the template you will likely, at minimum, get follow-up questions, but very possibly a request for new data.  In order to fulfill FDA’s request regarding low positives, you are required to report the Ct values for all positive samples in your study.

While this seems minor, many companies did not ensure that the Ct value was recorded for the positive sample they were purchasing. Most companies purchasing samples confirmed the PCR assay used was EUA authorized. This remediation may not be an issue for a large company as they would likely has direct access to a lab with a comparator device, but smaller companies were relying on the ability use these samples without the need to test them on any other device beyond their own.  In many cases this has forced smaller companies or universities to purchase additional samples and re-run the experiment.

FDA officials have indicated in conversations that reporting of Ct values for clinical agreement is a long-standing request for EUAs, but we have not been able to independently confirm this assertion.

When purchasing samples for a retrospective clinical agreement study you should be able to report the following for samples tests with a high-sensitivity PCR assay: source of the samples, results for each tested sample with Ct values, specimen type, and initial test date

3.  Make sure that your PCR comparator is a high-sensitivity test

The three main diagnostic technologies for the pandemic are:  PCR, Antigen, and Antibody tests.  In each of the templates for these devices you will find a version of the following statement:

Positive percent agreement should be calculated in comparison to an EUA RT-PCR test. We recommend using only a high sensitivity EUA RT-PCR assay which uses a chemical lysis step followed by solid phase extraction of nucleic acid (e.g., silica bead extraction). If available, FDA recommends selecting a comparator assay that has established high sensitivity with an internationally recognized standard or FDA SARS-CoV-2 Reference Panel.

For nearly a year now, we at HPM have been trying to find out what ‘high sensitivity’ meant in this context.  In conversations with FDA, we have been told to propose an assay as the comparator and FDA would then comment.

We recently were finally provided with an answer to this ‘high sensitivity’ question.  To be considered a ‘high sensitivity’ test, we now understand that the assay must have an LoD that is at or lower than 18,000 NDU/mL according to FDA’s Reference Panel.  Setting the threshold to 18,000 NDU/mL effective eliminates at least 17 PCR assays from being used as a comparator for other tests.  We expect the true number to be higher as the list containing reference panel results has not been updated since October 2020.  We were not provided with an explanation as to why 18,000 NDU/mL was an appropriate cutoff, but we will take whatever clarity we can get from the review staff.

This unpublicized requirement disproportionally impacts smaller companies. A larger company has access to more resources and can more easily shoulder the burden of retesting samples or purchasing new ones that were tested with an appropriate comparator, but it can be fatal for smaller companies without these resources.

If you did not pick a test that is high-sensitivity as FDA defines it, it is not your fault, as FDA has not previously provided guidance as to which of the authorized tests can be used as a comparator. The adverse consequences of selecting the wrong assay are real.  Companies run the risk of their data being considered unsuitable to support an EUA.

4.  What does it mean to have a sufficient number of low positive samples

In communications with FDA, the agency has insisted that the clinical agreement study consist of not only a range of viral loads and Ct values as stated in the template but, at minimum, 20% low positive specimens.

The requirement for at minimum 20% low positive samples is not found in the following templates:

• Molecular Diagnostic Template for Commercial Manufacturers(updated July 28, 2020)
• Molecular Diagnostic Template for Laboratories(updated July 28, 2020)
• Template for Manufacturers of Molecular and Antigen Diagnostic COVID-19 Tests for Non-Laboratory Use(July 29, 2020)

In FDA’s templates for PCR assays one can find only the following recommendation when FDA describes the Clinical Evaluation study:

Specimens representing a wide range of viral load including low positive samples should be tested.

We refer to this request for 20% ‘low positives’ as a requirement since FDA has flagged studies as insufficient and requested additional data if a retrospective clinical study does not have at least 20% low positives samples.

One can find only a single reference to a specific amount of low positives in the Antigen Template for Test Developers (October 26, 2020):

Retrospective specimens should be reflective of the natural distribution of SARS-CoV-2 viral loads, and approximately 10-20% of the clinical specimens should be low positives (i.e., RT-PCR Ct counts >30), as has been observed in other sequentially enrolled clinical studies.

As you can see from the excerpt above, the only specific guidance can is found in a template for a different assay technology where the requested amount of low positives is as little as 10%.  If you do follow this guidance and present a study with 10-20% low positives, your study will likely be deemed inadequate.

5.  Risks of picking a too‑sensitive comparator assay

The gut reaction in response to the above may to be over-correct and pick the absolute highest sensitivity device possible.  As stated in section 1 above, low positive is defined by the comparator assay. If you pick a comparator that is more sensitive than your test, you will not only increase the likelihood of discordance, but you will also narrow the window of what it means for a sample to be ‘low positive’. You then run the risk of not meeting the minimum requirement to have 20% of the samples in that range.  It is recommended that you choose a comparator with a similar LoD to your assay that is also still considered ‘high-sensitivity’ per FDA’s Reference Panel.

Based on the foregoing, we have the following recommendations to FDA to improve the EUA process:

1. Issue a statement defining ‘high-sensitivity’ and ‘low positives’ as this lapse is causing unnecessary delay in bringing new products to market and is creating needless inefficiency.
2. Issue updated templates with study design requirements that are consistent with FDA’s expectations for EUA authorization. Some of the templates have not been versioned in 9-11 months.
3. Commit to updating the EUA templates on a regular basis for the duration of the emergency to prevent this problem from recurring.
4. Commit to allowing pending EUAs to resolve issues via new testing as a post‑authorization requirement rather than applying unpublicized requirements retroactively.
5. Update the SARS-CoV-2 Reference Panel Comparative Data Table, as it is 6 months out of date and data from new assays or updated data from existing assays are almost certainly sitting in the shadows.

We strongly urge all developers of PCR assays that are about to submit an EUA or those that have an EUA pending to review the details of your clinical agreement study to make sure that your data will not get flagged by the reviewer.  We are finding that gap analyses are more important than ever to reduce review delays for EUA products.