Outliers in method validation and transfer
Two examples
Let's start with an example. During validation of an analytical method the linearity should be evaluated. Therefore 5 concentrations with 3 replicates each are examined (in order to evaluate linearity and trueness in one approach). At the examination level corresponding to 120% test concentration, one value is considerably higher than the other two ones. However, even taking this value into account, the acceptance criterion can be met. But the question is what to do. Is it necessary to repeat the entire 120% concentration with all three replicates, are we allowed to exclude the conspicuous one and calculate the mean by using just the two good ones, or do we have to repeat the whole linearity experiment?
Another example is a comparability study of a potency assay during a method transfer. For this, both the receiving site and the sending site determine the potency of a drug using a defined number of samples of exactly the same batch using a bioassay. In the bioassay used, the sample is applied in triplicates. For both sites, of course, the result should be within the specification and the relative standard deviation (RSD) across all measured samples from both sites should not exceed a predefined value. Unfortunately, one of the three replicates of one sample of the sending site deviates very much from the other two and averaged, the final result would be OOS (= out of specification). What is the correct procedure here?
For both examples, opinions differ on how to solve the problem.
The OOS Guideline of the FDA
The Guidance for Industry "Investigating Out-of-Specification (OOS) Test Results for Pharmaceutical Production" issued by the US Food and Drug Administration (FDA) in 2006 may provide some orientation, even if it is related to test results for batch release, and therefore no direct statements regarding applicability to method validation or transfers are made. Thus, after the discovery of an unexpected result the suspected sample should be kept, if possible, and a laboratory error analysis should be initiated. This has the purpose to identify the root cause. For this purpose, the laboratory manager and the analyst should sit together and check the analysis just performed in detail, preferably e.g. using a laboratory error analysis checklist. This includes the experimental procedure:
- have the correct reagents and standards been used
- have the dilutions and solutions been prepared correctly
- did something obviously go wrong (e.g. something spilled),
but also the raw data, all calculations, the question of whether the devices used were calibrated and qualified and whether the system suitability test (SST) was proper, to name just a few examples.
Furthermore, it makes sense to carry out a new analysis as soon as possible by a second analyst also experienced in the method using the sample kept (if possible) or another sample of the same batch. The purpose of this is to check whether the first analyst's handling error or a device error could have been the underlying root cause. In case of a definite laboratory error, the first result can be invalidated and the retest result used as final result.
The guideline also briefly discusses the use of outlier tests. Before application we should shortly think about the definition of an outlier according to this guideline: „On rare occasions, a value may be obtained that is markedly different from the others in a series obtained using a validated method. Such a value may qualify as a statistical outlier. An outlier may result from a deviation from prescribed test methods, or it may be the result of variability in the sample.” The use of an outlier test should be described in advance in the corresponding standard operating procedure (SOP) and, in case of high variability bioassays, may be appropriate to exclude a value from the calculation.
Let’s return to the examples...
This sounds applicable for the second example, as it is a bioassay with some variability in the results and which was - for sure - validated long time ago at the sending site. Since the problem occurred at the sending site and the method is used for product release, the conspicuous result should be considered as an OOS resulting in initiation of the laboratory error analysis discussed above. If, after thorough laboratory error analysis, the root cause couldn’t be identified and an outlier test application for this bioassay has been defined in a SOP, then an outlier test may be applied to check whether it is ok to exclude this strange result of one replicate from the calculation of the final result. To what extent an outlier test really makes sense when using only 3 values is another question… It's just an example. Of course, the exact outlier test to be applied should also be described in the SOP. Further information on outlier tests can be found in USP chapters <111> for bioassays and <1010> for general data analysis.
The first example can’t be considered as OOS, because the method is currently being validated thus no official specification can be available at this time. Official specifications take into account the analytical variability verified during method validation as well as process variability evaluated during production of some clinical batches. Nevertheless, it is highly recommended to initiate a laboratory error analysis to analyze the root cause. Even if it is not directly required for method validation - to my knowledge - it should be a matter of course for any conspicuous lab result to search for the underlying cause and wanting to understand the problem. In our example, it was discovered that there was a device issue at the end, which had to be fixed first resulting in a huge delay of the measurement of the last replicate compared to the other two previous ones. Due to the very small sample volume, liquid may have been evaporated, resulting in a much more concentrated result. Due to the clearly identified root cause, it is possible to exclude this one value from the calculation with proper justification while a much more conservative approach would be a repetition of the entire linearity experiment.
In case of an unclear root cause, the topic is a little bit more difficult. Simple and clear is the fact that there is no reason to exclude the outlier, as the result, even with the outlier, meets the acceptance criterion. It may not be so pretty, but an exclusion or a repetition just for the sake of "having beautiful results" is simply unacceptable from a regulatory point of view and tastes somehow like testing into compliance thus doesn’t seem to be trustworthy. An unidentifiable cause could indicate a robustness problem of the method itself. Therefore, we should think about the impact that might have on subsequent batch release analysis and if this can be tolerated as part of the normal method variability. If not, then we must take a step backwards and optimize the method regarding its robustness.
This leads to the point that you should perform method validation at a time when you are pretty sure that the method works really well, since everything else indicates not the best method development. In this context, so-called pre-validation tests or feasibility studies in late method development are helpful in order to avoid any unpleasant surprises during validation. This also protects against over-ambitious goals in form of too narrow, unrealistic acceptance criteria.
Documentation of incorrect results
Finally, a few words about documentation. For reasons of transparency, it is generally advisable to always list all values in the validation report, including exclusions. They might be presented in brackets and provided with a footnote for explanation. However, it can also be argued that only the repeated results should be presented with appropriate justification and the erroneous results and their raw data should be kept e.g. with the re-training records of the analyst who has generated the erroneous results. For this reason, it should already be noted in the validation plan how to deal with arising problems (atypical results or non-compliance with the acceptance criteria). In a specific deviation section of the validation report the issues will then be detailed. The detailedness is up to you. A few explanatory words and, if applicable and necessary, a reference to the number of an opened deviation or at least the laboratory error analysis are certainly not bad. Conversely, this doesn’t mean automatically that every single problem must be presented in absolute detail in the validation report. If it has already been detected during performance that e.g. a dilution has been prepared incorrectly, this can be recorded in the laboratory journal and the experiment can be re-scheduled without any notice in the validation report. The boundary between a simple note in the laboratory journal and opening a deviation can be drawn if the erroneous result is related to method performance. For obvious human or device-related failure causes, a simple note in the laboratory journal can be sufficient.
OOS versus OOT
Last but not least, a few words should be said about OOS versus OOT. By OOS we mean all results that are out of specification, and we usually think of the routine test results of release and stability analysis of drug (product) batches.
And what is OOT? By OOT (= out of trend) we mean those results that are not expected in the context of trending analyses. Let's take an example for explanation: For the determination of the shelf life of a drug product, a stability study was performed prior to submission, in which the investigated parameter (e.g. the content) shows a characteristically decreasing gradient, at the end of which, however, all results are still within specification (i.e. there is no OOS). If, after approval, we continue to monitor shelf life during ongoing stability studies and we would now observe a significantly greater decline, we would speak of OOT results, since there is a deviation from the expected gradient. The result could be that the shelf life would be significantly shortened because the content would fall below the lower specification limit well before the expected time. Thus, OOT results occur before an OOS result can occur. Early detection of OOT results thus enables appropriate actions to be taken in good time.
Update June 1st, 2022
Meanwhile, the FDA has revised the above-mentioned OOS Guideline from 2006. The new document, published in May 2022, can be found here. Also helpful is the information on handling OOS results published by the British Medicines & Healthcare products Regulatory Agency (MHRA) in the form of a presentation and updated in 2017.
These two recent warning letters / 483 from the FDA are examples of how important thorough and extensive investigations of OOS results are to the authorities.