?
Toward reliable false discovery rate control in classification problems under distribution shift
Background
Domain shifts or batch effects can significantly degrade the performance of machine learning-based classifiers in software production without any warning. This can particularly be problematic in medical applications which involve human lives. The shift changes the shape of the prediction score distribution of the test samples compared to that of the training samples; hence, decision boundary re-calibration would be required to conform to the expected performance standards.
Method and findings
We present a simple, yet robust heuristic method to calculate valid p-values (uniformly distributed) for raw prediction scores of test samples under data shift in general classification problems. Our method is based on an adjustment of the test prediction scores so that the test null distribution (approximated distribution of the prediction scores of negative test samples) aligns to the training null distribution (actual distribution of prediction scores of the negative training samples). We termed it the Test Null Adjustment (TNA) method. Then, empirical p-values of the adjusted test prediction scores can be calculated with respect to the training null distribution to be used in any statistical applications which require test samples accompanied by valid p-values. We demonstrate the usability of our TNA method in FDR control with Benjamini-Hochberg (BH) protocol.
Conclusion
The main advantage of the TNA method is that its application does not require any domain specific knowledge. It works in binary and multi-label classification scenarios, it is fully data-driven, and it operates in the 1-dimensional space of the prediction scores ensuring stable numerical calculations. We have performed four experimental tests on the biomedical image analysis domain to demonstrate that the TNA method is able to mitigate the bias in FDR induced by data shift in real-life applications. TNA is available at https://github.com/Aborevsky01/TNA.