Reliable Statistical Guarantees for Conformal Predictors with Small Datasets

• ArXiv number: 2512.04566

Surrogate models (including deep neural networks and other machine learning algorithms in supervised learning) are capable of approximating arbitrarily complex, high-dimensional input-output problems in science and engineering, but require a thorough data-agnostic uncertainty quantification analysis before these can be deployed for any safety-critical application. The standard approach for data-agnostic uncertainty quantification is to use conformal prediction (CP), a well-established framework to build uncertainty models with proven statistical guarantees that do not assume any shape for the error distribution of the surrogate model. However, since the classic statistical guarantee offered by CP is given in terms of bounds for the marginal coverage, for small calibration set sizes (which are frequent in realistic surrogate modelling that aims to quantify error at different regions), the potentially strong dispersion of the coverage distribution around its average negatively impacts the relevance of the uncertainty model's statistical guarantee, often obtaining coverages below the expected value, resulting in a less applicable framework. After providing a gentle presentation of uncertainty quantification for surrogate models for machine learning practitioners, in this paper we bridge the gap by proposing a new statistical guarantee that offers probabilistic information for the coverage of a single conformal predictor. We show that the proposed framework converges to the standard solution offered by CP for large calibration set sizes and, unlike the classic guarantee, still offers relevant information about the coverage of a conformal predictor for small data sizes. We validate the methodology in a suite of examples, and implement an open access software solution that can be used alongside common conformal prediction libraries to obtain uncertainty models that fulfil the new guarantee.

Related research projects

CSxAI

Complexity science for understanding AI: from the dynamics of complex networks to collective effects of interacting neural networks

I.P.: Lucas Lacasa, Víctor M. Eguíluz

The CSxAI project engages researchers at IFISC in a (non-oriented research project) on the interdisciplinary challenge of applying tools and concepts of complexity science (dynamical systems, statistical physics, and network science) to …

NEHIL

Neuromorphic-Enhanced Heterogeneously-Integrated FMCW LiDAR

I.P.: Miguel C. Soriano

The NEHIL project, an EU-Korea partnership, is set to transform the landscape of digital technologies through groundbreaking neuromorphic architectures and advanced heterogeneous integration such as LiDAR systems. This collaborative initiative aims to …

MdM-IFISC-2

Maria de Maeztu 2023-2026

I.P.: Ernesto Estrada, Ingo Fischer, Emilio Hernández-García, Rosa Lopez, Claudio Mirasso, Jose Javier Ramasco, Raúl Toral, Roberta Zambrini

After 15 years of its existence, IFISC can point to a proven track record of impactful research. The previous 2018-2022 MdM award has significantly enhanced the institute's capabilities, as demonstrated by an …