Conformalized Selective Regression
2402.16300
0
0
Abstract
Should prediction models always deliver a prediction? In the pursuit of maximum predictive performance, critical considerations of reliability and fairness are often overshadowed, particularly when it comes to the role of uncertainty. Selective regression, also known as the reject option, allows models to abstain from predictions in cases of considerable uncertainty. Initially proposed seven decades ago, approaches to selective regression have mostly focused on distribution-based proxies for measuring uncertainty, particularly conditional variance. However, this focus neglects the significant influence of model-specific biases on a model's performance. In this paper, we propose a novel approach to selective regression by leveraging conformal prediction, which provides grounded confidence measures for individual predictions based on model-specific biases. In addition, we propose a standardized evaluation framework to allow proper comparison of selective regression approaches. Via an extensive experimental approach, we demonstrate how our proposed approach, conformalized selective regression, demonstrates an advantage over multiple state-of-the-art baselines.
Create account to get full access
Overview
- This paper provides instructions for authors on the formatting and submission requirements for a particular publication.
- It covers general formatting guidelines, authorship rules, and specific requirements for the structure and content of the manuscript.
- The instructions aim to ensure a consistent and standardized format across all submitted works.
Plain English Explanation
The paper outlines the guidelines that authors must follow when submitting their work to be considered for publication. This includes rules around formatting the manuscript, such as the font, spacing, and page layout. It also covers the authorship requirements, specifying how many authors can be listed and the information that must be provided about each one.
Additionally, the instructions detail the structure and content that the manuscript must have, such as the sections that must be included (e.g. introduction, methods, results) and the type of information that should be presented in each. This ensures a consistent format across all submissions, making it easier for editors and reviewers to process the papers.
The goal of these guidelines is to streamline the publication process and ensure that all accepted works meet a certain standard of quality and presentation. By following these instructions, authors can increase the chances of their paper being successfully reviewed and accepted for publication.
Technical Explanation
The paper outlines the formatting and submission requirements for authors interested in publishing their work in a particular venue. It covers general guidelines for the manuscript, such as page limits, font sizes, and spacing.
The authorship section specifies the rules around listing authors, including the maximum number allowed and the information that must be provided for each (e.g. name, affiliation, contact details).
The instructions also detail the required structure and content of the manuscript, such as the sections that must be included (e.g. introduction, methods, results, discussion) and the type of information that should be presented in each. This ensures a consistent format across all submissions, facilitating the review and publication process.
Additionally, the paper provides guidance on formatting citations and references, as well as the submission process and any relevant deadlines.
Critical Analysis
The instructions provide a comprehensive and well-structured set of guidelines for authors, covering all the key aspects of manuscript preparation and submission. The clear and detailed nature of the requirements is likely to help ensure a consistent quality and format across all accepted works, benefiting both the authors and the publication.
However, one potential limitation is the risk of overly restrictive formatting rules stifling creativity or preventing authors from effectively communicating their research. The instructions should strike a balance between standardization and flexibility, allowing for some variation in presentation while maintaining the necessary level of consistency.
Additionally, the instructions could be improved by providing more specific examples or sample manuscripts to help authors better understand the expected formatting and content. This would further facilitate the submission process and reduce the risk of rejection due to non-compliance with the guidelines.
Overall, these instructions appear to be a well-designed set of guidelines that should help authors successfully navigate the submission process and increase the chances of their work being accepted for publication.
Conclusion
The paper outlines the detailed formatting and submission requirements for authors interested in publishing their research in a particular venue. By providing clear and comprehensive instructions, the guidelines aim to ensure a consistent quality and presentation across all accepted works.
The instructions cover key aspects such as manuscript formatting, authorship rules, and the required structure and content of the submission. This standardization benefits both the authors and the publication, streamlining the review and publication process.
While the instructions are thorough, it is important to maintain a balance between consistency and flexibility, allowing authors to effectively communicate their research while adhering to the necessary formatting requirements. Overall, these guidelines appear to be a well-designed set of instructions that should help authors successfully navigate the submission process.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🏷️
Calibrated Selective Classification
Adam Fisch, Tommi Jaakkola, Regina Barzilay
0
0
Selective classification allows models to abstain from making predictions (e.g., say I don't know) when in doubt in order to obtain better effective accuracy. While typical selective models can be effective at producing more accurate predictions on average, they may still allow for wrong predictions that have high confidence, or skip correct predictions that have low confidence. Providing calibrated uncertainty estimates alongside predictions -- probabilities that correspond to true frequencies -- can be as important as having predictions that are simply accurate on average. However, uncertainty estimates can be unreliable for certain inputs. In this paper, we develop a new approach to selective classification in which we propose a method for rejecting examples with uncertain uncertainties. By doing so, we aim to make predictions with {well-calibrated} uncertainty estimates over the distribution of accepted examples, a property we call selective calibration. We present a framework for learning selectively calibrated models, where a separate selector network is trained to improve the selective calibration error of a given base model. In particular, our work focuses on achieving robust calibration, where the model is intentionally designed to be tested on out-of-domain data. We achieve this through a training strategy inspired by distributionally robust optimization, in which we apply simulated input perturbations to the known, in-domain training data. We demonstrate the empirical effectiveness of our approach on multiple image classification and lung cancer risk assessment tasks.
6/24/2024
Conformal online model aggregation
Matteo Gasparin, Aaditya Ramdas
0
0
Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification without making strong distributional assumptions. It wraps around any black-box prediction model and converts point predictions into set predictions that have a predefined marginal coverage guarantee. However, conformal prediction only works if we fix the underlying machine learning model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: for a given problem, which of the plethora of prediction methods (random forests, neural nets, regularized linear models, etc.) should we conformalize? This paper proposes a new approach towards conformal model aggregation in online settings that is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
5/3/2024
🔮
Self-Consistent Conformal Prediction
Lars van der Laan, Ahmed M. Alaa
0
0
In decision-making guided by machine learning, decision-makers may take identical actions in contexts with identical predicted outcomes. Conformal prediction helps decision-makers quantify uncertainty in point predictions of outcomes, allowing for better risk management for actions. Motivated by this perspective, we introduce textit{Self-Consistent Conformal Prediction} for regression, which combines two post-hoc approaches -- Venn-Abers calibration and conformal prediction -- to provide calibrated point predictions and compatible prediction intervals that are valid conditional on model predictions. Our procedure can be applied post-hoc to any black-box model to provide predictions and inferences with finite-sample prediction-conditional guarantees. Numerical experiments show our approach strikes a balance between interval efficiency and conditional validity.
4/23/2024
↗️
Conditional validity of heteroskedastic conformal regression
Nicolas Dewolf, Bernard De Baets, Willem Waegeman
0
0
Conformal prediction, and split conformal prediction as a specific implementation, offer a distribution-free approach to estimating prediction intervals with statistical guarantees. Recent work has shown that split conformal prediction can produce state-of-the-art prediction intervals when focusing on marginal coverage, i.e. on a calibration dataset the method produces on average prediction intervals that contain the ground truth with a predefined coverage level. However, such intervals are often not adaptive, which can be problematic for regression problems with heteroskedastic noise. This paper tries to shed new light on how prediction intervals can be constructed, using methods such as normalized and Mondrian conformal prediction, in such a way that they adapt to the heteroskedasticity of the underlying process. Theoretical and experimental results are presented in which these methods are compared in a systematic way. In particular, it is shown how the conditional validity of a chosen conformal predictor can be related to (implicit) assumptions about the data-generating distribution.
5/1/2024