Heteroscedastic Preferential Bayesian Optimization with Informative Noise Distributions
2405.14657
0
0
🛠️
Abstract
Preferential Bayesian optimization (PBO) is a sample-efficient framework for learning human preferences between candidate designs. PBO classically relies on homoscedastic noise models to represent human aleatoric uncertainty. Yet, such noise fails to accurately capture the varying levels of human aleatoric uncertainty, particularly when the user possesses partial knowledge among different pairs of candidates. For instance, a chemist with solid expertise in glucose-related molecules may easily compare two compounds from that family while struggling to compare alcohol-related molecules. Currently, PBO overlooks this uncertainty during the search for a new candidate through the maximization of the acquisition function, consequently underestimating the risk associated with human uncertainty. To address this issue, we propose a heteroscedastic noise model to capture human aleatoric uncertainty. This model adaptively assigns noise levels based on the distance of a specific input to a predefined set of reliable inputs known as anchors provided by the human. Anchors encapsulate partial knowledge and offer insight into the comparative difficulty of evaluating different candidate pairs. Such a model can be seamlessly integrated into the acquisition function, thus leading to candidate design pairs that elegantly trade informativeness and ease of comparison for the human expert. We perform an extensive empirical evaluation of the proposed approach, demonstrating a consistent improvement over homoscedastic PBO.
Create account to get full access
Overview
- Preferential Bayesian Optimization (PBO) is a framework for efficiently learning human preferences between different designs or options.
- Traditionally, PBO has used homoscedastic noise models to represent the uncertainty in human preferences, which fails to capture the varying levels of uncertainty people may have about different comparisons.
- This paper proposes a heteroscedastic noise model that can adaptively assign different noise levels based on how difficult it is for the human to compare specific pairs of designs.
Plain English Explanation
Preferential Bayesian Optimization (PBO) is a technique that helps find the best design or option by learning from a person's preferences. For example, if you're designing a new product, PBO can figure out which features the user likes best by having them compare different prototypes.
The original PBO approach assumed that the person's uncertainty, or "noise," in their preferences was the same no matter what options they were comparing. However, in reality, people often have more certainty about some comparisons than others. A chemist, for instance, might be very confident comparing two glucose-related compounds but struggle to compare alcohol-based ones.
This paper introduces a new way of modeling the person's uncertainty that can adjust to how difficult the comparison is. The model uses "anchors" - examples of comparisons the person is very confident about - to determine the appropriate level of uncertainty for each new comparison. This allows PBO to explore new designs that balance being informative for the person and being easy for them to evaluate.
The paper shows that this new approach consistently outperforms the original PBO method in experiments, demonstrating the importance of accounting for varying levels of human uncertainty.
Technical Explanation
Preferential Bayesian Optimization (PBO) is a sample-efficient framework for learning human preferences between candidate designs. Traditionally, PBO has relied on homoscedastic noise models to represent human aleatoric uncertainty (i.e., inherent randomness in the person's preferences). However, these models fail to capture the varying levels of uncertainty people may have when comparing different pairs of candidates.
To address this limitation, the authors propose a heteroscedastic noise model that can adaptively assign noise levels based on the distance of a specific input to a predefined set of "anchor" inputs provided by the human. These anchors encapsulate the person's partial knowledge and offer insights into the comparative difficulty of evaluating different candidate pairs.
The authors integrate this heteroscedastic noise model into the acquisition function used in PBO, which guides the search for the next candidate design to evaluate. This allows the optimization process to balance informativeness and ease of comparison for the human expert. The authors demonstrate the effectiveness of their approach through extensive empirical evaluation, showing consistent improvement over the homoscedastic PBO baseline.
Critical Analysis
The paper's key contribution is the introduction of a heteroscedastic noise model to capture varying levels of human uncertainty in PBO. This is a valuable advancement, as the original homoscedastic assumption can lead to suboptimal exploration of the design space and underestimation of the risks associated with human uncertainty.
However, the paper does not address the potential challenge of eliciting reliable anchor points from human experts. Determining the appropriate set of anchors and ensuring they accurately reflect the person's partial knowledge may require additional effort and could introduce new sources of bias. The authors could have discussed strategies for mitigating these issues, such as using causal Bayesian optimization to learn the anchors from data or incorporating formal safety guarantees into the optimization process.
Additionally, the paper does not explore the potential trade-offs between the increased modeling complexity of the heteroscedastic noise model and the computational overhead it may introduce. Investigating the computational efficiency and information-theoretic considerations of the proposed approach could provide valuable insights for practitioners.
Conclusion
This paper presents a novel heteroscedastic noise model for Preferential Bayesian Optimization (PBO) that can adapt to varying levels of human uncertainty in design comparisons. By incorporating information about the person's partial knowledge through anchor points, the model can more accurately capture the difficulty of different comparisons and guide the optimization process accordingly.
The authors demonstrate the effectiveness of their approach through extensive empirical evaluation, showing consistent improvements over the traditional homoscedastic PBO. This work contributes to the broader effort of developing more effective and user-centric optimization techniques, with potential applications in fields such as product design, engineering, and decision-making support systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🛠️
Principled Preferential Bayesian Optimization
Wenjie Xu, Wenbin Wang, Yuning Jiang, Bratislav Svetozarevic, Colin N. Jones
0
0
We study the problem of preferential Bayesian optimization (BO), where we aim to optimize a black-box function with only preference feedback over a pair of candidate solutions. Inspired by the likelihood ratio idea, we construct a confidence set of the black-box function using only the preference feedback. An optimistic algorithm with an efficient computational method is then developed to solve the problem, which enjoys an information-theoretic bound on the total cumulative regret, a first-of-its-kind for preferential BO. This bound further allows us to design a scheme to report an estimated best solution, with a guaranteed convergence rate. Experimental results on sampled instances from Gaussian processes, standard test functions, and a thermal comfort optimization problem all show that our method stably achieves better or competitive performance as compared to the existing state-of-the-art heuristics, which, however, do not have theoretical guarantees on regret bounds or convergence.
5/30/2024
Preferential Multi-Objective Bayesian Optimization
Raul Astudillo, Kejun Li, Maegan Tucker, Chu Xin Cheng, Aaron D. Ames, Yisong Yue
0
0
Preferential Bayesian optimization (PBO) is a framework for optimizing a decision-maker's latent preferences over available design choices. While preferences often involve multiple conflicting objectives, existing work in PBO assumes that preferences can be encoded by a single objective function. For example, in robotic assistive devices, technicians often attempt to maximize user comfort while simultaneously minimizing mechanical energy consumption for longer battery life. Similarly, in autonomous driving policy design, decision-makers wish to understand the trade-offs between multiple safety and performance attributes before committing to a policy. To address this gap, we propose the first framework for PBO with multiple objectives. Within this framework, we present dueling scalarized Thompson sampling (DSTS), a multi-objective generalization of the popular dueling Thompson algorithm, which may be of interest beyond the PBO setting. We evaluate DSTS across four synthetic test functions and two simulated exoskeleton personalization and driving policy design tasks, showing that it outperforms several benchmarks. Finally, we prove that DSTS is asymptotically consistent. As a direct consequence, this result provides, to our knowledge, the first convergence guarantee for dueling Thompson sampling in the PBO setting.
6/24/2024
Provably Efficient Bayesian Optimization with Unbiased Gaussian Process Hyperparameter Estimation
Huong Ha, Vu Nguyen, Hung Tran-The, Hongyu Zhang, Xiuzhen Zhang, Anton van den Hengel
0
0
Gaussian process (GP) based Bayesian optimization (BO) is a powerful method for optimizing black-box functions efficiently. The practical performance and theoretical guarantees of this approach depend on having the correct GP hyperparameter values, which are usually unknown in advance and need to be estimated from the observed data. However, in practice, these estimations could be incorrect due to biased data sampling strategies used in BO. This can lead to degraded performance and break the sub-linear global convergence guarantee of BO. To address this issue, we propose a new BO method that can sub-linearly converge to the objective function's global optimum even when the true GP hyperparameters are unknown in advance and need to be estimated from the observed data. Our method uses a multi-armed bandit technique (EXP3) to add random data points to the BO process, and employs a novel training loss function for the GP hyperparameter estimation process that ensures consistent estimation. We further provide theoretical analysis of our proposed method. Finally, we demonstrate empirically that our method outperforms existing approaches on various synthetic and real-world problems.
6/7/2024
Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Prior
Shuyu Cheng, Yibo Miao, Yinpeng Dong, Xiao Yang, Xiao-Shan Gao, Jun Zhu
0
0
This paper studies the challenging black-box adversarial attack that aims to generate adversarial examples against a black-box model by only using output feedback of the model to input queries. Some previous methods improve the query efficiency by incorporating the gradient of a surrogate white-box model into query-based attacks due to the adversarial transferability. However, the localized gradient is not informative enough, making these methods still query-intensive. In this paper, we propose a Prior-guided Bayesian Optimization (P-BO) algorithm that leverages the surrogate model as a global function prior in black-box adversarial attacks. As the surrogate model contains rich prior information of the black-box one, P-BO models the attack objective with a Gaussian process whose mean function is initialized as the surrogate model's loss. Our theoretical analysis on the regret bound indicates that the performance of P-BO may be affected by a bad prior. Therefore, we further propose an adaptive integration strategy to automatically adjust a coefficient on the function prior by minimizing the regret bound. Extensive experiments on image classifiers and large vision-language models demonstrate the superiority of the proposed algorithm in reducing queries and improving attack success rates compared with the state-of-the-art black-box attacks. Code is available at https://github.com/yibo-miao/PBO-Attack.
5/30/2024