A pragmatic policy learning approach to account for users' fatigue in repeated auctions

Overview

• Online advertising banners are sold through real-time auctions
• The value of each additional banner shown to a user decreases
• Basic machine learning (ML) models can predict the decrease in value, but this is not enough to optimize bids
• Most bidders in the literature are "impatient" and do not fully account for the repeated nature of the auctions
• Impatience induces a cost that is important to consider

Plain English Explanation

When companies want to display ads on websites, they often use real-time auctions to buy the advertising "space" or "banners." The more banners a user sees, the less valuable each additional banner becomes to that user. Basic machine learning models can predict this decrease in value, but this knowledge alone is not enough to place bids that maximize the long-term profit.

Most bidders in previous research have been "impatient" - they only focus on winning the current auction and do not fully consider how winning now will impact the value of future auctions. This impatience comes at a cost, which the researchers demonstrate through an offline analysis and by showing that mitigating the cost of impatience leads to better business metrics.

Technical Explanation

The researchers note that the decreasing marginal value of advertising banners can be detected by basic ML models, which can predict how previously won auctions will impact the current auction's opportunity value. However, simply using these predictions to maximize the expected payoff of the current auction leads to an "impatient" policy that does not fully account for the repeated nature of the auctions.

The researchers provide two empirical arguments for the importance of this cost of impatience. First, they conduct an offline counterfactual analysis, and second, they demonstrate a notable improvement in key business metrics by mitigating the cost of impatience through policy learning. This aligns with prior work on the need for more realistic modeling of ad auctions.

Critical Analysis

The researchers acknowledge that learning is not enough to produce bids that fully account for the impact of winning the current auction on future values. They provide empirical evidence for the importance of considering this cost of impatience, but do not explore the underlying reasons in depth.

It would be interesting to see further research on the psychological and behavioral factors that contribute to impatient bidding strategies, as well as the potential cognitive biases or heuristics that may lead bidders to focus solely on short-term gains. Additionally, the researchers could investigate how different auction mechanisms or market structures might influence the prevalence and cost of impatient bidding.

Conclusion

This research highlights the need for bidding strategies in online advertising auctions to consider the repeated and dynamic nature of the auctions, rather than simply optimizing for the current auction. By accounting for the cost of impatience, the researchers demonstrate that businesses can improve their key metrics, pointing to the potential for more sophisticated reinforcement learning approaches to drive innovation in this space.