ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures

Overview

• This paper presents ClimRetrieve, a new benchmark dataset for evaluating information retrieval systems on corporate climate disclosures.
• The dataset contains over 31,000 documents from the financial filings of 400 publicly-traded companies, along with a set of 50 manually-curated information retrieval queries.
• The authors also provide baseline retrieval performance using several common IR models, establishing a benchmark for future research in this domain.

Plain English Explanation

The paper introduces a new dataset called ClimRetrieve that can be used to test and compare the performance of information retrieval (IR) systems when searching through corporate climate disclosures. Corporate climate disclosures are reports published by companies that describe their efforts to address climate change.

The ClimRetrieve dataset includes over 31,000 documents from the financial filings of 400 public companies, along with 50 sample search queries that someone might use to find information about a company's climate-related activities. The authors have already run several common IR models on the dataset and reported their baseline performance, providing a starting point for other researchers to build upon.

This dataset can be valuable for improving the retrieval of climate-related information from corporate disclosures, which is an important task for investors, policymakers, and the general public who want to understand how companies are addressing climate change. Having a standardized benchmark can help drive progress in this area of retrieval-augmented generation.

Technical Explanation

The ClimRetrieve dataset was constructed by the authors by scraping and processing over 31,000 financial documents from the SEC filings of 400 publicly-traded companies. These documents contain the companies' voluntary climate-related disclosures, which can be used to assess their environmental impact and sustainability efforts.

To create retrieval queries for the dataset, the authors manually curated a set of 50 information needs that someone might have when searching for climate-related information about a company. These queries cover a range of topics, from greenhouse gas emissions and renewable energy usage to climate risk mitigation strategies and sustainability reporting.

The authors then establish baseline retrieval performance on the ClimRetrieve dataset using several common IR models, including BM25, TF-IDF, and a language model-based approach. They evaluate the models' ability to rank the relevant documents highly for each query, using standard IR metrics like Normalized Discounted Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR).

This benchmark can serve as a starting point for exploring the nexus between retrievability and query generation strategies in the context of corporate climate disclosures. Researchers can use the ClimRetrieve dataset to develop and test new IR techniques, with the goal of improving the discoverability and accessibility of climate-related information in corporate filings.

Critical Analysis

The ClimRetrieve dataset represents a valuable contribution to the field of information retrieval, as it provides a standardized benchmark for evaluating systems on a real-world task with significant societal impact. The authors have taken care to construct a high-quality dataset and establish reasonable baseline performance, which should facilitate further research and development in this area.

One potential limitation of the dataset is the relatively small number of manually-curated queries (50). While this set covers a range of information needs, it may not be comprehensive, and the performance of IR models could be sensitive to the specific queries chosen. Expanding the query set or exploring collaborative retrieval-augmented generation approaches could help address this.

Additionally, the dataset only includes climate-related disclosures from publicly-traded companies, which may not be representative of the full landscape of corporate sustainability reporting. Extending the dataset to include private companies, non-profit organizations, or international entities could broaden its applicability and provide a more holistic view of climate-related information retrieval challenges.

Overall, the ClimRetrieve dataset represents a valuable contribution to the field and has the potential to drive progress in the important task of improving access to climate-related information in corporate disclosures.

Conclusion

The ClimRetrieve dataset provides a new benchmark for evaluating information retrieval systems on the task of surfacing climate-related information from corporate financial filings. By making this dataset publicly available, the authors have created an opportunity for researchers to develop and test novel IR techniques that can enhance the discoverability and accessibility of critical climate data.

The baseline performance reported in the paper establishes a starting point for future work, and the dataset's focus on a real-world problem with significant societal implications makes it a valuable resource for the field. As the need for transparent and actionable climate information continues to grow, tools and systems powered by the ClimRetrieve dataset can play a crucial role in empowering investors, policymakers, and the public to better understand and address the climate challenge.