[Citation needed] Data usage and citation practices in medical imaging conferences

Overview

• Medical imaging research often focuses on methodology, but the quality and validity of the results depend heavily on the datasets used
• Creating datasets is a lot of work, so researchers often use publicly available datasets
• However, there is no standard way to cite the datasets used in scientific papers, making it difficult to track their usage

Plain English Explanation

The papers on medical imaging techniques usually describe the methods and algorithms used, but the quality and reliability of the results depend a lot on the datasets that were used in the research. Collecting and preparing datasets for medical imaging is a huge amount of work, so researchers often turn to publicly available datasets instead of creating their own.

The problem is that there is no common way for researchers to cite the datasets they used in their scientific papers. This makes it hard to keep track of how these publicly available datasets are being used across the field. To address this issue, the researchers created two open-source tools that can help detect when datasets are used in papers. They applied these tools to study the usage of 20 publicly available medical datasets in papers from two major conferences, MICCAI and MIDL.

Technical Explanation

The researchers developed two tools to help track the usage of publicly available medical datasets in scientific papers:

1. A pipeline that uses the OpenAlex database and full-text analysis to detect when datasets are cited or mentioned in papers.
2. A PDF annotation software that they used to manually label whether datasets were present in the papers they studied.

They applied these tools to examine the usage of 20 different publicly available medical datasets in papers published at the MICCAI and MIDL conferences between 2013 and 2023. The researchers calculated the proportion and trends over time of three types of dataset presence in the papers: cited, mentioned in the full text, and both cited and mentioned.

Their analysis revealed that a limited set of datasets tend to be heavily used, demonstrating a concentration of dataset usage. The researchers also observed inconsistent citing practices across papers, making it difficult to automate the tracking of dataset usage.

Critical Analysis

The researchers acknowledge that their study is limited to the datasets and papers they examined, and that there may be other publicly available medical datasets and citing practices not captured in their analysis.

Additionally, the manually annotated dataset used in their study could be subject to human error or bias. Automating the detection of dataset usage in papers remains challenging due to the lack of standardization in how datasets are cited.

Further research is needed to develop more robust and comprehensive methods for tracking the usage of publicly available datasets across the medical imaging field. Establishing best practices for dataset citation could greatly improve transparency and accountability in this area of research.

Conclusion

This study highlights the importance of dataset usage in medical imaging research and the challenges in tracking how publicly available datasets are being utilized. The researchers developed tools to analyze dataset citations and mentions, revealing a concentration of usage for a limited set of datasets and inconsistent citing practices.

While their findings are limited in scope, this work underscores the need for greater standardization and transparency around dataset usage in the field. Addressing these issues could lead to more robust and reliable medical imaging research, as well as better opportunities for collaboration and reproducibility.