Fine-tuning Large Language Models for Automated Diagnostic Screening Summaries

Introduction

Related Works

This paper focuses on abstractive summarization, which involves using new words and phrases to create a summary, even if they weren't present in the original text. Large language models (LLMs) like GPT, BART, and T5 have shown promise for abstractive summarization, but their application in the medical and psychological domains requires further exploration to address inaccuracies without domain-specific knowledge.

The paper discusses the evolution of abstractive summarization techniques, including sequence-to-sequence models, attention mechanisms, and transformer-based encoder-decoder architectures. It also mentions the use of pointing mechanisms for word copying from the source document.

Regarding dialogue summarization, the paper notes that while models like BART and GPT-3 demonstrate exceptional performance on general-purpose tasks, they often require additional domain-specific conversation/dialogue data to understand dialogues better. The lack of publicly available appropriate datasets for medical dialogues, particularly in the psychological domain, creates a challenge for generating accurate abstractive summaries.

The paper highlights recent advancements in automatic medical dialogue summarization, including the use of LSTM and transformer models to generate single-sentence summaries from doctor-patient conversations, as well as the application of hierarchical encoder-tagger models to identify and extract meaningful utterances. However, the paper suggests that conversations in the psychological domain present unique challenges due to their longer length and more detailed patient responses, requiring specialized approaches beyond those employed in typical dialogue processing.

Methodology

Figure 1 provides a high-level overview of the methodology. The detailed description of the methodology sub-components is as follows:

The researchers identified the lack of a standardized Mental State Examination (MSE) questionnaire and reviewed existing options. They created a preliminary 18-question version tailored for students, covering key components like socialness, mood, attention, memory, frustration tolerance, and social support. The researchers then sought input from clinical psychiatrists to refine the questionnaire, resulting in a finalized 12-question version.

The researchers conducted a data collection study at their institute, obtaining ethical approval and recruiting student participants. Participants completed the MSE questionnaire, with their responses transformed into simulated doctor-patient conversations. This process generated 300 conversations with 7200 total utterances.

To facilitate training supervised deep-learning models for summarizing the conversations, the researchers developed a structured summary template that underwent review and revision. They then evaluated several pre-trained transformer-based models, including BART, T5, and models fine-tuned on conversational and medical text data, for their suitability in summarizing the collected MSE conversations.

Experiments

The paper adopted the ROUGE metric as the primary evaluation criterion for automated summarization. The ROUGE metric compares the automated summary generated from the trained model with the reference summary. While ROUGE is well-suited for assessing syntactical textual similarities, it fails to capture semantic similarities between summaries. To address this limitation, the authors conducted qualitative analysis using ratings from clinical and non-clinical annotators to assess the semantic similarities between reference and model-generated summaries.

The analysis showed that the fine-tuned BART-large-CNN model achieved the highest ROUGE values, outperforming other models. However, the model-generated summaries slightly lacked in completeness and relevance compared to human-generated summaries. The model-generated summaries also contained more missing information, and a higher degree of contradiction was observed, although the model did not exhibit hallucination.

The inter-rater agreement analysis showed moderate agreement and strong correlation among clinical annotators, while non-clinical annotators demonstrated good reliability in their assessments.

Generalization

The researchers used the publicly available D4 dataset to assess the generalizability of their best fine-tuned BART-large CNN model. They had three independent non-clinical reviewers rate the summaries generated by the model for ten randomly selected conversations from the D4 dataset. The evaluation parameters included completeness, relevance to the medical context, fluency, clarity, missingness, hallucination, contradiction, and repetitions.

The D4 dataset was in Chinese, so the researchers used Google Translate to translate the conversations to English. They extracted ten doctor-patient conversations, assigned dummy participant identifiers, and shared the translated English conversations and the corresponding fine-tuned BART-large-CNN model-generated summaries with the reviewers.

The review found that the best fine-tuned model's summary scored well in relevance, fluency, clarity, and repetition. However, the generated summary was slightly lacking in terms of missing information, hallucination, and contradiction. Tables in the appendix present example dialogues from the D4 dataset and the corresponding summaries generated by the fine-tuned BART-large-CNN model.

Comparison with the previous work

The paper represents the first attempt to summarize psychological conversation data, which differs from traditional text summarization but shares similarities with dialogue summarization. The model trained on the authors' dataset achieved a ROUGE-L score of 0.764, outperforming the previous work by Yao et al. (2022) which had a score of 0.26. The fine-tuned model also produced fluent and comprehensive summaries when applied to the dataset used by Yao et al. (2022).

The paper presents a comparative report with existing research in doctor-patient conversation analysis. The results show that the fine-tuned model outperforms the existing work by Yao et al. (2022), Krishna et al. (2021), Michalopoulos et al. (2022), and Zhang et al. (2021) in terms of the ROUGE metric. However, it is important to note that the previous studies used different datasets and had their own specific objectives beyond solely summarizing entire conversations. The current work primarily aims to generate summaries of psychological conversations, which presents its own challenges, such as dealing with lengthy conversation data and resulting in longer utterances.

Conclusion

The paper investigates using pre-trained transformer models to generate summaries of psychological patient conversations. This task faces several challenges, including limited publicly available data, significant domain shift from typical pre-training text, and unstructured lengthy dialogues.

The paper demonstrates that fine-tuning transformer models on a specific dataset can generate fluent and adequate summaries, even with limited training data. The resulting models outperform pre-trained models and surpass the quality of previously published work on this task.

The paper evaluates transformer models for handling psychological conversations, compares pre-trained and fine-tuned models, and conducts extensive and intensive evaluations.

Ethical Consideration

The study used anonymized numerical identifiers to store sensitive personal information about the participants, such as name, age, and email address, to ensure their privacy. The study was approved by the ethics committee of the host institute.

The pre-trained model demonstrated effectiveness on the dataset, and the models used were able to learn from a small number of conversations in fewer epochs. This suggests that utilizing pre-trained models tailored to specific tasks may yield better results than developing models from scratch, which can require large datasets and more training time.

The researchers initially hypothesized that the BART-large-xsum-samsum model trained on dialogue summarization data would perform better than other summarization models. However, they found that the BART-large-CNN model outperformed in terms of all ROUGE metrics, indicating that their hypothesis was incorrect. Further exploration is warranted.

This work presented the best fine-tuned summarization models for generating accurate and concise summaries from mental health conversations to assist overburdened psychiatrists. The primary intention is not to replace doctors but to serve as an assistant, offering concise summaries of patients' mental health. This approach holds promise for implementation in low-income countries with a shortage of mental health professionals, but further research is necessary to address privacy concerns and ensure the accuracy of the data utilized.

Limitations

This work achieved a better ROUGE score by comparing the generated and human reference summaries. However, the work has several limitations:

1. The study did not incorporate the physical behavior and appearance of the participants during the MSE (multimodal summarization evaluation). This could be addressed by activating the front camera or webcam of participants' phones while recording their responses.

2. There were instances where the participants' utterances were unclear to the reviewers. In real-world scenarios, a doctor would typically ask the patient to repeat or explain further. This poses a challenge in the study. The issue could potentially be mitigated by testing the user's response for fluency and completeness after each utterance and prompting the user to elaborate if the model detects an issue.

Appendix A Appendix

The paper summarizes the technical details and performance of the BART-large-CNN, T5 large, and BART-large-xsum-samsum models used for generating summaries of psychological conversations. Key points:

• The models were trained using the following hyperparameters: max token length of 1024 tokens, 500 warmup steps, 0.01 weight decay, evaluation strategy of steps every 500 steps, save steps of 1 million, and gradient accumulation of 16 steps. Training was done on an NVIDIA A100-PCIE-40GB GPU, with an average training time of 45 minutes.

• ROUGE evaluation metrics showed the models achieved strong performance, with the BART-large-CNN model performing the best across ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-L-SUM scores.

• Human evaluation by both clinical and non-clinical annotators was also conducted, assessing criteria like completeness, relevance, fluency, clarity, missingness, hallucination, contradiction, and repetition. The results showed the generated summaries were of high quality.

• Inter-rater reliability between the annotators was measured using Cohen's Kappa and Pearson's correlation, indicating strong agreement.

Overall, the paper demonstrates the effectiveness of large language models like BART and T5 for summarizing psychological conversations, with the BART-large-CNN model performing the best.