Automatic Alignment of Discourse Relations of Different Discourse Annotation Frameworks

Introduction

The summary is as follows:

Discourse relations are important for achieving coherence in text. Previous studies have shown the benefits of incorporating discourse relations in tasks like sentiment analysis, text summarization, and machine comprehension. Automatic discourse relation classification is a key part of discourse parsing, which is done using frameworks like Rhetorical Structure Theory (RST) and the Lexicalized Tree-Adjoining Grammar for Discourse (D-LTAG). These frameworks have been used to create large discourse corpora like the RST Discourse Treebank and the Penn Discourse Treebank.

Creating discourse corpora requires significant linguistic knowledge and is costly. However, these discourse frameworks share similar understandings of discourse relations and their role in discourse building. An approach to enlarge discourse corpora is to align existing corpora built on different frameworks so they can be used jointly. Aligning discourse relations across different annotation frameworks remains a challenging task.

The text provides an example of RST-style discourse annotation, showing how textual spans are connected by asymmetric discourse relations, with some relations represented by undirected parallel lines. The full text is recursively annotated with these discourse relations.

This paper discusses the challenges in aligning discourse relations between two different annotation frameworks, RST-DT and PDTB. The key points are:

1. RST-DT and PDTB have different assumptions about discourse units and higher-level structures. RST uses clauses as elementary discourse units (EDUs), while PDTB focuses on semantic properties of arguments.

2. RST enforces a tree structure, while PDTB only considers local relations without commitment to a higher-level structure.

3. RST-DT uses 78 relation types that can be divided into subject matter and presentational relations, while PDTB has a three-level sense hierarchy and allows multiple sense labels per argument pair.

4. These differences make it challenging to investigate the relationship between discourse relations in the two frameworks, even when using corpora annotated in multiple frameworks in parallel.

5. Previous approaches have relied on string matching to align PDTB arguments and RST EDUs, but this is limited by the strong nuclearity hypothesis.

6. The paper proposes a fully automatic method using label embedding techniques to explore correlations between relations in RST-DT and PDTB without the need for argument matching.

Related Work

The provided text discusses research on mapping discourse relations across different frameworks. It categorizes existing research into three types:

a) Identifying commonly used relations across frameworks by analyzing definitions and examples. b) Introducing fundamental concepts for analyzing relations across frameworks. c) Directly mapping discourse relations based on corpora annotated in multiple frameworks.

The text then focuses on the third approach, summarizing studies in this direction. Key findings include:

• Differences in annotation operationalization and relation granularity lead to many-to-many mappings between frameworks.
• Segmentation differences present challenges, but using the strong nuclearity hypothesis can help address this.
• Explicit discourse connectors in one framework can often be matched to relations in another framework.
• Attempts have been made to induce implicit relations in one framework from annotations in another, but challenges remain due to segmentation differences.

The text then discusses label embeddings, which have been shown to be useful in computer vision and NLP tasks. Key points include:

• Label embeddings can address issues with one-hot encoding, such as lack of robustness and failure to capture label semantics.
• Label embeddings can be obtained from external sources or learned during model training, incorporating additional information such as label hierarchy or textual descriptions.
• Label embeddings have been shown to be effective in data-imbalanced settings and zero-shot learning.

Method

This paper presents a method for learning label embeddings that can be used to correlate two discourse annotation frameworks D1 and D2. The key steps are:

1. Represent the input sequences X using a pre-trained language model like BERT. The [CLS] token representation is used as the sequence representation.

2. Explore different options for encoding the labels, including using BERT, RoBERTa, random initialization, and incorporating label descriptions. The label embeddings are learnable.

3. Apply an instance-centered contrastive loss and a label-centered contrastive loss to learn the label embeddings. These losses help mitigate issues with small batch sizes.

4. Add a label-embedding-based cross-entropy loss and a canonical multi-class cross-entropy loss to further improve the label embeddings and the input representations.

To evaluate the quality of the learnt label embeddings, the paper proposes computing a correlation matrix M between the label embeddings and class representation proxies. The average of the main diagonal values in M is used as an overall measure of embedding quality.

The paper also describes baseline approaches for relation classification and label embedding learning to compare against the proposed method.

Experiments

Here is a summary of the provided text:

The researchers focus on 16 relations for RST and PDTB L2 senses with more than 100 instances to learn label embeddings. They preprocess the data by binarizing the RST trees and mapping the 78 relations to 16 classes. For PDTB, they use sections 2-20 for training, 0-1 for validation, and 21-22 for testing.

The researchers run each model 3 times with different random seeds and report the mean and standard deviation. They use the AdamW optimizer, set the learning rate to 1e-5, and adopt early stopping. The temperature for the contrastive losses is set to 0.1.

The results show the performance on PDTB is better than RST. Adding label embeddings generally lowers F1 compared to using cross-entropy loss only, likely due to data sparsity. The random label encoder performs best on classification accuracy but has the lowest quality label embeddings.

To improve RST, the researchers use back translation data augmentation, which increases F1 and the quality of the label embeddings. Separate experiments on PDTB explicit and implicit relations show the label embeddings are more representative for explicit relations.

An ablation study finds the label-centered contrastive loss is most important, followed by the instance-centered contrastive loss and the canonical cross-entropy loss. This differs from prior work.

RST-PDTB Relation Mapping

The provided section of the paper presents the results of mapping the relations between Rhetorical Structure Theory (RST) and Penn Discourse Treebank (PDTB). The key points are:

• Table 5 shows the results of mapping 11 RST relations and 12 PDTB explicit relations. The two relations with the highest cosine similarity scores (greater than 0.10) are presented.

• From the RST perspective, a PDTB relation can be identified as having a much higher cosine similarity score (≥ 0.40) than the others for most RST relations.

• From the PDTB perspective, the mapping results are not symmetric due to differences in relation distributions.

• For the extrinsic evaluation, the authors relabeled PDTB explicit relations with RST labels based on the cosine similarity results and the relation distribution. They compared their results with those provided by Costa et al. (2023).

• Using an ensemble model, the performance with the authors' method is slightly higher than the results reported in Costa et al. (2023), with an accuracy of 63.13% and F1 of 47.95%.

Conclusions

This paper describes a method for automatically aligning discourse relations from different frameworks. The approach uses label embeddings that are learned alongside input representations during a classification task. This helps address the challenge of segmentation differences, which has been a significant issue in prior studies. The results are evaluated both intrinsically and extrinsically. The authors note that the method's performance is affected by the amount of data available, and they had to exclude some relations due to insufficient training data to learn reliable label embeddings. The paper suggests comparing the method to a theoretical proposal like ISO 24617-8 as future work. Additionally, the authors state the method may extend beyond just discourse relation labeling to aligning any label sets, opening up potential applications in various scenarios.

Acknowledgments

The authors express gratitude to the anonymous reviewers for their helpful feedback and suggestions. They also thank Mark-Jan Nederhof for productive discussions and Craig Myles for recommending the use of the diagonal entries of the normalized correlation matrix as a metric.

Bibliographical References

There is no text provided to summarize.

Language Resource References

The text summarizes two discourse treebank resources: the RST Discourse Treebank and the Penn Discourse Treebank Version 3.0. The RST Discourse Treebank was distributed by the Linguistic Data Consortium (LDC) in 2002 and contains annotated text resources. The Penn Discourse Treebank Version 3.0 was distributed by the LDC in 2019 and is a newer version of the discourse treebank resource.