Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Induction

Overview

• Historical linguists have long used a process of reconstructing words in an ancestor language and then writing programs to convert those reconstructed words into words in the descendent languages.
• This process is error-prone and time-consuming, so prior research has tried to automate parts of it.
• This paper proposes a new approach that uses large language models to generate sound law programs from examples of sound changes.

Plain English Explanation

Historical linguists study how languages change over time. They do this by reconstructing the words of ancient "parent" languages and then trying to figure out how those words evolved into the words we see in modern "child" languages.

To do this, linguists have traditionally written custom programs that take the reconstructed ancient words and convert them step-by-step into the modern words. They base these programs on observations of how specific sounds in the ancient words changed over time.

However, writing these conversion programs is difficult and error-prone. Researchers have previously tried to automate parts of this process computationally, but fewer have tackled the core challenge of inducing the sound laws that govern how sounds change.

This paper proposes a new approach that uses powerful language models to generate these sound law programs directly from examples of sound changes. The key idea is to cast the problem as a form of "programming by example" that the language models can learn to solve.

Technical Explanation

The paper frames the task of Sound Law Induction (SLI) as a "Programming by Examples" problem. Specifically, the authors propose generating Python programs that can convert reconstructed words (protoforms) into their attested descendant forms (reflexes) based on examples of such sound changes.

To do this, the authors experiment with various large language models (LLMs) and evaluate their ability to generate sound law programs from example pairs of protoforms and reflexes. They also explore methods for generating synthetic data to fine-tune the LLMs for this task.

The authors compare their LLM-based approach to existing automated SLI methods, finding that while the LLMs lag behind in performance, they can complement the existing techniques in useful ways. The LLM-generated programs may be less accurate but more flexible and generalizable.

Critical Analysis

The paper presents a novel approach to automating a critical task in historical linguistics, but it acknowledges several limitations. The LLM-generated programs are not as accurate as those produced by specialized SLI algorithms, and the authors note that further research is needed to improve their performance.

Additionally, the synthetic data generation methods proposed may not fully capture the nuances and complexities of real-world sound changes, which could limit the models' ability to generalize. The authors suggest that combining their approach with existing SLI techniques could be a fruitful area for future work.

One could also question whether LLMs are the optimal solution for this problem, given that they are not specifically designed for the task of program generation. Alternative approaches that more directly incorporate linguistic knowledge may prove more effective in the long run.

Conclusion

This paper presents a promising new direction for automating the process of Sound Law Induction in historical linguistics. By leveraging the impressive capabilities of large language models, the authors have shown that it is possible to generate sound law programs from examples of sound changes.

While the current performance of this approach lags behind specialized SLI algorithms, the flexibility and generalizability of the LLM-generated programs could make them a valuable complement to existing techniques. Continued research in this area has the potential to significantly streamline and accelerate historical linguistic analysis, ultimately leading to a better understanding of how languages evolve over time.