In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models

Overview

• This paper introduces a novel technique called In2Core for selecting a "coreset" of the most influential training examples during fine-tuning of large language models (LLMs) on instruction-following tasks.
• The key idea is to leverage influence functions to identify the training examples that have the largest impact on the model's performance, and use this to construct a smaller, more efficient coreset for fine-tuning.
• Experiments show that In2Core can achieve comparable performance to full fine-tuning, but with significantly fewer training examples, leading to faster and more data-efficient fine-tuning.

Plain English Explanation

When training large language models like GPT-3 or BERT on specific tasks, a common approach is "fine-tuning" - taking the pre-trained model and further training it on a smaller, task-specific dataset. This allows the model to specialize and perform better on the target task.

However, fine-tuning can be computationally expensive and time-consuming, especially when the task-specific dataset is large. The key insight of this paper is that not all training examples are equally important - some have a much bigger impact on the final model performance than others.

The researchers developed a technique called In2Core that uses "influence functions" to identify the most influential training examples. Influence functions measure how much each training example contributes to the final model. By selecting a small "coreset" of the top influential examples, the researchers showed they could achieve comparable performance to full fine-tuning, but with significantly fewer training examples.

This makes the fine-tuning process much faster and more data-efficient, which is especially important for tasks where labeled data is scarce or expensive to obtain. The paper demonstrates the effectiveness of In2Core on several instruction-following language tasks.

Technical Explanation

The core innovation of this paper is the In2Core method for coreset selection during fine-tuning of large language models on instruction-following tasks. The key steps are:

1. Compute Influence Functions: The researchers use influence functions to estimate the influence of each training example on the final model parameters. Influence functions measure how much changing a particular training example would change the final model.

2. Select a Coreset: Based on the influence scores, In2Core selects a small "coreset" of the most influential training examples. This coreset is then used to fine-tune the pre-trained language model, instead of the full training set.

3. Fine-tune on the Coreset: The researchers fine-tune the pre-trained model using only the selected coreset of examples. They show this achieves comparable performance to full fine-tuning, but with significantly fewer training examples.

The paper evaluates In2Core on several instruction-following language tasks, including instruction-following on the HumanEval and Anthropic Corpus datasets. The results demonstrate the data efficiency benefits of the coreset approach compared to full fine-tuning.

Critical Analysis

The key strength of this work is the clever use of influence functions to identify the most important training examples for fine-tuning. This allows the models to be trained much more efficiently, which is crucial for scaling up to large language models and real-world tasks.

One potential limitation is that the influence function computations can be computationally expensive, especially for very large language models. The paper does not provide a detailed analysis of the computational overhead of this step.

Additionally, the paper focuses on instruction-following tasks, but it's unclear how well In2Core would generalize to other fine-tuning scenarios. Further research is needed to understand the broader applicability of this technique.

Finally, the paper does not explore the robustness or generalization of the selected coresets. It's possible that models fine-tuned on a small coreset could be more brittle or have reduced out-of-distribution performance compared to full fine-tuning.

Conclusion

This paper presents a promising new technique called In2Core that leverages influence functions to enable data-efficient fine-tuning of large language models on instruction-following tasks. By selecting a small coreset of the most influential training examples, In2Core can achieve comparable performance to full fine-tuning, but with significantly fewer training examples.

This has important implications for scaling up language models to real-world applications, where labeled data can be scarce or expensive to obtain. The paper demonstrates the effectiveness of In2Core on several benchmark datasets, but further research is needed to explore its broader applicability and potential limitations.

Overall, this work represents an exciting advance in the field of efficient fine-tuning of large language models, and could pave the way for more data-efficient and scalable approaches to specializing these powerful models for specific tasks and domains.