Berkeley-nest
Models by this creator
📉
Starling-LM-7B-alpha
549
Starling-LM-7B-alpha is a large language model developed by the Berkeley NEST team. It is based on the Openchat 3.5 model, which in turn is based on the Mistral-7B-v0.1 model. The key innovation of Starling-LM-7B-alpha is that it was trained using Reinforcement Learning from AI Feedback (RLAIF), leveraging a new dataset called Nectar and a new reward training and policy tuning pipeline. This allows the model to achieve state-of-the-art performance on the MT Bench benchmark, scoring 8.09 and outperforming every model to date except for OpenAI's GPT-4 and GPT-4 Turbo. Model inputs and outputs Starling-LM-7B-alpha is a text-to-text model, taking natural language inputs and generating text outputs. The model uses the same chat template as the Openchat 3.5 model, with the input formatted as Human: {input}\n\nAssistant: and the output being the generated text. Inputs Natural language prompts**: The model can accept a wide variety of natural language prompts, from open-ended questions to task-oriented instructions. Outputs Generated text**: The model outputs generated text that is relevant to the input prompt. This can include responses to questions, explanations of concepts, and task completions. Capabilities Starling-LM-7B-alpha demonstrates strong performance on a variety of benchmarks, including MT Bench, AlpacaEval, and MMLU. It outperforms many larger models like GPT-3.5-Turbo, Claude-2, and Tulu-2-dpo-70b, showcasing its impressive capabilities. The model is particularly adept at tasks that require language understanding and generation, such as open-ended conversations, question answering, and summarization. What can I use it for? Starling-LM-7B-alpha can be used for a variety of applications that require natural language processing, such as: Chatbots and virtual assistants**: The model's strong performance on conversational tasks makes it well-suited for building chatbots and virtual assistants. Content generation**: The model can be used to generate a wide range of text-based content, from articles and stories to product descriptions and marketing copy. Question answering**: The model's ability to understand and respond to questions makes it useful for building question-answering systems. Things to try One interesting aspect of Starling-LM-7B-alpha is its use of Reinforcement Learning from AI Feedback (RLAIF) during training. This approach allows the model to learn from a dataset of human-generated rankings, which can help it better understand and generate responses that are more aligned with human preferences. Experimenting with different prompts and tasks can help you explore how this training approach affects the model's behavior and outputs.
Updated 5/28/2024
🤿
Starling-RM-7B-alpha
95
Starling-RM-7B-alpha is a reward model developed by the berkeley-nest team. It was trained from the Llama2-7B-Chat model using the method of training reward models described in the instructGPT paper. The model was further trained on the berkeley-nest/Nectar dataset, a preference dataset based on GPT-4 preferences. The Starling-RM-7B-alpha model outputs a scalar reward score for any given prompt and response pair. Responses that are more helpful and less harmful will receive a higher reward score. This reward model is likely biased towards GPT-4's preferences, including longer responses and certain response formats. Similar models developed by the berkeley-nest team include the Starling-RM-34B and Starling-LM-7B-alpha models. The Starling-RM-34B model is trained on the same method but uses the larger Yi-34B-Chat as the base model, while the Starling-LM-7B-alpha is a language model trained using the berkeley-nest/Starling-RM-7B-alpha reward model. Model Inputs and Outputs Inputs Prompt**: A piece of text that the model will evaluate and provide a reward score for. Response**: A candidate response to the provided prompt. Outputs Reward Score**: A scalar value representing the model's assessment of how helpful and harmless the given response is for the prompt. Capabilities The Starling-RM-7B-alpha model is able to assess the helpfulness and harmlessness of text responses based on the training data and methodology used. It can be used to rank and compare different responses to the same prompt, favoring those that are more aligned with the preferences in the training data. The model's performance is benchmarked on datasets like Truthful QA, Chatbot Arena Conversations, and PKU's Safe-RLHF, with the Starling-RM-34B model outperforming the Starling-RM-7B-alpha across all these metrics. What Can I Use It For? The Starling-RM-7B-alpha model can be used as part of a reinforcement learning pipeline to train large language models to be more helpful and less harmful. By providing reward scores for model outputs during training, the model can be optimized to generate responses that are aligned with the preferences in the training data. This type of reward model can also be used to evaluate the outputs of other language models, helping to identify responses that may be problematic or undesirable. The model could potentially be integrated into chatbot or virtual assistant applications to help ensure the system behaves in a way that is beneficial to users. Things to Try One interesting thing to try with the Starling-RM-7B-alpha model is to compare its reward scores for different responses to the same prompt. This could help surface nuances in how the model assesses helpfulness and harmlessness. It would also be worth exploring how the model's performance compares to the larger Starling-RM-34B model, and whether the differences in reward scores align with human assessments. Additionally, it could be insightful to probe the model's biases by crafting prompts or responses that play to the preferences in the berkeley-nest/Nectar dataset, and see how the reward scores are affected. This could shed light on the model's limitations and areas for improvement.
Updated 5/28/2024