A week ago we announced RedPajama, a project to create leading open-source models. We released the first step in the project a training dataset of over 1.2 trillion tokens following the LLaMA recipe.
Today we shared progress on training our first model on this dataset, a 7B parameter model using the Pythia architecture. So far we are a bit less than 50% through the training - 440B parameters. We published HELM benchmark results on 16 different scenarios for this checkpoint, showing the model accuracy to be quite high for this stage of training.
Hi! I lead Product at Together. We will be releasing a full suite of models trained on this data starting with the first models in the coming weeks. We will release RedPajama base models and RedPajama instruction-tuned models. All of the models will be released under the Apache 2.0 license, allowing commercial use.
Therefore, anyone will be able to fine-tune the RedPajama models using Vicuna or other datasets, given they will be fully open-source.
The RedPajama instruction-tuned models will be fine-tuned only with instruction labels from human labelers and OpenChatKit feedback (). We feel this will keep these models fully "clean" for use in commercial applications without using the output of other commercial models like were used in Alpaca or Vicuna. However, we'll be excited to see all the great fine-tunes created by the open community and are eager to see how close open-source models can get to the quality of leading commercial models over time!!
Hi! I lead Product at Together. We will be releasing a full suite of models trained on this data starting with the first models in the coming weeks. We will release base models and instruction-tuned models. All of the models will be released under the Apache 2.0 license, allowing commercial use.
Hi Everyone, I work at Together. Today we released OpenChatKit: an open-source base to create chatbots for various applications. More than a model release, this is the beginning of an open source project. We are releasing a set of tools and processes for ongoing improvement with community contributions.
1. An instruction-tuned large language model, fine-tuned for chat from EleutherAI’s GPT-NeoX-20B with over 43 million instructions on 100% carbon negative compute available under Apache-2.0 license on Hugging Face.
2. A set of customization recipes to fine-tune the model to achieve high accuracy on your tasks documented and available as open-source under the Apache-2.0 license on Github, along with code to recreate our model results.
3. An extensible retrieval system enabling you to augment bot responses with information from a document repository, API, or other live-updating information source at inference time, with open-source examples for using Wikipedia or a web search API.
4. A moderation model, fine-tuned from GPT-JT-6B, designed to filter which questions the bot responds to, also available under the Apache-2.0 license on Hugging Face.
We collaborated with the tremendous communities at @laion_ai and Ontocord to create the training dataset used for these models, also released as open-source. Read the full details on LAION's blog post!