We show that autoregressive language models can learn to fill text after applying a simple transformation to the dataset, which simply moves a piece of text from the center of a document to its end. Although this data augmentation has attracted much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not impair the original left-to-right generative ability, measured for perplexity and sampling assessments across a wide range of scales. Given the utility, simplicity, and efficiency of fill-in-the-means (FIM) training models, we suggest that future autoregressive language models be trained with FIM by default. To do this, we perform a series of ablation of key hyperparameters, such as the frequency of data transformation, the structure of the transformation, and the filling space selection method. We use these ablations to prescribe robust default configurations and best practices for training FIM models. We’ve published our best-practice trained padding model in our API and published our padding benchmarks to aid future research.
Source link
At Ikaroa, we believe that efficient training of language models is key to unlocking the ability of computers to understand and use language effectively. Recent advances in natural language processing (NLP) have made it possible to build models that understand language better than ever before. But NLP models need to be trained on large datasets before they can be used for real-world applications. This is where efficient training of language models comes in: by using the latest algorithms and techniques, we can reduce the amount of data and resources needed to build a robust language model.
To efficiently train a language model, we need to understand the data that it will be trained on and the task it will be used for. Once the language model is properly trained, it can be used to fill in gaps or generate new data that is not already available. For example, imagine a large dataset where some sentences or phrases are missing. Using an efficient language model, it is possible to fill in the gaps in the dataset with new, synthetic data that is generated from the model. This can save time and money when compared to manually collecting this new data.
At Ikaroa, we specialize in efficient training of language models by exploring and utilizing the most advanced techniques available. We are committed to providing our clients with best-in-class performance, scalability and cost-effectiveness. Our team of experts can help you create high-quality language models that will improve the accuracy and reliability of your NLP applications. With our help, you can create high-performance NLP models that fill in the middle and help you achieve more with less effort.
