Definition
Models trained on large-scale language data that can predict, generate and process language sequences.
This entry focuses on modern large-scale pretrained language models. The 2020 GPT-3 paper demonstrated task adaptation by an autoregressive model in few-shot contexts. It is a representative example, not the starting point of language modeling or pretraining.
Pretraining, contextual prompting and post-training serve different purposes. Generating fluent text does not guarantee factual accuracy, nor does it alone establish general intelligence.
Principles and impact
Problems solved
- Reuses representations and generation capabilities acquired through large-scale language training to handle a variety of language tasks.
Principles
- Learns language distributions through training objectives such as sequence prediction, and generates outputs based on the input context.
Improvements
Not recorded
Significance
- Enabled one model to serve multiple kinds of language tasks through prompting and adaptation.
History and applications
People
Tom B. Brown
Organizations
Not recorded
Events
Not recorded
Applications
Text generation、Question answering、Adaptation to language tasks
Limitations
- May generate incorrect content and is constrained by training data, context and task distributions.
Upstream and downstream relations
Upstream
Sources: Training Language Models to Follow Instructions with Human Feedback
Sources: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Sources: Language Models are Few-Shot LearnersAttention Is All You Need
Downstream
Sources: ReAct: Synergizing Reasoning and Acting in Language Models