pretraining
AI, ML & GenAI interview questions tagged pretraining, across every topic.
5 questions · 0 unlocked for you
Concepts behind "pretraining"
The curriculum that explains the ideas these questions test.
Foundational
What an LLM Is: Next-Token Prediction and the Training PipelineAn LLM is a function that maps a sequence of tokens to a probability distribution over the next token, called in a loop. Three stages turn that function into an assistant: pretraining on a huge corpus buys knowledge and fluency, supervised fine-tuning teaches it to answer rather than continue, and preference alignment teaches it which answer a human prefers. AI, ML, and GenAI engineer interviews probe this because capability comes from pretraining while behavior comes from post-training, and almost every production complaint is a behavior complaint.🧠 Foundations of LLMs & GenAI
Foundational
Foundation Models and the Pretrain-Adapt ParadigmA foundation model is a single large model pretrained on broad data and then adapted to many tasks, replacing the old habit of training one bespoke model per task. The scarce resource moved: it used to be labeled data and training compute, and now it is evaluation and context. AI, ML, and GenAI engineer interviews probe this because it explains why a prototype takes an afternoon while a reliable product still takes a quarter, and because knowing when a gradient-boosted tree still beats an LLM is a senior signal.🧠 Foundations of LLMs & GenAI
Core
Scaling LawsScaling laws say model loss drops predictably as a power law in parameters, data, and compute, which is why bigger models trained on more data reliably improve. The Chinchilla result showed that for a fixed compute budget you should scale parameters and training tokens together (roughly equally), meaning prior large models were under-trained. This reshaped how compute is allocated and why smaller, data-heavy models hold up. AI, ML, and GenAI interviews probe it because it underlies model-selection and the data-vs-size economics.🧠 Foundations of LLMs & GenAISign in
Core
Transfer LearningTransfer learning reuses a model pretrained on a large general corpus as the starting point for a new task, so you inherit learned features rather than training from scratch. The two modes are feature extraction (freeze the backbone, train only a new head) and fine-tuning (unfreeze some layers and keep training), and the choice hinges on how much labeled data you have and how far the new domain has drifted. AI, ML, and GenAI engineer interviews probe it because it is the default for vision and NLP when labels are scarce, and because candidates often fine-tune when they should freeze, or the reverse.📊 Evaluation & ML FoundationsSign in