80How do you choose the data mixture for pretraining an LLM, and what does domain reweighting buy you?▼hardGoogle DeepMindMetaMistral1 replies◆ premiumThe proportions of web, code, books, and math in pretraining quietly set downstream skills. The signal is knowing how mixtures get chosen, why upsampling helps, and how methods like DoReMi automate it.Open full answer →
81How do you do continued pretraining to adapt an LLM to a new domain without forgetting general ability?▼hardMetaDatabricksSnowflake1 replies◆ premiumContinued pretraining adds domain knowledge that fine-tuning cannot, yet done carelessly it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.Open full answer →
