99How does a vision-language model connect an image encoder to an LLM, and where does it fail?▼hardOpenAIGoogle DeepMindMeta2 replies◆ premiumVLMs such as GPT-4V and LLaVA attach a vision encoder to a language model through a projector. The signal is the image-tokens-as-prefix design, the alignment training, and the resolution and hallucination failure modes.Open full answer →
110Explain multimodal fusion: early, late, and cross-attention. Which architecture would you pick, and when?▼mediumOpenAIGoogle DeepMindMeta◆ premiumFour fusion families, and the choice between them is decided by one question most candidates never ask. Naming CLIP and LLaVA is table stakes; the score comes from knowing which one cannot reason and which one eats your context window.Open full answer →
134How do you fine-tune a vision-language model, and what do you freeze?▼hardOpenAIGoogle DeepMindMeta◆ premiumA VLM is three parts and the interview is entirely about which ones you train. The staged recipe, why the vision encoder almost always stays frozen, and the silent failure where your model learns to answer from the text prior and never looks at the image.Open full answer →
