10How do you evaluate an LLM, and why are benchmarks and LLM-as-judge both unreliable?▼hard★ EssentialOpenAIAnthropicGoogle2 repliesunlockedEvaluation is the toughest and most underrated part of shipping LLMs. The signal is understanding why public benchmarks mislead, why LLM-as-judge carries bias, and how to build a task-specific eval you can genuinely trust.Open full answer →
71What's the difference between evaluating a model and evaluating the product around it, and why do you need both?▼mediumOpenAIAnthropicScale AI2 replies◆ premiumA model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that run only model evals get blindsided. Here is the distinction that matters.Open full answer →
97How would you evaluate a coding agent, and why is a single SWE-bench score not enough to trust it?▼hardOpenAIAnthropicCognition◆ premiumCandidates who cite one leaderboard number fail the follow-up. The interviewer wants you to name what SWE-bench measures, what it cannot (long-horizon multi-file refactors, real tool use, business logic), and which complementary benchmarks close those gaps.Open full answer →
03Design an evaluation and guardrail stack for an LLM feature: jailbreaks, toxicity, and hallucination.▼hard★ EssentialAnthropicOpenAIGoogle2 repliesunlockedGetting an LLM feature to production safely is an evaluation problem first and a model problem second. What lands is a layered eval-plus-guardrail design with honest, segmented metrics rather than a single 'safety classifier.' Here is how to measure and defend each failure mode.Open full answer →
