training serving skew

Two problems wear one costume here: how you would have caught this before full rollout, and how you debug it after the fact. Cover both and you show senior judgment. This walks through the staged-rollout and root-cause playbook.

Online models need features that are fresh and computed the same way training computed them. The classic failure is training-serving skew that quietly degrades the model. The signal is the online/offline split built on one shared definition. Here is the design.

A feature store is simple to define and simple to botch. The signal is naming the precise bug it prevents (training-serving skew) and the point-in-time correctness that blocks label leakage. Here is the full picture.

Same model, two answers: clean offline, ugly online. The cause is almost always a feature computed differently across the two paths. Here is the diff-based hunt that localizes it to a single column.