MLOps in Practice: Getting Models Out of Notebooks and Into Revenue
Training-serving skew is the classic killer
The model performs beautifully in the notebook and poorly in production, and the reason is usually that features were computed differently in each place. In training, someone aggregated a customer's last 90 days of orders with a clean SQL query over a complete table. In production, that aggregate is computed from a partially-updated cache. The fix is to compute features once, in one place, and serve the same definitions to both paths — whether via a feature store or simply a shared library with tests. This single discipline prevents more failures than any other.
Version the data, not just the code
Reproducing a model six months later requires the code, the parameters, the environment and the exact training data. Teams routinely have the first three and not the fourth, which makes debugging a production regression guesswork. Snapshot training datasets, record the query and time bounds that produced them, and store that alongside the model artefact. When a regulator or a customer asks why the model made a decision in March, you need to be able to reconstruct March's model.
Deploy behind a flag and shadow first
Run the new model in parallel with the current one, serving live traffic but not acting on outputs, and compare distributions for a week or two. Shadow deployment catches feature pipeline breakage, latency surprises and unexpected input distributions with zero customer risk. Then move to a small traffic percentage with automatic rollback on error-rate or latency thresholds. This is standard software practice that machine learning teams often skip because the deployment is treated as a data science deliverable rather than a production change.
Monitor inputs, not only accuracy
Ground truth often arrives weeks late — you learn whether a churn prediction was right when the customer either leaves or doesn't. Meanwhile, monitor the inputs: feature distributions, null rates, category cardinality, prediction distribution. Input drift is visible immediately and is a leading indicator of accuracy decay. Most production model failures we've been called in to diagnose were upstream data changes, not model degradation, and they were visible in the input statistics days before anyone noticed the business metric.
Decide retraining policy deliberately
Scheduled retraining on a cadence is simple and predictable. Triggered retraining on drift detection is more responsive and more complex to operate. Both are better than the common third option, which is retraining when someone remembers. Whichever you choose, the retraining pipeline must include automatic evaluation against a held-out set and a gate that refuses to promote a worse model — automated retraining without an automated quality gate is a mechanism for silently deploying regressions.
Keep the operational footprint small
A team of three does not need a nine-component platform. Managed training and serving, a model registry, a scheduled pipeline, and monitoring will carry most organisations a long way. Add components when a specific pain justifies them. The most common self-inflicted wound in this space is building a platform for the scale you hope to reach, then spending your engineering capacity maintaining it instead of shipping models.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch