Modern Data Stack Migration: Snowflake, Databricks, or Neither
Pick the platform based on workload, not vendor comparison charts
If your workload is overwhelmingly SQL analytics and BI serving, a cloud warehouse is the straightforward answer and the ecosystem tooling around it is mature. If you have substantial data science, streaming and unstructured data alongside analytics, a lakehouse approach avoids maintaining two parallel platforms. If you are a mid-sized company with a few terabytes and predictable reporting needs, both may be overkill and a well-tuned managed Postgres or a cheaper columnar option will do the job for a fraction of the cost. The honest recommendation for smaller data volumes is often 'less platform than you were sold'.
Table formats matter more than they sound
Open table formats have reduced the lock-in question significantly — storing data in an open format on object storage means your compute engine becomes a more replaceable decision. That is worth structuring for even if you have no intention of switching, because it also lets different teams use different engines against the same data. The trade-off is a bit more operational complexity around table maintenance and compaction, which managed services increasingly handle for you.
Migrate consumers, not tables
The failure pattern is lifting every table across, then discovering nobody knows which of the 400 legacy tables anyone still reads. Instead, start from the outputs: list the dashboards, reports and downstream systems that matter, trace their lineage back, and migrate those paths first. Typically 20% of the legacy estate serves 90% of real usage, and the rest can be retired rather than migrated. Query logs from the old system will tell you which is which, and that analysis usually pays for itself immediately.
Run in parallel with automated reconciliation
For a defined window, both systems produce the same reports and an automated job compares them row by row, flagging differences above a tolerance. This is tedious and it is the only thing that gives finance and operations confidence to cut over. Set an explicit end date for the parallel period at the start, with criteria for cutover, because parallel running without a deadline becomes permanent and doubles your costs indefinitely.
Cost control has to be designed in
Consumption pricing means a single badly written dashboard refreshing every five minutes can cost more than a team of analysts. Set up per-team resource separation and budgets from day one, alert on anomalous spend daily rather than discovering it in the monthly invoice, and materialise expensive repeated queries instead of recomputing them. The teams that get surprised by cloud data bills are almost always the ones who turned on the platform before turning on the guardrails.
Governance is the part that outlives the migration
A catalogue with ownership, definitions and freshness expectations for the tables that matter is worth more a year later than any platform choice. If 'what does active customer mean' has three answers in three dashboards, migrating them all to a faster warehouse gives you the same disagreement, faster. Fix definitions during the migration while you have everyone's attention — it is the only time that political work is easy to justify.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch