A pipeline that lives in notebooks works until someone has to change it. Logic is spread across cells, the run order is in someone's head, and nobody can say which downstream table breaks if a column is renamed. That was the state of the legacy ETL I re-architected from Databricks to dbt. This post covers the approach I used, in the order I'd use it again.
Inventory before you move anything
The first deliverable was a list, not a model. For every notebook I recorded:
- the tables it read and the tables it wrote
- whether it ran full refreshes or incremental loads
- what scheduled it, and what ran after it
- who consumed each output table
That list became the dependency graph dbt would later draw for us. It also showed which notebooks were dead: outputs nobody read, kept running out of habit. Those were deleted instead of migrated.
Pick the layers up front
dbt is easiest to live with when every model has an obvious home. I used three layers:
| Layer | Purpose | Example |
|---|---|---|
staging | One model per source table. Rename, cast, nothing else. | stg_orders |
intermediate | Joins and business logic that more than one mart needs. | int_orders_enriched |
marts | Tables people query, shaped around a business question. | fct_daily_revenue |
The rule that saves the most time: staging models only touch one source, and marts never read sources directly. When a source schema changes, exactly one model needs editing.
Translate logic, not code
Notebook code tends to mix three things: extraction, transformation and side effects such as writing files or sending alerts. Only the transformation belongs in dbt. Extraction moved to the ingestion layer, and side effects moved to the orchestrator.
For PySpark transformations, I rewrote the logic as SQL rather than porting it line by line. A cell that builds a DataFrame through ten chained calls is usually one select with a couple of CTEs. Where logic was repeated across notebooks, it became a macro or an intermediate model, so there is one definition to change.
Tests are the migration's acceptance criteria
Each mart got tests before it got consumers:
models:
- name: fct_daily_revenue
columns:
- name: revenue_date
tests:
- not_null
- unique
- name: order_count
tests:
- not_null
Generic tests catch nulls and duplicate keys. The tests that matter most during a migration are reconciliation checks: the new model and the old table should agree on row counts and key totals for the same date range. I wrote those as singular tests and deleted them once the old pipeline was retired.
Run both until the numbers match
Cutting over in one step is how migrations lose trust. Instead, the old notebooks and the new dbt models ran in parallel, writing to separate schemas. A daily comparison flagged any difference. Most of the differences came from one of three places:
- Time zones. The notebooks and the warehouse disagreed about what "today" meant.
- Null handling. Spark and SQL treat nulls in joins and aggregates differently in edge cases.
- Silent fixes. Someone had patched a notebook by hand at some point, and the fix only existed in that notebook.
Each difference was either a bug in the new model or an undocumented rule in the old one. Both are worth finding before consumers switch over.
What changed afterwards
The transformations became plain SQL that goes through code review like any other change. Lineage comes from dbt docs instead of from memory. Tests run in CI, so a broken assumption fails a pull request instead of a dashboard. Onboarding someone to the pipeline went from a walkthrough of notebooks to reading the model directory.
If you're planning a similar move, start with the inventory and the reconciliation checks. The SQL is the easy part.
