Designing a dbt Pipeline for Consent Withdrawal
First published at Wednesday, 9 September 2026
Designing a dbt Pipeline for Consent Withdrawal
I work as a software architect at FernUniversität in Hagen on LEAD:FUH, a learning-analytics data platform handling highly sensitive student data. This post is adapted from our internal engineering documentation. Data platforms are a new domain for me, and this is what standard architecture discipline produced when the requirement "students can withdraw consent at any time" met a dbt pipeline.
The requirement that shapes everything
Our platform processes student data only with consent, and consent can be withdrawn at any time. Withdrawal has teeth: The student's pseudonymised data must disappear from the platform, promptly and completely.
At the same time GDPR is clear that irreversibly anonymised data is no longer personal data. A course's completion rate aggregated over hundreds of students does not become illegal because one of them withdrew consent. And losing years of research statistics to every withdrawal would make the platform useless for some of its purposes.
So two things must be true at once:
Withdrawal purges everything pseudonymised about the student.
Properly anonymised aggregates survive withdrawal.
The naive approach is one pipeline with a cleanup script which deletes rows on withdrawal. It fails the moment you look at lineage: If an anonymised aggregate is computed downstream of pseudonymised tables, then purging those tables either breaks the aggregate's next rebuild or silently changes its history. You end up choosing between a GDPR violation and data loss.
The conclusion we drew: Consent withdrawal is an architectural property of the dependency graph, not a deletion feature.
Two lanes through the warehouse
Our dbt project (the "Core Zone") is organised as three vertical lanes and four horizontal layers. The layers are the familiar dbt progression raw → staging → intermediate → marts [1] . The lanes determine deletion behaviour:
┌─────────────┐
│ sources │
└──────┬──────┘
│
┌──────▼──────┐
│ raw │ pseudonymized on ingestion
└──────┬──────┘
│
┌──────▼──────┐
│ staging │ last shared layer
└──┬───────┬──┘
│ │
pseudonymized│ │anonymized
│ │
┌──────▼──┐ ┌──▼──────┐
│ int_* │ │ int_* │ independent from here
└──┬──────┘ └──────┬──┘
│ │
┌──▼──────┐ ┌──────▼──┐
│ marts │ │ marts │
└─────────┘ └─────────┘The pseudonymised lane holds data with pseudonymous IDs, reversible via a separately held consent mapping. On withdrawal this lane is purged.
The anonymised lane holds irreversibly anonymised data with no path back to a person. It is never deleted on withdrawal, by design and with legal backing.
A third, small consent lane holds the consent state itself and the audit trail. On withdrawal it is updated, not deleted, because the record that consent was withdrawn must itself be kept.
Both lanes branch at staging and build independently from there. This brings us to the rule which makes the whole design work.
The one rule that must hold
An anonymised model must never reference a pseudonymised model.
If an anonymised aggregate read from a pseudonymised intermediate, purging that lane would break the aggregate's rebuild or, worse, quietly recompute history without the withdrawn student, making the "anonymised" numbers a function of current consent state. Both lanes therefore descend from the shared staging layer and nothing later.
What I like about this rule is its shape: It is a dependency-direction rule, the same species as "the domain layer must not import the infrastructure layer" in application architecture. It is checkable mechanically against the dbt graph. A CI job walks the manifest and fails on any ref() from the anonymised lane into the pseudonymised one. Twenty years of layered-architecture instincts translated directly, only the stakes changed from maintainability to legality.
Materialization follows deletion semantics
The standard advice for materializing dbt models is driven by query performance and build cost. Consent withdrawal adds a third force, and for us it wins:
Intermediate models are views. A view has no stored copy, so consent filtering and purges take effect at read time, immediately. A materialized intermediate would hold a stale copy of a withdrawn student's rows until the next run, a window in which we would be processing data without consent.
Marts are tables. Consumers need predictable query performance at the serving edge. The cost: After a purge, the pseudonymised marts must be rebuilt promptly. This is a single targeted dbt run, small enough to schedule aggressively.
Staging is views, with one exception: When merging activity from multiple source systems into one staging model, that model is a table. Both lanes must see an identical snapshot of the merged input, or the pseudonymised and anonymised lanes drift apart in subtle ways.
None of these choices is what a performance-first tuning pass would produce. All of them fall out naturally once you ask of every model not "how fast does it build" but "what happens to it when a student withdraws consent tomorrow".
Defense in depth: the reaper and the gateway
You may have spotted the gap: Marts are tables, so between a withdrawal and the next rebuild there is a stale materialized copy of the withdrawn student's rows. The lane structure alone does not close that window. Two more mechanisms do, and the three together are the actual design:
The gateway filters at serving time. All data leaves the platform through a serving gateway, and the gateway checks consent state on every request. The moment consent is withdrawn, the student's pseudonymised data stops being served, immediately, regardless of what is still sitting in a materialized table. There is no second door around it.
A reaper job purges at storage level. A scheduled job physically deletes withdrawn students' rows from the pseudonymised lane and triggers the mart rebuilds. This is the mechanism which makes the deletion real instead of merely invisible: Backups age out, tables get rebuilt, and after the reaper has run, the data is gone, not filtered.
Each layer covers a different failure mode. The dependency rule guarantees nothing anonymised ever depends on data which can vanish. The gateway guarantees withdrawn data is never exposed, even during the stale window. The reaper guarantees the data eventually ceases to exist. Remove any one of them and there is a scenario (a delayed rebuild, a bypassed filter, a forgotten copy) where consent withdrawal silently stops meaning what we promised students it means.
Summary
Privacy requirements are usually bolted onto pipelines as filters and cleanup jobs, and as an earlier post argued about classification, bolted-on privacy decays into guesswork. The consent architecture is the same lesson at the structural level: When deletion semantics are a hard requirement, they belong in the dependency graph, where a rule can be enforced by CI, not in operational procedures, where a rule can be forgotten.
We let consent handling decide the shape of the pipeline instead of adding it to a finished one. The result is almost boring: Two lanes, one forbidden dependency direction, views where immediacy matters and tables where speed does, a gateway which filters and a reaper which deletes. Boring scales, and here boring also complies.
References
- 1
- dbt Labs. "How we structure our dbt projects." docs.getdbt.com/best-practices/how-we-structure/1-guide-overview — the staging / intermediate / marts layering this post builds on.
Subscribe to updates
There are multiple ways to stay updated with new posts on my blog:
