Business Continuity: From Black Friday to Semester Start
First published at Wednesday, 21 October 2026
Business Continuity: From Black Friday to Semester Start
I work as a software architect at FernUniversität in Hagen on LEAD:FUH, a learning-analytics data platform handling highly sensitive student data. This post is adapted from our internal operations guidelines.
Unlike the data-platform topics elsewhere in this series, this is the domain I bring with me. I have done business continuity management (BCM) for large e-commerce platforms, environments where an hour of downtime has a price tag with many digits and where "where is the backup and when did you last restore it" is a question somebody asks you with a stopwatch running. After enough years of that, I cannot build a system without infrastructure as code, full automation, and a recovery story I have actually tested. By now this is muscle memory.
Nobody demands this here. A university research project has no auditor breathing down its neck, no revenue-per-minute dashboard, and no contractual recovery time objectives. We are aligning the platform with ISO 27001 [1] anyway, because from my experience the controls are simply what professional operations look like, and doing them from day one is dramatically cheaper than retrofitting them when somebody eventually does ask. And somebody (a data protection officer (DPO), a security office, a future operator) always eventually asks.
But you cannot copy an e-commerce BCM handbook onto a university platform. The discipline transfers, the priorities do not.
What stays: the non-negotiables
The core of the operational model would look familiar to any of my former colleagues. One principle carries everything:
Everything reconstructible from Git plus database backups equals full recovery capability.
The platform can be fully restored by a single engineer with access to the GitOps repository, the backup storage, and cluster credentials. Everything is infrastructure as code, reconciled by ArgoCD: The cluster state is the repository state, and a manual fix on a server will simply be overwritten on the next sync. Whatever is not in Git does not exist.
Recovery is tiered, and the tiers are designed up front instead of being discovered during an incident: Stateless services restore by Git sync alone. Databases restore from backup plus Git sync. The ingestion landing zone is deliberately not backed up at all, because it can always be re-ingested from the source systems, and backing up a re-creatable cache would only slow down every drill. Deciding what not to protect is as much a BCM decision as the backups themselves.
The e-commerce scar tissue shows most clearly with the backups: A backup which has never been restored is just a hope. A weekly automated job restores the latest backup into a scratch database, verifies the critical tables, and alerts on failure. Security patching is automated the same way. Dependency updates flow in nightly through Renovate, tiered by blast radius: Stateless app images auto-merge, databases require a human and a maintenance window.
None of this is exotic. It is the standard kit, applied without anyone requiring it.
What inverts: criticality
In commerce the criticality ranking is availability first, then everything else. The shop being down is the worst case, full stop.
On this platform the worst case is different in kind. Our analytics being unavailable for a day is an inconvenience: Students do not lose anything and researchers wait. But this platform processes student data under consent, and consent can be withdrawn at any time. This makes the consent table, a small, unglamorous PostgreSQL table, the most critical component of the entire system. If students cannot grant or revoke consent, or if the platform loses track of who has withdrawn it, we do not have an availability problem. We have a General Data Protection Regulation (GDPR) problem, and "the backup was eight hours old" is not a sentence anyone wants to say to a DPO.
The protection budget follows this ranking. The consent database gets the most aggressive backup cadence and retention on the platform, and its backup is the one the weekly restore test verifies by name. The alerts which page immediately are "consent service unavailable" and "backup failed repeatedly", while "dashboard slow" can wait. A machine learning score being a day late is, by explicit policy, a next-business-day issue.
The same inversion applies to time. E-commerce peak is Black Friday, ours is semester start. Outside business hours this platform does not need a pager. This luxury took me a while to accept, and it is itself a BCM decision: An honest "best effort outside business hours" is worth more than a fictional 24/7 commitment nobody is staffed to keep. Recovery objectives follow accordingly. A four-hour recovery point would be absurd for an order pipeline and is entirely proportionate for learning analytics. Writing relaxed objectives down felt strange after years of shaving minutes, but objectives you state and meet are the only ones that help you in an actual incident.
What is new: built to be handed over
One requirement has no e-commerce equivalent at all. A research project ends. The platform, if it earns its keep, does not. It transitions to the university's central IT for long-term operation. The people who built it will not be the people running it, and pretending otherwise is how universities accumulate infrastructure nobody dares to touch after the project team disperses.
This changes design priorities today, not at handover time:
Operating the platform must not require understanding it. Day-to-day operations like restarts, backup monitoring, restores, and rollbacks are runbook-driven and deliberately require no Python, no dbt, and no machine learning knowledge. The skill list for operators is short and standard: Kubernetes basics, database administration basics, Git.
Handover must be demonstrated, a signed document is not enough. The transition has acceptance criteria: The receiving team performs a full recovery from backup, a supervised rollback, and a single-database restore as drills, before accepting responsibility. A runbook which has never been executed by the person who will depend on it is fiction.
Escalation outlives the project. Tiered support with a defined path back to the architecture level is agreed on while the builders are still in the building.
E-commerce taught me to design for the 3am incident. The university version is designing for the incident after the team is gone, which is the more demanding constraint. It is also the same instinct which made us choose boring web standards for the frontend.
Summary
The BCM toolkit of infrastructure as code, GitOps, tested restores, automated patching, and drilled runbooks is professional practice, mandate or no mandate. What must be re-derived per context is the criticality model: What is actually unacceptable to lose, when your real peak season is, and who will be operating the system in five years. At a university those answers were different enough that copying the old handbook would have protected the wrong things.
References
- 1
- ISO/IEC 27001:2022. Information security, cybersecurity and privacy protection – Information security management systems – Requirements. The control catalogue we align with voluntarily; for German public-sector contexts, IT-Grundschutz by the Federal Office for Information Security (BSI) covers comparable ground.
Subscribe to updates
There are multiple ways to stay updated with new posts on my blog: