A data pipeline fails in two ways and only one of them wakes anyone up, which is why the dangerous failure is rarely the one that stops a job and raises an alert. The silent failure keeps running, writes plausible but wrong numbers into a table somebody trusts, and gets discovered weeks later by a person questioning a figure in a report they were about to present.
The industry is currently making the second kind more likely. The dbt analytics engineering survey of 363 practitioners found 72 percent prioritising AI-assisted coding while only 24 percent prioritise AI-assisted pipeline management, which covers testing, observability and quality controls. Teams are accelerating how fast pipelines get built without matching investment in knowing whether they are correct.
What are the components of a data pipeline?
Every pipeline has four stages regardless of the tooling: extraction from a source, movement into storage, transformation into a usable shape, and delivery to whatever consumes it. The architectural decisions that matter are where the transformation happens and what the pipeline guarantees when a stage fails partway through.
That second question separates ETL from ELT more usefully than the acronyms do. Transforming before loading means less storage and a narrower blast radius when logic is wrong, because the bad data never lands. Transforming after loading means the raw record survives, so a logic error is fixable with a rerun instead of a re-extraction you may not be able to repeat.
| Stage | Decision that matters | Cost of getting it wrong |
|---|---|---|
| Extraction | Full load or incremental, and how change is detected | Missed or duplicated records at source |
| Loading | Raw retained or discarded after transform | A logic error becomes unrecoverable |
| Transformation | Before or after landing, and where it runs | Compute cost, or an unfixable mistake |
| Delivery | Push or pull, and the freshness contract | Consumers acting on stale data unaware |
How should you choose between batch and streaming?
Choose by how quickly a decision depends on the data, not by how modern the architecture sounds. Batch suits anything consumed on a human cadence, meaning reports, models retrained weekly and most analytics. Streaming earns its substantially higher operational cost only when a delay of minutes changes an outcome, such as fraud scoring or live personalisation.
The hidden cost of streaming is not the infrastructure, it is the failure semantics. A batch job that fails can be rerun from a known point. A stream that drops or duplicates events during an outage requires you to have decided in advance whether the system guarantees at-least-once or exactly-once delivery, and to have built consumers that behave correctly under whichever you chose.
Signals that batch is sufficient:
- Consumers look at the data on a daily or weekly rhythm
- Source systems themselves only update periodically
- Reprocessing a window is acceptable and cheap
- Nobody can name a decision that changes within the hour
How do you design pipeline orchestration?
Orchestration decides what runs, in what order, under what conditions, and what happens when a step fails. Model dependencies explicitly rather than by schedule, because a job timed to start after another is not the same as a job that waits for it to succeed, and the difference surfaces the first time the upstream run takes longer than usual.
Scheduling by clock is the most common structural mistake in a pipeline that otherwise works. It holds until the day an upstream job runs slow, at which point the downstream job runs happily against yesterday's data and produces output that is wrong in a way nothing detects. Dependency-based triggering costs slightly more to set up and removes an entire category of silent failure.
What orchestration must define:
- The dependency graph, not just the schedule
- Retry policy per step, with backoff and a ceiling
- What a partial failure leaves behind, and whether it is safe
- Who is alerted, for which failures, and what they are expected to do
- The maximum acceptable lateness before a run is abandoned rather than delayed
How do you recover from pipeline failures?
Recovery depends on three properties designed in from the start: idempotency so a rerun produces the same result rather than duplicates, replayability so you can reprocess a defined window, and isolation of bad records so one malformed row does not stop everything behind it. Pipelines missing these are not repaired after a failure, they are rebuilt.
Idempotency is the one most often skipped and most expensive to retrofit. A pipeline appending rows on each run will silently double its output when a retry fires, and the duplication is usually found in a report rather than a log. Writing by a deterministic key and upserting makes reruns safe, which in turn makes every other recovery mechanism usable.
| Mechanism | What it protects against | What it requires |
|---|---|---|
| Idempotent writes | Duplicates from retries and reruns | A deterministic key per record |
| Replay from a window | Logic errors found after the fact | Retained raw input and a time boundary |
| Dead letter handling | One bad record halting the run | A quarantine destination and a review owner |
| Backfill as a routine | Gaps from outages or late-arriving data | Capacity planning and a correctness check |
Backfills deserve explicit planning rather than improvisation. A backfill spanning months can cost more compute than a year of normal operation and can overwhelm downstream systems expecting a day's volume, which turns a recovery into a second incident.
Validation inside the run, meaning schema checks and contract tests at each transform boundary, is what catches the silent failures early. The broader judgement about whether data is fit to train on or decide with sits in our guide to data readiness signals.
What drives the running cost of a pipeline?
Running cost is driven by how much data moves, how often it moves, and how much is recomputed that did not change. Most expensive pipelines are not expensive because of volume but because they reprocess everything on every run when an incremental approach would touch a fraction of it.
Three habits account for most avoidable spend. Full refreshes where incremental loads would work, transformations running on raw data that is filtered immediately afterwards, and retention of intermediate outputs nobody reads. Each is invisible on a monthly invoice that reports totals rather than causes.
Where the money usually goes:
- Full table scans caused by missing partitioning or clustering
- Recomputing unchanged historical partitions on every run
- Streaming infrastructure for data consumed daily
- Intermediate tables retained indefinitely with no owner
- Over-provisioned warehouses sized for a monthly peak
What separates a pipeline that scales from one that breaks?
Pipelines that scale share four properties: they are incremental by default, idempotent on every write, explicit about their dependencies, and observable at the record level rather than only at the job level. Pipelines that break are usually fine at first and fail once volume, source count or team size crosses a threshold nobody was watching.
Observability at the record level is the property most often missing. Knowing a job succeeded tells you nothing about whether it processed the expected number of rows, whether a source silently returned an empty set, or whether a column arrived null for the first time. Those are the conditions that produce confident wrong answers.
The check most teams do not run. Compare row counts and key distributions against the previous run, every run, and alert on deviation beyond a threshold. A job that succeeds while processing a tenth of the usual volume is the single most common silent failure.
Should you build a pipeline or buy one?
Buy when your sources are common and your transformations are standard, build when either is unusual enough that configuration becomes harder than code. Managed connectors handle well-known sources well and become expensive or limiting at high volume or against unusual systems, which is where the economics invert.
The decision is rarely permanent and does not have to be uniform. Most mature setups are hybrid, using managed ingestion for standard sources and custom code where the source is proprietary or the volume makes per-row pricing untenable. Who maintains which half is the question that actually needs answering, and our comparison of data engineering roles sets out where that responsibility belongs.
Building pipeline capability
The engineers worth having on this are the ones who ask what happens on the second run before they ask what the schema looks like. That instinct is the difference between a pipeline that survives its first outage and one that silently corrupts a month of history.
Our matching works against your source systems, your volume and the recovery guarantees you actually need rather than a generic data engineering brief.
