There’s a pipeline running somewhere in your company tonight that will download the same data it downloaded last night. And the night before. All of it, every time, so it can throw most of it away.
Nobody planned this. The script started as a proof of concept: pull everything, load everything, done in forty seconds. Then the source grew. Now the nightly run takes twenty minutes, the API provider is rate-limiting you, and the run occasionally dies at record 480,000 of 500,000 and starts over from record one. Full reload doesn’t fail loudly. It just gets a little more expensive every day, forever.
Why full reload survives
Full reload persists because it has one great property: it’s correct by brute force. There’s no state to corrupt and no gap to explain. When someone asks whether the data is right, the answer is yes, trivially, because you just rebuilt all of it.
Incremental extraction is where the real bugs live, and in thirty years of building this stuff I think I’ve been bitten by every one of them. The watermark that advances before the load commits, so a mid-run crash silently drops a window of records. The updated_at filter with a timezone mismatch that skips an hour of changes twice a year — you find that one in November, if you’re lucky. The retry that replays a batch into an append-only table and doubles a day’s revenue. My personal favorite: the hand-rolled checkpoint file that lives on one machine, in one directory, until somebody rebuilds the container and the pipeline quietly starts over from 2019.
So teams make a reasonable trade. Full reload is wasteful but safe, incremental is efficient but fragile, and wasteful-but-safe wins until the bill or the rate limit forces the issue.
I think the trade is framed wrong. The fragility was never intrinsic to incremental sync. It comes from where the state lives.
Agents made an old problem urgent
What changed is who writes the script. Increasingly it wasn’t your team. You described the source in a sentence, an agent generated the extraction code, and the code works. This is a real, current workflow, and it’s a good one.
But agent-written scripts have a property human-written ones don’t: they’re disposable by design. The source API adds a field, you regenerate. The pagination changes, you regenerate. You switch models and the new one writes it differently — fine, the script was never precious. Regeneration is the maintenance model.
Now ask what happens if that script also manages its own sync state. A timestamp in a local file, a cursor stashed in a variable, a checkpoint table it created for itself. Regenerate the script and the state is gone, or worse, half-gone. The new script doesn’t know about the old script’s checkpoint file. It starts from zero, or it invents a new checkpoint scheme, and now you have two. Every regeneration is a small migration nobody performed. The more freely you rewrite the code, which is the whole point of agent-generated pipelines, the more often you lose your place.
The code and the state have different lifespans, so they can’t live in the same place. The script is disposable. The bookmark is not. Disposable things get regenerated; durable things get held by the platform.
What the separation buys you
Put the sync state in the platform — outside the script, outside the container, outside anything an agent might rewrite — and the contract gets simple. Each run, the platform hands the script its bookmark: last timestamp seen, last ID processed, a continuation cursor, whatever fits the source. The script fetches from there and reports back where it ended. That’s the entire interface. The script stays stateless, which means it stays regenerable.
It also means the classic incremental bugs stop being the script author’s problem and become invariants you enforce once. The bookmark advances only on success, so a failed run leaves it untouched and the retry re-fetches the same window. No silent gaps, regardless of what the script’s error handling looks like. The script doesn’t have error handling for state, because it doesn’t own any.
The other half of the design is absorbing the overlap. At-least-once delivery means the same record can show up twice, so the destination has to upsert on key fields rather than blindly insert. This isn’t optional. A pipeline an agent operates will be re-run — on failure, on suspicion, on a whim — and every re-run has to be safe to repeat.
Backfills get their own rule: they don’t move the bookmark. Someone re-pulling last March by hand shouldn’t teleport the nightly schedule back five months. An explicit range is a read, not a checkpoint.
And the state has to be visible. When you suspect the data is wrong, the difference between a mystery and a five-minute fix is being able to see the bookmark, see when it was committed, and reset it for a clean re-fetch. Hand-rolled state hides in a file on a box somewhere. Platform state sits in the UI, and the agent can read and rewind it through the same tools you can.
None of this is novel. It’s the same discipline the Singer and Airbyte crowds converged on for connector state years ago. What’s new is who benefits: an agent that regenerates its extraction script on a Tuesday afternoon and loses nothing, because it never held anything.
Memory belongs to the system
There’s a broader pattern here, and it keeps showing up wherever agents touch infrastructure. Agents are ephemeral. Their context resets, their scripts get rewritten, their sessions end. Anything that has to survive — a memory, a code history, a sync position — needs a durable layer underneath them, one the agent uses but doesn’t carry.
This is the road we went down with Datris recently. Scheduled taps get a platform-held bookmark, committed only after a successful run, visible and resettable from the UI and from the agent’s own tools, with upserts absorbing the overlap on re-fetch. When the assistant generates a tap for a source that supports change detection, the script comes out incremental by default: modified-since filters, ID watermarks, cursors, content hashing, in that order of preference. But the argument doesn’t depend on any one product. Go find the pipeline in your shop that re-downloads the world every night. Then ask where its bookmark would live if it had one — and what happens to that bookmark the next time someone rewrites the script.
Todd Fearn is the founder of Datris.ai, an open-source, agent-native data platform built on the Model Context Protocol, and he runs IData Corporation, a data engineering consultancy for financial services firms. He has spent about thirty years building production data infrastructure inside institutions like Goldman Sachs, Bridgewater Associates, Deutsche Bank, and Freddie Mac.