Write-Ahead Logs on S3

How a durable history in S3 lets read replicas scale independently of write coordination, using immutable changes, snapshots, and conditional publication.

August 2026

Cursor’s Git at any scale describes Continuity, a Git storage system whose durable history lives in a write-ahead log (WAL) in S3. Servers reconstruct local repositories from that history. The service can add read replicas as demand grows, while writes become durable through S3 rather than through agreement among those replicas.

I find the independence of those two paths particularly useful. A repository can be large while a push adds comparatively little data. Many servers may need the complete repository locally to serve reads efficiently, but making every copy participate in accepting a push would tie write coordination to the number of readers. An authoritative history in object storage lets the copies follow accepted changes at their own pace.

The general pattern combines immutable changes, periodic snapshots, and a small mutable head that identifies the accepted history. A writer stores the new bytes, then conditionally advances that head. Readers reconstruct their working state from the published history and check whether they have caught up before serving a request that requires fresh data.

This allows read capacity to grow without adding participants to the commit protocol. The application relies on S3’s consistency and conditional writes for publication, avoiding a separate coordination database or an application-managed consensus group for that history. The useful boundary is equally specific: writes to one history still pass through one head, and the application must be able to reconstruct and validate its state from the changes it records.

How Continuity handles a push

In Continuity, a push produces an immutable entry containing its packfile, the bundle of Git objects being transferred. The host writes those bytes to local disk while uploading them to S3. A separate WAL index records accepted entries in order. An uploaded packfile becomes part of the repository’s published history only when the index references it.

The host prepares the Git reference transaction against the repository state it intends to extend. This checks the expected branch values and holds the relevant local locks. It then conditionally publishes the new WAL index. After successful publication, it commits the local reference transaction and acknowledges the push. The durable publication is what lets another host recover the accepted push if this host fails.

Other replicas catch up from S3. Before serving a read, a replica checks the index and applies missing history. Notifications help it catch up earlier, while checking the authoritative index makes missed notifications recoverable. Replicas can be added for a busy repository or removed when idle without changing where accepted data lives.

The reusable idea is that one durable history can support many independently maintained working copies. Git supplies a particular representation of changes and rules for validating them. To apply the architecture elsewhere, those properties must exist in the application’s own state model.

Snapshots and incremental changes

A log is useful when recording an update is substantially cheaper than replacing the state it changes. A full snapshot might occupy gigabytes, while an accepted operation contributes a much smaller batch of new data. Uploading a complete snapshot after every operation would repeatedly transfer almost identical contents. Recording the changes preserves the same progression with less repeated work.

Readers still need a starting point. A snapshot captures the state at a known version, and the ordered changes after it form a tail that brings the snapshot forward. A server can reconstruct version 42 by loading snapshot 40 and applying changes 41 and 42. A server already at version 41 needs only the final change.

Consider a dataset made up of immutable data files in an application’s S3 bucket. Its working state is the set of files currently included in the dataset. A snapshot records that set, and each change records additions and removals. The following layout is an illustrative protocol for this article, rather than Continuity’s own file format:

datasets/events/
├── data/part-003.parquet
├── data/part-013.parquet
├── data/part-014.parquet
├── snapshots/snapshot-c8e2.json
├── changes/candidate-a7c9.json
├── changes/candidate-f3b6.json
└── head.json

Suppose version 41 includes part-003.parquet and part-013.parquet. A writer wants to replace the first file with part-014.parquet, leaving the second alone. After uploading the new data file, it records the proposed change in changes/candidate-f3b6.json:

{
  "operationId": "op-a",
  "baseStateVersion": 41,
  "add": ["data/part-014.parquet"],
  "remove": ["data/part-003.parquet"]
}

Applying this entry to version 41 would produce a file set containing part-013.parquet and part-014.parquet. Removing a reference does not immediately delete the old file from S3; an older reader may still need it. The large data files stay immutable, while a small entry describes the transition between dataset versions.

This requires a precise replay rule. Given an accepted prior state and an entry, reconstruction must produce the intended next state. Generated identifiers, timestamps, or outside results needed for that reconstruction must be recorded rather than generated again during replay. The log can preserve resolved decisions, but replaying it should not repeat an external payment or another side effect.

For small state or infrequent updates, copying a consistent snapshot may be simpler. The log earns its extra machinery when the size difference and update frequency make full replacements expensive, especially when many readers need to maintain local copies.

Publishing through a shared head

Immutable uploads alone cannot tell a reader which changes were accepted. A writer may crash after an upload, or two writers may prepare conflicting successors. The head resolves that ambiguity by naming the snapshot and ordered tail that readers should use:

{
  "revision": 57,
  "stateVersion": 42,
  "snapshot": {
    "version": 40,
    "key": "snapshots/snapshot-c8e2.json"
  },
  "changes": [
    { "version": 41, "key": "changes/candidate-a7c9.json" },
    { "version": 42, "key": "changes/candidate-f3b6.json" }
  ]
}

This head describes version 42, after the file replacement is accepted. Snapshot 40 and change 41 reconstruct the prior file set; change 42 applies the replacement. The head shown here contains the reconstruction fields. We will add the evidence needed for safe retries below.

The object names use illustrative shortened identifiers. Each distinct candidate change or snapshot gets a unique immutable key, independent of the state version it might represent. Two writers proposing version 42 can therefore upload different candidates; the head determines which one becomes accepted. The operation ID identifies the logical request and remains the same if a retry needs a rebuilt candidate.

The head is the entry point for discovery. An object becomes part of the accepted history when a successful head update references it. Until then, readers ignore it. The writer must finish uploading both the change entry and every new data file it references before publishing the head. Otherwise, a reader could discover an accepted version that it cannot reconstruct.

A writer reads the current head and its ETag, the comparison token returned by S3. It validates the proposed operation against the corresponding state, uploads its immutable data, and prepares a successor head. It then submits that replacement with If-Match, supplying the ETag from its read.

S3 accepts the replacement only if the head still matches that token. This is compare-and-swap (CAS). If another writer has advanced the head, the stale attempt fails instead of erasing the newer history. The ETag is an opaque comparison token; the example’s state version identifies application state.

The separate revision increases on every head replacement, including maintenance that leaves application state unchanged. Within a history, the head is never reset to an earlier revision. Restoring older data must publish a new revision rather than reinstate old head bytes, so a delayed comparison cannot become valid again merely because the application returns to an earlier state.

Strong read-after-write consistency and conditional writes let the data and the publication record live in S3. The application can delegate this serialization point to the storage service instead of keeping the head in a separate database. Coordination remains necessary, but it is supplied through the storage API.

What conditional writes add

Strong consistency alone does not protect a read-modify-write sequence. Two writers can both read version 41, prepare different replacements, and overwrite the head in succession. Every S3 read could be perfectly current while the second replacement still drops the first writer’s accepted change. The missing condition is that each replacement must depend on the head its writer actually examined.

S3 introduced strong read-after-write consistency in December 2020. The write conditions followed in 2024: create-if-absent writes in August, then ETag-based conditional replacement in November. Together, these primitives support the immutable uploads and mutable head used here without a separate service to serialize that head.

Independent read replicas

Once the head is authoritative, a writer can finish publication without waiting for every read replica to apply the change. A slow replica delays its own readiness. It does not have to delay acceptance for replicas that are already caught up or for the writer publishing the next change.

Adding a replica therefore means giving another server enough history to reconstruct a local working copy. It reads the head, loads the snapshot, and applies the tail. Subsequent changes arrive through the same path. The writer continues publishing to the same storage objects regardless of how many replicas are following them.

This also separates durability from the number of running copies. If the complete accepted history is in S3, a service can discard idle working copies and rebuild them later. A busy history can have many replicas, a quiet one can have one, and an inactive one can have none. Keeping a warm copy is a latency and cost decision.

Read scaling is still bounded by resources. Each replica needs compute and local storage, and each must receive and apply the changes it serves. Starting many replicas at once consumes download bandwidth and storage requests. The architectural benefit is that increasing the number of readers does not enlarge an application-level consensus group or require the writer to collect more replica acknowledgments.

Read freshness

A replica can be behind even when the history in S3 is strongly consistent. A service has to decide when that lag is acceptable. Requests for a fixed historical version can use a copy of that version. Requests requiring current state need a check against the authoritative head.

A replica can remember the ETag of the head it has fully applied and send a conditional read using If-None-Match. A 304 Not Modified response confirms that the head still matches. If S3 returns a newer head, the replica downloads and applies the missing history before answering. It serves a coherent version, including any write completed before that authoritative check.

A write may arrive while the replica is serving the request. The read can still be consistent with the version established by its check; it need not chase a moving head forever. The application must keep the local view coherent while reading it, rather than expose a partially applied transition.

Notifications can reduce the amount of catching up a request triggers. They are a performance aid because the head supplies the final freshness check. If S3 is unavailable, a replica promising that check cannot silently substitute its cached copy. It must fail, wait, or use an explicitly weaker stale-read contract.

Concurrent writers

The freshness check tells readers which state to use. Writers need that same state for a different reason: deciding whether a proposed change is valid. In the dataset example, a replacement requires the old file to still belong to the current dataset. Other applications may check an account balance, an expected document revision, or a database page layout.

S3 does not evaluate those rules. It checks the head’s ETag, which ties publication to the state against which the application performed its validation. A valid candidate must remain private until publication succeeds; serving it early could expose a change that loses the race.

Two writers may both prepare a successor to version 41. Both immutable uploads can succeed, but only one head replacement can match the original ETag. If writer A publishes first, writer B receives 412 Precondition Failed and has to inspect the new history.

Writer B cannot simply overwrite the new head with its old proposal. It must preserve A’s accepted change and determine whether its own intent is still valid. Some operations can be evaluated again against the newer state; others must be rejected. Any new state-dependent output needs to be rebuilt under a new immutable key rather than changing bytes already stored.

Suppose A publishes our replacement of part-003.parquet with part-014.parquet. If B wanted to add an unrelated file, that addition may still be valid against version 42. If B also intended to replace part-003.parquet, its precondition has failed: that file is no longer current. Simply appending B’s old change could keep two incompatible replacements or lose A’s work. The application must reject B or recompute its request under explicitly defined conflict rules.

This distinction determines how much ownership machinery the application needs. A design whose operations can safely be revalidated may allow several servers to attempt publication and let conditional writes resolve their races. Routing writes preferentially to one server can reduce contention without making that routing decision the source of correctness.

Page changes from independently modified SQLite databases, for example, cannot simply be concatenated into one valid history. Page-based replication commonly uses a single write owner, with fencing to prevent a former owner from publishing stale work. The general pattern avoids a separate coordination service for publishing a history; it does not make every possible state representation safe for concurrent writers.

Retries after a lost response

A timeout does not tell a writer whether publication failed. S3 may have accepted the new head before its response disappeared. Retrying the logical operation without checking could apply it twice, even though each individual conditional write behaved correctly.

Recovery needs stable identity. A logical operation keeps the same operation ID across attempts. Before its first publication attempt, the service durably records the request contents and the state version from which it was first issued. Every retry uses that same record, even after a process restart. In our example, op-a was issued from version 41, so its earliest possible accepted version is 42.

{
  "operationId": "op-a",
  "issuedFromVersion": 41,
  "request": {
    "type": "replaceFiles",
    "add": ["data/part-014.parquet"],
    "remove": ["data/part-003.parquet"]
  }
}

One way to store that request record is a create-if-absent object keyed by the operation ID. Competing attempts read and reuse whichever record was created first, checking that its request contents match. They must not replace its initial version with the latest head version. A hash of a canonical request representation, with generated values fixed, lets the service detect the same ID being reused for different intent.

The example retains these immutable request records even after their acceptance receipts expire. Reclaiming them would need an enforced request-expiry rule as well: deleting a record must not let the same ID return with a newer initial version. The bounded receipt window limits the mutable head’s size, while retained request records remain a per-operation storage cost.

Recording acceptance with the change

The request record proves what was attempted. To prove acceptance, the head can retain a small receipt binding the operation ID to the request hash, accepted version, and result. The same conditional replacement publishes both the state change and its receipt. This follows AWS’s guidance on idempotent APIs: recording the request token and applying the mutation must be atomic.

For the worked example, add these fields to the version-42 head shown earlier. The hashes and IDs are shortened for readability. op-prev identifies the preceding change at version 41; op-a is our file replacement:

{
  "receiptCoverageFrom": 41,
  "recentReceipts": [
    {
      "operationId": "op-prev",
      "requestHash": "sha256:19c4...",
      "acceptedVersion": 41,
      "result": { "stateVersion": 41 }
    },
    {
      "operationId": "op-a",
      "requestHash": "sha256:7ab0...",
      "acceptedVersion": 42,
      "result": { "stateVersion": 42 }
    }
  ]
}

receiptCoverageFrom means that every accepted operation from that state version through the current one has a retained receipt. Here the coverage is 41 through 42. Every successor preserves all receipts in its advertised range. It can discard an older prefix only by advancing the coverage boundary, so a retry can distinguish a missing operation from missing evidence.

These fields make the write procedure more precise. Every attempt follows the same checks against one observed head:

1. Load the stable request record, then read the current head and its ETag.
2. If a receipt contains the operation ID, verify the request hash and return the recorded result. A different hash is an error.
3. If the ID is absent, check that receiptCoverageFrom is no greater than issuedFromVersion + 1, the operation’s earliest possible accepted version. Otherwise return an unknown outcome.
4. Validate the request against the state described by that head. Build the candidate and upload every new object it needs, using immutable keys.
5. Prepare a successor that preserves the required receipts, appends this operation’s receipt, and references its candidate. Advance the state version and head revision.
6. Publish with If-Match against the ETag from the read. On a failed comparison or a lost response, return to the head read with the same request record.

A retry may overlap the original attempt. If both use the same ETag, only one can publish. The loser reads the new head and checks for the receipt before attempting the mutation again. Finding no receipt establishes nonacceptance only through the checked version; a delayed original attempt may still complete afterward. Repeating the same checks and conditional publication makes that race safe.

Recovering after another writer commits

Suppose A publishes op-a as version 42 and loses the response. B then reads version 42 and publishes an unrelated addition as version 43. Its head carries A’s receipt forward while version 42 remains covered. A can read version 43, find its own receipt, and return the recorded version-42 result without applying the replacement again.

The failure point determines what the request has established. Other writers may keep advancing the head throughout recovery:

Bounding the receipt window

For a concrete retention rule, the example can retain receipts for the latest configured number of accepted state versions. At version 43, a two-version window covers 42 and 43, including every operation in either version if writes are batched. This bounds the number of covered versions without relying on clock synchronization. It does not promise a fixed number of minutes: a busy history advances the window faster.

For op-a, the stored initial version is 41 and the earliest possible acceptance is 42. If receiptCoverageFrom is 42 or lower, an absent receipt proves the operation has not committed through the observed head. If the boundary is 43, absence is inconclusive: the operation might have committed at 42 and been forgotten. The service must return an unknown outcome rather than apply it again.

The coverage check must precede every publication attempt, including one resumed after a long pause. An old prepared head will fail its ETag comparison if the window has moved; rebuilding against the new head requires checking coverage again. Assigning the old request a fresh operation ID or a newer initial version would discard that protection.

New histories start at state version zero, with an empty receipt list and coverage beginning at version one. The initial snapshot is uploaded first, and the head is created with If-None-Match: *. The first mutation can then pass the coverage check for version one.

As the history grows, receipt size depends on batch size and result size as well as the number of retained versions. Large results can live in immutable objects referenced by receipts; a fixed retry duration needs a time-based retention policy sized for the write rate. The head and its retention policy must fit the expected workload before this example becomes an implementation.

This example guarantees repeatable results for accepted mutations. A validation rejection changes no logged state; reproducing rejected results would require recording them too. If S3 cannot be read, recovery may have to leave the outcome unresolved. Bounded retries with backoff limit the work spent waiting for usable evidence.

The guarantee also stops at the stored state. An external payment or message has its own outcome, which cannot be made atomic with an S3 head update by recording an intention. Those effects need the external service’s idempotency mechanism or a reconciliation process.

Compaction and retention

Replica creation becomes progressively slower if every new server must replay the entire lifetime of a history. Periodic snapshots move the reconstruction starting point forward. After snapshot 900 is published, a new replica can load it and apply only the changes after that version.

snapshot 40 + changes 41..900
                ↓ compact
snapshot 900 + changes 901..

A maintenance worker builds the snapshot from a known coherent state and uploads it before publishing its reference. If writes advance the head while the snapshot is being built, the replacement must preserve their newer tail or retry against a fresh head. Maintenance follows the same conditional publication rule as ordinary writes.

Compaction changes the reconstruction path without changing what the application accepted. In the worked protocol, it advances the head revision while leaving the state version and receipt coverage unchanged. Even if an operation disappears from the active replay tail, its receipt remains available for duplicate detection. Replay and request recovery therefore have separate retention rules.

Old accepted entries and data files may still be needed by active readers, lagging replicas, or historical recovery policies. They can be deleted only when no supported reader or recovery path needs them. A replica whose required tail has been retired must restore from a newer snapshot. Removing a file from the current dataset, compacting its log entry, and deleting its stored bytes are distinct operations.

Unpublished objects require a separate cleanup argument. A delayed writer might still attempt to publish one, so age and absence from the current head do not prove it is safe to remove. A conservative design retains those objects until an implementation-specific rule establishes that no outstanding attempt can publish them.

Snapshot frequency balances storage and work against startup time. Frequent snapshots cost more to produce; a long tail costs more to replay on every new replica. This is part of operating elastic read capacity, especially when sudden demand may require many cold copies at once.

Write throughput and transaction boundaries

Read replicas scale independently, while each head still serializes publication for one history. That is a deliberate tradeoff. A small conditional update can coordinate a much larger state, but the rate of accepted transitions remains constrained by storage latency, contention, and the work needed to validate changes.

Batching can publish several operations in one transition and amortize that cost. Separate histories can also have separate heads, allowing unrelated repositories, projects, or customers to advance independently. Increasing readers for one history and distributing writes across independent histories are different ways of scaling.

An update to one tenant database does not change the ETag of another tenant’s head. There is no global order across these histories. Shared storage and network resources still impose limits, but the application can choose which writes must serialize together by choosing the scope of each head.

That choice also determines what becomes visible atomically. Suppose a dataset revision changes its file manifest, schema, and statistics. The writer uploads all three immutable objects, then publishes their references together in one head. A reader captures one head version and uses all three references from it:

{
  "revision": 88,
  "stateVersion": 73,
  "manifest": "objects/manifest-d92a.json",
  "schema": "objects/schema-e7f1.json",
  "statistics": "objects/stats-b6c4.json"
}

This is an alternative head layout showing only the publication fields. S3 still writes one key at a time. The application gets atomic visibility of the set because readers discover every member through the same head; reading separate head versions for the schema and manifest would lose that guarantee.

A head for each dataset partition allows partitions to advance independently but cannot atomically change two partitions. One dataset-wide head could publish both changes together, while requiring both writers to compete at the same key. Two independent conditional replacements cannot provide that all-or-nothing transition. Cross-history transactions, global uniqueness, or a shared high-frequency queue require additional machinery and may be better served by an established database or coordination system.

The latency requirement matters just as much. A workload that needs very low commit latency may need a replicated log closer to its writers, with object storage receiving history afterward. That can be a good architecture, but its acknowledgment and recovery paths differ from the S3-authoritative publication model described here.

Other applications

The strongest candidates combine expensive-to-copy state, smaller replayable changes, and substantial demand for local reads. Git makes those properties visible together. Other systems can share the storage shape while needing different rules for generating changes or accepting writes.

A customer’s SQLite database may be several gigabytes while a transaction changes only a few pages. SQLite already produces a local WAL. Preserving its committed changes and consistent snapshots elsewhere allows the database to be reconstructed on another server, and replicas can support additional reads. A typical page-based service uses a single write owner and still needs a safe replication layer and a clear remote-durability boundary.

Litestream illustrates that replication layer. Its background replication coordinates with SQLite through transactions, including an internal lock table used around checkpoints, and captures page data into replication files. Its repository documentation explains the locking. Asynchronous backup supplies recovery history; it does not by itself make an application commit wait for S3 publication.

SlateDB builds an embedded storage engine around object storage, immutable files, and local caching. Analytical formats such as Delta Lake and Apache Iceberg provide established ways to publish new views of large tables. These systems are useful starting points when their data model fits; sharing the pattern does not mean they use this example head format or commit protocol.

Closing thoughts

What I take from Continuity is the freedom to size the read fleet around demand. A busy repository can have many local copies, and an idle one can have none. Both retain the same durable foundation. Adding a server changes where the service can answer reads without changing how a writer publishes the next accepted version.

That is a useful option for systems with large working state, smaller replayable changes, and many readers. S3 can hold the history and provide the conditional update that orders publication, allowing the application to avoid operating a separate consensus group or coordination database for that job. The fit depends on whether each history can tolerate the write latency and whether replicas can reconstruct and catch up quickly enough to serve their readers.

When those conditions hold, the service can treat local copies as capacity it allocates rather than unique state it must preserve. That is the broader idea I would carry into another design: keep the accepted history durable in one authoritative store, and let the machines serving it come and go with the work.

References

1. Vicent Martí, Git at any scale, Cursor, August 2026.
2. Amazon Web Services, Amazon S3 data consistency model.
3. Amazon Web Services, Conditional writes and conditional reads.
4. Malcolm Featonby, Making retries safe with idempotent APIs, AWS Builders’ Library.
5. SQLite, Write-Ahead Logging.
6. Litestream, How it works and source database changes.
7. SlateDB introduction.
8. Delta Lake transaction log protocol.
9. Apache Iceberg, Reliability.