Managing Consistency

BucketDB handles distributed data using an immutable block structure combined with a Write-Ahead Log (WAL) mechanism. Because it is designed to run over decoupled object storage like S3, it natively offers tunable consistency.

The Role of readStalenessMs

The configuration property readStalenessMs (default: 1000) controls the maximum time, in milliseconds, that query results are allowed to be stale.

When a query is executed, BucketDB checks if its in-memory cache of the WALs is older than readStalenessMs. If it is, the system performs a sequence-guessing get loop against the underlying storage driver to discover and fetch newly committed WALs, merging them into the active query context to ensure the most recent writes are reflected.

High Staleness (Lower Cost)

If your application reads heavily but writes infrequently, or if having data delayed by a few seconds is acceptable, you can increase readStalenessMs (e.g., to 5000 or 30000).

• Pros: Drastically reduces the number of get operations sent to S3, reducing operational costs and lowering average query latency.
• Cons: Clients might not see a write immediately after committing it.

Zero Staleness (Immediate Consistency)

Setting readStalenessMs: 0 forces BucketDB to perform a freshness check before every single query.

• Pros: Immediate read-after-write consistency.
• Cons: Without custom driver optimization, this will issue multiple S3 get operations per query, resulting in high API costs and latency.

Advanced Pattern: The Shared Volume Driver

Because BucketDB allows you to inject custom storage drivers via the driver configuration, you can achieve zero-staleness reads without incurring the S3 penalty by utilizing a fast, shared coordinate layer (like a local shared volume, an EFS mount, or a Redis instance).

Example Architecture

Imagine you are running a deployment of 3 instances with a shared network volume. You can implement a custom StorageDriver wrapper that:

1. Pass-through Immutable Data: Passes all reads and writes for data/ and indexes/ directly to S3 (since these blocks are immutable and heavily cached).
2. Mirror Hot Paths: Whenever writing a new WAL or Block0, the driver writes it to both S3 and the local shared volume.
3. Local Reads for Hot Paths: Whenever BucketDB requests Block0 or a WAL file via get, the driver intercepts this and reads exclusively from the shared volume.

The Result

By using this pattern with readStalenessMs: 0, BucketDB will ask for a WAL refresh on every query. The custom driver instantly resolves this from the shared volume. You get instant read-after-write consistency with zero additional S3 costs and significantly lower latency.