Cluster Operations

BucketDB is designed to scale horizontally across multiple stateless nodes without requiring complex consensus protocols like RAFT or Paxos. It achieves this using a timesharing authority model with all commits given a MISC (Monotonically Increasing System Clock).

The Timesharing Model

In a distributed system, multiple nodes might try to merge writes from the Write-Forward log into the immutable data blocks simultaneously. This would cause race conditions and data corruption. To prevent this, only one node at a time is allowed to act as the "Leader".

How Authority is Assigned

Instead of electing a leader via a consensus algorithm, BucketDB assigns leadership strictly based on time and a node's assigned ID.

1. When configuring a cluster, each node is assigned a unique nodeId (from 0 to 255).
2. The writeForwardInterval (default 5000 milliseconds) determines how often the timesharing window rotates.
3. The cluster dynamically coordinates based on the number of active nodes. The current window index modulo the active ringSize dictates which node holds authority.

stateDiagram-v2
    [*] --> Node0
    Node0 --> Node1: Window + 5000ms
    Node1 --> Node2: Window + 5000ms
    Node2 --> Node0: Window + 5000ms

The Role of the Leader

The node currently holding authority becomes the Leader. Its sole responsibility is to run the background Write-Forward Service.

• It scans the committed/ prefix in the storage driver.
• It pulls down any pending batches (from any node).
• It merges those rows into new immutable data blocks.
• It atomically swaps the block pointers in Block 0.
• It deletes the pending batches from the log.

Note: ALL nodes can accept write requests and flush them to the committed/ log at any time. The timesharing mechanism only restricts who is allowed to write those logs forward into permanent blocks.

Dynamic Scaling and Replicas

Because the cluster coordinates dynamically, you can scale your database tier automatically.

Overlaps and Gaps (Split-Brain)

When a cluster scales dynamically (e.g., a node crashes or a new one boots up), nodes will not discover the change at the exact same millisecond.

• The Gap: For a few seconds, there may be a leadership window that no node thinks it owns. The Write-Forward log simply builds up slightly, which is harmless.
• The Overlap: Two nodes might temporarily both believe they are the leader of the current window.

In traditional databases, an overlap causes catastrophic split-brain corruption. In BucketDB, it is completely harmless.

Because Block 0 updates are guarded by strict Storage Driver ETag conditional writes, if two nodes run the Write-Forward process simultaneously, the slower node will receive an ETag mismatch when it tries to save the new block pointers. It will silently discard its redundant work and back off, treating the Timesharing ring purely as an optimization to prevent wasted compute, rather than a hard concurrency lock.

Performance Implications During Gaps

The Write-Ahead gap may have an impact on query performance.

When a query is executed, BucketDB must guarantee eventual consistency. To do this, the local QueryEngine will overlay pending commits. If the committed/ prefix temporarily grows larger than normal because the leader missed its turn, queries will have to process more batches. This may cause a noticeable latency spike in read queries for the duration of the gap.

Read Replicas (Edge Nodes)

A massive architectural feature of BucketDB's timesharing model is the ability to run pure Read Replicas.

Because any node can read data and append to the committed/ log, you can spin up 50 nodes globally (e.g., on edge locations) to serve lightning-fast reads. However, you don't want an edge node in Tokyo merging S3 blocks for a bucket hosted in Virginia.

To solve this, simply assign your edge nodes a nodeId that falls outside the active cluster's designated Timeshare ring, or do not run the WriteForwardService on them. They will serve reads from the in-memory cache and overlay local writes into the S3 WAL, but will never wake up to perform heavy background block merges, drastically reducing cloud egress costs and processing latency.