Schema Migrations

BucketDB enforces strict immutable structures. This means you cannot modify an existing schema version. Once version: 1 is written, the data blocks are locked to that binary layout forever.

Instead, BucketDB uses a Copy-on-Write Blue/Green Migration Pattern. This is handled natively by the cluster's timesharing authority via the Storage-Level Migration Daemon to guarantee zero downtime.

The Problem with Modifying Schemas

In traditional RDBMS systems, adding a column often locks a table, preventing reads/writes until the server rewrites the physical pages. In a highly distributed environment, locking an S3 bucket is impossible and downloading/rewriting all blocks synchronously would cause massive timeouts.

In earlier versions of BucketDB, developers had to write manual ETL scripts or handle versioned rows dynamically at query time. As of v0.2.0, the database handles this entirely in the background at the storage layer.

The Zero-Downtime Migration Process

Because migrating gigabytes of storage blocks takes time, you must orchestrate your application deployment alongside the BucketDB schema daemon.

Step 1: Define Version 2 (V2)

In your application code, register the new schema.

// Register V2 with the new field
db.registerSchema({
  name: 'users',
  version: 2, // Increment version
  fields: {
    id: { typeId: 10, maxLength: 36 },
    email: { typeId: 10, maxLength: 255 },
    age: { typeId: 1 } // New field
  },
  primaryKey: 'id'
});

Step 2: Deploy Dual-Schema Readers

Deploy an update to your application. At this stage:

• Your application logic must be prepared to read documents in either V1 or V2 format (e.g., handling missing fields gracefully or utilizing BucketDB's array aliases for backwards compatibility).

Step 3: Trigger the Storage Daemon

You trigger the background daemon by updating the target version.

// This writes the new target version into the Target Versions Section of Block 0
await db.setTargetSchemaVersion('users', 2);

What happens internally?

The node holding the timesharing authority (the Leader) spins up the SchemaDaemon.

1. Detection: It detects a discrepancy between the target version in Block 0 and the actual version of the data blocks in the index.
2. Blue/Green Transformation: It reads the old V1 data blocks, translates the rows into the new V2 binary layout (padding new fields with default values, and dropping removed fields), and writes them to entirely new S3 data blocks.
3. Atomic Pointer Swap: Once all blocks are transformed, it performs an atomic conditional PUT on Block 0. It swaps the active block pointers from the old schema version to the new one.
4. Garbage Collection: The old S3 data blocks are now orphaned (no longer referenced by Block 0) and will be safely purged by the standard db.gc() process.

During this time, your application is still reading and writing normally. The QueryExecutor simply reads the old blocks until the atomic pointer swap occurs.

Step 4: Deploy V2 Writers

Once the migration is fully complete (all blocks in the Block 0 index indicate V2), deploy a second update to your application. Now, your application logic should write documents using the V2 schema exclusively.

Step 5: Deprecate V1 (Cleanup)

Once you are certain no rogue clients are still caching or writing V1 data, you can remove any V1 fallback logic from your codebase.

Run db.gc() to purge the orphaned V1 S3 data blocks, freeing up storage capacity.