Dartvel change capture: keep a warehouse in step with your data

You mark a data model as captured and name where its copies go. Dartvel records the writes, delivers them, and catches a destination up after it was down.

Add capture: true to the data model. You read and write it the same way as before. Saving a record also records the change.

// Captured: every insert, update, delete, soft delete and restore is
// recorded, in the order it committed, and delivered to the destinations
// dartvel.capture in pubspec.yaml names. The email is named in each change
// and never carried in it. The subject is whose shipment it is, so an
// erasure reaches the row and every copy made from it.
@DVModel(subject: DVSubject.self, capture: true)
class const _Shipment({
  required final String id,
  required final String reference,
  required final int total,
  @DVModel.sensitiveField() required final String customerEmail,
});

capture:
  retention: 7d
  destinations:
    warehouse:
      type: database
      connection: WAREHOUSE_URL
      models: [Shipment]
      lagThreshold: 10m

connection is the name of a secret. The value stays out of the pubspec, and the server reads WAREHOUSE_URL from its environment when it starts. A URL written into the pubspec stops the build with DV-CDC-007.

type: database copies each data model into another database, one collection per model, holding the newest version of each record.

models is optional. Leave it out and the destination gets every captured data model. A name that is not a captured data model stops the build with DV-CDC-008.

Changes arrive in commit order, each with a sequence number, the record version, the tenant and the transaction id.

Inside DV.transaction a change is recorded only when the transaction commits. A rollback delivers nothing.

Writes made in Studio, and a device's offline writes replayed on the server, are captured like any other save.

A change names a sensitive field that changed but never carries its value. The log does not store it, and no destination has a place for it.

If a change cannot be recorded, the save is undone and throws DVCaptureWriteError. A saved record always has its change in the log.

The server delivers on the job queue. Each destination has its own queue, so one that is down does not hold up the others.

A destination with no copy yet is backfilled from the records already stored, in chunks, one job each. So is a data model you add to a destination later. The copy then follows new changes from where it began.

A destination that was down for longer than the retention window is backfilled again, so it never has a gap (DV-CDC-002).

If a destination's secret is not set, the server says so with DV-CDC-009 and delivers to the others. Its changes wait in the log until the retention window passes.

A server started without a role runs these jobs itself. In a deployment with worker processes, the workers run them.

The server measures how far behind each destination is and reports it at /metrics as dartvel_dv_capture_lag_seconds and dartvel_dv_capture_lag_changes. Past lagThreshold it logs DV-CDC-003, so a copy that is hours old is noticed.

When DV.Privacy erases a person, the logged values for their records are removed and every destination drops its copy at the same time, without waiting for the next delivery. If a destination cannot be reached, the erasure is marked incomplete. See Privacy and erasure.

GETTING STARTED

APP

DATA

BACKEND

OPERATIONS

SHIPPING

REFERENCE

Change capture

Mark a data model as captured

Say where the copies go

What a destination receives

Delivery, retries and backfill

Watch the lag

Erasure reaches the copies

Status