Snapshot Agent

A snapshot agent is a process that runs on an origin system and is responsible for providing snapshots of the origin application's data on demand.

The snapshot agent connects to the Kafka snapshot request topic as a consumer to the snapshot response topic as a publisher.

When a snapshot request is received, the snapshot agent

1. Unloads each of the configured data files to a portable format.
2. Compresses the exported data using LZ4 compression to an LZ4 archive in an outer TAR archive.
3. Uploads the TAR archive to network storage (usually an S3 bucket).
4. Sends a "snapshot complete" message to the snapshot response topic.
5. Deletes the local copy of the TAR archive.

The name of the TAR archive includes the current offset in the main event topic so that the exact point in the timeline of data changes the snapshot the data is associated with.

The snapshot agent for Windows and Linux is written in Rust. The snapshot agent for OpenVMS is written in VMS-C.