JSON Agent

JSON Agent is a .NET 10 application that can be deployed on Windows and Linux to consume messages from the CDC topic (in which data is represented as base-64 encoded DBL records), transform the data into a more widely consumable JSON format, and republish messages containing the JSON data into an alternate topic.

Startup

Unlike other agents, JSON Agent does not hydrate or repair a local replica, so there is no “already hydrated” or “missing data” branch during startup. Its startup work is focused on bootstrap and readiness. It determines which profile to use, opens its local state database, restores its locally protected secrets, loads configuration, and brings up its TLS-protected management endpoint. If ConfigServer is configured, it registers if necessary, authenticates, downloads the profile configuration, and obtains a TLS certificate if one is not already installed. If ConfigServer is not being used, or cannot be reached, it falls back to locally cached or manually maintained configuration files.

Once configuration is available, the agent validates that it can actually perform CDC-to-JSON conversion. It loads the configured record-helper assembly, verifies that helper classes exist for every enabled table definition, initializes the helper used to identify the correct table for multi-record files, checks that Kafka is reachable, and creates its Kafka consumer/producer helper. If any of those checks fail, startup stops and the agent does not enter processing. If they succeed, the agent is ready to consume CDC traffic. At that point, it resumes from the committed Kafka position for its configured consumer group or from the earliest retained message if that consumer group has no committed position yet.

Main Processing

In its main loop, JSON Agent polls the configured CDC topic, waits up to the configured consumer wait time for a message, and if nothing arrives, sleeps for the configured idle interval before polling again. While paused, it stays running and manageable but does not take another message. A pause takes effect between messages, so a message already being transformed is allowed to finish.

For each valid CDC message, the agent identifies the target table, decodes the base-64 record data with the generated helper classes, and builds a JSON version of the CDC message. The published message keeps the CDC metadata and operation type, replaces the file identifier with the resolved table name, converts the key data into text, and exposes the current and original record images as JSON objects. The agent then writes that JSON message to the configured output topic. Only after Kafka confirms the write does it commit the source CDC message. In practice, this gives the agent at-least-once behavior: it avoids losing source changes, but a restart can produce a duplicate JSON message if publishing succeeded and the source commit did not. Malformed messages, messages with missing record data, or messages that cannot be matched to a helper are logged, counted, and skipped. If publishing fails, the source message is not committed, so it can be retried later.

Shutdown

When JSON Agent is run as a Windows Service, normal shutdown should be initiated through Service Control Manager. When it is run under Linux service management such as systemd, it should be stopped through the service manager, for example, with systemctl stop. The same orderly shutdown path is also used when an administrator selects Stop in the management UI or sends POST /api/stop to the management API. If a developer has started the agent manually in a console, pressing Ctrl+C triggers the same shutdown flow.

During shutdown, the agent changes its visible state to Stopping, cancels its processing loop, and begins shutting down the host. There is no separate drain or catch-up phase. The loop stops before taking another message, Kafka resources are disposed, and the management site exits with the process. Because the source CDC message is only committed after a successful JSON publish, anything not fully published and committed before shutdown can be retried on the next start.

Management UI

JSON Agent includes a secure browser-based management UI for day-to-day operations. After signing in with the admin password, an administrator can use the status page to see the current state, profile, input and output topics, poll counts, received and sent message counts, failure counts, and last message offset, file, and resolved table. That page also supports auto-refresh and provides Pause, Resume, and Stop controls.

The log page shows recent log output and live log updates, with controls to pause the live view, choose the maximum number of displayed entries, enable or disable color coding, clear the browser view, and download the visible log text. The settings page allows an administrator to change the consumer wait time, the idle sleep time, the minimum allowed admin-secret length, and whether Swagger API documentation is exposed. It also allows the admin secret to be changed and shows read-only runtime settings such as the helper assembly path, Kafka consumer group, Kafka client ID, output topic, and logging options. An admin page allows the Data Protection key ring to be exported for secure backup.

Most of what the management UI does is implemented by authenticated calls to the management API. That means the built-in UI is only one client of the management surface, and the same interfaces can be used to build custom dashboards, automation, and monitoring tools.

Management API

The management API covers authentication, agent control, status, log access, settings, and encryption-key export. Its main functions are security and configuration.

Security

The API is protected in layers. It runs on the agent’s TLS-enabled management endpoint, and startup fails if the TLS certificate or password is unavailable. The admin password is stored encrypted in the local database using ASP.NET Core Data Protection; on Windows, the key ring can also be protected to the local machine with DPAPI. A successful sign-in creates a short-lived JWT whose lifetime is controlled by configuration, and the same login also sets a Secure, HttpOnly, SameSite=Strict browser cookie. Protected routes accept either the bearer token or that secure cookie. The agent’s UI pages, API routes, and live log hub all require authorization. Unauthenticated browser requests for HTML are redirected to the login page, while API clients receive a 401 Unauthorized error. LaunchPad integration is further restricted to localhost and uses a short-lived, one-time ticket. If Swagger UI is enabled, it can be used to explore the API, but protected operations still require a valid admin token.

Configuration

JSON Agent supports both ConfigServer bootstrap and local-file bootstrap. It looks for a ConfigServer URL and profile from command-line options, then environment variables, and finally local app settings. If ConfigServer is configured, it registers if needed, authenticates, downloads MasterConfig.json and ServerConfig.JSONAgent.json, and requests a TLS certificate if one is not already present locally. Successful downloads are cached in the local profile directory so the agent has a last-known-good copy available for later offline startup.

If ConfigServer is not being used, the agent simply loads those same configuration files from the local profile directory. On Windows, that's under %ProgramData%\Synergex\SIP\<profile>, and on Linux it's under /etc/synergex/sip/<profile>. Those files can either be cached copies from a previous ConfigServer startup or manually maintained local files. If ConfigServer is configured but unavailable, the agent falls back to those local copies instead of failing immediately. In addition, when settings are changed through the management UI, the agent saves them locally first and then, if the configuration originally came from ConfigServer, attempts to sync them back to ConfigServer. If that sync fails, the local save is kept and a warning is returned. Startup only fails when the required configuration files or TLS materials are missing or unusable.

Periodic Services

JSON Agent supports Periodic Services which allows you to configure one or more recurring background tasks that run on a cron-style schedule alongside normal agent processing.