Worker Monitoring¶

Host Administration provides separate Application host management and Worker monitoring tabs. Host administrator access is required. Switching tabs preserves unsaved host settings and expanded monitoring details. Monitoring is read-only; it does not change worker settings.

The Cluster nodes view shows cluster health, worker totals, and individual processes, including nodes without workers. Each process has a machine or pod label and a process identity that changes whenever it restarts. Multiple entries with the same machine label can represent different processes or a recent restart.

Each process shows CPU usage, thread count, available disk space in MB, health status, roles, and measurement time. Expand a process to inspect health checks and their data. Worker processes also show running jobs, waiting jobs, completion rate, job errors, database and Redis measurements, and recent capacity changes with their reasons. Use these together when investigating a growing job queue: a low completion rate alone does not indicate a fault when jobs normally take a long time.

Capacity Settings¶

Configure each agent in appsettings.json, then restart that agent to apply these settings:

1
2
3
4
5
6
7
8

{
  "Agent": {
    "Workers": {
      "MaxThread": 64,
      "Adaptive": true
    }
  }
}

The example maximum of 64 is not a sizing recommendation. Select a limit using observed database, Redis, and downstream service capacity. I/O-heavy jobs may benefit from a limit above the processor count; increasing it can also place more load on those services.

Prefetch means jobs already assigned to a node and waiting to run. This waiting capacity follows the current running capacity and cannot exceed MaxThread. When capacity decreases, running jobs are allowed to finish and waiting jobs are not discarded. Counts can temporarily exceed a reduced target while existing work completes.

Limits apply separately to each node. Adding nodes increases the cluster's total capacity; there is no fixed cluster-wide job limit. Job priority, schedules, retries, node/group restrictions, and tenant eligibility continue to apply.

These agent settings require a restart. The worker-type overrides in Operations → Workers can still be saved without restarting.

Automatic Capacity Changes¶

Automatic mode increases capacity gradually when the current capacity is occupied, more jobs are ready, and recent database and Redis measurements indicate healthy operation. Low traffic or missing measurements do not justify an increase. Errors and timeouts suspend increases; sustained resource pressure can reduce capacity. After a decrease, further increases are paused to allow the system to settle.

A single slow query or isolated timeout does not by itself cause a general capacity reduction. Repeated timeouts and sustained resource pressure can reduce capacity. Inspect the affected resource and operation type when investigating a reduction, such as database connection opening or Redis commands. Cancellations are reported separately from timeouts.

Recent capacity changes show the previous and new limits, the time, and the reason. If capacity decreases repeatedly, inspect the affected database or Redis service and downstream dependencies before raising MaxThread.

Reading Resource Measurements¶

Resource measurements cover the recent 60-second window. Connection operations and database commands are listed separately. Database command rows distinguish normal, long, and unlimited configured timeouts; these categories are not a universal slow-query threshold.

• p95 estimates the duration within which 95% of measured operations complete. The displayed value is an upper-bound estimate, so use it to identify trends rather than as an exact duration. Cluster totals do not average node p95 values.
• Connection opening time can include waiting for a connection, establishing a physical connection, authentication, and network delay. A rise does not prove that the database connection pool is exhausted.
• Open application DB connections counts connections used by the application; it is not the database server's total connection count or the provider's pool capacity.
• In-flight DB commands and the oldest tracked command duration help identify ongoing database activity, including commands still reading results.
• Pending Redis operations helps identify requests waiting to be sent or completed. Interpret it alongside latency and timeout rates, rather than relying on Redis connection counts alone.

Cluster capacity and completion rates combine currently running, reachable nodes. Resource error and timeout rates use combined operation counts, so a quiet node and a busy node do not receive equal weight. Monitoring works without an external telemetry server.

Node Status and Stale Data¶

The health endpoint and this screen show the same cluster information. Status normally refreshes approximately every 10 seconds. After 30 seconds without an update, a node shows Communication delayed; after 60 seconds it shows Unreachable. Old processes disappear after 10 minutes.

Check the measurement time as well as the node's last update. Missing or out-of-date health measurements are marked with a warning. Worker measurement warnings apply only to worker nodes; nodes without workers have no worker capacity or job counters.

An unreachable entry does not prove that the process has stopped. Check the service or pod and its network, Redis, and database connectivity. If a refresh fails, the last available information remains visible with an availability warning. Use the service or pod status to confirm whether the node is running before taking recovery action.

Stopping Nodes and Recovering Jobs¶

For a planned stop, Emakin stops taking new jobs, returns waiting prefetched jobs for processing elsewhere, and gives running jobs a fixed 30-second shutdown budget. Jobs that cannot finish in time are cancelled where possible and become eligible for retry or recovery. Ensure the service manager or Kubernetes termination grace period allows this budget with additional time for the rest of application shutdown.

After an unexpected process loss, its unfinished jobs normally become eligible for recovery after about two minutes, plus the recovery check and database processing delay. Healthy long-running jobs are not reassigned just because they are old. Keep node clocks synchronized, for example with NTP, to avoid premature or delayed recovery.

An interrupted job can repeat an external action if that action succeeded before completion was recorded. For integrations where duplicate actions matter, use idempotency keys or duplicate detection. Recovery does not guarantee that external actions occur exactly once.

Related Pages¶

• System Health Checks
• Host Administration
• Worker Settings
• Background Jobs