Signals, alerts, and recovery

KubeAtlas v1.6 exposes product-neutral Prometheus text metrics at /metrics. The chart does not install Prometheus, a ServiceMonitor or PodMonitor CRD, Grafana dashboards, or a vendor-specific monitoring agent. Connect the existing monitoring system used by your DevOps team to the ClusterIP Service and own its retention, routing, and escalation policy there.

:::warning Protect the monitoring and application surfaces

KubeAtlas has no built-in authentication. Keep the Service cluster-internal or put every exposed route, including /metrics, behind external authentication and authorization. KubeAtlas intentionally stores and displays broad cluster information: ConfigMap values, workload specifications, RBAC names, annotations, and topology can all be sensitive. Only Kubernetes Secret values and runtime credentials are excluded. An unauthenticated KubeAtlas endpoint is therefore still a cluster-wide information leak. Read Authentication is your job before exposing it.

:::

What the signals mean

Metric	Meaning
kubeatlas_go_memory_limit_bytes	Go's current soft runtime-managed-memory boundary. The chart derives it from the container limit; compare it with resources.limits.memory when investigating memory pressure.
kubeatlas_go_heap_alloc_bytes, kubeatlas_go_heap_live_bytes, kubeatlas_go_heap_goal_bytes	Current object allocations (including uncollected objects), objects marked live by the previous GC, and the GC heap-size target. Live bytes are not a current heap dump.
kubeatlas_go_heap_free_bytes, kubeatlas_go_heap_released_bytes, kubeatlas_go_heap_unused_bytes	Free heap not yet returned to the OS, free heap already returned, and space reserved for objects but unused. These help distinguish object retention from heap retention/fragmentation.
kubeatlas_go_heap_stacks_bytes, kubeatlas_go_runtime_total_bytes, kubeatlas_go_gc_cycles_total	Stack reservation, total read-write memory mapped by Go, and completed GC cycles. Total mapped memory is not RSS; GC cycles reset with the process.
kubeatlas_informer_synced	1 after the initial informer cache sync. It does not return to 0 after a later API outage, so do not use it alone as a freshness signal.
kubeatlas_graph_observation_state{state="..."}	One-hot graph state: initializing, synced, degraded, or stale. Exactly one series is 1.
kubeatlas_kubernetes_api_reachable	Result of the latest bounded, read-only Kubernetes API probe. In multi-cluster mode it is 1 only when every attached member responds.
kubeatlas_kubernetes_api_last_success_timestamp_seconds	Last successful API probe, or 0 before the first success.
kubeatlas_storage_reachable	Result of the latest bounded storage probe. Tier 2 probes PostgreSQL; Tier 1 memory is local to the process.
kubeatlas_storage_durable	1 for PostgreSQL Tier 2 and 0 for in-memory Tier 1. Reachable memory is not durable storage.
kubeatlas_storage_last_success_timestamp_seconds	Last successful storage probe, or 0 before the first success.
kubeatlas_snapshot_queue_drop_total	Snapshot events dropped because the asynchronous queue was full. Present only when snapshots are enabled.
kubeatlas_snapshot_write_failed_total	Snapshot events abandoned after the retry budget. Present only when snapshots are enabled.
kubeatlas_otel_dropped_total	Spans dropped because the opt-in OTLP queue was full. Present only when OTel is enabled.
kubeatlas_backup_status_available	1 when the optional operator-maintained backup timestamp marker is valid; 0 when it is disabled, missing, or invalid.
kubeatlas_backup_last_success_timestamp_seconds	Timestamp reported by that marker, or 0 when unavailable.
kubeatlas_backup_age_seconds	Age of the reported backup in seconds, or 0 when unavailable. Pair it with kubeatlas_backup_status_available; zero never means a fresh backup.

The graph state is derived as follows:

• initializing: the informer has not completed initial sync, or the first API probe has not completed.
• synced: initial sync completed and the latest API probe succeeded.
• degraded: the latest API probe failed, but the most recent success is newer than operations.staleAfter (default two minutes).
• stale: the probe is still failing after that boundary, or no successful probe has ever occurred after initial sync.

A quiet cluster remains synced: freshness comes from a read-only API probe, not from how recently a resource happened to change.

Resource processing lag

synced and /readyz do not prove that every queued informer event has reached the graph. Zero snapshot drops or write failures do not prove this either: resource writes precede the asynchronous snapshot queue. When testing change visibility, check that a known resource version reaches the graph within the declared deadline, including across a resync cycle.

By default, informers replay their local cache every ten minutes. A replay of an already-persisted UID and resource version no longer rewrites the resource or emits duplicate snapshot history. This is a per-process optimization, not durable event deduplication across application restarts. Failed resource writes remain eligible for retry, and unchanged resources still re-derive edges so late selector targets and failed edge writes can be reconciled. Resync is not an API relist and does not replace watch recovery.

Retain delayed-change evidence separately from snapshot queue metrics. Do not disable resync or extend a failed visibility deadline merely to pass a test.

Alert examples

These are plain PromQL examples, not installed alert rules. Adjust durations and backup objectives to your operating policy.

# Graph data has crossed the configured stale boundary.
kubeatlas_graph_observation_state{state="stale"} == 1

# Durable Tier 2 storage is currently unreachable.
(kubeatlas_storage_durable == 1)
and
(kubeatlas_storage_reachable == 0)

# Snapshot work was dropped or permanently failed in the last five minutes.
increase(kubeatlas_snapshot_queue_drop_total[5m]) > 0
or
increase(kubeatlas_snapshot_write_failed_total[5m]) > 0

# Opt-in OTel work was dropped in the last five minutes.
increase(kubeatlas_otel_dropped_total[5m]) > 0

# No valid operator backup marker exists for a durable deployment.
(kubeatlas_storage_durable == 1)
and
(kubeatlas_backup_status_available == 0)

# The operator-reported backup is older than 24 hours.
(kubeatlas_backup_status_available == 1)
and
(kubeatlas_backup_age_seconds > 86400)

Suggested starting severities:

• page on sustained Tier 2 storage failure and a stale graph if operators rely on KubeAtlas during incidents;
• ticket on dropped snapshot/OTel work, then page if the counter continues to rise;
• set backup-marker and backup-age severity from the actual recovery point objective, not from the example 24-hour threshold.

Metrics contain fixed state labels and numeric values only. They do not contain resource names, namespaces, probe errors, Secret values, database passwords, kubeconfigs, tokens, or backup contents.

Publish a backup timestamp marker

KubeAtlas does not take backups. Your backup job may update a small ConfigMap only after the backup command and its integrity checks succeed:

kubectl create configmap kubeatlas-backup-status \
  --namespace kubeatlas \
  --from-literal=last-successful="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --dry-run=client -o yaml \
  | kubectl apply -f -

helm upgrade kubeatlas oci://ghcr.io/lithastra/charts/kubeatlas \
  --namespace kubeatlas \
  --reuse-values \
  --set operations.backupStatus.configMapRef.name=kubeatlas-backup-status \
  --set operations.backupStatus.configMapRef.key=last-successful

The file accepts RFC 3339 or positive Unix seconds. The chart mounts only that key, read-only. Put no backup payload, credentials, destination URL, object key, or customer data in this ConfigMap. KubeAtlas reads at most 128 bytes and does not log invalid contents or parsing errors.

This marker is an operator attestation, not independent proof that the archive exists or can be restored. Update it only after the backup is complete, and pair it with scheduled restore drills.

Kubernetes API interruption

Observed signals:

1. kubeatlas_kubernetes_api_reachable becomes 0 after the next probe.
2. Graph state moves to degraded, then stale after staleAfter.
3. kubeatlas_informer_synced and /readyz remain successful because they record the completed initial sync; existing graph reads may be stale.

Operator response:

1. Confirm the control plane using your Kubernetes distribution's supported diagnostics. Do not restart KubeAtlas merely to clear the metric.
2. Keep the KubeAtlas endpoint protected while stale data is being served.
3. Restore API reachability and watch for kubeatlas_kubernetes_api_reachable 1 plus graph state synced.
4. Escalate if both signals have not recovered within 120 seconds after the API server is responsive.

For a drill, use the distribution's supported control-plane interruption procedure in a disposable environment. Confirm the target and recovery procedure first. Keep /metrics reachable independently of the API server; workstation-specific control-plane orchestration is not part of this repository.

PostgreSQL interruption

Observed signals:

1. On Tier 2, kubeatlas_storage_durable remains 1 and kubeatlas_storage_reachable becomes 0.
2. Graph reads that require PostgreSQL may fail; /healthz and /readyz do not claim database health.
3. Snapshot write failures can rise if the interruption exceeds the writer's retry budget. The queue may fill and drop events under sustained outage.

Operator response:

1. Inspect the CNPG Cluster, primary Pod, Services, and PVC. Do not delete a PVC as a reconnection test.
2. Restore a Ready primary using the database operator's supported procedure.
3. Require kubeatlas_storage_reachable 1 and a successful graph read within 120 seconds after the primary is Ready.
4. If dropped-work counters moved, record the observation gap and run the recovery procedure appropriate to your retention and backup policy.

The disposable embedded-Tier-2 drill is:

kubectl port-forward -n kubeatlas service/kubeatlas 18080:80
bash test/chaos/pg-disconnect.sh

See Persistence for the supported logical backup and restore path. High-availability coordination remains outside the v1.6 boundary; the application chart supports one KubeAtlas replica.

Evidence gates

When RSS exceeds its gate, retain the failed evidence and compare numerical heap/GC observations from the same Pod UID and workload. Rising RSS alone does not prove an object leak, and a later low reading does not invalidate a failed window. Check live heap across completed GC cycles, free/unreleased heap, stack reservation, and the workload timeline. Do not force GC, change GC or memory-limit settings, or raise the threshold merely to obtain a passing run. The runtime metrics expose only aggregate numbers; they do not expose heap contents or add a profiling endpoint.

Include the security-check endpoints when reproducing allocation pressure, not just cluster, namespace, and blast-radius reads. Namespace SVG/PNG exports now read only the namespace subgraph, and orphan detection filters resource reads at the store while still checking incoming references across namespaces. Diagnose cycle detection intentionally remains cluster-wide to preserve cycles that cross namespace boundaries. On the built-in stores it uses a metadata-only snapshot: all structured resource metadata and edges remain available, while the private Raw payload is omitted before PostgreSQL decoding. The ordinary snapshot API and stored objects remain unchanged. Wrappers without the optional capability retain their own snapshot behavior, including visibility restrictions. A namespace filter is not permission to omit cross-namespace dependencies. Compare identical request mixes and allow natural GC and scavenging to run. A short reduction in peak RSS is diagnostic evidence, not a replacement for a completed endurance evaluation under a frozen workload and declared limits.

• Pull requests run the required frozen Kubernetes 1.34, 1.35, and 1.36 Tier 2 matrix, plus required snapshot-overload evidence.
• A manual Release preflight composes artifact construction with that entire frozen candidate matrix. Every result must be definitive before release.
• Weekly public clean cluster anonymously resolves the current GitHub Release, verifies its checksum, pulls the matching public OCI Chart, checks versioned documentation, and installs both Tier 1 and Tier 2 on a fresh vanilla Kubernetes cluster.
• Failure and scheduled evidence contains bounded cluster state, logs, and metrics. A randomized raw/base64 Secret sentinel and the CNPG runtime password are scanned before retained evidence is accepted.

The exact automation inventory is maintained in the chaos README.