Kigi-CLI/crates/codegen/kigi-tui/docs/user-guide/24-monitoring-usage.md

Monitoring Usage (External OpenTelemetry)

Status: alpha. The schema below is versioned (kigi_code.schema.version = v1); additive changes may occur without notice, renames/removals will bump the version.

Kigi CLI can export usage metrics and events to your organization's own OpenTelemetry collector, so platform teams can monitor adoption, token consumption, tool-permission decisions, and errors across the fleet — without any data flowing through SpaceXAI.

The external stream is:

• Off by default, and requires a double opt-in (a master switch and an explicit exporter selection).
• Content-free by default: no prompts, no code, no file paths (extension only), no tool arguments, no bash commands, and MCP/skill/plugin names collapsed to categories. Optional content gates re-enable some of these.
• Structurally separate from SpaceXAI-internal telemetry: its exporters carry only the headers you configure, never SpaceXAI credentials.
• Independent of SpaceXAI data-retention opt-outs: it works even when telemetry is disabled and for ZDR (zero-data-retention) teams — those settings govern SpaceXAI-side retention; the external stream is governed solely by your own OTEL configuration.

Quick start

export KIGI_EXTERNAL_OTEL=1                  # master switch
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf  # or grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.corp.example:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <collector-token>"
kigi

KIGI_EXTERNAL_OTEL=1 alone enables nothing — you must also select at least one exporter. Conversely, the OTEL_* vars alone enable nothing without the master switch.

Environment variables

Variable	Default	Meaning
KIGI_EXTERNAL_OTEL	0	Master switch. Distinct from KIGI_TELEMETRY_ENABLED, which controls SpaceXAI-internal product analytics — the two govern opposite-pointing data flows.
OTEL_METRICS_EXPORTER	none	otlp | console | none.
OTEL_LOGS_EXPORTER	none	otlp | console | none. Gates the event stream.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf	http/protobuf | grpc.
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318 for HTTP, http://localhost:4317 for gRPC	Base endpoint. For http/protobuf, /v1/logs and /v1/metrics are appended per the OTLP spec; for grpc, the collector endpoint is used as-is.
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT / ..._METRICS_ENDPOINT	—	Signal-specific overrides, used verbatim. For gRPC these should normally be collector endpoints without /v1/... paths.
OTEL_EXPORTER_OTLP_HEADERS (+ signal-specific variants)	—	Collector auth (k=v,k2=v2). The only headers the external exporters send, and the only supported collector-auth mechanism (no config-file headers key — tokens never live on disk).
OTEL_EXPORTER_OTLP_TIMEOUT	10000 (ms)	Export timeout.
OTEL_METRIC_EXPORT_INTERVAL	60000 (ms)	Metric export interval.
OTEL_BLRP_SCHEDULE_DELAY (or alias OTEL_LOGS_EXPORT_INTERVAL)	5000 (ms)	Log batch interval.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE	delta	delta | cumulative.
OTEL_METRICS_INCLUDE_SESSION_ID	1	Attach session.id to metrics (cardinality opt-out).
OTEL_METRICS_INCLUDE_VERSION	0	Attach app.version to metrics.
OTEL_LOG_USER_PROMPTS	0	Content gate: prompt text on kigi_code.user_prompt (60 KB cap, secret-scrubbed).
OTEL_LOG_TOOL_DETAILS	0	Content gate: tool parameters (4 KB cap), full file paths, verbatim MCP/skill/plugin names. Bash command text is never exported in v1, even with this gate.

OTEL_RESOURCE_ATTRIBUTES is deliberately ignored: the resource is built from a fixed, audited attribute set.

Migration note: older releases could share OTEL_EXPORTER_OTLP_* with the product's own analytics pipeline. That behavior is deprecated: when KIGI_EXTERNAL_OTEL is set, product analytics ignores those vars, and the CLI refuses to activate the external stream in any configuration where product analytics already consumed them — your collector only receives the external stream you opted into.

Config file

Org defaults live under the existing [telemetry] table in config.toml (env vars win). The keys are otel_-prefixed peers of the other [telemetry] settings:

[telemetry]
otel_enabled = true
otel_metrics_exporter = "otlp"
otel_logs_exporter = "otlp"
otel_endpoint = "https://collector.corp.example:4318"
otel_protocol = "http/protobuf"  # or "grpc"
otel_log_user_prompts = false   # admins can pin these via requirements
otel_log_tool_details = false

The config keys are otel_* under [telemetry]; the env vars keep their standard OTEL names (KIGI_EXTERNAL_OTEL, OTEL_*) for ecosystem interop, so the two layers use deliberately different namespaces. The otel_protocol config key maps to OTEL_EXPORTER_OTLP_PROTOCOL.

There is deliberately no headers key: supply collector auth via OTEL_EXPORTER_OTLP_HEADERS so tokens are never stored on disk.

Managed deployments can additionally enable org-wide telemetry by distributing the [telemetry] otel_* keys through kigi setup managed config / requirements pins, or force-disable it fleet-wide with the same local config layers (external_otel_disabled, content-gate locks).

Resource attributes

Attribute	Value
service.name	kigi-cli
service.version, client.version	build/client versions
app.entrypoint	cli | headless | agent
terminal.type	terminal emulator brand
kigi_code.schema.version	v1

Identity attributes (user.id, and organization.id / team.id / deployment.id when known) are attached per metric data point and per event once authentication completes. prompt.id (per-prompt UUID) appears on events only, never metrics.

Metrics (meter scope ai.xai.kigi_code)

Metric	Unit	Attributes
kigi_code.session.count	{session}	base attrs only
kigi_code.token.usage	{token}	type = input | output | reasoning | cache_read; model
kigi_code.turn.count	{turn}	outcome = completed | cancelled | error; model
kigi_code.tool.decision	{decision}	tool_name, decision = allow | deny | cancelled | followup, access_kind, permission_mode
kigi_code.tool.usage	{call}	tool_name, outcome
kigi_code.error.count	{error}	error_category, model

There is no cost.usage metric: join kigi_code.token.usage with your own price sheet. lines_of_code.count and active_time.total are planned for a later phase.

tool_name values: built-in tool names pass verbatim; MCP tools collapse to mcp_tool and other non-built-in tools to custom_tool unless OTEL_LOG_TOOL_DETAILS=1.

Events (OTLP log records)

Every event carries event.sequence, session.id, turn_number (in-turn), prompt.id, plus the identity attributes. Gate legend: details = requires OTEL_LOG_TOOL_DETAILS, prompts = requires OTEL_LOG_USER_PROMPTS; everything else always exports while the stream is active.

event.name	Attributes
kigi_code.session_start	model, permission_mode, mcp_server_count, plugin_count, skill_count, hook_count, memory_enabled, is_git_repo, client_identifier
kigi_code.session_end	duration_secs, turn_count, tool_call_count, compaction_count, model
kigi_code.user_prompt	prompt_length, model, screen_mode? (fullscreen | inline | minimal | headless | other); prompt (prompts)
kigi_code.turn_completed	outcome, duration_ms, tool_call_count, model, error_category?, cancellation_category?
kigi_code.api_request	model, duration_ms, stop_reason?, input_tokens, output_tokens, reasoning_tokens, cache_read_tokens
kigi_code.api_error	error_category, model, status_code?, duration_ms?
kigi_code.tool_result	tool_name, outcome, success, duration_ms, file_extension; tool_parameters, file_path (details)
kigi_code.tool_decision	tool_name, decision, access_kind, permission_mode, source
kigi_code.mcp_server_connection	status, transport_type, duration_ms, tool_count?, error_type?; mcp_server.name (details; collapsed to mcp_server otherwise)
kigi_code.permission_mode_changed	to_mode, trigger
kigi_code.skill_activated	skill_source; skill.name (details)
kigi_code.plugin_loaded	install_kind?, success, error_category?; plugin_name (details)
kigi_code.compaction	duration_ms, tokens_before, tokens_after, model?
kigi_code.subagent	phase = launched | completed, subagent_type?, outcome?, duration_ms?
kigi_code.auth	auth_method
kigi_code.internal_error	error_type (class only — no message, no location)
kigi_code.model_switched	from_model, to_model, success, error_code?

Privacy model

Three independent fail-closed mechanisms guard the wire format:

1. A typed schema: attribute keys are a closed enum; nothing outside it can be attached.
2. Emit-time redaction: every string passes a secret-shape scrub and a home-directory scrub, with truncation (512→128 chars per value, 4 KB tool params, 60 KB prompt cap).
3. Export-time validators: any record carrying a non-schema key, a closed-gate key, or an unscrubbed secret shape is dropped before leaving the process; metric exports with out-of-schema attribute keys are dropped entirely.

Never exported: bash command text, error message bodies, prompt text (without the gate), file paths (without the gate), api_key.id, machine fingerprints, email addresses, subscription tier.

Example collector config

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  prometheus:
    endpoint: 0.0.0.0:9464

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: []   # point at your log backend (loki, elasticsearch, …)

Example queries (PromQL, with the Prometheus exporter above):

# Tokens by model and type across the org, 1h rate
sum by (model, type) (rate(kigi_code_token_usage_total[1h]))

# Sessions per team per day
sum by (team_id) (increase(kigi_code_session_count_total[1d]))

# Tool-permission denial ratio
sum(rate(kigi_code_tool_decision_total{decision="deny"}[1h]))
  / sum(rate(kigi_code_tool_decision_total[1h]))

Debugging

Set OTEL_LOGS_EXPORTER=console / OTEL_METRICS_EXPORTER=console to print redacted records to stderr (suppressed in agent/headless entrypoints to keep captured logs clean). Export errors never surface in the TUI; check the debug log.