Kigi-CLI/crates/codegen/kigi-tui/docs/user-guide/24-monitoring-usage.md

# Monitoring Usage (External OpenTelemetry)

> **Status: alpha.** The schema below is versioned (`grok_code.schema.version = v1`);
> additive changes may occur without notice, renames/removals will bump the
> version and be called out in the changelog.

Grok CLI can export usage **metrics** and **events** to your organization's
own OpenTelemetry collector, so platform teams can monitor adoption, token
consumption, tool-permission decisions, and errors across the fleet — without
any data flowing through SpaceXAI.

The external stream is:

- **Off by default**, and requires a *double opt-in* (a master switch **and**
  an explicit exporter selection).
- **Content-free by default**: no prompts, no code, no file paths (extension
  only), no tool arguments, no bash commands, and MCP/skill/plugin names
  collapsed to categories. Optional content gates re-enable some of these.
- **Structurally separate** from SpaceXAI-internal telemetry: its exporters carry
  only the headers you configure, never SpaceXAI credentials.
- **Independent of SpaceXAI data-retention opt-outs**: it works even when
  `telemetry` is disabled and for ZDR (zero-data-retention) teams — those
  settings govern SpaceXAI-side retention; the external stream is governed solely
  by your own OTEL configuration.

## Quick start

```bash
export KIGI_EXTERNAL_OTEL=1                  # master switch
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf  # or grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.corp.example:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <collector-token>"
grok
```

`KIGI_EXTERNAL_OTEL=1` alone enables **nothing** — you must also select at
least one exporter. Conversely, the `OTEL_*` vars alone enable nothing
without the master switch.

## Environment variables

| Variable | Default | Meaning |
|---|---|---|
| `KIGI_EXTERNAL_OTEL` | `0` | Master switch. Distinct from `KIGI_TELEMETRY_ENABLED`, which controls SpaceXAI-internal product analytics — the two govern opposite-pointing data flows. |
| `OTEL_METRICS_EXPORTER` | `none` | `otlp` \| `console` \| `none`. |
| `OTEL_LOGS_EXPORTER` | `none` | `otlp` \| `console` \| `none`. Gates the event stream. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | `http/protobuf` \| `grpc`. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` for HTTP, `http://localhost:4317` for gRPC | Base endpoint. For `http/protobuf`, `/v1/logs` and `/v1/metrics` are appended per the OTLP spec; for `grpc`, the collector endpoint is used as-is. |
| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` / `..._METRICS_ENDPOINT` | — | Signal-specific overrides, used verbatim. For gRPC these should normally be collector endpoints without `/v1/...` paths. |
| `OTEL_EXPORTER_OTLP_HEADERS` (+ signal-specific variants) | — | Collector auth (`k=v,k2=v2`). The **only** headers the external exporters send, and the only supported collector-auth mechanism (no config-file headers key — tokens never live on disk). |
| `OTEL_EXPORTER_OTLP_TIMEOUT` | `10000` (ms) | Export timeout. |
| `OTEL_METRIC_EXPORT_INTERVAL` | `60000` (ms) | Metric export interval. |
| `OTEL_BLRP_SCHEDULE_DELAY` (or alias `OTEL_LOGS_EXPORT_INTERVAL`) | `5000` (ms) | Log batch interval. |
| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `delta` | `delta` \| `cumulative`. |
| `OTEL_METRICS_INCLUDE_SESSION_ID` | `1` | Attach `session.id` to metrics (cardinality opt-out). |
| `OTEL_METRICS_INCLUDE_VERSION` | `0` | Attach `app.version` to metrics. |
| `OTEL_LOG_USER_PROMPTS` | `0` | Content gate: prompt text on `grok_code.user_prompt` (60 KB cap, secret-scrubbed). |
| `OTEL_LOG_TOOL_DETAILS` | `0` | Content gate: tool parameters (4 KB cap), full file paths, verbatim MCP/skill/plugin names. Bash command text is **never** exported in v1, even with this gate. |

`OTEL_RESOURCE_ATTRIBUTES` is deliberately ignored: the resource is built
from a fixed, audited attribute set.

> **Migration note:** older releases could share `OTEL_EXPORTER_OTLP_*` with
> the product's own analytics pipeline. That behavior is deprecated: when
> `KIGI_EXTERNAL_OTEL` is set, product analytics ignores those vars, and the
> CLI refuses to activate the external stream in any configuration where
> product analytics already consumed them — your collector only receives the
> external stream you opted into.

## Config file

Org defaults live under the existing `[telemetry]` table in `config.toml`
(env vars win). The keys are `otel_`-prefixed peers of the other
`[telemetry]` settings:

```toml
[telemetry]
otel_enabled = true
otel_metrics_exporter = "otlp"
otel_logs_exporter = "otlp"
otel_endpoint = "https://collector.corp.example:4318"
otel_protocol = "http/protobuf"  # or "grpc"
otel_log_user_prompts = false   # admins can pin these via requirements
otel_log_tool_details = false
```

The config keys are `otel_*` under `[telemetry]`; the **env vars keep their
standard OTEL names** (`KIGI_EXTERNAL_OTEL`, `OTEL_*`) for ecosystem
interop, so the two layers use deliberately different namespaces. The
`otel_protocol` config key maps to `OTEL_EXPORTER_OTLP_PROTOCOL`.

There is deliberately no `headers` key: supply collector auth via
`OTEL_EXPORTER_OTLP_HEADERS` so tokens are never stored on disk.

Managed deployments can additionally enable org-wide telemetry by distributing
the `[telemetry]` `otel_*` keys through `grok setup` managed config /
requirements pins, or force-disable it fleet-wide with the same local config
layers (`external_otel_disabled`, content-gate locks).

## Resource attributes

| Attribute | Value |
|---|---|
| `service.name` | `grok-cli` |
| `service.version`, `client.version` | build/client versions |
| `app.entrypoint` | `cli` \| `headless` \| `agent` |
| `terminal.type` | terminal emulator brand |
| `grok_code.schema.version` | `v1` |

Identity attributes (`user.id`, and `organization.id` / `team.id` /
`deployment.id` when known) are attached per metric data point and per event
once authentication completes. `prompt.id` (per-prompt UUID) appears on
events only, never metrics.

## Metrics (meter scope `ai.xai.grok_code`)

| Metric | Unit | Attributes |
|---|---|---|
| `grok_code.session.count` | `{session}` | base attrs only |
| `grok_code.token.usage` | `{token}` | `type` = `input` \| `output` \| `reasoning` \| `cache_read`; `model` |
| `grok_code.turn.count` | `{turn}` | `outcome` = `completed` \| `cancelled` \| `error`; `model` |
| `grok_code.tool.decision` | `{decision}` | `tool_name`, `decision` = `allow` \| `deny` \| `cancelled` \| `followup`, `access_kind`, `permission_mode` |
| `grok_code.tool.usage` | `{call}` | `tool_name`, `outcome` |
| `grok_code.error.count` | `{error}` | `error_category`, `model` |

There is no `cost.usage` metric: join `grok_code.token.usage` with your own
price sheet. `lines_of_code.count` and `active_time.total` are planned for a
later phase.

`tool_name` values: built-in tool names pass verbatim; MCP tools collapse to
`mcp_tool` and other non-built-in tools to `custom_tool` unless
`OTEL_LOG_TOOL_DETAILS=1`.

## Events (OTLP log records)

Every event carries `event.sequence`, `session.id`, `turn_number` (in-turn),
`prompt.id`, plus the identity attributes. Gate legend: **details** =
requires `OTEL_LOG_TOOL_DETAILS`, **prompts** = requires
`OTEL_LOG_USER_PROMPTS`; everything else always exports while the stream is
active.

| `event.name` | Attributes |
|---|---|
| `grok_code.session_start` | `model`, `permission_mode`, `mcp_server_count`, `plugin_count`, `skill_count`, `hook_count`, `memory_enabled`, `is_git_repo`, `client_identifier` |
| `grok_code.session_end` | `duration_secs`, `turn_count`, `tool_call_count`, `compaction_count`, `model` |
| `grok_code.user_prompt` | `prompt_length`, `model`, `screen_mode?` (`fullscreen` \| `inline` \| `minimal` \| `headless` \| `other`); `prompt` (**prompts**) |
| `grok_code.turn_completed` | `outcome`, `duration_ms`, `tool_call_count`, `model`, `error_category?`, `cancellation_category?` |
| `grok_code.api_request` | `model`, `duration_ms`, `stop_reason?`, `input_tokens`, `output_tokens`, `reasoning_tokens`, `cache_read_tokens` |
| `grok_code.api_error` | `error_category`, `model`, `status_code?`, `duration_ms?` |
| `grok_code.tool_result` | `tool_name`, `outcome`, `success`, `duration_ms`, `file_extension`; `tool_parameters`, `file_path` (**details**) |
| `grok_code.tool_decision` | `tool_name`, `decision`, `access_kind`, `permission_mode`, `source` |
| `grok_code.mcp_server_connection` | `status`, `transport_type`, `duration_ms`, `tool_count?`, `error_type?`; `mcp_server.name` (**details**; collapsed to `mcp_server` otherwise) |
| `grok_code.permission_mode_changed` | `to_mode`, `trigger` |
| `grok_code.skill_activated` | `skill_source`; `skill.name` (**details**) |
| `grok_code.plugin_loaded` | `install_kind?`, `success`, `error_category?`; `plugin_name` (**details**) |
| `grok_code.compaction` | `duration_ms`, `tokens_before`, `tokens_after`, `model?` |
| `grok_code.subagent` | `phase` = `launched` \| `completed`, `subagent_type?`, `outcome?`, `duration_ms?` |
| `grok_code.auth` | `auth_method` |
| `grok_code.internal_error` | `error_type` (class only — no message, no location) |
| `grok_code.model_switched` | `from_model`, `to_model`, `success`, `error_code?` |

## Privacy model

Three independent fail-closed mechanisms guard the wire format:

1. A **typed schema**: attribute keys are a closed enum; nothing outside it
   can be attached.
2. **Emit-time redaction**: every string passes a secret-shape scrub and a
   home-directory scrub, with truncation (512→128 chars per value, 4 KB tool
   params, 60 KB prompt cap).
3. **Export-time validators**: any record carrying a non-schema key, a
   closed-gate key, or an unscrubbed secret shape is dropped before leaving
   the process; metric exports with out-of-schema attribute keys are dropped
   entirely.

Never exported: bash command text, error message bodies, prompt text
(without the gate), file paths (without the gate), `api_key.id`, machine
fingerprints, email addresses, subscription tier.

## Example collector config

```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  prometheus:
    endpoint: 0.0.0.0:9464

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: []   # point at your log backend (loki, elasticsearch, …)
```

Example queries (PromQL, with the Prometheus exporter above):

```promql
# Tokens by model and type across the org, 1h rate
sum by (model, type) (rate(grok_code_token_usage_total[1h]))

# Sessions per team per day
sum by (team_id) (increase(grok_code_session_count_total[1d]))

# Tool-permission denial ratio
sum(rate(grok_code_tool_decision_total{decision="deny"}[1h]))
  / sum(rate(grok_code_tool_decision_total[1h]))
```

## Debugging

Set `OTEL_LOGS_EXPORTER=console` / `OTEL_METRICS_EXPORTER=console` to print
redacted records to **stderr** (suppressed in `agent`/`headless` entrypoints
to keep captured logs clean). Export errors never surface in the TUI; check
the debug log.