Every chart and number on the Observability dashboard is defined here in plain English. If a metric on screen isn't covered, file an issue — this glossary should be complete.
#Throughput
#Requests per minute (RPM)
What it is: Count of API calls — both successful and failed — landing on the gateway per minute, attributable to an API key or workspace.
Unit: req/min.
How it's computed: Total request records over the selected window divided by the number of minutes in that window. Each completed request — success or error — counts once.
When this is high: Heavy traffic. Watch for quota headroom and rate-limit pressure on the relevant key.
When this is low: Quiet traffic. Confirm clients are actually online before assuming health — a flat zero often means a misrouted base URL, not calm.
#Tokens per second (TPS)
What it is: Output tokens generated per second across active streams.
Unit: tokens/sec.
How it's computed: For an aggregate view across a model, it's the sum of completion_tokens across all requests in the window divided by the wall-clock seconds covered by those requests. For a single streaming request, it's the streaming rate measured between the first and last emitted token.
When this is high: The model and the path between client and provider are decoding briskly. Good.
When this is low: Slow generation. Causes range from a saturated upstream provider to congested proxy pipes to clients that aren't draining the stream fast enough.
#Latency
#Time to first token (TTFT)
What it is: Milliseconds from the moment the gateway receives a request to the moment the first byte of response is written back to the client.
Unit: ms.
How it's computed: first_byte_written_at − request_received_at per request, then surfaced as p50, p95, and (optionally) p99 percentiles over the window. Captures queueing time, cold-start time, and the model's prefill phase.
When this is high: The user is waiting before they see anything. Look at upstream service time first — if upstream is also high, the model is the bottleneck; if upstream is fine, queueing or proxy overhead is the cause.
When this is low: Snappy starts. Users perceive the model as responsive even before generation completes.
#Inter-token latency
What it is: Milliseconds between consecutive output tokens during streaming.
Unit: ms.
How it's computed: Wall-clock gap between adjacent tokens emitted on a single streaming response, averaged across the stream and then aggregated to percentiles across all streaming requests in the window. Captures decode speed.
When this is high: The model is decoding slowly or the network path is buffering. Streaming feels choppy.
When this is low: Smooth, sustained streaming. The output flows.
#Total request duration
What it is: Milliseconds from request received to response fully written.
Unit: ms.
How it's computed: end_time − start_time per request, stored as latency_ms. Surfaced as average and as p50, p95, p99 percentiles, with optional comparison against the provider-reported upstream service time.
When this is high: Long-running requests. Could be large completions, slow upstream, or stalled streams. Use the upstream comparison to attribute the time.
When this is low: Fast end-to-end completion. Pair with token counts to confirm the request actually did work and didn't truncate early.
#Reliability
#Success rate
What it is: Percentage of requests that returned a 2xx response and a non-error finish_reason.
Unit: %.
How it's computed: (total_requests − error_count) / total_requests × 100, where error counts include any request with finish_reason: error or status_code >= 400.
When this is high: Healthy traffic. The expected steady state for a production workload is above 99%.
When this is low: Something is actively broken. Drill into the error-code breakdown to localize the cause.
#Error rate
What it is: Percentage of requests that returned a 4xx or 5xx response, or whose finish_reason was recorded as error.
Unit: %.
How it's computed: error_count / total_requests × 100. The dashboard sub-breaks errors by status code (400, 401, 403, 404, 408, 429, 500, 502, 503, 504) and by finish_reason.
When this is high: Investigate the dominant error code. 401/403 point at credentials; 429 points at quota or rate limits; 5xx points at upstream provider trouble.
#Missing completion
What it is: Percentage of requests that returned a successful status (2xx, no error finish_reason) but never persisted a final completion field — meaning the stream started but the gateway never recorded a [DONE] event within the timeout window.
Unit: %.
How it's computed: Requests matching the success filter (no error finish reason, no 4xx/5xx) AND missing the completion field, divided by total successful requests, times 100.
When this is high: Upstream instability, client-side disconnects, or proxy buffering. The request technically started successfully — which is why this is distinct from error rate — but the gateway never saw a clean end. Check upstream provider status pages and any intermediate proxy timeouts. Repeated missing completions on a single API key often signal a client that disconnects before draining the stream.
When this is low: Streams are completing cleanly.
#Finish reason breakdown
What it is: Count of completed requests grouped by finish_reason — stop, length, content_filter, tool_calls, error, and not_specified.
Unit: count (and % of total).
How it's computed: Terms aggregation over the finish_reason field with not_specified as the fallback for records that don't report one.
When this is high (for length): Many requests are hitting max_tokens before completing. Consider raising the cap.
When this is high (for content_filter): Many requests are tripping safety filters. Worth inspecting the prompts.
#Spend
#Input tokens
What it is: Cumulative count of prompt tokens consumed over the selected time window.
Unit: tokens.
How it's computed: Sum of prompt_tokens across all matching requests. Available as a single number, as a trend over time, and broken down by model.
When this is high: Workloads with long contexts, RAG pipelines stuffing retrieved chunks, or repeated system prompts. Prompt caching can cut this if the same prefix repeats.
When this is low: Short prompts, or low call volume.
#Output tokens
What it is: Cumulative count of completion tokens generated over the selected time window.
Unit: tokens.
How it's computed: Sum of completion_tokens across all matching requests, surfaced both as a total and as a daily/hourly trend depending on the selected window.
When this is high: Long generations or high request volume. Output tokens are typically priced higher than input — they're the bigger cost lever.
When this is low: Short answers or low call volume.
#Total tokens
What it is: Sum of input and output tokens — the headline usage number.
Unit: tokens.
How it's computed: Sum of total_tokens across all matching requests.
When this is high: Heavy usage. Pair with cost to see whether the spend is concentrated on expensive models or simply spread across high volume.
When this is low: Quiet workload.
#Cost
What it is: Estimated dollar spend across all requests in the window.
Unit: USD ($).
How it's computed: (input_tokens × input_rate) + (output_tokens × output_rate), computed per request at log time using the rate sheet for that model, then summed across all requests. The dashboard surfaces this as a single total, a trend over time, and a top-models leaderboard ordered by cost descending.
When this is high: Concentrated on a few expensive models, or broad high-volume traffic. Drill into the top-models view to see which model owns the spend.
When this is low: Quiet, or running cheap models exclusively.
#Composition
#Modality mix
What it is: Breakdown of requests by request_type — text, image, audio, or embedding.
Unit: count (and % of total).
How it's computed: Terms aggregation over the request_type field, with text as the fallback for legacy records that predate explicit modality tagging.
When this is high (for non-text modalities): Image, audio, and embedding requests are usually priced and rate-limited differently from text. A surge here can shift cost dynamics quickly.
#Filters and dimensions
Per-API-key filtering, status-code filtering as a chart dimension, finish_reason filtering, and time windows longer than 30 days are not wired today.
The dashboard filter bar exposes four core controls today:
- Model — multi-select; populated from the workspace's active model set in the selected window.
- Request type — multi-select;
text,image,audio,embedding. - Time range — presets
now-15m,now-1h,now-24h,now-7d,now-30d. 30 days is the max. - Include prompts & responses — toggle that controls whether request/response bodies are loaded alongside the metric rows.
Three advanced filters are available where the surface supports them:
- Request ID — exact-match lookup for a single request.
- Streaming — restrict to streaming or non-streaming requests.
- Max-tokens range — min/max numeric bounds.
Sort orders on the requests list: time_desc (default), time_asc, lat_desc, lat_asc, cost_desc, cost_asc, tokens_desc, tokens_asc.
Per-request rows carry: id, startTime, endTime, model, latencyMs, totalTokens (split into prompt/completion), totalCost, finishReason, requestType, stream, statusCode, plus modality extras (imageCount, audioVoice, inputLength) on the matching request types. The dashboard composes the metrics above from this row set.
Metric values may lag actual usage by 30–90 seconds because log records flow through asynchronous aggregation before they're queryable. Treat the dashboard as near-real-time, not real-time. For sub-second confirmation that a specific request landed, query the individual request by ID instead.
#Alerts, export, and integrations
#Alerts
Not implemented today. When shipped, alerts will trigger on workspace-level thresholds (error rate, P99 latency, spend).
#Export
Not implemented today. CSV / JSON export of filtered metrics planned.
#Integrations (Datadog / Grafana / Prometheus)
Not implemented today. OTLP-compat metrics emission planned.