Hosted observability

Open your organization’s dashboard and alerts, see who can change what in them, and register a destination of your own.

Every organization gets its own Grafana workspace, with one managed dashboard and three managed alerts. What you see belongs to your organization alone.

Open your workspace

Servo creates the workspace the first time a member asks for it, so there is nothing to install or import. A signed-in member of your organization asks with GET /v1/observability/workspace, and Identity and auth covers signing in. The reply carries status and operational_entry_url, the dashboard address with your organization already selected.

Open operational_entry_url in a browser once status is ready, which means Servo has read the dashboard back from Grafana. provisioning and degraded mean it is being prepared or repaired, so ask again in a moment, and disabled means your organization is disabled. An organization admin can have Servo check it again with POST /v1/observability/workspace/reconcile.

Print a dashboard address under each of your deployments from your computer:

your computer

servo deployment list --links

The command also prints a Rerun address. Use operational_entry_url for the dashboard with your organization selected.

Roles in Grafana

Your role in Servo sets your role in Grafana, and Servo keeps the two in step.

In Servo	In Grafana	May
Organization admin	Editor	Build dashboards and alert rules of their own
Member	Read-only role	Look at dashboards and alerts, and change none

Neither role is Grafana’s Admin, the only role that can edit a data source, so nobody in your organization can repoint one at another organization’s data.

Servo reads your organization’s member list when it creates the workspace and every five minutes after that. It adds a new member and removes a departed one as it happens, and it re-checks a member each time they ask for the workspace. A person you remove loses access at the change or the next read. A changed role reaches Grafana at the next check or when that member next asks for the workspace, not at the change.

The performance dashboard

The read-only dashboard in Servo Managed shows action timing across your deployments. It carries the servo-managed tag, covers six hours, and refreshes every five minutes.

Panel	Answers
Actions and Errors	How many action requests the endpoint processed, and how many ended in an error
Server and Model p50 and p95	How long the endpoint took per action all told, and how long the model itself took
H264 decode p50 and p95	How long camera decoding took, for cameras sent as H.264
Endpoint round trip p50 and p95	How long the client waited on the network and endpoint together
Time breakdowns at p50	Server: queue, model, camera decode, and other work. Client: capture, encode, request, wire, and response
Server p95 latency over time	Whether the slow tail is drifting
What ran	Which model, GPU, host, data type, compile mode, and embodiment made the actions

The server total already includes queue, decode, and model time, so do not add nested stages together.

The dashboard is built from metrics only. The workspace also has Tempo and Loki data sources for traces and logs, and a log line that carries a trace id links to its trace.

Robot metadata comes from the authenticated inventory API: stable ID, name, type, site, labels, readiness, and last-seen state. Logs and traces use robot.id to scope robot views; Prometheus metrics do not repeat robot IDs.

Servo re-applies this dashboard and the three alert rules each time it checks your workspace, about every five minutes, so an edit to them does not last. Servo never changes a dashboard or rule it did not create, so build your own beside them.

Managed alerts

Three alert rules sit beside the dashboard, and all are advisory: they show state in Grafana and never change what a robot does.

Alert	Fires when
Servo — control deadline margin exhausted	The margin between queued actions and the wait for the next chunk is below zero in the worst five percent of chunks over the last five minutes
Servo — released action missing controller receipt	More actions were released than controller receipts were recorded over the last five minutes
Servo — OTLP ingestion appears dark	No spans reached the workspace for ten minutes, so the other alerts have nothing to read

Each rule must hold for one minute before it fires. A rule with no data counts as firing, so a rule shows firing until the signal it reads first arrives.

Servo creates the rules and does not configure where a firing alert is sent. GET /v1/observability/views lists the dashboard and the deadline and receipt alerts with their versions and the question each answers.

How isolation works

Isolation is enforced where the data lives, not by a filter in a dashboard.

• Each Servo organization has its own Grafana organization, and its data sources send your organization’s identity to the log, trace, and metric stores.
• Telemetry arrives with a write-only token for your organization. A gateway checks the token and sets the tenant itself, so a sender cannot name another organization.
• Your workspace’s members are mirrored from your organization’s member list. The only other members are Servo’s own administrator accounts, which are not on that list and are kept.
• When an organization is disabled, Servo removes every member from its workspace.
• The hosted API takes your organization from your sign-in, never from the request, and an object owned by another organization looks the same as one that does not exist.

Register your own destination

An organization admin can register an OpenTelemetry destination of your own beside the hosted workspace. Registering records the destination and does not change where your robots or endpoints send telemetry.

• POST /v1/observability/destinations registers an HTTPS endpoint with a credential that Servo stores write-only. Listing shows the endpoint origin and status, never the credential.
• PUT /v1/observability/destinations/{id} repoints or rotates it, resets verification to pending, and revokes the old credential when you supply a new one. It also re-enables a disabled destination, which needs a new credential.
• POST /v1/observability/destinations/{id}/verify reports verified only when Servo can query a unique probe back from the destination. No query-back check is wired in, so it reports pending with the code query_back_not_configured, and it refuses a disabled destination.
• DELETE /v1/observability/destinations/{id} disables the destination and revokes its credential.
• Admins read every change in the audit log at GET /v1/observability/audit, which never holds a credential.

A robot process that connects with an endpoint grant exports to the route in that grant, ahead of any OpenTelemetry settings already in its environment. A worker that Servo starts is given the hosted route. SERVO_TELEMETRY_PROFILE=off turns a process’s Servo export off.

You now have

• A workspace for your organization, and a role in it that follows your role in Servo.
• A dashboard that shows where each action spent its time, and alerts for late control, missing receipts, and silent telemetry.
• The rules that keep your data apart from every other organization’s, and an admin route to register a destination of your own.

Next

Identity and auth

Sign in as yourself, and grant lab-mates a role on a robot.

First motion

Preflight model inputs, then run a customer-controlled episode.