Most of what I have built so far is the first level of agent observability: which tools, skills, and MCPs my agents are running, how much, and when. My Claude Code and Codex agents push OpenTelemetry into a central stack on my own Kubernetes cluster, so there is now a real picture of agent activity sitting in one place. That is a great starting point.
What the activity layer covers
Level 1 answers “what happened.” Which skill fired, in which session, how often, at what cost, over what window. A DevX team could watch that feed and understand how everyone’s agents are actually operating.
The questions it does not answer
The real questions are the next layer up. How do I know the right things are running? How do I know whether any of this is becoming effective, whether it is good, whether it can be improved? Level 1 tells you activity. Level 2 has to tell you whether that activity is correct, effective, and improving.
I am not sure yet what that second level looks like in detail. Maybe it is a set of additional things inside OTel, built on top of the activity data I am already collecting. That is what I want to research next.
One box, then a team, then an organization
The reason the split matters is that the first level scales the easy way. It works for me across all my sessions on one box. If it works there it should work across a whole team’s agents, and then across another organization’s agents, all of them reporting to one place so a DevX team can monitor and improve agent usage across the org.
That is how an engineering org would learn whether the skills and tooling it builds are actually getting adopted and actually getting better, where the missed opportunities are, and whether the better outputs coming from the senior people can be pulled back into the shared tooling so everyone benefits.
My theory, and it is still a theory: org-level learning, the same way a compound-learning corpus works, combined with this observability, becomes institutional knowledge. Used well, that is a real competitive edge.
The gateway gives the activity layer away
An LLM gateway is the chokepoint every call passes through, so it hands you an activity layer for free as a byproduct: spend, tokens, latency, model used, per-key attribution, one dashboard. Hosted or self-hosted open source, it makes no difference, because the data is there by virtue of happening on the wire. OpenRouter alone is processing about 25 trillion tokens a week, up 5x in six months, and every one of those calls gets counted on the way through.
It is not quite the activity layer I have been collecting, though. A gateway sees the calls, never which skill decided to make them. So what I called Level 1 is really two layers, and only the bottom one comes free. A third sits above both:
- Wire and call. Model, tokens, cost, latency per API call. The gateway gives this away.
- Harness and execution. Which tool, skill, or MCP ran, session shape, accepts and rejects, step duration. Invisible to a gateway, because it happens inside the agent and never touches the wire.
- Workflow intelligence. Did the workflow improve, what did it cost, where did it stall, across humans and systems of record as well as agents.
That third layer is not my framing. It came up in conversation: enterprises do not want agent observability, they want workflow intelligence, because agents are only one actor in the workflow and nobody wants another dashboard.
That is the same split, reached from the market side instead of the engineering side. My bet is that the defensible ground starts at the harness.
Governance decides which product this is
The data captured does not decide whether this is healthy DevX or surveillance. The governance wrapper decides that: who can see what, at what level of aggregation, for what stated purpose, and what the org commits to never doing with it. Same telemetry, opposite products.
That wrapper is the part worth building carefully, because it is what an org keeps.
Collecting the activity data is not the product. Whatever tells you that activity is getting better is, and I do not have that yet.