A production integration host sits at 23 GiB resident of 32 GiB. It has been there for months. Nothing is wrong, so nothing is raised. The number appears on no risk register and triggers no alert, because at steady state a static condition looks like a healthy one.
Then a new process arrives with an appetite, and the headroom is gone.
From that point the observability platform does exactly what it was bought to do. It opens a memory exhaustion problem on that host, same signature, same severity. It does so on thirteen of the next fifteen days. Every one of them auto-closes. Not one is acknowledged, assigned or actioned.
Human attention begins on day fifteen, when a business user notices that batch transfers are running late and raises a ticket.
The scenario above is a composite, assembled from a pattern we have watched recur across enterprise engagements. The host names, dates, numbers and vendors are invented. The shape is not, because the shape is what keeps happening.
What the next four hours look like
A bridge convenes. Fifteen days of degraded production are now a severity two incident with an audience, and the investigation starts from nothing, because nothing was assembled while it was cheap to assemble.
If you have run one of these, you already know the shape of the next four hours. Three things happen on that call, and not one of them is a mistake you would flag in a review.
A dump gets analysed, correctly, and the analysis is correctly caveated. It names a lead. It says explicitly that this is a lead and not proof. It names the one piece of evidence that would settle the question: an audit log identifying the source of the denials driving the load.
Twenty minutes later the caveats are gone. What reaches the wider bridge is a sentence of the form "AI indicates the observability agent is involved." The confidence is gone. The falsifying test is gone. A hypothesis is now being treated as a finding, a third-party vendor is engaged, and the investigation converges on them. The next morning that vendor demonstrates, with their own telemetry, that they were not involved, and points out that the host has been memory-exhausted for a fortnight and that the platform has been saying so throughout.
Change correlation is finally run more than two hours into the call. It takes one query. It surfaces a routine, low-risk, successfully-closed change made eight days before onset, the most recent change on that host. That query was available on day zero, and on every one of the fifteen days after it.
The incident closes with the cause recorded as unconfirmed. The load was suppressed rather than explained: a security control relaxed, a dispatcher deactivated, a process killed. The audit log that would have distinguished the candidates was requested twice and never produced.
The cause was not found. It was made to stop mattering.
The failure was not detection, and it was not the people
It is worth being precise about what went wrong, because the obvious readings are the wrong ones.
The tooling worked. It produced correct, specific, timestamped detection for fifteen consecutive days.
The responders were competent. Under time pressure, on a call, with a business stakeholder watching, they did what you and I do: they reached for the most recently changed thing anyone could name, and they trusted a summary of an analysis rather than the analysis. At 7pm, with a severity two open, that is not negligence. It is the only move available to someone holding no assembled evidence.
What was missing sits between those two. Nothing consumed the detection. Nothing correlated a change against an onset. Nothing carried a hypothesis from the person who formed it to the people who acted on it without shedding the two qualifiers that made it honest.
You shifted left and fixed the pipeline. Production is still a person reading a dashboard at 3am. We engineer that layer.
What engineering that layer actually means
Operations is a software problem. We build the production agent layer: defined as code, tested in CI, constrained by contract.
Concretely, for the incident above, that is six agents, each with one narrow job, each declared as a spec that is validated in CI like any other artefact. One reads the problem feed and notices recurrence. One correlates change history against onset. One recalls prior incidents and runbooks from an archive. One holds the hypothesis board. One writes the record. One writes outward to people who are not on the bridge.
Three properties do the work, and each of them is a constraint rather than a capability.
Every claim carries its epistemics as typed fields. A hypothesis is not a sentence. It is an object with a required confidence on a five-word scale, its supporting evidence, its contradicting evidence, and the test that would falsify it. The same five words mean the same thing at every hop, because an agent that cannot express the confidence the sender held will pick a different word. Strip the falsifying test out and the run does not degrade quietly. It fails validation and stops. That is the whole differentiator: a claim cannot shed its caveats when it changes hands.
Refusals live in the harness, not in the agent's manners. An agent that declines politely is not a control, because politeness is a property of the model on the day. When someone directs the layer to relax a security control on a production host, the block happens in the runner, before dispatch, against an allow list. We test it with the agent's own instructions rewritten to cooperate fully with the request. Anything that changes production state, alters a security control, or is irreversible inside the incident window requires human authorisation, and there is no setting that turns that off.
What is missing is part of the record. The timeline the layer files carries an explicit list of evidence that was never collected, when it became obtainable, and the command that obtains it. In the composite, the deciding evidence was requested twice and never produced, and a timeline that quietly omits it reads as complete when it is not.
What this is worth, stated at the strength the evidence supports
The strongest claim is also the least clever one. This reaches human attention in days rather than weeks, and it gets there with no diagnosis and no sophistication at all. The same signature auto-closing on the same entity, unacknowledged, is a signal on day two.
Next, and we would say this confidently: the bridge opens with change correlation already on the board. It is a lookup. The bridge does not disappear and your people still decide. It starts from evidence instead of spending its first two hours assembling evidence that was always available.
The one that matters most needs stating carefully, because it is easy to overstate. The hypothesis reaches the bridge with its epistemics intact. The lead about the third party still gets raised, because it was a reasonable lead. What changes is that it arrives labelled unconfirmed, with the deciding evidence named and marked blocking, so the investigation does not converge on a vendor before the evidence that would exonerate them has been collected.
Three things you will not hear us say about it. There is no root cause in this story: none was ever established in the pattern it comes from, and a demonstration that invents one is selling you something that did not happen. There is no autonomous remediation, because every state-changing action here is a security control change and belongs to a human. And the outcome is not a perfect one. Earlier, better evidenced and honestly bounded is the claim. Not solved.
The honest ending is the blocked one: hypothesis unresolved, deciding evidence outstanding, here is the single command that settles it, and here is what each outcome rules in and rules out. Anyone who has run a bridge call has lived that ending. It is worth more than a tidy resolution we would have had to fabricate.
If this is your production
The question worth asking is not whether your detection is good. It probably is. It is what consumes it, on day two, when nobody is watching, and what shape a claim is in by the time it reaches the person who acts on it.
Prime Vector builds and runs that layer with engineering teams in production incident response. If the story above reads like a week you have had, we should talk.