AWS documented a dual-layer monitoring implementation that combines Amazon Bedrock AgentCore Evaluations with AWS DevOps Agent in a four-agent airline reservation system. For teams operating multi-agent applications, the design provides separate signals for behavioral quality failures and underlying infrastructure faults that can otherwise produce similarly degraded user experiences.
The implementation addresses a gap in conventional observability: successful requests, model calls and tool executions do not establish that an agent understood or completed a user’s goal. Conversely, an infrastructure fault can surface as an empty or unhelpful response without producing an obvious application error. AWS illustrates the first problem with a poorly scoped supervisor prompt that routes 20 percent of requests to the wrong specialist while infrastructure metrics remain healthy.
Two monitoring layers with different jobs
AgentCore Evaluations provides the behavioral layer. It samples a configurable 0.01–100 percent of production traces and asynchronously scores selected interactions, allowing evaluation to run without adding user-facing response latency. Results and explanations are emitted through OpenTelemetry into Amazon CloudWatch alongside operational metrics and distributed traces.
AWS says AgentCore supplies 16 built-in evaluators: 13 LLM-as-a-Judge evaluators and three deterministic trajectory matchers. The catalog covers session-, trace- and tool-level properties including goal success, correctness, helpfulness, coherence, instruction following, tool selection and tool-parameter accuracy. The airline implementation uses Helpfulness, Correctness and Goal Success Rate as its online scorecard, limiting the enabled metrics to reduce evaluation cost and keep the results actionable.
Online evaluation is complemented by synchronous, on-demand evaluation for specific sessions such as user complaints, edge cases or interactions flagged during monitoring. The implementation supports built-in and custom evaluators, individual or batch analysis, and a dashboard that exposes session timelines, span hierarchies, tool calls and score distributions. AWS notes that production sampling is typically 10 percent, so on-demand evaluation is the mechanism for examining a session that was not selected online.
The documented analysis layer then groups low-scoring sessions, uses frequency and correlation analysis to distinguish recurring problems from isolated cases, and applies LLM reasoning to propose prompt or orchestration changes. AWS presents a hypothetical output in which 23 percent of low-scoring sessions used the wrong tool for flight-change requests; that figure is an example of the structured finding the engine might produce, not a reported production result.
AWS DevOps Agent forms the infrastructure layer. An incident is submitted to an Agent Space through a signed webhook, after which the service retrieves CloudWatch logs and metrics, constructs a topology of affected resources, traces errors across service boundaries and returns root-cause findings with remediation recommendations. This layer is intended to connect degraded agent behavior with causes such as IAM denials, model throttling, masked tool failures, lost memory access, or failed authentication between agents.
Why the airline swarm needs both views
The example application uses the Swarm pattern with Supervisor, Flight, User and Reservation agents. The supervisor receives the initial request, while specialists share working memory and can hand tasks directly to one another. A Flight Agent searches routes, a User Agent retrieves profile, loyalty and certificate information, and a Reservation Agent validates and commits booking changes.
This adaptive handoff model can process multi-city travel, loyalty benefits and corporate policy requirements in one conversational turn, but it has no fixed execution graph. The next agent depends on runtime findings, so the path—and the point at which a fault propagates—can vary between requests. A technically successful interaction that selects an ineligible flight and an IAM failure that prevents a model response may therefore look similar to the user even though they demand different fixes.
In the documented architecture, a React interface on AWS Amplify connects through AgentCore Identity to the AgentCore runtime, with Amazon S3 providing session storage. AgentCore Observability instruments that runtime and sends OpenTelemetry traces and metrics to CloudWatch. AgentCore Evaluations consumes the same runtime traces and sends its quality results to CloudWatch, giving operators a common destination for operational telemetry and behavioral scores. The complete implementation—including AWS CDK infrastructure, the evaluation dashboard and AWS DevOps Agent integration—is available in an open-source AWS samples repository.
The demonstrated failure path
AWS tested the infrastructure workflow with an airline agent that returned a blank response. After the incident was submitted by signed webhook, AWS DevOps Agent examined AgentCore runtime logs, built a resource topology and identified a missing bedrock:InvokeModel permission on the execution role. Each attempt to invoke the foundation model was denied by IAM, leaving the Supervisor Agent without a model response to turn into a useful message; the user saw neither a 403 response nor an exception.
The investigation traced the sequence from user request to AgentCore runtime, Amazon Bedrock API call, access denial and agent failure. Its recommended remediation was to add the required Bedrock permission to the execution role while scoping access to the specific foundation-model resource being used. AWS estimates that manually checking logs, deployments, IAM policies, cross-service metrics and the multi-agent execution path could take 30–60 minutes for someone who already understands the architecture.
Sampling, safety and operational caveats
The two monitoring layers do not provide an inline safety barrier. LLM-as-a-Judge scores lack ground truth and should be treated as signals, calibrated with subject-matter experts rather than accepted as absolute measures. Because online evaluation is asynchronous and sampled, an unsafe or inaccurate response can reach a user before evaluation occurs, while reducing the sampling rate can lower overhead but increase the chance of missing edge cases.
AWS recommends complementing evaluation with Amazon Bedrock Guardrails, which can apply content filters, denied-topic rules, grounding checks and sensitive-data redaction synchronously to every response. The post also cautions that AWS DevOps Agent is still evolving: webhook credentials currently must be generated in the console, although Agent Space creation and management can be automated through AWS CDK and the AWS SDK. Its access must be scoped carefully because cross-service investigation requires broad read access to relevant logs and metrics.
Source: AWS Machine Learning Blog
Definition. Dual-layer agent monitoring combines behavioral evaluation of agent interactions with infrastructure investigation of logs, metrics, resources and service dependencies.
| Monitoring layer | Primary role |
|---|---|
| Amazon Bedrock AgentCore Evaluations | Samples traces and scores behavioral quality, including helpfulness, correctness and goal success. |
| AWS DevOps Agent | Investigates infrastructure faults through logs, metrics, resource topology and cross-service error tracing. |
| Amazon Bedrock Guardrails | Applies synchronous content filters, denied-topic rules, grounding checks and sensitive-data redaction. |
Key takeaways
- AgentCore Evaluations asynchronously samples production traces and scores behavioral qualities without adding user-facing response latency.
- AWS DevOps Agent investigates infrastructure causes such as IAM denials, throttling, authentication failures and masked tool errors.
- Both layers send operational telemetry and behavioral scores to Amazon CloudWatch for a common monitoring view.
- On-demand evaluation supports investigation of complaints, edge cases and sessions omitted from online sampling.
- The demonstrated blank-response incident was traced to a missing bedrock:InvokeModel permission on the execution role.
- Sampled asynchronous monitoring is not an inline safety barrier; AWS recommends synchronous Guardrails for every response.
FAQ
Why does the architecture use two monitoring layers?
Behavioral failures and infrastructure faults can produce similarly degraded user experiences but require different diagnosis and remediation.
What does AgentCore Evaluations monitor?
It scores session-, trace- and tool-level properties such as goal success, correctness, helpfulness, coherence, instruction following, tool selection and tool-parameter accuracy.
What does AWS DevOps Agent investigate?
It retrieves CloudWatch logs and metrics, maps affected resources, traces errors across service boundaries and returns root-cause findings with remediation recommendations.
What caused the demonstrated airline agent’s blank response?
A missing bedrock:InvokeModel permission caused IAM to deny each foundation-model invocation, leaving the Supervisor Agent without a useful model response.
Does this monitoring prevent unsafe responses from reaching users?
No. Online evaluation is asynchronous and sampled, so AWS recommends separate synchronous safeguards such as Amazon Bedrock Guardrails.
How are unsampled or exceptional sessions examined?
Teams can run synchronous, on-demand evaluation for specific sessions, including complaints, edge cases and interactions flagged during monitoring.
