Amazon Web Services has unveiled a new pipeline featuring Bedrock AgentCore that deploys, tests, and rejects agent updates before reaching production, aiming to catch subtle regressions through automated evaluation steps integrated with GitHub Actions.
Amazon Web Services is positioning Bedrock AgentCore as a way to put an automated quality gate in front of agent releases, using GitHub Actions to deploy, test and reject changes before they reach production. The company’s latest example centres on an agent that runs on AgentCore Runtime, talks to tools through an MCP server protected by OAuth, and is checked by an evaluation step that can block a pull request when scores fall below a set threshold.
The approach matters because agent regressions are often subtle. A prompt tweak, a model change or a revised tool configuration can degrade answers without breaking the deployment itself. AWS says the pipeline is designed to catch that class of failure by deploying a development stack, invoking the agent with a fixed set of prompts, scoring the resulting traces and then deciding whether the merge should proceed.
At the centre of the design is AgentCore Evaluations, which AWS documentation describes as the platform’s assessment layer for online, on-demand and batch evaluation. For this workflow, the relevant mode is on-demand evaluation, which accepts session trace data and returns scores from built-in or custom evaluators. The service uses OpenTelemetry traces and can judge behaviour across dimensions such as correctness, helpfulness and tool selection, while also supporting ground-truth inputs for tasks where expected responses or tool trajectories are known.
AWS also stresses that the evaluation layer is separate from the live runtime. That distinction is important for CI. The pipeline can invoke the agent, collect traces and then evaluate the session without needing to keep the test system running for the scoring step itself. The company’s documentation says batch evaluation is better suited to multiple sessions and online evaluation to production monitoring, while on-demand evaluation is the fit for immediate checks against a specific run.
The harder engineering problem is authentication. The article describes an MCP server that enforces role-based access control, yet CI runners do not have a human user context. To address that, the example uses a shared Amazon Cognito pool with two flows: client credentials for machine-to-machine use in the pipeline and authorisation code flow for interactive users. AWS says the MCP middleware distinguishes between token types, allowing CI tokens to reach all tools while keeping role-gated tools restricted for normal users.
The implementation combines that auth pattern with CDK infrastructure, a Strands agent, an MCP server and a single evaluation script. According to the post, the script obtains a token, waits for the runtimes to become ready, invokes the agent over HTTPS with a bearer token, and then runs evaluators such as GoalSuccessRate, Correctness, ToolSelectionAccuracy and ToolParameterAccuracy. AWS says the workflow posts results back to the pull request and tears the stack down afterwards, so a failed score can stop the merge rather than merely report on it.
The company also notes practical limits. LLM-as-judge scoring can vary slightly between runs, trace propagation takes time, and the full pipeline can take around 10 minutes. Even so, AWS argues that an automated gate is better than manual review alone, and its own documentation on AgentCore evaluations supports the broader pattern: evaluate specific sessions on demand, keep production monitoring separate, and use ground truth when you need stricter checks on expected behaviour.
Disclaimer: This content is intended for informational purposes only. Readers are advised to exercise their own judgement, conduct due diligence, or consult a qualified expert before acting on any information provided.
