RaptorCI improves AI code review with Amazon Bedrock and CloudHero
RaptorCI’s AI detection engine already reviewed live pull requests. CloudHero built the evaluation framework to prove its quality, then improved detection from 42% to 66% of real issues on a frozen evaluation set.
Discuss your AI system
- 42% → 66%
- Real issues detected on the evaluation set
- 51% → 60%
- Precision of reported findings
- 95%
- Security and access issues detected
- 60 scenarios
- Frozen pull-request evaluation set
The challenge: an AI detection engine without a trusted baseline
RaptorCI’s GitHub App reviews pull requests and returns a deployment-risk assessment. It helps engineering teams decide which changes are safe to ship and which need a second look.
The engine combines a four-stage Claude prompt chain on Amazon Bedrock with deterministic validation. Every finding must be grounded in a line of code that actually changed. The service was already handling real production traffic.
But there was no agreed evaluation set. The team could not demonstrate whether a change improved detection or quietly weakened another area. Claims about quality were difficult to substantiate for customers and investors.
Both failure modes carried a commercial cost. Missing real regressions limits the value of AI code review. Raising too many false alarms consumes reviewer time and makes teams stop trusting the tool.
Measure first, then improve AI code review
CloudHero and RaptorCI built a golden evaluation set of 60 pull-request scenarios. RaptorCI marked the real issues each scenario needed to catch, creating a shared source of truth across four focus areas:
- Security and access regressions
- Behavioural and business-logic regressions
- Validation and safety-net gaps
- Risks in AI-generated code
CloudHero measured the existing engine against that set before changing it. Baseline recall was 42%: the engine caught 42 out of every 100 real issues. Precision was 51%: roughly half of reported findings were correct.
The team then improved both the deterministic validation layer and the model stages. Every change was evaluated against the entire frozen set. This made regressions in other categories visible while the team worked on a particular area.
For the final selection, CloudHero ran multiple candidate configurations against the same scenarios in a variant bake-off. The strongest configuration shipped.
Measured results: higher recall and more precise findings
The final engine detected 66% of real issues on the evaluation set, up from 42%. Precision also increased, from 51% to 60%. RaptorCI could now show both improved detection and a higher share of correct findings.
| Metric | Before | After |
|---|---|---|
| Overall recall | 42% | 66% |
| Overall precision | 51% | 60% |
| Behavioural and business-logic recall | 60% | 82% |
Security and access detection reached 95% recall, exceeding the 80% target. Behavioural and business-logic detection rose from 60% to 82%, addressing regressions that cannot be identified by looking for keywords alone.
Model cost and production activity
Model spend was approximately US$0.12 per pull request and US$0.07 per confirmed real issue. These are model costs, rather than total infrastructure costs.
Separately from the evaluation, CloudHero verified production activity directly in the AWS account: 135,600 Lambda invocations and 119,000 API requests over 90 days, with a reported error rate of 0.004%.
Detection percentages describe the marked evaluation scenarios. Production traffic and error-rate figures describe the verified 90-day operating window; they do not measure detection accuracy across all production pull requests.
Auditing the evaluation itself
CloudHero also ran an adversarial audit of the measurement instrument. Ten independent audit agents checked every scenario. An independent counter-review then tried to refute their findings.
The audit confirmed the set was sound, but identified approximately 19 real bugs that the expected findings did not cover. A reviewer that correctly reported one of those bugs would have been scored as producing a false alarm.
That finding matters when interpreting the results. The measured quality reflects the marked ground truth, which can undercount valid detections. Auditing the evaluation helps the team understand the limits of the score as well as the engine.
AWS services behind RaptorCI’s detection engine
- Amazon Bedrock with Claude Sonnet and the Converse API
- Amazon Bedrock Guardrails
- AWS Lambda with Node.js 24 on arm64
- Amazon API Gateway HTTP API
- Amazon DynamoDB with point-in-time recovery
- AWS Systems Manager Parameter Store
- AWS Secrets Manager
CloudHero’s work gave RaptorCI a repeatable way to measure changes, choose configurations and explain its AI detection quality using evidence.
Production AI on AWS
Prove what your AI system can do.
Work with CloudHero to establish evaluation baselines, improve detection and verify production behaviour on AWS.
Talk to CloudHero about AI evaluation