Security agents can already coordinate tools and pursue extended objectives, but many deployments still begin with known alerts, fixed workflows and familiar attack behavior. A more difficult challenge is discovering what existing defenses fail to see, creating detections for those gaps and confirming that the new rules work beyond the attack trace that inspired them.
NVIDIA and CrowdStrike explored that challenge through an agentic offense-defense system evaluated in an isolated environment modeled on NVIDIA accelerated computing infrastructure. The design joined Nemotron open models, CrowdStrike Falcon telemetry and specialized agent harnesses in a recurring test-and-improve loop. Nemotron 3 Ultra orchestrated defensive work, while a fine-tuned Nemotron 3 Super generated and repaired detections as a bounded specialist.
The results suggest that model specialization, grounding and validation can matter as much as the underlying reasoning model. However, the study was limited to one scenario family, small detection sets and restricted benign traffic. It should therefore be read as a directional system-level case study, not a general cybersecurity benchmark.
Turning offense and defense into a continuous learning loop
Conventional red-and-blue exercises involve several manual transitions. An offensive team runs an attack, defenders inspect the resulting evidence, detection engineers revise coverage and the offensive team retests the system. Those handoffs restrict how many attack variations and defensive iterations can be examined.
The evaluated agentic architecture connected those activities in a closed loop inside a representative environment. Each run produced an offensive action trace and Falcon sensor telemetry. Defensive agents used that evidence to reconstruct events, locate visibility or detection gaps and produce candidate rules. Validated detections then informed another independently seeded attack, allowing the offensive side to search for an alternative path with knowledge of the updated defense.
The loop contained four stages:
- Execute and capture: A red-agent harness selected and ran an attack path based on a threat-informed objective. The harness recorded its actions while Falcon endpoint sensors collected telemetry.
- Process and reconstruct: A blue-agent harness received the action trace, telemetry and wider attack context. Grounded with information about available data sources and CrowdStrike detection-engineering expertise, it assessed what could be reconstructed, which existing detections fired and where gaps remained.
- Generate and validate: The defensive workflow authored candidate detections. A validation harness checked them, replayed them against the recorded telemetry and returned failures for correction. Candidates that passed could be deployed to the detection engine.
- Retest and adapt: A separately seeded attack pursued the same objective after deployment. Detection and alert context went back to the red-agent harness, which explored other attack or evasion paths and supplied fresh evidence for another defensive cycle.
This process was intended to continue until the offensive harness could find no further viable route in the modeled environment. That endpoint applies only to the environment and objective under test; it does not establish that every real-world route has been eliminated.
Building a representative test environment
NVIDIA supplied a sanitized natural-language specification representing its accelerated computing infrastructure. An agent-assisted process converted that description into an isolated target environment instrumented with Falcon platform sensors.
The evaluation measured progress through observable attack milestones, action traces and sensor telemetry rather than accepting the agent’s account of its own success. NVIDIA security experts reviewed the environment and threat paths for realism. The same reviewed environment was then used across attack runs, detection tests and evaluation metrics, giving the configurations a consistent basis for comparison.
Specializing the defensive harness
Producing a plausible-looking query was not enough. The defensive harness had to turn attack evidence into detections that were syntactically valid, grounded in available telemetry and behaviorally robust. Six mechanisms supported that goal:
- Schema knowledge: Agents could enumerate supported Falcon sensor schemas, fields and query syntax, reducing the risk of invented fields or invalid queries.
- Telemetry grounding: Red-agent traces, Falcon data and wider attack context tied conclusions to observed events and relationships.
- Specialized authoring: A customized Nemotron 3 Super generated and repaired detections as a bounded expert, separate from the longer orchestration task.
- Artifact linting: Automated checks rejected syntax errors, unsupported fields and rules bound to particular IP addresses, hosts, users or subnets. Feedback encouraged behavioral signals instead of environment-specific strings.
- Detection replay: Candidates ran against captured attack telemetry. A query that returned no matching event was rejected and sent back for repair, even if it otherwise appeared valid.
- Independent review: A separate judge working with fresh context assessed behavioral alignment, robustness and the appropriate use of multiple signals.
Failures generated structured feedback for another attempt. This made correction part of the workflow and encoded several checks that detection engineers would ordinarily perform through manual review.
Customizing Nemotron for orchestration and detection generation
The optimized pipeline divided responsibilities between two models. Nemotron 3 Ultra reconstructed attack sequences, planned the detection-engineering workflow and invoked tools. When the system needed to write or repair a detection, a customized Nemotron 3 Super handled that narrower job. The separation kept broad orchestration apart from specialized query authoring.
CrowdStrike used Nemotron 3 Super as the foundation for its NL2LogScale model. Its development combined continual pretraining on cybersecurity knowledge, supervised fine-tuning and reinforcement learning with verifiable rewards. Fine-tuning used 9,349 detection-generation and multistep-repair examples covering 59 programmatically generated error types. The data included request rephrasings produced by Nemotron 3 Super, genuine Falcon LogScale execution errors and quality-reviewed Nemotron 3 Ultra reasoning traces.
For reinforcement learning, NVIDIA NeMo Gym validated and executed generated queries in Falcon LogScale. Invalid queries received actual engine errors and could undergo as many as five repair attempts; unresolved attempts earned zero reward. Valid generated queries and their references ran over the same synthetic logs. The F1 overlap between their returned events determined the reward, so training valued matching execution results rather than syntax alone. NVIDIA NeMo RL then supported group relative policy optimization for model updates.
The source also reports that CrowdStrike’s Blue Solano defensive model was more accurate than the leading proprietary frontier model it tested at 99% lower cost. That statement comes from CrowdStrike internal evaluations and is not independently corroborated by the supplied evidence.
Evaluating the system from backtest to live fire
A rule that matches the exact telemetry used during authoring may fail when the same underlying behavior is executed again. The evaluation consequently used two stages: backtesting against the recorded attack and live-fire testing against eight new attacks from the same scenario family.
Every passing detection was deployed without modification to the live engine. An independent third-party model judged whether a match reflected the intended attack behavior, and the reported results were averaged across independently seeded authoring sessions.
Backtesting against the recorded attack
Nemotron 3 Ultra operating with the default harness produced detections with a mean recorded-attack detection rate of 16.5% across eight independently seeded sessions. The optimized open pipeline retained Ultra but added the tuned harness, customized Nemotron 3 Super, domain context, tools and validation. It reached 41.9% across six sessions, described as a 2.5-times improvement.
This comparison does not isolate the contribution of either model. Both the harness and the model stack changed, so the improvement belongs to the complete optimized pipeline rather than to a single-model ablation.
Evaluating generalization through live-fire testing
The live-fire stage deployed 11 backtest-passing detections from the optimized Nemotron pipeline and 35 from the complete frontier system against eight unseen attacks. Five of the 11 open-pipeline detections fired on at least one attack, a rounded rate of 45%. Ten of the 35 frontier-system detections fired, or 29%.
The source says the open pipeline averaged 2.6 “detections per detection,” versus 1.1 for the frontier system, although it does not further define that metric label. The validation funnel supplied a clearer view of quality. A detection had to fire, remain quiet on available test traffic and pass independent review for behavioral grounding, multiple signals and the absence of environment-specific strings before it could qualify as gold.
Four of the five firing open-pipeline detections stayed quiet, and those detections covered all eight attacks. Nine of the 10 firing frontier detections stayed quiet and covered seven of eight attacks. After independent review, three open-pipeline detections qualified as gold and collectively retained coverage of all eight attacks. No frontier-system detection reached gold status.
Interpreting the results and their limits
The frontier system generated more detections that passed backtesting, but the optimized open pipeline produced a higher proportion that fired in live testing and was the only configuration to yield gold detections. Those outcomes support the system design within the reported experiment, but they do not establish universal superiority.
Only one scenario family was evaluated, and both detection sets were small. Generalization across other scenarios therefore remains unknown. The benign traffic available for the quietness check was limited, so those results cannot stand in for production false-positive measurements. Three of the eight live-fire runs also experienced harness failures. They were retained because complete telemetry was still produced, but the failures are material when assessing operational reliability.
Applying the pattern to other specialized agents
The architecture points to a reusable pattern: assign broad planning and tool orchestration to a reasoning model, give a post-trained model a narrowly bounded expert task, and use the harness to control context, tools, feedback and validation. Four principles summarize the approach:
- Define measurable tasks and give each model an explicit responsibility.
- Post-train bounded specialists with domain evidence and rewards tied to verifiable outcomes.
- Ground outputs in authoritative context, then test them with deterministic checks, realistic replay and independent review.
- Evaluate the entire workflow under realistic conditions, with security experts governing scenarios, guardrails and the use of validated outputs.
CrowdStrike is pursuing this offense-defense coevolution pattern through SafeMind. NVIDIA identifies Nemotron 3, NeMo Megatron Bridge, NeMo Gym and NeMo RL as components for customizing and evaluating other specialized agents. The central lesson of the case study is that an adaptive security system depends on more than autonomous reasoning: it needs observable evidence, constrained specialization, executable tests, independent quality gates and candid measurement of failure.
Source: Adapted from the NVIDIA Developer Blog article “Building an Adaptive Agentic Cybersecurity System with NVIDIA Nemotron.”
