Security Testing at Machine Speed

We've spent a decade building offensive security tooling at Trickest. Bug bounty platforms, automation frameworks, workflow orchestration. Across every release, every customer conversation, every incident report, the same pattern has held: attacker speed curves up and defender speed stays flat. The gap AI is opening now makes everything before it look like a rounding error.

Attackers are getting faster. Defenders are buying AI agents that promise to close the gap and getting a different problem dressed up as a solution.

1. The Asymmetry

Start with the ratio. CrowdStrike's 2026 Global Threat Report tracked the average eCrime breakout time at 29 minutes in 2025. The fastest: 27 seconds. IBM's 2025 Cost of a Data Breach Report put the average breach lifecycle at 241 days. Twenty-nine minutes versus eight months. That is the gap.

The vulnerability pipeline confirms the acceleration. Rapid7's 2026 Global Threat Landscape Report found exploited high and critical severity vulnerabilities more than doubled year over year, surging 105%. The median time from vulnerability publication to CISA's Known Exploited Vulnerabilities listing dropped to 5.0 days. VulnCheck found that 32.1% of flaws were weaponized before detection or within 24 hours of disclosure in the first half of 2025. Google Threat Intelligence Group tracked 90 zero-days exploited in the wild in 2025, up from 78 the year before. Enterprise exploitation reached an all-time high, and for the first time, commercial surveillance vendors were attributed to more zero-day exploitation than state-sponsored espionage groups.

AI didn't invent phishing. It removed the skill floor, and the numbers on the ground reflect the industrialization. IBM's 2026 X-Force Threat Intelligence Index found that AI can generate a convincing phishing email in five minutes, a task that takes experienced human operators sixteen hours: a 192x efficiency gain. KnowBe4's 2025 Phishing Threat Report found 82.6% of all phishing emails now contain AI-generated elements. Okta documented attackers using generative AI to build complete phishing sites in under 30 seconds. The classic tells, typos, weird phrasing, broken English, have vanished. WormGPT, FraudGPT, and a growing catalog of dark-web LLMs sell for roughly $100 to 200 a month on cybercrime forums. Real products with real customers.

State-sponsored actors are using the same tools. Microsoft and OpenAI's joint threat intelligence documented this as early as February 2024. By October 2025, Microsoft's annual Digital Threats Report tracked over 200 instances in a single month where foreign governments used AI to generate fake online content, a tenfold increase since 2023. OpenAI's February 2025 threat intelligence update documented threat actors from North Korea, China, Russia, and Iran integrating LLMs into reconnaissance, social engineering, and malware development. The pattern is consistent across every report: AI didn't create new attack categories. It industrialized the existing ones.

The researcher side mirrors the attacker side, accelerating in both directions at once. HackerOne's 9th Annual Hacker-Powered Security Report documented a 210% surge in valid AI vulnerability reports year-over-year, a 540% increase in prompt injection attacks, and 560+ valid reports from fully autonomous hackbots. The platform calls it the "hackbot arms race." A fifth of the reports in one major program were outright AI-generated noise, per Synack's research team. Among the noise, real findings are being discovered faster by AI-augmented researchers. The side that integrates AI into an operational system first wins the speed advantage.

Attackers are using AI. That part is settled. Defenders now choose between black-box agents and deterministic automation, and the choice carries real consequences.

2. The False Promise of AI Agents

The vendor response to this asymmetry has been predictable: fight AI with AI. Ship an autonomous agent. Launch it at your attack surface. It'll find the vulnerabilities, tell you what to fix, close the gap. That pitch sounds compelling until you examine what these agents actually produce.

Non-Determinism Is a Feature of LLMs, and a Bug in Security

HackerOne published a model benchmark in May 2026 comparing frontier LLMs on vulnerability triage. GPT-5.5 returned "Fabricated" on its first run of a vulnerability assessment, then correctly identified it as "Valid" on a rerun with identical inputs. Claude models returned consistent verdicts, but the HackerOne team noted that 75% of errors were shared by two or more models. The gap between a generic agent and an optimized one, they found, "far exceeds the gap between models."

The practical consequence: run the same AI agent against the same vulnerability with the same input and you get different answers on different runs. In security, that's a failure mode. A non-deterministic validation pipeline means you can't reproduce findings, can't audit decisions, can't prove to a regulator that a specific vulnerability was tested and verified. You have a machine's opinion, and the machine changes its mind.

Synack published a piece in June 2026 called "Nobody's in the Cockpit" that put numbers on the problem. They cited the curl project shutting down its bug bounty program after finding that 95% of submissions were not valid, with about a fifth outright AI-generated noise. Major bug bounty platforms now report 60 to 80% of submissions are invalid, driven by AI-generated false positives. Their conclusion: "LLMs have no concept of truth. They assemble security-sounding language into something that resembles a finding. It looks professional and it has CVSS scores. But it's often wrong."

On Vectara's Hallucination Leaderboard, the industry's most-cited factual consistency benchmark, the numbers are stark. On the harder dataset using longer, more complex real-world documents, GPT-5.5 landed at 9.3% hallucination. Gemini 3.1 Pro at 10.4%. Claude Opus 4.6 at 12.2%. GPT-5.4 Mini led at 5.5%. The models marketed as the most capable, the reasoning models, performed worse on harder content. In security, a 10% hallucination rate means one in ten findings is fiction. You still verify everything the AI tells you, which means you haven't saved any work. You've added an AI-shaped triage layer on top of your existing triage problem.

The Accountability Problem

Non-determinism in security testing isn't just an operational concern. It's becoming a regulatory liability.

DORA, the EU's Digital Operational Resilience Act, has been in force since January 2025. Chapter IV, Articles 24 through 27, mandates that financial entities establish a "sound and comprehensive digital operational resilience testing programme." Critically, entities must "establish internal validation methodologies to ascertain that all identified weaknesses, deficiencies or gaps are fully addressed." That sentence carries weight: you must prove, with a documented methodology, that every finding was verified and every fix validated. A black-box AI agent that produces non-reproducible output doesn't satisfy that requirement.

NIS2, the EU's updated cybersecurity directive, runs in parallel. Article 21(f) mandates "policies and procedures to assess the effectiveness of cybersecurity risk-management measures." Not tool output. Policies and procedures. A documented, repeatable process.

The SEC's cyber disclosure rules create the same pressure in the US. When a material cybersecurity incident triggers board-level scrutiny, the question isn't "did you have a tool running" — it's "what testing did you perform, when, and can you prove it?" An AI agent that changes its verdicts between runs turns a compliance exercise into a credibility problem.

The EU AI Act is establishing risk categories for AI systems, and security AI systems that make autonomous decisions about infrastructure access or vulnerability classification are moving toward high-risk classification. That brings mandatory human oversight, transparency requirements, and conformity assessments. Pointing a black-box agent at production and trusting its output is no longer just an engineering risk. It's a compliance risk.

The pattern across all four regulations is the same: they demand evidence, not trust. A system that can't reproduce its own output can't produce evidence. The agent vendors selling "set it and forget it" security are selling a product that, by design, fails the test regulators are now writing into law.

3. The Deterministic Alternative

Deterministic workflows, built by humans who understand the methodology, executed at machine speed, with AI assisting in building and triage instead of replacing human judgment.

This approach rests on something the agent vendors can't offer: reproducibility.

A deterministic security test produces the same result every time you run it against the same target with the same parameters. In the Trickest model, a security test is a DAG, a directed acyclic graph of nodes. Each node is a concrete operation: run a port scan, check a certificate, enumerate subdomains, fuzz an endpoint, validate a finding. The graph defines the methodology. When you run the graph, you get the same execution every time. If something changes, a new CVE in your stack, a new subdomain in your attack surface, the graph surfaces the delta. Not a different opinion from a different model run.

This matters for three practical reasons.

Reproducibility. You run the same test on Tuesday and Thursday and trust that any difference in output reflects a real change in your environment, not a model hallucination.

Auditability. Every step is recorded. You can show exactly what was scanned, with what tool, with what parameters, and what the output was. DORA compliance becomes a property of the system, not a documentation exercise you run after the fact.

Accountability. A human designed the methodology. When a finding is reported, you know what logic produced it. When a false negative happens, you can improve the methodology instead of retraining a model and hoping.

Human-Designed, Machine-Executed

The methodology comes from your team. From the engineers who know your stack, your risk profile, and your adversary model. Not from a vendor's opinion of best practice or a model trained on everyone else's data.

If your team has a specific way of validating authorization bypass, a sequence of requests they send and edge cases they check, that methodology becomes a workflow. It runs identically whether you trigger it once a quarter or once an hour. If your team discovers a new technique for testing a particular technology, they add a node to the graph. The methodology improves, and the improvement is permanent. Institutional, not trapped in someone's head.

AI assists in building and composing. The Trickest agent can build workflows from natural language descriptions. You describe the methodology, it composes the graph. It reads a skill file, a documented testing procedure, and builds the DAG. Every node it places is visible. Every edge is inspectable. You can modify, extend, or reject anything it produces. The agent accelerates the build process. The human owns the methodology.

This inverts the usual pitch. Instead of pointing an AI at production and trusting its output, you use AI to build the workflow with you. The workflow runs deterministically. You get the speed of AI-assisted creation with the reproducibility of deterministic execution.

The Primitive Model

The platform operates on a small set of primitives:

Nodes. The atomic units: tools (Nmap, Nuclei, httpx, ffuf, your own scripts), modules (pre-built testing components), splitters (fan-out points that dispatch work across hundreds of machines), scripts (any language, any logic). You use the tools you trust.

Edges. Data flows between nodes. The output of a subdomain enumeration feeds into a port scan. The port scan feeds into service fingerprinting. The fingerprinting feeds into targeted vulnerability scanning. Structured data, not text you have to grep.

Graph. The DAG that encodes your methodology. Build it in a visual editor, generate it with the agent, or compose it in code. The graph is the methodology.

Fleet. The execution infrastructure. Managed machines, your own self-hosted hosts, or both. 30-second spin-up. Tear-down when the run ends. No idle VPS bills.

This model doesn't replace your team's expertise. It amplifies it. Your best engineer's methodology, running across hundreds of machines, on schedule, with structured output. That's what "machine speed" actually means.

4. What This Looks Like

A realistic offensive security workflow on the platform:

[Asset Inventory] → [Subdomain Enumeration] → [Live Host Discovery]
                                                    ↓
                                              [Port Scanning]
                                                    ↓
                              ┌─────────────────────┴─────────────────────┐
                              ↓                                           ↓
                      [Web Service Probe]                        [SSL/TLS Analysis]
                              ↓                                           ↓
                      [Vulnerability Scan]                       [Certificate Validation]
                              ↓                                           ↓
                              └─────────────────────┬─────────────────────┘
                                                    ↓
                                              [Result Aggregation]
                                                    ↓
                                         [AI-Assisted Triage]
                                                    ↓
                                              [Report Generation]

Each arrow is a structured data flow, not a shell pipe. Each node produces output that the next node can consume without parsing. The splitter at stage 4 fans the web service probing across 300 parallel runners. The methodology scales from 10 subdomains to 10,000 without modification.

The scheduling layer runs this graph on whatever cadence you need. Weekly recon that diffs against last week's run so you see what changed. Triggered runs when a new CVE drops in your stack. Continuous monitoring of your subsidiaries' attack surfaces after an M&A event. The methodology stays the same. You dial the speed.

The AI-assisted triage step at the end isn't making autonomous decisions. It's suggesting prioritization based on the structured output the workflow produced: CVSS scores, exploitability indicators, asset criticality from your CMDB. A human reviews the suggestions and makes the final call. The AI speeds up the triage. The human owns the decision. Because every step is logged, you have a complete audit trail from discovery to remediation.

5. The Speed Math

The security industry loves saying "at scale" without defining it. Here are actual numbers.

The Trickest platform has executed over 15 million jobs across more than 500,000 machines. The fleet spins up in 30 seconds. A single workflow can fan out across 300 parallel runners, each a dedicated machine executing its node independently, with results flowing back into the graph in real time.

What that means operationally:

Subdomain enumeration that takes one security engineer 4 hours on a laptop takes 4 minutes on 60 parallel runners.

Port scanning across a class B network that would take 2 days serially completes in under an hour. The results stream into the graph immediately, triggering the next stage while scanning is still in progress.

Vulnerability scanning that used to be a quarterly event because it took 3 weeks to run end-to-end now runs weekly. The same workflow that tested 100 assets last week tests 100,000 this week. The only difference is fleet size.

The economics follow the same curve. Splitters dispatch work horizontally with no queue manager, no central bottleneck, no serial dependency between parallel nodes. The graph scheduler walks deep DAGs without a depth penalty. Branch, merge, splitter, custom container, it's all the same engine.

Output lands in structured tables backed by ClickHouse. You don't spend post-processing time grepping through tool logs. The data is queryable the moment the run completes. "Show me every finding with CVSS >= 7.0 across all assets tagged 'production' that appeared in the last 4 weeks" is a database query, not a grep-and-spreadsheet exercise.

The Right Conversation

The model benchmarks are clear: AI can find some vulnerabilities faster than humans, unreliably. The real question is whether you can build a testing program that stays deterministic, reproducible, and auditable while running at machine speed.

You can. Not by pointing an opaque AI agent at production and trusting its output. By encoding your team's methodology as deterministic workflows, running them at machine speed, and using AI where it helps: building the workflows and prioritizing the results.

Your team designs the methodology, encoding expertise as executable workflows rather than outsourcing to a vendor's opinion of best practice. Those workflows run across hundreds of machines in parallel, on schedule, with structured output. AI assists in generating workflows from natural language descriptions and suggesting triage priorities from the resulting data, but the human owns every decision and every decision can be reproduced, audited, and defended to a regulator.

AI helps attackers scale. That's real, and it's not reversing. The defender's answer isn't a black-box AI agent making autonomous security decisions. It's deterministic workflows, built by humans who understand the methodology, executed at machine speed.

We built this. 15 million jobs in, across half a million machines, this is what works.

See how deterministic security testing works in practice. Explore the Trickest platform — build your first workflow from a natural language description or deploy one from the community library in minutes.