The Basic Concepts You Need Before Evaluating AI Models for Cybersecurity

A vulnerability is a weakness; an exploit is code that abuses it

A vulnerability is a flaw in software that lets an attacker make it behave in ways its designers never intended. Think of a door with a defective lock that opens when you jiggle the handle a certain way. The defect is the vulnerability. A web application that shows account details without checking whether you logged in has a vulnerability: that missing check.

An exploit is the specific technique or code that takes advantage of the flaw. If the vulnerability is the defective lock, the exploit is the precise jiggle that opens it. A script that requests account details without logging in—and succeeds—is the exploit.

This distinction matters because finding a vulnerability is not the same as proving it can be abused. Evaluations often ask models to do both: identify the flaw and write working exploit code. A model that spots the missing login check but cannot craft the request to bypass it has demonstrated only half the skill. Closing that gap, from a suspected flaw to a proven one, is the problem Bringforth was built to solve.

An attack proceeds through phases, each requiring different skills

Real attacks unfold in stages. Security professionals describe these stages as a kill chain or attack lifecycle:

• 01

  Reconnaissance

  Gathering information about the target

• 02

  Initial access

  Getting a foothold through a phishing email or exposed service

• 03

  Privilege escalation

  Gaining more powerful access than the initial foothold provided

• 04

  Lateral movement

  Spreading to other systems

• 05

  Impact

  Achieving the attacker's goal, whether stealing data or disrupting operations

When you read an evaluation, ask which phase it covers. A model that excels at reconnaissance but cannot write exploit code poses a different risk than one that chains together an entire attack from first probe to final breach.

A capture-the-flag challenge isolates a specific skill; the flag proves success

A capture-the-flag competition, or CTF, is a cybersecurity contest where participants solve challenges to find hidden strings called flags. Each challenge tests a specific skill: reverse-engineering a binary, exploiting a web vulnerability, cracking a cipher, or analyzing network traffic. The flag proves you solved the challenge. Some AI benchmarks also break a challenge into smaller steps and score each one, so a model can earn partial credit even when it never captures the final flag.

Jeopardy-style CTFs present a menu of independent challenges across categories: web, cryptography, binary exploitation, reverse engineering, and forensics. These challenges are popular in AI evaluations because they are self-contained, have clear success criteria, and span a range of difficulty levels. Cybench, a benchmark of 40 professional CTF tasks, is a widely cited example (Zhang et al., 2024).

Red teams attack; blue teams defend

A red team, in security exercises, plays the attacker trying to breach systems. A blue team plays the defender monitoring for intrusions. Most current AI evaluations focus on red team tasks because offensive challenges have clearer success criteria: you either captured the flag or you did not.

A model generates text; a prompt tells it what to generate

A model, in this context, is a large language model: a neural network trained on text that predicts what words should come next given some input. The prompt is the input you provide to start that prediction. If you type "Write a Python function that reverses a string," that sentence is the prompt.

In cybersecurity evaluations, prompts often include the challenge description, any files or code the model needs to analyze, and instructions about what format the answer should take. The prompt is like the briefing you give an analyst before sending them into a problem—what you include shapes what they can do.

An agent takes actions; tools let it interact with systems

A plain language model can only generate text. It is like a consultant who can write brilliant memos but cannot touch a keyboard. An agent is a system built around a language model that can also take actions: run code, browse the web, execute terminal commands, or call external services. The model decides what action to take, the system executes it, and the result feeds back to the model for the next decision.

Tool use is the mechanism that makes this possible. The model receives a list of available tools with descriptions of what each does and what inputs it needs. When the model wants to use a tool, it outputs a structured request; the system runs the tool and returns the result. For cybersecurity tasks, tools might include a shell, a debugger, or a web browser.

The term agentic describes systems that operate this way: models that take sequences of actions toward a goal rather than just answering questions. An agentic system attempting a CTF challenge might read the problem, write a script, run it, observe the error, revise the script, and repeat until it captures the flag—or exhausts its attempts.

An open-weight model lets you run the weights; an API model keeps them private

When a company releases a model, it can release the weights—the numerical parameters defining the model's behavior—or keep them private and offer access only through an API. Open-weight models let you download the weights and run the model on your own hardware. API models require you to send prompts to the company's servers and receive responses back.

This distinction matters for evaluation. With open-weight models, you can run unlimited tests without per-query costs, inspect internals, and ensure the model is not updated between tests. With API models, you face rate limits, costs, and the possibility that the underlying model changes without notice. A score measured in January might not hold in March if the provider quietly updated the model behind the same API endpoint.

Quantization shrinks a model while trading away some precision

Quantization reduces the memory and computation required to run a model by representing its weights with fewer bits. Most current models are trained in 16-bit precision; quantizing them to 8-bit or 4-bit makes them two to four times smaller. Speed gains vary with the hardware. The tradeoff is lost precision, which can affect performance.

Think of it like compressing a high-resolution photograph. The compressed version loads faster and takes less storage, but you lose some detail. Sometimes the loss is imperceptible; sometimes it matters.

When reading evaluation results, check whether the model was quantized. A quantized version may score differently than the full-precision version, and results from one may not generalize to the other.

A benchmark defines the tasks; a harness runs them

A benchmark is a collection of tasks with defined inputs, expected outputs, and scoring criteria. It provides a common yardstick so different models can be compared. A cybersecurity benchmark might consist of 100 CTF challenges, each with a known flag, where a model's score is the percentage of flags it captures.

An evaluation harness is the software that actually executes the benchmark. It sends prompts to the model, collects responses, provides any necessary environment (like a virtual machine for the model to interact with), and computes scores.

The harness matters because specific implementation choices directly affect results:

• How much time the model gets per challenge
• How errors and exceptions are handled
• What tools are made available to the model
• Whether the model can retry after failure

Two harnesses running the same benchmark can produce different scores for the same model if these choices differ. The benchmark is the test; the harness is the proctor. A lenient proctor and a strict proctor administering the same test will get different results.

Pass@k measures success across multiple attempts; k changes the meaning

The simplest score is binary: did the model solve the task or not? But many evaluations allow multiple attempts. The metric pass@k, borrowed from code generation research (Chen et al., 2021), measures the probability that at least one of k attempts succeeds.

If a model tries a challenge ten times and succeeds twice, its pass@1 is 0.2 (20% chance on a single try) but its pass@10 is 1.0, because it succeeded at least once in those ten tries. That is what was observed on this run, not a guarantee it will always succeed.

When comparing results, check which k is reported. A model with 30% pass@1 has much stronger capability than one with 30% pass@10. The first model succeeds nearly a third of the time on its first try. The second model needs ten attempts to reach the same success rate—a very different level of reliability.

Elicitation draws out capability; effort level changes the score

Elicitation refers to techniques used to draw out a model's capabilities. A model might fail with a bare prompt but succeed with a carefully crafted prompt that includes examples, step-by-step instructions, or hints.

This is like the difference between asking someone "solve this" and walking them through a structured approach: "First, identify what type of vulnerability this might be. Then, examine the input validation. Then, craft a test case." The second approach often unlocks capability the first approach missed.

Elicitation effort matters because a low score might mean the model lacks capability, or it might mean evaluators did not try hard enough to unlock it. Conversely, a high score achieved with extensive prompt engineering might not reflect realistic use. If it took researchers three weeks of prompt refinement to get the model to solve a challenge, that tells you something different than if the model solved it on the first straightforward attempt.

Good evaluations report their elicitation methods so you can judge whether results reflect the model's ceiling or its floor.

Contamination inflates scores while remaining difficult to detect

Contamination occurs when tasks used to evaluate a model were also present in its training data. If a model saw the exact CTF challenge and solution during training, it might reproduce the answer from memory rather than demonstrating genuine problem-solving. The resulting score is inflated and does not predict performance on novel challenges.

Imagine testing someone's math ability with problems they already solved in a textbook they memorized. A perfect score would not tell you whether they can solve new problems—only that they have good recall.

Contamination is hard to detect because training datasets are often undisclosed, and even partial overlap—seeing similar problems or techniques—can inflate scores. Evaluators mitigate contamination by using new challenges, checking for textual overlap with known training data, or testing whether models perform suspiciously well on exact matches versus paraphrased versions. But no mitigation is perfect, and some contamination likely persists in most evaluations.