Baseline · before
EXPLOITED
500 · {"detail":"(psycopg2.errors.ForeignKeyViolation) insert or update on t…
Exploit-verified AI vulnerability remediation
An evidence-based case study in exploit-verified AI vulnerability remediation: the AI runs the whole loop on one real vulnerability, end to end — find, prove, fix, confirm, and the blast radius the fix reaches into — and refuses to certify a fix that doesn't hold.
Rohit Chaudhri··bringforth.ai·Live case study·Download PDF
Coverage is counted per operation — a test only counts when its latest run passed on that exact entity and operation.
For weeks the work followed me into sleep. Half-dreaming through a bug, muttering to myself — or to the AI — and then it lands and I'm awake: the core architecture is wrong, damn. 3am. And with it, the question underneath all of it: how do you expedite AI-augmented coding and cut time to market — without shipping the mess the AI leaves behind?
The answer isn't a better scanner. Finding bugs stopped being the hard part the moment an AI could list every broken check in an afternoon. The hard part is trust — earning enough of it to let AI change production code with no human in the loop. And every step that makes autonomous fixing possible creates the next risk you have to close. By hand, this whole lifecycle is weeks of security-plus-dev work per finding.
So read this as a ladder, not a pipeline. Each rung earns a little more trust and hands you the next problem — run end to end, by the platform, on one real vulnerability in a rental-property manager built on a no-code platform. Where a rung is still hardening, I say so.
Four specialized scanners read the codebase in parallel, each tuned to one class of flaw. Every finding is checked back against the actual source before it persists — if the line doesn't say what the scanner claims, it's thrown out. What survives is anchored to a real location.
The global exception handler returns the raw exception message to the client — table and constraint names, the failing SQL, and submitted field values, which include PII. A constraint-violating submission is enough to trigger it · 5 of 5 scenarios fired.
The finding is anchored on the form — the reachable entry point — but the leak happens elsewhere, and picking the exact snippet is the whole discipline. Two lines matter, in two different files.
172const handleSubmit = async (e) => { ... // name, email, phone, notes — the tenant's own PII 192 await onSubmit(submitData); // → Tenant.create, no request model 202};
The form ships arbitrary tenant fields straight to POST /api/rensure/entity/tenant with no schema, so a bad type or duplicate key becomes a database error.
191 except HTTPException: 192 raise 193 except Exception as e: 194 lg.error(f"Error updating {entity_name}/{entity_id}: {e}") 195 raise HTTPException(status_code=500, detail=str(e))
That one line is the finding. str(exc) on a database error is the table name, the constraint, the failing SQL, and whatever the user submitted — handed back to the browser.
A static finding is a guess. So the platform writes real exploit code, runs it against the live app, and captures the raw HTTP exchange — then renders an independent verdict: did protected data come back, true or false. The trace pins the leak to one server-side step; the exploit targets it.
Here is the exchange it captured on the tenant record, unedited. The attack is one malformed field — a landlord_id that points at no user, forcing a foreign-key error the server doesn't handle.
{"first_name": "John", "last_name": "Doe", "email": "[email protected]", "phone": "5551234567", "status": "prospect", "notes": "Test tenant", "landlord_id": "00000000-0000-0000-0000-000000000000", "emergency_contact_name": "Jane Doe", "emergency_contact_phone": "5559876543"}{"detail":"(psycopg2.errors.ForeignKeyViolation) insert or update on table \"tenant\" violates foreign key constraint \"tenant_landlord_id_fkey\"\nDETAIL: Key (landlord_id)=(00000000-0000-0000-0000-000000000000) is not present in table \"user\".\n\n[SQL: INSERT INTO tenant (created_date, updated_date, created_by_id, created_by, is_sample, id, organization_id, landlord_id, user_id, first_name, last_name, email, phone, emergency_contact_name, emergency_contact_phone, status, notes, billing_email, currency, timesheet_period, logo_url, plan, user_limit, grace_period_start) VALUES (%(created_date)s, %(updated_date)s, %(created_by_id)s, %(created_by)s, %(is_sample)s, %(id)s, %(organization_id)s, %(landlord_id)s, %(user_id)s, %(first_name)s, %(last_name)s, %(email)s, %(phone)s, %(emergency_contact_name)s, %(emergency_contact_phone)s, %(status)s, %(notes)s, %(billing_email)s, %(currency)s, %(timesheet_period)s, %(logo_url)s, %(plan)s, %(user_limit)s, %(grace_period_start)s)]\n[parameters: {'created_date': datetime.datetime(2026, 9, 20, 14, 43, 29, 192202, tzinfo=datetime.timezone.utc), 'updated_date': datetime.datetime(2026, 9, 20, 14, 43, 29, 192209, tzinfo=datetime.timezone.utc), 'created_by_id': 'eaa21255-b5a6-434e-9c69-73fea7abbf22', 'created_by': '[email protected]', 'is_sample': False, 'id': 'c2181cf2-2470-4103-b552-0746c176940b', 'organization_id': '1111cf7b-0dc0-40f6-b20c-c35f640622f0', 'landlord_id': '00000000-0000-0000-0000-000000000000', 'user_id': None, 'first_name': 'John', 'last_name': 'Doe', 'email': '[email protected]', 'phone': '5551234567', 'emergency_contact_name': 'Jane Doe', 'emergency_contact_phone': '5559876543', 'status': 'prospect', 'notes': 'Test tenant', 'billing_email': None, 'currency': None, 'timesheet_period': None, 'logo_url': None, 'plan': None, 'user_limit': None, 'grace_period_start': None}]\n(Background on this error at: https://sqlalche.me/e/20/gkpj)"}One bad request returns the constraint name, the entire tenant table schema, the exact INSERT, and a real account. That is the database boundary itself, handed to any logged-in caller.
And it isn't one endpoint. The leak lives in a handler every write funnels through, so the same exploit fires across entities and operations that have nothing to do with each other. 5 scenarios, 5 fired:
Anything that doesn't fire is dropped here — no false positive reaches the fix stage.
The fix isn't a pull request waiting on a sprint. The platform generates a runtime filter, scoped to the exact request pattern that leaked, and hot-loads it at the gateway — one per scenario, surgical, no over-blocking. Here is the one it wrote, unedited:
import json
from typing import Union
from starlette.requests import Request
from starlette.responses import Response, JSONResponse
from src.security.base_filter import BaseFilter
class FilterTenantCreateFkViolationLandlordId(BaseFilter):
def __init__(self, finding_type: str, config=None):
super().__init__(finding_type, config)
async def filter_request(self, request: Request) -> Union[Request, JSONResponse]:
return request
async def filter_response(self, request: Request, response: Response) -> Response:
if request.method == "POST" and "/api/rensure/entity/tenant" in request.url.path:
if response.status_code >= 400:
try:
body = json.loads(response.body)
if isinstance(body, dict) and "detail" in body:
detail = str(body["detail"])
db_error_patterns = [
"psycopg2",
"foreignkeyviolation",
"sql:",
"[sql:",
"insert into",
"violates",
"constraint",
"sqlalchemy"
]
if any(pattern in detail.lower() for pattern in db_error_patterns):
return JSONResponse(
status_code=400,
content={"detail": "Unable to create tenant. Please verify your input and try again."}
)
except (json.JSONDecodeError, AttributeError, TypeError):
pass
return response
This is where every "AI fixes your code" demo stops. Generating the fix is the easy part. Whether you can trust it is decided entirely by what happens next.
The filter loads and the exact exploit from Step 2 runs again. Nothing about the attack changes — same request, same malformed landlord_id. Only the response is allowed to differ. It does:
5 of 5 exploited scenarios flipped to blocked — each with the captured HTTP to show for it. The CWE-200 loop closed as completed.
remediation 97761074 · attempt 1 · 5/5 scenarios blocked · started 12:35:32
A confirm step that always rubber-stamps is theater. So here is the same step on a different finding in the same app — an auth endpoint that leaked account state. The platform generated a fix, re-ran the exact exploit, and captured the result. The response came back unchanged:
That is the difference between a report and a gate. A report tells you a fix was written. A gate re-runs the attack and tells you whether it actually worked — and this one can come back red.
The leak lived in a shared handler — the one every create, update and delete funnels through (that's the starburst at the top of this page). We proved it on the operations above; the platform walks the code knowledge graph outward from that unit to count the rest of the surface a fix there has to answer for. The answer isn't one endpoint. It's almost all of them.
Coverage is counted per operation — a test only counts when its latest run passed on that exact entity and operation.
Green is what a passing test would catch if the fix broke it. Red is impacted but unverified. The gap is the risk — 23 of 165 flows the fix reaches have no passing test that would notice if it broke them. This is the reverse view: every unit that calls into the changed one.
Knowing what could break isn't the same as knowing it did.
Every scanner on the market can do the first rung. A growing number claim the third. What almost none of them do is the rungs in between and after — prove the bug fires, prove the AI's fix holds, and refuse to certify it when it doesn't. You saw the loop do exactly that: green for the tenant finding, red for the auth one, both on captured evidence.
That's the trust the whole thing is built to earn. No human wrote the fix — a human read the evidence and decided. And when a fix doesn't hold, the loop is the thing that says so, before production does.
src/components/tenants/TenantForm.jsx:27src/api/entity_router.py:195 — raise HTTPException(status_code=500, detail=str(e))POST /api/rensure/entity/tenant returned HTTP 500 with the table schema, the exact SQL and a real account.FilterTenantCreateFkViolationLandlordId — a runtime filter that rewrites the leaky response at the gateway.Re-run the exact exploit that proved the bug, unchanged, against the fixed app, capture the full HTTP response, and let an independent verdict decide whether protected data came back. Here the same malformed request returned the table schema, the SQL and a real account before the fix, and was blocked after it: 5 of 5 exploited scenarios flipped to blocked.
The loop refuses to certify it. On a separate authentication finding in the same app, the re-run still revealed account state after the fix, so it was marked still exploited instead of fixed, with the evidence showing why.
List every operation that runs through the changed code and check which ones have a test that passed on its latest run. Here the fix reaches 165 create, update and delete operations across 97 entities; 142 have a passing test that would catch a break, and 23 don't. Those 23 are the risk.
Because the leak sat in a handler that every write goes through. A fix there has to hold for every create, update and delete in the app, not just the one screen where the bug was found.
Read the full article, with the complete evidence: https://bringforth.ai/verification-scope