Blog · November 22, 2026 · 7 min read

What an AI-Led Attack Looks Like:
Four Hours With a Patient Machine

About a hundred signups in four hours, and nothing else changed. It was a language model with tools. What that looks like, and what to change.

A single milling machine running alone at night in a vast, empty machine shop, one amber work light glowing inside its closed enclosure.The same empty machine shop on a bright morning, one milling machine at its center under pale light from tall windows.

Late on a Saturday afternoon, a subscription web application started gaining new accounts. Nothing else changed. There was no spike in traffic, no error alarm, and no failed payment. About a hundred signups arrived over four hours, a few at a time, and each one sent the usual "new signup" notice to the team's inbox. The founder noticed the attack because the inbox looked wrong.

This account is drawn from a real incident. Details that would identify the organization have been changed or left out. The behavior has not been changed, because the behavior is the point: this was not a person at a keyboard, and it was not a scanner. It was a language model with tools, working through an application the way a junior penetration tester would, except that it never got tired or bored and it cost its operator almost nothing.

It Introduces Itself

The first sign was the names. A scanner registers test1, test2, and test3. This visitor named each account for the experiment it was about to run: one for a mass-assignment test, one for a request-forgery proof, one for SQL injection, one simply called a probe with a number on the end. Display names read "QA Probe" and "PT Admin."

Then it left notes. A free-text field that fed the company's internal defect tracker received a ticket describing itself as a canary for an authorized security test, with a reference number. Nobody had authorized anything. The note was a cover story, written in case a person read it, and it was written in fluent, polite, slightly formal English. A fixed script does not improvise a justification. A language model does it without being asked.

It Reads the Manual First

Before it tried anything clever, the visitor asked three of the application's services for their machine-readable route listings, the kind an API framework publishes by default. Some answered. Those listings gave it every route, every parameter, and every expected type.

After that, almost nothing it sent was malformed. In one stretch of about 650 requests, it touched more than 230 different routes. A scanner hammers a short list of known-bad paths thousands of times. This visitor walked the whole building once, trying each door with the right kind of key.

It Does the Chores

The application required a verified email address for most features. That stops a casual script. It did not stop this.

The visitor registered with disposable mailboxes, fetched each verification message, followed the link, signed in, and only then began testing as a verified customer. It did that about forty times. Each one is a small multi-step errand across two systems, with waiting in between. People find that tedious, and old tools cannot do it at all. For an agent, it is one more tool call.

It Thinks in Bursts

The request log has a rhythm no load test produces. Requests came in short bursts of one or two related calls, followed by a pause that was typically around 40 seconds, followed by another burst that built on the last answer.

A trial endpoint answered that the offer was gone, so the next call tried a differently named activation route. A guess at an administrative path under the billing API failed, so it tried /me, and then /self. Each step was a reaction to what the server had just said. The pause is the model reading the response and deciding what to do next.

Your error messages are now read by a patient, literate adversary. Every precise validation error is a hint, and it will use every one.

It Works in Parallel and Comes Back

At several points, different source addresses pursued different goals in the same second: one working through billing routes while another probed a separate service. That looks like several agents under one coordinator, each with its own assignment.

When the two source addresses were blocked at the edge, the activity stopped. It resumed 21 minutes later from a pool of about 40 addresses in 19 countries, a mix of cloud hosts and commercial proxies, and went straight to the area it had not finished: payments. It opened real checkout sessions with the payment provider and left them unpaid, apparently to study the flow.

What It Was Looking For

Read as a whole, the session was a competent, unhurried test plan:

  • Could a normal account promote itself by adding a role field to a profile update?
  • Could it read another customer's records by changing an identifier?
  • Would any field that takes a web address make the server fetch an internal one, such as the cloud metadata service?
  • Would injection strings in search and filter parameters change a query?
  • Could a free account start a trial twice, or reach a paid feature that costs the company money on each call?
  • Would text typed into a support form reach an internal tool, or an internal AI agent, with more authority than the customer who typed it?

Most of these failed. Access checks held, queries were parameterized, and no customer data was read. A few things did not hold: route listings were public, an internal mail service let any signed-in customer read template source, and a bug report could choose where it landed in the internal tracker.

The last question on that list deserves the most attention. An attacker that is a language model writes very good text. If any field a stranger can type into is later read by one of your own AI agents with tools, the stranger is now talking to your agent. Treat that text as hostile data, never as instructions.

Why Nobody Noticed

Nothing broke. The visitor stayed under every per-address rate limit, because it was slow by design. Error rates stayed normal, because its requests were well formed. The application had alarms for outages and none for strange behavior. The only signal was a human reading an inbox.

That is the uncomfortable part. Defenses built for volume, such as rate limits and blocklists, assume an attacker in a hurry. This one was content to send a request every few seconds for four hours.

What to Change

Count things that should be boring. Alarm on signups per hour, on refused requests per source, and on failed logins across the whole system. The thresholds matter less than having any. A hundred signups in an afternoon is trivial for a large product and impossible for a small one, and only you know which you are.

Stop publishing the map. Turn off route listings, interactive API documentation, and metrics endpoints in production, or put them behind sign-in. Check every service, including the internal ones that are quietly reachable from outside.

Put a ceiling on signup. Refuse disposable mailbox domains, refuse self-signup at your own staff domains, add a bot challenge, and set a hard hourly ceiling that raises an alarm when it trips. Email verification alone is a chore an agent will do.

Make errors less helpful to strangers. Keep precise validation messages for authenticated, trusted callers. Give everyone else a generic refusal, and never reveal whether an account exists.

Gate anything that costs money. A free, unverified account should not be able to spend your AI, voice, or messaging budget. Put a verified identity and a daily cap in front of every metered route.

Treat stranger-written text as untrusted everywhere it travels. Follow each public text field to its last reader. If that reader is an internal tool or an AI agent, strip its authority: fixed destinations, no instructions honored, no secrets within reach.

Write the block before you need it. The first block took minutes because the logs existed and the edge could be changed quickly. The return through a proxy pool was stopped by matching on what the requests had in common, not where they came from. Know in advance which request features you can match on.

The Economics Have Changed

A thorough manual penetration test costs real money and a week of a specialist's time, so most small applications were never worth one to an attacker. That protection is gone. An agent can run the same test plan against any application on the internet for the price of the tokens, at any hour, and try again next week with what it learned.

The practical conclusion is plain. Assume your application will receive a competent test from someone you did not hire. The findings will be the same ones a good consultant would have given you. The only choice left is who reads the report first.

Read the report
before they do.

The AWS Security, Compliance & Governance Audit finds what is exposed in your AWS organization, and the six-week Hardening closes every critical finding, verified by the same scan.

The next conversation

Let's define the result.

Tell Charles which challenge is in front of you. He will scope the work and make the result he will measure clear before you sign.

Book a 15-minute call with Charles

Remote engagements nationwide.
Serving clients since 2020.