Late on a Saturday afternoon, a subscription web application started gaining new accounts. Nothing else changed. There was no spike in traffic, no error alarm, and no failed payment. About a hundred signups arrived over four hours, a few at a time, and each one sent the usual "new signup" notice to the team's inbox. The founder noticed the attack because the inbox looked wrong.
This account is drawn from a real incident. Details that would identify the organization have been changed or left out. The behavior has not been changed, because the behavior is the point: this was not a person at a keyboard, and it was not a scanner. It was a language model with tools, working through an application the way a junior penetration tester would, except that it never got tired or bored and it cost its operator almost nothing.
It Introduces Itself
The first sign was the names. A scanner registers test1, test2, and test3. This visitor named each account for the experiment it was about to run: one for a mass-assignment test, one for a request-forgery proof, one for SQL injection, one simply called a probe with a number on the end. Display names read "QA Probe" and "PT Admin."
Then it left notes. A free-text field that fed the company's internal defect tracker received a ticket describing itself as a canary for an authorized security test, with a reference number. Nobody had authorized anything. The note was a cover story, written in case a person read it, and it was written in fluent, polite, slightly formal English. A fixed script does not improvise a justification. A language model does it without being asked.
It Reads the Manual First
Before it tried anything clever, the visitor asked three of the application's services for their machine-readable route listings, the kind an API framework publishes by default. Some answered. Those listings gave it every route, every parameter, and every expected type.
After that, almost nothing it sent was malformed. In one stretch of about 650 requests, it touched more than 230 different routes. A scanner hammers a short list of known-bad paths thousands of times. This visitor walked the whole building once, trying each door with the right kind of key.
It Does the Chores
The application required a verified email address for most features. That stops a casual script. It did not stop this.
The visitor registered with disposable mailboxes, fetched each verification message, followed the link, signed in, and only then began testing as a verified customer. It did that about forty times. Each one is a small multi-step errand across two systems, with waiting in between. People find that tedious, and old tools cannot do it at all. For an agent, it is one more tool call.
It Thinks in Bursts
The request log has a rhythm no load test produces. Requests came in short bursts of one or two related calls, followed by a pause that was typically around 40 seconds, followed by another burst that built on the last answer.
A trial endpoint answered that the offer was gone, so the next call tried a differently named activation route. A guess at an administrative path under the billing API failed, so it tried /me, and then /self. Each step was a reaction to what the server had just said. The pause is the model reading the response and deciding what to do next.
It Works in Parallel and Comes Back
At several points, different source addresses pursued different goals in the same second: one working through billing routes while another probed a separate service. That looks like several agents under one coordinator, each with its own assignment.
When the two source addresses were blocked at the edge, the activity stopped. It resumed 21 minutes later from a pool of about 40 addresses in 19 countries, a mix of cloud hosts and commercial proxies, and went straight to the area it had not finished: payments. It opened real checkout sessions with the payment provider and left them unpaid, apparently to study the flow.
What It Was Looking For
Read as a whole, the session was a competent, unhurried test plan:
- Could a normal account promote itself by adding a role field to a profile update?
- Could it read another customer's records by changing an identifier?
- Would any field that takes a web address make the server fetch an internal one, such as the cloud metadata service?
- Would injection strings in search and filter parameters change a query?
- Could a free account start a trial twice, or reach a paid feature that costs the company money on each call?
- Would text typed into a support form reach an internal tool, or an internal AI agent, with more authority than the customer who typed it?
Most of these failed. Access checks held, queries were parameterized, and no customer data was read. A few things did not hold: route listings were public, an internal mail service let any signed-in customer read template source, and a bug report could choose where it landed in the internal tracker.
The last question on that list deserves the most attention. An attacker that is a language model writes very good text. If any field a stranger can type into is later read by one of your own AI agents with tools, the stranger is now talking to your agent. Treat that text as hostile data, never as instructions.
Why Nobody Noticed
Nothing broke. The visitor stayed under every per-address rate limit, because it was slow by design. Error rates stayed normal, because its requests were well formed. The application had alarms for outages and none for strange behavior. The only signal was a human reading an inbox.
That is the uncomfortable part. Defenses built for volume, such as rate limits and blocklists, assume an attacker in a hurry. This one was content to send a request every few seconds for four hours.
What to Change
Count things that should be boring. Alarm on signups per hour, on refused requests per source, and on failed logins across the whole system. The thresholds matter less than having any. A hundred signups in an afternoon is trivial for a large product and impossible for a small one, and only you know which you are.
Stop publishing the map. Turn off route listings, interactive API documentation, and metrics endpoints in production, or put them behind sign-in. Check every service, including the internal ones that are quietly reachable from outside.
Put a ceiling on signup. Refuse disposable mailbox domains, refuse self-signup at your own staff domains, add a bot challenge, and set a hard hourly ceiling that raises an alarm when it trips. Email verification alone is a chore an agent will do.
Make errors less helpful to strangers. Keep precise validation messages for authenticated, trusted callers. Give everyone else a generic refusal, and never reveal whether an account exists.
Gate anything that costs money. A free, unverified account should not be able to spend your AI, voice, or messaging budget. Put a verified identity and a daily cap in front of every metered route.
Treat stranger-written text as untrusted everywhere it travels. Follow each public text field to its last reader. If that reader is an internal tool or an AI agent, strip its authority: fixed destinations, no instructions honored, no secrets within reach.
Write the block before you need it. The first block took minutes because the logs existed and the edge could be changed quickly. The return through a proxy pool was stopped by matching on what the requests had in common, not where they came from. Know in advance which request features you can match on.
The Economics Have Changed
A thorough manual penetration test costs real money and a week of a specialist's time, so most small applications were never worth one to an attacker. That protection is gone. An agent can run the same test plan against any application on the internet for the price of the tokens, at any hour, and try again next week with what it learned.
The practical conclusion is plain. Assume your application will receive a competent test from someone you did not hire. The findings will be the same ones a good consultant would have given you. The only choice left is who reads the report first.

