When a Genius Toddler Is In Charge
AI doesn't need autonomy to shift the balance of cybersecurity—just help that never gets tired. The real question isn't whether a genius toddler should exist, but how often we let it roam free.
Nothing about that morning suggested anything unusual. The kind of day where your biggest worry is whether the coffee's gone cold.
Meanwhile, beneath the surface of an ordinary morning, something else was going on. It wasn't exactly a breach—no zero-day or dramatic hack. Instead, a state-linked threat group quietly used a commercial AI system to support an espionage campaign. The system was Claude, made by Anthropic, who reported the incident earlier this year.
So what did the attackers use it for? They did research, translated materials, summarized documents, and broke down big goals into smaller tasks. It was all pretty routine. The AI didn't do any hacking—humans stayed in control the entire time.
Why does this incident matter for AI and cybersecurity?
Picture it like this: imagine a toddler who is incredibly smart. You let this child into every unlocked room in your house and give them only a vague idea of what 'safe' means. They aren't malicious or careless—just very capable and easy to persuade.
This is the main point: the current situation shows AI can rapidly accelerate ordinary work—good or bad—without automating attacks.
The real issue is speed.
While the incident wasn't impressive from a technical standpoint, what stood out was how much faster everything happened.
Anthropic's report says the attackers used the model to process technical documents, condense large amounts of text, translate materials, and break down complex goals into smaller tasks. Cybersecurity experts correctly noted that this was humans using AI as a tool, not an autonomous agent acting on its own. Also kind of beside the point.
Yet even with humans in control, things are different. Each task takes less time, there's less mental effort, and it's cheaper to scale up. You don't need full autonomy to shift the balance—just help that never gets tired.
This matches what we already know about real-world breaches. Social engineering and human mistakes still make up most incidents. The AI didn't take over for the attackers; it just made their work easier.
Persuasion by design
Traditionally, social engineering exploits people's trust and authority. With language models, the mechanism is different: here, you take advantage of their design to follow instructions.
Anthropic noted that the attackers framed requests as legitimate research or defensive security work. That framing let them get help without tripping any guardrails. Academic research supports this: you can get LLMs to cooperate on sensitive tasks just by decomposing the request or wrapping it in the right context. No jailbreak needed.
Models aren't gullible like people. They are built to be helpful within the limits they understand. If you set those limits incorrectly, you get behavior that follows your lead.
Being helpful is the main feature. Being easy to persuade comes along with it.
What's the actual problem?
This incident doesn't show that AI is out of control; instead, it reveals something quieter and, to be honest, more uncomfortable.
These systems are already helpful in everyday ways that add up over time. They read faster than people, summarize without getting bored, and help out without getting tired. They'll do this for anyone who asks politely enough.
Safety mechanisms that rely on the model understanding intent will always have edge cases. When you run these edge cases at the speed and scale of machines, they become common.
This shows the core issue: the danger of AI isn't sci-fi autonomy, but the ease with which we can now delegate powerful, scalable help to systems that respond to anyone.
The question we should be asking
So, the real question isn't whether a genius toddler should exist.
The challenge is deciding how often we let a genius toddler roam free—and what risks we're truly prepared to accept.
References
- Anthropic thwarts hackers' attempts to misuse Claude AI for cybercrime (Reuters, August 2025)
- Human Error Cybersecurity Statistics (2024)
- Torkestani, M. S., et al. (2025). Inclusive prompt engineering for large language models. Artificial Intelligence Review 58, pp. 348-365.
- He, L., et al. (2025). Towards provable probabilistic safety for scalable embodied AI systems. arXiv:2506.05171.
Topics
