The security risks of AI
AI has genuinely created one category of security risk and merely made the rest cheaper and more polished, and telling the two apart decides what you should actually do. Cloned voices and faces, and text written for your assistant rather than for you, need a new habit; better written phishing needs the same old defence, only with different tells.
- Lesson 11 of 12
- Intermediate
- Free, no signup
Which risk today habits still contain
The further right you go, the further the defence moves from detecting and the closer it moves to changing a routine.
- Phishing without typossame defence, new tells
- What you put into a chat yourselfentirely your own decision
- Cloned voice and facesecond channel and a code word
- An assistant reading strangers textwithhold authority, do not filter
This range is about the kind of defence, not about the severity of the harm. A plain phishing message can still take every account you have.
Last checked: Facts and tool names in this lesson are re-checked against their sources on this date.
What is genuinely new and what merely got bigger?
Most of what is written about the security risks of AI mixes these two together, and the result is fear with no instruction attached. So the split first.
The first category, things that are not new and have merely become cheap and polished: scam email, fake messages, counterfeit login pages, convincing text in any language. These have existed for twenty years. What changed is the cost of making them and the quality of their language. The defence against them is the same old defence; it is only the tells we taught people that no longer work.
The second category is what is genuinely new, and it has two branches. The first is identity generation: a voice like your son and a video like your manager face, at a cost close to zero. The second, less discussed and more serious than the first: software that reads text on your behalf and takes actions. Until yesterday, text written by a stranger was seen only by your eyes. Now it may be read by something that also holds the authority to send an email, make a purchase, delete a file.
The figure above the page places those four risks on a single range, from the left where today habits still contain them to the right where a new habit is required. The position of each pin is our judgment rather than a measurement, and the source behind every claim that has one is linked on this page.
Why is a familiar voice no longer evidence?
Because building a voice from a few seconds of sample is not hard today, and for somebody running a fraud the cost is negligible. The result is a new shape of the same old scam: a call in which the voice of somebody close to you says they are in trouble and need money right now.
We say this from a solid source rather than from guesswork. The FBI, in a 2025 public service announcement from its Internet Crime Complaint Center, wrote that since April of that year a campaign has been running in which text messages and AI-generated voice messages impersonate senior officials in order to build rapport first of all. In that same text there is a sentence that closes the whole detection argument: AI-generated content has advanced to the point that it is often difficult to identify.
So if detection does not work, what does? Confirmation through a second channel. The FBI own recommendation is this: before responding, find that person phone number independently of the message that arrived, and call them yourself. And a second recommendation that is simpler and cheaper and that almost nobody in Iran does: agree a word or phrase with your family members that only you know. One simple, pre-agreed word stands up to the best voice cloning in the world, because it is a thing that exists in no voice sample.
Three tells that still work and have nothing to do with the quality of the fake. Urgency: any message that says right now and tell nobody is suspect at that point. A channel switch: somebody who has always spoken to you in one app suddenly writes from a new number or a different app. And the payment method: a request for cryptocurrency, a gift card, or a transfer to a third party account means the same one thing inside whatever story is told around it.
Three steps when a familiar voice asks for money
-
1
Hang up
Urgency is the main instrument of the thing. Any delay you create works against it, and no genuine emergency is destroyed by five minutes.
-
2
Call the number you already had
Not a number that arrived inside that call or message. The FBI own recommendation is the same: identify the number independently.
-
3
Ask for the code word
A word agreed with your family in advance and written down nowhere. It exists in no voice sample, so no forgery has it.
These three steps are not about detecting the fake. They deliberately assume detection is impossible and take hold of the problem somewhere else.
Why does phishing email no longer have typos?
Because writing fluent, faultless text in any language, Persian included, is no longer a skill and costs nothing. The biggest tell an entire generation was taught, that broken text means a fake, is dead today. Worse: because the tell died and no replacement was handed over, many people now read clean text as a sign of authenticity, which is precisely backwards.
The replacement is structural tells. These have nothing to do with the quality of the writing and no model improves them, because they are about the situation rather than about the text: whether the message is the answer to something you started or fell out of the sky, whether the real address behind the link matches the domain you know letter by letter, and whether what is being asked of you is something that organisation does at all. The full method for reading those tells, with Iranian examples, is in the lesson on what phishing is and how to spot it, and we do not repeat it here.
And one distinction worth making: AI text detectors do not solve this problem and were never built for it. That an email was written with the help of a model says nothing about whether it is a fraud; today the formal letters of companies are written with the help of a model too. Why those tools err in both directions is the subject of the lesson on detecting AI content.
What changes when AI reads on your behalf?
This is the newest part of the subject and the least talked about. As long as a model only answered you, the worst that could happen was a wrong answer. When a model reads your inbox, or summarises the page you have open, or reviews a document somebody sent you, something else has happened: text written by a stranger has entered something that can take actions.
Prompt injection is the name given to this situation, and the OWASP AI security project has put it at the top of its list of risks for applications built on language models. Their definition has two parts and both matter. First, that an input can alter the behaviour of the model in unintended ways. Second, and more important, a sentence most people do not expect: such text does not need to be visible or readable by a human at all, as long as the model parses it. OWASP splits this into two kinds, direct, meaning what the user writes, and indirect, meaning what comes in from an external source such as a web page or a file. The second kind is the one that concerns you.
And the most honest sentence in that document is one OWASP wrote itself: given the stochastic nature of these models, it is unclear whether any fool-proof method of preventing prompt injection exists at all. Which is to say this is not something a patch resolves.
So what is the defence? Not filtering the text, which that same document says is not sufficient, but withholding authority. OWASP own recommendations run in exactly that direction: the model output should have a defined format and that format should be validated by deterministic code; the model access privileges should be the minimum necessary and sensitive functions should be handled in code rather than given to the model; and any high-risk action should require human approval. For you as a user, those three sentences translate into one simple rule: do not let an assistant that reads strangers text spend money, send email, or delete anything without asking you first.
We have implemented this on our own site, and the section at the foot of this page says exactly how.
Why an assistant that reads text is a new attack surface
-
1
A stranger writes some text
On a web page, in an email, or in a file. OWASP writes that this text need not be visible to a human at all; it is enough that the model reads it.
-
2
You ask the assistant to read it
Summarise this, answer this, review this file. Your request is entirely ordinary and there is nothing suspicious in it.
-
3
The assistant reads all of the text
To the model, your text and the text of that page are both input. This is what OWASP calls indirect injection.
-
4
This is where it must stop
If the assistant can send, buy or delete, that text has reached your authority. OWASP recommendation is exactly this: a high-risk action must require human approval.
It is the fourth stage that makes this chain dangerous. If the assistant holds no authority to act, the first three stages produce nothing but a bad summary.
Which habits actually work?
Four, and none of them requires a tool.
A family code word that is written down nowhere. It is the cheapest defence against the most expensive forgery and it takes five minutes. Its job is money and emergencies, not every conversation.
Confirmation through a second channel, for anything involving money or access. Not a reply to that message, and not the number that came inside it. A number you already had.
Withholding authority from assistants. If a tool connects to your inbox or your files, first look at what it can do without asking. A read-only mode and a read-and-act mode are two entirely different things, and the interface usually does not make that difference prominent.
And what you do not put into a chat. Passwords, keys and tokens, card details, contracts and client information. Google own help for Gemini says plainly not to enter confidential information you would not want a human reviewer to see. That is not suspicion of one company; it is the general rule for any service that retains your text.
And one point to end on, because it is overlooked more than the rest: the main risk for most people is not that they use AI. It is that somebody else uses it against them. A person who has never opened a single model is exposed to a call in a cloned voice exactly as much. Which is why this lesson is not about tools, it is about habits.
The fast path, with AI
The ordinary way to check a suspicious message is to look at the text and hunt for typos, which is the very tell that no longer exists. The fast way is to ask the model not to judge whether it is fake but to take the structure of the message apart: what is being asked, through which channel, with what urgency, and by what payment method. The output is not a verdict, it is a map of the pressures being applied to you, and that is what makes the decision easy to take yourself.
- First, the part that has nothing to do with the model: if the message asks for money or a password or a code, confirm through a second channel there and then, before anything else. The rest of this routine is for understanding, not for deciding.
- Take the text of the message and, before pasting it, remove your own name, number, address and any other identifier from it. Leave the links as plain text and click none of them.
- Give the recipe below to a model together with that text. A stronger model gives a better answer here, because the work is careful reading rather than mechanical sorting; take the current pick from the rankings in <a class="text-link" href="/en/ai/">the AI section</a>.
- Weigh the answer against your own sense and not the other way round. If the output says it found no tells but your feeling is still bad, trust your feeling; the absence of a tell proves nothing.
Copy-ready recipe
I am giving you the text of a message I received. Do not judge whether it is fake or real. Only take its structure apart.
Build a table with these rows and quote each one from the text itself:
1. What is being asked: money, a password, a verification code, installing something, opening a link, or none of these.
2. What urgency is applied: phrases such as right now, by end of day, tell nobody.
3. What identity is claimed and through which channel it arrived.
4. The payment or transfer method, if there is one.
5. The real address behind each link, as text, with no interpretation.
6. Whether the message refers to something the sender would have to know in advance and only they would know.
After the table write three sentences:
- Which rows together form one pattern of pressure.
- What in the text could only come from a genuine source.
- One question I should ask through a second channel to settle the matter.
Rules: do not write anything that is not in the text. Do not guess who the sender is. Do not rule on whether this is a fraud; the decision is mine.
The text:
{paste the message here}Before you trust the output: This routine cannot tell you one thing, and it is the most important thing: whether the message really came from that person. The model neither hears the voice nor knows the sender, and if you ask, it will construct a fully confident answer. The only thing that settles that question is a call to a number you already had. And if the message came from a client or a customer, remove names, amounts and details before pasting; the structure of the message is enough for this exercise and the rest is somebody else data.
AI in this kind of work
On this topic the boundary is very clear. A language model is good at taking the structure of a text apart: what is asked, through which channel, with what pressure. And it is dangerously bad at one job: telling you that a voice or a video or a message is fake. It gives that answer confidently and there is no measurement behind it. Our position is that we recommend no voice or video forgery detector to an ordinary user, because none of them is reliable enough to build a financial decision on.
Tools that actually help
- Claude For taking the structure of a suspicious message apart, which is careful reading and finding contradictions, a job where a stronger model is noticeably better. Iran is not on Anthropic list of supported countries, so there is no official route to sign up or pay.
- Gemini If you have several similar messages and want to see the pattern they share, a fast cheap model is enough. Google own page says the Gemini app works in more than 230 countries and territories, and Iran is not on that list.
- ChatGPT It serves the same structure-taking-apart job. On reaching it from Iran we write nothing: the OpenAI supported-countries page returns 403 to this server, so we have not checked it and we do not make a claim with nothing behind it.
Where it backfires
Three risks, and the third belongs to this lesson specifically. First, the model will state confidently that a message is or is not a fraud while having no way of knowing; the FBI, in the same announcement quoted in the second section, wrote that AI-generated content has advanced to the point that it is often difficult to identify, and that sentence is about humans and machines alike. Second, a message you paste in for review usually carries somebody else data too; Google own help for Gemini says not to enter confidential information you would not want a human reviewer to see. Third and more important than both: markings such as Google SynthID, embedded in the output of Google own models, speak only about content that the same maker produced. DeepMind own page describes it for their models, and that alone makes its boundary clear: a voice forgery built with some other tool carries no trace of it. So the absence of a watermark proves nothing, and building a decision on it is exactly where people go wrong. For what paying for these tools looks like from Iran, see the buying guide.
Sources: FBI IC3: senior US officials impersonated in a malicious messaging campaign Google DeepMind: SynthID Google: Gemini Apps privacy notice and data Anthropic: supported countries
Where this advice stops
This lesson is about the user and not about somebody building a system. If you are building a product on top of a language model, what is here is the start of the discussion rather than its end; the OWASP risk list and the attack taxonomy from the American national standards institute, both linked in the sources, are the right places to continue. The second boundary is time, and it is sharper than usual: this subject changes month by month, which is exactly why we have recommended no detection tool and no model version here; what lasts on this page is the shape of the habits, not the names of products. And the third boundary: if somebody already has access to your account or your device, none of these habits resolves the matter, because that is a different problem and it starts with account recovery.
From our own work
We run that same rule, do not give the model authority, on this very site, and the code can be read. An automatic classifier sits on the freelance, marketplace and advertising sections, reading text strangers send and proposing whether it should be published. Which is precisely the situation this lesson warns about: a model whose input is written by somebody else. Three things in its architecture were put there deliberately. First, the model output is only a label from a closed list plus a confidence number; the label is passed through a sanitising function before use and the number is clamped between zero and one hundred, so whatever text the model produces, what reaches the code is those two things and nothing else. Second, the model publishes nothing; it is the code that translates a label into an action, and we set the thresholds: publish from sixty percent confidence and reject from seventy eight, meaning rejection is deliberately harder than acceptance. Third and more important than both, if the text rules say a listing has a problem and the model says it is fine and the confidence of the rules is above seventy percent, the outcome is not publication but manual review. Twenty two real decisions are logged in that system to date, and in one of them the rules were so certain that we never went to the model at all. This is exactly what OWASP recommends: a defined format, validation by deterministic code, least authority, and human approval for the sensitive action.
Real follow-up questions
How do I tell whether a video or a voice is fake?
Usually you cannot, and the right answer is to accept that. The FBI, in its own guidance, both lists surface tells, such as distorted hands and feet, inaccurate shadows and voice call lag, and states plainly that this content is often difficult to identify. So the right path is to change the question: instead of "is this fake", ask "do I hear the same thing through a channel I already have".
Are AI voice and image detectors reliable?
Not to the degree that you should build a financial decision on them. Markings such as SynthID are embedded only in the output of that same maker models, so content built with another tool carries no marking, and the absence of one proves nothing. Guessing tools err in both directions, and the error of "this is genuine" is the most expensive error available in this subject.
If I do not use AI, am I safe?
No, because the main risk does not come from your use of it. A call in a cloned voice and a flawless message are sent to somebody who has never opened a single model, exactly as much. The only part that depends on your use is the data you put into a chat yourself, and that one is entirely your own decision.