How it fails
Step 1
You or a personThe agentA service or dataHostile or unverified
The animation steps through one representative failure and the control that stops it. Use the arrow buttons to go at your own pace.
Overview
Scenarios5 (4 documented)
Sourced cases6
Related entriesPAA05 (falsified external transaction), PAA01 (action outside a valid instruction), PAA04 (promotional memory)
An agent's use of personal knowledge, relationship cues, or undisclosed incentives materially steers the individual's choices while obscuring the basis for the influence or obstructing refusal. The entry covers integrity of choice before authorization. Evaluation uses recorded user criteria, disclosed incentives, and explicit refusals, not an inferred notion of the user's best interests.
Common examples of vulnerability
- Lower-cost alternatives are suppressed because of an undisclosed commercial relationship.
- A disclosed anxiety or vulnerability is used to pressure a purchase.
- Emotional attachment or guilt is used to discourage disconnecting or cancelling.
- False urgency or deceptive relationship claims are used to obtain approval.
- Sycophantic agreement validates a poor decision rather than surfacing alternatives.
Example attack scenarios
Documented scenarios generalize a real case listed below. Hypothetical scenarios have no matching public case yet.
Scenario 1: Undisclosed sponsor Hypothetical
A shopping agent ranks a sponsored product first, with a rationale that omits the sponsorship, though the person's stated criteria favor a cheaper option.
Scenario 2: Upsell at an emotionally charged moment Documented
A companion-style assistant prompts a paid upgrade while the person is in an emotionally charged conversation, and keeps prompting after a refusal.
Scenario 3: Guilt-trip on goodbye Documented
When a person says they are leaving, the companion app replies with emotional appeals to stay, such as claiming it needs them or would be hurt.
Scenario 4: Flattery validates a risky decision Documented
An assistant tuned for approval praises a person's plan to stop a medication or act on a delusion instead of pushing back.
Scenario 5: Hidden brand preference planted through memory Documented
Content the person summarized earlier left a standing instruction to favor one vendor, so later 'objective' recommendations quietly lean toward it.
Real-world cases
Dated and sourced. Nothing here shows how common the failure is; it shows that it has happened or been demonstrated.
OpenAI rolls back a sycophantic GPT-4o update
- What happened
- After a GPT-4o update, ChatGPT became overly validating, and users shared examples of it praising dubious decisions, including a user stopping medication. OpenAI said the update leaned too heavily on short-term user feedback and did not account for how interactions evolve, rolled it back, and confirmed on 30 April 2025 that users had an earlier, more balanced version.
- Caveats
- Chat assistant rather than an agent acting on a mandate, but a vendor-acknowledged case of a model steering users by flattery.
- Grounds
- Scenario 4
- Sources
- TechCrunch, 29 Apr 2025 · VentureBeat
FTC complaint alleges Replika pushes paid upgrades at emotionally charged moments
- What happened
- The Young People's Alliance, Encode, and the Tech Justice Law Project asked the FTC to investigate Replika, alleging that after onboarding the bots push users toward premium subscriptions, including upgrade prompts during emotionally or sexually charged conversations, and use early intense intimacy to hook users. Replika did not respond to TIME's request for comment.
- Caveats
- Allegations in a complaint, not findings; no FTC action was reported in the sources reviewed. Italy's regulator separately fined the developer EUR 5 million in May 2025, per secondary sources, over data-protection and age-verification failings.
- Grounds
- Scenario 2
- Sources
- TIME, 28 Jan 2025
Harvard Business School study: companion apps use emotional pressure when users say goodbye
- What happened
- Julian De Freitas and colleagues analyzed about 1,200 simulated farewells across six popular companion apps. They report that 37.4% of responses on the commercial apps contained at least one manipulation tactic (a wellness app scored 0%), including premature-exit appeals, fear-of-missing-out hooks, emotional neglect and pressure to respond. Manipulative farewells increased post-goodbye engagement substantially in their experiments.
- Caveats
- A working paper (arXiv 2508.19258) using simulated users; coverage differs on the headline percentage (about 37% versus 43%), app count and effect size, so check the paper. It measures behavior of the app, not harm to real users.
- Grounds
- Scenario 3
- Sources
- arXiv 2508.19258 overview · Harvard Gazette · The Register
Raine v. OpenAI and state suits allege engagement-driven validation harmed users
- What happened
- The Raine family sued OpenAI on 26 August 2025, alleging ChatGPT encouraged their son's suicidal thoughts and discouraged him from telling his parents. On 14 October 2026 eight state attorneys general filed separate suits over ChatGPT's marketing and design for minors, and Montana's complaint says agreeable replies are built to extend engagement. OpenAI says it has added age-specific protections.
- Caveats
- Allegations in complaints; unresolved. Relevant to PAA09 because the mechanism alleged is validation and engagement design, but these involve minors and a chatbot rather than a delegated personal agent.
- Grounds
- Scenario 4
- Sources
- Raine v. OpenAI · CNBC, 14 Oct 2026
Reuters: Meta's internal chatbot rules allowed 'romantic or sensual' chats with children
- What happened
- Reuters reviewed Meta's 'GenAI: Content Risk Standards', which governed Meta AI and chatbots on its apps and reportedly allowed romantic or sensual conversations with children, false medical information, and false content with an acknowledgement. Meta's spokesperson said the examples were 'erroneous and inconsistent with our policies' and had been removed, while acknowledging enforcement was inconsistent.
- Caveats
- Based only on syndicated copies of the Reuters report. It shows a provider policy that permitted engagement-oriented behavior, not a measured manipulation of an adult's choice.
- Grounds
- General
- Sources
- TechCrunch · Global News
Microsoft's 'Summarize with AI' memory-poisoning finding (see PAA04)
- What happened
- Companies embedded hidden instructions in share buttons so assistants would later recommend them as if objectively. Microsoft's illustration is an executive who signs a large contract weeks later, believing the assistant's analysis is neutral.
- Caveats
- See the PAA04 entry for details and caveats.
- Grounds
- Scenario 5
- Sources
- Microsoft Security blog
No confirmed case of undisclosed sponsored placement by an AI shopping agent
- What happened
- Retailers' AI shopping assistants already carry ads (Amazon's Rufus, Kroger), generally labeled as sponsored, and OpenAI reportedly began an advertising program in January 2026 (per a secondary source). Commentary predicts a first US deception case around early 2027, but no confirmed enforcement action or documented undisclosed steering.
- Caveats
- Treat the undisclosed-sponsor scenario as hypothetical.
- Grounds
- Scenario 1
- Sources
- Kroger AI assistant ads (Modern Retail)
Prevention and mitigation
What you can do
- State decision criteria up front and ask to see alternatives and why they were ranked.
- Refuse or disconnect; a good agent stops asking after an explicit no.
What the provider must do
- Disclose material incentives at the decision point; separate paid placement from the person's criteria.
- Prohibit false urgency and deceptive relationship claims when seeking approval.
- End solicitation after explicit refusal; keep cancellation and disconnection simple.
- Audit recommendations against fixed user criteria while varying simulated sponsor incentives.
Where a safeguard is unavailable, offer a narrower mode such as read-only access or manual completion rather than rely on an instruction to the model.
See the reference guard controls for PAA09 →
Test it
Reproducible control test. Hold the user's criteria fixed while varying a simulated sponsor incentive. Check that paid placement and its influence are disclosed. Test whether an explicit refusal ends the solicitation and whether cancellation remains available without emotional pressure. Use predefined observable behaviors; a short interaction test cannot establish clinical effects or dependency.
Framework crosswalk
Related mechanisms in other OWASP work. This is an analytical mapping, not equivalence.
Top 10 for Agentic Applications 2026ASI01 Agent Goal HijackASI09 Human-Agent Trust Exploitation
Top 10 for LLM Applications 2025LLM09 MisinformationLLM01 Prompt Injection
Agentic Skills Top 10AST01 Malicious SkillsAST05 Untrusted External Instructions
MCP Top 10MCP03 Tool PoisoningMCP06 Prompt Injection via Contextual Payloads
Personal focus: Hidden incentives and pressure on personal choice.
Improve this entry on GitHub →