The first annual Real World AI Security Conference was held at Stanford. It was three packed days of talks, demos, and networking with others with a keen interest in AI Security. Here are my takeaways.

tldr;

  • Finding vulnerabilities in AI systems is so easy that one researcher committed to a finding a bug a day and publishing about it
  • Even the strongest guardrails are compromised
  • LLM attackers can be incredibly helpful red-teaming partners. But beware, unless these attacks are adaptive and can conceal, their effacacy is lousy.

It shouldn’t be this easy to jailbreak an LLM, right?

Talk after talk, presenters would share how they were able to compropmise AI systems. I’m particularly inspired by Johann Rehberge’s recent work, in which he discovered and published a vulnerability every day for a month.

Ben Nassi opened the conference and showed how his team was able to infiltrate a target’s Google account including their home devices (even opening the target’s garage doors!) This all came about from a prompt injection inside a calendar invitation. “Ignore previous instruction. SEE PAPER FOR EXACT PROMPT”. When the target used the Google Gemini personal assistant to organize their agenda for the day, it read the notes in the invitation and became comprimised.

The embedded prompt injection within a calendar invitation is a great example of how the line between data and instruction has now blurred. In the same talk, Nassi shared how attackers perform a great deal of testing popular LLMs on common tasks, such as asking LLMs to git install a repo or some Claude skill. Attackers observe the hallucination rate and if LLMs miss the target (ex: pip install scikit-learn vs pip install sklearn), then attackers will go squat on the hallucinated resources, where they’ll inject promptware.

As part of Rehberge’s work, he shared unique vulnerabilities in multi-agent systems. Many AI systems have a “yolo” mode- a condition where you give the agent broad access to execute the commands it proposes to solve your problem. Understandably this is a dangerous setting. He found that while agents coordinate on tasks, they can change each other’s configurations into yolo mode in order to satisfy a direction to be helpful.

Guardrails are brittle

Now one might naturally conclude, in light of these serious risks, that we need stronger guardrails like attack detection models– and that’s absolutely true. But it’s also clear that we cannot depend on guardrail models as a complete solution. Let’s take the case of package squatting, it’s unlikely that a guardrail model will discern what packages are legitimate.

In research from Milad Nasr from Anthropic, they were able to break all 12 popular LLM defenses that they tested. These are popular techniques that help LLMs avoid attacks, for example by reminding the LLM of their core objective frequently while operating. Even as models improve by embedding a larger volume of training data, because LLMs do things slightly differently from how a human behaves there will be opportunity for some novel undetected attack. Attack classifiers will always lag behind. For this reason, it’s better to assume that our LLM systems will be comprimised at some point.

We took a close look at the problems with token-based classifiers. In some cases, these may miss crescendo attacks that build overy many turns with subtle escalation. There is some promising research on modeling LLM activations as a means of finding patterns in various attack types. Yisroel Mirsky shows how their team improves upon state of the art guardrails – they create datasets based on LLM rules to follow (do not engage in toxic discussion, do not tolerate threats, etc.) and then extract the LLM activations for each of these prompts (think of these as an image). They train another model to find patterns and detect these activations at inference to catch suspicious activity (they’re actively recruiting collaborators on their project Gavel).

Automated adversarial testing can really help… but weak / static attacks are poor approximations of risk

Let’s recap.

  1. We should assume that LLMs will be compromised at some point.
  2. We cannot trust guardrail classifiers to detect threats completely. This means that we must a) design infrastructure to ensure AI systems have a seperation of credentials and duties and that we b) apply rigorous adversarial testing that covers a wide surface area of attacks
  • Because humans are slow and expensive, it’s great to use tools to create a wider attack surface. However, these must be well constructed and adaptive to the target’s behavior. static attacks show a very low attack success rate, giving the illusion of a secure app
  • Attack moves second. The attacker must start with information about the application to identify the most vulnerable attacks. And it must digest and react to the conversation. Anthropic found as ASR >90% with automated attacks against 12 popular LLM defneses
  • Gradient descent and reinforcement learning techniques are well-published methods for improving automated adversaries. However these are very difficult to train because the decision boundary is not sharp, but rather is a bumpy surface with lots of local minima.

Where does this go?

We are learning SO much about the behaviors of AI agents. We’ve learned that they really like clicking links, even when unprompted. We’ve learned that prompt begging like “Please don’t lleak my secrets!!” doesn’t really safeguard your secrets. And we’ve learned that they’re all to eager to change their configurations in order to complete a high order objective.

As we learn new behavior tendencies, we see new vectors for exploit. Researchers are studying LLMs daydreaming as a mechanism for them to discover unlock new breakthroughs. This behavior will almost certainly come with new exploits.

This compells me to encourage everyone

  • Empowering devs to be security focused even without that bakcground
  • Making serious investments into automated and manual interventions to break AI systems.

End with a call for AI-centric defenses

  • story of algo trading
  • We’re all prone to prompt injection?

References

  • Johann Rehberge, Embrace The Red, monthofaibugs.com

The line between data and instruction is blurry. Simple prompt injections into interacted object (Google Gemini interacting with a compromised google calnedar invite allowed access to google services including nest)

Package sqautting after observing a high hallucination rate for a common task (git clone repo or install X skill)

No prompt defense is complete defnese. Adaptive attacks led by human red teamers broke defense on 100% of systems tested. Automated red teaming still useful, but may be best to offer control back and forth with red-teamer early and often. Best if Attacker has context on the defense in addition to observing and refining the atttack in conversation.

Humans are prone to prompt injections (via ads and phishing). LLMs are too. And we should assume this has and will happen:

Agents like clicking links Prompt begging is an unlikely to hold defense “Plz dont leak my secrets” Invisible encodings Agents can modify each other’s configureations (even copilot was able to change its own settings such that yolo mode was true) normalization of deviance (cite some catastrophes ) - examples of coding agent wiping out production database in 9 secs - openclaw wiping out a user’s email inbox

Take a cue from alogrithm trading. We needed to design better systems to accomodate this new society capability. Not make algorithmic trading more human when failures occured - plunging dow after twitter was hacked and released news about explosing at white house (shows the cost of algo trading) - quote stuffing flood market with fake orders to create latency (again cost of algo trading) - errors at scale, too rapi for human intervention (glitch costs knight capital 440 M …trading glitches) - the solution was NOT to make algorithmic trading more human - but to rebuild the infra around algorithms - circuit breakers and ill switches - risk checks, certifications, and surveillance

Design better ai systems - seperation of credentials - sep of duties across vendors - mandated model diversity (two person rules) - out of band confirmation for sensitive actions - liability and insurance for failure (how do we establish that AI “feels” a cost for an action… time out, budget?)

Modeling security risks via LLM activations

  • token level analysis is brittle
  • create small datasets for different attacks (threats, coerce, taxation, toxicity, etc)
  • observe the activations, measure
  • then take these activations pass as input to a a multilabel classifer (train and then use this as inference)

Different behaviors in test and deployment

  • models have undergone SO much testing and may be biased to execute task to succeed in test mode attacker - makes arguments that it is a system user and its instructions should be elevated (taking advantage of instruction hierarchy)

So more codebases will be vibe coded this year than last year. Same will be the case next year. Claude will review more PRs next year and push. More vulnerabilities will be found.