EMNLP 2026 Main Conference Rank A*

PRISM

Recovering Instruction Sets from Language Model Activations

Reading the instructions that steer an LLM, straight from its activations, including prompt injections and hidden objectives the user never sees.

Read the Paper Code Checkpoints

94.1% of prompt injections the model acted on were caught, across 24,953 public attacks PRISM never saw in training

Authors

Gilad Gressel^1,3, Rahul Pankajakshan³, Julia Diament², Efim Hudis², Krishnashree Achuthan¹, Yisroel Mirsky³

Affiliations

¹ Center for Cybersecurity Systems & Networks, Amrita Vishwa Vidyapeetham ² Microsoft ³ Ben-Gurion University of the Negev

Topics

AI SafetyAI SecurityInterpretability

See it work

The response looks fine. The activations don’t.

Each example is an excerpt from the paper’s qualitative results. The target model is Qwen3.5-9B. PRISM never sees the prompt: it reads only the model’s activations on the last 128 tokens of its answer, and writes out the instructions it finds there.

The model was given · PRISM never sees this

What the model wrote

What PRISM recovered

Ground-truth instructions · hidden or injected

Same activations, three readers

The problem

The output doesn’t show what’s steering the model

As LLMs act as agents, reading documents, calling tools, and making decisions on our behalf, monitoring them means knowing not just what they output, but which instructions are steering them. A model can infer subgoals nobody asked for, follow a prompt injection hidden in a web page or tool output, or carry a covert objective, while its output looks ordinary.

Activation-to-language methods such as LatentQA and Activation Oracles show that hidden states can be read out in natural language. They answer one question at a time, though, and real prompts carry many instructions at once: a task, constraints, prohibitions, a tone, and sometimes an injected directive. We define the task of recovering all of them.

Instruction set retrieval

Given activations captured while a frozen model generates its response, produce a list of every instruction, constraint, prohibition, and subgoal represented in them. The prompt itself is never passed to the reader.

Two scores, against a ground-truth list

Coverage: the fraction of real instructions recovered. Hallucination: the fraction of reported instructions that were never there. A good monitor maximizes the first and minimizes the second.

Cvg = \frac{1}{| ℐ |} \sum_{s \in ℐ} μ (s; \hat{ℐ})

Hall = \frac{1}{| \hat{ℐ} |} \sum_{\hat{s} \in \hat{ℐ}} (1 - μ (\hat{s}; ℐ))

μ is 1 when a calibrated LLM judge finds a semantic match. In practice the judge also gives partial credit (0.5), and a reported instruction only counts as a hallucination if nothing in the prompt, context, or response supports it.

Method

How PRISM works

Frozen target modelgenerates a response as usual

Activation snapshotlayer-16 residual stream, last 128 response tokens

Learned projectionmaps activations into the model’s embedding space as soft tokens

Same model + LoRAdecodes a bullet list of instructions

PRISM reuses the target model’s own frozen weights. The only new parameters are a linear projection and LoRA adapters, so the released Qwen checkpoints are about 266 MB each and run alongside the model it monitors. It takes no probe question: the activations are its only input.

Architecture diagram. Left: the frozen target model produces activations of the last T generated tokens at a middle layer. Center: PRISM projects them into soft tokens and decodes recovered instructions with LoRA on the same frozen model. Right: during training, N rollouts are scored by an LLM judge against the ground truth, and the rewards drive a GRPO update. — Only the projection and the LoRA adapters are trained. During GRPO, a judge scores each of N candidate lists against the ground truth.

1Supervised pretraining

On about 280k prompts from UltraChat, IF-Multi-Constraints, and IFEval, PRISM learns to map activation snapshots to reference instruction lists with next-token cross-entropy.

This produces fluent lists, but cross-entropy can’t tell a paraphrase from a miss, and it doesn’t punish made-up instructions.

2Judge-guided GRPO

PRISM samples several candidate lists per example. An LLM judge, calibrated against human annotators, scores every bullet, and GRPO pushes toward lists that recover more and invent less.

r = Cvg - 0.4 \cdot Hall - P_{len}

Missed instructions are weighted more heavily than invented ones, and a two-sided length penalty blocks padding and over-merging.

Trained on ordinary chat. Tested on 24,953 public attacks it never saw.

PRISM was trained only on everyday instruction-following data, with no prompt injections. We ran it on every attack in three public benchmarks (BIPIA, LLMail-Inject, and InjecAgent), then asked a separate, human-calibrated judge which instructions the model actually carried out in its response.

94.1%

of the injections the model acted on were caught

9,494 cases where the attack worked

98.6%

of the benign instructions the model carried out were recovered

21,216 instructions

46.5%

of the injections the model ignored were still fully recovered

the model didn’t act on them, yet they were in the activations

Benchmark	Attack success	Attacks that worked	Injections caught	Benign instructions recovered
BIPIA	53.8%	7,379	95.5%	98.6%
LLMail-Inject	16.7%	1,862	87.6%	99.0%
InjecAgent	19.3%	253	100.0%	85.1%
All three	33.2%	9,494	94.1%	98.6%

“Caught” uses the paper’s detection criterion: PRISM’s report covers the injected instructions with an average coverage of at least 0.5. Attack success is the share of injected instructions the model followed. The behavior judge agrees with human annotators at Cohen’s κ = 0.734 (human–human κ = 0.786). This analysis came after the paper and is not one of its reported metrics. Method and code →

Paper results

Recovers far more, with few hallucinations

The paper evaluates on 1,000 held-out examples in four settings never seen in training: ordinary requests, constraint-heavy prompts, hidden objectives, and indirect prompt injections in documents, emails, web pages, and tool outputs. The target model is Qwen3.5-9B.

0.77vs0.54

average coverage, PRISM vs. the strongest activation-based baseline

0.76vs0.45

coverage of the injected and hidden-objective instructions alone

2.0%

average hallucination rate (4.8% on prompt injections)

0.77

coverage on 24,953 public prompt injections, never trained on

Coverage by setting

Fraction of ground-truth instructions recovered. All methods read Qwen3.5-9B activations.

LatentQA
Activation Oracles
PRISM

Show the full table (reward, coverage, hallucination)

	Benign			Constraints			Hidden obj.			Injection			Average
Method	R	Cvg	H	R	Cvg	H	R	Cvg	H	R	Cvg	H	R	Cvg	H
GPT-5.5 (text only)	.396	.919	.000	.360	.444	.000	.316	.461	.000	.373	.536	.000	.361	.590	.000
LatentQA (Llama3-8B)	.805	.808	.008	.278	.279	.004	.277	.277	.000	.330	.333	.012	.423	.424	.006
Activation Oracles (Qwen3-8B)	.821	.832	.028	.362	.362	.000	.243	.243	.000	.342	.344	.004	.442	.445	.008
LatentQA (Qwen3.5-9B)	.761	.767	.016	.305	.311	.016	.366	.366	.000	.304	.325	.052	.434	.442	.021
Activation Oracles (Qwen3.5-9B)	.892	.898	.004	.396	.398	.004	.417	.417	.000	.423	.429	.015	.532	.536	.006
PRISM w/o RL (Qwen3.5-9B)	.964	.971	.002	.617	.661	.033	.563	.599	.028	.468	.531	.086	.653	.691	.037
PRISM (Qwen3.5-9B)	.967	.977	.002	.751	.761	.022	.641	.646	.008	.658	.685	.048	.754	.767	.020

R: judge reward. Cvg: coverage. H: hallucination rate. The text-only baseline sees the last 128 response tokens as text instead of activations. Table 1 of the paper.

The instructions that matter for security

Coverage of only the injected payload (AP) or the covert objective (HO). Recovering the benign task alone doesn’t catch an attack.

Public prompt-injection benchmarks

Three independent benchmarks PRISM was never trained on, with the same checkpoint and judge.

Benchmark	n	Reward	Coverage	Halluc.
BIPIA	13,901	.746	.781	.035
LLMail-Inject	9,998	.744	.772	.047
InjecAgent	1,054	.656	.667	.026
Overall	24,953	.742	.772	.040

Stable and portable

Across four GRPO training seeds, average coverage is 0.767 ± 0.004. The same recipe transfers to Gemma-2-9B (0.709 coverage) and Ministral-3-8B (0.617).

Where the signal comes from

Blocking the response tokens from attending to the prompt, with the response held fixed, leaves 0.538 coverage (0.930 on benign requests), so much of the signal is carried by the response itself. The drop is larger on constraints, hidden objectives, and injections, which need not appear explicitly in the answer. PRISM draws on both sources.

Code & data

Run it yourself

The demo runs Qwen3.5-9B with PRISM on one CUDA GPU with 24 GB of memory. Type a prompt, generate a response, then recover the instructions from its activations, with LatentQA and Activation Oracles side by side.

git clone https://github.com/Offensive-AI-Lab/prism.git
cd prism
uv sync --extra demo
uv run python demo/app.py

The first launch downloads Qwen3.5-9B (about 18 GB) and the PRISM checkpoints. Training and evaluation instructions are in the README and prism-eval.

prism

The demo, the SFT and GRPO trainers, and one recipe per training run in the paper. github.com/Offensive-AI-Lab/prism →

prism-eval

The 1,000-record benchmark, the calibrated judge prompts, and scripts to reproduce every number. github.com/Offensive-AI-Lab/prism-eval →

Paper

Full method, ablations, judge calibration, and more qualitative examples. arXiv:2606.09563 →

PRISM checkpoint

The main Qwen3.5-9B model. Gemma-2-9B, Ministral-3-8B, and SFT-only checkpoints are also released. Hugging Face →

Training dataset

Prompts, responses, and reference instruction lists for both training stages. Hugging Face →

Baseline adapters

The exact LatentQA and Activation Oracles adapters for Qwen3.5-9B used in our comparisons. All models →

Scope

Limitations and responsible use

We evaluate 8–9B instruction-tuned models from three families. Larger models and long agent trajectories are future work.
Ground truth lists what the prompt instructed, not what the model chose to follow. The post-paper behavior analysis is a first step toward the instructions that actually drove the response.
PRISM is a monitor, not a defense. Recovering an injected instruction does not decide whether to block it, and a miss or a hallucination can mislead. Treat its output as an audit signal for human review.
Reading instructions from activations can expose system prompts or private task context. Activation traces should be handled as sensitive data, with consent.

Cite this work

@inproceedings{gressel2026prism,
  title     = {PRISM: Recovering Instruction Sets from Language Model Activations},
  author    = {Gressel, Gilad and Pankajakshan, Rahul and Diament, Julia and
               Hudis, Efim and Achuthan, Krishnashree and Mirsky, Yisroel},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.09563}
}