COMPASS Research Group
COoperative Machine intelligence for People-Aligned Safe Systems
I am joining the ELLIS Institute as a Principal Investigator where I will be co-affiliated with the Max-Planck Institute for Intelligent Systems in Tübingen as an independent research group leader. I will found the COMPASS research group, focused on developing safe, aligned, and steerable AI agents with emphasis on security, human aspects, and cooperative multi-agent systems. 📢 I will be hiring!
About Me
Hi! I am an AI security researcher at Microsoft. Previously, I completed my PhD at CISPA Helmholtz Center for Information Security, advised by Prof. Dr. Mario Fritz, and I obtained my MSc degree at Saarland University.
I am interested in the broad intersection of AI with security, safety, and sociopolitical aspects. This includes the following areas:
1) Understanding, probing, and evaluating the failure modes of AI models, their biases, emergent risks, and their misuse scenarios.
2) How to design mitigations, system defenses, white-box control methods, and reasoning enhancements to counter such risks.
3) Leveraging AI agents for good: scientific discovery and advancing our society.
Our previous work, in 2023, was the first to identify the indirect prompt injection vulnerability in LLM-integrated applications, and in 2020, to propose and call for watermarking generative AI.
I am interested in the broad intersection of AI with security, safety, and sociopolitical aspects. This includes the following areas:
1) Understanding, probing, and evaluating the failure modes of AI models, their biases, emergent risks, and their misuse scenarios.
2) How to design mitigations, system defenses, white-box control methods, and reasoning enhancements to counter such risks.
3) Leveraging AI agents for good: scientific discovery and advancing our society.
Our previous work, in 2023, was the first to identify the indirect prompt injection vulnerability in LLM-integrated applications, and in 2020, to propose and call for watermarking generative AI.
AI Security
AI Safety
AI & Society
AI Ethics
Prompt Injection
Multi-agent Safety
Cooperative AI
Human-AI Interaction
Join the COMPASS Research Group!
You can find my research statement here. I am open to broad topics on AI safety, interpretability, reasoning, evals, agentic risks and opportunities. If you are interested in these research directions, I would love to hear from you!
Open Positions:
• For a postdoc position, please send an email with a CV and research statement
• I will likely admit PhD students via the following channels:
• Max Planck & ETH Center for Learning Systems (CLS)
• International Max Planck Research School for Intelligent Systems (IMPRS-IS)
• ELLIS PhD Program
• Research internships and visits, please fill out this form
Please reach out for questions!
Open Positions:
• For a postdoc position, please send an email with a CV and research statement
• I will likely admit PhD students via the following channels:
• Max Planck & ETH Center for Learning Systems (CLS)
• International Max Planck Research School for Intelligent Systems (IMPRS-IS)
• ELLIS PhD Program
• Research internships and visits, please fill out this form
Please reach out for questions!
Highlights
December 2024
Successfully defended my PhD with grade: Summa cum laude!
Feb 2024
I have joined Microsoft as an AI Security Researcher!
December 2023
Our paper "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" received the best paper award at AISec'23 workshop!
Selected Publications
For the full list, please refer to my Google Scholar page.
Linear Control of Test Awareness Reveals Differential Compliance in Reasoning Models
Arxiv 2025
Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models
Arxiv 2025
A Theory of Decision Sampling in LLMs: Part Descriptive and Part Prescriptive
ACL 2025 Main conference 🏆 Oral & panel discussion
Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation
NeurIPS Datasets and Benchmarks 2024
Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition
NeurIPS Datasets and Benchmarks 2024 🏆 Spotlight
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
AISec Workshop at CCS 2023 🏆 Best Paper Award
Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systems
USENIX Security 2023
Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources
CVPR 2022
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding
S&P 2021
Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data
ICCV 2021 🏆 Oral
Media Coverage & Outreach
Major Media Interviews & Features
Vice, Wired, Zeit, MIT Technology Review
2023
Our work on "indirect prompt injection" has been featured as interviews with myself/authors in Vice, Wired, Zeit, MIT Technology Review, and CISPA communication channels.
Podcasts & Documentaries
Various Media Outlets
2022 - 2024
Y-Kollektiv Documentary (2023): "ChatGPT: What happens when the AI takes over?"
CyberWire Podcast (2023): "A dark side to LLMs"
CISPA tl;dr Podcast (2022): "Deepfakes and Fingerprinting"
Industry & Research Blogs
Microsoft Security Response Center, Montreal AI Ethics Institute
2023 - 2024
Microsoft Security Response Center (MSRC) blogs: "Announcing the Adaptive Prompt Injection Challenge (LLMail-Inject)" and "Announcing the winners of the Adaptive Prompt Injection Challenge (LLMail-Inject)"
Montreal AI Ethics Institute: Featured work on "LLM-deliberation"
Montreal AI Ethics Institute: Featured work on "LLM-deliberation"
Policy & Industry Impact
Government & Industry Organizations
2023 - Present
Our work on "indirect prompt injection" has been featured by policymakers and practitioners including the German Federal Office for Information Security, NIST, OWASP, MITRE, Microsoft's AI bug bar, and many others, introducing new terminologies for the entire research and tech fields.
Academic Service
Program Committee Member
Multiple Conferences
2023 - 2025
Program Committee: IEEE SaTML'24, AISec'23 and '24 Workshop, USENIX Security'25, AAAI'25, CCS'25
Reviewer: ICLR'25, ICML'24, ICLR'24, NeurIPS'23, ICML'23 Neural Conversational AI Workshop, ICCV'23, CVPR'23, ECCV'22, CVPR'22, IEEE TPAMI (2024, 2022, 2021), ICLR'21 Workshop on Synthetic Data Generation Quality, Privacy, Bias
Reviewer: ICLR'25, ICML'24, ICLR'24, NeurIPS'23, ICML'23 Neural Conversational AI Workshop, ICCV'23, CVPR'23, ECCV'22, CVPR'22, IEEE TPAMI (2024, 2022, 2021), ICLR'21 Workshop on Synthetic Data Generation Quality, Privacy, Bias
Competition Organization
IEEE SaTML Challenges
2024 - 2025
Lead Co-organizer: IEEE SaTML'25 challenge "LLMail-Inject: Adaptive Prompt Injection Attacks"
Co-organizer: IEEE SaTML'24 challenge "LLM CtF"
Co-organizer: IEEE SaTML'24 challenge "LLM CtF"
Grant Reviewing & Consulting
Various Organizations
2024 - Present
Grant Reviewing: Cooperative AI
Consulting: UK AI Safety Institute
Consulting: UK AI Safety Institute
Invited Talks & Panels
Firewalls to Secure Dynamic LLM Agentic Networks
Brave, Google DeepMind, Qualcomm
2025
Presenting recent research on securing LLM agent networks across multiple industry venues.
Panel: Women in AI Security Workshop
The Alan Turing Institute
2025
Participated as a panelist discussing trends, challenges, diversity, and inclusion in AI security research.
Panel: Implementation and Evaluation of a Research Paper
AI Saturdays Lagos
2024
Participated as a panelist discussing prescriptives and research experiences.
Towards Aligned, Interpretable, and Steerable Safe AI Agents
TU Graz, UMass Amherst Security and Privacy Seminar, CISPA, ELLIS Institute
2025
Series of talks on developing safer and more controllable AI agent systems.
On the Security of Real-World LLM-Integrated Applications
European Symposium on Security and Artificial Intelligence
2024
Invited talk on security vulnerabilities in production LLM systems.
On New Security and Safety Challenges Posed by LLMs
HIDA PhD Meet-up (Keynote), MLSec seminars
2024
Keynote presentation on emerging security challenges in large language models and evaluation methodologies.
Compromising LLMs: The Advent of AI Malware
Black Hat USA 2023
2023
Major industry conference presentation on LLM security vulnerabilities and attack vectors.
On Evaluating Language Models and Their Security Implications
Vector Institute, ETH ZĂĽrich
2023
Research talks on methodologies for evaluating LLM safety and security properties.
LLM-Deliberation: Evaluating LLMs with Interactive Multi-Agent Negotiation
SIGSEC talk
2023
Presentation on novel evaluation frameworks using multi-agent systems for LLM assessment.
Panel: Security of Generative AI and Generative AI in Security
DIMVA Conference
2023
Invited panelist discussing the dual aspects of generative AI security challenges and applications.
Multi-modal Fact-checking: Out-of-Context Images
UCL Information Security seminars, Max Planck Institute
2022
Research presentations on detecting and countering misinformation through multi-modal approaches.
Experience
AI Security Researcher
Microsoft Security Response Center (MSRC), Microsoft Research Cambridge, UK
2024 - Present
Conducting AI security and safety research. Assessing AI security vulnerabilities reported through Microsoft's AI Bug Bounty program.
PhD in Computer Science
CISPA Helmholtz Center for Information Security, Germany
2019 - 2024
Advisor: Prof. Dr. Mario Fritz. External reviewers: Prof. Dr. Battista Biggio and Prof. Dr. Florian Tramèr. Grade: Summa cum laude.
Research Assistant
Max Planck Institute for Informatics, Germany
2017 - 2019
Advised by Prof. Dr. Andreas Bulling. Research on brain-computer inteface, ML, and human-computer interaction.
Quality Assurance Engineer
Mentor Graphics (Currently, Siemens EDA), Egypt
2013 - 2017
Software quality assurance and testing for electronic design automation tools.