Responsible Disclosure: LLM Chatbot Data Breach (Medical Sector)
Critical data privacy vulnerability found in an AI-powered customer support chatbot
The Discovery
While evaluating an AI-powered customer support chatbot at a regulated medical e-commerce company, I discovered a critical data privacy vulnerability: the underlying LLM had unrestricted access to the company's customer database, exposing sensitive personal and medical-adjacent order data through crafted prompts.
The Response
- Responsible Disclosure: Reported the vulnerability directly to the company through proper channels
- Escalation: When the issue was not adequately remediated, escalated the matter to the Bavarian Data Protection Authority (Bayerisches Landesamt für Datenschutzaufsicht)
- Regulatory Context: The exposure implicated GDPR obligations around personal data processing and data minimization
Why It Mattered
This case is a concrete example of a class of risk that becomes common as companies bolt LLMs onto existing systems without proper access controls: giving a language model direct, unscoped database access effectively removes the authorization boundary that would normally protect customer data. It reinforced my focus on building AI integrations with the same rigor — scoped credentials, least privilege, and auditability — that any production backend system requires.
Details that could identify the company or its customers are withheld out of respect for the disclosure process and affected users.