How Do You Extend an Application Security Program to Your AI/ML Applications

Top 10 attack vectors for LLMs and comparison with traditional application threat vectors:

1. Prompt Injection – Threat actor’s ability to modify the prompt given to LLM from a web page or interface that is supposed to have built in prompts to perform actions. This modification of prompt could result in exfiltration of private data, deletion of proprietary data or provide incorrect results to queries like risk analysis. This can be prevented using traditional defense mechanisms.
2. Insecure Output Handling – Threat actors can attack systems that use automation based on LLM output to generate output that is automated using exec or using plugins to cause exfiltration of private data, deletion of proprietary data or create backdoor by connecting to third party servers. This can be prevented by traditional defense mechanisms using input validation.
3. Training Data Poisoning – Similar to Data poisoning of ML, providing invalid data to model, having access to model training with unsuspecting users providing incorrect data or deliberate malicious data to train the model. This is a form of supply chain security and chain of custody addressed by traditional defense mechanisms.
4. Model Denial of Service – Threat actor can send poisoned queries using recursive resource usage, queries with query stream that exceeds context window, repetitive long inputs, large volume of variable length inputs causing high resource consumption with filled up context windows for each of the queries. This is specific to LLM models that needs to be handled by the Models.
5. Supply Chain Vulnerabilities – Using models no longer supported, poisoned crowd sourced data, vulnerable pre-trained models cause the model to perform with incorrect or skewed results that can be taken advantage of by threat actors in production. Traditional chain of custody and validation of source and provenance can be used to address this type of vulnerability.
6. Sensitive Information Disclosure – Incomplete or improper filtering in LLM responses, overfitting or Skewed data used for training can cause unintended disclosure of PII by well crafted queries by user with knowledge of metadata of training set or other users PII for a legitimate query for one user. This is specific to LLM models that need to be handled by monitoring responses for PII and private data.
7. Insecure Plugin Design – Plugins are driven by LLM model and input to plugin are driven by LLM. This could result in prompt resulting in input generation for plugin that can lead to data exfiltration, data deletion, or providing access to the system to malicious users. This is similar to insecure output handling, handled by traditional prevention techniques like strict parameterized inputs with validation by plugin, vulnerability checks on plugins, ASVS practices, SAST, DAST checks, requiring manual authorization for any sensitive plugins.
8. Excessive Agency – Ability to interface with other systems and take actions in response to prompts can lead to unintended consequences due to hallucination, compromised plugin, or poorly performing model. This is similar to insecure plugin design which can be prevented by ensuring boundaries of authorizations allowed for automation based on the actor and ensuring validity of commands from LLM.
9. Overreliance – LLMs can be authoritative even with hallucinations, bugs in generated code can be hard to detect causing problems downstream when LLM output is used without proper validation. This is specific to LLM models that needs to be handled by monitoring responses.
10. Model Theft – Unauthorized user gaining access to enterprise network, insider threat, attack vector for model extraction with carefully crafted queries issued in large numbers. This is prevented with combination of traditional defense mechanisms and LLM specific techniques of robust model for adversarial queries

As can be seen from above, only Denial of Service, Sensitive Information Disclosure, and Model Theft are the ones that require new techniques for prevention. MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS) is based on real world attack observations that can be mapped to the techniques exploiting vulnerabilities detailed by OWASP. Garak is an interesting project we are working with to provide comprehensive security posture for AI applications. Stay tuned for updates on AI security and other exciting enhancements at https://www.opsmx.com/secure-software-delivery/ .