Building a Gen AI Security Framework, Part 1: The Foundation
Part 1 of 4 in a series on building a practical, regulator-ready risk framework for generative AI.
Executive Summary
- Generative AI is now built into customer service tools, coding assistants, and internal knowledge systems across most companies. Few of those companies have a real plan for managing the risk it introduces.
- This part lays the foundation for that plan. It uses the NIST AI Risk Management Framework (AI RMF) as the structural base, brings in the Generative AI Profile (NIST AI 600-1) to name the risks specific to gen AI, applies NIST SP 800-30 to rate how serious each risk is, and uses OWASP and MITRE ATLAS to ground those ratings in how these systems are actually attacked.
- The AI RMF gives you four functions to organize risk work: Govern, Map, Measure, Manage.
- NIST AI 600-1 names twelve risk categories specific to generative AI.
- Naming a risk is not the same as rating it. SP 800-30 gives a repeatable method for that, and OWASP and MITRE ATLAS supply the real-world attack detail that makes the rating credible.
- Later parts in this series build on this foundation: Part 2 covers how to scale this analysis across many use cases, Part 3 covers mapping risks to controls and evidence, and Part 4 walks one use case through the entire framework end to end.
Why This Needs Its Own Framework
Most companies already have a risk management process for IT systems. Generative AI does not fit cleanly into it.
A traditional application has a fixed set of inputs and outputs. A gen AI system can be asked almost anything, in almost any phrasing, and its response is not fully predictable even to the people who built it. That changes what a risk assessment needs to look for. The threat is not only whether someone can break into the system. It also includes whether someone can talk the system into doing something it should not do.
This is why NIST built a separate framework for AI, and a separate profile specifically for generative AI. Both are free, publicly available, and already widely referenced by regulators and auditors. There is no need to build a new framework from scratch. There is a real need to put the existing pieces together correctly.
1. Start With the Foundation: NIST AI RMF
The NIST AI Risk Management Framework, formally NIST AI 100-1, was released in January 2023. It organizes AI risk management into four functions.
| Function | What it does |
|---|---|
| Govern | Sets ownership, policy, and accountability for AI risk across the organization. |
| Map | Builds context: what the system does, who uses it, what data it touches, what could go wrong. |
| Measure | Tests and monitors the system against defined metrics, not just at launch but on an ongoing basis. |
| Manage | Decides which risks to accept, reduce, or escalate, and runs incident response when something goes wrong. |
These four functions are the skeleton of the whole program. They tell you what to do. They do not tell you what to look for in a generative AI system specifically. That is the job of the Generative AI Profile.
2. Name the Risks: NIST AI 600-1
In July 2024, NIST published the Generative AI Profile, NIST AI 600-1. It maps the four RMF functions onto twelve risk categories that show up specifically in generative AI systems.
| # | Risk category | What it covers |
|---|---|---|
| 1 | CBRN information | Easier access to knowledge that could support chemical, biological, radiological, or nuclear harm. |
| 2 | Confabulation | Confident but false output, commonly called hallucination. |
| 3 | Dangerous or hateful content | Generation of violent, radicalizing, or discriminatory material. |
| 4 | Data privacy | Leakage of personal, health, or biometric data through training data or outputs. |
| 5 | Environmental impact | Resource cost of training and running large models. |
| 6 | Harmful bias or homogenization | Unequal performance across groups, or output that flattens diversity of thought. |
| 7 | Human-AI configuration | Over-reliance on the system, or people treating it as more capable than it is. |
| 8 | Information integrity | Content that blurs fact and opinion at scale. |
| 9 | Information security | New attack surface created by the model itself, including the system prompt and its integrations. |
| 10 | Intellectual property | Output that reproduces copyrighted or licensed material without rights. |
| 11 | Obscene or abusive content | Generation of illegal or exploitative imagery. |
| 12 | Value chain and component integration | Risk introduced by third-party models, plugins, and data sources with limited visibility. |
For a security-focused program, categories 4, 8, 9, and 12 tend to matter most day to day. A complete Map exercise still walks through all twelve, since the categories that feel less urgent, such as bias or environmental cost, still create legal and reputational exposure.
3. Rate What You Find: NIST SP 800-30
Naming a risk is not the same as knowing how serious it is. NIST SP 800-30, Guide for Conducting Risk Assessments, gives a repeatable method for rating risk using two factors: likelihood and impact.
Likelihood asks how probable it is that a threat will occur and lead to harm. For a gen AI system, this means asking how easy it is for someone to manipulate the model through its inputs, and how exposed the system is to outside users.
Impact asks what happens if it does. A leaked internal memo is bad. A leaked customer health record is worse, and may trigger legal obligations.
SP 800-30 supports three ways to combine these factors:
- Qualitative: rating each factor on a five-point scale, very low to very high, and combining them through a lookup matrix.
- Semi-quantitative: scoring each factor numerically, often one to five, multiplying the scores, then mapping the result back to a qualitative band.
- Quantitative: assigning real dollar values to likelihood and impact, producing an actual expected loss figure.
Most organizations start qualitative, since it is fast to apply consistently across a large inventory of AI use cases. Quantitative scoring gets reserved for the handful of highest-priority risks, where a dollar figure is needed to justify a budget decision or explain exposure to leadership. Part 4 of this series shows a full worked calculation using all three methods on the same example.
4. Ground the Rating in Real Attack Patterns
A likelihood rating is only useful if it reflects how these systems actually get attacked. Two references do the heavy lifting here.
The OWASP Top 10 for LLM Applications lists the most common ways generative AI applications get exploited in practice.
| ID | Risk |
|---|---|
| LLM01 | Prompt Injection |
| LLM02 | Sensitive Information Disclosure |
| LLM03 | Supply Chain |
| LLM04 | Data and Model Poisoning |
| LLM05 | Improper Output Handling |
| LLM06 | Excessive Agency |
| LLM07 | System Prompt Leakage |
| LLM08 | Vector and Embedding Weaknesses |
| LLM09 | Misinformation |
| LLM10 | Unbounded Consumption |
Each of these maps onto the Measure and Manage functions of the RMF. If your Map step identified information security as a risk category, this list tells you specifically what to test for.
MITRE ATLAS, the Adversarial Threat Landscape for AI Systems, adds another layer. Built on the same model as MITRE ATT&CK, it catalogs real-world adversary tactics and techniques against AI systems, drawn from documented incidents rather than theory. Where OWASP tells you what can go wrong in an LLM application, ATLAS tells you how attackers actually go about it, step by step. That detail is what lets you justify a likelihood rating with evidence instead of a guess.
5. Put It Together: A Working Process
A practical framework, built from these pieces, follows five steps.
- Govern. Assign an owner for gen AI risk and set a policy for what needs review before deployment.
- Map. Inventory every gen AI use case and score it against the twelve risk categories in AI 600-1.
- Test. Check each use case against the OWASP Top 10 and reference MITRE ATLAS to understand realistic attack paths.
- Rate. Apply the SP 800-30 method to score likelihood and impact, and rank use cases by overall risk.
- Manage. Fix or accept the highest-priority risks first, document the decision, and set a review cycle.
Gen AI risk does not stay still. New model versions and new integrations change the picture, so this is a repeating cycle, not a one-time project.
Conclusion
NIST AI RMF, the Generative AI Profile, SP 800-30, OWASP, and MITRE ATLAS give you something rare in this space: a framework that is grounded in recognized risk management standards and current with how these systems are actually attacked. Neither piece works well alone. Together, they give you a defensible, repeatable way to answer the question every board and regulator is now asking, which is how you know your gen AI systems are safe to use.
This is only the foundation. Naming and rating risk is the starting point, not the finish line. Part 2 of this series covers how to scale this analysis across a growing number of gen AI use cases without redoing the full exercise every time. Part 3 covers mapping these risks to actual controls and the evidence that proves those controls work. Part 4 carries one use case through the entire framework from start to finish.
References
- NIST. AI Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. https://www.nist.gov/itl/ai-risk-management-framework
- NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- NIST. Guide for Conducting Risk Assessments, SP 800-30 Revision 1, September 2012. https://csrc.nist.gov/pubs/sp/800/30/r1/final
- OWASP Gen AI Security Project. OWASP Top 10 for LLM Applications, 2025. https://genai.owasp.org/llm-top-10/
- MITRE. ATLAS, Adversarial Threat Landscape for Artificial-Intelligence Systems. https://atlas.mitre.org/
- Weidinger, L., et al. “Taxonomy of Risks Posed by Language Models.” Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT 22), pages 214 to 229. https://dl.acm.org/doi/10.1145/3531146.3533088
- “Prompt Injection Attacks in Large Language Models and AI Agent Systems: A Comprehensive Review of Vulnerabilities, Attack Vectors, and Defense Mechanisms.” Information, MDPI. https://www.mdpi.com/2078-2489/17/1/54