AI Research - Security and Resilience

The trustworthiness of AI technologies depends in part on how secure they are. The NIST AI Risk Management Framework (AI RMF) identifies “Secure and Resilient” as one of the primary characteristics of AI trustworthiness. NIST conducts research and develops guidelines that improve the security and resilience of AI applications and mitigate or manage their  vulnerabilities. 

Some cybersecurity risks related to AI systems are common (or identical) to cybersecurity risks across  software development and deployment. Overlapping risks include security concerns related to the confidentiality, integrity, and availability of the system and its training and output data – along with the general security of the underlying software and hardware for AI systems. NIST develops  a wide array of cybersecurity standards, guidelines, best practices, and other resources which complement its portfolio of AI activities.

The security and resilience of AI technologies is an area of active research, and challenges and potential solutions are changing very rapidly. For example, existing frameworks and guidance are unable to comprehensively address security concerns related to evasion, model extraction, membership inference, availability, or other machine learning attacks. They also do not account for the complex attack surface of AI systems or other security abuses enabled by AI systems.

AI technologies also have the potential to transform cybersecurity. They offer the prospect of giving defenders new tools that can address security vulnerabilities and even as they can enhance the capabilities of those seeking to target organizations and individuals through information technology (IT) and operational technology (OT) attacks.

Examples of AI security efforts underway by NIST follow:

Platform development and demonstration capability

NIST researchers are developing a platform, Dioptra, that is intended to be a shared resource to further AI research across NIST and the community. The platform serves as a testbed to research and develop metrics and best practices to assess vulnerabilities of AI models and the effectiveness of defenses against AI.  

Secure Software Design 

AI systems are built and operate on software. Security concerns of any data or information system apply to AI systems. In addition to the security concerns of traditional software, it is important to govern, map, measure, and manage AI-specific risks.  NIST worked with interagency collaborators – and sought feedback from the broader community through open and transparent processes – to develop a companion to the NIST Secure Software Development Framework (SSDF). NIST held a virtual workshop on Secure Development Practices for AI Models and released a draft Generative AI companion resource that incorporates secure development practices for generative AI and dual-use foundation models. That document was finalized after receiving public comments. 

Taxonomy and Terminology: Attacks and Mitigations

In January 2024, NIST published a report on  Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2023).  The report develops a taxonomy of concepts and defines terminology in the field of adversarial machine learning.