Date Published: July 31, 2025
Comments Due:
Email Questions to:
Author(s)
Harold Booth (NIST), Nakia Grayson (NIST), Michael Dimond (MITRE), Michael Ekstrom (MITRE), John Kent (MITRE), Daniel Lee (MITRE), Jasmine O'Hannon (MITRE), Shahzad Rajput (MITRE), Kenneth Sandlin (MITRE), Theresa Suloway (MITRE), Kevin Wacome (MITRE)
Announcement
The NIST National Cybersecurity Center of Excellence (NCCoE) has re-issued the initial public draft of NIST IR 8579. Originally published in June, the document was revised to improve the document’s demonstration of the enhanced abilities of an RAG-based LLM tool over a generic LLM.
The public comment period for the publication has been extended and will close at 11:59 pm EDT on September 11, 2025.
The NCCoE identified a potential application for a chatbot to support its mission and developed a secure, internal-use chatbot to assist NCCoE staff with discovering and summarizing cybersecurity guidelines tailored to specific audiences or use cases.
The chatbot was built using retrieval-augmented generation (RAG)-based LLM technology. This approach combines techniques from information retrieval and natural language generation, enabling the chatbot to provide more focused, contextually relevant responses by leveraging a repository of cybersecurity knowledge, including previous NCCoE publications. Compared to search engines, LLM-based chatbots provide more contextually relevant and precise responses by understanding the nuances of natural language queries.
This report provides a point in time examination of the NCCoE Chatbot, outlining the NCCoE’s approach to developing the tool, as well as the NCCoE’s response to specific security challenges. In addition, this report provides an overview of the chatbot and its supporting technologies so that other organizations might consider the benefits of their use.
We encourage you to review this draft and provide comments by September 11, 2025. If you have any questions, please email the team at [email protected].
Chatbots are emerging as alternative interfaces for structured information retrieval and internal knowledge access. Chatbots can utilize the capabilities of large language models (LLMs) to help interpret user-provided input and provide responses to a variety of requests. This paper describes the development of an LLM chatbot by the National Cybersecurity Center of Excellence (NCCoE) at NIST to enable internal search across its published cybersecurity guidance. The paper provides a point-in-time examination of the tool’s development process, including the architecture, the system configuration, and the NCCoE’s approach to addressing cybersecurity challenges throughout the design and deployment lifecycle. Specific attention is given to threats such as prompt injection, hallucinations, data exposure, and unauthorized access. The paper also discusses the mitigations applied, including local deployment, access controls, and validation filters. This paper is not intended to serve as implementation guidance. Instead, it documents technical decisions, observed limitations, and risk-informed safeguards that shaped the prototype. It provides an overview of the chatbot and its supporting technologies so that other organizations might consider the benefits of their use.
Chatbots are emerging as alternative interfaces for structured information retrieval and internal knowledge access. Chatbots can utilize the capabilities of large language models (LLMs) to help interpret user-provided input and provide responses to a variety of requests. This paper describes the...
See full abstract
Chatbots are emerging as alternative interfaces for structured information retrieval and internal knowledge access. Chatbots can utilize the capabilities of large language models (LLMs) to help interpret user-provided input and provide responses to a variety of requests. This paper describes the development of an LLM chatbot by the National Cybersecurity Center of Excellence (NCCoE) at NIST to enable internal search across its published cybersecurity guidance. The paper provides a point-in-time examination of the tool’s development process, including the architecture, the system configuration, and the NCCoE’s approach to addressing cybersecurity challenges throughout the design and deployment lifecycle. Specific attention is given to threats such as prompt injection, hallucinations, data exposure, and unauthorized access. The paper also discusses the mitigations applied, including local deployment, access controls, and validation filters. This paper is not intended to serve as implementation guidance. Instead, it documents technical decisions, observed limitations, and risk-informed safeguards that shaped the prototype. It provides an overview of the chatbot and its supporting technologies so that other organizations might consider the benefits of their use.
Hide full abstract
Keywords
artificial intelligence (AI); chatbot; cybersecurity; large language model (LLM); machine learning; retrieval-augmented generation (RAG)
Control Families
None selected
