SecureBERT 2.0: Cisco’s next-gen AI model powering cybersecurity applications

[ad_1] In the present day, we're excited to share that the SecureBERT 2.0 mannequin is accessible on HuggingFace and GitHub with an accompanying analysis paper. This launch marks a major milestone, constructing on the already broadly adopted SecureBERT mannequin to unlock much more superior cybersecurity purposes. Simply see this unparalleled efficiency throughout real-world duties: <img fetchpriority="high" decoding="async" class="aligncenter wp-image-480787 " src="https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_-1024x646.jpg" alt="" width="959" height="605" srcset="https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_-300x189.jpg 300w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_-1024x646.jpg 1024w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_-768x485.jpg 768w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_-1536x969.jpg 1536w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V3.Image1_.jpg 1845w" sizes="(max-width: 959px) 100vw, 959px"/>

In 2022, the primary SecureBERT mannequin was launched by Ehsan and a group of researchers from Carnegie Mellon College and UNC Charlotte as a pioneering language mannequin designed particularly for the cybersecurity area. It bridged the hole between general-purpose NLP fashions like BERT and the specialised wants of cybersecurity professionals—enabling AI techniques to grasp the technical language of threats, vulnerabilities, and exploits.

By December 2023, SecureBERT ranked among the many prime 100 most downloaded fashions on HuggingFace out of the roughly 500,000 fashions then out there on the repository. It gained important recognition throughout the cybersecurity neighborhood and stays in lively use by main organizations, together with the MITRE Menace Report ATT&CK Mapper (TRAM) and CyberPeace Institute.

On this weblog, we’ll mirror on the influence of the unique SecureBERT mannequin, element the numerous developments made in SecureBERT 2.0, and discover some real-world purposes of this highly effective new mannequin.

The influence of the unique SecureBERT mannequin Safety analysts at enterprises and companies dedicate an amazing period of time to parsing by numerous safety alerts to establish, analyze, categorize, and report on potential threats. It’s an vital course of that, when completed completely manually, is time-consuming, costly, and susceptible to human error.

SecureBERT gave researchers and analysts a software that might course of safety studies, malware analyses, and vulnerability write-ups with contextual accuracy by no means earlier than potential. Even at this time, it serves as a useful software for cybersecurity specialists at a few of the world’s prime companies, universities, and labs.

Nevertheless, SecureBERT had a number of limitations. It struggled to deal with long-context inputs equivalent to detailed menace intelligence studies and mixed-format information combining textual content and code. Since SecureBERT was skilled on RoBERTa-base, a traditional BERT encoder with a 512-token context restrict and no FlashAttention, it was slower and extra memory-intensive throughout coaching and inference. In distinction, SecureBERT 2.0, constructed on ModernBERT, advantages from an optimized structure with prolonged context, quicker throughput, decrease latency, and decreased reminiscence utilization.

With SecureBERT 2.0, we addressed these gaps in coaching information and superior the structure to ship a mannequin that was much more succesful and contextually conscious than ever. Whereas the unique SecureBERT was a standalone base mannequin, the two.0 model contains a number of fine-tuned variants specializing in numerous real-world cybersecurity purposes. <img loading="lazy" decoding="async" class="aligncenter wp-image-480779 size-full" src="https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V2Image2.png" alt="" width="936" height="360" srcset="https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V2Image2-300x115.png 300w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V2Image2-768x295.png 768w, https://blogs.cisco.com/gcs/ciscoblogs/1/2025/10/V2Image2.png 936w" sizes="auto, (max-width: 936px) 100vw, 936px"/>

Introducing SecureBERT 2.0

SecureBERT 2.0 brings larger contextual relevance and area experience for cybersecurity, understanding code sources and programming logic in a method its predecessor merely couldn't. The important thing here's a coaching dataset that's bigger, extra numerous, and strategically curated to assist the mannequin higher seize delicate safety nuances and ship extra correct, dependable, and context-aware menace evaluation.

Whereas giant autoregressive fashions equivalent to GPT-5 excel at producing language, encoder-based fashions like SecureBERT 2.0 are designed to grasp, symbolize, and retrieve info with precision—a elementary want in cybersecurity. Generative fashions predict the subsequent token; encoder fashions rework total inputs into dense, semantically wealthy embeddings that seize relationships, context, and that means with out fabricating content material.

This distinction makes SecureBERT 2.0 supreme for high-precision, security-critical purposes the place factual accuracy, explainability, and velocity are paramount. Constructed on the ModernBERT structure, it makes use of hierarchical long-context encoding and multi-modal text-and-code understanding to investigate advanced menace information and supply code effectively.

Let’s check out how SecureBERT 2.0 helps safety analysts in real-world purposes.

Actual world purposes of SecureBERT 2.0 Think about you're a SOC analyst tasked with investigating a suspected provide chain compromise. Historically, this may contain correlating open-source intelligence, inside alerts, and vulnerability studies in a course of which may take a number of weeks of handbook information evaluation and cross-referencing.

With SecureBERT 2.0, you'll be able to merely embed all related property—studies, codes, CVE information, and menace intelligence, for instance—within the system. The mannequin instantly surfaces connections between obscure indicators and beforehand unseen infrastructure patterns.

This is only one potential state of affairs of many; SecureBERT 2.0 can help and streamline a wealth of potential safety purposes:

Menace Intelligence Correlation: Linking indicators of compromise throughout a number of sources to uncover marketing campaign patterns and adversary ways Incident Triage & Alert Prioritization: Embedding alerts and studies to detect duplicates, associated incidents, or recognized CVEs—lowering noise and analyst workload Safe Code & Vulnerability Detection: Figuring out dangerous patterns, insecure dependencies, and potential zero-day vulnerabilities in supply code Semantic Search & RAG for Safety Ops: Offering context-aware retrieval throughout inside information bases, menace feeds, and documentation for quicker analyst response Coverage and Compliance Search: Enabling correct semantic lookup throughout giant regulatory and governance corpora

Not like generative LLMs that create textual content, SecureBERT 2.0 interprets and constructions info to ship quicker inference, decrease compute prices, and reduce the chance of hallucination. This makes it a trusted basis mannequin for enterprise, protection, and analysis environments the place precision and information integrity matter most.

Underneath the hood of SecureBERT 2.0 There are three elements to the SecureBERT 2.0 structure that make this mannequin such a major development: its ModernBERT basis, its information enlargement, and smarter strategy to pretraining.

SecureBERT 2.0 is powered by ModernBERT, a next-generation transformer designed for long-document processing. Prolonged consideration mechanisms and hierarchical encoding enable the mannequin to seize each fine-grained syntax and high-level construction—crucial for analyzing lengthy, multi-section safety studies.

The mannequin is skilled on 13 instances extra information than the unique SecureBERT with a brand new corpus that features curated safety articles and technical blogs, filtered cybersecurity information, code vulnerability repositories, and incident narratives. In complete, this dataset covers 13 billion textual content tokens and 53 million code tokens.

Lastly, a microannealing pretraining curriculum step by step transitions from curated to real-world information, balancing high quality and variety. Focused masking teaches the mannequin to foretell essential safety actions and entities like “bypass,” “encrypt,” or “CVE,” strengthening area illustration.

The efficiency of SecureBERT 2.0 is a marked enchancment over its predecessor and different evaluated fashions throughout benchmarks; the main points may be present in full analysis paper.

Trying forward: AI for safety at Cisco SecureBERT 2.0 demonstrates what’s potential when structure and information are purpose-built for cybersecurity. It joins different fashions, just like the generative Basis-Sec-8B from Cisco’s Basis AI group, as a part of Cisco’s continued dedication to making use of AI responsibly inside the area of cybersecurity.

We're excited to share this mannequin with the world, to see a few of the progressive methods it is going to be embraced by the safety neighborhood, and to proceed exploring potential usages for taxonomy creation, information graph technology, and different cutting-edge purposes.

You will get began with the SecureBERT 2.0 mannequin on HuggingFace and GitHub at this time, and dig into our analysis paper for extra element and efficiency benchmarking.