Enterprise AI operates in conversations — multilingual, multi-turn, and context-dependent. A guardrail that performs only on English single-turn prompts cannot protect what enterprises are actually building. In a recent independent benchmark by ML6 on 80,000 Dutch-language prompts, Cisco AI Defense led the cohort of providers with the highest F1 score.
01 A Note on Semantics
AI safety labels only work when everyone agrees on what they mean. Language is inherently semantically diffuse; intent, context, and linguistic nuance shape interpretation, and, consequently, the true label.
Cisco addresses this through constitutional definitions: precise, per-technique operational specifications that serve as the single source of truth for classification, model training, and customer-facing explanations. This approach reduces inter-model disagreement by up to 57× compared to paragraph-level definitions. Because the spec is machine-enforced, it applies with equal precision in French, Japanese, or Arabic.
The taxonomy distinguishes intent from content: a conversation can carry harmful intent without harmful output (a probed-and-refused attack), or harmful content without adversarial intent (model misbehavior on a benign request). That distinction is essential in production, where the same surface language can mean very different things depending on conversational context.
02 Security Has Moved Into the Conversation
In AI systems, ordinary language is the control plane. A malicious instruction can look identical to a user request; a benign phrase can look suspicious out of context. Attacks rarely arrive in a single prompt — real adversaries iterate, reframe refusals, and escalate gradually across turns. Cisco research across 15 frontier models found that every model tested shows meaningful multi-turn vulnerability, with attack success rates that bear no consistent relationship to single-turn benchmarks.
This means the security perimeter must move outside the model. Cisco AI Defense validates inputs and outputs in production, classifying the intent and active direction of each conversation — not just the surface content of each message. Guardrails are tailored to the specific vulnerabilities of each model and application, and applied at the point where AI behavior is actually shaped: the live exchange between user, model, data, and tools.
03 The Multilingual Reality Check
The ML6 benchmark put multilingual performance into sharp relief. Testing on 80,000 Dutch-language prompts — including prompt injection, policy bypass, ambiguous instructions, and realistic enterprise interactions — Cisco AI Defense achieved the highest F1 score in the cohort: 0.845.
To highlight Cisco’s multilingual capabilities – in this post we sample and share results on an augmented version of LMSYS Chat-1M and WildChat — two widely used open-source conversational datasets representing realistic enterprise chat traffic. The data was augmented with conversations from eight additional languages with a similar distribution as LMSYS and WildChat. The ground truth labels for this dataset were generated using Cisco AI’s security and safety taxonomy. The ML6 benchmark used a separate Dutch-specific dataset assembled independently; the two evaluations are complementary, not directly comparable.
Cisco AI Defense was evaluated on a multilingual, augmented conversational dataset derived primarily from the LMSYS Chat-1M and WildChat corpora. The evaluation set consists predominantly of benign, general-purpose conversations, along with an adversarial subset representing approximately 14% of the labeled examples. The dataset had approximately 5,800-5,900 conversations per language. FPR is measured on this specific adversarial evaluation mix; on a real-world distribution it would be much lower. Results are presented with English first, Dutch second, followed by the remaining languages ordered by F1 score.
F1 ranges from 0.796 (Arabic) to 0.860 (Portuguese) — a tight spread across nine typologically diverse languages, from Latin-script European languages to Arabic and Japanese. That consistency reflects the constitutional taxonomy at work: when a definition is precise and machine-enforced, the signal transfers across languages reliably. The same operational specification governs whether a prompt injection is written in French, Japanese, or Arabic.
Each curve is the achievable recall-vs-FPR frontier for Cisco AI Defense per language, across all threshold combinations. Higher and further left is stronger. Legend shows AUC per language.
04 Protection Without Friction
A guardrail with high recall but poor precision is not a security product — it is an availability problem. In the ML6 benchmark, another guardrail solution under test reached 0.327 recall but only 0.453 F1, as false alarms collapsed precision to 0.737. Cisco achieved 0.843 recall and 0.847 precision simultaneously — the highest F1 in the cohort. That balance requires a threat model precise enough to distinguish an adversarial instruction from a legitimate but emphatic user request.
Each marker is one language, positioned by its recall and false-positive rate. F1 scores shown in the legend. The shaded region marks the ideal operating zone — high recall with low false positives.
The FPR figures in the table — 2.3–5.8% across languages — are measured on an evaluation mix that is roughly 14% adversarial. On a predominantly benign production population, the effective FPR would be much lower. More meaningful than the absolute values is their cross-language stability: the narrow range across nine languages indicates the constitutional taxonomy produces consistent signal rather than silently trading precision for recall as users switch languages. Operating thresholds are configurable without retraining, allowing organizations to tune the precision-recall tradeoff to their specific risk profile.
05 Real-Time Protection
A guardrail that cannot keep pace with production traffic will not stay in the critical path. Enterprise AI applications have response-time SLAs; users notice latency; and in agentic pipelines, per-hop overhead compounds. Security that adds seconds per request gets disabled or bypassed.
Cisco AI Defense is built to sit in the live interaction without becoming the bottleneck. At p90 = 40 ms and p99 = 250 ms per request, the security check adds overhead that is imperceptible to end users and compatible with real-time conversational SLAs across chatbots, copilots, and agentic pipelines.
Runtime protection is not a point-in-time test. AI applications evolve continuously: models are updated, RAG sources shift, agents acquire new tools, and attack techniques adapt. Pre-deployment evaluation establishes a baseline; runtime guardrails maintain it under live production conditions, for every user, in every language, across every model and application the enterprise runs — regardless of vendor or deployment framework.
What Enterprises Should Take Away
Enterprise AI is multilingual and multi-turn by design. Security must match that reality. Cisco AI Defense addresses this from first principles:
- A constitutional taxonomy that produces consistent, explainable signal across languages and attack types.
- Conversational-native detection that classifies the intent and active direction of an exchange, not just its surface content.
- Multilingual by design — consistent detection across languages and scripts, because the taxonomy that drives the guardrail is language-agnostic.
- A precision-recall balance that protects the enterprise without punishing legitimate users.
- Runtime performance designed for production — p90 latency of 40 ms per request, compatible with real-time conversational SLAs.
For organizations scaling AI, the goal is not simply to block more. It is to preserve trust — protecting users, data, models, and business processes while keeping the conversation open for everyone who deserves to have it.
Related reading: Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation · Proprietary Problems: No Frontier Model Is Multi-Turn Immune · ML6 Enterprise Guardrail Benchmark