Anthropic Project Fetch 第二阶段：Claude Opus 4.7 自主操控机器狗速度超人类团队约 20 倍
Anthropic 公布 Project Fetch 第二阶段结果：Claude Opus 4.7 在无人类协助下，完成上一年人类团队做过的全部任务，速度比最快人类团队快约 20 倍。
AI安全动态
Anthropic推荐指数4040
网信办 · 网络安全推荐指数1212 市场监管总局会同中央网信办、国家铁路局约谈7家涉火车票销售第三方平台
6月11日，市场监管总局会同中央网信办、国家铁路局依法约谈携程、同程、去哪儿、飞猪、美团、智行火车票、高铁管家等7家第三方平台，针对其不当宣传“候补帮抢”及付费选座业务、不当诱导“买长乘短”或“买短乘长”、不当收集和使用用户个人信息等不规范经营问题。
Anthropic精选推荐指数8888 Anthropic 红队实测大模型加速 N-day 漏洞利用开发
Anthropic 前沿红队评估大模型从公开补丁自动开发 N-day 利用的能力，最强模型 Mythos Preview 在 18 个 Firefox 补丁上自主构建 8 个可执行代码利用，在 21 个 Windows 内核补丁上产出 8 条从低权限到 SYSTEM 的完整提权链。
推荐理由：Anthropic 红队用 Firefox 与 Windows 内核补丁实测前沿模型自动生成 N-day 利用，给出成功率、耗时与 API 成本，可据此重估补丁窗口。
Hugging Face Blog推荐指数8181 NVIDIA 发布 Nemotron 3.5 Content Safety 多模态内容安全模型
NVIDIA 发布 Nemotron 3.5 Content Safety，基于 Gemma 3 4B IT 微调，将多模态输入、12 语言覆盖、自定义企业策略执行与可审计推理统一到单次推理调用中。
Anthropic精选推荐指数8080 Anthropic 发布 LLM ATT&CK Navigator，将 832 个账号的 AI 网络攻击映射到 MITRE ATT&CK
Anthropic 前沿红队发布 LLM ATT&CK Navigator，把 2025 年 3 月至 2026 年 3 月间因违反使用政策被封禁的 832 个账号的 13,873 次恶意行为映射到 MITRE ATT&CK V18，覆盖全部 14 个战术和 482 个子技术。
推荐理由：Anthropic 把 832 个被封账号的 AI 滥用行为映射到 MITRE ATT&CK，并给出 ARiES 风险评分方法，可看攻击者如何从造工具转向自主编排。
Anthropic精选推荐指数8282 Anthropic 用 ExploitBench、ExploitGym 与 SCONE-bench 测量大模型开发漏洞利用的能力
Anthropic 前沿红队发布研究，用 ExploitBench、ExploitGym 和更新版 SCONE-bench 三个基准测量大模型开发漏洞利用的能力，Claude Mythos Preview 在三者上均优于所有其他受测模型。
推荐理由：Anthropic 红队用三个漏洞利用基准量化 Claude Mythos Preview 的能力，读者可据此看到模型攻破 V8 沙箱与内核的实际差距。
Hugging Face Blog Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality
TL;DR: Two new Apache 2.0 multilingual embedding models built on ModernBERT — a 97M-parameter compact model that beats every open sub-100M multilingual embedder on MTEB Multilingual Retrieval (60.3), and a 311M full-size model that scores 65.2 on MTEB Multilingual Retrieval (#2 among open models under 500M parameters) with Matryoshka support. Both cover 200+ languages, are tuned on 52 languages, handle 32K-token context (64x R1), and add code retrieval across 9 programming languages. In this post: Enterprise-Ready by Design · A Strong Sub-100M Multilingual Model · What Changed from R1 · Training the Full-Size 311M Model · Building the compact 97M Multilingual Model · Benchmark Results · Matryoshka Embeddings · Deployment Options · For Framework Integrators · Which Model Should You Use? · Try The Models Multilingual embedding models face a persistent tension: broad language coverage usually comes at the cost of model size, and small models usually sacrifice languages. If you work across languages — retrieval-augmented generation over multilingual corpora, cross-lingual search, code retrieval in international teams — you've likely had to choose between a model that's fast enough and one that's good enough. The Granite Embedding Multilingual R2 release narrows that gap considerably. We're releasing two new multilingual embedding models: granite-embedding-311m-multilingual-r2 — A 311M-parameter full-size model with 768-dimensional embeddings, Matryoshka dimension support, and top-tier multilingual retrieval quality. granite-embedding-97m-multilingual-r2 — A 97M-parameter compact model with 384-dimensional embeddings that delivers strong retrieval quality for its size. Both models support 200+ languages with enhanced retrieval quality for 52 languages and programming code , handle context lengths up to 32,768 tokens (a 64x increase over their R1 predecessors), and are released under the Apache 2.0 license. They work out of the box with sentence-transformers and transformers , require no task-specific instructions, and are compatible as drop-in replacements in LangChain , LlamaIndex , Haystack , and Milvus with a one-line model name change. For frameworks currently using an English-only default, that one line gives every user in your community support for 200+ languages — no API changes, no new dependencies, no code changes required on their end. Both models ship with ONNX and OpenVINO weights for CPU-optimized inference. 52 enhanced-support languages (click to expand) The underlying encoder was pretrained on text from 200+ languages, producing general-purpose embeddings for any of them. The following 52 languages receive explicit retrieval-pair and cross-lingual training for higher-quality retrieval: Albanian (sq), Arabic (ar), Azerbaijani (az), Bengali (bn), Bulgarian (bg), Catalan (ca), Chinese (zh), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Icelandic (is), Indonesian (id), Italian (it), Japanese (ja), Kazakh (kk), Khmer (km), Korean (ko), Latvian (lv), Lithuanian (lt), Malay (ms), Marathi (mr), Norwegian (no), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Serbian (sr), Slovak (sk), Slovenian (sl), Spanish (es), Swahili (sw), Swedish (sv), Tagalog (tl), Telugu (te), Thai (th), Turkish (tr), Ukrainian (uk), Urdu (ur), Uzbek (uz), Vietnamese (vi). Additionally, the models are trained on programming code (Python, Go, Java, JavaScript, PHP, Ruby, SQL, C, C++) and support cross-lingual code retrieval. Enterprise-Ready by Design Both embedding models are trained on a mixture of IBM‑curated datasets, publicly available data, and internally generated or synthetic data. Public web‑derived data used in training is selected and filtered using IBM‑developed quality, deduplication, and governance processes intended to reduce risk in downstream commercial use. We intentionally avoid the use of the MS‑MARCO training dataset and datasets with explicit non‑commercial licensing restrictions. The models are pretrained using GneissWeb , an IBM‑curated dataset derived from publicly available web content and processed using IBM’s data preparation and governance tooling—along with additional IBM‑curated and other publicly available sources. Datasets undergo IBM governance review to assess licensing considerations, ownership signals, and personal data risks. These processes are designed to contribute to responsible use and enterprise deployment. A Strong Sub-100M Multilingual Model The standout of this release is granite-embedding-97m-multilingual-r2 . At 97 million parameters, it scores 60.3 on Multilingual MTEB Retrieval across 18 languages — the highest retrieval score we've found for any open multilingual embedding model under 100M parameters. The next-best model in that size class, multilingual-e5-small, scores 50.9 on the same benchmark — a +9.4 point gap on a mature benchmark. At roughly one-third the size of the 311M full-size model, it retains the majority of its retrieval quality across multilingual, code, and long-document benchmarks — a +12.2 point gain on MTEB Multilingual Retrieval over its direct predecessor, driven by a new architecture, better training data, and a novel pruning methodology (more on that below). The full-size granite-embedding-311m-multilingual-r2 scores 65.2 on the same benchmark, a +13.0 point gain over its R1 predecessor. What Changed from R1 The Granite Embedding Multilingual R1 models were built on XLM-RoBERTa encoders with a 512-token context window. The R2 generation is a ground-up rebuild: ModernBERT is a recent encoder architecture that revisits the original BERT design with techniques from the last five years of transformer research. The shift brings several practical benefits: alternating attention lengths reduce computation on long sequences (improves throughput on long sequences significantly), rotary position embeddings allow the 32K context window without the positional interpolation hacks that plague older architectures, and Flash Attention 2.0 support speeds up encoding on modern GPUs. The new multilingual tokenizers are worth highlighting. Rather than reusing XLM-RoBERTa's 250K-token vocabulary, we adopted existing tokenizers with strong multilingual and code coverage. The 311M model uses the Gemma 3 tokenizer (262K tokens); the 97M model starts from the GPT-OSS tokenizer and prunes it down to a compact 180K-token vocabulary that preserves broad multilingual coverage while reducing the embedding table's parameter footprint. Tokenizer efficiency matters more than people realize — a 32K-token window sounds impressive until your tokenizer burns half of it encoding a single paragraph of Thai. Training the Full-Size 311M Model The 311M model is a 22-layer ModernBERT encoder with a 262K-token multilingual vocabulary, trained through a multi-stage pipeline: Knowledge distillation : The model learns from multiple teacher models simultaneously. The teachers are Granite 3.3 Instruct and Mistral v0.2 Instruct decoder models, further finetuned for text embeddings, which transfer retrieval-specific knowledge into the 311M encoder architecture. Contrastive fine-tuning : Standard contrastive training on multilingual retrieval pairs — queries matched with relevant and hard-negative passages across 52 languages and code — sharpens the model's ability to distinguish relevant from irrelevant results. Model merging : After training, we merge checkpoints from different training stages and configurations. This combines the strengths of models optimized for different objectives (e.g., multilingual breadth vs. English depth) into a single set of weights without additional training compute. Matryoshka Representation Learning : The model is trained with Matryoshka objectives so that its 768-dimensional embeddings can be truncated to 512, 384, 256, or 128 dimensions with minimal quality loss (see Matryoshka Embeddings below). The result is a model that scores 65.2 on MTEB Multilingual Retrieval and 56.3 on the overall average — a +14.5 point average gain over its R1 predecessor. Building the compact 97M Multilingual model The 97M model is trained through a combination of vocabulary selection and knowledge distillation : Vocabulary selection : The 262K-token vocabulary is reduced to a purpose-trained 180K-token vocabulary that preserves broad multilingual coverage while cutting the embedding table size substantially. Knowledge distillation : The pruned model is then finetuned using knowledge distillation from multiple teacher models (including a Granite 4.1 8B and Mistral Instruct decoder-based teacher) and contrastive training to improve retrieval quality. This approach transfers retrieval-specific knowledge from multiple strong teachers, while reducing the model parameters without sacrificing language coverage. The result is a highly efficient compact model — scoring 60.3 on MTEB Multilingual Retrieval vs. 65.2 for the full-size model, while being approximately 3x smaller. Benchmark Results Multilingual Retrieval Performance across the main benchmark suite sorted by model size. Scores are averages across tasks within each benchmark (higher is better): Model Params Active Params Embed Dim MTEB Multilingual Retrieval (18) Code (12) English Retrieval (10) LongEmbed (6) RaR-b (17) F2LLM-v2-80M 80M 32M 320 50.1 68.0 47.5 31.7 17.9 multilingual-e5-small 118M 22M 384 50.9 53.5 46.5 38.8 20.3 granite-embedding-107m-multilingual (R1) 107M 11M 384 48.1 40.7 47.9 34.3 17.1 paraphrase-multilingual-MiniLM-L12-v2 118M 22M 384 36.6 23.5 35.9 20.9 10.9 jina-embeddings-v5-text-nano 212M 113M 768 63.3 71.2 58.8 63.6 25.2 harrier-oss-v1-270m 268M 100M 640 66.4 62.4 52.1 64.9 32.9 multilingual-e5-base 278M 86M 768 52.7 52.6 49.0 40.5 23.4 granite-embedding-278m-multilingual (R1) 278M 86M 768 52.2 48.5 51.5 37.7 18.9 embeddinggemma-300m 308M 106M 768 62.5 68.7 54.6 55.4 26.1 gte-multilingual-base 305M 113M 768 57.2 57.5 50.8 62.1 19.0 snowflake-arctic-embed-m-v2.0 305M 113M 768 54.8 55.2 58.4 55.4 23.3 multilingual-e5-large 560M 304M 1024 53.7 55.8 51.5 40.4 25.4 text-embedding-3-small (OpenAI, API only) — — 1536 50.7 — 53.8 53.6 23.2 granite-embedding-97m-multilingual-r2 97M 28M 384 60.3 60.4 50.1 65.6 24.9 granite-embedding-311m-multilingual-r2 311M 110M 768 65.2 (#2) 63.8 (#3) 52.6 (#5) 71.7 (#1) 28.0 (#2) A few things stand out: The 97M R2 model beats multilingual-e5-base and gte-multilingual-base (~300M parameter models) on average and on most individual benchmarks, despite being roughly 3x smaller. paraphrase-multilingual-MiniLM-L12-v2 — a widely-used framework default — scores 36.6 , a full +23.7 points behind the 97M R2 model, which is also slightly smaller (97M vs 110M parameters) with the same 384-dimensional output. LongEmbed is the biggest R1-to-R2 gain : +31.3 points for the 97M model, +34.0 for the 311M. This is the direct payoff of the 32K context window — R1's 512-token limit meant your legal contract was being judged by its first page. Many practical multilingual workloads involve long documents (legal contracts, technical manuals, research papers, multi-page reports) that R1 simply could not see in full. Code retrieval improves dramatically : +19.7 (97M) and +15.3 (311M) over R1, reflecting the new code training set, larger context window, and better training methodology. In the broader competitive field , harrier-oss-v1-270m leads on MTEB Multilingual Retrieval (66.4) and RaR-b (32.9), while jina-embeddings-v5-text-nano leads on Code (71.2) and English Retrieval (58.8). The 311M Granite model is competitive on average (56.3) and leads on LongEmbed (71.7), while offering substantially higher encoding throughput than jina-embeddings-v5-text-nano (see speed table below). Speed and Throughput Encoding speed matters for production workloads, especially when you're indexing millions of documents or need low-latency query encoding. We measured latency and throughput on a single NVIDIA H100 GPU using 512-token chunks: The 97M model encodes over 2,500 documents per second — comparable throughput to multilingual-e5-small — while delivering substantially higher retrieval quality. The 311M model, at ~1,800 docs/sec, performs better than jina-embeddings-v5-text-nano on retrieval quality (65.2 vs. 63.3) at over 5.5x the encoding speed (note: speed numbers are computed with the latest transformer code, which had a speed regression vs the last 4.57 version - for both the Jina and granite models - see our technical report for details). harrier-oss-v1-270m offers the best combination of speed and retrieval score among the competitors listed here. Matryoshka Embeddings (311M) The 311M model supports Matryoshka Representation Learning , which lets you truncate embeddings from the full 768 dimensions down to 512, 384, 256, or 128 with graceful quality degradation. This is useful when storage, memory, or similarity-computation cost is a concern — a 256-dimensional embedding takes one-third the storage of a 768-dimensional one, and cosine similarity is proportionally cheaper to compute. Here's how retrieval quality holds up across embedding dimensions: The quality loss from dimension reduction is remarkably small. Cutting from 768 to 256 dimensions — a 3x reduction in storage and similarity-computation cost — drops MTEB Multilingual Retrieval by just 0.5 points (65.2 → 64.7) and Code Retrieval by 0.5 points (63.9 → 63.4). Even at 128 dimensions (a 6x reduction ), the model still scores 63.7 on MTEB Multilingual Retrieval and 62.3 on Code — retaining over 97% of its full-dimension performance. In practice, this means you can substantially reduce your index size and search latency with minimal impact on result quality. (Note,results in the above picture were evaluated with a context length of 1024 for English and Multilingual Retrieval, and 8192 for Code). For comparison, the 311M model truncated to 384 dimensions (the same dimensionality as the 97M model's native output) still outperforms the 97M model across all three benchmarks. If you need 384-dimensional embeddings and can afford the 311M model's encoding cost, Matryoshka truncation is the stronger option. from sentence_transformers import SentenceTransformer model = SentenceTransformer( "ibm-granite/granite-embedding-311m-multilingual-r2" ) # Full 768-dimensional embeddings full = model.encode([ "example text" ]) print (full.shape) # (1, 768) # Truncated to 384 dimensions small = model.encode([ "example text" ], truncate_dim= 384 ) print (small.shape) # (1, 384) The 97M model does not support Matryoshka — 384 dimensions is already compact. Cross-lingual Retrieval Average performance on cross-lingual tasks within MTEB Retrieval. Belebele measures cross-lingual passage matching across 122 languages; MLQA measures extractive cross-lingual question answering retrieval across 7 languages. Model Belebele Retrieval MLQA Retrieval granite-embedding-107m-multilingual (R1) 55.1 60.5 granite-embedding-278m-multilingual (R1) 62.2 63.0 granite-embedding-97m-multilingual-r2 52.9 60.5 granite-embedding-311m-multilingual-r2 66.5 67.1 The 311M R2 model gains +4.3 on Belebele and +4.1 on MLQA over its R1 predecessor, showing improved cross-lingual transfer at the larger scale across both benchmarks. The 97M R2 model scores lower on Belebele (52.9 vs 55.1, −2.2) while matching its R1 predecessor on MLQA (60.5). The Belebele gap is a tradeoff inherent in the pruning and vocabulary reduction process — the R2 model's training prioritized the broader 18-language MTEB Multilingual Retrieval set (where it gains +12.2 over R1) and long-document retrieval (+31.3), while the smaller vocabulary (180K vs. 250K tokens) and reduced layer count (12 vs. 22) affect narrow cross-lingual transfer tasks. If cross-lingual transfer across many language pairs is your primary use case, the full-size 311M model is the better choice. Deployment Options Both models ship with multiple deployment paths for production use. Install the core library with: pip install sentence-transformers Sentence Transformers (recommended for most users): from sentence_transformers import SentenceTransformer, util model = SentenceTransformer( "ibm-granite/granite-embedding-97m-multilingual-r2" ) queries = [ "What is the tallest mountain in Japan?" , # English "Wer hat das Lied Achy Breaky Heart geschrieben?" , # German "ドイツの首都はどこですか？" , # Japanese ] passages = [ "富士山は、静岡県と山梨県にまたがる活火山で、標高3776.12 mで日本最高峰の独立峰である。" , # Japanese "Achy Breaky Heart is a country song written by Don Von Tress." , # English "Berlin ist die Hauptstadt und ein Land der Bundesrepublik Deutschland." , # German ] q_emb = model.encode(queries) p_emb = model.encode(passages) print (util.cos_sim(q_emb, p_emb)) # Each query scores highest against its matching passage — across languages LangChain ( pip install langchain-huggingface ): from langchain_huggingface import HuggingFaceEmbeddings embeddings = HuggingFaceEmbeddings( model_name= "ibm-granite/granite-embedding-97m-multilingual-r2" ) docs = embeddings.embed_documents([ "富士山は日本最高峰の独立峰です。" , "Mount Fuji is Japan's highest peak." , ]) query = embeddings.embed_query( "What is Japan's tallest mountain?" ) # Drop-in replacement anywhere LangChain accepts an Embeddings object LlamaIndex ( pip install llama-index-embeddings-huggingface ): from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.core import Settings embed_model = HuggingFaceEmbedding( model_name= "ibm-granite/granite-embedding-97m-multilingual-r2" ) Settings.embed_model = embed_model # applies globally to any index or pipeline Haystack ( pip install sentence-transformers haystack-ai ) from haystack.components.embedders import ( SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder, ) from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever from haystack.dataclasses import Document from haystack.document_stores.in_memory import InMemoryDocumentStore doc_embedder = SentenceTransformersDocumentEmbedder( model= "ibm-granite/granite-embedding-97m-multilingual-r2" ) query_embedder = SentenceTransformersTextEmbedder( model= "ibm-granite/granite-embedding-97m-multilingual-r2" ) doc_embedder.warm_up() query_embedder.warm_up() # Embed and index documents document_store = InMemoryDocumentStore() result_docs = doc_embedder.run(documents=[ Document(content= "富士山は日本最高峰の独立峰です。" ), Document(content= "Mount Fuji is Japan's highest peak." ), Document(content= "Achy Breaky Heart is a country song written by Don Von Tress." ), Document(content= "Berlin ist die Hauptstadt und ein Land der Bundesrepublik Deutschland." ), ]) document_store.write_documents(result_docs[ "documents" ]) # Embed query and retrieve result_query = query_embedder.run(text= "What is Japan's tallest mountain?" ) retriever = InMemoryEmbeddingRetriever(document_store=document_store) results = retriever.run(query_embedding=result_query[ "embedding" ], top_k= 2 ) for doc in results[ "documents" ]: print ( f" {doc.score: .3 f} {doc.content} " ) # 0.961 Mount Fuji is Japan's highest peak. # 0.913 富士山は日本最高峰の独立峰です。 Milvus ( pip install pymilvus sentence-transformers ) from pymilvus import MilvusClient from sentence_transformers import SentenceTransformer model = SentenceTransformer( "ibm-granite/granite-embedding-97m-multilingual-r2" ) # Use "./milvus.db" for local persistence or a server URI for production client = MilvusClient( ":memory:" ) client.create_collection(collection_name= "multilingual_docs" , dimension= 384 ) docs = [ "富士山は日本最高峰の独立峰です。" , "Mount Fuji is Japan's highest peak." , "Achy Breaky Heart is a country song written by Don Von Tress." , "Berlin ist die Hauptstadt und ein Land der Bundesrepublik Deutschland." , ] embeddings = model.encode(docs).tolist() client.insert( collection_name= "multilingual_docs" , data=[{ "id" : i, "vector" : emb, "text" : doc} for i, (emb, doc) in enumerate ( zip (embeddings, docs))], ) query_emb = model.encode([ "What is Japan's tallest mountain?" ]).tolist() results = client.search( collection_name= "multilingual_docs" , data=query_emb, limit= 2 , output_fields=[ "text" ], ) for hit in results[ 0 ]: print ( f" {hit[ 'distance' ]: .3 f} {hit[ 'entity' ][ 'text' ]} " ) # 0.961 Mount Fuji is Japan's highest peak. # 0.913 富士山は日本最高峰の独立峰です。 Both models also ship with pre-converted ONNX and OpenVINO weights for optimized CPU/accelerator inference, work as embedding endpoints via vLLM ( vllm serve ... --task embed ), and can be converted to GGUF for Ollama using llama.cpp . See the model cards for full deployment examples. For Framework Integrators If you maintain an embedding framework, vector store, or RAG pipeline library and are evaluating these models as a default, here's what you need to know: License : Apache 2.0, trained without MS-MARCO Drop-in behavior : No task-specific instruction prefix required — behaves like all-MiniLM-L6-v2 at the API level. Existing code that calls .encode() works unchanged. Dimensionality : 384-dimensional output (97M) and 768-dimensional output (311M), matching the most common existing defaults. No index migration required. Model size : The 97M model's weights are 195 MB (safetensors) — less than half the size of paraphrase-multilingual-MiniLM-L12-v2 (471 MB), the most common multilingual default. The quantized ONNX weights are just 98 MB, comparable to all-MiniLM-L6-v2 (91 MB) while covering 200+ languages. CPU-friendly : Ships with ONNX and OpenVINO weights for optimized CPU inference. No GPU dependency for a getting-started tutorial. Multilingual by default : If your current default is English-only, this is a one-line swap that gives every user in your community support for 200+ languages — without touching their code. Stable identifier : ibm-granite/granite-embedding-97m-multilingual-r2 on Hugging Face, maintained by IBM under the Granite model family. To discuss adopting these models as a default in your project, open an issue at ibm-granite/granite-embedding-models . Which Model Should You Use? These two multilingual models are part of the broader Granite Embedding R2 family, which also includes two high-performing English-focused models: granite-embedding-english-r2 (149M parameters) and granite-embedding-small-english-r2 (47M parameters). If your data is predominantly English, the English models offer higher retrieval quality on English benchmarks at a smaller footprint, since they don't need to allocate capacity across 200+ languages. If you need... Use Best multilingual retrieval quality granite-embedding-311m-multilingual-r2 Flexible embedding dimensions (storage/speed tradeoff) granite-embedding-311m-multilingual-r2 (Matryoshka) Maximum throughput / edge deployment / low latency granite-embedding-97m-multilingual-r2 Best cross-lingual transfer across many language pairs granite-embedding-311m-multilingual-r2 Predominantly English data granite-embedding-english-r2 or granite-embedding-small-english-r2 Try The Models Both models are available now on Hugging Face under the IBM Granite Embedding collection : granite-embedding-311m-multilingual-r2 granite-embedding-97m-multilingual-r2 You can try the small models interactively (on CPU) via a Granite Embedding demo here on Hugging Face Spaces, or run the full examples notebook in Google Colab: You can access our detailed technical report covering the full training methodology, per-language evaluations, and pruning ablations here Granite Multilingual Embedding R2 report . For questions, feedback, or issues, visit ibm-granite/granite-embedding-models on GitHub. Framework maintainers: If you'd like to adopt these models as a default in your project, open an issue at ibm-granite/granite-embedding-models — we're happy to help with integration, testing, and any questions about licensing or deployment. Give them a try, and if the embeddings spark joy, smash that ❤️ button on Hugging Face. Our models have feelings too, and every +1 keeps them warm at night.
OWASP精选推荐指数7878 OWASP ASI06 负责人：AI Agent 的记忆也是攻击面
Idan Habler 作为 OWASP ASI06: Memory & Context Poisoning 条目共同负责人撰文，结合 Cisco 对 Claude Code 的研究（漏洞代号 MemoryTrap）指出。
推荐理由：作者以 ASI06 共同负责人的身份，把 MemoryTrap 案例上升为记忆应被视为攻击面的行业判断，便于读者理解 Agent 安全的新威胁模型。
Hugging Face Blog Adding Benchmaxxer Repellant to the Open ASR Leaderboard
"When a measure becomes a target, it ceases to be a good measure." (Goodhart’s Law) TLDR : Appen Inc. and DataoceanAI have provided high-quality English ASR datasets covering scripted and conversational speech over multiple accents. To prevent potential risks of benchmaxxing or test-set contamination, we will keep these datasets private for a high-quality measure of performance on multiple tasks. We’re not updating the average WER at this time : by default, the leaderboard’s Average WER remains computed on public datasets only. You can optionally include the private datasets using the toggle to see their impact 👀 Since its launch in September 2023, the Open ASR Leaderboard has been visited over 710K times. We’re blown away by the community’s interest and motivation to keep pushing speech recognition 🗣️ Two words sum up the objectives (but also challenges) in maintaining a benchmark like the Open ASR Leaderboard: Standardization : models can have different conventions for their usage and outputs, e.g. with/without punctuation and casing. Datasets have the same challenges and can be structured differently. To this end, all test sets have been gathered into a single dataset on the Hub for easy access and previewing. Moreover, to standardize model outputs and dataset transcripts, we use a normalizer that (among other things) removes punctuation and casing, and maps to American spelling. It is based on the normalizer of Whisper . Openness : the UI code and evaluation scripts are open-sourced. This has helped not only to incorporate new models, but also to improve the quality of the evaluation procedure through community feedback and contributions. Standardization and openness are essential for meaningful benchmarking, but they also make benchmarks more susceptible to benchmark-specific optimization ("benchmaxxing"), where models improve leaderboard performance without corresponding gains in real-world robustness. As models and use cases evolve, the Open ASR Leaderboard will continue incorporating high-quality datasets and new evaluation settings to better reflect real-world performance and improve robustness against benchmark-specific optimization. As discussed in our report , there is no single "catch-all" ASR model: some perform better on American English, others on diverse accents and multilingual settings, while others are optimized for speed or conversational audio. Different applications also prioritize different capabilities, so a model that performs less well on one dimension is not necessarily a worse model overall. The goal of the Open ASR Leaderboard is to capture these nuances and provide a more holistic view of ASR performance. New high-quality, private datasets To this end, we have worked with Appen Inc. and DataoceanAI to curate high-quality datasets for ASR benchmarking. Below is some information on the various splits. Dataset Accent Duration [h] Male (%) / Female (%) Style Transcription Appen Scripted AU Australian 1.42 49 / 51 Read Punctuated, cased. Appen Scripted CA Canadian 1.53 52 / 48 Read Punctuated, cased. Appen Scripted IN Indian 1.02 49 / 51 Read Punctuated, cased. Appen Scripted US American 1.45 49 / 51 Read Punctuated, cased. Appen Conversational IN Indian 1.37 51 / 49 Conversational, spontaneous Punctuated, disfluencies. Appen Conversational US003 American 1.64 49 / 51 Conversational, spontaneous Punctuated, cased, disfluencies. Appen Conversational US004 American 1.65 49 / 51 Conversational, spontaneous Punctuated, disfluencies. DataoceanAI Scripted US American 2.43 54 / 46 Read Punctuated, cased (proper nouns), disfluencies. DataoceanAI Scripted GB British 2.43 47 / 53 Read Punctuated, disfluencies. DataoceanAI Conversational US American 8.82 NA Conversational, spontaneous Punctuated, disfluencies. DataoceanAI Conversational GB British 5.96 NA Conversational, spontaneous Punctuated, disfluencies. Below are sample audio showing the variety of content (scripted, conversational, acronyms, disfluencies, proper nouns). While private datasets may sound contrary to the spirit of openness, we believe that incorporating such datasets will increase the trustworthiness of the Open ASR Leaderboard, as they are less likely to be exploited for benchmaxxing, whether by model developers who explicitly use the public test sets or who try to find training data that closely resembles a particular dataset to boost their score in the macroaverage. With these datasets, we can also provide targeted metrics to highlight gaps and biases between controlled and often saturated settings (scripted, American accent) and more nuanced conditions (conversational and non-American accents). Below is a screenshot of the new "Private data" tab. Below is how each column is computed. "Average WER" computes a macroaverage of the data provider averages, so that they are weighted equally. "Avg Scripted" performs a macroaverage of all scripted datasets. "Avg Conversational" performs a macroaverage of all conversational datasets. "Avg US" performs a macroaverage of all datasets with American accents. "Avg non-US" performs a macroaverage of all datasets with non-American accents. We intentionally do not provide a score on each split, to avoid model developers from boosting their score with a specific data provider or accent. How can I evaluate my model on this data? Get your model on the Open ASR Leaderboard, and we'll run the evaluation! As before, the process for adding a model to the leaderboard takes place on the Open ASR Leaderboard GitHub : Open a pull request, and a model checklist will appear. As before, you should report your results on the public datasets. We will verify the results on the public sets and compute the metrics on the private ones. Confirm the results we’ve obtained. While you wait for your model to be added to the Open ASR Leaderboard, you can self-report your metrics on the public sets by adding a YAML file like this to your model card. Your model will then appear on an (unverified) leaderboard that appears on the dataset page (see screenshot below). More on this approach to decentralized evaluation can be read here . Do models trained on the data providers have an advantage? They could. We’ve asked Appen and DataoceanAI to not provide this data to their clients. But even if they do not provide this exact data, data from a similar distribution could still help the model on the corresponding evaluation set (similar to benchmaxxing by optimizing for a challenging task from the public sets). To this end, having multiple data providers balances out the advantage a model may get from having used data from one of the providers. And we are open to more data providers and eval sets for the "Private data" tab! Moreover, to ensure that the private sets do not affect the model ranking, we’ve defaulted the Average WER to not include the Private sets in its macroaverage. In the screenshot below, you can see that "Private data" is toggled off. This means that the macroaverage across datasets does not include it. Simply toggle on "Private data" splits to include them in the macroaverage. The "Rank Δ" column shows how the ordering changes relative to the default macroaverage configuration. Including or excluding public datasets also changes the macroaverage, allowing users to tailor the evaluation to the use cases and data distributions most relevant to their application. What's next? We’re excited to hear the community’s feedback on how the new track and dataset toggling features help users identify the model(s) that best fit their application(s). We’re also looking into evaluations that better reflect real-world noisy conditions, and you can expect some news on that 😉 While preparing the private evaluation sets, we took extra care to ensure consistent audio and transcript quality across datasets, including developing tooling to identify challenging cases such as low signal-to-noise conditions or transcript mismatches, since these factors can meaningfully affect WER. More on that in a future post!
Hugging Face Blog精选推荐指数6060 Hugging Face 实测 OpenAI Privacy Filter：用 gradio.Server 构建三个 PII 脱敏 Web 应用
Hugging Face 博客作者用 OpenAI 本周在 Hub 上开源的 Privacy Filter（1.5B 参数、50M 激活参数、Apache 2.0 许可。
推荐理由：原文用三个可运行示例演示了 Privacy Filter 的 PII 检测能力与 gradio.Server 的前后端分工，读者可直接照搬这套工程做法。
Hugging Face Blog QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard
QIMMA validates benchmarks before evaluating models, ensuring reported scores reflect genuine Arabic language capability in LLMs. If you've been tracking Arabic LLM evaluation, you've probably noticed a growing tension: the number of benchmarks and leaderboards is expanding rapidly, but are we actually measuring what we think we're measuring? We built QIMMA قمّة (Arabic for "summit"), to answer that question systematically. Instead of aggregating existing Arabic benchmarks as-is and running models on them, we applied a rigorous quality validation pipeline before any evaluation took place. What we found was sobering: even widely-used, well-regarded Arabic benchmarks contain systematic quality issues that can quietly corrupt evaluation results. This post walks through what QIMMA is, how we built it, what problems we found, and what the model rankings look like once you clean things up. 🔍 The Problem: Arabic NLP Evaluation Is Fragmented and Unvalidated Arabic is spoken by over 400 million people across diverse dialects and cultural contexts, yet the Arabic NLP evaluation landscape remains fragmented. A few key pain points have motivated this work: Translation issues. Many Arabic benchmarks are translations from English. This introduces distributional shifts. Questions that feel natural in English become awkward or culturally misaligned in Arabic, making benchmark data less representative of how Arabic is naturally used. Absent quality validation. Even native Arabic benchmarks are often released without rigorous quality checks. Annotation inconsistencies, incorrect gold answers, encoding errors, and cultural bias in ground-truth labels have all been documented in established resources. Reproducibility gaps. Evaluation scripts and per-sample outputs are rarely released publicly, making it hard to audit results or build on prior work. Coverage fragmentation. Existing leaderboards cover isolated tasks and narrow domains, making holistic model assessment difficult. To illustrate where QIMMA sits relative to existing platforms: Leaderboard Open Source Native Arabic Quality Validation Coding Eval Public Outputs OALL v1 ✅ Mixed ❌ ❌ ✅ OALL v2 ✅ Mostly ❌ ❌ ✅ BALSAM Partial 50% ❌ ❌ ❌ AraGen ✅ 100% ✅ ❌ ❌ SILMA ABL ✅ 100% ✅ ❌ ✅ ILMAAM Partial 100% ✅ ❌ ❌ HELM Arabic ✅ Mixed ❌ ❌ ✅ ⛰ QIMMA ✅ 99% ✅ ✅ ✅ QIMMA is the only platform combining all five properties: open source, predominantly native Arabic content, systematic quality validation, code evaluation, and public per-sample inference outputs. ⛰ What's in QIMMA? QIMMA consolidates 109 subsets from 14 source benchmarks into a unified evaluation suite of over 52,000 samples , spanning 7 domains: Domain Benchmarks Task Types Cultural AraDiCE-Culture, ArabCulture, PalmX MCQ STEM ArabicMMLU, GAT, 3LM STEM MCQ Legal ArabLegalQA, MizanQA MCQ, QA Medical MedArabiQ, MedAraBench MCQ, QA Safety AraTrust MCQ Poetry & Literature FannOrFlop QA Coding 3LM HumanEval+, 3LM MBPP+ Code A few things stand out about this design: 99% native Arabic content. The only exception is code evaluation, which is inherently language-agnostic. First Arabic leaderboard with code evaluation. QIMMA integrates Arabic-adapted versions of HumanEval+ and MBPP+, making it possible to assess coding capability with Arabic-language problem statements. Diversity in Domains and Tasks. QIMMA evaluates real-world competency areas including education, governance, healthcare, creative expression, and software development. 🔬 The Quality Validation Pipeline This is the methodological heart of QIMMA. Before running a single model, we applied a multi-stage validation pipeline to every sample in every benchmark. Stage 1: Multi-Model Automated Assessment Each sample was independently evaluated by two state-of-the-art LLMs: Qwen3-235B-A22B-Instruct DeepSeek-V3-671B We chose two models with strong Arabic capability but different training data compositions, so that their combined judgment is more robust than either alone. Each model scores a sample against a 10-point rubric , with binary scores (0 or 1) per criterion: A sample is eliminated if either model scores it below 7/10. Samples where both models agree on elimination are dropped immediately. However, where only one model flags a sample, it proceeds to human review in Stage 2. Stage 2: Human Annotation and Review Flagged samples are reviewed by native Arabic speakers with cultural and dialectal familiarity. Human annotators make final calls on: Cultural context and regional variation Dialectal nuance Subjective interpretation Subtle quality issues automated assessment may miss For culturally sensitive content, multiple perspectives are considered, since "correctness" can genuinely vary across Arab regions. ⚠️ What We Found: Systematic Quality Problems The pipeline revealed recurring quality issues across benchmarks; not isolated errors, but systematic patterns reflecting gaps in how benchmarks were originally constructed. By the Numbers Benchmark Total Samples Discarded Discard Rate ArabicMMLU 14,163 436 3.1% MizanQA 1,769 41 2.3% PalmX 3,001 25 0.8% MedAraBench 4,960 33 0.7% FannOrFlop 6,984 43 0.6% ArabCulture 3,482 7 0.2% MedArabiQ 499 1 0.2% GAT 13,986 1 ~0.0% 3LM STEM 2,609 1 ~0.0% AraDiCE-Culture 180 0 0.0% ArabLegalQA 79 0 0.0% AraTrust 522 0 0.0% Taxonomy of Issues Found ⚖️ Answer Quality False or mismatched gold indices, factually wrong answers, missing or raw text answers. 📄 Text & Formatting Quality Corrupt or illegible text, spelling and grammar errors, and duplicate samples. 💬 Cultural Sensitivity Stereotype reinforcement and monolithic generalizations about diverse communities. 🤝 Gold Answer Compliance Misalignment of gold answers with evaluation protocols. 💻 Code Benchmark: A Different Kind of Quality Work Code benchmarks required a different intervention. Rather than discarding samples, we refined the Arabic problem statements in 3LM's Arabic adaptations of HumanEval+ and MBPP+, leaving task identifiers, reference solutions, and test suites completely unchanged. The modification rates were striking: Benchmark Total Prompts Modified Unchanged Modification Rate 3LM HumanEval+ 164 145 19 88% 3LM MBPP+ 378 308 70 81% Modifications fell into five categories: Linguistic refinement : normalizing toward natural Modern Standard Arabic and consistent imperative style Clarity improvements : fixing ambiguous instructions and unclear constraints Consistency normalization : standardizing mathematical terminology, punctuation, and example formatting Structural corrections : fixing broken triple-quoted strings, indentation errors, corrupted text fragments Semantic refinements : clarifying whether ranges are inclusive/exclusive, preserving task intent ⚙️ Evaluation Setup Evaluation Framework QIMMA uses LightEval , EvalPlus and FannOrFlop as its evaluation framework, chosen for consistency, multilingual community adoption, and reproducibility. Metrics by Task Type Task Type Metric Benchmarks MCQ Normalized Log-Likelihood Accuracy AraDiCE-Culture, ArabicMMLU, ArabCulture, PalmX, 3LM STEM, MedArabiQ, GAT, MedAraBench, AraTrust Multi-select MCQ Probability Mass on Gold Choices MizanQA Generative QA F1 BERTScore (AraBERT v02) MedArabiQ, ArabLegalQA, FannOrFlop Code Pass@1 3LM HumanEval+, 3LM MBPP+ Prompt Templates QIMMA standardizes prompting by question format, with six template types: MCQ : generic multiple choice · MCQ-C : multiple choice with context passage · MCQ-I : multiple choice with specific instructions (GAT analogy/completion) · QA : generic open-ended QA · QA-C : QA with context · QA-F : fill-in-the-blank QA All prompts are in Arabic. For MizanQA and ArabCulture, benchmark-specific system prompts from the original papers are preserved. 🏆 Leaderboard Results Results as of April 2026; covering top 10 evaluated models. Visit the live leaderboard for current rankings. Rank Model AVERAGE AraDiCE-Culture ArabicMMLU ArabCulture PALMX 3LM STEM AraTrust MizanQA MedArabiQ ArabLegalQA GAT MedAraBench HumanEval+ MBPP+ FannOrFlop 🥇 1 Qwen/Qwen3.5-397B-A17B-FP8 68.06 82.78 77.54 61.75 83.91 88.67 90.04 73.36 47.30 54.94 55.89 47.97 67.68 76.72 44.33 🥈 2 Applied-Innovation-Center/Karnak 66.20 73.33 80.94 53.49 81.40 93.10 89.08 55.92 55.78 71.58 61.06 54.19 33.54 64.55 58.91 🥉 3 inceptionai/Jais-2-70B-Chat 65.81 78.89 81.29 83.24 83.73 87.96 90.23 71.78 52.79 69.60 51.67 50.89 19.51 43.65 56.13 #4 Qwen/Qwen2.5-72B-Instruct 65.75 77.22 73.78 63.83 77.77 87.55 88.51 63.49 50.06 70.74 55.90 44.19 37.20 72.75 57.51 #5 Applied-Innovation-Center/AIC-1 65.37 73.33 72.02 77.52 76.11 88.13 90.61 56.36 53.75 68.96 62.11 50.78 28.05 69.58 47.83 #6 Qwen/Qwen3.5-122B-A10B 64.84 74.44 73.17 37.78 81.46 86.18 86.97 64.01 47.04 55.11 50.90 52.49 65.24 72.43 60.54 #7 Sakalti/Ultiima-72B 64.49 78.33 72.28 68.79 76.75 83.70 89.08 60.44 44.58 69.12 46.91 42.25 39.02 74.07 57.56 #8 meta-llama/Llama-3.3-70B-Instruct 63.96 77.22 71.57 78.05 77.95 88.28 85.63 67.44 56.25 64.00 51.13 54.86 27.44 71.16 24.43 #9 Qwen/Qwen2.5-32B-Instruct 63.26 70.56 68.76 75.80 72.07 81.03 85.82 53.78 48.08 69.27 56.94 36.51 34.15 72.75 93.10 #10 FreedomIntelligence/AceGPT-v2-32B-Chat 61.14 76.67 70.62 79.79 74.46 84.88 86.97 63.89 49.96 71.46 56.04 47.32 23.78 54.50 15.56 Scale does not guarantee best performance. The top 10 spans models from 32B to 397B parameters, with several mid-size models outperforming larger ones on specific domains. Arabic-specialized models lead on cultural and linguistic tasks. Jais-2-70B-Chat ranks highest on ArabicMMLU and ArabCulture, while Karnak leads on 3LM STEM and ArabLegalQA. Coding remains the hardest domain for Arabic-specialized models. The top HumanEval+ and MBPP+ scores belong to multilingual models, with Qwen3.5-397B leading both. The Size-Performance Relationship Across the full leaderboard (46 models), a clear but imperfect size-performance correlation emerges. However, there are interesting exceptions: Arabic-specialized models often outperform size-matched multilingual models Instruction-tuned models consistently outperform their base counterparts except for Qwen3 Some smaller Arabic-specialized models (Fanar-1-9B, ALLaM-7B) outperform much larger multilingual models on specific domains 🌟 What Makes QIMMA Different To summarize the distinctive properties of QIMMA: Property Details Quality-first philosophy Validation runs before evaluation, not as an afterthought Multi-model validation Two LLMs with different training + human review for flagged cases 99% native Arabic Avoids translation artifacts almost entirely Multi-domain, multi-task 7 domains, 3 task types (MCQ, QA, code), 109 subsets Code evaluation First Arabic leaderboard to include code generation Full transparency Per-sample inference outputs publicly released, not just aggregate scores LightEval-based Unified, reproducible evaluation codebase Dialectal awareness Explicit handling of MSA vs. dialectal variation in prompts and rubrics 🔗 Resources 🏆 Leaderboard : QIMMA Leaderboard 💻 Code : GitHub 📄 Paper : Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation 🔖 Citation @misc{alqadi2026arabicbenchmarksreliableqimmas, title={Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation}, author={Leen AlQadi and Ahmed Alzubaidi and Mohammed Alyafeai and Hamza Alobeidli and Maitha Alhammadi and Shaikha Alsuwaidi and Omar Alkaabi and Basma El Amel Boussaha and Hakim Hacid}, year={2026}, eprint={2604.03395}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2604.03395}, }
Hugging Face Blog推荐指数5858 Hugging Face 撰文论述 AI 网络安全中开放性为何关键
Hugging Face 发布博文，围绕 Mythos 与 Project Glasswing 的发布，分析 AI 网络安全的现状并论证开放性的作用。文章指出 Mythos 的能力来自系统配方（大算力、软件数据训练的模型、漏洞探测与修补脚手架、一定自主性），而非单一模型，且较小模型嵌入具备深厚安全专长的系统也可能以更低成本实现类似效果。
OWASP精选推荐指数7171 OWASP FinBot CTF 上线，作为 Agentic Security Initiative 的动手实践平台
OWASP 宣布 FinBot CTF 平台正式上线，它是 OWASP GenAI Security Project 旗下 Agentic Security Initiative 的一部分，被定位为 Agentic AI 领域的 Juice Shop，围绕模拟的金融服务应用提供动手环境。
推荐理由：原文说明了 FinBot 的挑战范围与风险映射，读者可据此了解它如何把 Agentic Top 10 的抽象条目落到可操作的实测场景。
OWASP精选推荐指数7575 OWASP 发布 2026 年 Q1 GenAI 漏洞事件汇总报告
OWASP GenAI Security Project 发布 GenAI Exploit Round-up Report Q1 2026，覆盖 2026 年 1 月 1 日至 4 月 11 日。
推荐理由：报告逐条把事件映射到 OWASP LLM 与 Agentic Top 10，并指出多数 AI 安全事件尚未进入 CVE 体系，读者可据此理解传统漏洞管理与 AI 风险之间的缺口。
网信办 · 网络安全推荐指数4040 中央网信办、国家铁路局联合约谈7家涉火车票销售第三方平台
中央网信办、国家铁路局近日依据《网络安全法》《关键信息基础设施安全保护条例》联合约谈携程、同程、去哪儿、飞猪、美团、智行火车票、高铁管家等7家涉火车票销售第三方平台，要求不得利用自动化程序实施大规模、高频次抢票操作干扰铁路12306平台的安全核验措施。相关部门将加强技术监测，对利用技术手段干扰、危害12306平台安全的行为依法严肃查处。
Anthropic推荐指数4242 Anthropic 详解如何构建可信 AI Agent：从 Claude Code 的 Plan Mode 到四层组件安全设计
Anthropic 发布《Trustworthy agents in practice》，阐述其去年 8 月提出的可信 Agent 五原则（人类控制、价值对齐、交互安全、透明、隐私）如何落地到 Claude Code 和 Claude Cowork 等产品。
Anthropic推荐指数8888 Anthropic 红队评估 Claude Mythos Preview 的网络安全能力
Anthropic 红队发布对 Claude Mythos Preview 的网络安全能力评估，称该模型可在用户指令下发现并利用各大操作系统与浏览器中的零日漏洞，并已启动 Project Glasswing 向关键行业伙伴与开源开发者限量开放。
Hugging Face Blog A New Framework for Evaluating Voice Agents (EVA)
Introduction Conversational voice agents present a distinct evaluation challenge: they must simultaneously satisfy two objectives — accuracy (completing the user's task correctly and faithfully) and conversational experience (doing so naturally, concisely, and in a way appropriate for spoken interaction). These objectives are deeply intertwined: mishearing a confirmation code renders perfect LLM reasoning meaningless, a wall of options overwhelms a caller who can't skim spoken output, and delayed responses can pass every accuracy check while remaining unusable in practice. Existing frameworks treat these as separate concerns — evaluating task success or conversational dynamics, but not both. We introduce EVA, an end-to-end evaluation framework for conversational voice agents that evaluates complete, multi-turn spoken conversations using a realistic bot-to-bot architecture. EVA produces two high-level scores, EVA-A (Accuracy) and EVA-X (Experience), and is designed to surface failures along each dimension. EVA is the first to jointly score task success and conversational experience. We release EVA with an initial airline dataset of 50 scenarios covering flight rebooking, cancellation handling, vouchers, and more — the first in a planned series of domains. We also provide benchmark results for 20 cascade and audio-native systems, such as speech-to-speech (S2S) models and Large Audio Language Models (LALMs). Our biggest finding is that there is a consistent Accuracy-Experience tradeoff; agents that perform well on task completion tend to deliver worse user experiences, and vice versa. 🌐 Website — Explore the full framework, early results, and a demo. 💻 GitHub — Dive into the code, framework, and judge prompts. 📂 HF Dataset - Explore the dataset Background and Motivation The field currently lacks a framework that evaluates the full quality of voice agent interactions, as most existing efforts assess individual components in isolation. For example, AudioBench , SD-Eval , VoxEval , Kimi-Audio-Evalkit , VoiceBench and VoxDialogue evaluate core speech understanding capabilities for Speech-to-Text (STT) — transcription, paralinguistics, acoustic cues — but remain confined to single-turn, non-interactive settings. On the other hand, EmergentTTS-Eval and SHEET assess perceived speech quality using subjective listening tests (e.g., Mean Opinion Score). Beyond speech perception, FD-Bench , Talking Turns , Full-Duplex-Bench provide deeper analyses of conversational dynamics — interruptions, backchanneling, turn-taking — yet evaluate these in isolation from task-oriented tool use, leaving the relationship between dialogue quality and agentic capability unexamined. More recent efforts, notably VoiceAgentBench and CAVA , take steps towards evaluating the agentic capabilities of commercial voice agent systems, including tool-calling and complex instruction-following. However, these voice-agentic capabilities are not evaluated within complete conversational workflows that voice agents must navigate in practice: from initial user request through multi-step tool orchestration to final task resolution. The lack of frameworks that jointly capture accuracy and experience underscores the need for a framework that treats voice agent quality as an integrated whole. This means evaluating not only whether the task succeeded, but whether the agent communicated accurately, concisely, and naturally throughout, and surfacing how these dimensions trade off against one another in realistic deployment conditions. EVA The Framework End-to-end evaluation reveals interaction dynamics that are not apparent at the component level: whether the agent interrupts users during natural pauses in speech, whether it recovers smoothly when a user corrects a transcription error, or whether high latency disrupts the conversational flow enough to prompt users to repeat themselves or abandon the task entirely. EVA simulates multi-turn spoken conversations over live audio in which the agent must invoke appropriate tools, adhere to task-specific policies, and reach a deterministically verifiable end state. EVA evaluates voice agents using a bot-to-bot audio architecture composed of five core components: User Simulator — A conversational AI configured with a specific goal and persona that plays the role of a caller. It operates in audio using high-quality Text-to-Speech (TTS) models, ensuring the evaluation captures representative speech-understanding challenges in natural-sounding conversational speech and realistic turn-taking dynamics. Voice Agent — The voice agent being evaluated, built with Pipecat, an open-source Python framework for real-time voice applications. EVA supports both cascade architectures (STT → LLM → TTS) and audio-native models (S2S or LALM → TTS). Tool Executor — The engine that provides deterministic, reproducible tool responses via custom Python functions. It dynamically queries and modifies a predefined per-scenario database. Validators — A set of validation metrics that check that conversations are complete and that the user faithfully reproduced the intended behavior and speech, with no human annotation required. Any conversation that fails in this validation step is regenerated, ensuring that only valid, correctly executed conversations enter evaluation. This stands in contrast to approaches that rely on post-hoc human labeling to identify simulator errors. Metrics Suite — A suite of metrics evaluates the voice agent using the conversation recording, transcript, and tool call logs. Data Each test case (scenario) in our framework is an evaluation record, structured to make tests reproducible: User Goal — What the caller is trying to accomplish. Includes a highly specific user objective with an exact decision tree that guides the user simulator through the conversation, leaving no ambiguity about the intended outcome. User Persona — How the caller should behave — their speaking style, patience level, and personality traits. Scenario Database — The backend data the agent's tools will query. Ground Truth — The expected final state of the scenario database after a successful conversation. We release EVA with a synthetic airline dataset of 50 scenarios and 15 tools, spanning IRROPS rebooking, voluntary itinerary changes, cancellations, same-day standby, and compensation vouchers. Scenarios are designed to test temporal reasoning, policy-following, constraint satisfaction, and named-entity handling. See the full demo here . Evaluation Methodology EVA evaluates voice agents across two fundamental dimensions, EVA-A for accuracy, and EVA-X for experience. EVA also includes a set of diagnostic metrics. Unlike the primary metrics, these are not used directly to compare or rank models — rather, they offer granular insight into why a model scores the way it does, helping identify and understand specific failure modes (e.g., ASR, speech synthesis, etc.). We report pass@k (the probability that at least one of k runs succeeds) and pass^k (the probability that all k runs succeed) across three trials per scenario (k = 3), capturing both peak performance and behavioral consistency. EVA uses two evaluation methods: deterministic code-based metrics, which compute scores directly from structured data and are fast; and LLM-as-Judge metrics, which use Large Language Models (LLMs) to assess qualitative aspects of the conversation, or Large Audio Language Models (LALM) to evaluate speech directly. Each judge-based metric uses the model that performs best on a curated evaluation dataset for that specific metric. EVA-A: Accuracy Task completion alone is a necessary but insufficient measure of accuracy. An agent can reach the correct end state while fabricating a policy detail, misreading a confirmation code aloud, or hallucinating a flight number mid-conversation. These failures are invisible to a binary pass/fail check but directly harm users. EVA-A therefore measures three dimensions of accuracy: Task Completion [Deterministic] — Measures whether the agent correctly completed the task by comparing the expected end state of the scenario database against the actual end state after the conversation. Faithfulness [LLM-as-Judge] — Measures whether the agent's responses were grounded in its instructions, policies, user inputs, and tool call results — flagging fabrications, misrepresentations, policy violations, and hallucinations. Speech Fidelity [LALM-as-Judge] — Measures whether the speech system faithfully reproduced the intended text in spoken audio, with particular focus on entities critical to get right in a voice context, such as confirmation codes, flight numbers, and dollar amounts. This is the only metric in any end-to-end voice agent benchmark that evaluates the quality of the agent's own spoken output at the audio level. EVA-X: Experience Turn-taking timing matters, but it tells only part of the story. An agent can have perfect timing while overwhelming a caller with a wall of spoken options they cannot skim, or repeatedly asking for information already given. These failures degrade the experience without ever involving a mistimed response. EVA-X therefore measures three dimensions of experience: Conciseness [LLM-as-Judge] — Measures whether the agent's responses were appropriately brief and focused for spoken delivery, since phone users cannot skim, re-read, or scroll back through long responses. Conversation Progression [LLM-as-Judge] — Measures whether the agent moved the conversation forward effectively — avoiding repetition, retaining context across turns, and driving toward task completion without stalling. Turn-Taking [LLM-as-Judge] — Measures whether the agent spoke at the right time — neither interrupting the user nor introducing excessive silence after they finish speaking. Findings We evaluated 20 systems — proprietary and open-source, cascade and audio-native — and find a consistent accuracy-experience tradeoff: agents that perform well on task completion tend to deliver worse user experiences, and vice versa — a tradeoff invisible to benchmarks that score only task completion. No single configuration dominates both axes, confirming that accuracy and experience must be measured jointly. Additionally, we identified named entity transcription as a dominant failure mode. A single misheard character can cascade into an authentication failure and a full conversation breakdown. Also, multi-step workflows break agents in predictable ways. Rebooking a flight while preserving ancillary services — seats, baggage — is the dominant complexity breaker across all configurations. Finally, we observed that additional calibration is needed for real-world use cases. The gap between pass@3 and pass^3 is substantial across all configurations. Even agents that can complete a task often cannot do so consistently, which is critical for real-world success. View the early results here . Limitations EVA is designed to provide rigorous, end-to-end evaluation of conversational voice agents, but several limitations are important to acknowledge, across the framework, data, and metrics dimensions: Metrics — LLM-as-judge models carry inherent biases and may favor certain response styles independent of quality, with additional risk of systematic bias when the evaluated and judge models share a provider. While we validate our judges against labeled datasets and report accuracy measurements on our website, these alignment scores do not eliminate systematic bias entirely. Additionally, task completion is measured as binary, which does not capture partial credits and may understate the relative quality of systems that fail gracefully versus catastrophically. Simulation — The current release covers 50 English-language scenarios in a single domain (airline); results may not generalize to other domains, languages, or accents. Also, the user simulator may not perfectly replicate real caller behavior (e.g., disfluencies, hesitations, emotions) or guarantee full policy adherence. Framework — The user simulator relies on a single commercial provider whose voice characteristics may systematically favor certain ASR systems, and the bot-to-bot pipeline — including audio format conversions and real-time audio interfaces — may not fully represent production deployments. Also, full reproduction requires commercial API access, and latency measurements will vary across providers and infrastructure. What's Next On the evaluation side, we plan to add prosodic quality assessment (pronunciation, rhythm, expressiveness) — currently an open problem after finding very low alignment between LALM-as-Judge and human judgments. We also plan robustness testing under noisy conditions, diverse accents, multilingual users, and varied speaker behaviors, alongside affect-aware evaluation of how agents respond to user distress. In terms of data, we are developing additional domain datasets — each with distinct policy structures, named entity profiles, and conversational dynamics — and more complex scenarios involving compound requests, multi-step follow-ups, and longer conversational memory. On the tooling front, we will release a results and error analysis application that automatically identifies errors per metric and model, surfaces representative examples for exploration, and generates structured summaries of each model’s strengths and weaknesses. Finally, we intend to expand the leaderboard continuously to provide an up-to-date assessment of voice agent capabilities across the field. View more details about limitations and our upcoming roadmap here . Acknowledgements Core contributors include Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, and Hari Subramani. We also thank Lindsay Brin, Akshay Kalkunte, Joseph Marinier, Jishnu Nair, and Aman Tiwari for their careful data review and thoughtful contributions to the framework, and Fanny Riols, Anil Madamala, Sridhar Nemala, and Srinivas Sunkara for their management, leadership, and support throughout. We also extend our thanks to the PAVA and CLAE ServiceNow teams, whose prior work on evaluations and voice agents provided valuable inspiration for this project. Citation @misc{eva-2026, title={A New End-to-end Framework for Evaluating Voice Agents (EVA)}, author={Bogavelli, Tara and Gauthier Melançon, Gabrielle and Stankiewicz, Katrina and Bamgbose, Oluwanifemi and Nguyen, Hoang and Mehndiratta, Raghav and Subramani, Hari}, year={2026}, url={https://github.com/ServiceNow/eva} }
OWASP精选推荐指数6060 OWASP GenAI Security Project 在 RSA 2026 前发布 LLM 与 Agentic 安全新版指南
OWASP GenAI Security Project 宣布发布 Q2 2026 Updated Landscape Guide for LLM and Agentic Security，新增更新的厂商与工具生态文档以及新的 agentic 红队分类法，覆盖识别、度量、缓解和治理 AI 风险的全生命周期框架。
推荐理由：原文列出了新增的 Agentic 红队分类法和 MCP 安全指南等具体产出，读者可据此了解 OWASP 在 Agent 安全方向的框架布局。
Anthropic精选推荐指数7878 Anthropic 红队逆向分析 Claude 为 CVE-2026-2796 编写的漏洞利用
Anthropic 前沿红队公开了 Claude Opus 4.6 为 Firefox 的 CVE-2026-2796（已修复）编写漏洞利用的逆向分析过程。
推荐理由：Anthropic 红队复盘 Claude Opus 4.6 自主写出 Firefox 漏洞利用的完整过程，可看到模型如何一步步搭出利用原语。
腾讯玄武实验室精选推荐指数8080 腾讯玄武实验室提出 Agentic Coding 幽灵依赖供应链威胁并发布 Atuin 防护插件
腾讯玄武实验室提出 Agentic Coding 范式下的新型供应链安全威胁幽灵依赖，包含版本幽灵（LLM 倾向引入过时组件版本）和名称幽灵（LLM 捏造不存在的组件名）两类风险，并定义了利用过时组件 N-day 漏洞和针对幻觉组件名定向投毒两种攻击方式。
推荐理由：研究给出了 Agentic Coding 场景下幽灵依赖的威胁模型与实验数据，读者可据此理解 AI 自主供应链决策带来的新型攻击面及防御思路。
Hugging Face Blog OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
AI agents often perform impressively in controlled research settings, yet struggle when deployed in real-world systems where they must reason across multiple steps, interact with real tools and APIs, operate under partial information, and recover from errors in stateful, permissioned environments—highlighting a persistent gap between research success and production reliability. OpenEnv is an open-source framework from Meta and Hugging Face designed to address this challenge by standardizing how agents interact with real environments. As part of this collaboration, Turing contributed a production-grade calendar management environment to study tool-using agents under realistic constraints such as access control, temporal reasoning, and multi-agent coordination. In this post, we explore how OpenEnv works in practice, why calendars serve as a powerful benchmark for real-world agent evaluation, and what our findings reveal about the current limitations of tool-using agents. What Is OpenEnv? OpenEnv is a framework for evaluating AI agents against real systems rather than simulations . It provides a standardized way to connect agents to real tools and workflows while preserving the structure needed for consistent and reliable evaluation. OpenEnv uses a gym-oriented API ( reset , step , action , observations ) like OpenAI's Gymnasium . Also, OpenEnv uses a standard MCP tool call interface to connect to envs which provides a consistent interface across domains and simulation to production environments. The environments maintain state across multiple actions—enabling long-horizon reasoning—and can connect directly to real APIs and tools such as browsers, code repositories, or calendars. This shifts evaluation from "Can this work in a controlled demo?" to "Can this operate reliably in the real world?" The Calendar Gym: A Production-Grade Benchmark Calendar systems are deceptively complex. While scheduling a meeting seems simple, real-world calendar management requires agents to reason over time, permissions, multiple users, and incomplete information—often across several dependent steps. These properties make calendars a powerful testbed for evaluating tool-using agents outside controlled simulations. To ground OpenEnv in this kind of realistic, demanding use case, Turing built a production-grade calendar management environment referred to as the Calendar Gym . Rather than simulating scheduling in the abstract, it exposes agents to the same constraints they would face in real calendar systems: Access Control Lists across users and calendars, limited visibility into other users' state, and multi-step workflows where actions must be chained in the correct order. Agents interact with a rich set of calendar operations—from listing calendars to modifying events and permissions—and must handle failed actions, incorrect assumptions, and missing permissions. Each session runs in an isolated environment, enabling reliable comparisons across runs. Below is a code example of how to use the Calendar Gym. We explore the environment, discover available tools, list calendars, create an event, and print the result. from openenv_wrapper.client import MCPEnvClient from openenv_wrapper.data_models import MCPAction with MCPEnvClient.from_hub(base_url= "TuringEnterprises/calendar-gym" ) as client: # Connect and reset the environment result = client.reset() print ( "Reset successful:" , result.observation.success) # Discover available tools result = client.step(MCPAction(action_type= "ListToolsAction" )) print ( "Available tools:" , len (result.observation.tools_list)) # List calendars result = client.step(MCPAction( action_type= "ToolCallAction" , tool_name= "calendars_list" , arguments={} )) calendars = result.observation.tool_result[ "items" ] print ( "Calendars:" , calendars) # Create an event result = client.step(MCPAction( action_type= "ToolCallAction" , tool_name= "events_insert" , arguments={ "calendarId" : "primary" , "summary" : "Team Sync" , "start" : { "dateTime" : "2026-01-15T14:00:00Z" }, "end" : { "dateTime" : "2026-01-15T15:00:00Z" } } )) print ( "Event created:" , result.observation.success) Below is an excerpt of what the Calendar Gym returns when you call ListToolsAction . Each entry includes the tool name plus an input schema (what arguments the tool accepts). Click to expand output { "tools_list" : [ { "name" : "calendars_list" , "description" : "List calendars visible to the current user." , "input_schema" : { "type" : "object" , "properties" : { } , "additionalProperties" : false } } , { "name" : "events_insert" , "description" : "Create an event in a calendar." , "input_schema" : { "type" : "object" , "properties" : { "calendarId" : { "type" : "string" } , "summary" : { "type" : "string" } , "start" : { "type" : "object" , "properties" : { "dateTime" : { "type" : "string" } } , "required" : [ "dateTime" ] } , "end" : { "type" : "object" , "properties" : { "dateTime" : { "type" : "string" } } , "required" : [ "dateTime" ] } } , "required" : [ "calendarId" , "summary" , "start" , "end" ] } } ] } What We Learned Evaluating agents in the Calendar Gym revealed consistent patterns which were common across multiple domains. While agents often perform well on individual game like actions, reliability breaks down as tasks become longer, more ambiguous, and more constrained. Multi-step reasoning is the primary bottleneck. Agents struggle to correctly chain actions across longer workflows, suggesting that benchmarks need to test sustained reasoning over multiple dependent steps—not just single tool calls. Ambiguity significantly degrades performance. Agents achieved close to 90% success on tasks with explicit calendar identifiers, but success dropped to roughly 40% when the same tasks were phrased using natural language descriptions. Building stronger lookup and validation into agent loops—rather than relying on the LLM to resolve references unaided—appears essential. Correct tool choice isn't enough. Across failed interactions, more than half of errors stemmed from malformed tool arguments or incorrect ordering, even when the right tool was selected. Reliable agent behavior depends as much on execution quality and structured feedback as on tool selection—environment design matters. These challenges are not unique to scheduling and calendars. They reflect broader limitations that emerge whenever agents operate in changing systems over long periods of time, and they point toward evaluation frameworks that test permissions, partial observability, and multi-step workflows together. Looking Ahead OpenEnv provides a foundation for testing agents under realistic conditions, and the Calendar Gym demonstrates how seemingly simple domains can surface deep challenges in reasoning, ambiguity resolution, and tool use. By evaluating agents where failure is measurable and constraints are real, we gain clearer insight into what it takes to build agents that operate reliably in production. For a deeper dive into the Calendar Gym's design, benchmarking methodology, and quantitative results, explore the full technical article on Turing's site . To explore a clone of the Calendar Gym, visit the Calendar Gym space . Appendix: Common error cases in tool use In practice, tool integrations rarely fail in dramatic ways; they fail in small, predictable ones. When wiring up MCP tools to real APIs (like calendar operations), we encountered a handful of recurring issues. Specific error cases found in the wild Below are three common failure modes we’ve seen in production, along with representative error payloads and mitigation strategies. These examples illustrate not just what can go wrong, but how structured errors can help agents recover gracefully. 1. Schema validation errors (missing or malformed arguments) The agent calls a valid tool (e.g. events_insert ), but the arguments do not match the declared JSON schema. Missing required fields like calendarId Incorrect nesting of start / end Passing a string where an object is expected. Click to expand error payload { "ok" : false , "error_type" : "validation_error" , "tool_name" : "events_insert" , "message" : "Invalid arguments for tool 'events_insert'." , "details" : { "missing_required_fields" : [ "calendarId" , "end" ] , "invalid_fields" : [ { "field" : "start" , "expected_type" : "object" , "received_type" : "string" } ] } } We can mitigate this by providing one canonical example of a correct 'events_insert' call in your prompt. Return structured validation errors so the model can repair and retry instead of failing silently. 2. Permission / authorization errors (401/403) The tool call is syntactically correct, but the API rejects it due to insufficient permissions. Missing OAuth scopes Expired access token User lacks write access to the target calendar Click to expand error payload { "ok" : false , "error_type" : "permission_error" , "tool_name" : "events_insert" , "http_status" : 403 , "message" : "The authenticated user does not have write access to calendar 'primary'." , "remediation" : [ "Ensure the OAuth token includes calendar write scope." , "Verify the user has edit access to the target calendar." , "Reconnect the integration if the token has expired." ] } We can mitigate this by clearly documenting the required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user instead of retrying the same failing call. Clearly document required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user instead of retrying the same failing call. 3. Datetime / format errors (RFC3339 & timezone issues) The event is rejected by the API, or it is created at an unexpected time. Missing timezone offset Non-RFC3339 datetime format Incorrect nesting of start.dateTime or end.dateTime Mixing local time and UTC without specifying an offset Click to expand error payload { "ok" : false , "error_type" : "format_error" , "tool_name" : "events_insert" , "message" : "Invalid datetime format for field 'start.dateTime'." , "details" : { "received" : "02/11/2026 9:30 AM" , "expected_format" : "RFC3339 (e.g. 2026-02-11T09:30:00-05:00)" } } We can mitigate this by standardizing on RFC3339 with explicit timezone offsets (e.g. 2026-02-11T09:30:00-05:00). Include at least one correct datetime example in your documentation to anchor model behavior and reduce repair retries.
Anthropic推荐指数8282 Anthropic 评估并缓解 LLM 发现 0-day 漏洞的风险
Anthropic 前沿红队发布研究，称 Claude Opus 4.6 无需专用脚手架或定制提示即可在长期运行模糊测试的代码库中发现高危漏洞，并已找到并验证超过 500 个高危漏洞，开始向开源维护者报告且部分补丁已合入。
Hugging Face Blog Community Evals: Because we're done trusting black-box leaderboards over the community
TL;DR: Benchmark datasets on Hugging Face can now host leaderboards. Models store their own eval scores. Everything links together. The community can submit results via PR. Verified badges prove that the results can be reproduced. Evaluation is broken Let's be real about where we are with evals in 2026. MMLU is saturated above 91%. GSM8K hit 94%+. HumanEval is conquered. Yet some models that ace benchmarks still can't reliably browse the web, write production code, or handle multi-step tasks without hallucinating, based on usage reports. There is a clear gap between benchmark scores and real-world performance. Furthermore, there is another gap within reported benchmark scores. Multiple sources report different results. From Model Cards, to papers, to evaluation platforms, there is no alignment in reported scores. The result is that the community lacks a single source of truth. What We're Shipping Decentralized and transparent evaluation reporting. We are going to take evaluations on the Hugging Face Hub in a new direction by decentralizing reporting and allowing the entire community to openly report scores for benchmarks. At first, we will start with a shortlist of 4 benchmarks and over time we’ll expand to the most relevant benchmarks. For Benchmarks: Dataset repos can now register as benchmarks ( MMLU-Pro , GPQA , HLE are already live). They automatically aggregate reported results from across the Hub and display leaderboards in the dataset card. The benchmark defines the eval spec via eval.yaml , based on the Inspect AI format, so anyone can reproduce it. The reported results need to align with the task definition. For Models: Eval scores live in .eval_results/*.yaml in the model repo. They appear on the model card and are fed into benchmark datasets. Both the model author’s results and open pull requests for results will be aggregated. Model authors will be able to close score PR and hide results. For the Community: Any user can submit evaluation results for any model via a PR. Results get shown as "community", without waiting for model authors to merge or close. The community can link to sources like a paper, Model Card, third-party evaluation platform, or inspect eval logs. The community can discuss scores like any PR. Since the Hub is Git based, there is a history of when evals were added, when changes were made, etc. The sources look like below. To learn more about evaluation results, check out the docs . Model scores in the Hub Why This Matters Decentralizing evaluation will expose scores that already exist across the community in sources like model cards and papers. By exposing these scores, the community can build on top of them to aggregate, track, and understand scores across the field. Also, all scores will be exposed via Hub APIs, making it easy to aggregate and build curated leaderboards, dashboards, etc. Community evals do not replace benchmarks so leaderboards and closed evals with published results are still crucial. However, we believe it's important to contribute to the field with open eval results based on reproducible eval specs. This won't solve benchmark saturation or close the benchmark-reality gap. Nor will it stop training on test sets. But it makes the game visible by exposing what is evaluated, how, when, and by whom. Mostly, we hope to make the Hub an active place to build and share reproducible benchmarks. Particularly focusing on new tasks and domains that challenge SOTA models more. Get Started Read the docs: To learn more about evaluation results, check out the docs . Add eval results: Publish the evals you conducted as YAML files in .eval_results/ on any model repo. Check out the scores on the benchmark dataset . Register a new benchmark: Add eval.yaml to your dataset repo and contact us to be included in the shortlist. The feature is in beta. We're building in the open. Feedback welcome.
腾讯玄武实验室精选推荐指数8686 腾讯玄武实验室发布《AI网络爬虫安全白皮书》并开源SEChrome防护方案
腾讯玄武实验室发布《AI网络爬虫安全白皮书》，系统分析AI时代服务端浏览器/爬虫的攻击面，指出多款产品存在沙箱外远程代码执行（RCE）风险，相关产品服务端用户规模累计超10亿，涵盖多家头部科技企业。
推荐理由：白皮书梳理了服务端浏览器的完整攻击链与四个真实RCE案例，并给出可复用的纵深防御框架和开源方案，工程落地参考价值高。
Hugging Face Blog Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs
Arabic is one of the most widely spoken languages in the world, with hundreds of millions of speakers across more than twenty countries. Despite this global reach, Arabic is not a monolithic language. Modern Standard Arabic coexists with a rich landscape of regional dialects that differ significantly in vocabulary, syntax, phonology, and cultural grounding. These dialects are the primary medium of daily communication, oral storytelling, poetry, and social interaction. However, most existing benchmarks for Arabic large language models focus almost exclusively on Modern Standard Arabic, leaving dialectal Arabic largely under-evaluated and under-represented. This gap is particularly problematic as large language models increasingly interact with users in informal, culturally grounded, and conversational settings. A model that performs well on formal newswire text may still fail to understand a greeting, an idiomatic expression, or a short anecdote expressed in a local dialect. To address this limitation, our team introduces Alyah الياه (which means North Star ⭐️ in Emirati), an Emirati-centric benchmark designed to assess how well Arabic LLMs capture the linguistic, cultural, and pragmatic aspects of the Emirati dialect. Benchmark Motivation and Scope The Emirati dialect is deeply intertwined with local culture, heritage, and history. It appears in everyday greetings, oral poetry, proverbs, folk narratives, and expressions whose meanings cannot be inferred through literal translation alone. Our benchmark is intentionally designed to probe this depth. Rather than testing surface-level lexical knowledge, it challenges models on their ability to interpret culturally embedded meaning, pragmatic usage, and dialect-specific nuances. The benchmark covers a wide range of content, including common and uncommon local expressions, culturally grounded greetings, short anecdotes, heritage-related questions, and references to Emirati poetry. The goal is not only to measure correctness, but also to understand where models systematically succeed or fail when confronted with authentic Emirati language use. Dataset Structure Following further development and consolidation, the benchmark has been unified into a single dataset called Alyah . The final benchmark contains 1,173 samples , all collected manually from native Emirati speakers to ensure linguistic authenticity and cultural grounding. This manual curation step was essential to capture expressions, meanings, and usages that are rarely documented in written resources and are difficult to infer from Modern Standard Arabic alone. Each sample is formulated as a multiple-choice question with four candidate answers , exactly one of which is correct. Large language models were used to synthetically generate the distractor choices, after which they were reviewed to ensure plausibility and semantic closeness to the correct answer. To avoid positional bias during evaluation, the index of the correct answer follows a randomized distribution across the dataset. Below is the distribution of word count per query and candidate answers. Alyah spans a broad spectrum of linguistic and cultural phenomena in the Emirati dialect, ranging from everyday expressions to culturally sensitive and figurative language. The distribution across categories is summarized below. Category Number of Samples Difficulty Greetings & Daily Expressions 61 Easy Religious & Social Sensitivity 78 Medium Imagery & Figurative Meaning 121 Medium Etiquette & Values 173 Medium Poetry & Creative Expression 32 Difficult Historical & Heritage Knowledge 89 Difficult Language & Dialect 619 Difficult Below are examples of each category: This composition allows Alyah to jointly evaluate surface-level conversational fluency and deeper cultural, semantic, and pragmatic understanding, with a particular emphasis on dialect-specific language phenomena that remain challenging for current models. Model Evaluation Setup We evaluated a total of 54 language models , comprising 23 base models and 31 instruction-tuned models , spanning several architectural and training paradigms. These include Arabic-native LLMs such as Jais and Allam, multilingual models with strong Arabic support such as Qwen and LLaMA, and adapted or regionally specialized models such as Fanar and AceGPT. For each family, both base and instruction-tuned variants were evaluated in order to understand the impact of alignment and instruction tuning on dialectal performance. All models were evaluated under a consistent prompting and scoring protocol. Responses were assessed for semantic correctness and appropriateness with respect to Emirati usage, rather than literal overlap with a reference answer. This is particularly important for dialectal evaluation, where multiple valid phrasings may exist. For each question category, we estimated difficulty empirically based on model performance. Categories where most models struggled were labeled as harder, while those consistently answered correctly across model families were considered easier. This approach allows difficulty to emerge from observed behavior rather than from subjective annotation alone. Evaluation Results on Alyah (Emirati Dialect) We evaluate a broad set of contemporary Arabic and multilingual large language models on Alyah , using accuracy on multiple-choice questions as the primary metric. The evaluation covers 53 models in total, including 22 base models and 31 instruction-tuned models , spanning Arabic-native, multilingual, and regionally adapted systems. Below is a radar plot showing the performance of top models of different sizes per question category. These results are intended as reference measurements within the scope of Alyah, rather than absolute rankings across all Arabic benchmarks. Base Models Model Accuracy google/gemma-3-27b-pt 74.68 tiiuae/Falcon-H1-34B-Base 73.66 FreedomIntelligence/AceGPT-v2-32B 67.35 google/gemma-3-4b-pt 63.17 QCRI/Fanar-1-9B 62.75 tiiuae/Falcon-H1-7B-Base 60.78 meta-llama/Llama-3.1-8B 58.23 Qwen/Qwen3-14B-Base 57.29 inceptionai/jais-adapted-13b 56.01 Qwen/Qwen2.5-32B 53.03 FreedomIntelligence/AceGPT-13B 50.81 Qwen/Qwen2.5-72B 47.91 Qwen/Qwen2.5-14B 46.8 google/gemma-2-2b 41.86 tiiuae/Falcon3-7B-Base 41.43 Qwen/Qwen3-8B-Base 40.75 tiiuae/Falcon-H1-3B-Base 40.41 Qwen/Qwen2.5-7B 36.57 Qwen/Qwen2.5-3B 35.29 meta-llama/Llama-3.2-3B 35.12 inceptionai/jais-adapted-7b 33.5 Qwen/Qwen3-4B-Base 27.45 Instruction-Tuned Models Model Accuracy falcon-h1-arabic-7b-instruct 82.18 humain-ai/ALLaM-7B-Instruct-preview 77.24 google/gemma-3-27b-it 74.68 falcon-h1-arabic-3b-instruct 74.51 Qwen/Qwen2.5-72B-Instruct 74.6 CohereForAI/aya-expanse-32b 73.66 Navid-AI/Yehia-7B-preview 73.32 FreedomIntelligence/AceGPT-v2-32B-Chat 72.8 Qwen/Qwen2.5-32B-Instruct 71.61 tiiuae/Falcon-H1-34B-Instruct 71.1 meta-llama/Llama-3.3-70B-Instruct 69.74 QCRI/Fanar-1-9B-Instruct 69.22 tiiuae/Falcon-H1-7B-Instruct 65.13 CohereForAI/c4ai-command-r7b-arabic-02-2025 64.54 silma-ai/SILMA-9B-Instruct-v1.0 63.94 FreedomIntelligence/AceGPT-v2-8B-Chat 63.43 CohereLabs/aya-expanse-8b 61.21 yasserrmd/kallamni-2.6b-v1 61.13 yasserrmd/kallamni-4b-v1 60.7 microsoft/Phi-4-mini-instruct 58.57 tiiuae/Falcon-H1-3B-Instruct 57.12 silma-ai/SILMA-Kashif-2B-Instruct-v1.0 48.51 Qwen/Qwen2.5-7B-Instruct 45.44 google/gemma-3-4b-it 46.12 meta-llama/Llama-3.1-8B-Instruct 46.29 meta-llama/Llama-3.2-3B-Instruct 39.64 yasserrmd/kallamni-1.2b-v1 37.77 Qwen/Qwen3-4B 26.26 google/gemma-2-2b-it 26.00 Qwen/Qwen3-14B 26.00 Qwen/Qwen3-8B 25.66 Analysis and Observed Trends Figure 1: Models' accuracy across categories based on size. Figure 2: Models' accuracy across categories based on language. Several trends emerge from the evaluation. Instruction-tuned models generally outperform their base counterparts as shown in Figures 1 and 2. This is particularly the case on questions involving conversational norms and culturally appropriate responses (i.e. the Etiquette & Values Category). Furthermore, it is the case with questions that test imagery and figurative meaning. This can be attributed, to the model’s original strong capabilities with understanding MSA-based imagery and figurative language regardless of the dialect at hand. The models are able to draw patterns of non-literal description regardless of dialect. Generally, the most difficult categories for the models were consistently “Language and Dialect” and “Greeting and Daily expressions” across model sizes as shown in figure 1. These results reflect the current state of Emirati dialect presence in written media, as the dialect is mostly spoken rarely written, which explains its novelty relative to the evaluated models. Nonetheless, there is a clear benefit to instruct models with understanding the dialect (and the other evaluation categories) in comparison to their counterparts, especially in small and medium models. This is particularly noticeable with the Poetry and Creative Expression category, which is where the large instruct models performed marginally better than the smaller models. Figure 3: Evaluated models average accuracy. As shown in Figure 3, even strong multilingual models show notable degradation on the most challenging Alyah questions, suggesting that dialect-specific semantic knowledge is not easily acquired through generic multilingual training alone. It must be noted that while Arabic-native models tend to perform more robustly on culturally grounded content, their performances are not uniform across all categories (figure 2). In particular, questions involving implicit meanings and rare expressions remain difficult across nearly all evaluated models. This highlights a persistent gap between surface-level dialect familiarity and deeper cultural understanding. The high variance in performance across categories , where a model that excels at imagery and figurative meaning may still struggle with poetry or heritage-related creative questions, indicates that dialectal competence is multi-dimensional and cannot be captured by a single score. Figure 3 shows that the highest scoring large model in Jais-2-70B, followed by the two small models jais-2-8B and ALLaM-7B-instruct, which are all Arabic instruct-tuned models. Conclusion and Community Impact This benchmark represents a step toward more realistic and culturally grounded evaluation of Arabic language models. By focusing on the Emirati dialect, we aim to support the development of models that better serve local communities, institutions, and users in the UAE. Beyond model ranking, the benchmark is intended as a diagnostic tool to guide future data collection, training, and adaptation efforts. We invite researchers, practitioners, and the broader community to use the benchmark, explore the results, and share feedback. Community input will be essential to refining the dataset, expanding coverage, and ensuring that dialectal Arabic receives the attention it deserves in the evaluation of Large Language Models. Citation @misc{emirati_dialect_benchmark_2026, title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models}, author={Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid}, year = {2026}, month = {january}, }
Hugging Face Blog AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
AssetOpsBench is a comprehensive benchmark and evaluation system with six qualitative dimensions that bridges the gap for agentic AI in domain-specific settings, starting with industrial Asset Lifecycle Management. Introduction While existing AI benchmarks excel at isolated tasks such as coding or web navigation, they often fail to capture the complexity of real-world industrial operations. To bridge this gap, we introduce AssetOpsBench , a framework specifically designed to evaluate agent performance across six critical dimensions of industrial applications. Unlike traditional benchmarks, AssetOpsBench emphasizes the need for multi-agent coordination—moving beyond `lone wolf' models to systems that can handle complex failure modes, integrate multiple data streams, and manage intricate work orders. By focusing on these high-stakes, multi-agent dynamics, the benchmark ensures that AI agents are assessed on their ability to navigate the nuances and safety-critical demands of a true industrial environment. AssetOpsBench is built for asset operations such as chillers and air handling units. It comprises: 2.3M sensor telemetry points 140+ curated scenarios across 4 agents 4.2K work orders for diverse scenarios 53 structured failure modes Experts helped curate 150+ scenarios. Each scenario includes metadata: task type, output format, category, and sub-agents. The tasks designed span across: Anomaly detection in sensor streams Failure mode reasoning and diagnostics KPI forecasting and analysis Work order summarization and prioritization Evaluation Framework and Overall Feedback AssetOpsBench evaluates agentic systems across six qualitative dimensions designed to reflect real operational constraints in industrial asset management. Rather than optimizing for a single success metric, the benchmark emphasizes decision trace quality, evidence grounding, failure awareness, and actionability under incomplete and noisy data. Each agent run is scored across six criteria: Task Completion Retrieval Accuracy Result Verification Sequence Correctness Clarity and Justification Hallucination rate Across early evaluations, we observe that many general-purpose agents perform well on surface-level reasoning but struggle with sustained multi-step coordination involving work orders, failure semantics, and temporal dependencies. Agents that explicitly model operational context and uncertainty tend to produce more stable and interpretable trajectories, even when final task completion is partial. This feedback-oriented evaluation is intentional: in industrial settings, understanding why an agent fails is often more valuable than a binary success signal. Failure Modes in Industrial Agentic Workflows A central contribution of AssetOpsBench is the explicit treatment of failure modes as first-class evaluation signals in agentic industrial workflows. Rather than treating failure as a binary outcome, AssetOpsBench analyzes full multi-agent execution trajectories to identify where , how , and why agent behavior breaks down under realistic operational constraints. Failure analysis in AssetOpsBench is implemented through a dedicated trajectory-level pipeline ( TrajFM ), which combines LLM-based reasoning with statistical clustering to surface interpretable failure patterns from agent execution traces. This pipeline operates in three stages: (1) trajectory-level failure extraction using an LLM-guided diagnostic prompt, (2) embedding-based clustering to group recurring failure patterns, and (3) analysis and visualization to support developer feedback and iteration. Across industrial scenarios, recurrent failure modes include: Misalignment between sensor telemetry, alerts, and historical work orders Overconfident conclusions drawn under missing, delayed, or insufficient evidence Inconsistent aggregation of heterogeneous data modalities across agents Premature action selection without adequate verification or validation steps Breakdowns in multi-agent coordination, such as ignored inputs or action–reasoning mismatches Importantly, AssetOpsBench does not rely solely on a fixed, hand-crafted failure taxonomy. While a structured set of predefined failure categories (e.g., verification errors, step repetition, role violations) is used for consistency, the system is explicitly designed to discover new failure patterns that emerge in practice. Additional failure modes identified by the LLM are embedded and clustered automatically, allowing the taxonomy to evolve as new agent designs and behaviors are evaluated. To preserve industrial confidentiality, raw execution traces are never exposed. Instead, agents receive aggregated scores across six evaluation dimensions together with clustered failure-mode summaries that explain why an agent failed, without revealing sensitive data or intermediate reasoning steps. This feedback-driven design enables developers to diagnose weaknesses, refine agent workflows, and iteratively resubmit improved agents. This failure-aware evaluation reflects the realities of industrial asset management, where cautious, degradation-aware reasoning—and the ability to recognize uncertainty, defer action, or escalate appropriately—is often preferable to aggressive but brittle automation. Submit an Agent for Evaluation AssetOpsBench-Live is designed as an open, competition-ready benchmark , and we welcome submissions of agent implementations from the community. Agents are evaluated in a controlled, privacy-preserving environment that reflects real industrial asset management constraints. To submit an agent, developers first validate their implementation locally using a provided simulated environment, which includes representative sensor data, work orders, alerts, and failure-mode catalogs. Agents are then containerized and submitted for remote execution on hidden evaluation scenarios. Submitted agents are evaluated across six qualitative dimensions—task completion, accuracy, result verification, action sequencing, clarity, and hallucination—using a consistent, reproducible evaluation protocol. Execution traces are not exposed; instead, participants receive aggregated scores and structured failure-mode feedback that highlights where and why an agent’s reasoning or coordination broke down. This feedback-driven evaluation loop enables iterative improvement: developers can diagnose failure patterns, refine agent design or workflow structure, and resubmit updated agents for further evaluation. Both planning-focused and execution-focused agents are supported, allowing researchers and practitioners to explore diverse agentic designs within the same benchmark framework. Experiment and Observations We performed a community evaluation where we tested two tracks: Planning-oriented multi-agent orchestration Execution-oriented dynamic multi-agent workflow. Across 225 users and 300+ agents and leading open source models, here are the observations: Model Family Best Planning Score Best Execution Score Key Limitation GPT-4.1 68.2 72.4 Hallucinated completion on complex workflows Mistral-Large 64.7 69.1 Struggled with multi-hop tool sequences LLaMA-4 Maverick 66.0 70.8 Missed clarifying questions (fixable) LLaMA-3-70B 52.3 58.9 Collapsed under multi-agent coordination Note: None of the models could pass our evaluation criteria benchmark and get 85 points, which is the threshold for deployment readiness. Distribution of Failures Across 881 agent execution traces, failure distribution was as follows: Ineffective Error Recovery: 31.2% Overstated Completion: 23.8% Formatting Issues: 21.4% Unhandled Tool Errors: 10.3% Ignored Feedback: 8.0% Other: 5.3% Beyond this, 185 traces had one new failure pattern and 164 had multiple novel failures. Key Error Findings "Sounds Right, Is Wrong": Agents claim to have completed tasks (23.8%) and output success even after unsuccessful failure recovery (31.2%). AssetOps benchmarking is important to uncover this so that operators do not act upon incorrect information. Tool Usage: This is the biggest differentiator between high and low performing agents, with top agents having 94% tool accuracy compared to 61% of low performers. Multi-agent Multiplies Failures: Task accuracy between single agent (68%) vs multi-agent (47%) shows the complexity multi-agent brings with context loss, asynchronous issues, and cascaded failures. Domain Knowledge: Agents with access to failure mode databases and maintenance manuals performed better. However, RAG knowledge wasn’t always used correctly, suggesting a need for structured reasoning. Ambiguity: Missing sensors, conflicting logs, and vague operator descriptions caused the success rate to drop 34%. Agents must have clarification strategies embedded. Where to get started? Read our technical report AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance How to run AssetOpsBench locally - Video AssetOpsBench Local Execution Try out AssetOpsBench in the HuggingFace Space Playground Find More Detail AssetOpsBench GitHub , fork the repo and get started.
Anthropic精选推荐指数7272 Anthropic 红队：Claude Sonnet 4.5 在真实网络靶场中自主利用公开 CVE
Anthropic 前沿红队与 Incalmo 合作，在 25-50 台主机的模拟网络靶场上评测 Claude，发现 Claude Sonnet 4.5 在少数网络上无需此前几代所需的定制网络工具包即可成功。
推荐理由：Anthropic 红队披露 Claude Sonnet 4.5 在真实网络靶场中无需专用工具即可自主利用公开 CVE，并给出与上一代的对照数据。
Anthropic推荐指数4848 Anthropic 用 Claude 与基于属性的测试在 Python 生态中发现数百个 bug
Anthropic 开发了一个基于 Claude Code 自定义命令的 AI agent，通过推断代码应满足的通用属性并用 Hypothesis 编写基于属性的测试，在 NumPy、SciPy、Pandas 等热门 Python 包中发现数百个潜在 bug。
Anthropic推荐指数5858 Anthropic 与 PNNL 用 Claude 模拟攻击水处理厂，攻击链重建从数周缩短至三小时
Anthropic 与 Pacific Northwest National Laboratory（PNNL）合作，用 Claude 对水处理厂的高保真仿真环境进行对手模拟，PNNL 估计攻击链重建从数周缩短到三小时。
腾讯玄武实验室精选推荐指数6262 腾讯玄武实验室披露 ComfyUI-Manager 远程代码执行漏洞 CVE-2025-67303
腾讯玄武实验室发现 ComfyUI 官方扩展组件 ComfyUI-Manager 存在高危漏洞 CVE-2025-67303，远程攻击者可在无需账号的情况下实现远程代码执行并完全控制服务器，包括盗取用户数据和私有 AI 模型。
推荐理由：通告给出了漏洞根因、影响版本与迁移步骤，读者可据此判断自身 ComfyUI 部署是否受影响并完成修复。
Hugging Face Blog推荐指数7676 ServiceNow 发布 8B 安全护栏模型 AprielGuard，覆盖 Agent 工作流攻击检测
ServiceNow-AI 在 Hugging Face 发布 AprielGuard，一个 8B 参数的安全护栏模型，用于检测 16 类安全风险（涵盖毒性、仇恨、虚假信息、自残、非法活动等）以及提示词注入、越狱、思维链污染、上下文劫持、记忆投毒和多智能体利用链等对抗攻击，同时支持对 Agent 工作流中的工具调用与推理轨迹进行检测。
OWASP推荐指数5757 AIBOM Generator 开源工具正式捐赠给 OWASP GenAI 安全项目
AIBOM Generator 是一款可为 Hugging Face 模型生成 AI 软件物料清单（AIBOM）的开源工具，最初在 RSAC 2025 发布，现已列入 CycloneDX Tool Center。
Anthropic推荐指数5555 Anthropic 公布 Project Vend 第二阶段：Claude 经营小店实验
Anthropic 公布 Project Vend 第二阶段结果，Claudius 从 Claude Sonnet 3.7 升级到 Sonnet 4.0 和 4.5，并接入 CRM、库存管理、网页浏览等工具，还新增 CEO 智能体 Seymour Cash 和周边制作智能体 Clothius，业务表现明显改善，亏损周基本消除，门店扩展到旧金山、纽约和伦敦三地。
Hugging Face Blog The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator
It has become increasingly challenging to assess whether a model’s reported improvements reflect genuine advances or variations in evaluation conditions, dataset composition, or training data that mirrors benchmark tasks. The NVIDIA Nemotron approach to openness addresses this by publishing transparent and reproducible evaluation recipes that make results independently verifiable. NVIDIA released Nemotron 3 Nano 30B A3B with an explicitly open evaluation approach to make that distinction clear. Alongside the model card, we are publishing the complete evaluation recipe used to generate the results, built with the NVIDIA NeMo Evaluator library, so anyone can rerun the evaluation pipeline, inspect the artifacts, and analyze the outcomes independently. We believe that open innovation is the foundation of AI progress. This level of transparency matters because most model evaluations omit critical details. Configs, prompts, harness versions, runtime settings, and logs are often missing or underspecified, and even small differences in these parameters can materially change results. Without a complete recipe, it’s nearly impossible to tell whether a model is genuinely more intelligent or simply optimized for a benchmark. This blog shows developers exactly how to reproduce the evaluation behind Nemotron 3 Nano 30B A3B using fully open tools, configurations, and artifacts. You’ll learn how the evaluation was run, why the methodology matters, and how to execute the same end-to-end workflow using the NeMo Evaluator library so you can verify results, compare models consistently, and build transparent evaluation pipelines of your own. Building a consistent and transparent evaluation workflow with NeMo Evaluator A single, consistent evaluation system Developers and researchers need evaluation workflows they can rely on, not one-off scripts that behave differently from model to model. NeMo Evaluator provides a unified way to define benchmarks, prompts, configuration, and runtime behavior once, then reuse that methodology across models and releases. This avoids the common scenario where the evaluation setup quietly changes between runs, making comparisons over time difficult or misleading. Methodology independent of inference setup Model outputs can vary by inference backend and configuration, so evaluation tools should never be tied to a single inference solution. Locking an evaluation tool to one inference solution would limit its usefulness. NeMo Evaluator avoids this by separating the evaluation pipeline from the inference backend, allowing the same configuration to run against hosted endpoints, local deployments, or third-party providers. This separation enables meaningful comparisons even when you change infrastructure or inference engines. Built to scale beyond one-off experiments Many evaluation pipelines work once and then break down as the scope expands. NeMo Evaluator is designed to scale from quick, single-benchmark validation to full model card suites and repeated evaluations across multiple models. The launcher, artifact layout, and configuration model support ongoing workflows, not just isolated experiments, so teams can maintain consistent evaluation practices over time. Auditability with structured artifacts and logs Transparent evaluation requires more than final scores. Each evaluation run produces structured results and logs by default, making it easy to inspect how scores were computed, understand score calculations, debug unexpected behavior, and conduct deeper analysis. Each component of the evaluation is captured and reproducible. A shared evaluation standard By releasing Nemotron 3 Nano 30B A3B with its full evaluation recipe , NVIDIA is providing a reference methodology that the community can run, inspect, and build upon. Using the same configuration and tools brings consistency to how benchmarks are selected, executed, and interpreted, enabling more reliable comparisons across models, providers, and releases. Open evaluation for Nemotron 3 Nano Open evaluation means publishing not just the final results, but the full methodology behind them, so benchmarks are run consistently, and results can be compared meaningfully over time. For Nemotron 3 Nano 30B A3B , this includes open‑source tooling, transparent configurations, and reproducible artifacts that anyone can run end‑to‑end. Open-source model evaluation tooling NeMo Evaluator is an open-source library designed for robust, reproducible, and scalable evaluation of generative models. Instead of introducing yet another standalone benchmark runner, it acts as a unifying orchestration layer that brings multiple evaluation harnesses under a single, consistent interface. Under this architecture, NeMo Evaluator integrates and coordinates hundreds of benchmarks from many widely used evaluation harnesses, including NeMo Skills for Nemotron instruction-following, tool use, and agentic evaluations, as well as the LM Evaluation Harness for base model and pre-training benchmarks, and many more ( full benchmark catalog ). Each harness retains its native logic, datasets, and scoring semantics, while NeMo Evaluator standardizes how they are configured, executed, and logged. This provides two practical advantages: teams can run diverse benchmark categories using a single configuration without rewriting custom evaluation scripts, and results from different harnesses are stored and inspected in a consistent, predictable way, even when the underlying tasks differ. The same orchestration framework used internally by NVIDIA’s Nemotron research and model‑evaluation teams is now available to the community, enabling developers to run heterogeneous, multi‑harness evaluations through a shared, auditable workflow. Open configurations We published the exact YAML configuration used for the Nemotron 3 Nano 30B A3B model card evaluation with NeMo Evaluator. This includes: model inference and deployment settings benchmark and task selection benchmark-specific parameters such as sampling, repeats, and prompt templates runtime controls including parallelism, timeouts, and retries output paths and artifact layout Using the same configuration means running the same evaluation methodology. Open logs and artifacts Each evaluation run produces structured, inspectable outputs, including per‑task results.json files, execution logs for debugging and auditability, and artifacts organized by task for easy comparison. This structure makes it possible to understand not only the final scores, but also how those scores were produced and to perform deeper analysis of model behavior. The reproducibility workflow Reproducing Nemotron 3 Nano 30B A3B model card results follows a simple loop: Start from the released model checkpoint or hosted endpoint Use the published NeMo Evaluator config Execute the evaluation with a single CLI command Inspect logs and artifacts, and compare results to the model card The same workflow applies to any model you evaluate using NeMo Evaluator. You can point the evaluation at a hosted endpoint or a local deployment, including common inference providers such as HuggingFace , build.nvidia.com , and OpenRouter . The key requirement is access to the model, either as weights you can serve or as an endpoint you can call. For this tutorial, we use the hosted endpoint on build.nvidia.com . Reproducing Nemotron 3 Nano benchmark results This tutorial reproduces the evaluation results for NVIDIA Nemotron 3 Nano 30B A3B using NeMo Evaluator. The step-by-step tutorial, including the published configs used for the model card evaluation , is available on GitHub. Although we have focused this tutorial on the Nemotron 3 Nano 30B A3B, we also published recipes for the base model evaluation . This walkthrough runs a comprehensive evaluation suite of the published configs used for the model card evaluation for NVIDIA Nemotron 3 Nano 30B A3B using the following benchmarks: Benchmark Accuracy Category Description BFCL v4 53.8 Function Calling Berkeley Function Calling Leaderboard v4 LiveCodeBench (v6 2025-08–2025-05) 68.3 Coding Real-world coding problems evaluation MMLU-Pro 78.3 Knowledge Multi-task language understanding (10-choice) GPQA 73.0 Science Graduate-level science questions AIME 2025 89.1 Mathematics American Invitational Mathematics Exam SciCode 33.3 Scientific Coding Scientific programming challenges IFBench 71.5 Instruction Following Instruction following benchmark HLE 10.6 Humanity's Last Exam Expert-level questions across domains For Model Card details, see the NVIDIA Nemotron 3 Nano 30B A3B Model Card . For a deep dive into the architecture, datasets, and benchmarks, read the full Nemotron 3 Nano Technical Report . 1. Install NeMo Evaluator Launcher pip install nemo-evaluator-launcher 2. Set required environment variables # NVIDIA endpoint access export NGC_API_KEY="your-ngc-api-key" # Hugging Face access export HF_TOKEN="your-huggingface-token" # Required only for judge-based benchmarks such as HLE export JUDGE_API_KEY="your-judge-api-key" Optional but recommended for faster reruns: export HF_HOME="/path/to/your/huggingface/cache" 3. Model endpoint The evaluation uses the NVIDIA API endpoint hosted on build.nvidia.com : target: api_endpoint: model_id: nvidia/nemotron-nano-3-30b-a3b url: https://integrate.api.nvidia.com/v1/chat/completions api_key_name: NGC_API_KEY Evaluations can be run against common inference providers such as HuggingFace , build.nvidia.com , or OpenRouter , or anywhere that the model has an available endpoint. If you're hosting the model locally or using a different endpoint: nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o target.api_endpoint.url=http://localhost:8000/v1/chat/completions 4. Run the full evaluation suite Preview the run without executing using --dry-run : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ --dry-run From the examples directory, run the evaluation using the YAML configuration provided: nemo-evaluator-launcher run \ --config /path/to/examples/nemotron/local_nvidia_nemotron_3_nano_30b_a3b.yaml Note that for quick testing, you can limit the number of samples by setting limit_samples : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 5. Running an individual benchmark You can run specific benchmarks using the -t flag (from the examples/nemotron directory): # Run only MMLU-Pro nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_mmlu_pro # Run only coding benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_livecodebench # Run multiple specific benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_gpqa -t ns_aime2025 6. Monitor execution and inspect results # Check status of a specific job nemo-evaluator-launcher status # Stream logs for a specific job nemo-evaluator-launcher logs <job-id> Results are written to the defined output directory: results_nvidia_nemotron_3_nano_30b_a3b/ ├── artifacts/ │ └── <task_name>/ │ └── results.json └── logs/ └── stdout.log Interpreting results When reproducing evaluations, you may observe small differences in final scores across runs. This variance reflects the probabilistic nature of LLMs rather than an issue with the evaluation pipeline. Modern evaluation introduces several sources of non‑determinism: decoding settings, repeated trials, judge‑based scoring, parallel execution, and differences in serving infrastructure. All of which can lead to slight fluctuations. The purpose of open evaluation is not to force bit-wise identical outputs, but to deliver methodological consistency with clear provenance of evaluation results. To ensure your evaluation aligns with the reference standard, verify the following: Configuration : use the published NeMo Evaluator YAML without modification, or document any changes explicitly Benchmark selection : run the intended tasks, task versions, and prompt templates Inference target : verify you are evaluating the intended model and endpoint, including chat template behavior and reasoning settings when relevant Execution settings : keep runtime parameters consistent, including repeats, parallelism, timeouts, and retry behavior Outputs : confirm artifacts and logs are complete and follow the expected structure for each task When these elements are consistent, your results represent a valid reproduction of the methodology, even if individual runs differ slightly. NeMo Evaluator simplifies this process, tying benchmark definitions, prompts, runtime settings, and inference configuration into a single auditable workflow to minimize inconsistencies. Conclusion: A more transparent standard for open models The evaluation recipe released alongside Nemotron 3 Nano represents a meaningful step toward a more transparent and reliable approach to open-model evaluation. We are moving away from evaluation as a collection of bespoke, "black box" scripts, and towards a defined system where benchmark selection, prompts, and execution semantics are encoded into a transparent workflow. For developers and researchers, this transparency changes what it means to share results. A score is only as trustworthy as the methodology behind it and making that methodology public is what enables the community to verify claims, compare models fairly, and continue building on shared foundations. With open evaluation configurations, open artifacts, and open tooling, Nemotron 3 Nano demonstrates what that commitment to openness looks like in practice. NeMo Evaluator supports this shift by providing a consistent benchmarking methodology across models, releases, and inference environments. The objective isn’t identical numbers on every run; it’s confidence in an evaluation methodology that is explicit, inspectable, and repeatable. And for organizations that need automated or large‑scale evaluation pipelines, a separate microservice offering provides an enterprise‑ready NeMo Evaluator microservice built on the same evaluation principles. Use the published NeMo Evaluator evaluation configuration for an end-to-end walkthrough of the evaluation recipe. Join the Community! NeMo Evaluator is fully open source, and community input is essential to shaping the future of open evaluation. If there’s a benchmark you’d like us to support or an improvement you want to propose, open an issue, or contribute directly on GitHub. Your contributions help strengthen the ecosystem and advance a shared, transparent standard for evaluating generative models.
Anthropic精选推荐指数8282 Anthropic 红队发布 SCONE-bench：AI 智能体在模拟中挖出价值 460 万美元的智能合约漏洞
Anthropic 前沿红队发布 SCONE-bench 基准，包含 2020 至 2025 年间真实被利用的 405 个智能合约，用于评估 AI 智能体的漏洞利用能力。
推荐理由：Anthropic 红队用真实被黑合约构建基准，把智能体攻击能力换算成美元损失，并给出成本与收益的量化对照。
Hugging Face Blog Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks
While everyone (and their grandma 👵) is spinning up new ASR models, picking the right one for your use case can feel more overwhelming than choosing your next Netflix show. As of 21 Nov 2025, there are 150 Audio-Text-to-Text and 27K ASR models on the Hub 🤯 Most benchmarks focus on short-form English transcription (<30s), and overlook other important tasks, such as (1) multilingual performance and (2) model throughput, which can a be deciding factor for long-form audio like meetings and podcasts. Over the past two years, the Open ASR Leaderboard has become a standard for comparing open and closed-source models on both accuracy and efficiency . Recently, multilingual and long-form transcription tracks have been added to the leaderboard 🎉 TL;DR - Open ASR Leaderboard 📝 New preprint on ASR trends from the leaderboard: https://hf.co/papers/2510.06961 🧠 Best accuracy: Conformer encoder + LLM decoders (open-source ftw 🥳) ⚡ Fastest: CTC / TDT decoders 🌍 Multilingual: Comes at the cost of single-language performance ⌛ Long-form: Closed-source systems still lead (for now 😉) 🧑💻 Fine-tuning guides ( Parakeet , Voxtral , Whisper ): to continue pushing performance Takeaways from 60+ models As of 21 Nov 2025, the Open ASR Leaderboard compares 60+ open and closed-source models from 18 organizations , across 11 datasets . In a recent preprint , we dive into the technical setup and highlight some key trends in modern ASR. Here are the big takeaways 👇 1. Conformer encoder 🤝 LLM decoder tops the charts 📈 Models combining Conformer encoders with large language model (LLM) decoders currently lead in English transcription accuracy. For example, NVIDIA’s Canary-Qwen-2.5B , IBM’s Granite-Speech-3.3-8B , and Microsoft’s Phi-4-Multimodal-Instruct achieve the lowest word error rates ( WER ), showing that integrating LLM reasoning can significantly boost ASR accuracy. 💡 Pro-tip: NVIDIA introduced Fast Conformer , a 2x faster variant of the Conformer, that is used in their Canary and Parakeet suite of models. 2. Speed–accuracy tradeoffs ⚖️ While highly accurate, these LLM decoders tend to be slower than simpler approaches. On the Open ASR Leaderboard , efficiency is measured using inverse real-time factor (RTFx), where higher is better. For even faster inference, CTC and TDT decoders deliver 10–100× faster throughput , albeit with slightly higher error rates. This makes them ideal for real-time , offline , or batch transcription tasks (such as meetings, lectures, or podcasts). 3. Multilingual 🌍 OpenAI’s Whisper Large v3 remains a strong multilingual baseline, supporting 99 languages . However, fine-tuned or distilled variants like Distil-Whisper and CrisperWhisper often outperform the original on English-only tasks, showing how targeted fine-tuning can improve specialization ( how to fine-tune? Check out guides for Whisper , Parakeet , and Voxtral ). That said, focusing on English tends to reduce multilingual coverage 👉 a classic case of the tradeoff between specialization and generalization. Similarly, while self-supervised systems like Meta’s Massively Multilingual Speech (MMS) and Omnilingual ASR can support 1K+ languages, they trail behind language-specific encoders in accuracy. ⭐ While just five languages are currently benchmarked, we’re planning to expand to more languages and are excited for new dataset and models contributions to multilingual ASR through GitHub pull requests . 🎯 Alongside multilingual benchmarks, several community-driven leaderboards focus on individual languages. For example, the Open Universal Arabic ASR Leaderboard compares models across Modern Standard Arabic and regional dialects , highlighting how speech variation and diglossia challenge current systems. Similarly. the Russian ASR Leaderboard provides a growing hub for evaluating encoder-decoder and CTC models on Russian-specific phonology and morphology . These localized efforts mirror the broader multilingual leaderboard’s mission to encourage dataset sharing, fine-tuned checkpoints, and transparent model comparisons , especially in languages with fewer established ASR resources. 4. Long-form transcription is a different game ⏳ For long-form audio (e.g., podcasts, lectures, meetings), closed-source systems still edge out open ones. It could be due to domain tuning, custom chunking, or production-grade optimization. Among open models, OpenAI’s Whisper Large v3 performs the best. But for throughput, CTC-based Conformers shine 👉 for example, NVIDIA’s Parakeet CTC 1.1B achieves an RTFx of 2793.75 , compared to 68.56 for Whisper Large v3, with only a moderate WER degradation ( 6.68 and 6.43 respectively). The tradeoff? Parakeet is English-only, again reminding us of that multilingual and specialization tradeoff 🫠. ⭐ While closed systems still lead, there’s huge potential for open-source innovation here. Long-form ASR remains one of the most exciting frontiers for the community to tackle next! 🎤 The Show Must Go On Given how fast ASR is evolving, we’re excited to see what new architectures push performance and efficiency, and how the Open ASR Leaderboard continues to serve as a transparent, community-driven benchmark for the field, and as a reference for other leaderboards ( Russian , Arabic , and Speech DeepFake Detection ). We’ll keep expanding the Open ASR LeaderBoard with more models, more languages, and more datasets so stay tuned 👀 👉 Want to contribute? Head on over to the GitHub repo to open a pull request 🚀
腾讯玄武实验室推荐指数5757 腾讯玄武实验室详解量子计算机距离攻破 RSA-2048 还有多远
腾讯玄武实验室发布长文，从量子计算原理、超导量子计算机构造到破解 RSA-2048 的工程瓶颈，系统分析量子计算机距离攻破经典公钥密码还有多远。文章指出科学层面已无死胡同，真正的挑战集中在冷却、控制、布线、能耗与量子纠错实时实现等工程问题；以表面码纠错计算，运行 Shor 算法需要百万级物理量子比特。
Anthropic推荐指数3838 Project Fetch：Claude 能否训练机器狗？
Anthropic 前沿红队开展 Project Fetch 实验，将 8 名非机器人专家的研究人员随机分为两组，一组可用 Claude、一组不可用，任务是编程四足机器狗取回沙滩球。
腾讯玄武实验室精选推荐指数7373 腾讯玄武实验室阿图因引擎发现 gnark 零知识证明库签名可锻造性漏洞 CVE-2025-57801
腾讯玄武实验室的阿图因自动化漏洞挖掘引擎在零知识证明库 gnark 中发现高危签名可锻造性漏洞 CVE-2025-57801（CVSS 8.6），并联合上海交通大学 GOSSIP 实验室及郁昱教授团队完成复现。
推荐理由：文章解释了漏洞虽存在多年却大多未被触发的原因，读者可据此自查闭源项目是否使用了受影响的 Native Verification API。