跳到正文
2月4日周三
  1. Hugging Face Blog

    Community Evals: Because we're done trusting black-box leaderboards over the community

    TL;DR: Benchmark datasets on Hugging Face can now host leaderboards. Models store their own eval scores. Everything links together. The community can submit results via PR. Verified badges prove that the results can be reproduced. Evaluation is broken Let's be real about where we are with evals in 2026. MMLU is saturated above 91%. GSM8K hit 94%+. HumanEval is conquered. Yet some models that ace benchmarks still can't reliably browse the web, write production code, or handle multi-step tasks without hallucinating, based on usage reports. There is a clear gap between benchmark scores and real-world performance. Furthermore, there is another gap within reported benchmark scores. Multiple sources report different results. From Model Cards, to papers, to evaluation platforms, there is no alignment in reported scores. The result is that the community lacks a single source of truth. What We're Shipping Decentralized and transparent evaluation reporting. We are going to take evaluations on the Hugging Face Hub in a new direction by decentralizing reporting and allowing the entire community to openly report scores for benchmarks. At first, we will start with a shortlist of 4 benchmarks and over time we’ll expand to the most relevant benchmarks. For Benchmarks: Dataset repos can now register as benchmarks ( MMLU-Pro , GPQA , HLE are already live). They automatically aggregate reported results from across the Hub and display leaderboards in the dataset card. The benchmark defines the eval spec via eval.yaml , based on the Inspect AI format, so anyone can reproduce it. The reported results need to align with the task definition. For Models: Eval scores live in .eval_results/*.yaml in the model repo. They appear on the model card and are fed into benchmark datasets. Both the model author’s results and open pull requests for results will be aggregated. Model authors will be able to close score PR and hide results. For the Community: Any user can submit evaluation results for any model via a PR. Results get shown as "community", without waiting for model authors to merge or close. The community can link to sources like a paper, Model Card, third-party evaluation platform, or inspect eval logs. The community can discuss scores like any PR. Since the Hub is Git based, there is a history of when evals were added, when changes were made, etc. The sources look like below. To learn more about evaluation results, check out the docs . Model scores in the Hub Why This Matters Decentralizing evaluation will expose scores that already exist across the community in sources like model cards and papers. By exposing these scores, the community can build on top of them to aggregate, track, and understand scores across the field. Also, all scores will be exposed via Hub APIs, making it easy to aggregate and build curated leaderboards, dashboards, etc. Community evals do not replace benchmarks so leaderboards and closed evals with published results are still crucial. However, we believe it's important to contribute to the field with open eval results based on reproducible eval specs. This won't solve benchmark saturation or close the benchmark-reality gap. Nor will it stop training on test sets. But it makes the game visible by exposing what is evaluated, how, when, and by whom. Mostly, we hope to make the Hub an active place to build and share reproducible benchmarks. Particularly focusing on new tasks and domains that challenge SOTA models more. Get Started Read the docs: To learn more about evaluation results, check out the docs . Add eval results: Publish the evals you conducted as YAML files in .eval_results/ on any model repo. Check out the scores on the benchmark dataset . Register a new benchmark: Add eval.yaml to your dataset repo and contact us to be included in the shortlist. The feature is in beta. We're building in the open. Feedback welcome.

2月2日周一
  1. 腾讯玄武实验室86

    腾讯玄武实验室发布《AI网络爬虫安全白皮书》并开源SEChrome防护方案

    腾讯玄武实验室发布《AI网络爬虫安全白皮书》，系统分析AI时代服务端浏览器/爬虫的攻击面，指出多款产品存在沙箱外远程代码执行（RCE）风险，相关产品服务端用户规模累计超10亿，涵盖多家头部科技企业。

    推荐理由：白皮书梳理了服务端浏览器的完整攻击链与四个真实RCE案例，并给出可复用的纵深防御框架和开源方案，工程落地参考价值高。

1月27日周二
  1. Hugging Face Blog

    Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

    Arabic is one of the most widely spoken languages in the world, with hundreds of millions of speakers across more than twenty countries. Despite this global reach, Arabic is not a monolithic language. Modern Standard Arabic coexists with a rich landscape of regional dialects that differ significantly in vocabulary, syntax, phonology, and cultural grounding. These dialects are the primary medium of daily communication, oral storytelling, poetry, and social interaction. However, most existing benchmarks for Arabic large language models focus almost exclusively on Modern Standard Arabic, leaving dialectal Arabic largely under-evaluated and under-represented. This gap is particularly problematic as large language models increasingly interact with users in informal, culturally grounded, and conversational settings. A model that performs well on formal newswire text may still fail to understand a greeting, an idiomatic expression, or a short anecdote expressed in a local dialect. To address this limitation, our team introduces Alyah الياه (which means North Star ⭐️ in Emirati), an Emirati-centric benchmark designed to assess how well Arabic LLMs capture the linguistic, cultural, and pragmatic aspects of the Emirati dialect. Benchmark Motivation and Scope The Emirati dialect is deeply intertwined with local culture, heritage, and history. It appears in everyday greetings, oral poetry, proverbs, folk narratives, and expressions whose meanings cannot be inferred through literal translation alone. Our benchmark is intentionally designed to probe this depth. Rather than testing surface-level lexical knowledge, it challenges models on their ability to interpret culturally embedded meaning, pragmatic usage, and dialect-specific nuances. The benchmark covers a wide range of content, including common and uncommon local expressions, culturally grounded greetings, short anecdotes, heritage-related questions, and references to Emirati poetry. The goal is not only to measure correctness, but also to understand where models systematically succeed or fail when confronted with authentic Emirati language use. Dataset Structure Following further development and consolidation, the benchmark has been unified into a single dataset called Alyah . The final benchmark contains 1,173 samples , all collected manually from native Emirati speakers to ensure linguistic authenticity and cultural grounding. This manual curation step was essential to capture expressions, meanings, and usages that are rarely documented in written resources and are difficult to infer from Modern Standard Arabic alone. Each sample is formulated as a multiple-choice question with four candidate answers , exactly one of which is correct. Large language models were used to synthetically generate the distractor choices, after which they were reviewed to ensure plausibility and semantic closeness to the correct answer. To avoid positional bias during evaluation, the index of the correct answer follows a randomized distribution across the dataset. Below is the distribution of word count per query and candidate answers. Alyah spans a broad spectrum of linguistic and cultural phenomena in the Emirati dialect, ranging from everyday expressions to culturally sensitive and figurative language. The distribution across categories is summarized below. Category Number of Samples Difficulty Greetings & Daily Expressions 61 Easy Religious & Social Sensitivity 78 Medium Imagery & Figurative Meaning 121 Medium Etiquette & Values 173 Medium Poetry & Creative Expression 32 Difficult Historical & Heritage Knowledge 89 Difficult Language & Dialect 619 Difficult Below are examples of each category: This composition allows Alyah to jointly evaluate surface-level conversational fluency and deeper cultural, semantic, and pragmatic understanding, with a particular emphasis on dialect-specific language phenomena that remain challenging for current models. Model Evaluation Setup We evaluated a total of 54 language models , comprising 23 base models and 31 instruction-tuned models , spanning several architectural and training paradigms. These include Arabic-native LLMs such as Jais and Allam, multilingual models with strong Arabic support such as Qwen and LLaMA, and adapted or regionally specialized models such as Fanar and AceGPT. For each family, both base and instruction-tuned variants were evaluated in order to understand the impact of alignment and instruction tuning on dialectal performance. All models were evaluated under a consistent prompting and scoring protocol. Responses were assessed for semantic correctness and appropriateness with respect to Emirati usage, rather than literal overlap with a reference answer. This is particularly important for dialectal evaluation, where multiple valid phrasings may exist. For each question category, we estimated difficulty empirically based on model performance. Categories where most models struggled were labeled as harder, while those consistently answered correctly across model families were considered easier. This approach allows difficulty to emerge from observed behavior rather than from subjective annotation alone. Evaluation Results on Alyah (Emirati Dialect) We evaluate a broad set of contemporary Arabic and multilingual large language models on Alyah , using accuracy on multiple-choice questions as the primary metric. The evaluation covers 53 models in total, including 22 base models and 31 instruction-tuned models , spanning Arabic-native, multilingual, and regionally adapted systems. Below is a radar plot showing the performance of top models of different sizes per question category. These results are intended as reference measurements within the scope of Alyah, rather than absolute rankings across all Arabic benchmarks. Base Models Model Accuracy google/gemma-3-27b-pt 74.68 tiiuae/Falcon-H1-34B-Base 73.66 FreedomIntelligence/AceGPT-v2-32B 67.35 google/gemma-3-4b-pt 63.17 QCRI/Fanar-1-9B 62.75 tiiuae/Falcon-H1-7B-Base 60.78 meta-llama/Llama-3.1-8B 58.23 Qwen/Qwen3-14B-Base 57.29 inceptionai/jais-adapted-13b 56.01 Qwen/Qwen2.5-32B 53.03 FreedomIntelligence/AceGPT-13B 50.81 Qwen/Qwen2.5-72B 47.91 Qwen/Qwen2.5-14B 46.8 google/gemma-2-2b 41.86 tiiuae/Falcon3-7B-Base 41.43 Qwen/Qwen3-8B-Base 40.75 tiiuae/Falcon-H1-3B-Base 40.41 Qwen/Qwen2.5-7B 36.57 Qwen/Qwen2.5-3B 35.29 meta-llama/Llama-3.2-3B 35.12 inceptionai/jais-adapted-7b 33.5 Qwen/Qwen3-4B-Base 27.45 Instruction-Tuned Models Model Accuracy falcon-h1-arabic-7b-instruct 82.18 humain-ai/ALLaM-7B-Instruct-preview 77.24 google/gemma-3-27b-it 74.68 falcon-h1-arabic-3b-instruct 74.51 Qwen/Qwen2.5-72B-Instruct 74.6 CohereForAI/aya-expanse-32b 73.66 Navid-AI/Yehia-7B-preview 73.32 FreedomIntelligence/AceGPT-v2-32B-Chat 72.8 Qwen/Qwen2.5-32B-Instruct 71.61 tiiuae/Falcon-H1-34B-Instruct 71.1 meta-llama/Llama-3.3-70B-Instruct 69.74 QCRI/Fanar-1-9B-Instruct 69.22 tiiuae/Falcon-H1-7B-Instruct 65.13 CohereForAI/c4ai-command-r7b-arabic-02-2025 64.54 silma-ai/SILMA-9B-Instruct-v1.0 63.94 FreedomIntelligence/AceGPT-v2-8B-Chat 63.43 CohereLabs/aya-expanse-8b 61.21 yasserrmd/kallamni-2.6b-v1 61.13 yasserrmd/kallamni-4b-v1 60.7 microsoft/Phi-4-mini-instruct 58.57 tiiuae/Falcon-H1-3B-Instruct 57.12 silma-ai/SILMA-Kashif-2B-Instruct-v1.0 48.51 Qwen/Qwen2.5-7B-Instruct 45.44 google/gemma-3-4b-it 46.12 meta-llama/Llama-3.1-8B-Instruct 46.29 meta-llama/Llama-3.2-3B-Instruct 39.64 yasserrmd/kallamni-1.2b-v1 37.77 Qwen/Qwen3-4B 26.26 google/gemma-2-2b-it 26.00 Qwen/Qwen3-14B 26.00 Qwen/Qwen3-8B 25.66 Analysis and Observed Trends Figure 1: Models' accuracy across categories based on size. Figure 2: Models' accuracy across categories based on language. Several trends emerge from the evaluation. Instruction-tuned models generally outperform their base counterparts as shown in Figures 1 and 2. This is particularly the case on questions involving conversational norms and culturally appropriate responses (i.e. the Etiquette & Values Category). Furthermore, it is the case with questions that test imagery and figurative meaning. This can be attributed, to the model’s original strong capabilities with understanding MSA-based imagery and figurative language regardless of the dialect at hand. The models are able to draw patterns of non-literal description regardless of dialect. Generally, the most difficult categories for the models were consistently “Language and Dialect” and “Greeting and Daily expressions” across model sizes as shown in figure 1. These results reflect the current state of Emirati dialect presence in written media, as the dialect is mostly spoken rarely written, which explains its novelty relative to the evaluated models. Nonetheless, there is a clear benefit to instruct models with understanding the dialect (and the other evaluation categories) in comparison to their counterparts, especially in small and medium models. This is particularly noticeable with the Poetry and Creative Expression category, which is where the large instruct models performed marginally better than the smaller models. Figure 3: Evaluated models average accuracy. As shown in Figure 3, even strong multilingual models show notable degradation on the most challenging Alyah questions, suggesting that dialect-specific semantic knowledge is not easily acquired through generic multilingual training alone. It must be noted that while Arabic-native models tend to perform more robustly on culturally grounded content, their performances are not uniform across all categories (figure 2). In particular, questions involving implicit meanings and rare expressions remain difficult across nearly all evaluated models. This highlights a persistent gap between surface-level dialect familiarity and deeper cultural understanding. The high variance in performance across categories , where a model that excels at imagery and figurative meaning may still struggle with poetry or heritage-related creative questions, indicates that dialectal competence is multi-dimensional and cannot be captured by a single score. Figure 3 shows that the highest scoring large model in Jais-2-70B, followed by the two small models jais-2-8B and ALLaM-7B-instruct, which are all Arabic instruct-tuned models. Conclusion and Community Impact This benchmark represents a step toward more realistic and culturally grounded evaluation of Arabic language models. By focusing on the Emirati dialect, we aim to support the development of models that better serve local communities, institutions, and users in the UAE. Beyond model ranking, the benchmark is intended as a diagnostic tool to guide future data collection, training, and adaptation efforts. We invite researchers, practitioners, and the broader community to use the benchmark, explore the results, and share feedback. Community input will be essential to refining the dataset, expanding coverage, and ensuring that dialectal Arabic receives the attention it deserves in the evaluation of Large Language Models. Citation @misc{emirati_dialect_benchmark_2026, title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models}, author={Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid}, year = {2026}, month = {january}, }

1月21日周三
  1. Hugging Face Blog

    AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality

    AssetOpsBench is a comprehensive benchmark and evaluation system with six qualitative dimensions that bridges the gap for agentic AI in domain-specific settings, starting with industrial Asset Lifecycle Management. Introduction While existing AI benchmarks excel at isolated tasks such as coding or web navigation, they often fail to capture the complexity of real-world industrial operations. To bridge this gap, we introduce AssetOpsBench , a framework specifically designed to evaluate agent performance across six critical dimensions of industrial applications. Unlike traditional benchmarks, AssetOpsBench emphasizes the need for multi-agent coordination—moving beyond `lone wolf' models to systems that can handle complex failure modes, integrate multiple data streams, and manage intricate work orders. By focusing on these high-stakes, multi-agent dynamics, the benchmark ensures that AI agents are assessed on their ability to navigate the nuances and safety-critical demands of a true industrial environment. AssetOpsBench is built for asset operations such as chillers and air handling units. It comprises: 2.3M sensor telemetry points 140+ curated scenarios across 4 agents 4.2K work orders for diverse scenarios 53 structured failure modes Experts helped curate 150+ scenarios. Each scenario includes metadata: task type, output format, category, and sub-agents. The tasks designed span across: Anomaly detection in sensor streams Failure mode reasoning and diagnostics KPI forecasting and analysis Work order summarization and prioritization Evaluation Framework and Overall Feedback AssetOpsBench evaluates agentic systems across six qualitative dimensions designed to reflect real operational constraints in industrial asset management. Rather than optimizing for a single success metric, the benchmark emphasizes decision trace quality, evidence grounding, failure awareness, and actionability under incomplete and noisy data. Each agent run is scored across six criteria: Task Completion Retrieval Accuracy Result Verification Sequence Correctness Clarity and Justification Hallucination rate Across early evaluations, we observe that many general-purpose agents perform well on surface-level reasoning but struggle with sustained multi-step coordination involving work orders, failure semantics, and temporal dependencies. Agents that explicitly model operational context and uncertainty tend to produce more stable and interpretable trajectories, even when final task completion is partial. This feedback-oriented evaluation is intentional: in industrial settings, understanding why an agent fails is often more valuable than a binary success signal. Failure Modes in Industrial Agentic Workflows A central contribution of AssetOpsBench is the explicit treatment of failure modes as first-class evaluation signals in agentic industrial workflows. Rather than treating failure as a binary outcome, AssetOpsBench analyzes full multi-agent execution trajectories to identify where , how , and why agent behavior breaks down under realistic operational constraints. Failure analysis in AssetOpsBench is implemented through a dedicated trajectory-level pipeline ( TrajFM ), which combines LLM-based reasoning with statistical clustering to surface interpretable failure patterns from agent execution traces. This pipeline operates in three stages: (1) trajectory-level failure extraction using an LLM-guided diagnostic prompt, (2) embedding-based clustering to group recurring failure patterns, and (3) analysis and visualization to support developer feedback and iteration. Across industrial scenarios, recurrent failure modes include: Misalignment between sensor telemetry, alerts, and historical work orders Overconfident conclusions drawn under missing, delayed, or insufficient evidence Inconsistent aggregation of heterogeneous data modalities across agents Premature action selection without adequate verification or validation steps Breakdowns in multi-agent coordination, such as ignored inputs or action–reasoning mismatches Importantly, AssetOpsBench does not rely solely on a fixed, hand-crafted failure taxonomy. While a structured set of predefined failure categories (e.g., verification errors, step repetition, role violations) is used for consistency, the system is explicitly designed to discover new failure patterns that emerge in practice. Additional failure modes identified by the LLM are embedded and clustered automatically, allowing the taxonomy to evolve as new agent designs and behaviors are evaluated. To preserve industrial confidentiality, raw execution traces are never exposed. Instead, agents receive aggregated scores across six evaluation dimensions together with clustered failure-mode summaries that explain why an agent failed, without revealing sensitive data or intermediate reasoning steps. This feedback-driven design enables developers to diagnose weaknesses, refine agent workflows, and iteratively resubmit improved agents. This failure-aware evaluation reflects the realities of industrial asset management, where cautious, degradation-aware reasoning—and the ability to recognize uncertainty, defer action, or escalate appropriately—is often preferable to aggressive but brittle automation. Submit an Agent for Evaluation AssetOpsBench-Live is designed as an open, competition-ready benchmark , and we welcome submissions of agent implementations from the community. Agents are evaluated in a controlled, privacy-preserving environment that reflects real industrial asset management constraints. To submit an agent, developers first validate their implementation locally using a provided simulated environment, which includes representative sensor data, work orders, alerts, and failure-mode catalogs. Agents are then containerized and submitted for remote execution on hidden evaluation scenarios. Submitted agents are evaluated across six qualitative dimensions—task completion, accuracy, result verification, action sequencing, clarity, and hallucination—using a consistent, reproducible evaluation protocol. Execution traces are not exposed; instead, participants receive aggregated scores and structured failure-mode feedback that highlights where and why an agent’s reasoning or coordination broke down. This feedback-driven evaluation loop enables iterative improvement: developers can diagnose failure patterns, refine agent design or workflow structure, and resubmit updated agents for further evaluation. Both planning-focused and execution-focused agents are supported, allowing researchers and practitioners to explore diverse agentic designs within the same benchmark framework. Experiment and Observations We performed a community evaluation where we tested two tracks: Planning-oriented multi-agent orchestration Execution-oriented dynamic multi-agent workflow. Across 225 users and 300+ agents and leading open source models, here are the observations: Model Family Best Planning Score Best Execution Score Key Limitation GPT-4.1 68.2 72.4 Hallucinated completion on complex workflows Mistral-Large 64.7 69.1 Struggled with multi-hop tool sequences LLaMA-4 Maverick 66.0 70.8 Missed clarifying questions (fixable) LLaMA-3-70B 52.3 58.9 Collapsed under multi-agent coordination Note: None of the models could pass our evaluation criteria benchmark and get 85 points, which is the threshold for deployment readiness. Distribution of Failures Across 881 agent execution traces, failure distribution was as follows: Ineffective Error Recovery: 31.2% Overstated Completion: 23.8% Formatting Issues: 21.4% Unhandled Tool Errors: 10.3% Ignored Feedback: 8.0% Other: 5.3% Beyond this, 185 traces had one new failure pattern and 164 had multiple novel failures. Key Error Findings "Sounds Right, Is Wrong": Agents claim to have completed tasks (23.8%) and output success even after unsuccessful failure recovery (31.2%). AssetOps benchmarking is important to uncover this so that operators do not act upon incorrect information. Tool Usage: This is the biggest differentiator between high and low performing agents, with top agents having 94% tool accuracy compared to 61% of low performers. Multi-agent Multiplies Failures: Task accuracy between single agent (68%) vs multi-agent (47%) shows the complexity multi-agent brings with context loss, asynchronous issues, and cascaded failures. Domain Knowledge: Agents with access to failure mode databases and maintenance manuals performed better. However, RAG knowledge wasn’t always used correctly, suggesting a need for structured reasoning. Ambiguity: Missing sensors, conflicting logs, and vague operator descriptions caused the success rate to drop 34%. Agents must have clarification strategies embedded. Where to get started? Read our technical report AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance How to run AssetOpsBench locally - Video AssetOpsBench Local Execution Try out AssetOpsBench in the HuggingFace Space Playground Find More Detail AssetOpsBench GitHub , fork the repo and get started.

1月16日周五
  1. Anthropic72

    Anthropic 红队：Claude Sonnet 4.5 在真实网络靶场中自主利用公开 CVE

    Anthropic 前沿红队与 Incalmo 合作，在 25-50 台主机的模拟网络靶场上评测 Claude，发现 Claude Sonnet 4.5 在少数网络上无需此前几代所需的定制网络工具包即可成功。

    推荐理由：Anthropic 红队披露 Claude Sonnet 4.5 在真实网络靶场中无需专用工具即可自主利用公开 CVE，并给出与上一代的对照数据。

1月14日周三
1月12日周一
1月8日周四
1月6日周二
  1. 腾讯玄武实验室62

    腾讯玄武实验室披露 ComfyUI-Manager 远程代码执行漏洞 CVE-2025-67303

    腾讯玄武实验室发现 ComfyUI 官方扩展组件 ComfyUI-Manager 存在高危漏洞 CVE-2025-67303，远程攻击者可在无需账号的情况下实现远程代码执行并完全控制服务器，包括盗取用户数据和私有 AI 模型。

    推荐理由：通告给出了漏洞根因、影响版本与迁移步骤，读者可据此判断自身 ComfyUI 部署是否受影响并完成修复。

12月23日周二
  1. Hugging Face Blog76

    ServiceNow 发布 8B 安全护栏模型 AprielGuard，覆盖 Agent 工作流攻击检测

    ServiceNow-AI 在 Hugging Face 发布 AprielGuard，一个 8B 参数的安全护栏模型，用于检测 16 类安全风险（涵盖毒性、仇恨、虚假信息、自残、非法活动等）以及提示词注入、越狱、思维链污染、上下文劫持、记忆投毒和多智能体利用链等对抗攻击，同时支持对 Agent 工作流中的工具调用与推理轨迹进行检测。

12月19日周五
12月18日周四
  1. Anthropic55

    Anthropic 公布 Project Vend 第二阶段：Claude 经营小店实验

    Anthropic 公布 Project Vend 第二阶段结果，Claudius 从 Claude Sonnet 3.7 升级到 Sonnet 4.0 和 4.5，并接入 CRM、库存管理、网页浏览等工具，还新增 CEO 智能体 Seymour Cash 和周边制作智能体 Clothius，业务表现明显改善，亏损周基本消除，门店扩展到旧金山、纽约和伦敦三地。

12月17日周三
  1. Hugging Face Blog

    The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator

    It has become increasingly challenging to assess whether a model’s reported improvements reflect genuine advances or variations in evaluation conditions, dataset composition, or training data that mirrors benchmark tasks. The NVIDIA Nemotron approach to openness addresses this by publishing transparent and reproducible evaluation recipes that make results independently verifiable. NVIDIA released Nemotron 3 Nano 30B A3B with an explicitly open evaluation approach to make that distinction clear. Alongside the model card, we are publishing the complete evaluation recipe used to generate the results, built with the NVIDIA NeMo Evaluator library, so anyone can rerun the evaluation pipeline, inspect the artifacts, and analyze the outcomes independently. We believe that open innovation is the foundation of AI progress. This level of transparency matters because most model evaluations omit critical details. Configs, prompts, harness versions, runtime settings, and logs are often missing or underspecified, and even small differences in these parameters can materially change results. Without a complete recipe, it’s nearly impossible to tell whether a model is genuinely more intelligent or simply optimized for a benchmark. This blog shows developers exactly how to reproduce the evaluation behind Nemotron 3 Nano 30B A3B using fully open tools, configurations, and artifacts. You’ll learn how the evaluation was run, why the methodology matters, and how to execute the same end-to-end workflow using the NeMo Evaluator library so you can verify results, compare models consistently, and build transparent evaluation pipelines of your own. Building a consistent and transparent evaluation workflow with NeMo Evaluator A single, consistent evaluation system Developers and researchers need evaluation workflows they can rely on, not one-off scripts that behave differently from model to model. NeMo Evaluator provides a unified way to define benchmarks, prompts, configuration, and runtime behavior once, then reuse that methodology across models and releases. This avoids the common scenario where the evaluation setup quietly changes between runs, making comparisons over time difficult or misleading. Methodology independent of inference setup Model outputs can vary by inference backend and configuration, so evaluation tools should never be tied to a single inference solution. Locking an evaluation tool to one inference solution would limit its usefulness. NeMo Evaluator avoids this by separating the evaluation pipeline from the inference backend, allowing the same configuration to run against hosted endpoints, local deployments, or third-party providers. This separation enables meaningful comparisons even when you change infrastructure or inference engines. Built to scale beyond one-off experiments Many evaluation pipelines work once and then break down as the scope expands. NeMo Evaluator is designed to scale from quick, single-benchmark validation to full model card suites and repeated evaluations across multiple models. The launcher, artifact layout, and configuration model support ongoing workflows, not just isolated experiments, so teams can maintain consistent evaluation practices over time. Auditability with structured artifacts and logs Transparent evaluation requires more than final scores. Each evaluation run produces structured results and logs by default, making it easy to inspect how scores were computed, understand score calculations, debug unexpected behavior, and conduct deeper analysis. Each component of the evaluation is captured and reproducible. A shared evaluation standard By releasing Nemotron 3 Nano 30B A3B with its full evaluation recipe , NVIDIA is providing a reference methodology that the community can run, inspect, and build upon. Using the same configuration and tools brings consistency to how benchmarks are selected, executed, and interpreted, enabling more reliable comparisons across models, providers, and releases. Open evaluation for Nemotron 3 Nano Open evaluation means publishing not just the final results, but the full methodology behind them, so benchmarks are run consistently, and results can be compared meaningfully over time. For Nemotron 3 Nano 30B A3B , this includes open‑source tooling, transparent configurations, and reproducible artifacts that anyone can run end‑to‑end. Open-source model evaluation tooling NeMo Evaluator is an open-source library designed for robust, reproducible, and scalable evaluation of generative models. Instead of introducing yet another standalone benchmark runner, it acts as a unifying orchestration layer that brings multiple evaluation harnesses under a single, consistent interface. Under this architecture, NeMo Evaluator integrates and coordinates hundreds of benchmarks from many widely used evaluation harnesses, including NeMo Skills for Nemotron instruction-following, tool use, and agentic evaluations, as well as the LM Evaluation Harness for base model and pre-training benchmarks, and many more ( full benchmark catalog ). Each harness retains its native logic, datasets, and scoring semantics, while NeMo Evaluator standardizes how they are configured, executed, and logged. This provides two practical advantages: teams can run diverse benchmark categories using a single configuration without rewriting custom evaluation scripts, and results from different harnesses are stored and inspected in a consistent, predictable way, even when the underlying tasks differ. The same orchestration framework used internally by NVIDIA’s Nemotron research and model‑evaluation teams is now available to the community, enabling developers to run heterogeneous, multi‑harness evaluations through a shared, auditable workflow. Open configurations We published the exact YAML configuration used for the Nemotron 3 Nano 30B A3B model card evaluation with NeMo Evaluator. This includes: model inference and deployment settings benchmark and task selection benchmark-specific parameters such as sampling, repeats, and prompt templates runtime controls including parallelism, timeouts, and retries output paths and artifact layout Using the same configuration means running the same evaluation methodology. Open logs and artifacts Each evaluation run produces structured, inspectable outputs, including per‑task results.json files, execution logs for debugging and auditability, and artifacts organized by task for easy comparison. This structure makes it possible to understand not only the final scores, but also how those scores were produced and to perform deeper analysis of model behavior. The reproducibility workflow Reproducing Nemotron 3 Nano 30B A3B model card results follows a simple loop: Start from the released model checkpoint or hosted endpoint Use the published NeMo Evaluator config Execute the evaluation with a single CLI command Inspect logs and artifacts, and compare results to the model card The same workflow applies to any model you evaluate using NeMo Evaluator. You can point the evaluation at a hosted endpoint or a local deployment, including common inference providers such as HuggingFace , build.nvidia.com , and OpenRouter . The key requirement is access to the model, either as weights you can serve or as an endpoint you can call. For this tutorial, we use the hosted endpoint on build.nvidia.com . Reproducing Nemotron 3 Nano benchmark results This tutorial reproduces the evaluation results for NVIDIA Nemotron 3 Nano 30B A3B using NeMo Evaluator. The step-by-step tutorial, including the published configs used for the model card evaluation , is available on GitHub. Although we have focused this tutorial on the Nemotron 3 Nano 30B A3B, we also published recipes for the base model evaluation . This walkthrough runs a comprehensive evaluation suite of the published configs used for the model card evaluation for NVIDIA Nemotron 3 Nano 30B A3B using the following benchmarks: Benchmark Accuracy Category Description BFCL v4 53.8 Function Calling Berkeley Function Calling Leaderboard v4 LiveCodeBench (v6 2025-08–2025-05) 68.3 Coding Real-world coding problems evaluation MMLU-Pro 78.3 Knowledge Multi-task language understanding (10-choice) GPQA 73.0 Science Graduate-level science questions AIME 2025 89.1 Mathematics American Invitational Mathematics Exam SciCode 33.3 Scientific Coding Scientific programming challenges IFBench 71.5 Instruction Following Instruction following benchmark HLE 10.6 Humanity's Last Exam Expert-level questions across domains For Model Card details, see the NVIDIA Nemotron 3 Nano 30B A3B Model Card . For a deep dive into the architecture, datasets, and benchmarks, read the full Nemotron 3 Nano Technical Report . 1. Install NeMo Evaluator Launcher pip install nemo-evaluator-launcher 2. Set required environment variables # NVIDIA endpoint access export NGC_API_KEY="your-ngc-api-key" # Hugging Face access export HF_TOKEN="your-huggingface-token" # Required only for judge-based benchmarks such as HLE export JUDGE_API_KEY="your-judge-api-key" Optional but recommended for faster reruns: export HF_HOME="/path/to/your/huggingface/cache" 3. Model endpoint The evaluation uses the NVIDIA API endpoint hosted on build.nvidia.com : target: api_endpoint: model_id: nvidia/nemotron-nano-3-30b-a3b url: https://integrate.api.nvidia.com/v1/chat/completions api_key_name: NGC_API_KEY Evaluations can be run against common inference providers such as HuggingFace , build.nvidia.com , or OpenRouter , or anywhere that the model has an available endpoint. If you're hosting the model locally or using a different endpoint: nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o target.api_endpoint.url=http://localhost:8000/v1/chat/completions 4. Run the full evaluation suite Preview the run without executing using --dry-run : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ --dry-run From the examples directory, run the evaluation using the YAML configuration provided: nemo-evaluator-launcher run \ --config /path/to/examples/nemotron/local_nvidia_nemotron_3_nano_30b_a3b.yaml Note that for quick testing, you can limit the number of samples by setting limit_samples : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 5. Running an individual benchmark You can run specific benchmarks using the -t flag (from the examples/nemotron directory): # Run only MMLU-Pro nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_mmlu_pro # Run only coding benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_livecodebench # Run multiple specific benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_gpqa -t ns_aime2025 6. Monitor execution and inspect results # Check status of a specific job nemo-evaluator-launcher status # Stream logs for a specific job nemo-evaluator-launcher logs <job-id> Results are written to the defined output directory: results_nvidia_nemotron_3_nano_30b_a3b/ ├── artifacts/ │ └── <task_name>/ │ └── results.json └── logs/ └── stdout.log Interpreting results When reproducing evaluations, you may observe small differences in final scores across runs. This variance reflects the probabilistic nature of LLMs rather than an issue with the evaluation pipeline. Modern evaluation introduces several sources of non‑determinism: decoding settings, repeated trials, judge‑based scoring, parallel execution, and differences in serving infrastructure. All of which can lead to slight fluctuations. The purpose of open evaluation is not to force bit-wise identical outputs, but to deliver methodological consistency with clear provenance of evaluation results. To ensure your evaluation aligns with the reference standard, verify the following: Configuration : use the published NeMo Evaluator YAML without modification, or document any changes explicitly Benchmark selection : run the intended tasks, task versions, and prompt templates Inference target : verify you are evaluating the intended model and endpoint, including chat template behavior and reasoning settings when relevant Execution settings : keep runtime parameters consistent, including repeats, parallelism, timeouts, and retry behavior Outputs : confirm artifacts and logs are complete and follow the expected structure for each task When these elements are consistent, your results represent a valid reproduction of the methodology, even if individual runs differ slightly. NeMo Evaluator simplifies this process, tying benchmark definitions, prompts, runtime settings, and inference configuration into a single auditable workflow to minimize inconsistencies. Conclusion: A more transparent standard for open models The evaluation recipe released alongside Nemotron 3 Nano represents a meaningful step toward a more transparent and reliable approach to open-model evaluation. We are moving away from evaluation as a collection of bespoke, "black box" scripts, and towards a defined system where benchmark selection, prompts, and execution semantics are encoded into a transparent workflow. For developers and researchers, this transparency changes what it means to share results. A score is only as trustworthy as the methodology behind it and making that methodology public is what enables the community to verify claims, compare models fairly, and continue building on shared foundations. With open evaluation configurations, open artifacts, and open tooling, Nemotron 3 Nano demonstrates what that commitment to openness looks like in practice. NeMo Evaluator supports this shift by providing a consistent benchmarking methodology across models, releases, and inference environments. The objective isn’t identical numbers on every run; it’s confidence in an evaluation methodology that is explicit, inspectable, and repeatable. And for organizations that need automated or large‑scale evaluation pipelines, a separate microservice offering provides an enterprise‑ready NeMo Evaluator microservice built on the same evaluation principles. Use the published NeMo Evaluator evaluation configuration for an end-to-end walkthrough of the evaluation recipe. Join the Community! NeMo Evaluator is fully open source, and community input is essential to shaping the future of open evaluation. If there’s a benchmark you’d like us to support or an improvement you want to propose, open an issue, or contribute directly on GitHub. Your contributions help strengthen the ecosystem and advance a shared, transparent standard for evaluating generative models.

12月8日周一
12月1日周一
  1. Anthropic82

    Anthropic 红队发布 SCONE-bench：AI 智能体在模拟中挖出价值 460 万美元的智能合约漏洞

    Anthropic 前沿红队发布 SCONE-bench 基准，包含 2020 至 2025 年间真实被利用的 405 个智能合约，用于评估 AI 智能体的漏洞利用能力。

    推荐理由：Anthropic 红队用真实被黑合约构建基准，把智能体攻击能力换算成美元损失，并给出成本与收益的量化对照。

11月21日周五
  1. Hugging Face Blog

    Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks

    While everyone (and their grandma 👵) is spinning up new ASR models, picking the right one for your use case can feel more overwhelming than choosing your next Netflix show. As of 21 Nov 2025, there are 150 Audio-Text-to-Text and 27K ASR models on the Hub 🤯 Most benchmarks focus on short-form English transcription (<30s), and overlook other important tasks, such as (1) multilingual performance and (2) model throughput, which can a be deciding factor for long-form audio like meetings and podcasts. Over the past two years, the Open ASR Leaderboard has become a standard for comparing open and closed-source models on both accuracy and efficiency . Recently, multilingual and long-form transcription tracks have been added to the leaderboard 🎉 TL;DR - Open ASR Leaderboard 📝 New preprint on ASR trends from the leaderboard: https://hf.co/papers/2510.06961 🧠 Best accuracy: Conformer encoder + LLM decoders (open-source ftw 🥳) ⚡ Fastest: CTC / TDT decoders 🌍 Multilingual: Comes at the cost of single-language performance ⌛ Long-form: Closed-source systems still lead (for now 😉) 🧑‍💻 Fine-tuning guides ( Parakeet , Voxtral , Whisper ): to continue pushing performance Takeaways from 60+ models As of 21 Nov 2025, the Open ASR Leaderboard compares 60+ open and closed-source models from 18 organizations , across 11 datasets . In a recent preprint , we dive into the technical setup and highlight some key trends in modern ASR. Here are the big takeaways 👇 1. Conformer encoder 🤝 LLM decoder tops the charts 📈 Models combining Conformer encoders with large language model (LLM) decoders currently lead in English transcription accuracy. For example, NVIDIA’s Canary-Qwen-2.5B , IBM’s Granite-Speech-3.3-8B , and Microsoft’s Phi-4-Multimodal-Instruct achieve the lowest word error rates ( WER ), showing that integrating LLM reasoning can significantly boost ASR accuracy. 💡 Pro-tip: NVIDIA introduced Fast Conformer , a 2x faster variant of the Conformer, that is used in their Canary and Parakeet suite of models. 2. Speed–accuracy tradeoffs ⚖️ While highly accurate, these LLM decoders tend to be slower than simpler approaches. On the Open ASR Leaderboard , efficiency is measured using inverse real-time factor (RTFx), where higher is better. For even faster inference, CTC and TDT decoders deliver 10–100× faster throughput , albeit with slightly higher error rates. This makes them ideal for real-time , offline , or batch transcription tasks (such as meetings, lectures, or podcasts). 3. Multilingual 🌍 OpenAI’s Whisper Large v3 remains a strong multilingual baseline, supporting 99 languages . However, fine-tuned or distilled variants like Distil-Whisper and CrisperWhisper often outperform the original on English-only tasks, showing how targeted fine-tuning can improve specialization ( how to fine-tune? Check out guides for Whisper , Parakeet , and Voxtral ). That said, focusing on English tends to reduce multilingual coverage 👉 a classic case of the tradeoff between specialization and generalization. Similarly, while self-supervised systems like Meta’s Massively Multilingual Speech (MMS) and Omnilingual ASR can support 1K+ languages, they trail behind language-specific encoders in accuracy. ⭐ While just five languages are currently benchmarked, we’re planning to expand to more languages and are excited for new dataset and models contributions to multilingual ASR through GitHub pull requests . 🎯 Alongside multilingual benchmarks, several community-driven leaderboards focus on individual languages. For example, the Open Universal Arabic ASR Leaderboard compares models across Modern Standard Arabic and regional dialects , highlighting how speech variation and diglossia challenge current systems. Similarly. the Russian ASR Leaderboard provides a growing hub for evaluating encoder-decoder and CTC models on Russian-specific phonology and morphology . These localized efforts mirror the broader multilingual leaderboard’s mission to encourage dataset sharing, fine-tuned checkpoints, and transparent model comparisons , especially in languages with fewer established ASR resources. 4. Long-form transcription is a different game ⏳ For long-form audio (e.g., podcasts, lectures, meetings), closed-source systems still edge out open ones. It could be due to domain tuning, custom chunking, or production-grade optimization. Among open models, OpenAI’s Whisper Large v3 performs the best. But for throughput, CTC-based Conformers shine 👉 for example, NVIDIA’s Parakeet CTC 1.1B achieves an RTFx of 2793.75 , compared to 68.56 for Whisper Large v3, with only a moderate WER degradation ( 6.68 and 6.43 respectively). The tradeoff? Parakeet is English-only, again reminding us of that multilingual and specialization tradeoff 🫠. ⭐ While closed systems still lead, there’s huge potential for open-source innovation here. Long-form ASR remains one of the most exciting frontiers for the community to tackle next! 🎤 The Show Must Go On Given how fast ASR is evolving, we’re excited to see what new architectures push performance and efficiency, and how the Open ASR Leaderboard continues to serve as a transparent, community-driven benchmark for the field, and as a reference for other leaderboards ( Russian , Arabic , and Speech DeepFake Detection ). We’ll keep expanding the Open ASR LeaderBoard with more models, more languages, and more datasets so stay tuned 👀 👉 Want to contribute? Head on over to the GitHub repo to open a pull request 🚀

11月15日周六
  1. 腾讯玄武实验室57

    腾讯玄武实验室详解量子计算机距离攻破 RSA-2048 还有多远

    腾讯玄武实验室发布长文，从量子计算原理、超导量子计算机构造到破解 RSA-2048 的工程瓶颈，系统分析量子计算机距离攻破经典公钥密码还有多远。文章指出科学层面已无死胡同，真正的挑战集中在冷却、控制、布线、能耗与量子纠错实时实现等工程问题；以表面码纠错计算，运行 Shor 算法需要百万级物理量子比特。

11月12日周三
11月10日周一
  1. 腾讯玄武实验室73

    腾讯玄武实验室阿图因引擎发现 gnark 零知识证明库签名可锻造性漏洞 CVE-2025-57801

    腾讯玄武实验室的阿图因自动化漏洞挖掘引擎在零知识证明库 gnark 中发现高危签名可锻造性漏洞 CVE-2025-57801（CVSS 8.6），并联合上海交通大学 GOSSIP 实验室及郁昱教授团队完成复现。

    推荐理由：文章解释了漏洞虽存在多年却大多未被触发的原因，读者可据此自查闭源项目是否使用了受影响的 Native Verification API。

11月4日周二