跳到正文
4月9日周四
4月7日周二
4月6日周一
  1. AICSO.AI汇总77

    Meta 在重大 AI 数据泄露后暂停与 Mercor 合作

    Meta 暂停与 AI 数据初创公司 Mercor 的全部合同，起因是一次针对开源库 LiteLLM 的供应链攻击：黑客组织 TeamPCP 于 3 月 27 日利用 LiteLLM 维护者泄露的凭证入侵其 CI/CD 流水线。

    推荐理由：原文梳理了从开源依赖到多家 AI 厂商的连锁泄露路径，读者可据此理解共享第三方数据供应商的集中风险。

3月24日周二
  1. Hugging Face Blog

    A New Framework for Evaluating Voice Agents (EVA)

    Introduction Conversational voice agents present a distinct evaluation challenge: they must simultaneously satisfy two objectives — accuracy (completing the user's task correctly and faithfully) and conversational experience (doing so naturally, concisely, and in a way appropriate for spoken interaction). These objectives are deeply intertwined: mishearing a confirmation code renders perfect LLM reasoning meaningless, a wall of options overwhelms a caller who can't skim spoken output, and delayed responses can pass every accuracy check while remaining unusable in practice. Existing frameworks treat these as separate concerns — evaluating task success or conversational dynamics, but not both. We introduce EVA, an end-to-end evaluation framework for conversational voice agents that evaluates complete, multi-turn spoken conversations using a realistic bot-to-bot architecture. EVA produces two high-level scores, EVA-A (Accuracy) and EVA-X (Experience), and is designed to surface failures along each dimension. EVA is the first to jointly score task success and conversational experience. We release EVA with an initial airline dataset of 50 scenarios covering flight rebooking, cancellation handling, vouchers, and more — the first in a planned series of domains. We also provide benchmark results for 20 cascade and audio-native systems, such as speech-to-speech (S2S) models and Large Audio Language Models (LALMs). Our biggest finding is that there is a consistent Accuracy-Experience tradeoff; agents that perform well on task completion tend to deliver worse user experiences, and vice versa. 🌐 Website — Explore the full framework, early results, and a demo. 💻 GitHub — Dive into the code, framework, and judge prompts. 📂 HF Dataset - Explore the dataset Background and Motivation The field currently lacks a framework that evaluates the full quality of voice agent interactions, as most existing efforts assess individual components in isolation. For example, AudioBench , SD-Eval , VoxEval , Kimi-Audio-Evalkit , VoiceBench and VoxDialogue evaluate core speech understanding capabilities for Speech-to-Text (STT) — transcription, paralinguistics, acoustic cues — but remain confined to single-turn, non-interactive settings. On the other hand, EmergentTTS-Eval and SHEET assess perceived speech quality using subjective listening tests (e.g., Mean Opinion Score). Beyond speech perception, FD-Bench , Talking Turns , Full-Duplex-Bench provide deeper analyses of conversational dynamics — interruptions, backchanneling, turn-taking — yet evaluate these in isolation from task-oriented tool use, leaving the relationship between dialogue quality and agentic capability unexamined. More recent efforts, notably VoiceAgentBench and CAVA , take steps towards evaluating the agentic capabilities of commercial voice agent systems, including tool-calling and complex instruction-following. However, these voice-agentic capabilities are not evaluated within complete conversational workflows that voice agents must navigate in practice: from initial user request through multi-step tool orchestration to final task resolution. The lack of frameworks that jointly capture accuracy and experience underscores the need for a framework that treats voice agent quality as an integrated whole. This means evaluating not only whether the task succeeded, but whether the agent communicated accurately, concisely, and naturally throughout, and surfacing how these dimensions trade off against one another in realistic deployment conditions. EVA The Framework End-to-end evaluation reveals interaction dynamics that are not apparent at the component level: whether the agent interrupts users during natural pauses in speech, whether it recovers smoothly when a user corrects a transcription error, or whether high latency disrupts the conversational flow enough to prompt users to repeat themselves or abandon the task entirely. EVA simulates multi-turn spoken conversations over live audio in which the agent must invoke appropriate tools, adhere to task-specific policies, and reach a deterministically verifiable end state. EVA evaluates voice agents using a bot-to-bot audio architecture composed of five core components: User Simulator — A conversational AI configured with a specific goal and persona that plays the role of a caller. It operates in audio using high-quality Text-to-Speech (TTS) models, ensuring the evaluation captures representative speech-understanding challenges in natural-sounding conversational speech and realistic turn-taking dynamics. Voice Agent — The voice agent being evaluated, built with Pipecat, an open-source Python framework for real-time voice applications. EVA supports both cascade architectures (STT → LLM → TTS) and audio-native models (S2S or LALM → TTS). Tool Executor — The engine that provides deterministic, reproducible tool responses via custom Python functions. It dynamically queries and modifies a predefined per-scenario database. Validators — A set of validation metrics that check that conversations are complete and that the user faithfully reproduced the intended behavior and speech, with no human annotation required. Any conversation that fails in this validation step is regenerated, ensuring that only valid, correctly executed conversations enter evaluation. This stands in contrast to approaches that rely on post-hoc human labeling to identify simulator errors. Metrics Suite — A suite of metrics evaluates the voice agent using the conversation recording, transcript, and tool call logs. Data Each test case (scenario) in our framework is an evaluation record, structured to make tests reproducible: User Goal — What the caller is trying to accomplish. Includes a highly specific user objective with an exact decision tree that guides the user simulator through the conversation, leaving no ambiguity about the intended outcome. User Persona — How the caller should behave — their speaking style, patience level, and personality traits. Scenario Database — The backend data the agent's tools will query. Ground Truth — The expected final state of the scenario database after a successful conversation. We release EVA with a synthetic airline dataset of 50 scenarios and 15 tools, spanning IRROPS rebooking, voluntary itinerary changes, cancellations, same-day standby, and compensation vouchers. Scenarios are designed to test temporal reasoning, policy-following, constraint satisfaction, and named-entity handling. See the full demo here . Evaluation Methodology EVA evaluates voice agents across two fundamental dimensions, EVA-A for accuracy, and EVA-X for experience. EVA also includes a set of diagnostic metrics. Unlike the primary metrics, these are not used directly to compare or rank models — rather, they offer granular insight into why a model scores the way it does, helping identify and understand specific failure modes (e.g., ASR, speech synthesis, etc.). We report pass@k (the probability that at least one of k runs succeeds) and pass^k (the probability that all k runs succeed) across three trials per scenario (k = 3), capturing both peak performance and behavioral consistency. EVA uses two evaluation methods: deterministic code-based metrics, which compute scores directly from structured data and are fast; and LLM-as-Judge metrics, which use Large Language Models (LLMs) to assess qualitative aspects of the conversation, or Large Audio Language Models (LALM) to evaluate speech directly. Each judge-based metric uses the model that performs best on a curated evaluation dataset for that specific metric. EVA-A: Accuracy Task completion alone is a necessary but insufficient measure of accuracy. An agent can reach the correct end state while fabricating a policy detail, misreading a confirmation code aloud, or hallucinating a flight number mid-conversation. These failures are invisible to a binary pass/fail check but directly harm users. EVA-A therefore measures three dimensions of accuracy: Task Completion [Deterministic] — Measures whether the agent correctly completed the task by comparing the expected end state of the scenario database against the actual end state after the conversation. Faithfulness [LLM-as-Judge] — Measures whether the agent's responses were grounded in its instructions, policies, user inputs, and tool call results — flagging fabrications, misrepresentations, policy violations, and hallucinations. Speech Fidelity [LALM-as-Judge] — Measures whether the speech system faithfully reproduced the intended text in spoken audio, with particular focus on entities critical to get right in a voice context, such as confirmation codes, flight numbers, and dollar amounts. This is the only metric in any end-to-end voice agent benchmark that evaluates the quality of the agent's own spoken output at the audio level. EVA-X: Experience Turn-taking timing matters, but it tells only part of the story. An agent can have perfect timing while overwhelming a caller with a wall of spoken options they cannot skim, or repeatedly asking for information already given. These failures degrade the experience without ever involving a mistimed response. EVA-X therefore measures three dimensions of experience: Conciseness [LLM-as-Judge] — Measures whether the agent's responses were appropriately brief and focused for spoken delivery, since phone users cannot skim, re-read, or scroll back through long responses. Conversation Progression [LLM-as-Judge] — Measures whether the agent moved the conversation forward effectively — avoiding repetition, retaining context across turns, and driving toward task completion without stalling. Turn-Taking [LLM-as-Judge] — Measures whether the agent spoke at the right time — neither interrupting the user nor introducing excessive silence after they finish speaking. Findings We evaluated 20 systems — proprietary and open-source, cascade and audio-native — and find a consistent accuracy-experience tradeoff: agents that perform well on task completion tend to deliver worse user experiences, and vice versa — a tradeoff invisible to benchmarks that score only task completion. No single configuration dominates both axes, confirming that accuracy and experience must be measured jointly. Additionally, we identified named entity transcription as a dominant failure mode. A single misheard character can cascade into an authentication failure and a full conversation breakdown. Also, multi-step workflows break agents in predictable ways. Rebooking a flight while preserving ancillary services — seats, baggage — is the dominant complexity breaker across all configurations. Finally, we observed that additional calibration is needed for real-world use cases. The gap between pass@3 and pass^3 is substantial across all configurations. Even agents that can complete a task often cannot do so consistently, which is critical for real-world success. View the early results here . Limitations EVA is designed to provide rigorous, end-to-end evaluation of conversational voice agents, but several limitations are important to acknowledge, across the framework, data, and metrics dimensions: Metrics — LLM-as-judge models carry inherent biases and may favor certain response styles independent of quality, with additional risk of systematic bias when the evaluated and judge models share a provider. While we validate our judges against labeled datasets and report accuracy measurements on our website, these alignment scores do not eliminate systematic bias entirely. Additionally, task completion is measured as binary, which does not capture partial credits and may understate the relative quality of systems that fail gracefully versus catastrophically. Simulation — The current release covers 50 English-language scenarios in a single domain (airline); results may not generalize to other domains, languages, or accents. Also, the user simulator may not perfectly replicate real caller behavior (e.g., disfluencies, hesitations, emotions) or guarantee full policy adherence. Framework — The user simulator relies on a single commercial provider whose voice characteristics may systematically favor certain ASR systems, and the bot-to-bot pipeline — including audio format conversions and real-time audio interfaces — may not fully represent production deployments. Also, full reproduction requires commercial API access, and latency measurements will vary across providers and infrastructure. What's Next On the evaluation side, we plan to add prosodic quality assessment (pronunciation, rhythm, expressiveness) — currently an open problem after finding very low alignment between LALM-as-Judge and human judgments. We also plan robustness testing under noisy conditions, diverse accents, multilingual users, and varied speaker behaviors, alongside affect-aware evaluation of how agents respond to user distress. In terms of data, we are developing additional domain datasets — each with distinct policy structures, named entity profiles, and conversational dynamics — and more complex scenarios involving compound requests, multi-step follow-ups, and longer conversational memory. On the tooling front, we will release a results and error analysis application that automatically identifies errors per metric and model, surfaces representative examples for exploration, and generates structured summaries of each model’s strengths and weaknesses. Finally, we intend to expand the leaderboard continuously to provide an up-to-date assessment of voice agent capabilities across the field. View more details about limitations and our upcoming roadmap here . Acknowledgements Core contributors include Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, and Hari Subramani. We also thank Lindsay Brin, Akshay Kalkunte, Joseph Marinier, Jishnu Nair, and Aman Tiwari for their careful data review and thoughtful contributions to the framework, and Fanny Riols, Anil Madamala, Sridhar Nemala, and Srinivas Sunkara for their management, leadership, and support throughout. We also extend our thanks to the PAVA and CLAE ServiceNow teams, whose prior work on evaluations and voice agents provided valuable inspiration for this project. Citation @misc{eva-2026, title={A New End-to-end Framework for Evaluating Voice Agents (EVA)}, author={Bogavelli, Tara and Gauthier Melançon, Gabrielle and Stankiewicz, Katrina and Bamgbose, Oluwanifemi and Nguyen, Hoang and Mehndiratta, Raghav and Subramani, Hari}, year={2026}, url={https://github.com/ServiceNow/eva} }

3月18日周三
  1. OWASP60

    OWASP GenAI Security Project 在 RSA 2026 前发布 LLM 与 Agentic 安全新版指南

    OWASP GenAI Security Project 宣布发布 Q2 2026 Updated Landscape Guide for LLM and Agentic Security，新增更新的厂商与工具生态文档以及新的 agentic 红队分类法，覆盖识别、度量、缓解和治理 AI 风险的全生命周期框架。

    推荐理由：原文列出了新增的 Agentic 红队分类法和 MCP 安全指南等具体产出，读者可据此了解 OWASP 在 Agent 安全方向的框架布局。

3月9日周一
3月6日周五
2月28日周六
  1. 腾讯玄武实验室80

    腾讯玄武实验室提出 Agentic Coding 幽灵依赖供应链威胁并发布 Atuin 防护插件

    腾讯玄武实验室提出 Agentic Coding 范式下的新型供应链安全威胁幽灵依赖，包含版本幽灵（LLM 倾向引入过时组件版本）和名称幽灵（LLM 捏造不存在的组件名）两类风险，并定义了利用过时组件 N-day 漏洞和针对幻觉组件名定向投毒两种攻击方式。

    推荐理由：研究给出了 Agentic Coding 场景下幽灵依赖的威胁模型与实验数据，读者可据此理解 AI 自主供应链决策带来的新型攻击面及防御思路。

2月12日周四
  2. Hugging Face Blog

    OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

    AI agents often perform impressively in controlled research settings, yet struggle when deployed in real-world systems where they must reason across multiple steps, interact with real tools and APIs, operate under partial information, and recover from errors in stateful, permissioned environments—highlighting a persistent gap between research success and production reliability. OpenEnv is an open-source framework from Meta and Hugging Face designed to address this challenge by standardizing how agents interact with real environments. As part of this collaboration, Turing contributed a production-grade calendar management environment to study tool-using agents under realistic constraints such as access control, temporal reasoning, and multi-agent coordination. In this post, we explore how OpenEnv works in practice, why calendars serve as a powerful benchmark for real-world agent evaluation, and what our findings reveal about the current limitations of tool-using agents. What Is OpenEnv? OpenEnv is a framework for evaluating AI agents against real systems rather than simulations . It provides a standardized way to connect agents to real tools and workflows while preserving the structure needed for consistent and reliable evaluation. OpenEnv uses a gym-oriented API ( reset , step , action , observations ) like OpenAI's Gymnasium . Also, OpenEnv uses a standard MCP tool call interface to connect to envs which provides a consistent interface across domains and simulation to production environments. The environments maintain state across multiple actions—enabling long-horizon reasoning—and can connect directly to real APIs and tools such as browsers, code repositories, or calendars. This shifts evaluation from "Can this work in a controlled demo?" to "Can this operate reliably in the real world?" The Calendar Gym: A Production-Grade Benchmark Calendar systems are deceptively complex. While scheduling a meeting seems simple, real-world calendar management requires agents to reason over time, permissions, multiple users, and incomplete information—often across several dependent steps. These properties make calendars a powerful testbed for evaluating tool-using agents outside controlled simulations. To ground OpenEnv in this kind of realistic, demanding use case, Turing built a production-grade calendar management environment referred to as the Calendar Gym . Rather than simulating scheduling in the abstract, it exposes agents to the same constraints they would face in real calendar systems: Access Control Lists across users and calendars, limited visibility into other users' state, and multi-step workflows where actions must be chained in the correct order. Agents interact with a rich set of calendar operations—from listing calendars to modifying events and permissions—and must handle failed actions, incorrect assumptions, and missing permissions. Each session runs in an isolated environment, enabling reliable comparisons across runs. Below is a code example of how to use the Calendar Gym. We explore the environment, discover available tools, list calendars, create an event, and print the result. from openenv_wrapper.client import MCPEnvClient from openenv_wrapper.data_models import MCPAction with MCPEnvClient.from_hub(base_url= "TuringEnterprises/calendar-gym" ) as client: # Connect and reset the environment result = client.reset() print ( "Reset successful:" , result.observation.success) # Discover available tools result = client.step(MCPAction(action_type= "ListToolsAction" )) print ( "Available tools:" , len (result.observation.tools_list)) # List calendars result = client.step(MCPAction( action_type= "ToolCallAction" , tool_name= "calendars_list" , arguments={} )) calendars = result.observation.tool_result[ "items" ] print ( "Calendars:" , calendars) # Create an event result = client.step(MCPAction( action_type= "ToolCallAction" , tool_name= "events_insert" , arguments={ "calendarId" : "primary" , "summary" : "Team Sync" , "start" : { "dateTime" : "2026-01-15T14:00:00Z" }, "end" : { "dateTime" : "2026-01-15T15:00:00Z" } } )) print ( "Event created:" , result.observation.success) Below is an excerpt of what the Calendar Gym returns when you call ListToolsAction . Each entry includes the tool name plus an input schema (what arguments the tool accepts). Click to expand output { "tools_list" : [ { "name" : "calendars_list" , "description" : "List calendars visible to the current user." , "input_schema" : { "type" : "object" , "properties" : { } , "additionalProperties" : false } } , { "name" : "events_insert" , "description" : "Create an event in a calendar." , "input_schema" : { "type" : "object" , "properties" : { "calendarId" : { "type" : "string" } , "summary" : { "type" : "string" } , "start" : { "type" : "object" , "properties" : { "dateTime" : { "type" : "string" } } , "required" : [ "dateTime" ] } , "end" : { "type" : "object" , "properties" : { "dateTime" : { "type" : "string" } } , "required" : [ "dateTime" ] } } , "required" : [ "calendarId" , "summary" , "start" , "end" ] } } ] } What We Learned Evaluating agents in the Calendar Gym revealed consistent patterns which were common across multiple domains. While agents often perform well on individual game like actions, reliability breaks down as tasks become longer, more ambiguous, and more constrained. Multi-step reasoning is the primary bottleneck. Agents struggle to correctly chain actions across longer workflows, suggesting that benchmarks need to test sustained reasoning over multiple dependent steps—not just single tool calls. Ambiguity significantly degrades performance. Agents achieved close to 90% success on tasks with explicit calendar identifiers, but success dropped to roughly 40% when the same tasks were phrased using natural language descriptions. Building stronger lookup and validation into agent loops—rather than relying on the LLM to resolve references unaided—appears essential. Correct tool choice isn't enough. Across failed interactions, more than half of errors stemmed from malformed tool arguments or incorrect ordering, even when the right tool was selected. Reliable agent behavior depends as much on execution quality and structured feedback as on tool selection—environment design matters. These challenges are not unique to scheduling and calendars. They reflect broader limitations that emerge whenever agents operate in changing systems over long periods of time, and they point toward evaluation frameworks that test permissions, partial observability, and multi-step workflows together. Looking Ahead OpenEnv provides a foundation for testing agents under realistic conditions, and the Calendar Gym demonstrates how seemingly simple domains can surface deep challenges in reasoning, ambiguity resolution, and tool use. By evaluating agents where failure is measurable and constraints are real, we gain clearer insight into what it takes to build agents that operate reliably in production. For a deeper dive into the Calendar Gym's design, benchmarking methodology, and quantitative results, explore the full technical article on Turing's site . To explore a clone of the Calendar Gym, visit the Calendar Gym space . Appendix: Common error cases in tool use In practice, tool integrations rarely fail in dramatic ways; they fail in small, predictable ones. When wiring up MCP tools to real APIs (like calendar operations), we encountered a handful of recurring issues. Specific error cases found in the wild Below are three common failure modes we’ve seen in production, along with representative error payloads and mitigation strategies. These examples illustrate not just what can go wrong, but how structured errors can help agents recover gracefully. 1. Schema validation errors (missing or malformed arguments) The agent calls a valid tool (e.g. events_insert ), but the arguments do not match the declared JSON schema. Missing required fields like calendarId Incorrect nesting of start / end Passing a string where an object is expected. Click to expand error payload { "ok" : false , "error_type" : "validation_error" , "tool_name" : "events_insert" , "message" : "Invalid arguments for tool 'events_insert'." , "details" : { "missing_required_fields" : [ "calendarId" , "end" ] , "invalid_fields" : [ { "field" : "start" , "expected_type" : "object" , "received_type" : "string" } ] } } We can mitigate this by providing one canonical example of a correct 'events_insert' call in your prompt. Return structured validation errors so the model can repair and retry instead of failing silently. 2. Permission / authorization errors (401/403) The tool call is syntactically correct, but the API rejects it due to insufficient permissions. Missing OAuth scopes Expired access token User lacks write access to the target calendar Click to expand error payload { "ok" : false , "error_type" : "permission_error" , "tool_name" : "events_insert" , "http_status" : 403 , "message" : "The authenticated user does not have write access to calendar 'primary'." , "remediation" : [ "Ensure the OAuth token includes calendar write scope." , "Verify the user has edit access to the target calendar." , "Reconnect the integration if the token has expired." ] } We can mitigate this by clearly documenting the required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user instead of retrying the same failing call. Clearly document required OAuth scopes. Return structured, actionable remediation steps so the agent can guide the user instead of retrying the same failing call. 3. Datetime / format errors (RFC3339 & timezone issues) The event is rejected by the API, or it is created at an unexpected time. Missing timezone offset Non-RFC3339 datetime format Incorrect nesting of start.dateTime or end.dateTime Mixing local time and UTC without specifying an offset Click to expand error payload { "ok" : false , "error_type" : "format_error" , "tool_name" : "events_insert" , "message" : "Invalid datetime format for field 'start.dateTime'." , "details" : { "received" : "02/11/2026 9:30 AM" , "expected_format" : "RFC3339 (e.g. 2026-02-11T09:30:00-05:00)" } } We can mitigate this by standardizing on RFC3339 with explicit timezone offsets (e.g. 2026-02-11T09:30:00-05:00). Include at least one correct datetime example in your documentation to anchor model behavior and reduce repair retries.

2月5日周四
2月4日周三
  1. Hugging Face Blog

    Community Evals: Because we're done trusting black-box leaderboards over the community

    TL;DR: Benchmark datasets on Hugging Face can now host leaderboards. Models store their own eval scores. Everything links together. The community can submit results via PR. Verified badges prove that the results can be reproduced. Evaluation is broken Let's be real about where we are with evals in 2026. MMLU is saturated above 91%. GSM8K hit 94%+. HumanEval is conquered. Yet some models that ace benchmarks still can't reliably browse the web, write production code, or handle multi-step tasks without hallucinating, based on usage reports. There is a clear gap between benchmark scores and real-world performance. Furthermore, there is another gap within reported benchmark scores. Multiple sources report different results. From Model Cards, to papers, to evaluation platforms, there is no alignment in reported scores. The result is that the community lacks a single source of truth. What We're Shipping Decentralized and transparent evaluation reporting. We are going to take evaluations on the Hugging Face Hub in a new direction by decentralizing reporting and allowing the entire community to openly report scores for benchmarks. At first, we will start with a shortlist of 4 benchmarks and over time we’ll expand to the most relevant benchmarks. For Benchmarks: Dataset repos can now register as benchmarks ( MMLU-Pro , GPQA , HLE are already live). They automatically aggregate reported results from across the Hub and display leaderboards in the dataset card. The benchmark defines the eval spec via eval.yaml , based on the Inspect AI format, so anyone can reproduce it. The reported results need to align with the task definition. For Models: Eval scores live in .eval_results/*.yaml in the model repo. They appear on the model card and are fed into benchmark datasets. Both the model author’s results and open pull requests for results will be aggregated. Model authors will be able to close score PR and hide results. For the Community: Any user can submit evaluation results for any model via a PR. Results get shown as "community", without waiting for model authors to merge or close. The community can link to sources like a paper, Model Card, third-party evaluation platform, or inspect eval logs. The community can discuss scores like any PR. Since the Hub is Git based, there is a history of when evals were added, when changes were made, etc. The sources look like below. To learn more about evaluation results, check out the docs . Model scores in the Hub Why This Matters Decentralizing evaluation will expose scores that already exist across the community in sources like model cards and papers. By exposing these scores, the community can build on top of them to aggregate, track, and understand scores across the field. Also, all scores will be exposed via Hub APIs, making it easy to aggregate and build curated leaderboards, dashboards, etc. Community evals do not replace benchmarks so leaderboards and closed evals with published results are still crucial. However, we believe it's important to contribute to the field with open eval results based on reproducible eval specs. This won't solve benchmark saturation or close the benchmark-reality gap. Nor will it stop training on test sets. But it makes the game visible by exposing what is evaluated, how, when, and by whom. Mostly, we hope to make the Hub an active place to build and share reproducible benchmarks. Particularly focusing on new tasks and domains that challenge SOTA models more. Get Started Read the docs: To learn more about evaluation results, check out the docs . Add eval results: Publish the evals you conducted as YAML files in .eval_results/ on any model repo. Check out the scores on the benchmark dataset . Register a new benchmark: Add eval.yaml to your dataset repo and contact us to be included in the shortlist. The feature is in beta. We're building in the open. Feedback welcome.

2月2日周一
  1. 腾讯玄武实验室86

    腾讯玄武实验室发布《AI网络爬虫安全白皮书》并开源SEChrome防护方案

    腾讯玄武实验室发布《AI网络爬虫安全白皮书》，系统分析AI时代服务端浏览器/爬虫的攻击面，指出多款产品存在沙箱外远程代码执行（RCE）风险，相关产品服务端用户规模累计超10亿，涵盖多家头部科技企业。

    推荐理由：白皮书梳理了服务端浏览器的完整攻击链与四个真实RCE案例，并给出可复用的纵深防御框架和开源方案，工程落地参考价值高。

1月27日周二
  1. Hugging Face Blog

    Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs

    Arabic is one of the most widely spoken languages in the world, with hundreds of millions of speakers across more than twenty countries. Despite this global reach, Arabic is not a monolithic language. Modern Standard Arabic coexists with a rich landscape of regional dialects that differ significantly in vocabulary, syntax, phonology, and cultural grounding. These dialects are the primary medium of daily communication, oral storytelling, poetry, and social interaction. However, most existing benchmarks for Arabic large language models focus almost exclusively on Modern Standard Arabic, leaving dialectal Arabic largely under-evaluated and under-represented. This gap is particularly problematic as large language models increasingly interact with users in informal, culturally grounded, and conversational settings. A model that performs well on formal newswire text may still fail to understand a greeting, an idiomatic expression, or a short anecdote expressed in a local dialect. To address this limitation, our team introduces Alyah الياه (which means North Star ⭐️ in Emirati), an Emirati-centric benchmark designed to assess how well Arabic LLMs capture the linguistic, cultural, and pragmatic aspects of the Emirati dialect. Benchmark Motivation and Scope The Emirati dialect is deeply intertwined with local culture, heritage, and history. It appears in everyday greetings, oral poetry, proverbs, folk narratives, and expressions whose meanings cannot be inferred through literal translation alone. Our benchmark is intentionally designed to probe this depth. Rather than testing surface-level lexical knowledge, it challenges models on their ability to interpret culturally embedded meaning, pragmatic usage, and dialect-specific nuances. The benchmark covers a wide range of content, including common and uncommon local expressions, culturally grounded greetings, short anecdotes, heritage-related questions, and references to Emirati poetry. The goal is not only to measure correctness, but also to understand where models systematically succeed or fail when confronted with authentic Emirati language use. Dataset Structure Following further development and consolidation, the benchmark has been unified into a single dataset called Alyah . The final benchmark contains 1,173 samples , all collected manually from native Emirati speakers to ensure linguistic authenticity and cultural grounding. This manual curation step was essential to capture expressions, meanings, and usages that are rarely documented in written resources and are difficult to infer from Modern Standard Arabic alone. Each sample is formulated as a multiple-choice question with four candidate answers , exactly one of which is correct. Large language models were used to synthetically generate the distractor choices, after which they were reviewed to ensure plausibility and semantic closeness to the correct answer. To avoid positional bias during evaluation, the index of the correct answer follows a randomized distribution across the dataset. Below is the distribution of word count per query and candidate answers. Alyah spans a broad spectrum of linguistic and cultural phenomena in the Emirati dialect, ranging from everyday expressions to culturally sensitive and figurative language. The distribution across categories is summarized below. Category Number of Samples Difficulty Greetings & Daily Expressions 61 Easy Religious & Social Sensitivity 78 Medium Imagery & Figurative Meaning 121 Medium Etiquette & Values 173 Medium Poetry & Creative Expression 32 Difficult Historical & Heritage Knowledge 89 Difficult Language & Dialect 619 Difficult Below are examples of each category: This composition allows Alyah to jointly evaluate surface-level conversational fluency and deeper cultural, semantic, and pragmatic understanding, with a particular emphasis on dialect-specific language phenomena that remain challenging for current models. Model Evaluation Setup We evaluated a total of 54 language models , comprising 23 base models and 31 instruction-tuned models , spanning several architectural and training paradigms. These include Arabic-native LLMs such as Jais and Allam, multilingual models with strong Arabic support such as Qwen and LLaMA, and adapted or regionally specialized models such as Fanar and AceGPT. For each family, both base and instruction-tuned variants were evaluated in order to understand the impact of alignment and instruction tuning on dialectal performance. All models were evaluated under a consistent prompting and scoring protocol. Responses were assessed for semantic correctness and appropriateness with respect to Emirati usage, rather than literal overlap with a reference answer. This is particularly important for dialectal evaluation, where multiple valid phrasings may exist. For each question category, we estimated difficulty empirically based on model performance. Categories where most models struggled were labeled as harder, while those consistently answered correctly across model families were considered easier. This approach allows difficulty to emerge from observed behavior rather than from subjective annotation alone. Evaluation Results on Alyah (Emirati Dialect) We evaluate a broad set of contemporary Arabic and multilingual large language models on Alyah , using accuracy on multiple-choice questions as the primary metric. The evaluation covers 53 models in total, including 22 base models and 31 instruction-tuned models , spanning Arabic-native, multilingual, and regionally adapted systems. Below is a radar plot showing the performance of top models of different sizes per question category. These results are intended as reference measurements within the scope of Alyah, rather than absolute rankings across all Arabic benchmarks. Base Models Model Accuracy google/gemma-3-27b-pt 74.68 tiiuae/Falcon-H1-34B-Base 73.66 FreedomIntelligence/AceGPT-v2-32B 67.35 google/gemma-3-4b-pt 63.17 QCRI/Fanar-1-9B 62.75 tiiuae/Falcon-H1-7B-Base 60.78 meta-llama/Llama-3.1-8B 58.23 Qwen/Qwen3-14B-Base 57.29 inceptionai/jais-adapted-13b 56.01 Qwen/Qwen2.5-32B 53.03 FreedomIntelligence/AceGPT-13B 50.81 Qwen/Qwen2.5-72B 47.91 Qwen/Qwen2.5-14B 46.8 google/gemma-2-2b 41.86 tiiuae/Falcon3-7B-Base 41.43 Qwen/Qwen3-8B-Base 40.75 tiiuae/Falcon-H1-3B-Base 40.41 Qwen/Qwen2.5-7B 36.57 Qwen/Qwen2.5-3B 35.29 meta-llama/Llama-3.2-3B 35.12 inceptionai/jais-adapted-7b 33.5 Qwen/Qwen3-4B-Base 27.45 Instruction-Tuned Models Model Accuracy falcon-h1-arabic-7b-instruct 82.18 humain-ai/ALLaM-7B-Instruct-preview 77.24 google/gemma-3-27b-it 74.68 falcon-h1-arabic-3b-instruct 74.51 Qwen/Qwen2.5-72B-Instruct 74.6 CohereForAI/aya-expanse-32b 73.66 Navid-AI/Yehia-7B-preview 73.32 FreedomIntelligence/AceGPT-v2-32B-Chat 72.8 Qwen/Qwen2.5-32B-Instruct 71.61 tiiuae/Falcon-H1-34B-Instruct 71.1 meta-llama/Llama-3.3-70B-Instruct 69.74 QCRI/Fanar-1-9B-Instruct 69.22 tiiuae/Falcon-H1-7B-Instruct 65.13 CohereForAI/c4ai-command-r7b-arabic-02-2025 64.54 silma-ai/SILMA-9B-Instruct-v1.0 63.94 FreedomIntelligence/AceGPT-v2-8B-Chat 63.43 CohereLabs/aya-expanse-8b 61.21 yasserrmd/kallamni-2.6b-v1 61.13 yasserrmd/kallamni-4b-v1 60.7 microsoft/Phi-4-mini-instruct 58.57 tiiuae/Falcon-H1-3B-Instruct 57.12 silma-ai/SILMA-Kashif-2B-Instruct-v1.0 48.51 Qwen/Qwen2.5-7B-Instruct 45.44 google/gemma-3-4b-it 46.12 meta-llama/Llama-3.1-8B-Instruct 46.29 meta-llama/Llama-3.2-3B-Instruct 39.64 yasserrmd/kallamni-1.2b-v1 37.77 Qwen/Qwen3-4B 26.26 google/gemma-2-2b-it 26.00 Qwen/Qwen3-14B 26.00 Qwen/Qwen3-8B 25.66 Analysis and Observed Trends Figure 1: Models' accuracy across categories based on size. Figure 2: Models' accuracy across categories based on language. Several trends emerge from the evaluation. Instruction-tuned models generally outperform their base counterparts as shown in Figures 1 and 2. This is particularly the case on questions involving conversational norms and culturally appropriate responses (i.e. the Etiquette & Values Category). Furthermore, it is the case with questions that test imagery and figurative meaning. This can be attributed, to the model’s original strong capabilities with understanding MSA-based imagery and figurative language regardless of the dialect at hand. The models are able to draw patterns of non-literal description regardless of dialect. Generally, the most difficult categories for the models were consistently “Language and Dialect” and “Greeting and Daily expressions” across model sizes as shown in figure 1. These results reflect the current state of Emirati dialect presence in written media, as the dialect is mostly spoken rarely written, which explains its novelty relative to the evaluated models. Nonetheless, there is a clear benefit to instruct models with understanding the dialect (and the other evaluation categories) in comparison to their counterparts, especially in small and medium models. This is particularly noticeable with the Poetry and Creative Expression category, which is where the large instruct models performed marginally better than the smaller models. Figure 3: Evaluated models average accuracy. As shown in Figure 3, even strong multilingual models show notable degradation on the most challenging Alyah questions, suggesting that dialect-specific semantic knowledge is not easily acquired through generic multilingual training alone. It must be noted that while Arabic-native models tend to perform more robustly on culturally grounded content, their performances are not uniform across all categories (figure 2). In particular, questions involving implicit meanings and rare expressions remain difficult across nearly all evaluated models. This highlights a persistent gap between surface-level dialect familiarity and deeper cultural understanding. The high variance in performance across categories , where a model that excels at imagery and figurative meaning may still struggle with poetry or heritage-related creative questions, indicates that dialectal competence is multi-dimensional and cannot be captured by a single score. Figure 3 shows that the highest scoring large model in Jais-2-70B, followed by the two small models jais-2-8B and ALLaM-7B-instruct, which are all Arabic instruct-tuned models. Conclusion and Community Impact This benchmark represents a step toward more realistic and culturally grounded evaluation of Arabic language models. By focusing on the Emirati dialect, we aim to support the development of models that better serve local communities, institutions, and users in the UAE. Beyond model ranking, the benchmark is intended as a diagnostic tool to guide future data collection, training, and adaptation efforts. We invite researchers, practitioners, and the broader community to use the benchmark, explore the results, and share feedback. Community input will be essential to refining the dataset, expanding coverage, and ensuring that dialectal Arabic receives the attention it deserves in the evaluation of Large Language Models. Citation @misc{emirati_dialect_benchmark_2026, title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models}, author={Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid}, year = {2026}, month = {january}, }

1月21日周三
  1. Hugging Face Blog

    AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality

    AssetOpsBench is a comprehensive benchmark and evaluation system with six qualitative dimensions that bridges the gap for agentic AI in domain-specific settings, starting with industrial Asset Lifecycle Management. Introduction While existing AI benchmarks excel at isolated tasks such as coding or web navigation, they often fail to capture the complexity of real-world industrial operations. To bridge this gap, we introduce AssetOpsBench , a framework specifically designed to evaluate agent performance across six critical dimensions of industrial applications. Unlike traditional benchmarks, AssetOpsBench emphasizes the need for multi-agent coordination—moving beyond `lone wolf' models to systems that can handle complex failure modes, integrate multiple data streams, and manage intricate work orders. By focusing on these high-stakes, multi-agent dynamics, the benchmark ensures that AI agents are assessed on their ability to navigate the nuances and safety-critical demands of a true industrial environment. AssetOpsBench is built for asset operations such as chillers and air handling units. It comprises: 2.3M sensor telemetry points 140+ curated scenarios across 4 agents 4.2K work orders for diverse scenarios 53 structured failure modes Experts helped curate 150+ scenarios. Each scenario includes metadata: task type, output format, category, and sub-agents. The tasks designed span across: Anomaly detection in sensor streams Failure mode reasoning and diagnostics KPI forecasting and analysis Work order summarization and prioritization Evaluation Framework and Overall Feedback AssetOpsBench evaluates agentic systems across six qualitative dimensions designed to reflect real operational constraints in industrial asset management. Rather than optimizing for a single success metric, the benchmark emphasizes decision trace quality, evidence grounding, failure awareness, and actionability under incomplete and noisy data. Each agent run is scored across six criteria: Task Completion Retrieval Accuracy Result Verification Sequence Correctness Clarity and Justification Hallucination rate Across early evaluations, we observe that many general-purpose agents perform well on surface-level reasoning but struggle with sustained multi-step coordination involving work orders, failure semantics, and temporal dependencies. Agents that explicitly model operational context and uncertainty tend to produce more stable and interpretable trajectories, even when final task completion is partial. This feedback-oriented evaluation is intentional: in industrial settings, understanding why an agent fails is often more valuable than a binary success signal. Failure Modes in Industrial Agentic Workflows A central contribution of AssetOpsBench is the explicit treatment of failure modes as first-class evaluation signals in agentic industrial workflows. Rather than treating failure as a binary outcome, AssetOpsBench analyzes full multi-agent execution trajectories to identify where , how , and why agent behavior breaks down under realistic operational constraints. Failure analysis in AssetOpsBench is implemented through a dedicated trajectory-level pipeline ( TrajFM ), which combines LLM-based reasoning with statistical clustering to surface interpretable failure patterns from agent execution traces. This pipeline operates in three stages: (1) trajectory-level failure extraction using an LLM-guided diagnostic prompt, (2) embedding-based clustering to group recurring failure patterns, and (3) analysis and visualization to support developer feedback and iteration. Across industrial scenarios, recurrent failure modes include: Misalignment between sensor telemetry, alerts, and historical work orders Overconfident conclusions drawn under missing, delayed, or insufficient evidence Inconsistent aggregation of heterogeneous data modalities across agents Premature action selection without adequate verification or validation steps Breakdowns in multi-agent coordination, such as ignored inputs or action–reasoning mismatches Importantly, AssetOpsBench does not rely solely on a fixed, hand-crafted failure taxonomy. While a structured set of predefined failure categories (e.g., verification errors, step repetition, role violations) is used for consistency, the system is explicitly designed to discover new failure patterns that emerge in practice. Additional failure modes identified by the LLM are embedded and clustered automatically, allowing the taxonomy to evolve as new agent designs and behaviors are evaluated. To preserve industrial confidentiality, raw execution traces are never exposed. Instead, agents receive aggregated scores across six evaluation dimensions together with clustered failure-mode summaries that explain why an agent failed, without revealing sensitive data or intermediate reasoning steps. This feedback-driven design enables developers to diagnose weaknesses, refine agent workflows, and iteratively resubmit improved agents. This failure-aware evaluation reflects the realities of industrial asset management, where cautious, degradation-aware reasoning—and the ability to recognize uncertainty, defer action, or escalate appropriately—is often preferable to aggressive but brittle automation. Submit an Agent for Evaluation AssetOpsBench-Live is designed as an open, competition-ready benchmark , and we welcome submissions of agent implementations from the community. Agents are evaluated in a controlled, privacy-preserving environment that reflects real industrial asset management constraints. To submit an agent, developers first validate their implementation locally using a provided simulated environment, which includes representative sensor data, work orders, alerts, and failure-mode catalogs. Agents are then containerized and submitted for remote execution on hidden evaluation scenarios. Submitted agents are evaluated across six qualitative dimensions—task completion, accuracy, result verification, action sequencing, clarity, and hallucination—using a consistent, reproducible evaluation protocol. Execution traces are not exposed; instead, participants receive aggregated scores and structured failure-mode feedback that highlights where and why an agent’s reasoning or coordination broke down. This feedback-driven evaluation loop enables iterative improvement: developers can diagnose failure patterns, refine agent design or workflow structure, and resubmit updated agents for further evaluation. Both planning-focused and execution-focused agents are supported, allowing researchers and practitioners to explore diverse agentic designs within the same benchmark framework. Experiment and Observations We performed a community evaluation where we tested two tracks: Planning-oriented multi-agent orchestration Execution-oriented dynamic multi-agent workflow. Across 225 users and 300+ agents and leading open source models, here are the observations: Model Family Best Planning Score Best Execution Score Key Limitation GPT-4.1 68.2 72.4 Hallucinated completion on complex workflows Mistral-Large 64.7 69.1 Struggled with multi-hop tool sequences LLaMA-4 Maverick 66.0 70.8 Missed clarifying questions (fixable) LLaMA-3-70B 52.3 58.9 Collapsed under multi-agent coordination Note: None of the models could pass our evaluation criteria benchmark and get 85 points, which is the threshold for deployment readiness. Distribution of Failures Across 881 agent execution traces, failure distribution was as follows: Ineffective Error Recovery: 31.2% Overstated Completion: 23.8% Formatting Issues: 21.4% Unhandled Tool Errors: 10.3% Ignored Feedback: 8.0% Other: 5.3% Beyond this, 185 traces had one new failure pattern and 164 had multiple novel failures. Key Error Findings "Sounds Right, Is Wrong": Agents claim to have completed tasks (23.8%) and output success even after unsuccessful failure recovery (31.2%). AssetOps benchmarking is important to uncover this so that operators do not act upon incorrect information. Tool Usage: This is the biggest differentiator between high and low performing agents, with top agents having 94% tool accuracy compared to 61% of low performers. Multi-agent Multiplies Failures: Task accuracy between single agent (68%) vs multi-agent (47%) shows the complexity multi-agent brings with context loss, asynchronous issues, and cascaded failures. Domain Knowledge: Agents with access to failure mode databases and maintenance manuals performed better. However, RAG knowledge wasn’t always used correctly, suggesting a need for structured reasoning. Ambiguity: Missing sensors, conflicting logs, and vague operator descriptions caused the success rate to drop 34%. Agents must have clarification strategies embedded. Where to get started? Read our technical report AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance How to run AssetOpsBench locally - Video AssetOpsBench Local Execution Try out AssetOpsBench in the HuggingFace Space Playground Find More Detail AssetOpsBench GitHub , fork the repo and get started.

1月16日周五
  1. Anthropic72

    Anthropic 红队：Claude Sonnet 4.5 在真实网络靶场中自主利用公开 CVE

    Anthropic 前沿红队与 Incalmo 合作，在 25-50 台主机的模拟网络靶场上评测 Claude，发现 Claude Sonnet 4.5 在少数网络上无需此前几代所需的定制网络工具包即可成功。

    推荐理由：Anthropic 红队披露 Claude Sonnet 4.5 在真实网络靶场中无需专用工具即可自主利用公开 CVE，并给出与上一代的对照数据。

1月14日周三
1月12日周一
1月8日周四
1月6日周二
  1. 腾讯玄武实验室62

    腾讯玄武实验室披露 ComfyUI-Manager 远程代码执行漏洞 CVE-2025-67303

    腾讯玄武实验室发现 ComfyUI 官方扩展组件 ComfyUI-Manager 存在高危漏洞 CVE-2025-67303，远程攻击者可在无需账号的情况下实现远程代码执行并完全控制服务器，包括盗取用户数据和私有 AI 模型。

    推荐理由：通告给出了漏洞根因、影响版本与迁移步骤，读者可据此判断自身 ComfyUI 部署是否受影响并完成修复。

12月23日周二
  1. Hugging Face Blog76

    ServiceNow 发布 8B 安全护栏模型 AprielGuard，覆盖 Agent 工作流攻击检测

    ServiceNow-AI 在 Hugging Face 发布 AprielGuard，一个 8B 参数的安全护栏模型，用于检测 16 类安全风险（涵盖毒性、仇恨、虚假信息、自残、非法活动等）以及提示词注入、越狱、思维链污染、上下文劫持、记忆投毒和多智能体利用链等对抗攻击，同时支持对 Agent 工作流中的工具调用与推理轨迹进行检测。

12月19日周五
12月18日周四
  1. Anthropic55

    Anthropic 公布 Project Vend 第二阶段：Claude 经营小店实验

    Anthropic 公布 Project Vend 第二阶段结果，Claudius 从 Claude Sonnet 3.7 升级到 Sonnet 4.0 和 4.5，并接入 CRM、库存管理、网页浏览等工具，还新增 CEO 智能体 Seymour Cash 和周边制作智能体 Clothius，业务表现明显改善，亏损周基本消除，门店扩展到旧金山、纽约和伦敦三地。

12月17日周三
  1. Hugging Face Blog

    The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator

    It has become increasingly challenging to assess whether a model’s reported improvements reflect genuine advances or variations in evaluation conditions, dataset composition, or training data that mirrors benchmark tasks. The NVIDIA Nemotron approach to openness addresses this by publishing transparent and reproducible evaluation recipes that make results independently verifiable. NVIDIA released Nemotron 3 Nano 30B A3B with an explicitly open evaluation approach to make that distinction clear. Alongside the model card, we are publishing the complete evaluation recipe used to generate the results, built with the NVIDIA NeMo Evaluator library, so anyone can rerun the evaluation pipeline, inspect the artifacts, and analyze the outcomes independently. We believe that open innovation is the foundation of AI progress. This level of transparency matters because most model evaluations omit critical details. Configs, prompts, harness versions, runtime settings, and logs are often missing or underspecified, and even small differences in these parameters can materially change results. Without a complete recipe, it’s nearly impossible to tell whether a model is genuinely more intelligent or simply optimized for a benchmark. This blog shows developers exactly how to reproduce the evaluation behind Nemotron 3 Nano 30B A3B using fully open tools, configurations, and artifacts. You’ll learn how the evaluation was run, why the methodology matters, and how to execute the same end-to-end workflow using the NeMo Evaluator library so you can verify results, compare models consistently, and build transparent evaluation pipelines of your own. Building a consistent and transparent evaluation workflow with NeMo Evaluator A single, consistent evaluation system Developers and researchers need evaluation workflows they can rely on, not one-off scripts that behave differently from model to model. NeMo Evaluator provides a unified way to define benchmarks, prompts, configuration, and runtime behavior once, then reuse that methodology across models and releases. This avoids the common scenario where the evaluation setup quietly changes between runs, making comparisons over time difficult or misleading. Methodology independent of inference setup Model outputs can vary by inference backend and configuration, so evaluation tools should never be tied to a single inference solution. Locking an evaluation tool to one inference solution would limit its usefulness. NeMo Evaluator avoids this by separating the evaluation pipeline from the inference backend, allowing the same configuration to run against hosted endpoints, local deployments, or third-party providers. This separation enables meaningful comparisons even when you change infrastructure or inference engines. Built to scale beyond one-off experiments Many evaluation pipelines work once and then break down as the scope expands. NeMo Evaluator is designed to scale from quick, single-benchmark validation to full model card suites and repeated evaluations across multiple models. The launcher, artifact layout, and configuration model support ongoing workflows, not just isolated experiments, so teams can maintain consistent evaluation practices over time. Auditability with structured artifacts and logs Transparent evaluation requires more than final scores. Each evaluation run produces structured results and logs by default, making it easy to inspect how scores were computed, understand score calculations, debug unexpected behavior, and conduct deeper analysis. Each component of the evaluation is captured and reproducible. A shared evaluation standard By releasing Nemotron 3 Nano 30B A3B with its full evaluation recipe , NVIDIA is providing a reference methodology that the community can run, inspect, and build upon. Using the same configuration and tools brings consistency to how benchmarks are selected, executed, and interpreted, enabling more reliable comparisons across models, providers, and releases. Open evaluation for Nemotron 3 Nano Open evaluation means publishing not just the final results, but the full methodology behind them, so benchmarks are run consistently, and results can be compared meaningfully over time. For Nemotron 3 Nano 30B A3B , this includes open‑source tooling, transparent configurations, and reproducible artifacts that anyone can run end‑to‑end. Open-source model evaluation tooling NeMo Evaluator is an open-source library designed for robust, reproducible, and scalable evaluation of generative models. Instead of introducing yet another standalone benchmark runner, it acts as a unifying orchestration layer that brings multiple evaluation harnesses under a single, consistent interface. Under this architecture, NeMo Evaluator integrates and coordinates hundreds of benchmarks from many widely used evaluation harnesses, including NeMo Skills for Nemotron instruction-following, tool use, and agentic evaluations, as well as the LM Evaluation Harness for base model and pre-training benchmarks, and many more ( full benchmark catalog ). Each harness retains its native logic, datasets, and scoring semantics, while NeMo Evaluator standardizes how they are configured, executed, and logged. This provides two practical advantages: teams can run diverse benchmark categories using a single configuration without rewriting custom evaluation scripts, and results from different harnesses are stored and inspected in a consistent, predictable way, even when the underlying tasks differ. The same orchestration framework used internally by NVIDIA’s Nemotron research and model‑evaluation teams is now available to the community, enabling developers to run heterogeneous, multi‑harness evaluations through a shared, auditable workflow. Open configurations We published the exact YAML configuration used for the Nemotron 3 Nano 30B A3B model card evaluation with NeMo Evaluator. This includes: model inference and deployment settings benchmark and task selection benchmark-specific parameters such as sampling, repeats, and prompt templates runtime controls including parallelism, timeouts, and retries output paths and artifact layout Using the same configuration means running the same evaluation methodology. Open logs and artifacts Each evaluation run produces structured, inspectable outputs, including per‑task results.json files, execution logs for debugging and auditability, and artifacts organized by task for easy comparison. This structure makes it possible to understand not only the final scores, but also how those scores were produced and to perform deeper analysis of model behavior. The reproducibility workflow Reproducing Nemotron 3 Nano 30B A3B model card results follows a simple loop: Start from the released model checkpoint or hosted endpoint Use the published NeMo Evaluator config Execute the evaluation with a single CLI command Inspect logs and artifacts, and compare results to the model card The same workflow applies to any model you evaluate using NeMo Evaluator. You can point the evaluation at a hosted endpoint or a local deployment, including common inference providers such as HuggingFace , build.nvidia.com , and OpenRouter . The key requirement is access to the model, either as weights you can serve or as an endpoint you can call. For this tutorial, we use the hosted endpoint on build.nvidia.com . Reproducing Nemotron 3 Nano benchmark results This tutorial reproduces the evaluation results for NVIDIA Nemotron 3 Nano 30B A3B using NeMo Evaluator. The step-by-step tutorial, including the published configs used for the model card evaluation , is available on GitHub. Although we have focused this tutorial on the Nemotron 3 Nano 30B A3B, we also published recipes for the base model evaluation . This walkthrough runs a comprehensive evaluation suite of the published configs used for the model card evaluation for NVIDIA Nemotron 3 Nano 30B A3B using the following benchmarks: Benchmark Accuracy Category Description BFCL v4 53.8 Function Calling Berkeley Function Calling Leaderboard v4 LiveCodeBench (v6 2025-08–2025-05) 68.3 Coding Real-world coding problems evaluation MMLU-Pro 78.3 Knowledge Multi-task language understanding (10-choice) GPQA 73.0 Science Graduate-level science questions AIME 2025 89.1 Mathematics American Invitational Mathematics Exam SciCode 33.3 Scientific Coding Scientific programming challenges IFBench 71.5 Instruction Following Instruction following benchmark HLE 10.6 Humanity's Last Exam Expert-level questions across domains For Model Card details, see the NVIDIA Nemotron 3 Nano 30B A3B Model Card . For a deep dive into the architecture, datasets, and benchmarks, read the full Nemotron 3 Nano Technical Report . 1. Install NeMo Evaluator Launcher pip install nemo-evaluator-launcher 2. Set required environment variables # NVIDIA endpoint access export NGC_API_KEY="your-ngc-api-key" # Hugging Face access export HF_TOKEN="your-huggingface-token" # Required only for judge-based benchmarks such as HLE export JUDGE_API_KEY="your-judge-api-key" Optional but recommended for faster reruns: export HF_HOME="/path/to/your/huggingface/cache" 3. Model endpoint The evaluation uses the NVIDIA API endpoint hosted on build.nvidia.com : target: api_endpoint: model_id: nvidia/nemotron-nano-3-30b-a3b url: https://integrate.api.nvidia.com/v1/chat/completions api_key_name: NGC_API_KEY Evaluations can be run against common inference providers such as HuggingFace , build.nvidia.com , or OpenRouter , or anywhere that the model has an available endpoint. If you're hosting the model locally or using a different endpoint: nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o target.api_endpoint.url=http://localhost:8000/v1/chat/completions 4. Run the full evaluation suite Preview the run without executing using --dry-run : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ --dry-run From the examples directory, run the evaluation using the YAML configuration provided: nemo-evaluator-launcher run \ --config /path/to/examples/nemotron/local_nvidia_nemotron_3_nano_30b_a3b.yaml Note that for quick testing, you can limit the number of samples by setting limit_samples : nemo-evaluator-launcher run \ --config local_nvidia_nemotron_3_nano_30b_a3b.yaml \ -o evaluation.nemo_evaluator_config.config.params.limit_samples=10 5. Running an individual benchmark You can run specific benchmarks using the -t flag (from the examples/nemotron directory): # Run only MMLU-Pro nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_mmlu_pro # Run only coding benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_livecodebench # Run multiple specific benchmarks nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_gpqa -t ns_aime2025 6. Monitor execution and inspect results # Check status of a specific job nemo-evaluator-launcher status # Stream logs for a specific job nemo-evaluator-launcher logs <job-id> Results are written to the defined output directory: results_nvidia_nemotron_3_nano_30b_a3b/ ├── artifacts/ │ └── <task_name>/ │ └── results.json └── logs/ └── stdout.log Interpreting results When reproducing evaluations, you may observe small differences in final scores across runs. This variance reflects the probabilistic nature of LLMs rather than an issue with the evaluation pipeline. Modern evaluation introduces several sources of non‑determinism: decoding settings, repeated trials, judge‑based scoring, parallel execution, and differences in serving infrastructure. All of which can lead to slight fluctuations. The purpose of open evaluation is not to force bit-wise identical outputs, but to deliver methodological consistency with clear provenance of evaluation results. To ensure your evaluation aligns with the reference standard, verify the following: Configuration : use the published NeMo Evaluator YAML without modification, or document any changes explicitly Benchmark selection : run the intended tasks, task versions, and prompt templates Inference target : verify you are evaluating the intended model and endpoint, including chat template behavior and reasoning settings when relevant Execution settings : keep runtime parameters consistent, including repeats, parallelism, timeouts, and retry behavior Outputs : confirm artifacts and logs are complete and follow the expected structure for each task When these elements are consistent, your results represent a valid reproduction of the methodology, even if individual runs differ slightly. NeMo Evaluator simplifies this process, tying benchmark definitions, prompts, runtime settings, and inference configuration into a single auditable workflow to minimize inconsistencies. Conclusion: A more transparent standard for open models The evaluation recipe released alongside Nemotron 3 Nano represents a meaningful step toward a more transparent and reliable approach to open-model evaluation. We are moving away from evaluation as a collection of bespoke, "black box" scripts, and towards a defined system where benchmark selection, prompts, and execution semantics are encoded into a transparent workflow. For developers and researchers, this transparency changes what it means to share results. A score is only as trustworthy as the methodology behind it and making that methodology public is what enables the community to verify claims, compare models fairly, and continue building on shared foundations. With open evaluation configurations, open artifacts, and open tooling, Nemotron 3 Nano demonstrates what that commitment to openness looks like in practice. NeMo Evaluator supports this shift by providing a consistent benchmarking methodology across models, releases, and inference environments. The objective isn’t identical numbers on every run; it’s confidence in an evaluation methodology that is explicit, inspectable, and repeatable. And for organizations that need automated or large‑scale evaluation pipelines, a separate microservice offering provides an enterprise‑ready NeMo Evaluator microservice built on the same evaluation principles. Use the published NeMo Evaluator evaluation configuration for an end-to-end walkthrough of the evaluation recipe. Join the Community! NeMo Evaluator is fully open source, and community input is essential to shaping the future of open evaluation. If there’s a benchmark you’d like us to support or an improvement you want to propose, open an issue, or contribute directly on GitHub. Your contributions help strengthen the ecosystem and advance a shared, transparent standard for evaluating generative models.

12月8日周一
12月1日周一
  1. Anthropic82

    Anthropic 红队发布 SCONE-bench：AI 智能体在模拟中挖出价值 460 万美元的智能合约漏洞

    Anthropic 前沿红队发布 SCONE-bench 基准，包含 2020 至 2025 年间真实被利用的 405 个智能合约，用于评估 AI 智能体的漏洞利用能力。

    推荐理由：Anthropic 红队用真实被黑合约构建基准，把智能体攻击能力换算成美元损失，并给出成本与收益的量化对照。

11月21日周五
  1. Hugging Face Blog

    Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks

    While everyone (and their grandma 👵) is spinning up new ASR models, picking the right one for your use case can feel more overwhelming than choosing your next Netflix show. As of 21 Nov 2025, there are 150 Audio-Text-to-Text and 27K ASR models on the Hub 🤯 Most benchmarks focus on short-form English transcription (<30s), and overlook other important tasks, such as (1) multilingual performance and (2) model throughput, which can a be deciding factor for long-form audio like meetings and podcasts. Over the past two years, the Open ASR Leaderboard has become a standard for comparing open and closed-source models on both accuracy and efficiency . Recently, multilingual and long-form transcription tracks have been added to the leaderboard 🎉 TL;DR - Open ASR Leaderboard 📝 New preprint on ASR trends from the leaderboard: https://hf.co/papers/2510.06961 🧠 Best accuracy: Conformer encoder + LLM decoders (open-source ftw 🥳) ⚡ Fastest: CTC / TDT decoders 🌍 Multilingual: Comes at the cost of single-language performance ⌛ Long-form: Closed-source systems still lead (for now 😉) 🧑‍💻 Fine-tuning guides ( Parakeet , Voxtral , Whisper ): to continue pushing performance Takeaways from 60+ models As of 21 Nov 2025, the Open ASR Leaderboard compares 60+ open and closed-source models from 18 organizations , across 11 datasets . In a recent preprint , we dive into the technical setup and highlight some key trends in modern ASR. Here are the big takeaways 👇 1. Conformer encoder 🤝 LLM decoder tops the charts 📈 Models combining Conformer encoders with large language model (LLM) decoders currently lead in English transcription accuracy. For example, NVIDIA’s Canary-Qwen-2.5B , IBM’s Granite-Speech-3.3-8B , and Microsoft’s Phi-4-Multimodal-Instruct achieve the lowest word error rates ( WER ), showing that integrating LLM reasoning can significantly boost ASR accuracy. 💡 Pro-tip: NVIDIA introduced Fast Conformer , a 2x faster variant of the Conformer, that is used in their Canary and Parakeet suite of models. 2. Speed–accuracy tradeoffs ⚖️ While highly accurate, these LLM decoders tend to be slower than simpler approaches. On the Open ASR Leaderboard , efficiency is measured using inverse real-time factor (RTFx), where higher is better. For even faster inference, CTC and TDT decoders deliver 10–100× faster throughput , albeit with slightly higher error rates. This makes them ideal for real-time , offline , or batch transcription tasks (such as meetings, lectures, or podcasts). 3. Multilingual 🌍 OpenAI’s Whisper Large v3 remains a strong multilingual baseline, supporting 99 languages . However, fine-tuned or distilled variants like Distil-Whisper and CrisperWhisper often outperform the original on English-only tasks, showing how targeted fine-tuning can improve specialization ( how to fine-tune? Check out guides for Whisper , Parakeet , and Voxtral ). That said, focusing on English tends to reduce multilingual coverage 👉 a classic case of the tradeoff between specialization and generalization. Similarly, while self-supervised systems like Meta’s Massively Multilingual Speech (MMS) and Omnilingual ASR can support 1K+ languages, they trail behind language-specific encoders in accuracy. ⭐ While just five languages are currently benchmarked, we’re planning to expand to more languages and are excited for new dataset and models contributions to multilingual ASR through GitHub pull requests . 🎯 Alongside multilingual benchmarks, several community-driven leaderboards focus on individual languages. For example, the Open Universal Arabic ASR Leaderboard compares models across Modern Standard Arabic and regional dialects , highlighting how speech variation and diglossia challenge current systems. Similarly. the Russian ASR Leaderboard provides a growing hub for evaluating encoder-decoder and CTC models on Russian-specific phonology and morphology . These localized efforts mirror the broader multilingual leaderboard’s mission to encourage dataset sharing, fine-tuned checkpoints, and transparent model comparisons , especially in languages with fewer established ASR resources. 4. Long-form transcription is a different game ⏳ For long-form audio (e.g., podcasts, lectures, meetings), closed-source systems still edge out open ones. It could be due to domain tuning, custom chunking, or production-grade optimization. Among open models, OpenAI’s Whisper Large v3 performs the best. But for throughput, CTC-based Conformers shine 👉 for example, NVIDIA’s Parakeet CTC 1.1B achieves an RTFx of 2793.75 , compared to 68.56 for Whisper Large v3, with only a moderate WER degradation ( 6.68 and 6.43 respectively). The tradeoff? Parakeet is English-only, again reminding us of that multilingual and specialization tradeoff 🫠. ⭐ While closed systems still lead, there’s huge potential for open-source innovation here. Long-form ASR remains one of the most exciting frontiers for the community to tackle next! 🎤 The Show Must Go On Given how fast ASR is evolving, we’re excited to see what new architectures push performance and efficiency, and how the Open ASR Leaderboard continues to serve as a transparent, community-driven benchmark for the field, and as a reference for other leaderboards ( Russian , Arabic , and Speech DeepFake Detection ). We’ll keep expanding the Open ASR LeaderBoard with more models, more languages, and more datasets so stay tuned 👀 👉 Want to contribute? Head on over to the GitHub repo to open a pull request 🚀

11月15日周六
  1. 腾讯玄武实验室57

    腾讯玄武实验室详解量子计算机距离攻破 RSA-2048 还有多远

    腾讯玄武实验室发布长文，从量子计算原理、超导量子计算机构造到破解 RSA-2048 的工程瓶颈，系统分析量子计算机距离攻破经典公钥密码还有多远。文章指出科学层面已无死胡同，真正的挑战集中在冷却、控制、布线、能耗与量子纠错实时实现等工程问题；以表面码纠错计算，运行 Shor 算法需要百万级物理量子比特。

11月12日周三
11月10日周一
  1. 腾讯玄武实验室73

    腾讯玄武实验室阿图因引擎发现 gnark 零知识证明库签名可锻造性漏洞 CVE-2025-57801

    腾讯玄武实验室的阿图因自动化漏洞挖掘引擎在零知识证明库 gnark 中发现高危签名可锻造性漏洞 CVE-2025-57801（CVSS 8.6），并联合上海交通大学 GOSSIP 实验室及郁昱教授团队完成复现。

    推荐理由：文章解释了漏洞虽存在多年却大多未被触发的原因，读者可据此自查闭源项目是否使用了受影响的 Native Verification API。

11月4日周二