ServiceNow AI open-sources EVA-Bench Data 2.0 with 213 voice-agent scenarios across 3 domains

ServiceNow AI ships EVA-Bench Data 2.0

ServiceNow AI's research team has released EVA-Bench Data 2.0, an open benchmark designed to test AI agents in enterprise voice settings. According to a dev.to write-up from JudyAI Lab, which summarizes a ServiceNow AI post on the Hugging Face blog, the update grows the benchmark from a single domain to three, for a total of 213 evaluation scenarios and 121 tools — about four times the footprint of the earlier release.

The domains are airline customer service management (CSM), enterprise IT service management (ITSM), and healthcare human resources service delivery (HRSD). The write-up puts the scenario split at 50 for airline, 80 for ITSM and 83 for healthcare HRSD.

Scenarios derived from real support work

The stated design emphasis is realism. According to the write-up, every scenario was sifted from genuine telephone support workflows rather than invented for the test, and the tool schemas mirror production API specifications. The intent is to measure agents against the kind of requests they would actually receive on a support line.

The healthcare HRSD domain digs deepest into regulatory texture, anchoring scenarios in details of the US healthcare system such as NPI provider identifiers, FMLA family leave rules and insurance coverage terms. That density of business detail, the write-up suggests, is what an evaluation needs before it can separate systems meaningfully in a regulated field.

Solvability verified across three models

Before release, all 213 scenarios were checked against three frontier models — OpenAI's GPT-5.4, Google's Gemini 3.1 Pro and Anthropic's Claude Opus 4.6 — to confirm that each task is solvable. The write-up frames this multi-model check as a way to keep the benchmark demanding without favoring any single model family.

Open data, English for now

All three datasets are fully open source and loadable directly through Hugging Face Datasets. The write-up adds that the design principles and the scenario generation process are documented in the original post, which makes the release usable as a reference for teams building their own evaluation sets. The benchmark is currently English-only, reflecting its enterprise focus, and the team has announced a multilingual expansion to push beyond that limit.

Why it matters

Voice is quickly becoming a standard interface for enterprise agents, yet there is still no widely shared yardstick for how those agents perform on real support work. A benchmark anchored in actual call-center workflows, with tool schemas copied from production APIs, moves evaluation closer to predicting production behavior instead of abstract model capability.

Two design choices are worth borrowing for anyone assembling an eval dataset. Working backward from business workflows, rather than forward from a list of model capabilities, tends to surface the gaps that show up in deployment. And validating solvability across models from three different vendors guards against a benchmark that quietly flatters one of them. With the data, the schemas and the generation pipeline all public, EVA-Bench Data 2.0 works both as a measuring instrument and as a template for enterprise evaluation design.