Vercel Firewall now blocks AI crawlers and AI assistants separately

Separate controls for two kinds of AI traffic

Vercel Firewall can now log or block AI crawlers and AI assistants as independent categories, through the AI Bots managed ruleset. According to a post on the Vercel blog, the feature is available on all plans, and Vercel keeps the ruleset current as new bots appear, so customers do not have to maintain their own detection lists.

The distinction between the two categories is the point of the release. AI crawlers are the bots that gather content at scale for indexing and model training. AI assistants are the bots that fetch a page because a person took an action, such as running a search or asking a chat product a question. Splitting them lets a site take opposite positions on each group: the changelog's own example is denying crawlers entirely while assistants keep access, so those tools can still read pages when responding to people.

Setting it up

In the Vercel dashboard, the controls sit under a project's Security > Firewall > Rules section. Under Bot Management, users enable AI Bots and open its settings through a gear icon beside it. From there, an action can be assigned to each of the two categories, and the change is reviewed and pushed live like any other firewall rule.

The same configuration is scriptable through the Vercel CLI. An edit command targeting the ai-bots ruleset accepts a category flag, either ai-crawlers or ai-assistants, alongside an action flag such as deny or log. That makes it straightforward to codify a policy like blocking crawlers while only logging assistant traffic. The changelog shows the pending change being inspected with a diff command and applied with a publish command, which should fit easily into existing infrastructure-as-code workflows.

Why it matters

Automated traffic from AI companies has become a routine reality for anyone operating a website, and it pulls site owners in two directions at once. Many publishers do not want their pages harvested to train models they get nothing back from, but they do want their content reachable when someone asks an assistant or an AI search engine a question it can answer. A single block-all-AI-bots toggle cannot express that preference; two distinct categories can.

There is also a maintenance argument. Bot identification is a moving target, with new crawlers and assistants appearing on a regular basis. Vercel taking on that detection work at the firewall level means a site's policy stays effective over time without anyone hand-curating user agent blocklists. For teams already hosting on Vercel, the practical consequence is that AI traffic handling becomes a first-class, per-category firewall setting rather than something patched together afterwards.