deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Cloudflare makes Markdown for Agents generally available and adds measurement tools

Cloudflare's Markdown for Agents is now generally available with analytics, logs, a GraphQL dataset and tracing spans — but converted responses drop ETag and Last-Modified, breaking conditional requests.

Cloudflare makes Markdown for Agents generally available and adds measurement tools

What happened

Cloudflare made Markdown for Agents generally available on 9 November 2026 and shipped a measurement layer with it, according to a dev.to write-up. The dashboard gains an Analytics tab, sampled Logs, a Playground that compares Markdown against HTML, a GraphQL dataset and a tracing span for each conversion. The feature itself, first announced in February, converts HTML pages into Markdown at Cloudflare's edge so AI agents and crawlers get a compact document instead of a full page with navigation, footers and scripts.

How the conversion works

The mechanism is HTTP content negotiation. When a client lists text/markdown in its Accept header, Cloudflare fetches the HTML from the origin, converts the body on the fly and streams the result back. The converted response sets Content-Type to text/markdown and adds Accept to Vary, so caches hold separate variants of the two representations.

The output has a stable shape: a YAML frontmatter block built from the page's meta tags — the standard form first, Open Graph as a fallback — while navigation, footers, scripts and styles are stripped and any JSON-LD is preserved in a single fenced block at the end. For consumers, that predictability removes the need to write per-site parsers.

The feature covers Pro, Business and Enterprise plans plus SSL for SaaS at no cost. It converts HTML only and stops converting above 6 MiB of decompressed HTML.

What the new tools measure

Under Agent Readiness in the dashboard, the Analytics tab reports conversion attempts, a full conversion rate, partial and failed conversions, top converted pages, top user agents, failure reasons, content limits and an estimated token reduction across a seven-day window. Logs exposes sampled conversion events with filters for outcome, failure reason and content limits, and the Playground requests Markdown for a single path and shows it beside the HTML response, headers included.

The same data leaves the dashboard through the GraphQL Analytics API as markdownForAgentsAdaptiveGroups, at both zone and account scope, with queries limited to seven days of history and 1,000 groups per query. Cloudflare Traces adds a markdown_conversion span on sampled streaming conversions, carrying the outcome, input and output bytes, converter CPU time, the longest processing slice and the content-limit flags — the detail that attaches a conversion decision to an individual request timeline.

Token savings are byte estimates

Every token figure Cloudflare reports is an estimate: bytes divided by four and rounded down, weighted by sampling, and counted only over full conversions with a positive input estimate. The company publishes the formula for its reduction ratio and warns the estimate turns negative when the Markdown output is larger than its HTML input. As the dev.to piece argues, teams should read the number as a bandwidth proxy; a tokenizer-specific count requires running the target tokenizer, and accuracy drops on code, tables and identifier-heavy pages.

Header changes affect conditional requests

Security and cache headers survive conversion, including Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, Set-Cookie, the CORS headers and the caching headers. Body-derived headers do not: Content-Type becomes text/markdown, Content-Length is removed to support streaming, and ETag, Last-Modified, Content-Encoding, Content-Range and Transfer-Encoding are dropped because their values belong to the original HTML body.

Dropping the validators has consequences beyond the feature. A conditional request cannot be answered for a converted response, so an agent polling a page with If-None-Match pays for a full transfer every time, and a monitor sees a request pattern that differs from the HTML side. A team storing the Markdown downstream also loses the cheap way to tell whether a page changed, which pushes change detection onto its own content hashes.

Partial conversions are the number to watch

A full conversion means no content-limit flag fired — Cloudflare defines the outcome by those flags, and how faithfully content was preserved is a separate assessment. A partial conversion means a limit truncated the input or dropped the frontmatter or the JSON-LD. Because the 6 MiB cap applies to decompressed HTML, a large documentation page can cross it quietly: the agent still receives a document that parses but is missing the section it came for. Two further caveats from the reporting: the user-agent breakdown is client-supplied and Cloudflare verifies no bot identity behind it, and the reported page URLs exclude query strings and fragments.

Why it matters

Markdown for Agents changes the contract between websites and the machines that read them. Publishers on the supported plans can serve crawlers and AI agents a stripped, predictable representation through ordinary HTTP negotiation, without building their own extraction endpoints, and the new analytics turn that delivery path into an engineering decision with numbers attached. The trade-offs are real — no conditional requests, byte-derived token estimates and silent truncation above 6 MiB — but for teams optimizing how agents consume their sites, the partial-conversion rate and the token trend line are now the metrics worth watching.

  • #cloudflare
  • #markdown
  • #ai-agents
  • #http
  • #web-crawling

Related posts