Pydantic's Logfire shreds JSON attributes in Parquet for up to 1000x faster queries

Logfire, the observability platform from Pydantic, is in for a large speedup. According to an engineering post from the Pydantic team, an optimization the company calls dynamic shredding makes some queries more than 1,000 times faster — enough that lookups which used to hit a 30-second timeout now return in under a second. The change begins reaching all customers in February 2026, and the write-up has since surfaced on Hacker News's front page.

Semi-structured data in a columnar store

Logfire ingests OpenTelemetry telemetry, which mixes two kinds of data. Fixed-schema fields — timestamps, span names, trace and span IDs — exist for every span and are strongly typed. Semi-structured attributes such as HTTP status codes, user IDs or deployment regions arrive as key-value pairs attached to spans or resources, and they vary widely: some keys are standardized in the OpenTelemetry spec, while an application-specific one like user_id might be an integer in one service and a string in another.

The data is ultimately stored as Parquet files in object storage. The fixed fields map to typed columns, but attributes lived in a single column holding a JSON blob.

Why parsing speed wasn't enough

Pydantic's first approach leaned on raw parsing throughput: attributes stayed as JSON, and Jiter — a parser the team ranks as among the fastest in any language — chewed through them at query time. The post is candid that this optimized the wrong thing. The company was making the work faster rather than avoiding it.

Two costs dominated. Compression was poor, because a single varying attribute defeats Parquet's dictionary encoding, and even zstd struggles with highly variable, syntax-heavy JSON. Worse, the extra bytes meant extra IO, and on object storage the latency of initiating downloads swamps CPU time as the main driver of query runtimes.

Typed columns and statistics

Columnar engines such as DataFusion avoid unnecessary work in two ways: they read only the columns a query touches, and they use zone maps — min/max statistics recorded per row group and page — to skip chunks that cannot possibly match a filter. The post gives the example of a duration filter that cannot match any stored value being answered from statistics alone, with no scanning at all.

Logfire's first attempt to bring that machinery to attributes was static shredding: a hand-picked list of commonly queried keys, like http.response.status_code or url.full, extracted at ingestion time into dedicated typed columns. Queries written against the JSON path get rewritten internally to hit the shredded column, so users never need to know it exists. The typed columns compress well and take full advantage of pruning.

Where static shredding fell short

The static approach has significant limits, Pydantic acknowledges. Only a small, hardcoded set of attributes benefit, so anything outside the list pays the full JSON cost — and that becomes genuinely slow on projects with hundreds of gigabytes of data. Adding new shredded attributes is difficult. The internal columns also leak into user-facing output under an lf_attributes prefix whenever someone runs select *, exposing an implementation detail the team wanted hidden; the stated goal is for attributes to look like a single logical JSON column while the physical layout underneath changes.

Dynamic shredding

That tension sets up the post's central subject: dynamic shredding, the technique Pydantic credits with the headline speedups. The article walks through the data model, the original JSON design and the shortcomings of static shredding before laying out the new approach, and reports that early-access customers are already seeing the gains — one quoted customer described attribute queries as now feeling essentially instant.

Why it matters

The performance ceiling on observability tooling is usually set by semi-structured, high-cardinality attributes, not by the well-typed core fields. Pydantic's account is a useful case study in where that cost actually lives: not in CPU-bound JSON parsing, but in object-store IO that faster parsers can never remove. It also illustrates a design pattern with wider applicability — decoupling a logical schema, one JSON column, from a physical one of typed, prunable columns, so storage optimizations stay invisible to users. For Logfire customers the practical consequence is immediate: queries that timed out now come back, and attribute filtering becomes usable on datasets where it previously wasn't.