Blog post argues Pandas inefficiencies push data teams into distributed systems too early

The claim

A blog post titled "Pandas Should Go Extinct," published on eddie.codes and picked up on the Hacker News front page, makes a deliberately provocative argument: the widely used Python DataFrame library should be retired because its inefficiencies push developers toward distributed systems long before their workloads require them. According to the author, most teams will never need those systems, which he characterises as heavily marketed fixes for problems few organisations actually have.

The post maps out a typical adoption path. Analysts begin in Excel, move to Pandas somewhere in the gigabyte range, and work comfortably into the tens of gigabytes, at which point they hit memory limits, slow computations or an API the author describes as overgrown. The conventional answer is a step up to tools built for large-scale processing such as Spark, Databricks, Snowflake or Dask. The author's point is that a gap has opened between where Pandas runs out of steam — roughly around the 100GB mark — and the scale at which distributed systems are genuinely necessary, and that modern single-machine tools can now fill that gap.

Evidence from Amazon's own fleet

To support the claim that genuinely large-scale data is rarer than assumed, the post leans on a 2024 Amazon paper analysing the Redshift fleet, which published statistics on query runtimes and table sizes across Amazon's customer base. Working from a few stated assumptions — an average row size of 1KB, and clusters of ten machines each reading from S3 at 8GB/s — the author calculates that 94.68% of tables in the fleet hold less than 100GB, and that 86.9% of queries touch 80GB of data or less, since they complete in under a second at the assumed throughput. Even assuming much fatter 10KB rows, the corresponding table sizes remain around 1TB.

The conclusion the author draws is blunt: most practitioners have medium-sized data problems and need medium-data solutions. He also points readers to a deeper analysis of the same dataset by Jordan Tigani of MotherDuck, while noting that MotherDuck sells hosted DuckDB and therefore deserves some scepticism.

DuckDB and Polars as the replacements

The proposed successors are Polars, a Rust-based DataFrame library that feels familiar to Pandas users but behaves very differently at runtime, and DuckDB, an in-process analytics database the author likens to an analytics-oriented SQLite.

To illustrate the difference, the post benchmarks implementations of the 1 Billion Row Challenge, a contest to compute the minimum, mean and maximum readings from a one-billion-row CSV of weather station data, where the fastest accepted Java entry ran in 1.5 seconds. The original challenge used a 32-core, 128GB RAM bare-metal server; the author instead ran his tests on an AWS m7a.8xlarge with the same core count and memory, acknowledging that skipping dedicated hardware may affect reproducibility.

The technical distinction matters more than the raw numbers. In the post's example, the Pandas version loads the entire dataset into memory and then executes each step of the computation sequentially. The Polars version uses a lazy scan, streams the file in chunks, and only materialises results when collect is called. That allows it to build an optimised query plan in the style of a database engine, parallelise work across threads, and apply techniques such as pushing filters down before aggregation rather than loading every row first.

Why it matters

Tooling decisions carry compounding costs. Distributed systems add expense, operational burden and debugging complexity, and if the vast majority of analytical tables sit under 100GB, many teams adopting clusters are paying for headroom they will never use. If modern single-machine engines can cover that range while borrowing decades of database optimisation, the default escalation path from Pandas to Spark deserves to be questioned rather than assumed.

The discussion the piece has prompted also shows how contested the territory remains. The Redshift arithmetic rests on the author's own assumptions about row sizes and cluster configurations, and the benchmark ran on a 32-core server rather than a laptop. Vendors of distributed platforms have commercial reasons to emphasise scale, just as DuckDB-adjacent businesses have reasons to argue the opposite. Still, the post captures a real shift in the data community, where tools like Polars and DuckDB are moving from curiosities toward serious defaults for everyday analytical work.