PyO3 walkthrough shows how Rust extensions like pydantic-core power Python

A hands-on guide published on belderbos.dev, currently on the Hacker News front page, walks through how Rust code gets embedded in Python using PyO3 — the same framework behind pydantic-core, the Rust engine that handles validation in Pydantic v2. The example project is deliberately small: a JSON parser written in Rust and exposed so it can be imported like any ordinary Python package. The lasting lesson, though, is about where the performance actually goes.

Four steps from Rust to import

According to the post, the path from a Rust crate to a working Python module has four stages: write a normal Rust module, annotate it with PyO3 macros, let maturin compile and install it, then import the result.

Two macros do the wiring. #[pyfunction] and #[pymodule] are attribute macros, which the author likens to Python decorators because they rewrite the function they sit on, adding the glue that lets Python call into Rust and handling type conversions plus reference counting at the seam between the languages. Maturin then compiles the crate into a shared library — .so, .dylib or .dll depending on the platform — and installs it directly into the virtual environment, so a plain import works with no extra plumbing.

The parse stays on the Rust side

The example parser builds its output as a plain Rust enum, with variants covering null, booleans, numbers, strings, arrays and objects. That tree lives entirely in Rust; Python never sees it directly. The PyO3 layer on top is described as a thin conversion shim.

The exposed function is compact. It receives a token representing the interpreter (Python<'py>), runs the parse, and returns PyResult, which is Result with PyErr as the error type — so a failed parse surfaces in Python as an exception rather than a Rust error value. The real work happens in the parse call; the final .into_pyobject step is what builds the Python objects the caller asked for.

The return trip is the costly part

That final step is the heart of the article. Implementing PyO3's IntoPyObject trait means traversing the finished Rust tree and reconstructing it as native Python objects: a dict for each JSON object, a list for each array, a float or string for each leaf value. A document containing 100,000 values therefore triggers roughly 100,000 Python object allocations at the boundary — all of it after parsing has already finished. On large documents, the post argues, this materialisation loop rather than the parser itself can dominate end-to-end time.

The practical advice follows from that. If the function being ported returns a scalar, the boundary is small enough to ignore. If it returns a large structure, the conversion is the next thing to optimise once the algorithm is fast. Preallocating dictionaries helps only at the margins; the bigger win is architectural — return a lazy, Rust-backed view and construct Python objects only when the caller actually touches them.

Errors cross the same bridge

Return values are not the only thing that needs translating. A single From implementation from the parser's Rust error type to PyErr lets the question-mark operator do the work: malformed input raises a ValueError that carries the position where parsing broke, with one match arm per error variant. File handling comes free, since std::io::Error already maps to the matching Python exception — a missing path raises FileNotFoundError. Callers get native Python semantics without the Rust layer leaking through.

The post also cites results from the author's Python-to-Rust cohort, where students build exactly this kind of parser: one student's version reportedly beat CPython's C module on real-world fixtures, and another ran up to 3.5 times faster than a pure-Python equivalent. These are self-reported figures without published methodology, but they illustrate why the pattern attracts attention.

Why it matters

PyO3 is not a curiosity. Pydantic v2, the data-validation library many Python applications rely on, gets its speed from a PyO3-built core, so this architecture sits underneath widely used tooling. For teams weighing a similar port, the guide reframes the problem: making Rust run fast is the easier half. The half that decides whether the port pays off is what happens on the way out — the construction of Python objects from Rust values. That means profiling should cover the conversion layer, not just the algorithm, and designs that avoid materialising whole structures will often beat ones that merely parse quickly.