TokenPrint: an open-source 3D debugger for tracing tokens through transformers

A developer has published TokenPrint, an open-source 3D visual debugger for transformer and LLM inference that turns a model's internal computation into a navigable graph. According to a post on dev.to by the project's author, the motivation is that static diagrams, equations and source code describe a model's structure, but not the path an individual token takes through it.

Following a token through the network

The tool models inference as a path. A token starts as a discrete ID, is converted into an embedding, and then passes through normalization, attention, projections, non-linear transformations and residual connections across layers until the model emits its final logits. In the 3D view, developers can walk that path: select an operation, move between layers, and inspect the tensors and metadata attached to each step.

The set of inspectable components reads like a parts list for a modern decoder: token embeddings, positional information, RMSNorm, the Q, K and V projections, grouped-query attention, RoPE, attention scores, softmax, weighted value aggregation, the output projection, SwiGLU, MLP projections, residual streams, and logits and predictions. For grouped-query attention specifically, the tool can show the relationship between query heads and shared key/value groups instead of treating the block as one opaque unit.

Inspection views and tensor data

Selecting a component opens an inspection panel with a plain-language explanation of the operation, the equation it implements, its input and output dimensions, the number of learned parameters involved, and the tensor path in the running model. A projection, for instance, can surface a weight such as the layer-11 self-attention key projection, together with its shape, dtype, parameter count, layer index and runtime information.

There is also a dedicated tensor inspection workflow. The dev.to post gives the example of a layer-3 value-projection weight with shape 128 by 896 in float32, roughly 114,700 parameters, loaded from a local Hugging Face runtime — a deliberate link between the on-screen graph and the model's actual implementation.

Provenance labels separate real data from illustration

One of the more unusual design decisions is explicit provenance. Everything displayed is tagged in one of four categories: REAL for values taken directly from the model or runtime, DERIVED for values computed from real model data, CONCEPTUAL for educational representations, and SIMULATION for behaviour that is deliberately faked. The author's argument is that an attractive visualization is worthless if it silently fabricates model internals, so the labels make clear what is measured and what is merely illustrated.

The project is also intended as a teaching aid: each component is meant to explain what the operation computes, why the transformer needs it, what its inputs and outputs are, how it connects to the next operation, and which parameters and tensors are involved.

Roadmap: from viewer to experiment environment

TokenPrint is positioned as more than a viewer. The author describes an ambition to build an environment for inspecting and experimenting with transformer computation, with attention inspection, activation analysis, residual stream analysis, tensor inspection, layer and head experiments, ablations, activation patching, trace replay and model comparisons listed as target areas.

Planned directions include broader Hugging Face model support, better model capability detection, richer execution traces, remote inference, experiment workflows, and shareable traces that others can explore without reproducing the original runtime. The code is hosted on GitHub, and the project invites contributions in areas such as frontend and 3D visualization, transformer architecture support, inference backends, tensor instrumentation, debugging tools, visualization UX, educational content and research experiments. A hosted instance is available for trying it in a browser.

Why it matters

Transformer debugging and interpretability work has produced plenty of notebooks, 2D attention heatmaps and static diagrams, but comparatively few tools let a developer move through the whole computation graph interactively. By mapping diagram-level concepts onto concrete tensors, and by labelling which parts of a view are real model data versus illustration, TokenPrint bridges the gap between educational explanations and implementation-level debugging. The planned ablation and activation-patching features point toward mechanistic-interpretability workflows inside a visual UI rather than a Python session. One caveat: the details come from a single announcement post by the project's author, and the experimentation features are described as directions the project is heading rather than capabilities that ship today.