ggml 1.0.0 removes deprecated APIs, adds FP8 and broad backend speedups

ggml reaches 1.0

The ggml project, the tensor library that sits underneath llama.cpp and a wide range of local inference tools, has published version 1.0.0, its first major-version release. According to the release notes on GitHub, the version arrives after v0.29.0 and pairs a long-signalled cleanup of the public interface with new low-precision formats and performance work touching nearly every backend the library supports.

Breaking changes in the API

The most consequential item is the removal of the deprecated public API along with the old DIAG_MASK_INF and DIAG_MASK_ZERO operations. The RPC protocol has also moved to version 9: peers running incompatible versions are now rejected outright, and the new protocol loads faster while coping better with unreachable endpoints or failed handshakes. For anyone operating mixed-version setups, that protocol bump is itself a breaking change.

On the additive side, ggml_rope_set_truncate and ggml_rope_get_truncate expose control over rounding at the YaRN correction bound, a setting now honored consistently by the CUDA, Vulkan, WebGPU and OpenVINO backends. The TRI operation gained a backward pass, and the MUL_MAT src1 path picked up precision tests together with gating for ggml_prec_set_acc.

New formats and faster matmuls on CUDA

CUDA gains support for OCP FP8 and fused NVFP4 gate/up matrix multiplies with GLU inside the quantized matmul path, aimed at the newest compact checkpoint formats. Small-batch inference gets direct attention, with a few-row MMQ path covering batch sizes 2-24 and MMVF handling BF16 matrices with few rows. FlashAttention gained an sm70 configuration for head size 256 plus MMA head sizes 40 and 72. Mixture-of-experts models benefit from skipped empty stream-k tiles, a linearized MoE layout and NaN cleanup after bias. On the AMD side, HIP receives a fixed BLAS workspace and an MFMA lightning indexer for CDNA2 hardware.

Metal, Vulkan and Intel GPUs

Metal now fuses MoE routing and keeps fusable node groups within a single command buffer, adds a mul_mm path for K between 32 and 64, and makes radix Top-K deterministic when values tie. Fixes cover GROUP_NORM with ne03 greater than 1 and out-of-bounds accesses in the FlashAttention output and SOLVE_TRI kernels.

Vulkan gains an Intel FlashAttention prefill kernel, batched tensor copies, ReBAR detection, dequantized KV with CM2, and RDNA3/4 large-tile split-K along with large cooperative-matrix matmul on newer AMD drivers. Correctness fixes for MoltenVK, GROUP_NORM workgroup limits and FlashAttention scratch sizing are also included.

SYCL, mobile and browser backends

SYCL adds a device-side MoE scheduler, makes the oneDNN FlashAttention cache thread-safe, enables fused SDPA for Intel Alchemist at head_dim 64, and now rejects buffer allocations that would exceed available VRAM.

OpenCL receives a decomposed FlashAttention implementation for Adreno GPUs with improved decode performance, plus new multi-column GEMV kernels enabled by default on Adreno X2. WebGPU adds a batched tensor API and GROUP_NORM, MEAN, COUNT_EQUAL and PAD_REFLECT_1D kernels. Hexagon gains Q5_0 and Q5_1 quantization support, an optimized conv_2d, a new conv_3d and automatic libhtp loading. Smaller backends were touched too: CANN now fails early on unsupported buffer types, and ZDNN accepts non-zdnn tensors.

On CPU, MUL_MAT switches to F32 for src1 and FlashAttention accumulates the V term in F32 for better numerical accuracy, while ARM NEON enables a 4x4 simd_gemm tile.

Smarter scheduling

The scheduler no longer uploads inputs that a node never reads, and it prefetches only inputs produced by earlier splits, trimming avoidable host-to-device traffic. Separately, the MoE cache can compute a share of cache misses on the CPU and copy the results back asynchronously, a hybrid path that could matter for large mixture-of-experts models that spill beyond GPU memory.

Why it matters

ggml is the engine beneath llama.cpp and a long tail of downstream tools, so an API purge at 1.0 is not cosmetic. Anything linking against the library must adapt when it syncs, and mixed-version RPC clusters will refuse to talk to each other. Beyond the breakage, the release sketches where local inference is heading: FP8 and NVFP4 paths for newer checkpoints, fused MoE execution for expert models, and optimizations stretching from NVIDIA and AMD discrete GPUs through Apple Silicon, Intel Arc, phone chips and browsers. The 1.0 label signals that the project now considers its public interface stable enough to build on, which is meaningful for the many tools that depend on it.