StepFun opens weights of 600B-parameter MoE model with 1M-token context

StepFun has released the open weights of Step 5 Preview, a foundation model with 600 billion total parameters that activates roughly 27 billion per token and accepts a context of up to one million tokens. According to a post on dev.to, the release targets long-running agentic orchestration and multi-hop analytical problem solving, and the checkpoints are published on Hugging Face and ModelScope in BF16, FP8 and NVFP4 formats.

A deep, sparse design built for long contexts

The post describes the model's topology as favouring depth over width: 92 layers that alternate between two attention mechanisms in a 3:1 ratio. Three of every four layers use gated sliding-window attention, which confines the quadratic attention computation to a fixed-size horizon and reportedly makes speculative decoding with a draft model easier. Every fourth layer applies full global attention with grouped queries, keeping information from the start of a document reachable across many turns of dialogue and tool calls.

Sparsity is handled by a router that selects 27 billion parameters for each emitted token, drawing on shared experts for universal syntactic representations alongside specialised experts for abstract reasoning, a split the author credits with isolating noise between application domains. Together, the design is said to hold VRAM growth in check at the million-token mark and to reduce prefill costs for coding and audit workloads that involve repeated tool calls.

What it takes to run

In NVFP4 quantisation, a checkpoint occupies about 340 GB of VRAM, and the post says a four-accelerator cluster of Hopper or Blackwell GPUs can serve the model at more than 110 tokens per second. That footprint, the author argues, makes continuous ingestion and refactoring of large codebases, monorepo audits and the modelling of sensitive data feasible on hardware an organisation owns, without shipping industrial secrets to a third-party API.

Benchmark numbers, as reported

The post publishes a comparison against contemporary frontier models — OpenAI's GPT-6 Astra, DeepSeek 4.1, Anthropic's Claude Fable 5.1 and Google DeepMind's Gemini 3.8 Flash Cyber — attributing the results to community-run evaluations with deterministic graders and no human steering:

• SWE-bench Verified: 78.2%, fractionally above GPT-6 Astra (78.1%) and DeepSeek 4.1 (77.8%)
• AIME 2026: 94.1%, just behind GPT-6 Astra's 94.2%
• AgentBench: 87.6%, trailing Claude Fable 5.1 (88.1%) and GPT-6 Astra (87.8%)
• MMLU-Pro: 84.5%, against 84.9% for GPT-6 Astra

These figures come from a single blog post and could not be independently verified for this article; the technical report is the place to confirm them. On the post's own numbers, the model sits at or near the front of the field on software engineering and mathematics, and slightly behind the leaders on agentic robustness.

How it was trained

Beyond supervised fine-tuning, the post says StepFun built its training data with a multi-agent pipeline in which different agents took the roles of hypothesis formulator, sandboxed code executor and quality critic, working under guidelines set by human specialists. The agents are said to have simulated hundreds of thousands of realistic engineering scenarios, producing data the lab considers uncontaminated by public benchmarks. Alignment was then reinforced with a credit-assignment scheme applied over long trajectories. The post also credits a proprietary pretraining pipeline with overcoming the routing collapse and cross-node bandwidth bottlenecks that have historically affected sparse MoEs at this scale.

Why it matters

Open weights at this scale change the options available to teams building agentic systems. The post argues that organisations needing chained deductive reasoning over large codebases have had little choice but closed APIs, with the attendant exposure to terms-of-service changes, unpredictable latency and third-party data sovereignty. A 600B-parameter model with a million-token context that can be self-hosted on four accelerators — and that, on the reported numbers, trades blows with proprietary frontier models — makes that dependency optional. For anyone evaluating it, the sensible next step is the official technical report and independent reproductions of the benchmark claims.