Step3

2025-07-31 · Publisher : StepFun · Domain : Artificial Intelligence

StepFun opens Step3, a multimodal mixture-of-experts model with 321 billion parameters, 38 of them active, under Apache 2.0, designed with its inference system to reduce decoding cost on a variety of GPUs.

### Context In summer 2025, the inference cost of large MoEs has become a central topic: DeepSeek-V3 published a reference distributed deployment system, which StepFun takes as a point of comparison. StepFun, a young Chinese company specializing in multimodal models, publishes with Step3 both a model and the server architecture that goes with it. ### What the model brings - Multi-Matrix Factorization Attention (MFA): according to the report, it reduces the volume of memory accesses to the key-value cache by only about 10% compared with DeepSeek-V3's latent attention (MLA), but the estimated attention cost (table 6) is halved or more in many cases, thanks to a better-balanced arithmetic intensity. - Attention-FFN disaggregation (AFD): the attention layers and the expert layers run on separate GPU groups, which makes it possible to choose hardware suited to each part, including cheaper cards than the H800. - Announced throughput: 4,039 tokens per GPU per second in decoding on Hopper GPUs, at an average context of 4,096 tokens and under a target of 20 tokens per second per request, about 74% more than the DeepSeek-V3 reference according to the report. - Multimodal model: the language model is paired with a vision encoder, for a total of 321 billion parameters. - StepMesh communication library published for AFD (GitHub). ### Architecture and specifications - Architecture: MoE with 316 billion parameters for the language part (321 billion with vision), 38 billion active; 61 layers including 5 dense, hidden dimension of 7,168, MFA attention with 64 query heads. - Experts: 48 routed experts, 3 selected per token, plus 1 shared expert. Tokenizer: that of DeepSeek V3. - Context: 65,536 tokens maximum. Modalities: text and image. Published precision: BF16 (and block-wise FP8 version). - License: Apache 2.0. Weights: stepfun-ai/step3 and step3-fp8 on Hugging Face and ModelScope. Knowledge cutoff and languages: not disclosed. ### Cost and access API access is offered on StepFun's platform, but no pricing was consulted. The system report describes deployments of 32 to 48 Hopper GPUs in AFD mode (for example 2 attention GPUs for 2 FFN GPUs per group) and recommends cards at least as powerful as the NVIDIA L20 for the attention part. | Configuration (average context) | Hopper GPUs | Maximum throughput (tokens per GPU per second) | | --- | --- | --- | | DeepSeek-V3, official blog (4,989) | 144 | 1,850 | | DeepSeek-V3, profiling measurement (4,096) | 128 | 2,324 | | Step3, BF16 attention (4,096) | 40 | 3,321 | | Step3, FP8 attention (4,096) | 32 | 4,039 | | Step3, FP8 attention (8,192) | 48 | 2,643 | ### Benchmarks | Measure | Step3 | DeepSeek-V3 | Qwen3 MoE | | --- | --- | --- | --- | | Maximum decoding throughput (tokens/GPU/s, target 20 tokens/s per request) | 4,039 | 1,850 (blog) or 2,324 (profiling) | Not measured | | Theoretical decoding cost at 8K context (USD per million tokens) | 0.055 | 0.068 | 0.062 | | Theoretical decoding cost at 32K context (USD per million tokens) | 0.129 | 0.211 | 0.193 | Source: StepFun report arXiv 2507.19427, table 8, §7.3 and observation 1 of §4.2 (theoretical costs, StepFun analytical estimates, not measured). StepFun measurements; the DeepSeek-V3 figures are those that DeepSeek published (blog and profiling). The model card's capability scores are in an image: not included. No comparison with a closed model. ### Versus closed models The report does not compare Step3 to closed models: it targets decoding cost and measures it against DeepSeek-V3, an open model. On this point, StepFun claims 4,039 tokens per GPU per second against 1,850 (blog) or 2,324 (profiling) for DeepSeek-V3 under conditions described as similar, acknowledging that DeepSeek-V3 could be optimized further (quantization, kernels); these are vendor measurements. The comparison of capabilities with the closed models of the time is not available in text: it appears in an image on the model card. Since the price was not recorded, the cost gap with closed models is not established. The interest for an open-or-closed choice is architectural: a multimodal model with 38 billion active parameters under Apache 2.0, designed to be served on mixed-range GPUs, which matters for self-hosting at scale. ### Key takeaways Step3 is a milestone in joint model and infrastructure design: MFA and AFD aim to lower decoding cost. Its gains are measured against DeepSeek-V3 and not against closed models, whose comparison is missing in text.

Tags : LLM, Open Source, StepFun, Step, Apache 2.0, MoE, Multimodal