GitHub – Deepseek-ai/DeepSeek-V3

We provide DeepSeek-V3, a strong Mixture-of-Experts (MoE) language design with 671B overall criteria with 37B triggered for each token. To achieve efficient inference and cost-efficient training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly confirmed in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free method for load balancing and sets a multi-token prediction training goal for stronger efficiency. We pre-train DeepSeek-V3 on 14.8 trillion diverse and top quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning phases to totally harness its abilities. Comprehensive assessments expose that DeepSeek-V3 outperforms other open-source models and accomplishes efficiency equivalent to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires just 2.788 M H800 GPU hours for its complete training. In addition, its training procedure is incredibly stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or carry out any rollbacks.

2. Model Summary

Architecture: Innovative Load Balancing Strategy and Training Objective

– On top of the efficient architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free technique for load balancing, which reduces the efficiency deterioration that develops from encouraging load balancing.
– We examine a Multi-Token Prediction (MTP) goal and show it helpful to model efficiency. It can likewise be used for speculative decoding for inference velocity.

Pre-Training: Towards Ultimate Training Efficiency

– We create an FP8 combined accuracy training structure and, for the first time, verify the expediency and effectiveness of FP8 training on an incredibly large-scale design.
– Through co-design of algorithms, structures, and hardware, we conquer the communication bottleneck in cross-node MoE training, almost achieving complete computation-communication overlap.
This substantially enhances our training effectiveness and lowers the training costs, enabling us to even more scale up the design size without extra overhead.
– At an economical cost of just 2.664 M H800 GPU hours, we finish the pre-training of DeepSeek-V3 on 14.8 T tokens, producing the currently greatest open-source base design. The subsequent training stages after pre-training require just 0.1 M GPU hours.

Post-Training: Knowledge Distillation from DeepSeek-R1

– We present an innovative method to distill thinking capabilities from the long-Chain-of-Thought (CoT) design, specifically from among the DeepSeek R1 series designs, into basic LLMs, particularly DeepSeek-V3. Our pipeline elegantly includes the confirmation and reflection patterns of R1 into DeepSeek-V3 and especially improves its thinking efficiency. Meanwhile, we likewise maintain a control over the output design and length of DeepSeek-V3.

3. Model Downloads

The total size of DeepSeek-V3 designs on Hugging Face is 685B, which includes 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights. **

To make sure optimum performance and flexibility, we have partnered with open-source communities and hardware suppliers to supply several methods to run the design locally. For step-by-step guidance, have a look at Section 6: How_to Run_Locally.

For developers wanting to dive deeper, we advise exploring README_WEIGHTS. md for information on the Main Model weights and the Multi-Token Prediction (MTP) Modules. Please note that MTP assistance is presently under active development within the neighborhood, and we invite your contributions and feedback.

4. Evaluation Results

Base Model

Standard Benchmarks

Best outcomes are shown in bold. Scores with a space not exceeding 0.3 are considered to be at the very same level. DeepSeek-V3 achieves the very best efficiency on the majority of benchmarks, especially on mathematics and code jobs. For more assessment information, please inspect our paper.

Context Window

Evaluation results on the Needle In A Haystack (NIAH) tests. DeepSeek-V3 carries out well across all context window lengths up to 128K.

Chat Model

Standard Benchmarks (Models bigger than 67B)

All designs are evaluated in a setup that restricts the output length to 8K. Benchmarks consisting of fewer than 1000 samples are tested numerous times utilizing differing temperature level settings to obtain robust last results. DeepSeek-V3 stands as the best-performing open-source model, and also displays competitive efficiency versus frontier closed-source designs.

Open Ended Generation Evaluation

English open-ended conversation examinations. For AlpacaEval 2.0, we utilize the length-controlled win rate as the metric.

5. Chat Website & API Platform

You can talk with DeepSeek-V3 on DeepSeek’s official site: chat.deepseek.com

We also provide OpenAI-Compatible API at DeepSeek Platform: platform.deepseek.com

6. How to Run Locally

DeepSeek-V3 can be deployed in your area using the following hardware and open-source community software:

DeepSeek-Infer Demo: We supply an easy and lightweight demonstration for FP8 and BF16 reasoning.
SGLang: Fully support the DeepSeek-V3 model in both BF16 and FP8 reasoning modes, with Multi-Token Prediction coming quickly.
LMDeploy: Enables efficient FP8 and BF16 inference for regional and cloud release.
TensorRT-LLM: Currently supports BF16 reasoning and INT4/8 quantization, with FP8 support coming soon.
vLLM: Support DeepSeek-V3 design with FP8 and BF16 modes for tensor parallelism and pipeline parallelism.
AMD GPU: Enables running the DeepSeek-V3 model on AMD GPUs by means of SGLang in both BF16 and FP8 modes.
Huawei Ascend NPU: Supports running DeepSeek-V3 on Huawei Ascend devices.
Since FP8 training is natively adopted in our framework, we just provide FP8 weights. If you need BF16 weights for experimentation, you can use the provided conversion script to perform the improvement.

Here is an example of converting FP8 weights to BF16:

Hugging Face’s Transformers has not been directly supported yet. **

6.1 Inference with DeepSeek-Infer Demo (example just)

System Requirements

Note

Linux with Python 3.10 only. Mac and Windows are not supported.

Dependencies:

Model Weights & Demo Code Preparation

First, clone our DeepSeek-V3 GitHub repository:

Navigate to the reasoning folder and set up dependencies noted in requirements.txt. Easiest method is to utilize a package supervisor like conda or uv to create a new virtual environment and set up the reliances.

Download the model weights from Hugging Face, and put them into/ path/to/DeepSeek-V 3 folder.

Model Weights Conversion

Convert Hugging Face design weights to a particular format:

Run

Then you can chat with DeepSeek-V3:

Or batch reasoning on a provided file:

6.2 Inference with SGLang (advised)

SGLang currently supports MLA optimizations, DP Attention, FP8 (W8A8), FP8 KV Cache, and Torch Compile, delivering state-of-the-art latency and throughput performance among open-source structures.

Notably, SGLang v0.4.1 completely supports running DeepSeek-V3 on both NVIDIA and AMD GPUs, making it an extremely flexible and robust solution.

SGLang also supports multi-node tensor parallelism, enabling you to run this design on numerous network-connected makers.

Multi-Token Prediction (MTP) is in development, and progress can be tracked in the optimization strategy.

Here are the launch directions from the SGLang group: https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3

6.3 Inference with LMDeploy (suggested)

LMDeploy, a flexible and high-performance inference and serving structure customized for big language models, now supports DeepSeek-V3. It offers both offline pipeline processing and online deployment capabilities, effortlessly integrating with PyTorch-based workflows.

For extensive step-by-step directions on running DeepSeek-V3 with LMDeploy, please describe here: InternLM/lmdeploy # 2960

6.4 Inference with TRT-LLM (suggested)

TensorRT-LLM now supports the DeepSeek-V3 design, using precision options such as BF16 and INT4/INT8 . Support for FP8 is currently in development and will be launched soon. You can access the custom-made branch of TRTLLM particularly for DeepSeek-V3 support through the following link to experience the new features straight: https://github.com/NVIDIA/TensorRT-LLM/tree/deepseek/examples/deepseek_v3.

6.5 Inference with vLLM (advised)

vLLM v0.6.6 supports DeepSeek-V3 inference for FP8 and BF16 modes on both NVIDIA and AMD GPUs. Aside from standard strategies, vLLM offers pipeline parallelism enabling you to run this model on several makers connected by networks. For detailed assistance, please refer to the vLLM directions. Please do not hesitate to follow the improvement plan too.

6.6 Recommended Inference Functionality with AMD GPUs

In collaboration with the AMD team, we have actually achieved Day-One assistance for AMD GPUs utilizing SGLang, with complete compatibility for both FP8 and BF16 precision. For comprehensive assistance, please describe the SGLang instructions.

6.7 Recommended Inference Functionality with Huawei Ascend NPUs

The MindIE structure from the Huawei Ascend community has effectively adjusted the BF16 variation of DeepSeek-V3. For detailed guidance on Ascend NPUs, please follow the guidelines here.

7. License

This code repository is licensed under the MIT License. Making use of DeepSeek-V3 Base/Chat models undergoes the Model License. DeepSeek-V3 series (consisting of Base and Chat) supports industrial use.