DeepSeek-V3

DeepSeek-V3 is a cutting-edge Mixture-of-Experts (MoE) language model boasting 671 billion parameters, with 37 billion activated for each token. It incorporates Multi-head Latent Attention (MLA) and a unique load-balancing strategy that avoids auxiliary losses, alongside multi-token prediction training objectives. Pre-trained on a massive dataset of 14.8 trillion tokens, this model excels in complex reasoning, mathematical problem-solving, and code generation. The MLA architecture is used for inference, while the DeepSeekMoE architecture ensures cost-effective training. The load-balancing strategy efficiently distributes computational tasks across experts, preventing interference that could hinder model accuracy. The model's multi-token prediction training enhances data efficiency and accelerates inference through speculative decoding. DeepSeek-V3's performance has been rigorously validated on various benchmarks, surpassing other open-source models with scores of 88.5 and 75.9 on the MMLU and MMLU-Pro educational datasets, respectively, and 90.2 on the MATH-500 mathematical reasoning task. Impressively, DeepSeek-V3 achieved these state-of-the-art capabilities for a relatively low training cost of $5.576 million, utilizing just 2.788 million H800 GPU hours.