CUDA – Neuronix Lab

A deep teardown of DeepSeek-V3's Multi-Head Latent Attention (MLA): how it compresses the KV cache by 93.3% and breaks through the long-context VRAM wall, plus its auxiliary-loss-free MoE load balancing and Multi-Token Prediction acceleration strategy.

#CUDA #DeepSeek #KV-Cache

mla_attention_kernel.pypython

import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiHeadLatentAttention(nn.Module):
    def __init__(self, dim=7168, num_heads=128, head_dim=128, kv_lora_rank=512, qk_rope_dim=64):
        super().__init__()
        self.num_heads = num_heads
        self.head_dim = head_dim
        self.kv_lora_rank = kv_lora_rank
        self.qk_rope_dim = qk_rope_dim

Dr. Evelyn Vance 2025-02-14

Read

Inside DeepSeek-V3/R1: MLA Attention Compression and Multi-Token Prediction Kernels