LongCat-2.0

We introduce LongCat-2.0, a large-scale MoE language model with 1.6 trillion total parameters and ~48 billion activated per token — a substantial step up from previous LongCat models, accompanied by several architectural improvements.

Both the full training run and the large-scale deployment are built entirely on AI ASIC superpods.

Pretraining spans millions of accelerator-hours across more than 35 trillion tokens, with no rollbacks or irrecoverable loss spikes — demonstrating that we have the capability to conduct frontier-scale training on alternative hardware platforms.

Tech Blog

Introduction

To strengthen the model on long-horizon tasks, we introduce LongCat Sparse Attention and train LongCat-2.0 on hundreds of billions of tokens of 1M-context data. Together with dedicated post-training, this gives LongCat-2.0 strong performance on coding and agentic tasks.

LongCat-2.0 is deeply integrated with mainstream harnesses such as Claude Code, OpenClaw, and Hermes, delivering strong performance across code understanding, repository-level edits, automated task execution, and agentic workflows — providing developers with a more stable and efficient collaborative experience.

LongCat Sparse Attention

To address the output discontinuity and quadratic scoring bottleneck of the Lightning Indexer in DSA, we introduce LongCat Sparse Attention (LSA). LSA features three orthogonal improvements:

• Streaming-aware Indexing (SI) reshapes the token selection budget to combine hardware-aligned contiguous access with dynamic random selection. This turns fragmented memory access into predictable sequential reads, achieving coalesced HBM access and high effective bandwidth.
• Cross-Layer Indexing (CLI) leverages the empirical stability of attention saliency across adjacent layers to amortize indexing cost: a single indexing pass serves several consecutive layers at inference time, made possible by cross-layer distillation during training.
• Hierarchical Indexing (HI) uses a coarse-to-fine, two-stage scoring scheme — first a coarse recall via block-level approximate scoring, then fine-grained token selection within the recalled candidates — shrinking the candidate space the indexer must process per query.

All strategies seamlessly extend to the 3-step Multi-Token Prediction module for speculative decoding. For CLI, the target model shares an index every 2 layers, while all 3 MTP draft steps share a single pass.

N-gram Embedding

LongCat-2.0 inherits N-gram Embedding from LongCat-Flash-Lite, improving parameter utilization efficiency by expanding parameters in sparse dimensions orthogonal to MoE. 135B N-gram Embedding parameters are included in the model, which adheres to the following scaling principles:

• The sparsity of MoE has crossed the sweet spot. Given that the model’s sparsity has already reached approximately 97% even without considering the N-gram Embedding, the performance gain from scaling up experts by 135B parameters is negligible. In contrast, an N-gram Embedding of the equivalent parameter scale yields benefits far exceeding those of standard experts.
• The proportion of N-gram Embedding is constrained within an optimal range. Scaling experiments reveal that when n-gram embedding parameters consume an excessive share of the total parameter budget (over 50%), their advantage over scaling up experts diminishes. In LongCat-2.0, this proportion is strictly kept under 10%, operating well within a safe margin.

These two principles guarantee the robust superiority of N-gram Embedding compared to equivalent-sized pure MoE models. For inference, shifting parameters from experts to N-gram Embedding reduces large-batch decoding memory I/O, accelerating generation.

Scalable Training Infrastructure on Superpod Accelerators

LongCat-2.0 is pre-trained on over 50K AI accelerators, introducing significant system-level challenges due to both model and cluster scale. We address these challenges through systematic optimizations, achieving over 35% training throughput improvement while also enhancing reliability compared to a naive implementation.

• Determinism & Reliability: The system ensures training reproducibility through enforced determinism, enhances numerical reliability by optimizing foundational operators, and achieves seamless fault recovery via automated monitoring to fully secure stable production operations.
• Training at Scale: The system effectively overcomes hardware memory bottlenecks and boosts large-scale training efficiency by integrating 6D parallelism with super-node architectures, applying multi-dimensional memory optimizations, and pioneering the large-scale deployment of a customized Muon optimizer.
• Long Context Training: The system tackles large-scale long-context training challenges by implementing optimized in-house operators, scaling context length to 1M via an all-gather-based CP parallelism scheme, and meticulously designing compute-communication overlap to minimize synchronization overhead.

Inference Optimization on Superpod Accelerators

Serving a 1.6T-parameter model over a 1M-token context presents a significant challenge, particularly under tight constraints on HBM capacity, HBM I/O bandwidth, and inter-node interconnect bandwidth. We address this challenge through a stack of optimizations at the model, device, and deployment levels.

• Model-Specific Optimization: To optimize Attention and ScMoE, the system mitigates KV-cache overhead and memory pressure through indexer pipelining and KV-cache parallelism, while leveraging explicit per-core control to achieve fully parallel execution of dense and MoE branches.
• Accelerator-Oriented Optimization: The system optimizes overall performance by employing super kernels to minimize execution overhead, leveraging L2 cache for weight prefetching to hide I/O latency, and utilizing high-speed interconnects for efficient scale-up and scale-out distributed processing.
• Deployment & Serving: LongCat-2.0 optimizes serving performance through a prefill-decode disaggregated architecture that leverages tailored parallelism schemes—CPP and SP for prefill, KVP and large EP for decode—alongside asynchronous load balancing to overcome stage-specific bottlenecks efficiently.

Learning from Multiple Teachers

We introduce a specialized expert-group design in the post-training pipeline, organized into three categories: Agent Experts, Reasoning Experts, and Interaction Experts.

• Agent Experts focus on improving autonomous task execution in complex real-world scenarios. They achieve SOTA-level performance in fine-grained vertical domains such as code, work, and search. During training, we optimize not only end-to-end task success rates, but also the atomic capabilities that underpin agent robustness, including precise tool invocation, reliable parameter parsing in multi-turn API interactions, and self-correction mechanisms that mitigate infinite loops and repetitive calls.
• Reasoning Experts extend the model’s depth of logical reasoning and enable adaptive computation based on problem difficulty. These experts deliver strong performance on mathematics, STEM problem solving, and multi-hop reasoning tasks, improving the model’s ability to handle complex analytical scenarios.
• Interaction Experts focus on human alignment and user experience optimization. They improve fine-grained instruction following across diverse applications, suppress factual hallucination through advanced alignment techniques, and establish well-bounded safety mechanisms without compromising usefulness.

Finally, we adopt the MOPD architecture to integrate the strongest capabilities from these three expert groups. This fusion enables the final model to combine strong agentic execution, deep reasoning, and high-quality interaction, allowing it to accurately understand complex user needs and reliably complete challenging real-world tasks.

Evaluation Results

We evaluate LongCat-2.0 against leading proprietary models across agentic, coding, search, productivity and foundational capabilities. Unless noted with *, all scores are measured in-house under a unified harness.

Notes: * — cited from the model’s official report; - — no comparable public score.

Deployment

LongCat-2.0 can be deployed on both GPU and NPU platforms.

GPU

For GPU deployment, please refer to the SGLang cookbook.

NPU

For NPU deployment, please refer to SGLang-FluentLLM.

Chat Template

We provide a chat template for LongCat-2.0 in the tokenizer_config.json file, which can be used to encode a list of messages into a single string for model input.

Here is a brief example of how to use the template:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meituan-longcat/LongCat-2.0", trust_remote_code=True)

tools = [
    {
        "type": "function",
        "function": {
            "name": "func_add",
            "description": "Calculate the sum of two numbers",
            "parameters": {
                "type": "object",
                "properties": {
                    "x1": {"type": "number", "description": "The first number to add"},
                    "x2": {"type": "number", "description": "The second number to add"},
                },
                "required": ["x1", "x2"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "func_multiply",
            "description": "Calculate the product of two numbers",
            "parameters": {
                "type": "object",
                "properties": {
                    "x1": {"type": "number", "description": "The first number to multiply"},
                    "x2": {"type": "number", "description": "The second number to multiply"},
                },
                "required": ["x1", "x2"],
            },
        },
    },
]

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Calculate 1+1"},
    {
        "role": "assistant",
        "reasoning_content": "Calling func_add to calculate 1+1",
        # Note: unlike the standard OpenAI format, we expect `arguments` to be a dict rather than a string.
        "tool_calls": [
            {"type": "function", "function": {"name": "func_add", "arguments": {"x1": 1, "x2": 1}}},
        ],
    },
    {"role": "tool", "name": "func_add", "content": '{"ans": 2}'},
    {"role": "assistant", "reasoning_content": "The result is 2", "content": "2"},
    {"role": "user", "content": "Check your answer, is it correct?"},
]

# thinking mode on
prompt_think = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    enable_thinking=True,
    add_generation_prompt=True
)

# thinking mode on, keeping all reasoning content for better performance
prompt_full = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    enable_thinking=True,
    add_generation_prompt=True,
    save_reasoning_content=True
)

# thinking mode off, for better token efficiency
prompt_no_think = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    enable_thinking=False,
    add_generation_prompt=True
)

License Agreement

The model weights are released under the MIT License.

Any contributions to this repository are licensed under the MIT License, unless otherwise stated. This license does not grant any rights to use Meituan trademarks or patents.

Usage Considerations

This model has not been specifically designed or comprehensively evaluated for every possible downstream application.

Developers should take into account the known limitations of large language models, including performance variations across different languages, and carefully assess accuracy, safety, and fairness before deploying the model in sensitive or high-risk scenarios. It is the responsibility of developers and downstream users to understand and comply with all applicable laws and regulations relevant to their use case, including but not limited to data protection, privacy, and content safety requirements.

Nothing in this Model Card should be interpreted as altering or restricting the terms of the MIT License under which the model is released.

Contact

Please contact us at longcat-team@meituan.com or open an issue if you have any questions.