i14 Trains and Serves Laplace 3B on AWS Trainium2

News LLM Inference InfrastructureLLM Training InfrastructureAI Agent Infrastructure

  • i14’s Laplace 3B uses Gated Linear Attention, keeping its recurrent state at 29.5 MB per sequence regardless of context length.
  • i14 and AWS rewrote the GLA implementation with standard PyTorch operators and ran it on Trainium2 through TorchNeuron Native, retaining the model’s GLA design and Muon optimizer.
  • Training on one trn2.48xlarge instance reached about 120,000 tokens per second and sustained that rate through step 84,210.
  • With an unmodified vLLM engine, one trn2.48xlarge served 11,544 output tokens per second across 1,280 concurrent sequences.

Our take

Unlike softmax-attention models whose KV cache grows with context, Laplace 3B offers more predictable per-sequence memory use for long-running workloads. Its recurrent state compresses history, however, so retrieval quality needs validation for each application. The implementation depends on Trainium2 and TorchNeuron Native, and the reported results do not establish cost or performance relative to GPU deployments.

Sources

  1. Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch · AWS Machine Learning

The summary and analysis were generated by AI. The original sources are authoritative. AI content policy