Lewi Surge Preview
An open mixture-of-experts language model for programming, trained from scratch by one person on free and donated compute. The current model is Lewi Surge Preview.
- Parameters
- 3.45B total
- Active per token
- ~0.5B
- Experts
- 128, top-6 + 1 shared
- Context in training
- 128k tokens
Results
Measured on the base model before and after code-focused annealing. No fine-tuning tricks, no cherry-picked prompts: greedy decoding, standard prompts, local runs.
| Benchmark | Metric | Before | After annealing |
|---|---|---|---|
| HumanEval | pass@1 | 18.9% | 25.0% |
| MBPP (sanitized) | pass@1 | 23.7% | 29.6% |
| HellaSwag | acc_norm | 41.9% | 43.4% |
| ARC-Challenge | acc_norm | 26.5% | 27.4% |
Knowledge-heavy benchmarks (ARC) barely move at this size; that is what the next stages and the larger model are for.
Architecture
Sixteen layers in four identical blocks. Three layers of Gated DeltaNet linear attention keep a fixed-size memory; every fourth layer is full attention. Each layer routes tokens to 6 of 128 small experts plus one shared expert.
- Gated DeltaNet (linear attention)
- Full attention
- Experts used by one token
Why long context is cheap here
Only four of sixteen layers keep a key-value cache, with two KV heads each. That is about 4 KB per token: a 128k-token context fits in roughly half a gigabyte, so long documents and whole repositories run on consumer hardware.
Training log
- Stage 1A 1.4B model trained from scratch on English, code, math and Ukrainian.
- GrowthGrown to 3.45B: twice the experts, one more block of layers, without restarting.
- Code-first pretraining1.6B tokens of curated code, instructions and textbooks on an H200.
- Annealing1.4B tokens, more than half of them conversations, with the learning rate decayed to zero.
- Fine-tuningChat behaviour on 16k-token conversations.
- NowContext extension to 128k and a second fine-tuning pass, on TPU v5e.
What comes next
Reasoning and tools
Fine-tuning with step-by-step reasoning traces and tool calls.
Learning from tests
Reinforcement learning where code only earns reward when its unit tests pass.
Images and video
Vision encoders connected through trained projectors.
Lewi Surge 24B2A
The full-size model, grown from the preview.
Open by default
The training code is already public on GitHub. The weights will follow on Hugging Face under an open license, with evaluation scripts. Ukrainian is one of Lewi's first-class languages.