Exea Labs, a research lab run by high schoolers
An open language model, built by high schoolers.
Daily notes from building ExeaLM.
Latest · Day 2 · November 11, 2026
Seven pull requests from Aaron. One found a leak.
All seven are merged. One caught a leak worth up to 0.002 bits per byte.
Posts
Retrieval on real text needs a better search
The read works. The search finds the right article only 30% of the time.
5 min readPure TONN lost to Llama. The hybrid wins.
Two attention layers for every TONN layer. It beats a same-size Llama at equal compute, up to 0.1B.
5 min readAgents learn to talk. The model builds its own RAG.
Agents pass facts over the board 99.8% of the time. The model finds the right fact among 1,024 it has never seen.
5 min readStarting ExeaLM
An open 0.1B language model built by high schoolers, with retrieval and agent messaging inside its layers.
4 min readWhere ExeaLM stands
Ahead of a same-size Llama at 0.1B
Llama: 1.481. Same training time. One seed so far.
Trains at Llama's speed
Cost per token against Llama. Pure TONN was 1.54×.
Swarms help on real text
At least this much, after a leak fix. Two seeds.
Retrieval's open problem
How often the search finds the right article.
The model
Plates with circuits are TONN layers. The rest are attention. Point at a layer.
- Memory
- Short range
- Running state
- Gates and products
- Attention
Every third layer is TONN: a row of small circuits, with retrieval and a message board built in.
See insideWho builds it
Avneh Bhatia@AvnehSBhatia · Maintainer · 80 commits, Joshua SelvarajMaintainer, Aaron Chu@aarochu · 8 commits, Aariv Bhalla, Advaith Banigandlapati, Amanuel Semere Tsehay@amanx98, Anand B@Anandb71, Anirudh Ulabal@UAnirudh, Arnav Rohilla@midnightnothere, Arnav Saraf@Arnav-Saraf-Official, Avyank Vanamali@avyankvanamali-star, Devom Brahmbhatt@DevomB, Edmund Eric Gah@Eddiegah, Eknoor Singh Ahluwalia@eknoorahluwalia1, Hana Jannath, Ilhaan, Ishaan Mondal@IshaanM05, Ishaan Srinivasan@XytheRoblox, Jasraj Tulsi@jaztulsi, Luana Hua@238trimethylnonane, Naman Jain Kamalesh@man-coding-j, Naman Reddy@Naman20101, Raunak Mondal@raunak2007, Rishith Chennupati@rishith-c, Saad Sadaf@saad0-0hehe, Sehaj Randhir Singh@sehajr-singhs, Shivansh Behera@shivy7777, Sowndaryan Jayaprakash Anand@sowndyjay, Yash Patel@codeyash09
Sign up to contribute