Results

Every ExeaLM-versus-Llama run so far. Lower bits per byte is better.

ExeaLM

1.464

Llama

1.481

ExeaLM at 0.1B, same training time as a same-size Llama

Equal steps and equal compute

Fig. 01

Equal compute (same training time)Equal steps

48M, first try (half attention)0.1B, hybrid at lr 1.5e-3

The hybrid's lead over Llama by model size. Above zero, the hybrid is better.

Every run

Bits per byte on held-out text, lower is better. Each row is one training run. Point at a column name to sort.

#	Run	Model	Layout	Budget	lr	Seed
1	eqt_big_attn6	TONN	attention at 1, 2, 4, 5, 7, 8conv start	2,000 s	0.0015	0	1.365	1.435	1.327	1.655	1.446
2	eqt_big_attn6_s1	TONN	attention at 1, 2, 4, 5, 7, 8conv start	2,000 s	0.0015	1	1.367	1.437	1.328	1.650	1.446
3	eqt_big_llama_s1	Llama	all attention	2,000 s	0.001	1	1.369	1.439	1.333	1.665	1.451
4	eqt_big_llama	Llama	all attention	2,000 s	0.001	0	1.370	1.439	1.332	1.668	1.452
5	eqt_100m_hybrid_lr1e3	TONN	attention at 1, 2, 4, 5, 7, 8, 10, 11, 13conv start	3,000 s	0.001	0	1.373	1.450	1.339	1.692	1.464
6	eqt_big_hybrid_lr15e4	TONN	attention at 1, 3, 5, 7, 9conv start	2,000 s	0.0015	0	1.374	1.453	1.337	1.699	1.466
7	hconv_6k	TONN	attention at 1, 3, 5conv start	6,000 steps	0.002	0	1.390	1.455	1.343	1.676	1.466
8	eqt_big_hybrid	TONN	attention at 1, 3, 5, 7, 9conv start	2,000 s	0.002	0	1.379	1.456	1.341	1.712	1.472
9	llama_6k	Llama	all attention	6,000 steps	0.002	0	1.395	1.459	1.348	1.686	1.472
10	eqt_100m_hybrid	TONN	attention at 1, 2, 4, 5, 7, 8, 10, 11, 13conv start	3,000 s	0.0015	0	1.386	1.462	1.347	1.702	1.474
11	eqt_big_hybrid_lr1e3	TONN	attention at 1, 3, 5, 7, 9conv start	2,000 s	0.001	0	1.377	1.458	1.341	1.723	1.475
12	eqt_100m_llama	Llama	all attention	3,000 s	0.001	0	1.391	1.468	1.352	1.713	1.481
13	mid_hybrid	TONN	attention at 1, 3, 5, 7conv start	3,000 steps	0.002	1	1.413	1.481	1.368	1.702	1.491
14	big_hybrid_3k	TONN	attention at 1, 3, 5, 7, 9conv start	3,000 steps	0.002	0	1.395	1.477	1.360	1.735	1.492
15	mid_hybrid	TONN	attention at 1, 3, 5, 7conv start	3,000 steps	0.002	0	1.412	1.484	1.373	1.702	1.493
16	eqt_mid_hybrid	TONN	attention at 1, 3, 5, 7conv start	1,000 s	0.002	1	1.415	1.486	1.373	1.701	1.494
17	eqt_mid_hybrid	TONN	attention at 1, 3, 5, 7conv start	1,000 s	0.002	0	1.418	1.492	1.376	1.719	1.502
18	eqt_mid_llama	Llama	all attention	1,000 s	0.001	1	1.418	1.490	1.380	1.720	1.502
19	eqt_mid_llama	Llama	all attention	1,000 s	0.001	0	1.417	1.494	1.379	1.730	1.505
20	big_llama_3k	Llama	all attention	3,000 steps	0.001	0	1.417	1.496	1.381	1.727	1.505
21	mid_llama_lr1e3	Llama	all attention	3,000 steps	0.001	1	1.435	1.508	1.397	1.741	1.520
22	eqt_small_hybrid	TONN	attention at 1, 3, 5conv start	600 s	0.002	1	1.445	1.512	1.395	1.738	1.522
23	eqt_small_llama	Llama	all attention	600 s	0.002	0	1.440	1.511	1.394	1.745	1.522
24	mid_hybrid_lr1e3	TONN	attention at 1, 3, 5, 7conv start	3,000 steps	0.001	0	1.426	1.512	1.390	1.768	1.524
25	hyb135	TONN	attention at 1, 3, 5	3,000 steps	0.002	0	1.449	1.516	1.399	1.736	1.525
26	hyb135_search	TONN + search	attention at 1, 3, 5	3,000 steps	0.002	0	1.448	1.517	1.400	1.734	1.525
27	mid_llama_lr1e3	Llama	all attention	3,000 steps	0.001	0	1.434	1.514	1.398	1.754	1.525
28	hyb135_mem	TONN + memory	attention at 1, 3, 5	3,000 steps	0.002	0	1.449	1.518	1.402	1.734	1.526
29	hconv_25	TONN	attention at 2, 5conv start	3,000 steps	0.002	1	1.452	1.517	1.399	1.743	1.528
30	hconv_attn4	TONN	attention at 1, 2, 4, 5conv start	3,000 steps	0.002	1	1.448	1.516	1.399	1.749	1.528
31	hconv_search	TONN + search	attention at 1, 3, 5conv start	3,000 steps	0.002	1	1.448	1.516	1.401	1.751	1.529
32	hconv_024	TONN	attention at 0, 2, 4conv start	3,000 steps	0.002	0	1.451	1.522	1.405	1.741	1.530
33	hyb135_conv	TONN	attention at 1, 3, 5conv start	3,000 steps	0.002	1	1.451	1.519	1.402	1.748	1.530
34	eqt_small_hybrid	TONN	attention at 1, 3, 5conv start	600 s	0.002	0	1.450	1.524	1.404	1.746	1.531
35	hconv_25	TONN	attention at 2, 5conv start	3,000 steps	0.002	0	1.451	1.523	1.406	1.744	1.531
36	hconv_search	TONN + search	attention at 1, 3, 5conv start	3,000 steps	0.002	0	1.452	1.522	1.406	1.744	1.531
37	hyb135_conv	TONN	attention at 1, 3, 5conv start	3,000 steps	0.002	0	1.452	1.523	1.406	1.745	1.532
38	eqt_small_llama	Llama	all attention	600 s	0.002	1	1.449	1.523	1.401	1.754	1.532
39	hconv_attn4	TONN	attention at 1, 2, 4, 5conv start	3,000 steps	0.002	0	1.453	1.527	1.407	1.760	1.537
40	mid_llama	Llama	all attention	3,000 steps	0.002	1	1.447	1.522	1.401	1.777	1.537
41	mid_llama	Llama	all attention	3,000 steps	0.002	0	1.444	1.524	1.402	1.778	1.537
42	hconv_mem	TONN + memory	attention at 1, 3, 5conv start	3,000 steps	0.002	0	1.456	1.532	1.410	1.764	1.540
43	hconv_mem	TONN + memory	attention at 1, 3, 5conv start	3,000 steps	0.002	1	1.455	1.528	1.410	1.773	1.542
44	hconv_024	TONN	attention at 0, 2, 4conv start	3,000 steps	0.002	1	1.461	1.530	1.411	1.778	1.545
45	arch	Llama	all attention	3,000 steps	0.002	0	1.461	1.534	1.413	1.775	1.546
46	llama_lr3e3	Llama	all attention	3,000 steps	0.003	0	1.465	1.538	1.415	1.769	1.547
47	llama_lr1e3	Llama	all attention	3,000 steps	0.001	0	1.468	1.548	1.425	1.776	1.554
48	arch	Llama	all attention	3,000 steps	0.002	1	1.473	1.545	1.426	1.803	1.562
49	rp4	TONN	all TONN	3,000 steps	0.002	0	1.475	1.555	1.424	1.869	1.581
50	hyb135_search	TONN + search	attention at 1, 3, 5	3,000 steps	0.002	1	1.455	1.543	1.407	1.933	1.584
51	arch	TONN + memory	all TONN	3,000 steps	0.002	1	1.475	1.559	1.421	1.906	1.590
52	hyb135_s1_curve	TONN	attention at 1, 3, 5	3,000 steps	0.002	1	1.461	1.551	1.412	1.949	1.593
53	hyb135_s1	TONN	attention at 1, 3, 5	3,000 steps	0.002	1	1.461	1.551	1.412	1.950	1.594
54		TONN + search	all TONN	3,000 steps	0.002	0	1.479	1.566	1.422	1.948	1.604
55		TONN	all TONN	3,000 steps	0.002	0	1.483	1.570	1.424	1.959	1.609
56		TONN + search	all TONN	3,000 steps	0.002	1	1.474	1.563	1.417	1.993	1.611
57		TONN	all TONN	3,000 steps	0.002	1	1.477	1.567	1.419	2.000	1.616
58	arch	TONN + memory	all TONN	3,000 steps	0.002	0	1.491	1.578	1.433	1.964	1.616