CorX Labs — releases and notes

3 Billion Tokens on One GPU: How CorX1.5 Was Trained

Nathan — Sat, 01 Aug 2026 00:00:00 +0000

Everyone asks how 3 billion training tokens fit on a single GPU. They never had to. A technical breakdown of what actually has to fit in VRAM for a 158M-parameter model, the 6 GB memory-mapped token file the dataset really lives in, sequence packing, gradient accumulation, and why the whole pretraining run is only about 5,700 optimiser steps.

Announcement: CorX Labs releases CorX1.5

CorX Labs — Sat, 01 Aug 2026 00:00:00 +0000

CorX Labs, an independent AI research lab in Jamaica, has released CorX1.5 — a 158-million-parameter language model designed and trained entirely from scratch by a single developer on a single GPU, under the Apache 2.0 licence.

CorX1.5 — a 158M parameter language model built from scratch

Nathan — Sat, 01 Aug 2026 00:00:00 +0000

CorX1.5 is an open-source language model with roughly 158 million parameters, built entirely from scratch by a single developer in Jamaica. It is a custom decoder-only transformer using RoPE positional encoding, SwiGLU feed-forward layers and RMSNorm, with learned role embeddings, assistant-only loss masking, 10 layers, a 1,024-token context window and a 32,768-token custom BPE vocabulary.

Trained in three stages — pretraining on ~3B tokens across 25 open datasets, distillation from a larger teacher model, then reasoning training with step-by-step blocks. Released under Apache 2.0 on Hugging Face.