Documentation
What CorX Labs is, what this website is, and how CorX1.5 — our first open-source language model — was designed, trained and released.
1. Overview
CorX Labs is an independent AI research lab and research station in Jamaica, in the Caribbean. It builds language models from scratch — architecture, tokenizer, training pipeline and fine-tuning — instead of adapting an existing base model. CorX1.5 is its first public release: a decoder-only transformer of roughly 158 million parameters, trained from random weights on a single GPU by one developer, and published open-source under the Apache 2.0 licence.
CorX1.5 is not a fine-tune. Every layer of it was written and trained here: a custom BPE tokenizer with a 32,768-token vocabulary, a custom decoder-only transformer with role embeddings and assistant-only loss masking, and a three-stage training run covering pretraining, distillation and reasoning.
The model is actively developed. New training data — new topics, new examples, patches for gaps found in testing — is added on an ongoing basis rather than treating the release as a one-time event. Updated checkpoints are published as training continues.
- Lab
- CorX Labs
- Location
- Jamaica, Caribbean
- First release
- CorX1.5
- Licence
- Apache 2.0
2. About this website
corx-labs.com is the home of CorX Labs. It exists so that the lab, its work and its models have one clear, permanent address instead of living only inside a model repository. The site has four parts:
- Documentation — this page. What the lab is, what the models are, and how they were built.
- Our Products — every model CorX Labs has released, each with a short summary and a link to its full details and its Hugging Face page.
- About Us — who is behind the lab and why it exists.
- Contact — how to reach us for business inquiries and research questions.
Model weights and code are not hosted here. They live on Hugging Face, which is linked from every model page.
3. Architecture
CorX1.5 is a custom decoder-only transformer. It is not copied from an existing model family, and it does not share weights or a config with one.
| Component | Choice | Why |
|---|---|---|
| Positional encoding | RoPE | Relative position information that extrapolates cleanly across the context window. |
| Feed-forward | SwiGLU | Gated activation; better quality per parameter than a plain MLP. |
| Normalisation | RMSNorm | Cheaper and more stable than LayerNorm at this scale. |
| Role embeddings | Learned, per token | Every token is tagged as user, assistant or raw text at the architecture level, so the model always knows whose turn it is — most models infer this only from special tokens. |
| Loss masking | Assistant-only | The model is trained to predict only the tokens it should generate, not the user's side of the conversation. |
| Parameters | 157.8M | Small enough to train and iterate on a single GPU. |
| Layers | 10 | — |
| Context length | 1,024 tokens | — |
| Vocabulary | 32,768 custom BPE | Tokenizer trained here rather than reused from another model. |
4. Training pipeline
CorX1.5 was trained in stages, each one building on the checkpoint before it.
-
Pretraining
~3 billion tokens across 25 open datasets spanning code, math, reasoning, conversation and general web text. A held-out validation split with early stopping was used to catch overfitting before it set in.
-
Distillation
Fine-tuned on responses generated by a larger teacher model (Qwen2.5-Instruct), with automated quality filtering that keeps only well-formatted, non-repetitive, correctly-verified answers.
-
Reasoning training
Trained to show step-by-step work in
<think>...</think>blocks before answering, using verified worked arithmetic and debugging exercises containing a known, injected bug that the answer can be checked against. -
Ongoing additions
Training has not stopped. New topics and targeted fixes are added as gaps turn up in testing, and updated checkpoints are pushed to the model repository. The commit history on Hugging Face is the source of truth for what is current.
5. Capabilities & limits
What it is good at
- Holding a consistent identity and conversational style
- Short, direct answers to greetings and everyday questions
- Declining requests it genuinely cannot fulfil — no internet, no memory between chats, no tool access — instead of fabricating an answer
- Small worked arithmetic, shown step by step
- Recognising and explaining common Python bugs in a structured format
What it is not
- A source of broad factual knowledge or general reasoning outside its training patterns
- Reliable for complex or multi-step maths beyond basic arithmetic
- Suitable for production or safety-critical use of any kind
- A competitor to frontier or lab-scale models
For context: models in a similar parameter range from major labs are typically trained on trillions of tokens. CorX1.5 was trained on billions. That gap is the honest explanation for most of what it cannot do, and it is documented here rather than glossed over. This is an independent research and learning project, shared in that spirit — small, transparent, and built to understand every part of how it works.
6. Safety
CorX1.5 includes a hardcoded response layer for messages indicating self-harm or crisis. Those messages return a fixed set of crisis resources rather than a generated response.
This is implemented in code, not left to the model's judgment. At this size, a generated response cannot be relied on to handle a crisis message correctly, so the decision is taken out of the model's hands entirely.
7. Usage
CorX1.5 uses a custom architecture and is not compatible with
AutoModel.from_pretrained() out of the box. Load it with the model
definition included in the repository — modeling_corx.py or the provided
inference script — alongside the checkpoint and tokenizer files.
# 1. Get the repository (weights, tokenizer, model definition)
git lfs install
git clone https://huggingface.co/Sigmandndnns/CorX1.5-160M
cd CorX1.5-160M
# 2. Load with the bundled model definition, not AutoModel
python inference.pyThe repository on Hugging Face is the authoritative source for file names, the current inference script and the latest checkpoint. Start there:
8. FAQ
Can I load it with AutoModel?
No. CorX1.5 uses a custom architecture and is not compatible
with AutoModel.from_pretrained() out of the box. Load it with the model
definition included in the Hugging Face repository (modeling_corx.py or
the provided inference script) alongside the checkpoint and tokenizer files.
What are role embeddings?
Role embeddings are a learned signal that marks every token as user, assistant or raw text at the architecture level, so the model always knows whose turn it is. Most models infer this only from special tokens in the text stream.
What licence is CorX1.5 under?
Apache 2.0 — commercial use, modification and redistribution are all permitted with attribution.
Should I use it in production?
No. CorX1.5 is an independent research and learning project. It should not be relied on for broad factual knowledge, complex multi-step maths, or any production or safety-critical use.
9. Licence
CorX1.5 is released under the Apache 2.0 licence. You may use, modify and redistribute it, including commercially, provided you keep the attribution and licence notice. The licence text ships with the model repository.
Questions about a specific use? Get in touch.
Next
See what we have released.
Every model CorX Labs publishes, with its specs, its limits and a direct link to run it.