Vortex: Progress Report on a Compact Language-Model Research Program
A functioning research prototype and training pipeline focused on useful instruction following, school-level reasoning, and lightweight local inference from roughly 175 million trainable parameters.
A functioning research prototype and training pipeline focused on useful instruction following, school-level reasoning, and lightweight local inference from roughly 175 million trainable parameters.
Project objective and status
Vortex is an experimental language-model project focused on obtaining useful instruction following, explanations, school-level reasoning, and lightweight local inference from roughly 175 million trainable parameters.
The project now includes a parameter-efficient decoder architecture, a data preparation pipeline, a CUDA trainer, a post-training pipeline, and read-only evaluation and audit tools. The final model name and best deployable checkpoint remain undecided.
Architecture actually used for the long run
The selected long-run artifact uses a dense configuration: 12 transformer blocks, hidden size 1,024, 16 query heads and 4 key/value heads, SwiGLU intermediate size 3,664, RMSNorm, QK normalization, rotary position encoding, an 8,192-token vocabulary, and tied input and output embeddings.
The model contains exactly 174,942,720 parameters under this configuration. The selected checkpoint came from the dense architecture, not the earlier hybrid architecture exploration.
Pretraining progress
The long continued-pretraining run used a mixed corpus of FineWeb-derived text, DCLM-derived text, mathematics, and code, with snapshot and overlap controls intended to reduce obvious contamination.
The selected artifact is the step-187,500 checkpoint. Training was designed for an RTX 5060 Ti with 16 GB of VRAM using staged sequence lengths, gradient accumulation, activation checkpointing, BF16 computation, and validation and checkpoint gates.
What the training pipeline accomplished
The engineering progress is real and reusable. The model can be trained, resumed, evaluated, and checkpointed on the target class of hardware without a cloud-scale cluster.
The capability evidence is mixed. Small logged tests show some gains on sampled multiple-choice tasks, no gain on HellaSwag, and a regression on a small GSM8K sample. The raw checkpoint also produced a severe repetition loop in interactive use.
Current limitations and next work
Vortex should be described as a functioning research prototype and training pipeline—not as a finished general-purpose assistant. Later supervised fine-tuning candidates reduce some formatting and repetition problems, but still produce shallow explanations and incorrect code in some cases.
The next gains are more likely to come from verified data, teacher distillation, careful checkpoint selection, and independent evaluation than from simply extending continued pretraining.