Ornith-1.5-35B-A3B, REAP-192, mxfp4
The base model is the Ornith Team's, not mine. My work was compressing it to run on a 24 GB Mac mini, where the plain 4-bit conversion does not fit. I pruned each layer from 256 to 192 experts with REAP, quantised to mxfp4 while keeping the shared expert and router at 8 bits, and lowered the active experts per token from 8 to 7. There was no fine-tuning.
- decode
- 49.8 tok/s
- weights
- 14.2 GB
- 42-task suite, unpruned 3-bit 0.954
- 0.981
Limits. The task suite is finite and English only: coding, math, tool calls, structured output and format checks. Expect some loss on knowledge-heavy tasks. I could not run the bf16 original on any hardware I had, so there is no direct comparison with it. The build is tuned for one user at a time.