Sign in

Blog › Hacker News

Image: JESHOOTS-com / Pixabay
Hacker News · 3 min read

LookAtMyQuant Squeezes Inkling-2-Mini Into 5.4 GB on Apple Silicon

Published Nov 22, 2026

An open-source quantization compiler now packs a 26.4B-parameter looped mixture-of-experts model into six Apple Silicon builds, from 3.85 GB for phones to a 12.8 GB build that matches the teacher.

A new release of LookAtMyQuant, an MIT-licensed quantization compiler, shrinks Thinking Machines' Inkling-2-Mini from 18.89 GB of text weights into six builds that run on Apple Silicon. The smallest weighs 3.85 GB and targets 8 GB iPhones. The 5.4 GB build in the headline sits in the middle of the range, and the largest, at 12.76 GB, is claimed to match the original on every benchmark the author ran.

What the compiler does

Most quantization tools apply one scheme everywhere: pick a bit width, round the weights, accept the damage. LookAtMyQuant takes a different route. It measures how much each stored piece of a model hurts the output when compressed, then assigns a code to each piece so the whole model fits a size budget with the least total damage. The measurements are expressed as KL divergence from the original model's output distribution.

The project reports that on Gemma 4 E2B its GGUF reaches a KL 12 times lower than Q4_K_M at the same file size. Earlier releases applied the recipe to Qwen3.5-4B, shrunk to 1.23 GB, and to a 27B Qwen model, reduced from 53.7 GB to 7.2 GB.

Inkling-2-Mini in six sizes

Inkling-2-Mini is an unusual target: a 26.4B-parameter mixture-of-experts model with about 5.4B active parameters, where a block of layers is looped three times, giving 48 logical layer visits across 128 experts with top-8 routing. The author compresses it with trellis codes, row-level bit allocation and a joint fine-tuning pass over all experts and attention projections.

The six builds, by file size and bits per weight:

  • 3.85 GB at 1.18 bpp, aimed at 8 GB devices
  • 4.72 GB at 1.45 bpp, sized to fit under the 12 GB iPhone memory ceiling alongside a small drafter model
  • 5.38 GB at 1.65 bpp
  • 8.87 GB at 2.72 bpp
  • 11.18 GB at 3.43 bpp
  • 12.76 GB at 3.92 bpp

The two largest builds keep the most sensitive expert buckets in their native NVFP4 form with zero error. According to the project's own evaluation on GSM8K, HumanEval, vision tests and multilingual text windows, the 12.76 GB build matches the BF16 teacher on all 349 pooled checks while using about a third less memory. The 8.87 GB build keeps roughly 99 percent of teacher performance and, the author says, cuts KL by 2.34 times on English, chat and code compared with Unsloth's 9.14 GB UD-Q2_K_XL GGUF.

The sub-2-bit builds are the more striking result. The 5.38 GB build reaches 87.0% on GSM8K and 71.3% on strict HumanEval. The 3.85 GB build, which the author says collapses under plain post-training quantization to near-zero scores, recovers to 78.5% on GSM8K and 48.2% on HumanEval after the fine-tuning stage.

Running it

All builds run through a Metal runtime installed with a single pip command, exposed as a lamq command-line tool and a Python API compatible with mlx_lm. Optional 4-bit speculative drafters, about 179 MB, speed up decoding. On an Apple M5 Max the author reports serial decode between roughly 60 and 75 tokens per second, and lossless drafted decode between roughly 100 and 125 tokens per second depending on the build.

The repository also documents how to reproduce the pipeline from scratch. A small SmolLM2-360M walkthrough measures, allocates and quantizes in three commands, and the Qwen3.5-4B recipe is laid out as named jobs, with a stated cost of about $34 of GPU time.

Why it matters

On-device inference is limited by memory long before it is limited by compute. A sparse model with 5.4B active parameters is attractive for phones, but only if its full 26B of weights can be made to fit. Per-piece bit allocation rather than uniform rounding is the idea doing the work here, and it is open and reproducible, which makes it easy to compare against GGUF quantizations from other projects.

Two caveats apply. The benchmark numbers come from the author, not an independent party, and the HumanEval and GSM8K samples are modest in size. The runtime also targets Apple Silicon only, so users on other hardware cannot yet use these files.

What to watch next: whether the allocation method gets ported to common GGUF and CUDA toolchains, and whether independent evaluations confirm that the small builds hold up on tasks beyond the benchmarks reported.

Original reporting: github.com

Discussion

Loading comments …

The Repocharts newsletter

Once a week: the biggest repo clashes, the fastest risers and the AI and tech stories that matter. Free, no spam, unsubscribe with one click.

We send a confirmation link first (double opt-in). See our privacy policy.