| Model profile | ||
|---|---|---|
| Model | Size | Approximate size |
| Inkling SmallModel source: provider1,048,576 tokens max native context | 276Bactive 12B | 165.6 GB4-bit quantization |
| Compatible hardware Estimated speeds, not benchmark results: calculated from memory bandwidth and model size. Real results can differ significantly because there is no precise formula for deriving LLM generation speed from hardware specifications alone. | ||||
|---|---|---|---|---|
| Hardware | Memory | Bandwidth | vLLM | Estimated generation |
| M5 Ultra | 256 GBUnified RAM | 1200 GB/s | no | 63 tok/s4-bit quantization |
| DGX H200 | 1128 GBGPU HBM3e | 4800 GB/s | yes | 197 tok/s4-bit quantization |
| DGX Station | 748 GBCoherent Memory | 7100 GB/s | yes | 255 tok/s4-bit quantization |
| ET900N G3 | 748 GBCoherent Memory | 7100 GB/s | yes | 255 tok/s4-bit quantization |
| DGX B200 | 1440 GBGPU HBM3e | 8000 GB/s | yes | 274 tok/s4-bit quantization |
| GB200 NVL72 | 13400 GBGPU HBM3e | 8000 GB/s | yes | 274 tok/s4-bit quantization |
| GB300 NVL72 | 20000 GBGPU HBM3e | 8000 GB/s | yes | 274 tok/s4-bit quantization |
Evaluates Inkling-Small MLX Q9 using Inferencer App on M3 Ultra 512 GiB hardware, highlighting its status as a benchmark-topping open weight model with multimodal inference capabilities.
Tests Inkling-Small, a general-purpose multimodal model accepting text, image and audio inputs to generate text outputs.