{"full_name":"jundot/omlx","owner":"jundot","name":"omlx","shard":"1","url":"https://github.com/jundot/omlx","description":"LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar","homepage":"https://omlx.ai","topics":["apple-silicon","inference-server","llm","macos","mlx","openai-api"],"tag_pages":["apple-silicon","llm","macos","mlx","openai-api"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/64250138?u=e95cca863622c88961c53f9b3fffe2d5982c8a6f&v=4","created_at":"2026-02-13T14:13:27Z","pushed_at":"2026-11-15T09:50:11Z","metrics":{"stars":23653,"forks":2134,"watchers":113,"issues":2393,"prs":2570,"releases":138,"issues_open":321,"issues_closed":2072,"prs_open":182,"prs_closed":1027,"prs_merged":1361,"discussions":333},"growth":{"day":{"stars":19,"forks":6,"base_day":"2026-11-15"},"week":{"stars":325,"forks":44,"base_day":"2026-11-04"},"month":{"stars":789,"forks":75,"base_day":"2026-10-17"}},"history":{"dates":["2026-09-06","2026-09-07","2026-09-08","2026-09-09","2026-09-10","2026-09-11","2026-09-12","2026-09-13","2026-09-14","2026-09-15","2026-09-16","2026-09-17","2026-09-18","2026-09-19","2026-09-20","2026-09-21","2026-09-22","2026-09-23","2026-09-24","2026-09-25","2026-09-26","2026-09-27","2026-09-28","2026-09-29","2026-09-30","2026-10-01","2026-10-02","2026-10-03","2026-10-04","2026-10-05","2026-10-06","2026-10-07","2026-10-08","2026-10-09","2026-10-10","2026-10-11","2026-10-12","2026-10-13","2026-10-14","2026-10-15","2026-10-16","2026-10-17","2026-10-18","2026-10-19","2026-10-20","2026-10-21","2026-10-22","2026-10-23","2026-10-24","2026-10-25","2026-10-26","2026-10-27","2026-10-28","2026-10-29","2026-10-30","2026-10-31","2026-11-01","2026-11-02","2026-11-03","2026-11-04","2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[21434,21473,21507,21533,21562,21598,21638,21658,21684,21723,21767,21811,21843,21883,21919,21963,22002,22072,22141,22190,22238,22275,22302,22334,22369,22400,22425,22462,22489,22509,22539,22580,22610,22638,22655,22689,22715,22734,22764,22800,22837,22864,22886,22907,22935,22963,22977,22992,23014,23029,23044,23071,23102,23129,23158,23203,23228,23265,23289,23328,23498,23531,23560,23576,23605,23634,23653],"forks":[1828,1834,1835,1837,1840,1848,1854,1861,1873,1878,1882,1884,1886,1894,1900,1901,1901,1910,1920,1923,1929,1932,1934,1940,1944,1950,1958,1960,1962,1963,1969,1973,1980,1986,1990,1997,2003,2005,2012,2021,2030,2059,2096,2136,2171,2148,2048,2048,2051,2057,2060,2066,2070,2071,2074,2081,2081,2086,2087,2090,2117,2122,2123,2124,2126,2128,2134],"watchers":[110,111,112,112,112,113,114,114,114,114,114,114,114,115,115,115,114,115,116,115,117,116,116,117,117,117,117,116,116,116,117,117,117,117,117,118,118,118,117,117,117,118,118,116,118,118,116,117,116,114,114,115,114,113,114,114,115,115,115,115,null,114,114,114,113,113,113],"issues_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,321,323,317,324,329,321],"prs_open":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,173,162,150,165,178,182]},"boards":{},"same_owner":[],"issues":[{"number":114,"title":"m5 运行很慢","comments":27,"created_at":"2026-03-10T00:13:31Z"},{"number":1835,"title":"macOS 27: long-context inference 10x slower than macOS 26.6 (even after HOST_VM_INFO64 fix)","comments":16,"created_at":"2026-06-11T22:51:21Z"},{"number":1550,"title":"Qwen 3.6 MTP + batch of 2 = huge slowdown and corruption","comments":14,"created_at":"2026-05-30T15:54:38Z"},{"number":4213,"title":"0.7.0 release: Memory Guard dynamic ceiling issue killing workflow","comments":13,"created_at":"2026-10-02T17:24:09Z"},{"number":5223,"title":"Prefill throttle still inflates per_token after the enforcer reclaims the pool (single request, tight ceiling): follow-up to #4587 / #4603","comments":12,"created_at":"2026-11-13T00:53:16Z"},{"number":5075,"title":"Time To Full Decode Speed (TTFDS) and Stream Interrupt","comments":12,"created_at":"2026-11-04T10:46:49Z"},{"number":2150,"title":"Lightning MTP (PR #2113) regresses tgTPS by 8-12% on Qwen3.6-35B-A3B (MoE) vs. old depth-1 MTP, and is fully bypassed during continuous batching","comments":11,"created_at":"2026-07-09T19:20:30Z"},{"number":3697,"title":"Can mlx-lm/mlx-vlm be kept up-to-date?","comments":10,"created_at":"2026-09-16T08:54:53Z"}],"discussions":[{"number":5270,"title":"Per-request LoRA adapters, to use AutoTrust JEV-27B for both chat and decisions","comments":1,"category":"Ideas","created_at":"2026-11-14T18:26:45Z"},{"number":5256,"title":"Separate models selection for audio processing (tts & stt) and reasoning in Ask Ai","comments":0,"category":"Show and tell","created_at":"2026-11-14T07:17:17Z"},{"number":5249,"title":"Prefill Accelerator Mac Upgrade Help","comments":0,"category":"General","created_at":"2026-11-13T19:42:43Z"},{"number":5231,"title":"Parallel requests for decision model","comments":0,"category":"Ideas","created_at":"2026-11-13T09:27:01Z"},{"number":5220,"title":"LeapQuant support","comments":0,"category":"Ideas","created_at":"2026-11-12T22:40:04Z"},{"number":5217,"title":"Cross-instance prefix cache transfer for load-balanced oMLX replicas","comments":0,"category":"Ideas","created_at":"2026-11-12T21:20:02Z"},{"number":5215,"title":"Try same-prompt prefill across two CUDA helper hosts","comments":0,"category":"Ideas","created_at":"2026-11-12T19:10:46Z"},{"number":5214,"title":"Support discrete CUDA VRAM monitoring for mixed clusters","comments":0,"category":"Ideas","created_at":"2026-11-12T19:10:05Z"}],"latest_release":{"name":"0.8.0rc1","tag":"v0.8.0rc1","published_at":"2026-11-12T16:06:56Z","body":"# oMLX 0.8.0rc1\r\n\r\n> This is a release candidate. Please report any bugs through [GitHub Issues](https://github.com/jundot/omlx/issues/new/choose). If you need a stable release, please use [oMLX 0.7.3](https://github.com/jundot/omlx/releases/tag/v0.7.3).\r\n\r\nThe release candidate for oMLX 0.8.0 adds the Prefill Accelerator, which lets a PC with NVIDIA GPUs or a second Mac speed up prefill for a big model on your Mac, and makes DeepSeek V4.1 faster with many requests and on clusters.\r\n\r\n**Download:** [macOS 26 / 27](https://github.com/jundot/omlx/releases/download/v0.8.0rc1/oMLX-0.8.0rc1-macos26-27.dmg) | [macOS 15 Sequoia](https://github.com/jundot/omlx/releases/download/v0.8.0rc1/oMLX-0.8.0rc1-macos15-sequoia.dmg)\r\n<sub><i>* If you downloaded build 3727 right after release, the Cluster tab may not find a DGX Spark or NVIDIA PC, so please download it again.</i></sub>\r\n\r\n<img width=\"2400\" height=\"1350\" alt=\"image-1794476752259\" src=\"https://github.com/user-attachments/assets/dfa46522-dd0d-425d-8a16-ca2b3eac042c\" />\r\n\r\n## Prefill Accelerator with NVIDIA GPUs or a Mac\r\n\r\nA PC with NVIDIA GPUs, a DGX Spark or a second Mac with a faster GPU runs the first layers of each prompt chunk, while your Mac runs the rest, decodes and keeps the KV cache. Auto picks how many layers the helper takes from the measured speed of both machines and the link, and keeps the prompt on the Mac when a split would be slower (#5065, #5163, #5195).\r\n\r\nThe CUDA helper builds on the great CUDA kernels of [TensorFold](https://github.com/ashhart/TensorFold) by @ashhart and [mlx-serve](https://github.com/ddalcu/mlx-serve) by @ddalcu, and would not have been possible without them. Thank you both for sharing them in the open. The adapted kernels and their licenses are in [`third_party/`](https://github.com/jundot/omlx/tree/v0.8.0rc1/apps/omlx-cuda-helper/omlx_cuda_helper/third_party).\r\n\r\n> Prefill tok/s on a 64K prompt, oQ4e models, M3 Ultra 512GB with a CUDA helper over 2.5 GbE. Decode speed stays about the same.\r\n\r\n| Model | M3 Ultra alone | + DGX Spark | + 2x RTX A6000 |\r\n|---|---:|---:|---:|\r\n| Qwen3.8 27B | 352 | 1,062 (+202%) | 1,879 (+434%) |\r\n| Qwen3.8 Flash-Next | 1,370 | 2,715 (+98%) | 3,928 (+187%) |\r\n| DeepSeek V4.1 Flash | 1,085 | 2,412 (+122%) | 1,914 (+76%) |\r\n| GLM-5.3 Flash | 555 | 1,076 (+94%) | 978 (+76%) |\r\n| MiniMax M3.1 Flash | 312 | 504 (+62%) | 468 (+50%) |\r\n\r\nTurn on Global Settings > Advanced > Enable Distributed Inference, pair the helper in the Cluster tab, then pick it under Prefill Accelerator in the model's settings ([guide](https://github.com/jundot/omlx/blob/v0.8.0rc1/docs/experimental/prefill-accelerator.md)). An NVIDIA PC needs only the command below and the code from `docker logs omlx-helper`, with no SSH or oMLX install. It needs driver 580 or newer and an Ampere or newer GPU or a GB10, and has not been tested on Windows yet.\r\n\r\n```\r\ndocker run -d --name omlx-helper --restart unless-stopped --gpus all --network host -v omlx-helper:/data jundot/omlx-cuda-helper\r\n```\r\n\r\n<img src=\"https://github.com/user-attachments/assets/5ac66493-51a3-461f-9330-5f2b1f16b69c\" height=\"180\" alt=\"CUDA helper in the Cluster tab\"> <img height=\"180\" alt=\"Screenshot 2026-11-11 at 1 44 44 PM\" src=\"https://github.com/user-attachments/assets/8703d8b4-8f37-4c35-b649-7c833552ff79\" />\r\n\r\n## Faster DeepSeek V4.1 with Many Requests and Clusters\r\n\r\nWith the DSpark drafter on, DeepSeek V4.1 now verifies the drafts of several requests together, so the drafter keeps its speedup as requests add up (#5154). Tensor-parallel clusters now run the same fast prefill and decode kernels as a single Mac, and pipeline clusters run the drafter for several requests (#5086, #5154).\r\n\r\n| DeepSeek V4.1 Flash | Change | Hardware |\r\n|---|---|---|\r\n| oQ3e-mtp, DSpark on, 8 requests | Total decode 73.2 -> 107.9 tok/s (+47%) | M3 Ultra |\r\n| oQ3e-mtp, DSpark on, 2 requests, pipeline | Total decode 46.7 -> 61.7 tok/s (+32%) | M3 Ultra + M5 Max |\r\n| oQ4e, tensor parallel, 32K prompt (esti","is_prerelease":false},"readme":true,"readme_zh":false,"news":[{"title":"OMLX: Local LLM Server for Apple Silicon Macs","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=48093410","source":"Hacker News","lang":"en","date":"2026-05-11","points":4,"comments":0},{"title":"Show HN: oMLX – SSD-backed KV cache cuts coding agent TTFT from 90s to 1s on Mac","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=47247294","source":"Hacker News","lang":"en","date":"2026-03-04","points":4,"comments":1},{"title":"Show HN: oMLX – coding agents on local LLMs without the painful reprefill","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=47139158","source":"Hacker News","lang":"en","date":"2026-02-24","points":3,"comments":0},{"title":"OMLX – LLM Inference Server for Apple Silicon (Ollama for MLX)","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=47005152","source":"Hacker News","lang":"en","date":"2026-02-13","points":3,"comments":0},{"title":"OMLX v0.3.9 Stable Merges Native MTP (Multi-Token Prediction)","url":"https://github.com/jundot/omlx/releases/tag/v0.3.9","hn":"https://news.ycombinator.com/item?id=48232765","source":"Hacker News","lang":"en","date":"2026-05-22","points":2,"comments":0},{"title":"OMLX – Ollama for MLX (LLM Inference Server for Apple Silicon)","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=47024470","source":"Hacker News","lang":"en","date":"2026-02-15","points":2,"comments":0},{"title":"OMLX: Use a second Mac to speed up local LLM prefill 2.9x","url":"https://github.com/jundot/omlx/releases/tag/v0.7.4.dev1","hn":"https://news.ycombinator.com/item?id=50403212","source":"Hacker News","lang":"en","date":"2026-11-07","points":1,"comments":0},{"title":"Show HN: oMLX – Native Mac inference server that persists KV cache to SSD","url":"https://github.com/jundot/omlx","hn":"https://news.ycombinator.com/item?id=47080277","source":"Hacker News","lang":"en","date":"2026-02-19","points":1,"comments":0},{"title":"How to use oMLX: Keep models resident on your Mac and stop recalculating conversation continuations","url":"https://note.com/opensourcelab/n/na952cb406af0?hl=en","source":"note","lang":"en","date":"2026-09-07"},{"title":"TTL이라는 답을 찾아 아날로그로 속도를 측정한 이야기 (oMLX 편)","url":"https://note.com/aighost_log/n/nc6c80c85a19b?hl=ko","source":"note","lang":"en","date":"2026-08-16"},{"title":"Mac 本機 AI 編碼 agent 搭建指南：oMLX + OpenCode + Qwen 3.6 完整設定教學","url":"https://tw.news.yahoo.com/mac-%E6%9C%AC%E6%A9%9F-ai-%E7%B7%A8%E7%A2%BC-agent-230807301.html","source":"Yahoo奇摩新聞","lang":"zh","date":"2026-05-18"},{"title":"[May 2026 Latest] Which one is truly the fastest for local LLMs on Mac? A thorough comparison of mlx-lm / vllm-mlx / oMLX / Ollama / LM Studio (M3 Max 64GB actual measurements)","url":"https://note.com/zephel01/n/ncf2b8b652c01?hl=en","source":"note","lang":"en","date":"2026-05-13"},{"title":"Mac用户可以在oMLX中使用TurboQuant了，搭配Gemma-4-31B，谷歌全家桶实测很能打！","url":"https://news.qq.com/rain/a/20260408A03GLU00","source":"腾讯网","lang":"zh","date":"2026-04-08"},{"title":"oMLX 本地大語言模型伺服器 MacOS 平台地端 LLM 部署實作","url":"https://cyberq.tw/2026/03/10/omlx-local-large-language-model-server/","source":"cyberq","lang":"zh","date":"2026-03-09"},{"title":"Run LLMs on a Mac — 81× Faster Context Switching with oMLX","url":"https://news.google.com/rss/articles/CBMiQkFVX3lxTE41akZQSUs1R01YTzV6bjZyR3VnWHVJLWI3dWR1VlhoRWE1em5xa1R6TTZZbzh2azl5c3lPNTF0d2p5UQ?oc=5","source":"YouTube","lang":"en","date":"2026-09-17"},{"title":"DFlash 2 发布：解码速度近自回归 3 倍，多模型平均接受长度显著提升！","url":"https://news.google.com/rss/articles/CBMia0FVX3lxTE1WdGZsWXh6ZEJLejVKVW9Ea2tFRUVmNnlZTHdhdXR5T25hZ1hJck14SFNtWlplT3RTQW5NSXRFNjVxYlJGZ1B0VkdZS0tlSDQ2UHlwRFhOTElxV09lWjNiVG1SNy1LalY5OXpz?oc=5","source":"CSDN博客","lang":"zh","date":"2026-08-20"},{"title":"oMLX：为 Mac 优化的大语言模型推理工具，多特性助力高效开发与应用_IT界那些事儿","url":"https://news.google.com/rss/articles/CBMiYkFVX3lxTE1FX25Ba053YWNUOXFFV0tkeE1qUGZzWHJpOWdJVFZUQWw4ZU1qRlRfNElfdHYwdXlrLU9kRVFKTHE0emNra2pPWVNBWVJEeGJwUkJtclRkZ2E0UWNfcDFiNHB3?oc=5","source":"CSDN","lang":"zh","date":"2026-05-11"}],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}