{"full_name":"ashhart/TensorFold","owner":"ashhart","name":"TensorFold","shard":"1","url":"https://github.com/ashhart/TensorFold","description":"LLM Inference Engine for Metal, CUDA and Vulkan.","homepage":"https://tensorfold.dev","topics":["ai","ai-tools","llm","llm-inference","llm-tools"],"tag_pages":["ai","ai-tools","llm","llm-inference","llm-tools"],"language":"Zig","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/71890694?u=aa10855ba3e7c72e32ef1d2e9d1360a3521f2d17&v=4","created_at":"2026-06-19T20:27:39Z","pushed_at":"2026-11-15T19:22:23Z","metrics":{"stars":2253,"forks":521,"watchers":11,"issues":863,"prs":2593,"releases":63,"issues_open":229,"issues_closed":634,"prs_open":386,"prs_closed":1043,"prs_merged":1164,"discussions":0},"growth":{"day":{"stars":38,"forks":4,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[1838,2038,2072,2215,2253],"forks":[388,398,401,517,521],"watchers":[10,11,11,11,11],"issues_open":[183,190,223,217,229],"prs_open":[290,330,367,365,386]},"boards":{"daily_rate":150,"weekly_rate":70,"monthly_rate":149},"same_owner":[],"issues":[{"number":1491,"title":"Add native Metal support for LFM3-1.7B and LFM3-VL-2B MLX 4/6/8-bit checkpoints","comments":63,"created_at":"2026-10-27T18:50:31Z"},{"number":3136,"title":"Native CUDA: bring up Spark-X2.6-4B affine-4 inference on Pascal sm_61","comments":51,"created_at":"2026-11-13T02:33:22Z"},{"number":2445,"title":"Qwen3.8: integrate remaining Yukon CUDA and Metal improvements","comments":49,"created_at":"2026-11-07T13:24:54Z"},{"number":2406,"title":"Qwen3.8-27B on native Windows RTX 5090: 250 tok/s single-stream prose gate","comments":48,"created_at":"2026-11-07T00:11:46Z"},{"number":2433,"title":"Qwen CUDA: opt-in native NVFP4 prefill with preserved decode and end-to-end gates","comments":41,"created_at":"2026-11-07T10:02:33Z"},{"number":2667,"title":"Proposal: fine-tune DSpine-b16 on the 4-bit Qwen3.8-27B target, with `--record` for Qwen3.8 on CUDA","comments":26,"created_at":"2026-11-09T11:09:47Z"},{"number":678,"title":"Living Weights: conflicting facts, updates and forgetting (measurements + proposal)","comments":26,"created_at":"2026-10-11T15:32:31Z"},{"number":2741,"title":"Opt-in coherent execution witness for exact-response cache qualification","comments":25,"created_at":"2026-11-10T10:35:05Z"}],"discussions":[],"latest_release":{"name":"TensorFold 1.2.9","tag":"v1.2.9","published_at":"2026-11-15T01:22:59Z","body":"# TensorFold 1.2.9\n\n1.2.9 brings faster decode and long prompts, an Ollama-compatible API, and fixes for a wrong reply and a hang. Every\nspeedup below keeps replies unchanged.\n\n## Speed\n\n- **Nemotron on Metal** (without an MTP head): concurrent streams share one round. On an M5 Ultra, two streams decode\n  about 20% faster, four about 75%, eight 2.3 to 2.5 times and 24 streams 3.2 to 3.5 times (#3005).\n- **GLM-5.3-Flash on CUDA:**\n  - Long prompts' first token comes 0.6% to 13.7% sooner at TP=3 from 131,072 to 2,000,000 tokens, and 6.7% sooner at\n    TP=2 at 1,000,000; shorter prompts at TP=2 stay within about 2% either way (#3159).\n  - After a 1,040,384-token prompt, decode rounds are 5.05%, 9.36% and 11.36% shorter at 1, 4 and 8 streams (#2722).\n- **Qwen3.8-Flash-Next on HIP** (gfx1151): one stream decodes about 6% faster, four streams 2.8% to 4.6% (#2697).\n- **Fewer CUDA launches** for Flash Next draws (#2927), Nemotron 3.5 draws (15 a level instead of about 112, #3124) and\n  Qwen3.8-27B prompts (#2925).\n\n## Models and features\n\n- **An Ollama-compatible API:** `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps` and\n  `/api/version` on the same engine paths as the OpenAI routes (#3342).\n- **`tensorfold serve --config FILE.toml`** reads serve flags from a file, so a launch can be checked in (#2885).\n- **Linux:** an opt-in memory-pressure governor driven by PSI, `TENSORFOLD_MEMORY_PRESSURE=1` (#2156).\n- **Groundwork, not used by any served model yet:** a CUDA image path for the dense engine (#2768), dots.mocr's vision\n  kernels (#2767), and HIP's TCP fabric for two hosts without RDMA (#2961).\n- **An experimental Living Weights v2 controller** in `tools/living_weights_v2` (#3320).\n\n## Server and API\n\n- `/v1/models` lists each model's `input_types`. A video sent to a model without video is a 400, not a 500 (#3019).\n- Logprobs work with tools, stop strings and reasoning, and on the Responses API (#3206).\n- With thinking off, a request's reasoning effort reaches the template (#3017).\n- Replies say when they kept a tool call as text, in `tensorfold.kept_tool_calls` and two `/metrics` counters (#2814).\n- Streams can carry the committed token IDs, opt-in (#3285).\n\n## Fixes\n\n- **Nemotron Omni on CUDA:** a failed image request no longer leaks into the next reply (#2985).\n- **GLM-5.3-Flash at TP=3:** `prompt_logprobs` over about 100 tokens no longer hangs (#1571).\n- **Living Weights:** the drift gate no longer refuses a lesson over the writer's own wording (#2929).\n- **CUDA:** the release server refuses a GPU it has no kernels for, by name, at startup (#2914).\n\n## For developers\n\n- `zig build test-host` gathers the Linux host checks (#2095); flaky tests fixed (#2963, #3015, #3310, #3347, #3349,\n  #3355); docs for `TF_GLM_PROF` (#2990) and a Jammed XSFP4 NZ5 row (#2981).\n\n## What changes by default\n\n- **GLM on CUDA:** the `TF_GLM_INDEX_CUDA` kernels and `TF_GLM_DSEL` are on by default.\n- **Flash Next on CUDA:** sampled draws are batched by default; `TF_FLASHNEXT_BATCH_DRAWS=0` restores per-stream draws.\n- **Flash Next on HIP:** on one RDNA 3.5 GPU (gfx1150 to gfx1153) serving alone, MTP drafts stop under a 0.8\n  confidence floor; other HIP GPUs and families keep 0.3, and `--mtp-confidence` still overrides it.\n- **Server:** `--reasoning-effort` no longer changes a thinking-off prompt.\n- **Server:** a `video_url` part sent to a model without video gets HTTP 400 (`invalid_request_error`), not a 500.\n- **CUDA:** the release server refuses, by name at startup, a GPU it has no kernels for.\n\nThe first three keep replies unchanged. The fourth changes the prompt of a thinking-off request when the server sets\n`--reasoning-effort`. The last two turn a failure into a named refusal.\n\n## Thanks\n\n@Bizuayeu, @Defilan, @garvrao80, @gilby, @jimthedj65, @jishnuvenugopal, @laudney, Mia's AI Lab (@MiaAI-Lab),\n@Nipale-ai, @Shoestring-dev, @ss-cong, @tarsnake and @WebReflection.\n","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}