llm-web

lean, a small LLM inference engine in Rust and WGSL that runs the same code natively and in the browser.

lean is an LLM inference engine written in Rust with WGSL compute kernels on wgpu, one codebase that runs natively and compiled to WebAssembly. In the browser it picks WebGPU, CPU threads or a single CPU thread depending on what the device supports. It loads GGUF weights (Q4_0, Q4_1, Q8_0, Q6_K) for Qwen2, Qwen3 and Llama-architecture models, applies LoRA adapters at runtime, and its output matches Hugging Face transformers token for token. It is the engine behind the trucs.ai chat and voice demos and llm-life’s language-model methods. The public demo runs SmolLM2-360M-Instruct.

How it is built

lean is three Rust crates, each doing one job, compiled from one source to a native library and to WebAssembly, with no ML framework.

agent-web, chat pages
  lean-chat   messages to prompt
  lean        text in, text out
  lean-core   the forward pass

Agents such as agent-web sit on top of lean and add the tool-calling loop.

In a browser, lean runs on WebGPU, then on CPU threads, then on a single CPU thread. The choice is made by capability, never by measuring the device. The kernels are the same on every device, with no autotuning and no per-device code. Its output is token-exact against Hugging Face transformers: the greedy tokens match the reference on every backend.

Runs
browser, WASM, WebGPU, native
Model
Qwen/Qwen2.5-0.5B-Instruct, 0.5B params, ~430MB (Q4_0 GGUF), model licence Apache-2.0; code licence MIT
Demo (GitHub Pages)
https://idle-intelligence.github.io/llm-web/web/
Source
idle-intelligence/llm-web
Status
maintained

Used on: llm, llm + tts, stt + llm + tts

← Home