Device Pool plan
Answer me with artifact · 2026-10-28 15:22

New version 0.5.10

You have 0.5.5. Found when this page was made.

What's new:

Run in a terminal:

npx skills update answer-me-with-html -y

Or tell your agent:

Update answer-me-with-html
See what's new

Device Pool plan

Lead: Device Pool is a new Allternit product. It runs one large open model across the computers a user already owns, and serves it as the first Native backend in our router. We build it on Apache/MIT engines (mesh-llm, exo, llama.cpp), with no hardware box. It adds capacity, not speed, so the product must be honest about tokens/sec. Plan doc: Allternit Brain/Products/DevicePool.md.

A

How it works

activations, mesh tunnelactivations, mesh tunnelnext tokendraft tokens, embeddingsBackendKind::Nativesame model listMac (layers 1-30)PC + RTX GPU (layers31-60)Linux box (layers61-80)Head node /v1Phoneallternit-api routerChat, Bots, Gizzi

Each worker reports its free memory and link speed. A planner sizes the layer ranges and shows the expected t/s before the model loads.

B

What we already have

Piece Where State
Native backend slot cmd/allternit-api/src/llm_gateway/provider_routing.rs:752 Stub, never built
Node registry + scheduler cmd/allternit-api/src/fabric/ Per node only, no pooled model
Capability report cmd/gizzi-code/src/runtime/fabric/capability-catalog.ts Memory only, GPU/VRAM not confirmed
Mesh tunnel cmd/allternit-api/src/mesh_bridge.rs, runtime/server/mesh.ts Works, not load-tested for activations
Local model server cmd/gizzi-code/src/runtime/local-model-server.ts Single host (mlx_lm, GGUF)
Always-on daemon cmd/allternit-node/ Mac only
Self-update packages runtime.allternit.com Works on Mac, Windows, Linux
C

Engine options

Engine License Strength Weakness
mesh-llm Apache-2.0 Layer ranges over QUIC, E2E encryption, invite meshes, Mac/Linux/Windows Young, no client API auth
exo 1.0 Apache-2.0 Thunderbolt 5 RDMA: about 32 t/s Qwen3-235B on 4 Mac Studios macOS 26.2+ only, fragile setup
llama.cpp RPC MIT Runs on every backend No auth, CVE-2026-34159 RCE (fix b8492)
prima.cpp not verified Claims 5-17x lower latency on home devices Research code, authors' numbers
RAMDeck core Commons Clause Not usable Blocks commercial use
Which engine base do we spike first?
D

Speed facts

Setup Result Source
gpt-oss-120b, 5 devices 3.1 t/s RAMDeck vendor
gpt-oss-120b, 6 devices 1.37 t/s RAMDeck vendor
72B, split over 10GbE 11.1 to 5.9 t/s decode kjaiswal benchmark
7B prompt processing, 10GbE 76 to 318 t/s kjaiswal benchmark
Qwen3-235B, 4 Mac Studio, TB5 RDMA about 32 t/s Jeff Geerling, exo
iPhone 16 Pro, sustained load about half throughput after 2 iterations arXiv 2603.23640
Design rules from these numbers
  • Split a dense model only when it does not fit on one machine.
  • Prefer MoE models. Use spare devices for prompt processing and draft tokens.
  • Phones are helpers only, in the foreground.
  • Never expose raw RPC ports. Every hop goes through the mesh with pairing identity.
E

Plan outline

  1. Phase 0Spike, about 1 weekEoj's Mac and Test MacBook over Wi-Fi, gigabit and Thunderbolt. One dense 70B and one MoE model. Publish t/s and pick the engine.
  2. Phase 1MVPPool toggle in Desktop Settings > Computers. Worker ships as a runtime package. Head serves /v1 as the Native backend. Phone layout and PWA show pool status and chat through it. Docs in the same PR.
  3. Phase 2Smart rolesMoE expert sharding, speculative draft, prompt offload. Windows and Linux workers.
  4. Phase 3PhonesiOS app and PWA as foreground draft and embedding helpers.
  5. Phase 4Fast mode and teamsThunderbolt RDMA Mac clusters. Team pools in platform.allternit.com. Opt-in overflow to Allternit Cloud.
Start the Phase 0 spike now?
F

Business model (draft)

Open (Apache-2.0)
worker, head, planner. They run on user devices, per OpenSource.md
Closed
team pool management in the console, hosted overflow
Hardware
none. RAMDeck's box is the part the press calls redundant
Pitch
run big models on RAM you already own during the DRAM shortage, with measured speeds published
Pricing
not set. Goes through /quote and Stripe drafts after Eoj approves
How do we charge for it?
G

Open items

  • Product name. "Device Pool" is a working name. Check SurfaceNames.md before choosing.
  • The tweet link is queued in Research/.incoming/links.md for the next research sweep.
  • Not checked: RAMDeck funding raised, iOS background limits, prima.cpp license. Check these in Phase 0.