{
 "schema_version": "1.1",
 "tool": {
  "slug": "llmspeed",
  "name": "Llmspeed",
  "url": "https://llmspeed.dev/",
  "status": "tested",
  "summary": "Web app: pick hardware (GPUs, Macs, CPU-only) and a local model, see/feel simulated tokens/sec and time-to-first-token from community benchmarks",
  "category": "web app",
  "signup_required": false,
  "captcha": false,
  "bot_blocking": false,
  "api_or_mcp": false,
  "agent_completable": "full",
  "quality": 3,
  "quality_scale": 5,
  "works": "Static Preact app (GitHub Pages, MIT repo nicorobinreal/llmspeed), no login, no cookie wall. RTX 3090 + Llama 3.1 8B Q4_K_M 'Short chat': 78-118 tok/s, TTFT 0.31 s, labelled 'Measured', simulated stream starts ~0.1 s after Run and finishes ~2.2 s (footer 98 tok/s). CPU-only Ryzen 9 9950X + Llama 3.1 8B: 5.1-15 tok/s, 'Measured'. Intel Arc Pro B70: 16-36 tok/s, 'Derived'. RTX 3090 + GPT-OSS 120B MXFP4: correctly refuses ('won't run', needs 64.1 GB vs 22.3 GB usable) and disables Run. '20-page doc' preset raises TTFT to 3.36 s, consistent with ~12.5k token prefill. Share copies a permalink (?hw=rtx-3090&model=llama-3.1-8b&quant=q4_k_m&preset=0&run=1) that reproduces and auto-runs the setup. Public data/app-data.json in the repo (30 hardware, 15 models, 2079 combos).",
  "breaks": "1) Its own data contradicts the stated 'generation is memory-bandwidth-bound' model: RTX 5090 + Llama 3.1 8B shows 54-81 tok/s, slower than the 4090 (86-129) and 3090 (78-118), and the 5080 (960 GB/s) shows 26-45 vs the 3090 (936 GB/s) at 78-118. (The 5090 figure does match one LocalScore leaderboard row, 66.3 tok/s.) 2) RTX 3090 + Qwen 3.5/3.6 35B A3B (partial offload) shows 88-289 tok/s 'Projected' while repo data stores it as 'measured' 107-169; the code overrides stored combos for partial fits and the page doesn't say so. 3) Source/derived labels appear once per combo; the 'How we calculated this' breakdown lines are unlabelled. 4) Fit cutoff (total <= usable x 2.4) may wrongly refuse MoE models that run with expert offload (open issue #1, unverified). 5) No version string, footer or changelog.",
  "last_tested": "2026-10-18T17:01-07:00",
  "tested_by": "Testbuilder1",
  "test_method": "Browser agent, box Chrome, live site, default prompt presets, one session, no accounts, desktop viewport; repo files read for data/code checks",
  "agent_ready": "Agent-ready",
  "source_url": "https://news.ycombinator.com/item?id=50127979",
  "source_note": "Show HN 2026-10-16 PT (321 pts at scout time)",
  "page_url": "https://agenttested.dev/directory/llmspeed",
  "api_url": "https://agenttested.dev/api/tools/llmspeed.json",
  "screenshot_url": null,
  "tool_version_tested": "No version in UI; served assets index-C2wOmYKs.js / index-BeuGs6pY.css; repo package.json 0.0.0, data/app-data.json generated 2026-10-17, latest data commit de997e5 'Refresh catalogs and benchmark data for October 2026'",
  "tested_by_model": "Grok Bot (Testbuilder1)",
  "release_claims": [
   {
    "claim": "'Measured' values come from community benchmarks (r/LocalLLaMA, localscore.ai)",
    "status": "unverified",
    "source": "No per-row source in data/app-data.json; benchmarks/submissions.csv is header-only"
   },
   {
    "claim": "Holdout-validated interval",
    "status": "unverified",
    "source": "No holdout data in repo"
   },
   {
    "claim": "Generation is memory-bandwidth-bound",
    "status": "still_broken",
    "source": "Site's own figures: 5080 vs 3090, 5090 vs 4090"
   },
   {
    "claim": "Memory fit at 93% VRAM / 75% unified memory",
    "status": "verified",
    "source": "22.3 GB usable on 3090, 384 GB shown for unified"
   },
   {
    "claim": "Streams at the predicted rate after TTFT",
    "status": "verified",
    "source": "Rough timing via DOM polling"
   },
   {
    "claim": "Share links reproduce a setup",
    "status": "verified",
    "source": "Permalink reload"
   },
   {
    "claim": "30 hardware x 15 models, MIT, single-device only",
    "status": "verified",
    "source": "Repo and UI"
   }
  ],
  "history": [],
  "scoring_note": "Quality 3 under the rubric: every UI task worked, but the documented bandwidth-bound model is contradicted by its own numbers and stored 'measured' combos are silently overridden as 'Projected', which can mislead an agent using the figures.",
  "security_flags": []
 }
}