Skip to main content

4×

more firmware bugs fixed on the first flash

3.8×

fewer tokens on average, up to 13× on individual tasks

Same model

both agents ran Claude Haiku 4.5, identically configured
No public benchmark existed for AI agents on embedded IoT firmware, so we built one and published it as open source.

​
Setup

IoT-FirmwareDebugBench v0.1 runs on real nRF52840 DK and nRF52832 DK boards with NCS v3.2.1 (Zephyr 4.2.99). Six BLE-focused tasks across three difficulty levels, each with a precisely injected bug, a defined reproduction procedure, and a known correct fix. Both agents run the same model, Claude Haiku 4.5, reasoning disabled, caching identical. This isolates one variable: the architecture around the model, not the model itself. Difficulty. L1: root cause readable in logs. L2: inference from BLE behavior or Kconfig. L3: correlating state across two devices or full session timelines.

​
Results

Bug-closed rate by threshold, Adsum vs Claude Code

Bug-closed rate by flash threshold (BC@k = bug closed within k flashes).

​
What the gap shows

  • The loaded workflow makes log capture a first-class step, not one the agent can skip under pressure.
  • On one task the baseline burned 27M tokens and lost the original symptom; Adsum resolved it at a 148.7k-token peak.
  • The gap widens with difficulty: parity at L2, Adsum 1/2 vs 0/2 at L3.

​
Watch it head to head

Both agents, same model, same real embedded bug:

Full benchmark report

Per-task breakdown, limitations, and how to run it yourself.