THE NOT-PASS ENCYCLOPEDIA

The NOT PASS Encyclopedia

Behind every success are many times as many failures. Here's every way AutoDev LG has stumbled while building an app, counted by kind. It updates itself every day from the run records.

114
app-building runs
18
PASS
96
NOT PASS
84%
chance of failure

Failure ranking

Public tests: tests the worker writes to check its own code. Acceptance tests: tests written by the test writer that give the final verdict on whether the app matches the spec.

#1

Couldn't write the tests

24
test_invalid · 25% · taskboard 18 · INVADERS 6 · first Sep 17 · latest Sep 29

At the test-writing stage, the pass/fail tests (Acceptance tests) couldn't be written correctly, so it ended before any coding.

"Can't build the app before the answer key exists."

Episode: In inv18 on 9/28, Qwen3.8-27B, then Gemma-4-26B-A4B-it, then gpt-oss-120b each took a turn — and all three dropped out here.

#2

Failed the final tests

23
acceptance_diagnosed · 24% · taskboard 23 · INVADERS 0 · first Sep 17 · latest Sep 29

The app got built, but failed the Acceptance tests that judge whether it matches the spec. It got as far as diagnosing why, then stopped.

"The type that trips right before the finish line."

#3

Failed its own tests

16
public_diagnosed · 17% · taskboard 11 · INVADERS 5 · first Sep 17 · latest Sep 29

The Public tests the worker wrote to check its own code didn't pass. It diagnosed why, then stopped.

"Sets its own exam, fails its own exam."

Episode: In inv01 on 9/26, a screen test the worker wrote crashed inside the test itself.

#4

Stopped working midway

11
runtime_failed · 11% · taskboard 7 · INVADERS 4 · first Sep 18 · latest Sep 29

The run itself stopped — a timeout, or the workload hit its cap.

"Ran out of steam."

Episode: In inv16 on 9/27, a test run timed out after 180 seconds. inv22 on 9/29 ran for about 2 h 8 min before hitting OpenHands' workload cap.

#5

Stopped: fix needed a wider scope

8
scope_reviewed · 8% · taskboard 4 · INVADERS 4 · first Sep 20 · latest Sep 29

It tried to repair the app, but decided the fix couldn't be done within the files it was allowed to touch.

"It knows the problem is somewhere else."

#6

Gave up on the repair

6
public_repair_stopped · 6% · taskboard 4 · INVADERS 2 · first Sep 14 · latest Sep 29

It tried to repair the failing tests, but made no progress and gave up.

"The more it fixed, the deeper it sank."

#7

Test rewrite rejected

3
acceptance_test_regeneration_rejected · 3% · taskboard 2 · INVADERS 1 · first Sep 25 · latest Sep 27

It tried to rewrite the Acceptance tests, but the check rejected the new ones.

"No do-overs on the do-over."

#8

Dropped out at the requirements

2
requirements_traceability_invalid · 2% · taskboard 2 · INVADERS 0 · first Sep 16 · latest Sep 17

In the very first step, the requirements couldn't be matched to the spec's items, so design never started.

"Tripped on the starting line."

#9

Failed the final spec check

2
spec_compliance_fail · 2% · taskboard 2 · INVADERS 0 · first Sep 23 · latest Sep 25

The tests passed, but the final check of the finished app against the spec found a mismatch.

"Perfect score on the tests — forgot to write its name on the paper."

#10

Test-harness repair rejected

1
acceptance_harness_recovery_rejected · 1% · taskboard 1 · INVADERS 0 · first Sep 21 · latest Sep 21

A proposed repair to the harness that runs the Acceptance tests was rejected by the check.

"The repair quote didn't get approved."

Counted from the same run records as the dev log (stopped runs excluded).