Testing a Game with 13,800 Checks

Why so many tests #

Our developer is an AI that starts every session without memory of the last one, and on a busy day it commits 40 to 70 times. Later in the project, several AI agents edited the same repository at once. Without a safety net, every one of those changes could quietly break something built two days earlier. So the test suite isn't an afterthought. It went into the very first commit together with movement and the first arc attack, and every feature since has added to it.

At the time of writing, tests/run_tests.gd holds 341 test functions, which run about 13,800 checks in a little over a minute. Both numbers have a story, and the check count is the less trustworthy of the two. We'll come back to that.

The runner: a few dozen lines, no framework #

We don't use a test framework. The runner is a SceneTree script that Godot runs headless:

func _run_all() -> void:
	for method in get_method_list():
		var name: String = method["name"]
		if name.begins_with("test_"):
			print("- ", name)
			call(name)
			_cleanup()
	print("\n%d checks, %d failed" % [_checks, _failures])
	quit(1 if _failures > 0 else 0)

A test is any method named test_*. It spawns what it needs and calls _check(condition, message). _cleanup() frees every actor, damage number, loot drop and world it left behind, so each test starts clean. Two environment variables were added once the suite got big: TEST_ONLY=<substring> runs the matching tests, and TEST_TIMES=1 prints each test's time and the slowest forty. The first test ever written is still there and still typical:

func test_arc_hits_only_enemies_in_front_and_in_range() -> void:
	var player := _spawn(Actor.Team.PLAYER, Vector3.ZERO)
	var front := _spawn(Actor.Team.ENEMY, Vector3(1.5, 0, 0))
	var front_edge := _spawn(Actor.Team.ENEMY, Vector3(1, 0, 1))  # 45 deg, inside 120 arc
	var behind := _spawn(Actor.Team.ENEMY, Vector3(-1.5, 0, 0))
	var too_far := _spawn(Actor.Team.ENEMY, Vector3(5, 0, 0))
	var ally := _spawn(Actor.Team.PLAYER, Vector3(1, 0, 0))

That this test is easy to write is a direct result of an architecture rule: "Combat functions are pure queries: no side effects, so they can be unit tested."

Gotcha 1: engine errors don't fail tests #

A GDScript runtime error prints SCRIPT ERROR and carries on. The test that triggered it may still pass. So the command in CLAUDE.md always greps the log as well:

"$G" --headless --path . -s res://tests/run_tests.gd 2>&1 | tee /tmp/t.log | tail -1; grep -E "^(SCRIPT )?ERROR" /tmp/t.log

"All passed" with an error line in the log counts as a failure. Several real bugs, for example freed-object errors, first showed up only as an ERROR line.

Gotcha 2: headless runs never compile shaders #

This one cost a lot of embarrassment for one line of code. On day six, an agent added a uniform called shade to the sprite shader. A shared shader include for cloud shadows already had a local variable called shade. The sprite shader stopped compiling, and every sprite in the game (the player and every NPC) rendered as a flat white rectangle.

Every test passed. Every headless smoke run passed. A headless Godot never compiles a shader, so the error never happened there. The human found it in their own running game and asked, dryly: "you do realize that our player model is a white wall?" The fix was a rename (shade_tint). The lesson went into CLAUDE.md the same day:

Headless runs never compile shaders: after any shader edit (incl. shader strings in .gd files) take a windowed screenshot of main.tscn and grep its log for ^ERROR.

Agents working in parallel at the time noticed it too. One reported that its first screenshots of the ice dungeon showed white walls where the player should be, and re-shot them after the fix. Screenshots caught what tests couldn't, but only because someone looked.

Where 13,800 checks come from (and why the number lies) #

Most checks come from tests that loop over data:

func test_items_have_models_icons_and_upgrade_glow() -> void:
	for data: ItemData in ItemDb.items():
		_check(ItemLook.icon(data) != null, "%s has an icon" % data.id)
		var model := ItemLook.make_model(data)
		_check(model != null, "%s has a model" % data.id)

That's one function and hundreds of checks. Every item has an icon and a model. Every monster type has a valid model with its animations. Every quest points at real NPCs and monsters. Every NPC, townsperson and companion name is unique (the lore has a rule about that). Every armor slot has a tier at least every ten levels. Every ranged monster's telegraph is long enough to dodge. A busy town square (63 NPC sprites) stays inside its video memory budget. When the AI adds 41 armor pieces or ten dungeons in one commit, these tests check every one of them without anyone writing a new test.

Now the honest part. On the morning of day four the suite had 11,933 checks and took 168 seconds. Most of that time came from 38 tests that each built the real 2.6 x 3 km Sanctuary Vale, at about 3.9 seconds a build. An agent moved those tests onto a small dev-only test map. The suite dropped to 12 seconds, and the check count dropped to 4,992. Nothing was lost: two tests make one check per spawned monster or spawn slot, and the big map simply had far more monsters (3,093 checks from one test alone, against 123 on the test map). It took two more days of new content to climb back to 13,800.

So a check count measures how much data a test happens to walk over, not how much behaviour is covered. We still print it because a sudden drop is a useful alarm. We don't brag about it.11 The title of this post is a small exception.

Regression tests reproduce the bug first #

When a bug is found, the fix comes with a test that reproduces the original failure. One of the first: dead monsters are freed while a Shadow's spawn list still holds them, and a typed lambda then errors. The test does exactly that:

	spawned[0].free()
	spawned[1].die()
	_check(shadow.alive_count() == shadow.wave_size - 2, "freed and dead mobs no longer count (got %d)" % shadow.alive_count())
	shadow.spawn_wave()
	_check(shadow.alive_count() == shadow.wave_size * 2 - 2, "next wave still spawns")

The lesson also went into CLAUDE.md, so the AI doesn't write that kind of code again. The same pattern held for a GPU problem later on: when big maps ran out of GPU descriptors,22 The performance post has the story. the fix came with a test that a map's props share a handful of materials and that a far-away monster releases its model.

Tests never touch the player's files #

The game writes options, window layouts and tutorial progress to user://. A test run that overwrote the human's key bindings would be a nasty surprise, so the runner points every one of those at a throwaway file before anything else runs:

func _initialize() -> void:
	WindowDragger.layout_path = "user://test_ui_layout_%d.cfg" % randi()  # never touch the player's layout
	LootHint.progress_path = "user://test_tutorial_%d.cfg" % randi()
	GameSettings.path = "user://test_options_%d.cfg" % randi()  # never the player's options

Save tests set SaveSystem.root_dir to a temp directory, and the terrain disk cache can be switched off for tests. The same care applies to processes: the human often has the editor or the game open while the AI works, so test processes are always wrapped in timeout, and the AI never kills a Godot process it didn't start.

A map just for tests #

Real maps get reshaped all the time. Map 1 has been rebuilt from scratch, then shrunk, then dressed with nearly a thousand pieces of clutter. Tests that depended on "the pond at (45, -8) in the Vale" would break with every redesign. So there's a dev-only map, the Proving Grounds, generated by a Python script: a safe square with shops and every NPC role, training dummies, every kind of liquid, an arena, "The Wilds" (a hunting strip with packs, Shadows, a road and mountains), and a "zoo" with one passive monster of every type. Maps with a dev_ prefix load by id but stay out of the world, achievements and exploration. CLAUDE.md says: "Prefer it for new tests and scenarios: the real maps get reshaped."※margin noteF10 takes you there in a debug build.

A related rule came from an early surprise. Maps sit 5000 m apart, and their offsets come from sorted map ids, so adding a new map file shifted every map after it and broke tests that used raw coordinates. Tests now always place things with map.to_global(local). With 20 dungeons added as map files, that rule has been worth a lot.

Tests outside Godot #

Not everything runs in the engine. The dungeon generator33 It has its own post. is plain Python, and it has its own selftest.py: pressure plates and beams built from templates, latching gates, guide mazes, shifting walls, illusion walls, ice slides and hazard marks. More importantly, the generator refuses to write a dungeon its solver can't finish, so an unsolvable layout never reaches the game at all. The balance model is Python too and is checked the same way a spreadsheet is: by rerunning it after every data change and reading the charts.

A combat simulation instead of a playtest (almost) #

Balancing 20 dungeons and 90 levels by hand would take weeks of play. So tests/sim_combat.gd runs the real game code headless as fast as the CPU allows (about 70 times real time): real Actors, real monster AI and boss phases, real skills, real dungeon mechanics. Bot players drive their characters through the same command_* methods as the player controller. They swing the combo, cast their school's skills, drink potions when low, and step out of every telegraphed attack after a 0.3 second "human" reaction time. Whatever still hits them is what a careful player can't avoid.

It fights every hunting zone, every boss and every dungeon room, per school, solo, with companions and in a party of four, and prints tables: time to kill, damage taken and by which attack, potions, deaths and the biggest single hit. The first run found that nothing in the game could hurt you and nothing lived (packs died to the opening skill burst in under 1.5 seconds), and that one boss took 150 to 230 seconds because the bots kept hitting him while he was shielded.44 The balance post has the numbers. The bots can't walk mazes or solve puzzles, so dungeon run times still need a human with a stopwatch.

Smoke runs and network bots #

Unit tests check pieces. Two cheaper checks cover the whole:

• Smoke runs: start the start menu headless for 60 frames, and the game for 300, then grep for errors. This catches problems that only happen when everything is live together: autoload order, scene wiring, a missing resource.
• Network bots: tests/net_test.sh runs a host and a client that play on their own (fight, loot, party up, enter the dungeon), then a passworded dedicated server. The multiplayer post covers them. They now run on a random port with their own log folder, so parallel runs don't collide.

Screenshots: how an AI checks visuals #

Tests can't tell you that the grass is the wrong green or a window covers the hotbar. For that, the AI takes screenshots and looks at them itself. tests/screenshot.gd renders any scene for a few frames and saves a PNG. On top of it there are scenario scripts: scripted sessions that set something up and photograph it. There were 15 when this series was first drafted. There are now 44: every skill of a school, the inventory, the upgrade window, every NPC up close, the animals in town, the story's finale, the scenic spots of the Vale, and one per batch of dungeons that photographs the gate, every room, each mechanic in action, the boss and a top-down overview.

That last kind changed how visual review scales. Ten dungeons produce a few hundred screenshots, and the coordinator can't look at every one carefully. So each dungeon batch also produced contact sheets (all of a kit's pieces on one image, all of a dungeon's monsters on another), and the review starts there.

For visual changes the rule is before-and-after: a screenshot of the current state, the change, the same shot again, and a look at both. Many visual commits show iteration in small steps. The lens effect, for instance, got three follow-up commits within three minutes ("only at the outer corners", "half the corner blur", "a bit more color fringing").

Bugs found along the way #

From the commit messages and agent reports:

• Ranged monsters never set their home position and walked with NaN directions.
• Monsters across a map border stood frozen like a wall, because only the player's map was awake.
• A moved window kept its biggest size: the skill window got stuck at full height.
• Moving an actor between maps inside a physics callback segfaulted Godot. That one is now a rule in CLAUDE.md.
• Two agents each added a test helper named _room_mid to the same test file, and the whole suite stopped parsing until one of them renamed theirs.55 The coordinator post has more on parallel agents.
• New class_name scripts aren't known until Godot refreshes its class cache, so a suite that suddenly "can't find" a class usually needs --import, not a fix. That line is also in CLAUDE.md now.

What tests can't do #

The CLAUDE.md line is honest about the limits: tests and smoke runs don't cover visuals or game feel. "Ask the user to playtest anything about feel." Whether the combo has a good rhythm, whether a blow seems to land when its damage number appears, whether a full sprint bar should last 5 seconds or 25 (it went from one to the other two minutes after sprint landed): these are questions for a human with the game in their hands. The combat sim narrows the questions; it doesn't answer them.

Takeaways #

• No framework needed. A few dozen lines of runner and a _check function go a long way.
• Loop over your data. One test function per rule, not per item, gives thousands of checks for free and catches broken references between files. Just don't read too much into the total.
• Grep for engine errors, and compile your shaders. A green headless run with a broken shader isn't green.
• Isolate the player's files and processes. Tests that touch real settings or kill a running editor destroy trust quickly.
• Simulate what you can't play. Bots that use the same commands as the player turn "is this boss too hard?" into a table.
• Make the AI look at its work. Screenshots turn "I changed the shader" into "I changed the shader and here is what it looks like now".

What's next #

Next is the content pipeline: how every monster, weapon, building and tree in the game is built by Python scripts in headless Blender, and what went wrong with the first batch.