Dekiro Dev Blog

Part 20 of 28

Multiplayer Bugs Only a Friend Can Find

A server-authoritative game, a bot that plays it, thousands of tests, and still: a friend joins and sinks into the hills, can't pick anything up, and everyone freezes when someone takes a portal. Six bugs that needed a second person, or at least a second point of view, and what each one taught.

On the Windows build it seems that items as well as other players are all below the actual surface, so on the Y axis 0. What could that be?

— the developer

Why the bots didn't find them #

The multiplayer post described the setup: single player is a local server, every actor has a network role, and tests/net_test.sh starts a host, a client and a dedicated server with bots that walk, fight, trade, die and come back. It runs after every network change and reports engine errors. It's good at what it was built for: race conditions, missing spawns, RPCs to peers that can't see an entity yet.

What it can't do is notice that something looks wrong, and it only goes where it's told. Most of the bugs below lived in exactly those two gaps. Three came from the developer's evening with a friend on the Windows build, one from playing alone, and two were caught by thinking about what a friend's machine would do differently. This post follows on from the multiplayer post,1 "Server-Authoritative Multiplayer in Godot, Single Player Included" (03) has the roles, the spawn function and the network bot. and the order is roughly from "embarrassing" to "subtle".

A friend at y = 0 #

The report came in one sentence (the epigraph above), then a second:

Can't seem to pick up any items and so on.

The first suspicion was the items. A dropped item takes its height from the ground like everything else, so the first checks measured that: in the editor, on a hill, items landed exactly on the ground. In the exported game, in single player, too. So it wasn't export-specific and it wasn't items. It was something only a joining player sees, and reproducing it with two game instances was slow, so the investigation went to the code that receives a client's position instead. On the host:2 Line 1909 of scripts/actors/actor.gd, in _rpc_owner_state, unchanged since the multiplayer commit be14886 of day two. Hills came with 21a6b5b the next afternoon.

net_pos = Vector3(pos.x, 0.0, pos.z)

When a client sends its position, the host keeps x and z and sets the height to zero. That line was written on day two, in the commit that added multiplayer, when every map was flat and zero was the ground. Rolling hills arrived the next day. Characters don't collide with the terrain in this game: an actor snaps its y to map.height_at() after moving. But a joined player's copy on the host (a PROXY, in the post's terms) doesn't move itself; it takes the client's samples. So for eleven days, every joined player stood at height zero on the host, which is inside any hill.

That one line caused all three symptoms:

  • Players sank into the ground. The host sends its copy's position to everyone else, so every other player saw the friend at height zero, which on a hill means buried.
  • No pickups. Loot is collected on the server when your character's body overlaps the pickup's area. The host had the friend's body under the hill, so on any slope it never touched anything.
  • "Items under the surface" was the same effect seen from the other side: the friend stood at the right height on their own screen, the items were where they should be, and the host disagreed.
On the host, a joined player's copy stood at y = 0, inside the hill, under the loot's pickup area. The fix: the host puts them on its own ground

Why didn't the bots find it? Because towns are flat. The terrain generator flattens the ground at safe zones, coasts, portals and buildings, and the network test's bots spend their time in town. On flat ground zero is correct, so everything worked. The bug needed someone to walk out into the hills and try to pick something up.

The fix3 Commit 4feeb21: 11 lines in actor.gd, 24 in the tests. takes the height from the host's own ground, not from the client:

func apply_owner_state(pos: Vector3, dir: Vector3, sent_msec: int) -> void:
	if Vector2(pos.x - net_pos.x, pos.z - net_pos.z).length() > MAX_OWNER_STEP:  # ground distance: hills don't count
		_rpc_teleport.rpc_id(owner_peer, net_pos)  # too far: put them back
		return
	net_pos = Vector3(pos.x, map.height_at(Vector3(pos.x, 0.0, pos.z)) if is_instance_valid(map) else pos.y, pos.z)

Trusting the client's height would have been the quick fix, and the wrong one in a server-authoritative game: a client could then float. The second change fixes a related bug: the "you moved too far, back you go" check measured in 3D, so on a steep hill the difference in height alone could look like a cheat. It now measures distance along the ground only. The receiving code moved out of the RPC into its own function so a test can feed it.

The test, test_joined_players_stand_on_the_ground_on_the_server, finds a hill on the test map, sends a sample with height zero, then one with a fake height 30 m up, and checks the player stands on the ground both times.

Four characters on a forest slope (a staged shot from the Steam screenshot set): the kind of ground where joined players used to disappear

"ConnectToGlobalUser failed" #

The same evening started with a different report: "It seems there's a Steam bug." Then the error text: "ConnectToGlobalUser failed." Steam's own message for "this program can't talk to your Steam account" is not self-explanatory, and the logs on the machine showed nothing. The coordinator guessed Family View, Steam's PIN-locked parental mode, which blocks exactly this connection. A few minutes later the developer confirmed it:

Ah, never mind, this is me: the family guard was active in Steam.

Not a game bug, then. But other players will run into the same wall, and they won't have an AI to guess for them. So the start menu now translates the error.4 SteamService.explain() in scripts/autoload/steam_service.gd, commit a2db5ce; the start menu's Steam row calls SteamService.start() again every 5 seconds until it works. For ConnectToGlobalUser it says: make sure Steam is running and signed in, not in offline mode, with Family View unlocked; that the game and Steam run as the same user, neither "as administrator"; and on Linux, that a Flatpak or Snap Steam can't see games started outside it. It ends with ". Playing without Steam (join by address still works)", and keeps Steam's own text in brackets for anyone searching for it.

The second half of the fix matters more: the start menu now tries to connect to Steam again every 5 seconds. Before, Steam was initialized once at launch, so unlocking Family View or starting Steam after the game meant restarting the game. Now the Steam row just lights up. The test is small: test_steam_init_errors_are_explained checks the text names what to check and keeps Steam's message.

The host that froze #

";It lags a bit for other players if someone uses a portal." This is the playtest note that sounds smallest and was the most work.

The network agent measured before touching anything. A new test, tests/net_portal_test.sh, starts a host and two clients: a "mover" who travels through five maps nobody is on yet and two dungeon instances, and a "watcher" who stands in the Vale. The host logs its worst frame around each trip; the watcher logs the biggest gap between updates of another player. The first run on the development PC:

Trip Host's worst frame
Shadow Marsh (first load) 6,260 ms
Frostpeak 1,102 ms
Sunscar Desert 557 ms
Sun Tomb (dungeon) 79 ms
Crypt of Whispers (dungeon) 77 ms
back to the Vale (already loaded) 13 ms

Six seconds. "Lags a bit" was generous. The cause was the one everybody suspected: when someone travels to a map that isn't loaded, the host builds the whole map, terrain, tens of thousands of plants, clutter batches, buildings, NPCs and monsters, inside a single frame. In single player that's a loading screen and nobody cares. On a host, everyone else's game stops with it: their characters freeze, their monsters freeze, and then the world catches up in a jump.

The fix is to build maps in steps. GameMap now queues its build as a list of resumable steps, and the World drives them a slice per frame: 6 ms on a host with a screen, 12 ms on a headless server. Loops that place plants or clutter remember where they stopped and continue next frame; the terrain bake and the ground mesh run on a worker thread. The travelling player sees a short "Travelling..." screen and stays where they are until the map is ready, then arrives. Maps that are already loaded are instant. Single player still builds at once, because there's nobody else to stall.

Two tests guard it: test_map_built_in_steps_matches_built_at_once builds the test map and a dungeon both ways and compares a signature of every placed thing, line by line, because a step that resumes in the wrong place would quietly shift every plant after it; and test_travel_waits_for_a_map_built_in_steps checks the traveller waits and then arrives.

The host's worst frame when one player first travels to each place, before and after building maps in slices (headless host, an intermediate run while the work was going on)

The last stragglers were found the same way. Frostpeak kept one 550 ms frame until the scatter step learned to look up marsh pools and lakes in a grid instead of testing every plant against every pool, and dungeons cost 6 to 10 seconds on a host with a screen because merging the kit pieces read each tile's mesh back from the GPU; caching the pieces as plain arrays took the Crypt from 9.6 s to 0.3 s. Monster packs near the arriving player now appear one per slice, nearest first, instead of all in one frame.

The final run, same trips:

Before After
Headless host, worst frame 5.8 s (Marsh; 22 s when its terrain cache was missing), desert 7.5 s, Frostpeak 6.2 s 30 ms; nothing over 33 ms
Host with a screen, worst frame up to 7.2 s 72 ms
The watcher's longest gap between updates 28 s headless, 10.9 s with a screen 55 ms / 90 ms
The traveller's wait none (everyone else waited instead) 0.1-0.9 s for a dungeon, 2.4-6.4 s for a big map

The traveller now waits a few seconds on a "Travelling..." screen for a fresh map, which is the price of nobody else noticing.

A clock that disagreed #

This one was found without a friend, by imagining one. Townsfolk, animals, the desert caravan and the time of day don't go over the network. Every machine computes them from the wall clock: a 36-minute day, a routine at a time of the clock, a caravan stop at a time of the clock. So every player sees the caravan at the same well without a packet.

That works if every PC agrees on the time. The caravan agent wrote that its merchant trades "only while halted (plus 5 s slack for machines whose clocks differ a little)". A little is the normal case. But a friend's PC with a clock five minutes off would see the caravan at a different stop than the host, walk up to it, and be refused by the server, which thinks the caravan left long ago. The same friend would see a different time of day, and townsfolk in different places.

The fix is small. On join, the server's welcome message carries its wall clock, and the client keeps the difference:

WorldClock.server_skew = float(rules["clock"]) - Time.get_unix_time_from_system()

WorldClock.now() is the local time plus that skew, and everything timed by the wall clock reads it. The server resets its own skew to zero, so a host that was a client earlier in the session doesn't inherit someone else's offset. test_clients_take_the_wall_clock_from_the_server feeds a welcome two minutes ahead and checks that the clock, the townsfolk and the caravan follow.

Dashing out of the world #

The developer found this one alone, but it's a multiplayer bug in spirit:

With some of the skills like dash, one can move over boundaries such as the ocean's invisible wall or into mountains. Any way to fix this? I guess this would also work and break dungeons.

Dash and Charge simply teleported the character 6 to 8 m to the end point. Nothing in between was checked: not the coast's wall, not a mountain, not a dungeon wall or a locked door. In a single-player game that's an exploit. In a server-authoritative game it's worse, because the server itself was doing the teleport, so no amount of client checking would stop it, and a party member could dash past a puzzle door everyone else had to solve.

Actor.dash_end now walks the path in 0.4 m steps with a 0.35 m sphere and stops just before the first static body, the edge of the map's land, or the start of a mountain. Monsters don't block it, so Charge still goes through the enemies it hits. Knockback, pulls and the attack lunge already moved through normal physics and needed nothing. test_dash_stops_at_walls_and_the_edge_of_the_map covers four cases: an open dash goes the full distance, a monster in the way doesn't stop it, a wall stops it just in front, and a dash across the edge ends on land. The test itself had a bug of its own.5 It passed alone and failed after other tests, because maps that earlier tests left behind sat at the same spot. It now runs 41 km away from everything (maps sit 5 km apart).

The build that would have shown capsules #

The last one never reached a friend, which is the point. Asked for new Windows and Linux builds, the coordinator checked the export settings first and found the sprite atlases missing. Godot exports resources it knows about; plain .json files have to be listed in an export preset's include filter, and the filter had data/*.json and data/maps/*.json but not assets/sprites/*.json. Every sprite sheet's atlas, the file that says where each frame of each direction lives, would have been left out, and the game falls back to the capsule when a sheet has no atlas. The exported game would have run without a single error and shown every player and NPC as a coloured capsule.

What an export without the atlases would have looked like: the capsule fallback next to the sprite (from the HD-2D post)

Commit 21dd86c adds the path to every preset. It's also why the project's rulebook has a line about it:6 CLAUDE.md, the file every Claude session reads first; the first post explains it. "add *.json to the export preset's non-resource include filter, or data/*.json won't ship." Headless tests run from the project folder, where every file exists, so no test could see this. The check now is to look inside the built package before handing it over.

What they have in common #

Six bugs, and none of them was a race condition or a desync, the things network code is usually afraid of. They were all assumptions that held on the developer's machine:

  • The ground is at zero (true on day two, and in every town).
  • The person reading the error knows Steam's internals.
  • Building a map takes a moment (true when only you are waiting).
  • Every PC's clock agrees.
  • A short teleport can't leave the map.
  • The files in the project folder are the files in the build.

Takeaways #

  • Bots test what you point them at. The network bots stayed in flat towns, so the y = 0 bug was invisible to them for eleven days. Send at least one bot somewhere inconvenient.
  • Never take the client's height, and don't fix a bug by trusting the client. The server owns the ground; the fix put the player on it rather than believing them.
  • Measure the lag before fixing it. "Lags a bit" was a six-second freeze, and the numbers per map showed which maps mattered.
  • Presentation from a clock needs one agreed clock. One number sent on join keeps every machine's townsfolk, sun and caravan together.
  • Translate the platform's errors. "ConnectToGlobalUser failed" means nothing to a player; a list of four things to check does. And retry, so fixing the cause doesn't need a restart.
  • Look inside the package. A build that runs without errors can still be missing half its look.

What's next #

The friend's evening was also the first real playtest, and most of its notes weren't bugs but wishes. The playtest post goes through all of them, from potion numbers to a chat that was once "not needed".