Connection drops after 12-24 hours online

[ApexDevelopment]

I'm going to use this issue to track a problem I have been having where after the server is running for a long time, players lose connection.

What I know so far:

• It is not due to network changes on the server machine. It is on ethernet and its internal IP is unchanged.
• There is zero indication in the server console that anything has changed. It doesn't even show any disconnects, the player count stays the same.
• The server does not hang. It continues to accept messages and commands from the console.
• All joined peers receive a "lost connection to host" message.

What I don't know yet:

• Whether all peers receive this message and lose connection at the same time.
• Whether "lost connection to host" is the exact wording of the error they receive (this has only happened to my friends, not me).

If anyone else has experienced this please share any information you have so I can try to track this one down.

[ApexDevelopment]

Ok, as expected, it is some sort of timeout. This is the code that causes it on the client side:

func _on_host_check_timer_timeout():
	if backdrop: return 
	if not Network.PLAYING_OFFLINE and not Network.GAME_MASTER:
		var found_host = false
		for member in Network.LOBBY_MEMBERS:
			if member.steam_id == Network.KNOWN_GAME_MASTER:
				found_host = true
		
		if not found_host:
			PopupMessage._show_popup("Lost connection to Host.")
			Globals._exit_game()

After reading a similar issue on Cove, it seems this could be caused by the server losing connection to Steam. Maybe there's some kind of heartbeat or anti-idle that needs to be implemented to fix this?

[ncrypted-dev]

Hey, that sounds interesting. I can remember that I had the server running for at least 36-48 hours during a few tests without any issues.

I will check if other games that rely on Steams P2P-Networking have had similar issues.

Were there any players on the server when it "stopped" accepting requests?

[ApexDevelopment]

@ncrypted-dev Yes, that's actually the only examples I have of this. I have no data on what happens when someone tries to join when the server is in this state.

I hooked the SteamClient callbacks for disconnecting and reconnecting from the Steam servers and spun up my server again. We'll see if those log messages correspond to when my friends get kicked off.

[morgana-x]

I've heard according to a server host that this happens when steam goes offline for maintenance weekly,

"less than an hour every Tuesday between 10 p.m. and 1 a.m. UTC." in America

which could've been the root cause of this issue and why ncryped-dev was able to run it for 36-48 hours without issue rather than 12-24 hours

[ApexDevelopment]

@morgana-x Interesting, I'll keep an eye out to see if this correlates. Thanks for the heads up.