Make sure that you are playing on most recent Toribash version (5.14). If that's the case then wait until the bug gets patched, it's all you can do really. Restarting the game before joining a room would definitely work as a temporary fix.