Also are you sure there isnt any firewall that might block your connection?

If your single player works good, then there's something blocking the packets, not the game itself.
And I dont think you would have such slow net, that you would get only 10-20 fps on Toribash.