How can I use downloaded or self maded mods for an online server?

Some mods cannot be used on a server. Those ones are not multiplayer compatible mods.
Althouygh u can change to the ones that are compatible by going to main menu:

. Click Setup
.Click Mods
.Choose Mod

Correct, but the friend(s) you're going to play with need that mod too, that's why you got to send them the file. They have to save the file in this folder:

C:\Games\Toribash-x.xx\data\mod

That's my folder, don't know if it's like yours, but the most important is that you save in Toribash's mod data. (x.xx being the game version you have)