Pretty much what Neo said: If you turn it off, trades are a bit quicker. However, if you get scammed in any fashion, Valve will not assist you at all.
If you keep it on, Valve will support you in the event you get scammed - and it acts as an extra firewall into your inventory - at the cost of having to take a minute to verify the email.
Additionally, you are only required to use email verification when items are removed from your backpack, and not when items get put into it.