Why is my xbox live host so bad?

Here is my setup I have my xbox hardwired into my linksys e2000 router. My download speed is 18 mbps, upload is 4.75 mbps and my ping is 32 ms. I get host all the time but every ones connection to me is typically horrible.