Food will probably stick into their and it could grow bacteria which could spread to your gums and give you a major bacterial infection. If the bacteria does grow I suggest using an anti-bacterial to get rid of the bacteria. If you want to keep your retainer in good shape when you not using it, keep it in the case or set it in an anti-bacterial mouthwash to prevent a bacterial infection from even starting, but don't leave it in for more than 2 hours.
It's pretty disgusting and unhygienic. And i bet you wouldnt want to catch more infections.
bacteria. all i gotta say