china, I know exactly how you feel! It makes me physically ill. I also want to leave my husband every time I "catch" him. I explain it to him this way: when we were wed, we became one flesh. This sin contamintates me as well as him.
Hopefully, this won't happen again - with my husband or yours...