Should I stay or should I go?

Ok, I met my boyfriend in high school and I've been with him for 3 years. For 2 1/2 years we were basically inseparable, and then he went to work at a different place than I did and had 8+ hours a day away from me. Next thing I know he's hiding porn mags everywhere and not treating me well, I found emails from a girl from work talking about "their love." A manager gets fired from her job for editing the time sheet to get him loger lunches with her. He had his email to where I had the screen up and I didn't get to read the subjects hardly before he emptied the trash. There were like 20 0f them! She moved to a different state and other people at work are saying she is pregnant with his baby now that I work there too everyone acts like there is some big secret, except one big-mouth 19 year old. He swears he never cared about her. And believe me, she is not at all good looking, but I can't seem to trust him. How do I know he won't become who he was before? Now I constantly wonder what he's doing. Even him looking at another woman has me terrified and enraged. The worst part is all I can get from him is that nothing happened and I feel confused and out of control. He gets more irritated with me everytime I try to talk about it and says, "nothing happened. If you can't deal with it, leave." I get so angry over the stress that I want to leave, but the idea that I might be leaving over some obsessed girl that gave him a ride to lunch is enough to keep my foot in the door. He's manic depressive too, I think, so he gets moody and enraged when I try to talk about it. Should I be strong and work through it. or is this a common thing? This is my first serious relationship and we have 5 houses together, 2 dogs. No kids yet.