What should I do - tell him, or not??

OK, well…there’s this guy who I’m really good friends with, and it was perfect just the way it was, like that. But then, a week ago he asked me out. I said yes for two reasons – one, there is no harm (I thought) in giving him a chance, and two, I didn’t want to break his heart by saying no. But now I’m in an even worse situation – he really seems to like me, and is always holding my hand and stuff. And I feel sooo guilty because he thinks I like him too, but I actually don’t! I should have just said no when he asked me in the first place, but I didn’t. And that was really silly of me, because now I feel even worse. But I just don’t know how to break up with him. And the even stranger thing is, he is my second boyfriend, and whenever he holds my hand, my mind drifts back to my first boyfriend, who I actually really liked. I was talking to my first about my second, and he has been trying to help me out as well, but I just don’t know what to do…help, anyone?? Please? Thank you!