Avoiding the affair

I met a man in february. The moment he walked in the door he had my attention, and when he raised his head and looked into my eyes, we were both taken aback. After six months of being friends, sharing the same friends, music, likes and dislikes, philosophical debates and conversations, we both told each other that. We have since revealing this truth shared many words and discussed and weighed the circumstances and consequences. He has been married ten years to a woman who is opposite to him. They were truly happy and in love, but their differences have led them down different life styles and different directions. Neither is in love anymore, and it is apparent to most who know them. She is not involved in his life outside the home, and is not part of his friends and social life, and thinks little of him, or their sex life. We have not been able to pursue our passion and desire, especially as we are usually together with friends. Our friends are already suspicious, they have been since the beginning, and it is now frighteningly obvious to me and my consults that we act not like friends. He seeks me out when I am not talking to him, he praises me when I’m not in the room, I glow, and smile and light up the room when he is near. We both gaze into each other with admiration, affection and attentively lean on every word. We have since day one, which averted suspicion initially. Friends assumed we knew each other for years. He told me his wife had been suspicious as of the day we met. We both fell into our eyes and felt a most intense connection the moment we met, and he has shown himself awkward talking about me to her, which has fueled further suspicion in her as well as the fact that she knows how alike we are. He pulls near, and I back away, knowing the pain that this difficult desire and longing will bring. I fear the pain it could cause him and his life more than the pain it causes me to keep myself at a distance. I realize now though, it has been months of people whispering about us, his wife being suspicious, and our both secretly longing one another. The inevitability is screaming here!
The last time we saw each other was at a party I hosted. In the end it was left down to one of our mutual friends, and the two of us. He stayed as long as he could, hoping to be alone with me. We had one brief moment alone all night. He grabbed me then, and pulled me close, held me tightly pressed against me, and sighed. Then quickly pushed me away right before someone entered the room. He does not have children, he is not in love, though loves his wife, and I am already more involved in his life than she, though we are yet to be lovers. The words, the feelings, and the knowledge we share in them, is enough to make us lovers already. Had he not been married, I don’t think either of us would have through twice before going head first into this. Over the months the level of emotion has grown, and a romance, and a love, are becoming part of our reality. I don’t know what to do. We are both too far in to back out, and it is just a matter of time before one of us slips up, or someone gets suspicious and starts meddling. I’m already riddled with guilt, fear, and both of us are actively trying to be secret and aloof, while we think, while he sorts things out. What advice have you for me?