Should I be worried about his ex?

I have known my boyfriend for 30 years , but we have only been together for 4 months. We are deeply in love and he has discussed marriage and our future BUT he works with his ex. She is twenty years older than me and they work together and he left her three months before starting our relationship after being with her for 7 years , 2 of which he said he was unhappy. When we first started seeing eachother he was picking her up and taking her to work and home again and occassionally stay for a cup of tea. I tolerated it because I trust him.She started telling him that he’s no good at relationships etc and he would come home and get quite down on himself for a few days. This irritated me to the point I told him he should limit his contact with her as she makes him feel bad about himself. He told me he was stopping the transporting her around on a certain date. that date came and went and I knew he was still taking her etc because his times for texting and calling were still the same…he never volunteered the information though. Last night I asked him and he said , “ she said for the sake of our friendship we should still travel in and out together because otherwise we would just never see eachother”. as you can imagine I was very upset…not just because he was still doing this but also because he didnt tell me. To me , that sort of comment from her is not appropriate. I am also annoyed because he says he keeps his private life close to his chest…ie he has not made it clear to her , whilst she knows I exist , that we are serious and have made a commitment to eachother. Therefore this tells me that he doesnt want to hurt her feelings which also tells me he still cares. I told him he cannot have two relationships and he was incredulous…he said he loves me not her and he is 100% with me…but this cant be true. in my opinion every time a man goes out of his way to spend time with a woman he is having a relationship with her. I see him three days a week and she sees him every day. He is clearly having a close friendship with her and I am not happy about it. Its like the only thing that has changed between them is geography. Every time I bring it up (he is ten years older than me and she is twenty years older) I get told my opinion is that of immaturity. I dont agree and would love any opinions as to how I should deal with this. If I moan…then he has happy times with her and misery with me and she gets what she obviously wants…if I do nothing…he gets his dual life and I am unhappy.