Why does my boyfriend hang up on me when his ex is around?

I met my boyfriend in 2006 when we were both still unhappily married and living with our partners. Neither of us are divorced as yet but that will happen ..I’ve been living apart from my husband for over a year now and hes just moved out of his marital home recently into a place of his own.. thats when things were meant to improve - I was having to try and be very patient and understanding towards his wifes feelings for 3 years whilst he lived with her and its been the hardest thing to go thru..but thats how much I want a future together. Today was just a classic example of the rows we have. He goes over to his wifes house daily or nightly to see his children for a few hrs, and tonight I called him on his mobile phone to speak, I know shes never around because she goes out walking when hes there. However she was due in and he gets uncomfortable speaking to me when shes about, hes fine talking to me when his kids are there, anyway she walked in and he cut me off the phone immediately.. hes done this lots of times over the 4 years I’ve known him no matter whats going on, its like her feelings come first every time, and I’ve to put up with being cut off. I’ve told him how it makes me feel, and he’ll use the excuse of the reason he did it at that time eg: tonite was because he wanted to kiss his kids goodbye and his wife walked in and I was on the phone…it overwhelms him so he cuts me off. From past experience I know he’d cut me off whether the kids are there or not..she only has to appear… am I being over sensitive? should I understand that if she walks in I must be treated like that ? I’ve never hung up on him when my husband comes and visits his kids.