Do you think I'm a halfway decent straight woman? I actually defend my boyfriend's "wife" (transual)?

My friends think I'm a bad person for having an affair with a married man. But let me tell you something, this man's "wife"(an MTF transual) is overweight and he's always putting "her" down because of it. Instead of laughing along, I get FURIOUS with him and tell him to stop. His wife is a sweet "woman" and she doesn't deserve to be treated that way. She knows her husband is cheating on her but because of her lack of self-esteem, she doesn't complain. And I feel so bad for her. Just last week, my boyfriend invited to me THEIR house to have in THEIR bedroom while his wife was HOME! At first, I said okay but as I followed him upstairs, I heard this poor "woman" crying her eyes out in the living room. I started to cry too and I told him that there is no way I can do this. So, we left and went to my place instead. This woman cooks for him, cleans the house, fixes his lunch for work, irons his suits and pays half the bills in the household yet she's subjected to being called "fat" and "disgusting" all the time. Can he get it through his head that I am not screwing him to hurt his wife? I am screwing him because I love him. He treats me with love and respect and since his wife is not getting any from him, why won't he just leave her for me? You see, I'm not your average mistress, I get no thrill out of hurting my boyfriend's wife like some evil witches do. What should I do? I want him to stop hurting her and to be with me.