It's possible that LW was unhappy for a long time, then fell in love with another woman, and moved out of the family home before getting into a new relationship. He may be thinking that he didn't have an affair because he didn't get into a new sexual relationship before leaving his wife, while she's thinking "he left me for another woman, that makes him an adulterer."

Also, the son is 23, what does "I never want to leave him" even mean here? Even if the son still lives with his parents, he's old enough that there are no custody issues.