I think it might be independent of factual qualities at all. Like if someone made an observation without value judgment about, idk, analytical reasoning abilities or IQ scores, it could go into conversation without rancor. It's the assumption that this particular difference, unlike all the other ways you're not expected to be exactly equal to your partner, constitutes a gulf in your value as people, or at least a deal-breaker for compatibility that has any real significance. Nobody would say that someone married "below" them in artistic talent or lung capacity or hair curliness, you know? It's nonsensical.