In my experience of the kind of people who might say they want Von Neumann's baby, this isn't so much "I would have a baby with this famous person but not you", it's "I want my future progeny to be theoretical and mathematical, not biological". LW it's hard to tell without more context on the "extremely hypothetical" context - it sounds like the conversation might not have originally been about babies - but if that's what you meant, then it might be worth unpacking it to yourself and then to your boyfriend ("BF, when I said that I didn't mean I was lusting to change diapers in 1940s New Jersey, I meant I was more interested in abstract math than in babies. I hope that didn't make you think I would change my mind about having more real babies. It's still a firm no, whether you're a math genius or not.")

That said, if you really would be interested in having a (real) baby with a different (real) guy but not this one, whatever your reasons, you need to work through that with both him and yourself, too.