1. Unless the grandparents are outright abusive, your child very likely won't notice until much older, and probably won't care much even after that. (They probably won't notice or care for at least a couple of years if you don't see them, either.) Grandparents are different from parents.

2. Your wife, on the other hand, has noticed and does care. Try worrying more about her feelings than about your hopes and dreams for a child you haven't even met yet and probably won't be anything like you're expecting anyway. Your wife, however, you know.