Fair enough. It's a question that would be pretty common in my neighborhood (though the exchange would generally go more like, "Oh, I live at XX Y St with the cherry tree out front, which house is yours?" with the questioner volunteering the information first; asking without offering would come across a little strangely), but social norms differ from city to city and from block to block. I mean, I literally had this exact exchange with a (male) stranger yesterday and didn't think twice of it, so I guess I am a little surprised at how strongly others are reacting.