I would say picking up fruit or nuts from the ground that are on or adjacent to the sidewalk would be fine. In my childhood, you could go for a walk around the neighborhood and gather baskets of pecans from the streets and gutters, which seems obvious to me - but equally obvious that you can't go into the yard to get more.

I would not even consider picking fruit from bushes or trees in someone else's yard, nor flowers for that matter, regardless of where they were reachable from, and the suggestion strikes me as bizarre and a bit shocking.