They could have easily got it from you.
Have you ever had your wifi and GPS turned on at the same time?
Did you check the option to say you're willing to share the wifi positional data is acquires with google? (You can change your mind by going into location settings and unchecking the "use wireless networks" and then checking it again. It asks for confirmation each time).
Then again, even if you didn't, someone else walking past (doesn't have to be a streetview car) could easily have been.
It's actually a useful feature. It allows the device to locate your initial (vague) position very very quickly. A cold start on GPS can take half a minute or so without it, which is a damn long time when you're sat at the front of a queue of traffic trying to work out where you are.