Its the camera, stupid.
I'd expect exactly the same reaction from my neighbor if I attached a camera to a long pole and used that to look at his yard. (Maybe not exactly the same reaction -- not all Americans shoot stuff on sight.)(Also, not all of them think that everyone's a perv after their kids.) You have to use a bit of common sense when you're flying these things.
Model aircraft are slightly different from quadcopters in that they're not purpose built camera platforms. You can mount cameras on them but you get the same sort of view you'd get from a low flying aircraft -- you can tell where you are but you're not able to pick out details. Despite this you still have to give assurances to homeowners that you're not carrying cameras ("Thank you, drone owners").
Amazon's idea of using traffic space won't fly. They seem to think that model aircraft are toys that fly around in front of your face. Some are; a lot are more serious and need a lot more room to fly them. My interest is in gliders -- unpowered planes. These can be quite large -- 3 or 4 meter wingspan is typical but they're made larger -- and some specialist types can be flown quite fast ("but not near people") -- the current speed record for dynamic soaring is a little north of 500mph. People make them for all sorts of reasons, one is that they just like modelling but its also a useful research vehicle for testing out design concepts and construction techniques (no, we don't make them out of balsa....).