Well exactly
It's not a criticism of Kennedy Space Center, as such, but the fact that shuttles, specifically, are launched there (owing to their extra sensitivity to lightning risk, compared to most rockets). Weather has been one of the constant factors in delaying shuttle flights over the years. Tanking of propellant isn't even begun if there is a more than 20% chance of lightning within 5 miles of the launch pad.
White Sands is only 4° north of Canaveral, and both are more that 28° north of the equator, so the orbital velocity argument isn't that strong for Canaveral over White Sands, or some similar inland desert location.
One of the main reasons the Shuttle was less successful than promised was that turn around time between launches was much higher (meaning that there have been about a quarter, the number of flights, than were originally predicted), and reliability of launch time was much lower. Turn around times (among other things) could have been greatly helped if the things had been launched from the same site, they are often landed at, while reliability of launch envelope could have been improved by launching from a site with much less interesting weather.
However, as you say, you do get to watch them from your backyard, and who in their right minds would chose to live amidst cactus, when you can have palm trees?