Madison stood somewhat at odds with most of the framers of the US constitution in believing that the government should by necessity control the governed. This is a flawed premise to start from, as it assumes that the government allows the governed to remain at its behest, when the opposite is and should be the case. The majority of the US founding fathers were of the opinion that the government should be subservient to the people, who would exert the necessary control over that government, removing the need for the government to control itself. This attitude works as long as your government is composed of people who believe that the government should serve the governed rather than control them.
The last century has seen a progressive move toward the Madison way of seeing things, placing the governed in a subservient position to the government. The end result is that people go into government believing they have the right to do whatever they want, and the consequence is Dubya (whom I once supported unconditionally), Clinton, Nixon, Blair, Brown, the EU... Stalin hitler pol pot mao the japanese empire and god knows what else.
Start from the correct premise, that the government stands and falls only by the will of the governed, STICK to that premise, and you will have a truly representative and just government. The mistake came in listening to people like Madison rather than reading the words of the US declaration of independence:
"We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.
That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed"
A government should make no assumption of an implicit right to control the governed. The reason we're facing all these intrusions into our private life is precisely because people listened to people like Madison and his assumption that the government should have that control.