One-stop shopping for ID thieves; but necessary
The article makes it sound all so pointless and sinisiter and intrusive... Well, they're right on intrusive, but then, you don't have to ask for a security-clearance job, either.
The primary point of the SF-86 is to give the suits doing your background check a leg up, so they don't have to spend so much money checking you out. If they had to do it from scratch, it would take years, and cost millions - per clearance.
It's not JUST the leg up, either; those references that they could 'just look up' themselves? Who you pick as your references tells something specific about *you* - your judgement, that is. Did you pick morally-upstanding citizens? Or did you cite your drug-smoking trouble-making friends? (I have literally seen just exactly that on an SF-86!) If you lack the judgement to cudgel your brain for a few 'good citizen' refferences (or don't have any!), then the guys in suits can bin your application for 'demonstrated poor judgement' and save the taxpayers a lot of cash.
The more complete your SF-86, the less it costs for the government to make an informed decision about your judgement, trustworthiness, and lack of hostile influences - All of which are necessary. It generally works out pretty well - Despite some very public breaches of late, the VAST, overwhelming majority of security-cleared persons go abuot their jobs in a trustworthy manner, day in, day out, for their entire careers.
Which of course makes the breach of trust on the part of the government that much worse. I'll guarantee you that my name is in that pile of data. And my wife's. And my father's. My children. My siblings. And pretty much all of my friends. Every last one of us, betrayed by the government to whom I (and my wife, and my father) rendered faithful service.