Feeds

back to article Microsoft's own code should prevent an Azure SSL fail: So what went wrong?

Server 2012 is the Microsoft operating system that, in my opinion, makes cloud computing a reality. As far as I am concerned it is as big a leap over Server 2008 R2 as that OS was over Server 2003. With it you can build anything from a small cluster to a service as big as Microsoft's own Azure platform. Which is why I am …

COMMENTS

This topic is closed for new posts.

Page:

Bronze badge
Joke

What went wrong?

They took Eadon's advice and brought him in as a consultant...

17
3

This post has been deleted by a moderator

Silver badge

Re: What went wrong? @Radon

Is that why you're on the anti-MS soapbox? They didn't give you a job? That actually makes sense.

8
4
Bronze badge

Re: What went wrong?

I think it would be safe to say you hate windows. So why would they hire you?

1
2
Bronze badge
Mushroom

Re: What went wrong?

"Even if it was the cryptographic certificate upstream from the end nodes that expired, why wasn't the CSC server auto-renewing from elsewhere?"

The answer to that would be that a properly designed and secure PKI root certificate server architecture is not connected to the LAN.

In Microsoft's case, the approach is publically documented: http://blogs.technet.com/b/aviraj/archive/2011/10/04/microsoft-downloads-public-key-infrastructure-at-microsoft.aspx

3
2
Bronze badge
Happy

Re: What went wrong? @Radon

MS' HR is highly professional, yeah, they hired someone else, that very guy who slept through the ssl certificates renewal. What a ballmer.

6
0
Bronze badge
Happy

@1Rafayal

It actually makes a lot of sense. Suppose MS hire Eadon as a consultant.

MS: Eadon, what do you feel about us, Microsoft?

Eadon: you're incompetents mor*ns you always forget about the leap year day

MS: Urgently notify their leap year team about the upcoming leap year event.

----------------------------------

Otcome: no leap year embarrassment for MS ever occurs

MS: Eadon, did you change your opinion about us?

Eadon: Of course not, you're so unprofessional, you can easily forget to sign your own SSL certificates

MS: hastily texting, emailing and making phone calls attempting to wake up the dormant SSL/HTTPS certificate team

-------------------------------------

Result: This discussion doesn't take place

3
1
Anonymous Coward

Re: What went wrong?

I remember when Amazon's storage went down across multiple regions and it took them a lot longer than 9 hours to fix it!

http://www.nytimes.com/2012/12/27/technology/latest-netflix-disruption-highlights-challenges-of-cloud-computing.html?_r=0

0
0
Stop

Re: What went wrong?

Quite. Whilst it's certainly possible to automate certificate management, it's sort of like maintaining service uptime by sending service restart commands over telnet. Insecure, undesirable and just a little bit quaint.

0
0

Will this work with Symantec (Verisign) certs?

I buy a 1 year cert, but it will have an expiry date in 2018.

However it will be revoked this time next year if I don't pay them for another year.

I would like to buy the 5 years together, but beancounter says no.

0
0

So don't buy an over priced Verisign cert with an expiry date you cannot track.

2
0
Bronze badge

RE: I would like to buy the 5 years together, but beancounter says no.

The solution is right out of a BOFH's playbook.

Invite the beancounter into the IT lair, and motivate said beancounter with the exposed ends of a mains cord. His experience will be 'shocking' (to say the least), and hopefully it will instill a newly found sense of respect (or fear as some might put it) for the powers of IT.

If that doesn't get the beancounter to see the error of his ways, then arrange for a "data corruption" to occur in his payroll records. A salary that mysteriously changes to ZERO; and resists any attempts to "correct" it.

I mean, use your imagination.

1
0
Silver badge

> I'm not fully sure of the underpinnings of Azure; does it run on Server 2012?

According to wikipedia (yes, I know):

"Windows Azure has been described as a "cloud layer" on top of a number of Windows Server systems, which use Windows Server 2008 and a customized version of Hyper-V, known as the Windows Azure Hypervisor to provide virtualization of services. "

2
0

This post has been deleted by a moderator

Silver badge

Re: Sys admins should not be responsible for managing security, they implement it.

There's a standard admin response. "Yes I know it says administrator in my name and I demand oversight of every detail, but if it breaks that's not my problem".

3
1
Anonymous Coward

Re: Sys admins should not be responsible for managing security, they implement it.

It is a sysadmin's *problem* certainly. But it's never their fault.

2
0
Gold badge
Meh

Re: Sys admins should not be responsible for managing security, they implement it.

Ah. I believe that's commonly known as the "Nuremberg defence".

4
0
Paris Hilton

Re: Sys admins should not be responsible for managing security, they implement it.

Rubbish. In a large enterprise such as Microsoft, Information Systems and Information Security should occur parallel but apart from each other. This ensures that systems management decisions and systems security decisions happen apart while the actions can take place in chorus. Otherwise, security often takes a back seat to the day-to-day operations.

Even for the small guys like us, operations and security often get mis-prioritized between the two and the occasional SSL certificate expiration might go forgotten until the phone calls start coming in about not being able to access email or other services because we were working on a MySQL to Maria migration or new virtual server spin-up. Stayed too late working on it and, whoops!, the 6am phone calls hit us.

Keep operations and security separate in the management process and there is a much better chance of over-all success as each management focus pushes and coordinates priorities for the action team.

Paris, gotta keep 'em separated.

1
0

An accountant is in the wood pile almost certainly

This time of year there's almost certainly an accountant complaining that they can't authorize any expenditures, so, no you can't pay for the certificate. It will have to wait.

3
0
Holmes

SysAdmins versus Ops versus Sec

System admins deal with systems, ops usually deal with servers and infrastructures, sec deals with the security of them...

I'm sure back in the older days, certificate administration was someones full time occupation?

Theoretically, you would need triple the amount of Sec to cater for infrastructure, server and system?

A basic maths principle, I would have thought?

1
0
Facepalm

Re: SysAdmins versus Ops versus Sec

You'd think that would be the case. But what's stopping the Sys Admin creating a certificate validation spreadsheet/database of all the cert's in production/test, location etc and then assigning calendar reminders of up to 4 weeks before the cert becomes invalid? Surely that should be setup on birth of a certificate? It's not exactly rocket science; it's common sense and effective administration.

1
0

This post has been deleted by a moderator

Bronze badge

Re: SysAdmins versus Ops versus Sec

you are forgetting, or more likely not aware of the change team.

There will be some change manager in place to ensure that the certs meet what ever requirements are needed to get a new cert installed and on time. It is highly likely that someone from a PMO would also be involved to manage the idea of getting new certs out to where ever they need to go.

So, a security type can say a new cert is needed, an ops guy can provide the mechanism to get it in place and a sysadmin can look after the system. But without a PM and a CM, who looks after the process behind actually getting the cert in place?

Unless you feel that a security guy should do this as well?

1
0
Windows

Re: SysAdmins versus Ops versus Sec

Why mess around with a spreadsheet? Surely networking monitoring/syslog/SNMP must have the ability to say <8 weeks remaining on this cert and send out an alert/change an icon to yellow/something to draw attention to the issue?

0
0
Bronze badge

Re: SysAdmins versus Ops versus Sec

You obviously haven't ever worked with a project manager.

A project manager doesn't say "we can save money by getting a sysadmin to do it". That is not a process. That's what is said by the "boss" at an SME where your responsible for the maintenance of everything that uses electricity or moving parts.

A project manager breaks a job down into processes and individual parts that can be seperated out and safely delegated (often with break points to prevent a catastrophe if it's not bring done to standard) Those processes are then usually divided in such a way that they can be done by less expensively trained and competent staff. This is usually done by giving each position a narrow set of personal responsibility and severe penalties for stepping outside of the process.

The problem with Microsoft was probably that it wasn't anybodies job specifically to look after issues like this, so nobody did it. Alternately, someone did discover the issue but didn't have enough time to push the problem through the byzantine change management system designed to stop people the horde of relatively unskilled people breaking things they don't understand.

1
0
Bronze badge

Re: You obviously haven't ever worked with a project manager.

The best1 definition of a project manager I have ever seen is:

A project manager is the organizer of a series of clusterfucks resulting in a catastrophe, spoken with respect to either an Oracle or SAP implementation, and about some 'large accounting firm".

1 "Best" - lies in the eyes of the beholder.

1
0
Paris Hilton

Re: SysAdmins versus Ops versus Sec

This could be an explanation of what happened with CenturyLink a week or so ago. It let "embarqservices.net" domain expired which had an impact on its entire IP infrastructure, at least from a down-stream perspective.

I first noticed it when reverse resolution verification prevented me from connecting to one of my edge SSH machines. I hobbled in through the VPN to see what was going on. At first I though it might be a delegation problem with my provider, which happened before when CenturyLink gave its IP space to another customer, but a quick phone call to my old boss (now the administrator of my provider) determined otherwise. We found that it was affecting a large swath of CenturyLink, and syslogs being sent to me from CTL clients were showing IPs rather than names.

It took about 20 minutes to work our way back up to DNS: the servers listed for the CTL network are ns(x).embarqservices.net which had expired, as we figured out via a cached SOA pointing the domain to "pendingdeletion.com" or something of that nature. Funny enough, this saga started around 9:30pm when I noticed the problem, called my provider around 9:45pm, and we figured out the problem around 10:10pm, by then someone at CTL had renewed the domain around 10:00pm-ish. A few rndc flush commands and Windows cache flushes and it was like nothing had ever happened.

There is a contact listed for the domain. And I assume someone got the renewal emails. And my guess is when the domain expired someone was trying to get hold of the pockets without success and whipped out their own credit card to pay the renewal for a year. $35 is a small price to pay to keep your job, I suppose.

Paris, small price to pay...

1
0
Silver badge
Windows

One problem with the article...

"Not only can Microsoft sign its own damned certs, Server 2012 makes this whole process so simple web administrators will weep."

Which is of course assuming that Microsoft actually uses that stuff themselves. Be very careful there because Microsoft has a solid history of telling the world "A" while doing "B" themselves. For example; when they started pushing Exchange forward as the big Windows MTA which could easily rival Unix environments their own e-mail facility remained hosted by Unix for several years to come. Simply because Exchange wasn't capable to handle their load. Something which they tried hard to keep under wraps of course.

Just because Microsoft has released a new server doesn't mean they immediately started using it themselves, that line of thinking is IMO a bit silly. In fact; I would deem it much more likely that parts of MS are currently running on hybrid solutions; a Windows core which is being maintained in-house themselves and as such its no longer really server 2003 yet also not really server 2008; but instead a 'hybrid' sitting somewhere in between which got all the solid enhancements to their server line of products yet without the bloat.

Because just like any other Enterprise environment Microsoft knows that with every change you apply you always risk introducing a certain danger to the system. Considering that they also maintain their own OS updates; why not utilize that themselves as well ?

6
0
Bronze badge

Re: One problem with the article...

I would argue that you are pretty much right, but with one caveat; I dont think MS would use hybridised versions of its own server OS in mission critical platforms like Azure (for example).

I completely agree that these systems exist, and that they are being used, but MS needs to present a business use case for a lot of their new offerings, I think someone has already said Server 2012 will let you run an Azure like cloud if you wanted to do so. In my mind it would make a lot of sense to me for MS to run its cloud in the same way other would or could.

But, this is just my opinion, nothing more.

2
1
Bronze badge
Mushroom

Re: One problem with the article...

It is a known fact that Azure runs on a version of Hyper-V Server 2012....And that Microsoft almost always 'dogfood' their products internally before releasing them publically.

0
3
Anonymous Coward

Re: One problem with the article...

>It is a known fact that....Microsoft almost always 'dogfood' their products internally before releasing them publically.

Que? No comprehendo.

You seem to have buggered up your grammar and spelling there RICHTO. Allow me to help you:

Microsoft almost always releases 'dogfood' well before it should be releasing the products publicly.

5
0
Silver badge

Re: One problem with the article...

Maybe you've not heard the term before.

"Dogfooding" means "using your own products internally".

It is almost universally a good thing, as it saves the supplier money and helps find subtle bugs.

Aside from that, would you trust a supplier who doesn't trust their own products enough to rely on them for their own business?

It comes from "Eat your own dogfood".

2
2
Joke

"Server 2012 is the Microsoft operating system that, in my opinion, makes cloud computing a reality"

Thanks for the laughs!

10
0

Er,

Probably a stupid question, but is it possible that MS were indeed running a single CSC server, and it went titsup?

1
0

easy failure - designed to fail.

More like the root CA expired.

Regenerating one requires a manual intervention somewhere.

Of course, as soon as the root CA expires, so do ALL certificates signed by that root CA.

The next problem is how long is the root CA valid? If it is longer than the persons job, how does the responsibility get passed...

And if the root CA lasts 5 years, and you change personnel 3 times, how likely is it that the current person in charge knows they are responsible for a root CA that "just works".

Point and click admins don't know how certificates actually work. And that makes it easy to skip a simple message "the certificate is about to expire" as they just click to update one... Unless it is the root CA.

I would almost bet the person doing this spent the last 12 hours before the CA expired trying to find the documents on how and where to generate the new certificate, AND get all the updates distributed world wide.

2
0
Bronze badge
Mushroom

Re: easy failure - designed to fail.

The process and people are almost certainly well defined. However obtaining the required approvals, and getting the right 2 security people out of bed to the right datacentre and to physically issue the required Certs from the offline root server would probably take a while...Requesting a certificate is not supposed to be something that you need in an emergency.

1
2
Bronze badge

Re: Point and click admins

AKA, your average Windows Click Monkey.

3
2

Re: easy failure - designed to fail.

Designed to fail, because it's a single point of failure. I bet about everything else with the system is redundant.

The entire cert system needs to be ran off 2 different CA's, the entire system can run off one, but has a total fit about it (leading a person to correct the problem). Oh, and make sure the CA's expriy dates are significantly different.

0
1
Bronze badge
Mushroom

Re: easy failure - designed to fail.

Thanks for demonstrating your lack of understanding as to how PKI works. You can't run an "entire cert system" off 2 different CAs. That would by definition be two entirely separate and different cert systems...

The best you can do is cluster the CAs (If you run Windows you can anyway).

For offline root CAs, a regular secure backup is the best option.

0
5
Bronze badge

Re: easy failure - designed to fail.

Thanks for demonstrating your lack of understanding as to how PKI works. You can't run an "entire cert system" off 2 different CAs. That would by definition be two entirely separate and different cert systems...

There's no reason you couldn't provision a network of SSL/TLS servers from a dual X.509 PKI hierarchy with two CAs, a primary and a backup.

Simply give the servers two certificates, one signed by each CA. Give the root and server certs distinct expiration dates - 24 hours apart is probably safe, but there's no reason not to provide a wider margin, say a week. Have the servers check the expiration dates of their own "primary" certificate and its root (and any intermediate certs in the chain, of course) and if the expiration is dangerously close (as defined by an appropriate safety margin, based on things like your SSL/TLS session-resumption timeout), send the "secondary" certificate instead.

Obviously that means two root certs to distribute to the clients, rather than one, but typical clients have so many damn roots installed it hardly matters.

Then, a server that has to switch to the secondary should send an alert - SNMP, SCOM, whatever your operations console uses.

There's nothing magic about the "one certificate per server" rule. The client wants to see a server cert that has a subjAltName or CN that matches the FQDNS name of the server it's trying to connect to; that was signed by an issuer it recognizes (or is at the end of a chain that etc); that hasn't expired; that isn't listed on a CRL or dissed by OCSP; and so forth. It shouldn't care if it's not the same certificate it got the last time, and while there are clients that will notify you of the fact (eg Firefox with the Cert Patrol extension), it's quite acceptable.

0
0
Holmes

Inexcusable

Fill in the blank with any excuse you like, but the fact remains this was a simple thing to look after and totally avoidable. Its not like Microsoft doesn't have the resources to make sure the simple things are taken care of. Apologists be damned, Microsoft is poorly managed to allow this t happen...

5
0

Re: Inexcusable

Yup. Totally pointless article. The bottom line is Microsoft is badly managed and promotes a culture of incompetence, much like the software they unfortunately keep trying to inflict on people who don't know any better.

3
1
Silver badge

The real reason would be that, like any other organisation, they cant afford

to have their IT staff off having an expensive lecture in where the sodding icon has been moved every time the software is shuffled, sorry upgraded.

2
0
Bronze badge

Re: The real reason would be that, like any other organisation, they cant afford

I think the issue simply boils down to the fact there should have been either a system or process or both in place to have caught this well in advance.

How many people here need to deal with licence credentials for the software they work with? Spotting when they expire is a fairly run of the mill job for most people, even if it comes down to a calendar entry in Outlook.

1
1
Bronze badge

Re: The real reason would be that, like any other organisation, they cant afford

I think the issue simply boils down to the fact there should have been either a system or process or both in place to have caught this well in advance.

Yes. Ironically, Microsoft notified me back in October that my Azure ACS 2 certificate was going to expire - a few days from now. Apparently they're better at tracking certificate expiration for Azure customers than they are for themselves.

Also ironically, I haven't gotten around to replacing that certificate yet. It will most likely expire before I do. (I only have it for development purposes, and I haven't needed to do any ACS work in months.) In my defense, rolling the certificates over would take at least three or four minutes, which I could use instead to post comments no one really cares about on the Reg.

0
0
Anonymous Coward

They don't?

Microsoft doesn't have to worry about licensing Microsoft's own kit, so how exactly did this happen?

In my experience from a large Swedish Telecom business: It is *exponentially* harder to license another departments products that any external vendors!

"Aiding and Abetting The Enemy, that's what that is" -

Relying on stuff controlled entirely by the competition, those other managers looking for the next vacant slot in the hierarchy, is a *dumb* move.

Building an empire of vertical stove-pipes with own-versions of everything needed to ship the product one is responsible for, is the smart move.

Who cares that this costs money, those are *shareholders money*. Leaving too much of them on the table only makes the top executives more popular, thus difficult to remove, limiting the stream of future career opportunities.

In fact, Open Source remains the normal, safest and most career-conscious way to share common code and applications between profit centres.

3
0

The answer to the question at the end of the title of this article simply resides in the very first word of said title...

3
0
DJV
Facepalm

You beat me to it - though I was going to say the first 3 words...

1
0
Silver badge
Devil

So if you want a good, stable Cloud coputing system do you choose:-

a. Microsoft, A 'Software Company' or

b. Amazon, a bookshop?

Answers on a Valid Certificate...

2
0

This post has been deleted by a moderator

Page:

This topic is closed for new posts.