Microsoft secure Azure Storage goes down WORLDWIDE
Microsoft's Windows Azure storage cloud is having worldwide problems with secure SSL storage, probably because Redmond let the HTTPS certificate expire. Being 'in the pink' is not good news for Windows Azure, as this screenshot from the Windows Azure Service Dashboard attests (click to enlarge) The problems were first …
This topic is closed for new posts.
Re: There are so many reasons...
> 3. If your ISP is down, your business is down
No it isn't.
Communications is a little harder, but I can do plenty without net access.
Vic.
Azure waves of pain
Now is a good time to suss out exactly how many Microsoft properties trust their own "enterprise class cloud provider" enough to eat the dogfood.
SSL Certs - the new single point of failure
There must be a name for the law that says "whenever you think you've eliminated all single points of failure, another one arises"?
Re: SSL Certs - the new single point of failure
Murphy's Law
Re: SSL Certs - the new single point of failure
smiles.
Like the time the digger on the M1 dug up all the 'diverse' optical fibres that connected the UK south to the UK north and cut the UK internet in half?
Like the computer in charge of monitoring and switching connections between diverse links failed to connect ANY of them?
Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank..
Re: SSL Certs - the new single point of failure
"Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank.."
I used to work for a company that was affected by the IRA Manchester bomb around 20ish years ago. Equipment was trashed, but the backups were in the firesafe, and were fine..... It just took over 2 weeks before police/forensics/health and safety would let anyone in to get them!
In the meantime, all the disaster recovery team could do was rebuild the server (as new) and create new accounts for the staff.....
.. and nothing of value was lost.
Oh Microsoft. Why would anyone trust Azure anyway? It's not like they already tried storing a bunch of customer's data for Danger.. and lost a bunch of it. Whoops.
Re: .. and nothing of value was lost.
I think your facts are a bit off there. The danger systems were not running on Azure, or even on updated MS equipment. And it appears that it was an outage that was reported as a data loss story. I have seen no evidence that anyone actually lost data after the restore was completed.
Re: .. and nothing of value was lost.
@cpreston - WTF? You come out with some real crap. You think that Microsoft have mended their ways since the Danger debacle? That was caused by a clueless MS exec. She ordered the engineers NOT to back up prior to an upgrade, despite their strong warnings. And *wham* - the users of Dangers phones lost their data.
The point is this: Microsoft are project-manager - not engineer - driven. So you will always end up with Microsoft screwing things up due to poor management decisions, PHB style.
You cannot trust Microsoft to be competent, and ergo, you cannot trust Azure.
What did we just see? A world wide outage of the Azure cloud. Caused by what seems an error that a fool would avoid making.
If MS can't manager their PKI keys, what hope is there that they can keep your data safe, or keep your data secure? As I say, only Microsoft makes these amateurish mistakes due to it's "f*ck it, just deliver" project management led culture.
EPIC FAIL!
Re: .. and nothing of value was lost.
The people who support cloud services like Azure are the engineers that develop it - so they have an incentive to prevent it falling over. No one wants a call of this nature when they're the escalation point.
You can pretty much be assured that they will engineer a way to prevent this happening again - automated SSL certificate expiration checking anyone?
Re: .. and nothing of value was lost.
@AC 15:36 "You can pretty much be assured that they will engineer a way to prevent this happening again"
Hahahah they said that about security too, but Windows STILL needs a virus checker and it's 2013!
MS write software that is low quality and they have dysfunctional management. Yes, they will put in place some process where some droid checks the PKI certs get renewed before going tits up, but, you wait and see - there will be more disasters. It's in Microsoft's nature to make something "Good enough" but not Excellent. But "Good Enough" does not cut it with clouds.
You can't make your beta software a "Version 1.0" in the Cloud because Beta testing in the Cloud gets embarrassing - your bugs cause world wide outages, as happened here.
Re: .. and nothing of value was lost.
I never suggested Danger's data was on Azure, that predates this of course, I'm just suggesting that Microsoft have a proven failure at software as a service. Just ask anyone who's running Office 365!
Look on the bright side
At least they weren't holding it wrong.
This doesn't kill "the cloud" for me
After commenting on a few other comments, I thought I'd make my own. First, I don't work for MS or any cloud vendor, so spare me the ad hominem attacks.
What I am is a huge user of cloud technology. In fact, my entire company runs in the cloud. Our phone system, our file sharing system, our CRM system, our payroll and accounting system, etc are all cloud providers.
Am I without outages? Absolutely not. Am I without problems? Nope. Have I fired some cloud vendors over the last few years? Yep.
But...
Do I have to hire and manage an IT staff? Nope. When I found out I didn't like a product, was I stuck with it because I paid for it up front and have to wait for it to depreciate? Nope. I just fired that cloud vendor and got another. Do I have to do any planning when I expand and contract the services I use? Nope.
I am far happier with my company's IT services than I was when I ran my last company and we did it all ourselves. And when there's a problem? I just call the guy. (2 1/2 men reference if that wasn't obvious.)
This is possibly the dumbest reason a cloud vendor has ever had an outage. But I have to quote the Big Man, "Let you who is without sin cast the first stone." I love all the comments that say they'll never trust the cloud because of things like this. As if you'd never have an outage if you did your own IT. Riiiiiigggghhht.
Re: This doesn't kill "the cloud" for me
Good points cpreston,
Microsoft Azure is not the only cloud service available. And I have never seen or heard of a data center that has met a 100 % SLA. Not in my past 25 years in the business. We will have to wait for Skynet Singularity and the deletion of all humans before that happens.
All these big cloud players must have at least one MAJOR fail on their books (look at AWS in Northern Virginia, FFS) as do .... *cough" .. most, if not all ,privately managed data centers.
Some of these massive failures make the news, and some ... don't.
Statistically, they are most often caused by human error.
Afterwards, failure can be measured economically (the right way) or by the degree of exposure and ensuing media frenzy (the most common way).
I will admit this is a rather SPECTACULAR fail because of the company, number of data centers, customers and zones involved. The apparent simplicity of the reason for which it happened just makes it juicier (but when is there ever a good reason?).
"Too big to fail" might work for banks and investment houses. But that 21st century oxymoron will never apply to data centers, networks, nuclear power plants, ATC systems, etc. Some things just can't be covered up. Hidden or ignored flaws have a way of finding you, usually late at night , when you are sleeping, on vacation or having a coffee break.
Whilst some of us can still rush in during that bank holiday weekend and bring the payroll system back to life before Monday morning (after a long-ignored, "too expensive to fix", SPOF finally pukes all over the system) and leave quietly with no one the wiser, that doesn't make us particularly smarter, it just makes us lucky.
Fortune 500 managed systems proposing 24/7/365 availability are obviously held to a higher standard. I suspect some highly-placed corporate dweeb-bonehead-ms-drone-bean-counter (choose all that apply) decided that automatic cert renewal might be too *expensive*. Which could also explain the magnitude of the policy error.
School boy mistake indeed.
Personally, I would call it a hanging offense, at the very least worthy of a hefty and vigorous b*tch slapping followed by 4 year re-assignment to an Alaskan data center. Pity the poor fool who ends up taking the rap.
Full disclosure: yes I did see this happen (once) in our shop, to a little-used web service.
We were lucky and smarter afterwards. Policy put in place, etc. forgiven and forgotten, bla
IMHO cloud computing will become a much healthier and more viable eco-system when it is no longer dominated by a handful of big players. Accidents like this will improve cloud deployments in the long run, particularly if companies start hiring and listening to talented people again.
Remember there is more than one way to do cloud, here are just a few:
1) Go public all the way, preferably with non-core services first until you gain confidence. There are many options out there, just go look at them.
2) Set up your own private cloud first and think about what you can peel off to a public cloud, when and if it becomes economically justifiable
3) Some hybrid of the two.
Of course, step 0 is to conclusively prove that any proposed cloud option(s) are less expensive and troublesome to run than by using your conventional data center. That is not always easy to do. But as soon as you have that proof, ît is time to to consider cloud services and solutions in the same way you consider any IT purchase. Of course, some research, assembly and homework will be required.
Remember that as long as it's cheaper and easier to do cloud options, companies will.
Although some companies may be CAREFULLY re-thinking that Azure subscription (or at least re-negotiating it).
Signed,
A self -confessed, mildly obsessed, fanboi, cloud-tech lover
Re: This doesn't kill "the cloud" for me
@cpreston - you seem to be apologising for Microsoft's incompetence. If this was a one-off occurrence, then, sure, we can say, it was just a freak mistake, it can happen to anybody.
However, you seem to be turning a blind eye to the litany of dumb F**K UPS by MS. Everything they do is dumb these days, from a legion of security fails, to customer data loss on an unprecedented scale (Danger) to the very recent Azure screw up of a crash followed by a week trying to recover the data.
If you trust MS to have the competence in security, reliability, resilience to manage your data, then you must be a PHB.
Re: This doesn't kill "the cloud" for me
Interesting. cpreston did not refer to Azure in his post but to having used more than one vendor in the past. Some stuff he mentioned is not even available in Azure...
As for reliability: Not even the control units at a nuclear plant are rated at 100 percent and no engineer that knows his job will give you that. 99.999for highly redundant specialist units [2 sets of triples with voting] is the best I have seen, two nines are more likely
Re: This doesn't kill "the cloud" for me
@Eadon - A massive MS failure, which even MS fans are saying is a total screw up and somehow you still manage to be obnoxious enough to get a ton of downvotes. My hat is off to you.
Re: This doesn't kill "the cloud" for me
@mmeier - the argument, "nothing is perfect" is a meaningless argument. Why are you guys so forgiving towards such a dereliction of duty?
Some organisations are more competent than others. Amazon, Google, Salesforce, Apple etc. do not let their Cloud certs expire. I mean, it's such a fundamental aspect of security. Only Microsoft, with their casual attitude to security, could operate a cloud system that could FAIL in this way. It hardly garners confidence in Microsoft's motivation or capabilities to manage data securely.
If MS were running a hospital, everyone would be dying in the winter because they forgot to pay the heating bills. Would you trust such a hospital? Oh sure, they can operate on me, I'm sure they know what they're doing!
If you were an engineer, you would understand.
Re: This doesn't kill "the cloud" for me
@AC 17:54 - I'm not here to win a BEAUTY contest, I am here to tell the TRUTH! And THEY CANNOT HANDLE THE TRUTH
Re: This doesn't kill "the cloud" for me
Where have I been"forgiving"? By stating the fact that no engineer would guarantee 100 percent? Like cpreston l never mentioned MS nor could l use it on the job for legal reasons [same for Amazon, Google...] And actually the security is not endangered since no one can access the data :)
For private use it is currently Google App Engine since it supports Java and Dropbox since I got some space for free. In either case 99.9 would be more than enough [8h/year] andl could live with 99 [4 days/year]
As for engineer: Master in CompSci [Dipl. lnf.] of the TU Braunschweig and a quarter century experience with more OS and languages than the average Linux freak has used distributions
Re: This doesn't kill "the cloud" for me
Here to tell the truth? And make "i shagged your mum" claims? As further up this thread...
Re: This doesn't kill "the cloud" for me
I am doing nothing of the kind. I think what MS did then and before we spectalurly stupid. What I'm responding to is the litany of posts saying "I'll never trust THE CLOUD." MS is not the cloud, nor do MS's screwups ruin the cloud.
Re: This doesn't kill "the cloud" for me
Because I wasn't replying to the story as much as I was to the "I'll never trust the cloud" comments.
Re: This doesn't kill "the cloud" for me
It's not a meaningless argument when the point that is being discussed is that people are saying "I'll never trust 'the cloud' because of this MS screwup."
If your selection of a service or an IT department is that they never have an outage -- for whatever reason -- then you're going to have a hard time finding either.
Re: This doesn't kill "the cloud" for me
Yeah, and let's face it, Eadon's posts say "virgin" all over them
Re: This doesn't kill "the cloud" for me
Oh yeah, you can't handle the truth... until your biz goes down.
Re: This doesn't kill "the cloud" for me
Cloud vendors are interested in your business (the money you pay them), not your Business (what you do to have people pay you money).
They don't know your wants/needs/tolerance for outages, etc. etc. If commodity is all you need from IT, then fine.
If IT is mission critical to you (and that is an ever-expanding universe) then you have to think twice about giving up even indirect control over what happens to the infrastructure running your Business.
Outsourcing is one thing; you have a direct line of communication to people who should understand your Business and can react accordingly when making changes and when things go sideways.
Going to The Cloud completely removes you from the decision makers and actors, you have no control over the how and when and all you can do is hope your business is enough to keep them competent enough so they don't screw up your Business.
Re: This doesn't kill "the cloud" for me
Ah, so your ass *is* potentially on the line - I was wondering about that, reading yet another of your comments defending cloud services. You're not a cloud vendor, but you seem to imply that you're responsible for making technology decisions in your company, so if your current vendor has a similar titsup, someone can use this example as "you should have seen it coming" ammo against you. In any case, thanks for the disclosure.
The backlash against cloud services you see in the comments is really due to one simple promise: that switching to a cloud provider would effectively rid your IT infrastructure of single points of failure, modulo maybe a couple of minutes max downtime in the event of a very serious emergency. Problem, this is another cloud outage in the last couple of months with exactly the opposite characteristic.
In summary: I concede that cloud services are a Very Useful Thing, but the fact is they *have* been massively hyped and overadvertised. The current reaction is simply yet another IT silver bullet having a new one ripped.
Re: This doesn't kill "the cloud" for me
@AMPC
As someone who gets to pick the pieces up after such events as this, I can't help but disagree with all that you and cpreston have presented here.
I think cpreston summed up what cloud is, when he stated he didn't have to employ IT staff anymore. It's a non technical managers wet dream.
Now they get to sit in their plush offices doing less actual work (not managing them annoying technical people for example), pointing out to those who own the business how much money they are saving the company (by not paying for IT Staff, or equipment for example) whilst down playing the risks they are exposing the company to.
I think such people are deplorable, that they would be willing to gamble in some cases the very existence of a company, and all of the staffs jobs, so that they can have an easier life (not having those annoying technical people to deal with for example, no evaluations to do, no wages to pay, etc. etc)
The only possible way for any company to be able to ensure that data is secure, and available is to do it for themselves. That costs money, but it provides security for the existence of the company, and for the staff who rely upon the continued existence of the company for their jobs.
Anyone who is dumb enough to outsource the existence of their companies viability to some third party, deserves to fail. Any manager doing this for your company should be questioned very seriously about why they would want to expose your company to such risks.
Re: This doesn't kill "the cloud" for me
@Dave Dowell
I'm sorry you find me deplorable. I'm also sorry that you misunderstood my comment about hiring IT people. I am IT people with 20 years of experience in IT. I also now employ another IT person. But we choose to use our IT skills in different ways other than supporting the day to day IT operations of our small company (<10 employees). We have "outsourced" that to various other companies. I'm sorry if you think that means that I deserve to fail. I disagree.
My choices were a lot like other small companies:
1. Hire an IT generalist that will know a little bit about everything
2. Hire a bunch of specialists via cloud companies who know a lot about a few things
I chose #2, and I don't see how that makes me an idiot. I love how you think that people who are employed by your company are inherently less risky than cloud vendors paid by your company. It's a bias that I'm trying to expose.
Please tell me why having someone on my payroll makes them less likely to screw up than someone who works for a vendor that does nothing but the thing I hired them to do (e.g. run mail services for me).
A previous commenter mentioned about the lack of control you have in a cloud situation. That is true and it's not true. It's true in that you are one customer of thousands or millions, and you have little control over how they do IT. It's not true in that if you don't like how one cloud vendor does things, there are lots of other cloud vendors to take their place.
Re: This doesn't kill "the cloud" for me
@ cpreston
I'll let you in on a secret, the managers in the company you've out sourced your compute function to, do exactly what you do. They look around the market place and employ the cheapest resource they can get away with. Sure they could pay vast sums to employ proper specialists, but like you they have targets, and budgets, and so want their bottom line to look good, so that they can get their big bonus.
All them people you don't want to employ because you don't want to pay for their skill are just like the people running clouds. Except they don't know or care about your individual infrastructure. Your individual infrastructure is just one of many they run, it's no more personal to them than any of the other infrastructures they run. When it's in a heap on the floor, they're not completely dedicated to getting it back up, they're still making sure they deliver on the other infrastructures they run. They have targets to meet as well, and delivering on all them other infrastructures, even when yours in in a heap, is part of their targets.
Cloud has a place, indeed it can be a very useful resource, but any place you put it, which could result in your company not being able to survive in the event of a failure, isn't a place where cloud belongs. From your earlier statement about how you've put your entire company IT into cloud, that's what you've done. You've delegated the responsibility for your companies survival to some third party supplier. Who as I pointed out earlier is employing exactly the people you don't want to employ, but putting them under much more pressure to maintain lots of different infrastructures. They know less about your companies specific requirements, they know less about your companies specific needs, and they care less about both than anyone you employ who would be dedicated to understanding and caring about those things.
Still it's cheap, and you don't have to worry about planning, implementation and running costs. So who cares. The staff at your cloud providers certainly don't, they'll keep their jobs no matter how badly your company fails, they have other infrastructures to run after all, so their jobs are safe.
Re: This doesn't kill "the cloud" for me
Yes, cloud service has been overhyped, even more when it comes to small/medium companies and cloud. You exchange one Single Point of Failure(SPF) like your server for another (your internet connection). You also have to trust your choosen cloud provider that he can deliver availability as contracted. BUT:
Data centers in the banks typically sell 99.99 (around 1h/year) or 99.9 (9h/year) to internal customers for most systems(1) so that should be enough for most companies even if you may be more affected by planned downtimes(2), And to get the 99.99 figure they use two centers on different locations, double data lines and all the stuff.
Outsourcing to reduce management jobs does not work. You exchange local IT where you can do the "800pf gorilla"(3) with remote IT where you threaten with contract clauses. Language differences get added even for those who speak the queens english as their primary language(4). If you do, you do it for money and likely for a non-IT business.
If not for the lousy internet speeds (and german laws) I could see the craftsman in my hometown doing it. Small/medium companies in germany use "cloud" already for tax and salaries with Datev (not called a cloud but technically the same) so doing it for bills and general writing would not be a new concept. And those companies could live with a 99 percent reliability since they have no time critical stuff
(1) Back in the early 2000 even stock trading was in that range
(2) That time from internal IT included planned downtime that where put "late" and not affect business
(3) A 190+ cm, still muscular if "manly girth" equiped man leaning on ITs desk does motivate many admins...
(4) Most Indian IT workes speak something that resembles english as much as Texan does (Or Sächsisch resembels German)
Re: This doesn't kill "the cloud" for me
You're chucking about that word security again in the casual way IT people do to scare idiot managers. The problem quite possibly happened BECAUSE of security. i.e. only a certain level of admin could renew the certificate. That certain level of admin is also quite possibly not prone to doing boring humdrum day to day admin jobs like renewing certs.
Blue Pill? Red Pill?
Except that there is so much more to the "cloud" than Microsoft Azure.
First of all, I do not like using the term "cloud". It is basically a marketing buzz word and means whatever people want it to mean. Unfortunately, we need some kind of shorthand term so cloud it is, at least for today.
Second of all
Cloud != the end of IT as we know it.
Arguing that this Azure meltdown invalidates all cloud solutions is like saying the Toyota brake system recall invalidates the internal combustion engine.
Azure is a public "cloud" service that offers storage and compute resources for rent. For many companies, that can be incredibly useful and cost effective. Microsoft is one of multiple companies offering the same service. It is using technology that is readily available to private IT shops. This market is growing by 40 % annually.
Although Azure just failed at providing that service (for about 12 hours based on recent reports), it doesn't mean cloud services and technology can't be used successfully by people who know what their doing.
With the proper staff and know-how you can build your own Azure cloud and still save money. You may even do it better than MS (or not).
Like all tech, when used properly cloud solutions can save money and even generate new business.
But a LOT of people just can't or won't grok this.
I suspect that weavers and carriage drivers also refused to accept steam engines and cloth looms, but hey, whatcha gonna do?
Refusing to understand mega trends is a very counter-intuitive approach in a market where there is a massive shortage of people with cloud skills but clearly no shortage in ostrich techies.
A business needs IT systems that help them make and save money . This fundamental rule still applies to even the most "special" environment, particularly during a bust economy
You also imply that managers choosing cloud options have never thought about the risks. But I fail to see how managers can even assess these risks without some knowledge.
When was the last time a manager asked you about the risks associated with this or that redundant power supply, server clustering technology or the company's off-site backup strategy ?
Would "never", "once" or "hardly ever" be right answers?
Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.
I suggest you at least learn about cloud options so you can explain them to PHBs, otherwise someone else will.
It really is that simple.
Re: Blue Pill? Red Pill?
@ AMPC
Why is it cloud fanbois assume that you know nothing about cloud if you speak out about it being used badly?
You should avoid patronising people who express views you disagree with by implying they havn't got a clue what they're talking about, or sooner or later you might make the mistake of patronising someone who works in a cloud company.
Thank you for your advice, I am certain however that I understand cloud, so I shan't be rushing off to learn about it.
Re: This doesn't kill "the cloud" for me
> i.e. only a certain level of admin could renew the certificate.
Possibly, but *anyone* could *review* the certificate. Indeed, it ought to have been an automated task (I can do it for my customers - why can't a big cloud provider?)
> That certain level of admin is also quite possibly not prone to doing boring humdrum
> day to day admin jobs like renewing certs
If that is so, then there is a *serious* problem in the design of that infrastructure; if the man whose job it is to ensure such a core part of the service considers that job too "humdrum" for him to bother doing it - then he's in the wrong job...
Vic.
Re: Blue Pill? Red Pill?
Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.
That's very easy. You just point out how much more cash they would have if they didn't bother to pay any insurance premiums for the year.
Cloud has its part to play in the future of IT, but I've yet to see an instance where the purported savings didn't come at the expense of increased exposure to risk.
Vic.
This what happens...
...When you stray from your core business seeking a new revenue stream and you don't, apparently, know jack about what you're doing.
Everybody sees how M$ f***ed up yet again and thinks "the cloud is not an option". It's not *the* cloud. It's M$ who fucked up yet again in short succession, in their so-called cloud, which seems to be full of schoolboy errors and single points of failure.
The biggest SPOF sits at the helm of that company, still, for a reason which is beyond many people.
> It's not *the* cloud. It's M$ who fucked up yet again
Sure.
But the issue is about who cares..
If I screw up on customer site, the man infront of me cares. And that means I care.
If a cloud provider (possibly on a different continent) screws up, he's not nearly as motivated as I would be because he is remote from the problem. And the only people the end customer can talk to is a tech supoprt department who will know very little, have no ability to effect any fix, and probably won't even know who is trying to fix it.
Outsourcing your work to any remote provider means you will have less influence over them than you would if the job were done locally. Whether that makes a difference to the business case is something that needs to be managed and documented.
Vic.
All your eggs in one basket isn't a good idea
A real world demonstration that having all your eggs in one (third party) basket is asking for trouble.
O365 - backed by a financial guarantee ...
... yeah RIGHT. Microsoft promise to pay you the paltry amount you paid them for each user. The atrocious effect on your business is not their problem and never will be.
If you choose cloud, choose very, very carefully. Putting stuff there without understanding the risk to your business could be a career limiting move.
Ha ha ha. Ha ha ha ha ha. etc
This is the best laugh I've had in years. Made my day.
@plrndl
Penguin icon, schadenfreude and childish repetition of ha. Not the most original combination, is it? Is something with MS's name on it going wrong really so great for you? I can't even say I feel sorry for you.
Re: @plrndl
@AC 16:10 ha ha ha ha ha ha ha ha ha hehehe ha ha ha ha ha ha ha ha ha :-)
Data Utilities
Will it take a catastrophic failure of a Cloud service to make people (and government) demand some sort of regulation and accountability like other "public" utilities? The Internet was designed to cope with nuclear attack - doesn't feel like it today.
Jus' sayin'.
Re: Data Utilities
TCP/IP is designed with loosing parts of thenetwork infrastructure. This is an application layer problem.
Re: Data Utilities
@mmeier - so? Without the "application layer" of the stack working correctly on the servers then the internet is useless.
Fortunately for us all, however, the internet does "route" around damage, it can avoid servers running MS Azure. Though admittedly that's not coded into TCP/IP. Yet.
Re: Data Utilities
You will learn to understand the answer when you outgrow PFY status in a decade
This topic is closed for new posts.
