back to article Wells Fargo? Well fscked at the moment: Data center up in smoke, bank website, app down

Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers. Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being …

  1. Eric Olson

    This isn't the first time...

    I think there was a nationwide ATM outage in Feb and Apr of 2011, also caused by a single point-of-failure. Speculation was a bad software rollout, but I haven't been able to find anything more than the bank saying it was an "internal system."

    But hey... it's not like being able to query a bank account is important for customers, right?

    1. TheVogon Silver badge

      Re: This isn't the first time...

      Probably because US datacentres are seemingly run on a shoe string and have sprinkler systems as the cheapest form of fire suppression. So they have to kill to power if a fire is detected. Hence why no one uses sprinklers in DCs on most of the planet.

    2. TheVogon Silver badge

      Re: This isn't the first time...

      Did Wells Fargo by any chance do a TSB and outsourced support to a third world country?

  2. Marketing Hack Silver badge
    Flame

    Yep, just checked my accounts, and I am kind of borked....

    I can see my balances, but I can't move anything between accounts. Fortunately, I don't have any checks clearing at the moment.

    Cue the jokes about how "I can't see the difference in Wells Fargo's usual customer service!", but actually they have been pretty rock solid through the many years they have held my money, except that they really pissed me off with their F'd up online bill paying service when the rolled it out years ago.

    I have to do some grocery shopping, paid for via my ATM card, in a few hours. We'll see if I have to pay the store with another card.

    1. arctic_haze Silver badge

      Re: Yep, just checked my accounts, and I am kind of borked....

      I used to be a Wells-Fargo client long time ago and I can confirm that their website was rock solid and generally much more user friendly than the websites f banks I now use. Which does not change the fact that all company (banks / utilities etc.) website become more and more cluttered and unusable with time.

  3. Oengus Silver badge

    Job opening

    I see the position of Business Continuity Manager coming available soon.

    People wonder why I always have cash on me. With events like this happening more and more often or even when recently I was in a shopping centre that had a power failure earlier but their EFTPOS systems were off-line I was still able to purchase my weekly food (or more importantly booze)... The queues at the checkout were non-existent because they couldn't process card payments.

    1. Fatman Silver badge
      FAIL

      Re: Job opening

      <quote>I see the position of Business Continuity Manager coming available soon.</quote>

      You can probably add Executive VP of Operations to that list. (One would expect that PHB to supervise the BCM.)

      Icon, because it is appropriate.

      I wonder, did they hire anyone from the UK banking industry???

    2. Mr Benny

      Re: Job opening

      I doubt it. Management are masters in pointing the finger downwards. Most likely some poor grunts who were simply following moronic orders will be told to clear their desks while the actual MBA gimps responsible for the mess will get a bonus for "solving" the problem they created.

  4. Phil Endecott Silver badge

    “It is baffling that a single incident has set off such a chain reaction that it knocked out the entire internet-facing infrastructure of the bank.”

    To be read with a sarcastic tone of voice, right?

  5. SharpOB

    Wells Fargone

    ^ Nobody thought of this?

  6. LateAgain

    Smoke detection should just be a warning

    Every engineer knows that it's when the equipment runs out of smoke that it stops working.

    1. Mike 16 Silver badge

      Re: Smoke detection should just be a warning

      They should have had spares on hand:

      http://ibm-1401.info/#NewWorkR

      ( In the toolbox next to the left-handed monkey wrench and the snipe-nets)

      1. jake Silver badge

        Re: Smoke detection should just be a warning

        You must be the one who stole my metric die grinder.

  7. ma1010 Silver badge
    Devil

    The BOFH Strikes Again

    We're told a fire suppression system was activated...When the fire department got there, though, they found no evidence of a blaze.

    Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though. But I'm sure service will be restored after some quality pub time.

    1. swschrad

      Re: The BOFH Strikes Again

      Halon has not been manufactured for years, greenhouse gas, you know. Nitrogen and CO2 are in vogue. all of which will still have to be ventilated before the pocket-protector brigade can go in and try to bring the center up one power cord at a time.

      1. Oengus Silver badge

        Re: The BOFH Strikes Again

        Halon may not have been manufactured for years but there was no requirement to remove or replace the installed systems. Also there are other systems using other "not nice" gasses that replace Halon. While they are supposed to be "safe" I am not sticking around if they go off; also the force that they come out with can send loose objects flying around the room.

        1. phuzz Silver badge
          Flame

          Re: The BOFH Strikes Again

          I had to go in for 'health and safety training' at one of our data centres, and while talking us through what to do if the fire alarm goes off (turns out you should leave the building, good thing they told me that eh?), they told us that while you'd survive being in the room when the fire suppression went off (it uses some fancy new, non-deadly, chemical), you'd probably have both your ear drums ruptured by the air pressure suddenly increasing.

          1. Anonymous Coward
            Anonymous Coward

            Re: The BOFH Strikes Again

            normally you have a button to delay release of the gas if you're in the DC and its set off

            1. Amos1

              Re: The BOFH Strikes Again

              We had an entire data center "go quiet" once for a semi-related reason. The EPO button (emergency power off or Big Red Button) can be wired one of two ways, normally closed or normally open.

              Normally closed is similar to a light switch that is always on and flipping the switch kills the power. That's how this one was wired. Even though everything had preventive maintenance twice a year, "everything" did not include that 20 year-old push button and its wiring screws slowly loosened up.

              Then one day someone came into the data center and when the door closed behind them, the door next to the Big Red Button, it got very quiet.

              The electricians said someone had pushed the button and the person who walked in was worried he was going to lose his job, But the security cameras showed he was nowhere near it.

              The EPO button is now wired as normally open...

              1. TheVogon Silver badge

                Re: The BOFH Strikes Again

                A large investment bank I worked at had that happen a number of years ago by a Nigerian security guard That apparently could not read English. Pressed the EPO button next to the door.

            2. phuzz Silver badge

              Re: The BOFH Strikes Again

              The 'delay release' button is right next to the exit door, why stand there pressing the button when you can just leave? (unless you open the door and the corridor is on fire I suppose)

          2. Mr Benny

            Re: The BOFH Strikes Again

            You wouldn't survive, you'd suffocate. All these system displace oxygen which is the whole point of them.

            1. jake Silver badge

              Re: The BOFH Strikes Again

              Ah yes, the "displaces oxygen so you die of suffocation" myth. I suggest you eyeball a Halon 1301 fact sheet, Mr Benny. It's actually quite safe for humans ... at the concentrations used for fire suppression, no special precautions are required for evacuation.

              1. Mr Benny

                Re: The BOFH Strikes Again

                It is? I'd suggest you find a better fact sheet pal, perhaps one thats not written by the manufacturers.

                https://en.wikipedia.org/wiki/Bromotrifluoromethane

                "Human exposure to Halon 1301 can be toxic, affecting the central nervous system and other bodily functions."

                "It is considered good practice to avoid all unnecessary exposure to Halon 1301"

                1. jake Silver badge

                  Re: The BOFH Strikes Again

                  Oh, goody. Wiki. The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting, such as how many ppm it takes to cause the effects reported.

                  A fact sheet not written by the manufacturers? How about the National Institute of Health?

                  https://toxnet.nlm.nih.gov/cgi-bin/sis/search2/f?./temp/~KAhpfw:1:human

                  tl;dr version: "Halon 1301 is a medically safe agent for use in fire fighting in confined spaces.[Harrison JN et al; J Soc Occup Med 32 (1): 37-43 (1982)"

                  --or--

                  https://www.ncbi.nlm.nih.gov/pubmed/4072911 which describes "An accidental discharge of a Halon 1301 total flooding fire extinguishing system." Seems that in this case, no significant illness or injury due to the Halon exposure was found.

                  I wouldn't spec it if it were dangerous.

                  1. Mr Benny

                    Re: The BOFH Strikes Again

                    "The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting,"

                    I doubt some random guy off the street wrote that wiki article. But you want something more official?

                    Try this:

                    https://www.ncbi.nlm.nih.gov/pubmed/1345555

                    "An accidental discharge of a Halon 1301 system is reported... .... workers present during the halon discharge reported significantly more lightheadedness, headache, voice change, cough, and a fast heartbeat than did those who worked the later shift."

                    "I wouldn't spec it if it were dangerous."

                    Sure you would, you probably only care about cost.

                    1. jake Silver badge

                      Re: The BOFH Strikes Again

                      Whatever, dude. Believe what you like, no skin off my teeth.

      2. jake Silver badge

        Re: The BOFH Strikes Again

        Halon is no longer manufactured, true. But it IS still used. Use your .fav search engine to look up "recycled halon" and the rules & regs relating to it.

        I have had a recycled bromotrifluoromethane protected data center go "live" this year, in ultra-green California. (That's Halon 1301, if you care).

        1. Anonymous Coward
          Anonymous Coward

          Re: The BOFH Strikes Again

          @jake

          and California has about the most strict environmental laws in NA, unless I'm mistaken. I'm generally impressed with impressed when I hear about things that California is doing in the news.

          1. jake Silver badge

            Re: The BOFH Strikes Again

            I wouldn't call them strict, I'd call them draconian.

      3. TechDrone
        Alert

        Re: The BOFH Strikes Again

        If you're in an office and somebody sets off a dry powder extinguisher you don't want to hang around. Although most of the powder will go where it's pointed you also get a fine dust that spreads everywhere and you'll inevitably inhale some. For the next several hours you don't want to be too far from a toilet...

        1. Anonymous Coward
          Anonymous Coward

          Re: The BOFH Strikes Again

          I heard a story recently about a man who set off a powder extinguisher in his kitchen when something minor caught fire.

          On Christmas day. Just as his wife was getting the dinner out of the oven.

          All totally ruined obviously.

          Bet he’s still in the dog house.

        2. J. Cook Silver badge
          Pirate

          Re: The BOFH Strikes Again

          You also don't want to use a ten pound extinguisher to put out a very minor fire in a tiny closet. Ask me how I know... *coughs*

    2. Anonymous Coward
      Anonymous Coward

      Re: The BOFH Strikes Again

      “Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though.”

      Possibly just particularly challenging contract negotiations? When the BOFH provided his or her new rate, it was considered too high and they were asked to demonstrate their value to the business?

      A head count reduction through a middle management data centre visit and showing the business the damage that can be done when due care isn’t taken may have helped both parties reach an understanding.

      Although 5AM suggests some of the managers may not have knowingly consented to the site visit...

  8. jake Silver badge

    Wells Fargo is still a thing?

    Who knew.

    1. Mike 16 Silver badge

      Re: Wells Fargo is still a thing?

      Yup. Much like Bank of America, AT&T, and Atari. What's left of them is a name and some "goodwill" (in accountant-speak) after being acquired some years ago (In Atari's case, at least a half-dozen times).

      Common decency forbids my naming several (formerly) U.S. and British auto and motorcycle brands...

      (And don't get me started on "Craft beers" owned by mega-breweries)

    2. Evil_Goblin

      Re: Wells Fargo is still a thing?

      Very much so given the size and expense of the new Wells Fargo building they have just opened in London...

  9. Anonymous Coward
    Anonymous Coward

    Bollocks!

    Saying a prayer for my wife who works at one of their call centers - she has been front line for a lot of stuff over the past few years, and this means I had better get some sleep by the time she gets home, so I can wake up and help - my spouse is pretty strong, but when the bank has a bad day, everyone where she works has a worse one.

    1. jake Silver badge

      Re: Bollocks!

      Hint: Meet her at the door with a glass of her .fav plonk. For extra points, have a bath run and supper in the oven.

  10. YourNameHere

    Friday story

    Sounds like a story for your Friday story where you report on those things that you did by accident...

    1. Throatwarbler Mangrove Silver badge
      Trollface

      Re: Friday story

      . . . and then someone calls "fake!" because reasons.

  11. redpawn Silver badge

    Single point of failure

    Well... They got it down to one, can't get much better than that.

  12. Kevin McMurtrie Silver badge

    Hopefully everybody is still getting their 0.01% annual yield during the outage. That can add up to tens of cents over time.

    1. Anonymous Coward
      Anonymous Coward

      0.01%

      Well, it's Wells Fargo, so at least you're getting 0.01% on eight different accounts! (that was their previous scandal... pushing a million services on everyone)

  13. W.S.Gosset Bronze badge

    Hardcore admins

    > smoke was detected following routine maintenance

    Say what you like about them, that's some SERIOUS maintenance. I've never had a script set fire to a machine -- I must clearly be a lightweight.

    1. Oengus Silver badge
      Mushroom

      Re: Hardcore admins

      Just yesterday I had a laptop charger cable (C5) short and take out the circuit breaker and black out the whole section of the office. There was a flash, bang and a puff of smoke. I wasn't even doing maintenance. The laptop was charging on a desk beside me. The cable had been inspected and passed a few months ago. If it had been in the server room it could have triggered the smoke detection system. There is still a mark on the desk where the cable was sitting.

    2. Korev Silver badge
      Flame

      Re: Hardcore admins

      We had a GPU used for MD calculations overheat and catch fire!

  14. Anonymous Coward
    Anonymous Coward

    Where's there's smoke there's fire.... What's the rest of this story??

    1. Anonymous Coward
      Anonymous Coward

      modern VESDA system will detect before you even get visible smoke, very sensitive so you need to be careful what you take in the DC and have fly paper traps by the doors

      1. John Brown (no body) Silver badge

        "modern VESDA system will detect before you even get visible smoke, very sensitive so you need to be careful what you take in the DC and have fly paper traps by the doors"

        Modern? I had a tour of Kidde Fire 30 years ago and they were claiming that level of "fire" detection even then, ie, detecting the vapours pre-smoke or fire. Having said that, I'd imagine that level of detection is a moving target as newer or different materials are being used.

  15. Nate Amsden Silver badge

    working fine for me?

    I have been a wells fargo customer since the 90s.

    I don't use the online stuff often maybe once a week or something. But I just logged in, no issue(8:40 PM pacific time). Someone mentioned they couldn't move funds between accounts(2 checking accounts though the transfer doesn't use wells fargo it uses Zelle or something?? but it's fully integrated into the UI). I just did no issue. I don't really use the online banking for much more than viewing the balance and seeing the transactions.

    From those aspects everything seems perfectly normal. I even clicked on a check I wrote and the image for the check came up immediately.

    No errors anywhere.

    I do not, and have never used their mobile app maybe that is more impacted I don't know.

    1. Anonymous Coward
      Anonymous Coward

      Re: working fine for me?

      I have been a Wells customer since the 1970s, and have had great difficulty getting into their website these past two days, including this morning. There are reports today of payroll deposit delays, thankfully I get paid bi-weekly so they have a week to get their act together before I completely lose it.

      I may need to rethink where I bank.

  16. chivo243 Silver badge
    Facepalm

    Wells hastogo

    An ex-colleague was a contractor for WF back in the 90's and early 00's. Oh, the stories he would tell. Kludge this, fudge that. I will never put one cent in their incapable hands...

    1. Down not across

      Re: Wells hastogo

      Kludge this, fudge that. I will never put one cent in their incapable hands...

      What makes you think it is any better at any other institution?

  17. 89724102172714182892114I7551670349743096734346773478647892349863592355648544996312855148587659264921

    "fscked"

    Why don't we FCUK anymore?

    Trademarks are out of control

    1. John Brown (no body) Silver badge

      "Why don't we FCUK anymore?"

      Because it's an IT site, not a fashion site.

    2. M.V. Lipvig

      I like fscked better than fcuk anyway. It implies "two" things.

  18. tony2heads
    Joke

    bad update maybe??

    The one with the 'halt and catch fire' instruction

  19. Joe Harrison Silver badge
    Black Helicopters

    Datacenter fire just a cover story?

    A couple of nights ago half a dozen military helicopters carried out some kind of operation at a complex of buildings which included a big Wells Fargo office. This was alleged to be training exercise but it looks pretty hairy and dangerous if it was.

    https://youtu.be/slCN7oz-420

    1. Mike 16 Silver badge

      Re: Datacenter fire just a cover story?

      I was wondering if someone had recently seen Ocean's 11 (The Rat Pack Version) and thought it could be profitably updated.

      (TL;DR Latent hack to casino security is triggered by a widespread power failure. OK, blowing up a major power transmission tower is more dramatic than striking a match under the smoke detector, or maybe wearing particularly natsy cologne.)

      Maybe we should check if Benny Hill is somehow still active in heists (Does WFB have Italian branches?)

    2. John Brown (no body) Silver badge

      Re: Datacenter fire just a cover story?

      "https://youtu.be/slCN7oz-420"

      Looking at that, I'd say no way is that an exercise. If they need to train for that sort of situation, there are plenty of large military bases with street layouts and narrow landing zones they could practice in without doing it in a real city. There's too much potential for things to go horribly wrong, not to mention the possible damage from the helicopter down wash in such an enclosed space. You were right to use "alleged\" :-)

      1. jake Silver badge

        Re: Datacenter fire just a cover story?

        Of course it's a fucking exercise. Eyeball 1:14, see how the streets are conveniently barricaded at the intersection? See the city cops getting into position to take pictures right around 2:56, same intersection? Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll.

        Sorry to bust a good conspiracy, but use your eyes.

        1. John Brown (no body) Silver badge

          Re: Datacenter fire just a cover story?

          "Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll."

          You think they might have been under attack while leaving and so need "rear guard" cover in a US city? Whether it was for real or an exercise, they'd not be leaving an incident in a US city without being absolutely sure there was no threat behind them. It may well have been an exercise, but none of you reasons for it being so are confirmation. If it was an exercise, it was an incredibly irresponsible one.

    3. imanidiot Silver badge
      Facepalm

      Re: Datacenter fire just a cover story?

      Man, they call that dangerous and hairy nowadays?? They'd be able to fly 2 abreast in that street! Plenty of space, roads closed off. Calm clear weather. <sarcasm> Yeah, really dangerous... </sarcasm>

  20. fredesmite
    Facepalm

    A simple software upgrade to the UPS controllers !!!

    Don't worry

    It's just a 1 line bug fix

  21. Ken Mitchell

    Wells Fraudo

    Couldn't happen to a better bank than Wells Fraudo. And I was NOWHERE NEAR the place!

  22. Kanhef

    Of course there were backups

    They were in the next server rack over.

  23. fredesmite

    EVERYONE KNOWS WHEN THE SMOKE LEAKS OUT

    of those little black chips they stop working !

    smoke makes them work !

  24. DerekCurrie
    Paris Hilton

    Why use Wells Fargo?

    After a multitude of failings, both deliberate and inexplicable...

    After gross customer abuse...

    After contributing to the 'Great Recession' through malfeasance...

    After requiring a US taxpayer paid bailout after malfeasance...

    Do failures attract?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019