Wells Fargo? Well fscked at the moment: Data center up in smoke, bank website, app down Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers. Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being …
I think there was a nationwide ATM outage in Feb and Apr of 2011, also caused by a single point-of-failure. Speculation was a bad software rollout, but I haven't been able to find anything more than the bank saying it was an "internal system."
But hey... it's not like being able to query a bank account is important for customers, right?
I can see my balances, but I can't move anything between accounts. Fortunately, I don't have any checks clearing at the moment.
Cue the jokes about how "I can't see the difference in Wells Fargo's usual customer service!", but actually they have been pretty rock solid through the many years they have held my money, except that they really pissed me off with their F'd up online bill paying service when the rolled it out years ago.
I have to do some grocery shopping, paid for via my ATM card, in a few hours. We'll see if I have to pay the store with another card.
I used to be a Wells-Fargo client long time ago and I can confirm that their website was rock solid and generally much more user friendly than the websites f banks I now use. Which does not change the fact that all company (banks / utilities etc.) website become more and more cluttered and unusable with time.
I see the position of Business Continuity Manager coming available soon.
People wonder why I always have cash on me. With events like this happening more and more often or even when recently I was in a shopping centre that had a power failure earlier but their EFTPOS systems were off-line I was still able to purchase my weekly food (or more importantly booze)... The queues at the checkout were non-existent because they couldn't process card payments.
<quote>I see the position of Business Continuity Manager coming available soon.</quote>
You can probably add Executive VP of Operations to that list. (One would expect that PHB to supervise the BCM.)
Icon, because it is appropriate.
I wonder, did they hire anyone from the UK banking industry???
I doubt it. Management are masters in pointing the finger downwards. Most likely some poor grunts who were simply following moronic orders will be told to clear their desks while the actual MBA gimps responsible for the mess will get a bonus for "solving" the problem they created.
We're told a fire suppression system was activated...When the fire department got there, though, they found no evidence of a blaze.
Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though. But I'm sure service will be restored after some quality pub time.
Halon may not have been manufactured for years but there was no requirement to remove or replace the installed systems. Also there are other systems using other "not nice" gasses that replace Halon. While they are supposed to be "safe" I am not sticking around if they go off; also the force that they come out with can send loose objects flying around the room.
I had to go in for 'health and safety training' at one of our data centres, and while talking us through what to do if the fire alarm goes off (turns out you should leave the building, good thing they told me that eh?), they told us that while you'd survive being in the room when the fire suppression went off (it uses some fancy new, non-deadly, chemical), you'd probably have both your ear drums ruptured by the air pressure suddenly increasing.
We had an entire data center "go quiet" once for a semi-related reason. The EPO button (emergency power off or Big Red Button) can be wired one of two ways, normally closed or normally open.
Normally closed is similar to a light switch that is always on and flipping the switch kills the power. That's how this one was wired. Even though everything had preventive maintenance twice a year, "everything" did not include that 20 year-old push button and its wiring screws slowly loosened up.
Then one day someone came into the data center and when the door closed behind them, the door next to the Big Red Button, it got very quiet.
The electricians said someone had pushed the button and the person who walked in was worried he was going to lose his job, But the security cameras showed he was nowhere near it.
The EPO button is now wired as normally open...
It is? I'd suggest you find a better fact sheet pal, perhaps one thats not written by the manufacturers.
https://en.wikipedia.org/wiki/Bromotrifluoromethane
"Human exposure to Halon 1301 can be toxic, affecting the central nervous system and other bodily functions."
"It is considered good practice to avoid all unnecessary exposure to Halon 1301"
Oh, goody. Wiki. The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting, such as how many ppm it takes to cause the effects reported.
A fact sheet not written by the manufacturers? How about the National Institute of Health?
https://toxnet.nlm.nih.gov/cgi-bin/sis/search2/f?./temp/~KAhpfw:1:human
tl;dr version: "Halon 1301 is a medically safe agent for use in fire fighting in confined spaces.[Harrison JN et al; J Soc Occup Med 32 (1): 37-43 (1982)"
--or--
https://www.ncbi.nlm.nih.gov/pubmed/4072911 which describes "An accidental discharge of a Halon 1301 total flooding fire extinguishing system." Seems that in this case, no significant illness or injury due to the Halon exposure was found.
I wouldn't spec it if it were dangerous.
"The thing that anybody with an agenda can edit, and anybody can quote without understanding what they are quoting,"
I doubt some random guy off the street wrote that wiki article. But you want something more official?
Try this:
https://www.ncbi.nlm.nih.gov/pubmed/1345555
"An accidental discharge of a Halon 1301 system is reported... .... workers present during the halon discharge reported significantly more lightheadedness, headache, voice change, cough, and a fast heartbeat than did those who worked the later shift."
"I wouldn't spec it if it were dangerous."
Sure you would, you probably only care about cost.
Halon is no longer manufactured, true. But it IS still used. Use your .fav search engine to look up "recycled halon" and the rules & regs relating to it.
I have had a recycled bromotrifluoromethane protected data center go "live" this year, in ultra-green California. (That's Halon 1301, if you care).
If you're in an office and somebody sets off a dry powder extinguisher you don't want to hang around. Although most of the powder will go where it's pointed you also get a fine dust that spreads everywhere and you'll inevitably inhale some. For the next several hours you don't want to be too far from a toilet...
“Perhaps their BOFH used the Halon system to get rid of some annoying users or managers? Bad on allowing a total shutdown and cascade, though.”
Possibly just particularly challenging contract negotiations? When the BOFH provided his or her new rate, it was considered too high and they were asked to demonstrate their value to the business?
A head count reduction through a middle management data centre visit and showing the business the damage that can be done when due care isn’t taken may have helped both parties reach an understanding.
Although 5AM suggests some of the managers may not have knowingly consented to the site visit...
Yup. Much like Bank of America, AT&T, and Atari. What's left of them is a name and some "goodwill" (in accountant-speak) after being acquired some years ago (In Atari's case, at least a half-dozen times).
Common decency forbids my naming several (formerly) U.S. and British auto and motorcycle brands...
(And don't get me started on "Craft beers" owned by mega-breweries)
Saying a prayer for my wife who works at one of their call centers - she has been front line for a lot of stuff over the past few years, and this means I had better get some sleep by the time she gets home, so I can wake up and help - my spouse is pretty strong, but when the bank has a bad day, everyone where she works has a worse one.
Just yesterday I had a laptop charger cable (C5) short and take out the circuit breaker and black out the whole section of the office. There was a flash, bang and a puff of smoke. I wasn't even doing maintenance. The laptop was charging on a desk beside me. The cable had been inspected and passed a few months ago. If it had been in the server room it could have triggered the smoke detection system. There is still a mark on the desk where the cable was sitting.
"modern VESDA system will detect before you even get visible smoke, very sensitive so you need to be careful what you take in the DC and have fly paper traps by the doors"
Modern? I had a tour of Kidde Fire 30 years ago and they were claiming that level of "fire" detection even then, ie, detecting the vapours pre-smoke or fire. Having said that, I'd imagine that level of detection is a moving target as newer or different materials are being used.
I have been a wells fargo customer since the 90s.
I don't use the online stuff often maybe once a week or something. But I just logged in, no issue(8:40 PM pacific time). Someone mentioned they couldn't move funds between accounts(2 checking accounts though the transfer doesn't use wells fargo it uses Zelle or something?? but it's fully integrated into the UI). I just did no issue. I don't really use the online banking for much more than viewing the balance and seeing the transactions.
From those aspects everything seems perfectly normal. I even clicked on a check I wrote and the image for the check came up immediately.
No errors anywhere.
I do not, and have never used their mobile app maybe that is more impacted I don't know.
I have been a Wells customer since the 1970s, and have had great difficulty getting into their website these past two days, including this morning. There are reports today of payroll deposit delays, thankfully I get paid bi-weekly so they have a week to get their act together before I completely lose it.
I may need to rethink where I bank.
A couple of nights ago half a dozen military helicopters carried out some kind of operation at a complex of buildings which included a big Wells Fargo office. This was alleged to be training exercise but it looks pretty hairy and dangerous if it was.
https://youtu.be/slCN7oz-420
I was wondering if someone had recently seen Ocean's 11 (The Rat Pack Version) and thought it could be profitably updated.
(TL;DR Latent hack to casino security is triggered by a widespread power failure. OK, blowing up a major power transmission tower is more dramatic than striking a match under the smoke detector, or maybe wearing particularly natsy cologne.)
Maybe we should check if Benny Hill is somehow still active in heists (Does WFB have Italian branches?)
"https://youtu.be/slCN7oz-420"
Looking at that, I'd say no way is that an exercise. If they need to train for that sort of situation, there are plenty of large military bases with street layouts and narrow landing zones they could practice in without doing it in a real city. There's too much potential for things to go horribly wrong, not to mention the possible damage from the helicopter down wash in such an enclosed space. You were right to use "alleged\" :-)
Of course it's a fucking exercise. Eyeball 1:14, see how the streets are conveniently barricaded at the intersection? See the city cops getting into position to take pictures right around 2:56, same intersection? Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll.
Sorry to bust a good conspiracy, but use your eyes.
"Also note that none of the folks loading up are covering their rear ... one of the "rear guard" actually looks to be out for an evening stroll."
You think they might have been under attack while leaving and so need "rear guard" cover in a US city? Whether it was for real or an exercise, they'd not be leaving an incident in a US city without being absolutely sure there was no threat behind them. It may well have been an exercise, but none of you reasons for it being so are confirmation. If it was an exercise, it was an incredibly irresponsible one.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
