back to article 'Major incident' at Capita data centre: Multiple services still knackered

A major outage at a Capita data centre has knocked out multiple services for customers – including a number of councils' online services – for the last 36 hours. Some of the sites affected include the NHS Business Services Authority, which apologised on its website for the continuing disruption and said it hoped its systems …

Re: British Airways

These are low paying holiday makers, not business travellers

0
0
Bronze badge

As Battery Sergeant Major Williams said...

Oh dear, how sad, never mind.

2
0
Anonymous Coward

Wankers

1
0
Silver badge
Mushroom

What they didnt tell you about te cloud...

was that it was your security and ability to do business at all you were outsourcing.

2
0

To all Capita clients.

Did you buy DR? If you did, was it tested?

If not, then it is your fault.

(Ditto BA)

0
2
FAIL

Lots of Pension Schemes affected

Not just my pension scheme is affected... some other small schemes also affected: Teachers Pensions, NHS Pensions

(https://www.nhsbsa.nhs.uk/employer-hub/pensions-online: "Service disruption. We are sorry for the continuing disruption to our services and for any inconvenience this may be causing. We are hoping to have our systems available by noon on Friday so please try again at this time. Until then contact centre staff will, regrettably, be unable to assist with requests that require system access. Thank you for your patience.")... it's Sunday 28 May now...

good thing it's a long bank holiday weekend... an extra day to get the rubber bands and sticking plasters in place.

0
0
FAIL

Still down on Sunday 28 May 2017 06:45

My pension scheme still has no online service. So that's an outage of 72 hours (at least). Perhaps they'll refund the annual management charge?

0
0
Anonymous Coward

Beanclusterfsck?

Previous public sector role: engineers were taken aside early in job and it was explained very clearly that bad design, corner cutting or straightforward screw-ups could lead to loss of job and the possibility of personally facing civil or criminal charges (with no official support). Beancounters who force cuts? No sanctions at all. Middle or senior managers who back the bean counters? Safe as houses. Middle or senior managers who back the engineers? Non-existent, as 'the engineers are there to tell us how to do things, it's their fault if it goes wrong'.

DR for the individual bean counters and friends are well rehearsed, as others have said: sanctions against them seem to be totally absent - cuts can be made, "savings" made, yet when it goes wrong, those people are absent.

4
0
Silver badge
Mushroom

Self-inflicted Cyber Pearl Harbor

Always remember the terrible banking holiday of May 2017!

1) BA down

2) CAPITA and appendages down

3) Sainsbury's down

4) Theresa May generates hot air in parliament

"I felt a great disturbance in IT, as if thousands of servers nicely aligned in datacenters were suddenly engulfed in fault reports and red blinkenlights, then silenced. I fear terrible downtime has happened."

0
0

DR and BC are your parachutes.

Who jumps out of a plane without a parachute and a reserve?

1
0
Anonymous Coward

Re: DR and BC are your parachutes.

"Who jumps out of a plane without a parachute and a reserve?"

Good point- has BA got anything in the air to jump out of at the moment though?

2
0
Thumb Down

Still can't access my pension record

So access to my pension scheme website restored on Sunday 28 May. Sadly it seems that none of the security login credentials have been restored in the backend system: "There has been a problem logging you in as xxxxxxxx. Please contact us and confirm as much information as possible to help us find your account (e.g. name, date of birth, national insurance number, login name)". So now I have to wait until Tuesday 08:30 to contact Capita - because obviously they haven't provided any emergency operational cover over the bank holiday weekend to deal with customer concerns. And what are the chances that come Tuesday I'll be able to get through on the telephone?? Have Capita provided any press statement? Have Capita made any attempt to contact scheme members? Of course not. Appalling (but not surprised).

2
0
Bronze badge

Back off holiday tomorrow, hope the team I'm part of has fixed anything we have there.. Long way to go and fix stuff if not!

0
0
Silver badge

From what I've seen when outsourcing happens (school cleaning, meals contracts, payroll etc) the suits in an office far removed from the frontline agree a work and cost schedule based on what they dream the job is. But it's really just a fantasy. As in the cleaning contracts, x number of desks at 4 seconds/desk, Y sq metres of flooring at 2 seconds per sq metre and so on. I'm sure other types of contract are equally unrealistic. But this does not actually allow for how long it really takes to do the job, such as to make the surfaces clean. Things like removal of glue and paint, or the general griminess of places where there are kids. And it doesn't take into account how long it actually takes to get round a room full of normal classroom stuff, or to get from one room to the next. All the details that make actually doing the job in the allocated time impossible.I have no doubt that every contract glosses over lots of these sorts of real details, and that, as in my observations, the voices of staff who know what the job(s) entail are the least trusted or listened to.

2
0
FAIL

Access to pension scheme still not possible Tuesday 30 May 2017

So the website was restored on Sunday. Unfortunately the website failed when it tried to communicate with the system backend and members were asked to call the operational team - but of course the operational team were not working on Sunday and Monday due to the bank holiday weekend.

So I try again on Tuesday at 10am. The website is down again. I call the operational team; they ask for my National Insurance number - they are not aware that the website is down again. I ask to speak to a supervisor - supervisor tells me that the IT team are dealing with an incident caused by a major power failure. Supervisor tells me that the IT team are working to restore the website as the last step in the recovery process. However the supervisor has no timescales for when service will restart. She can only offer an apology.

1
0
Bronze badge

ON the other hand the cost nutral runway option at Gatwick is starting to look attractive, cos even BA can take off from there

0
0
FAIL

Pension scheme issues still not resolved 6 days later

So today the online access website reappears. I type in my login credentials and the website accepts the credentials but when the website tries to communicate with the backend the session fails with the message "There has been a problem logging you in as <username>. Please contact us and confirm as much information as possible to help us find your account (e.g. name, date of birth, national insurance number, login name).". So I follow the instructions on screen only to be told by the poor front line staff that they are aware of the problem and the IT team are still working to resolve the problem. I ask how I will know when the service has been restored - I'm told I will receive an e-mail but probing this a little further the poor front line staff tell me that a communication will be sent to all members who have had problems accessing their pension account in ... er... "a few weeks". When I question this timescale I'm politely told that my "concerns" will be passed on - who to I wonder - some distant black hole in some far away galaxy?. I'm also told that those members who have not had problems will NOT receive ANY communication about the problem - despite them paying the Capita annual management charge for the scheme. Communicating with all members would seem to me the "treating the customer fairly" approach - much vaunted in the Financial Services industry - but this would cost Capita money so there's no chance of Capita going anywhere near this suggestion.

I suspect the only reason that the website has been made available is to avoid the poor front line staff having to e-mail all the scheme forms and documents to disgruntled members - access to these doesn't require a successful login. But perhaps I'm growing cynical in my old age? (as you can see my username here was chosen wisely).

0
0
Facepalm

Teachers Pensions still affected 6 days later

Teachers Pensions Facebook message posted about 09:30 today (31 May 2017)

"While our systems continue to return to normal, we are still unfortunately experiencing delays. Please accept our apologies. We are working hard to try and get the systems working as normal and appreciate your patience at this time."

0
0
Happy

Capita just keep giving...

So now I get an e-mail from Capita's Chief Executive Andy Parker - who in March announced he was leaving Capita:

"I am very sorry for the service experience and difficulties you have had in accessing the Teachers Pensions website. The system issues that recently impacted Teachers Pensions were caused by a failure of the power to re-instate cleanly back from our emergency generator which had been operating as required in one of our data centres following a power outage in the local area, this then caused damage across connectivity and other elements requiring replacement and recovery of systems. Our IT team have worked continuously since the incident with full support from Capita’s senior management to fix the issue and restore all Teacher Pensions services as safely as possible. Our number one priority was to ensure payment systems were enabled to process payments to teachers and this was successfully restored to ensure all payments due to members were made as expected. Our attention then moved to ensure that the website and other services were restored. All services are now restored and functioning satisfactorally [sic].

Please rest assured that all personal data and records were safe and secure at all times throughout this incident. We are undertaking a full investigation of the incident and its root cause, and we will identify any further remedial actions that need to be taken to mitigate any re-occurance [sic].

I have passed on your comments regarding the customer service you experienced and I have asked the business to ensure that lessons are learnt from this and remedial actions taken to ensure customer service is delivered to the highest level at all times.

Kind regards,

Andy Parker

Chief Executive"

Now this would be reassuring but for one minor detail (and a few typos) - I'm not a member of the Teachers Pension scheme!! Just a lowly member of one of the other lesser pension schemes "managed" by Capita. Perhaps Mr Parker was distracted by a bee in his garden?

1
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2017