back to article Software glitch led to London Ambulance Service outage – report

A software glitch in the London Ambulance Service’s 999 call handling system, which prevented the system’s recycle bin from being emptied, led to an outage at the New Year – according to an official report. Control room staff were forced to log all emergency calls by pen and paper and pass information to ambulance crews over …

Oracle "Recycle bin"

https://docs.oracle.com/cd/B28359_01/server.111/b28310/tables011.htm

The recycle bin is actually a data dictionary table containing information about dropped objects. Dropped tables and any associated objects such as indexes, constraints, nested tables, and the likes are not removed and still occupy space. They continue to count against user space quotas, until specifically purged from the recycle bin or the unlikely situation where they must be purged by the database because of tablespace space constraints.

7
0
TRT
Silver badge

Re: Oracle "Recycle bin"

"When you drop a tablespace including its contents, the objects in the tablespace are not placed in the recycle bin and the database purges any entries in the recycle bin for objects located in the tablespace. The database also purges any recycle bin entries for objects in a tablespace when you drop the tablespace, not including contents, and the tablespace is otherwise empty."

Not exactly crystal clear.

2
0
Silver badge

Re: Oracle "Recycle bin"

So you ask the database to delete something, then you have to ask the database to *really* delete it. And if you forget to ask the database to *really* delete it, the system will eventually crash as it fills up.

So what's the point of the first delete request?

7
0
Silver badge
Joke

Re: Oracle "Recycle bin"

So you can decide if you want to commit

0
0
Silver badge

Re: Oracle "Recycle bin"

"So you ask the database to delete something, then you have to ask the database to *really* delete it."

Or maybe really, really delete it. Or even really, really really delete it.

My brief encounter with Oracle simply left me with the feeling that it was thoroughly obfuscated. I'm glad I was able to make my living with saner alternatives.

5
0
TRT
Silver badge

Re: Oracle "Recycle bin"

I got about 16 words into that piece of documentation and the voice in my head turned into that of Sir Humphrey Appleby, GCB, KBE, MVO, MA (Oxon).

9
0
Silver badge

Re: Oracle "Recycle bin"

We had that in dBase. I've seen people in the 21st century talking about "soft delete". In dBase, you could delete a row (record) from the data table (file), then change your mind and "undelete" it. You could even browse through the data with all the deleted data visible. I think the command to really delete it is "pack". And then it still probably was liable to be left on the disk, in space uncommitted from file use. But I think we also wrote a sub-routine to replace all the fields with blank space, to erase one record - or maybe it was a built-in command of "REPLACE BLANK". Then you'd delete it, maybe.

1
0

Really, "glitch" was it ? or did they just run out of tablespace ?

I love the Appendices in the report - I think "Empty the Recycle Bin" is my favorite action closely followed by "Disable the Recycle Bin functionality"

8
0
Silver badge
Coat

Mmm... Puts a whole new slant on "whose turn is it to empty the bins".

8
0
Anonymous Coward

Wow

I've never seen the recycle bin enabled on a production Oracle database before.

The outage was caused by incompetence.

13
0

Re: Wow

Exacly...

5
0
Anonymous Coward

Re: Wow

And surely the disk space monitoring would have generated timely alerts that there was a problem so that the incompetence could be caught before there was a system outage?

2
0

So the problem was a simple Oracle configuration issue and their solution is to employ two new top level bosses? Surely employing one Oracle expert techie would be cheaper and far, far more effective.

17
0

If you didn't have any IT expertise at the top, they may well not have realised that you need a decent Oracle DBA to run an Oracle database.

Good Oracle DBAs aren't cheap as Oracle is pretty complex.

3
0
LDS
Silver badge

"Good Oracle DBAs aren't cheap as Oracle is pretty complex"

So once you hired the new bosses no money are left to hire competent DBAs, and monitor the system proactively to catch issues before they become deadly.

7
0
Silver badge

Re: "Good Oracle DBAs aren't cheap as Oracle is pretty complex"

So once you hired the new bosses no money are left to hire competent DBAs, and monitor the system proactively to catch issues before they become deadly.

But at this stage you have a really good handle on the problem and can produce an accurate risk register and apply for more budget to recruit a cheap dba and another senior manager to maintain the ratios

3
0
Silver badge

I particularly liked the way their fourth review was to review what they should have learned from previous balls-ups and didn't.

4
0
Gold badge
Unhappy

At first I thought they were talking about the Windows recycle bin, but that made no sense,

as I've never heard of anyone having any trouble with it (just about the only piece of Windows I haven't heard of some problem about).

But this Oracle whatsit is a whole different ball game.

4
0
WTF?

I wonder just how old the servers and software actually are ?

These two recommendations made me cringe a bit:

R19 Update the CAD servers to a modern, supported platform that replaces the existing Itanium servers.

R20 Update the Oracle version to current (or one below as a default).

So probably still on Oracle 10g (10.x) if not earlier, but HP-UX or OpenVMS ?

0
0
Silver badge
IT Angle

Software glitch caused London Ambulance Service outage

Well that explains everything, it's this kind of technical reporting that I come here for :)

1
0
Pint

Re: Software glitch caused London Ambulance Service outage

I'll drink to that, is it Friday yet?

2
0
Silver badge
Thumb Up

R E S P E C T

I, Hans 1, also would like to "pay tribute to [the] staff who responded incredibly well, ensuring patients continued to receive care during our busiest time of the year.”

Above all, I think that these guys/gals deserve the greatest respect of all, even with a failing IT system, I am 100% sure they tried their best to get patients treated!

Thank you!

Love from hans1

Microsoft MHP

Adobbe MHP

2
1

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2017