A programming mistake or was it a case "Ah Mr. Corleone, I gather sir is interested in one of our 'discrete' accounts.... yes sir, we call it the 10-alpha account. It's much more convenient than Panama"
A programming blunder in its reporting software has led to Citigroup being fined $7m (£5m). According to the US Securities and Exchange Commission (SEC), that error [PDF] resulted in the financial regulator being sent incomplete "blue sheet" information for a remarkable 15 years – from May 1999 to April 2014. The mistake was …
No, you come at the grand total by a different route, checking at a different point.
For example, every time a transaction actually happens the stored procedure which implements it (and does all the security checks, and cannot be amended except with a *lot* of safeguards) also adds the amount to the grand total. There's various other possible ways.
Yeah, that's what should happen in the FMIS. But first, you're talking about reporting, not accounting, which although it uses some of the same data, doesn't do the checks that accounting does. You might have the reporting function do double entry checking, but why would you when the FMIS does that.
Secondly, you're talking about a bank that implemented a new coding system and didn't bother checking if it worked properly. For 15 years. That kind of precludes them from even thinking about doing a redundant check on the reporting data, let alone being able to implement it without sinking 30% of the world's cargo ships at the same time.
Who you calling an EBCDIC!
(Just kidding of course). Having seen something very similar happen at a large 3 letter NGO based in the land of gold and chocolate... I can see how this sort of stuff happens when code gets outsourced, then offshored and little bits of knowledge get lost along the way and then one day. BAM.
Twice in my career I've found such errors when inspecting code trying to debug it and finding serious errors in financial calculations. Both times the project management were in shock and took remedial action but hushed it up from going higher up the chain out of embarrassment.
Once when contracting to Ericsson AU - many years ago - I found a similar error... then mentioned it to my supervisor (very old lady near retirement, reporting to the CFO).
She told me to ignore it, and not mention it to anyone. Which I thought was weird.
I mentioned it to the CFO later on (as a young, dumb, clueless person does) who was shaken at the time.
Needless to say, I was immediately fired by my supervisor. Who naturally, probably had a good excuse to the CFO and never needed to follow up it.
At least, not in the few months until she retired. :/
Thankfully, I've learned to be more discrete over the years when finding corruption and things of similar ilk.
a) Document things properly
b) Have (distributed) backups, that others will access
c) Have a chat to the corruptor. I can/perhaps/maybe overlook the problem... but what do I get out of it if I do? (document this)
d) If possible, find someone external to report to. Else, Wikileaks/similar.
I'm thinking perhaps the original programmers didn't know or had forgotten about that and assumed ASCII expecting strcmp() or a similar string comparison function to work ok. Either that or they never expected letters to appear in those 3 positions. Either way, clearly their testing procedures sucked.
> The reporting logic treated letters, such as A, B, C, as being less than the number 0.
That seems unlikely to me. I'd believe it if they said they excluded anything over 89.
That or if they said it was a Y2K/Windows 9 type thing where it would look if the first two digits were 9 or 10 and skip it.
Almost there. The number of non-reported transactions was close to 27,000 ; the number of requests (that is, how many reports were created) is 2,300 . Meaning on average each report missed nearly 12 transaction. That's tiny but as you correctly note, there might (or might not) be large sums involved, or other signs of "noncompliant behaviour" which were effectively hidden from SEC.
What it does further highlight is how ineffective and useless their regulator(s) are.
For 15 years they have been receiving (albeit unintentionally) falsified information and haven't spotted it. Not even remotely acceptable. If they don't notice that without being prompted then how are they going to spot deliberate wrongdoing being covered up? What are their regulators actually regulating?!
"Banks manage thousands of transactions every minute."
That's transactions. I'm talking about 3 official requests from the government for transaction REPORTS every week for 15 years. Which to me seems a lot - ie - did they think Citi was up to no good and was keeping a close watch on them? Or is that amount just par for the course for banks?
It is completely irrelevant if it is a "lot" or not.
How would you feel if a bank 'lost' any of your transactions for reporting.
Let us not forget that it is the monitory value not just the transaction count that is the problem.
Sorry... we forgot to report this 1 billion $ transaction....
"How much do you guys want to bet that someone or some group within the company made more than 7 million bucks out of this?"
I wouldn't bet much at all. These were just normal transactions that due to a bug didn't get included. There is no reason to think any of them were suspicious or the bank made any extra money out of not forwarding them to the SEC. I know the banks are hardly shiny white , but in this case it clearly looks like cock up , not corruption.
Sometimes the only way to be certain that the production system actually performs in the same way as the development system is to exercise it with the same data. Without that check it is possible for the production system to have an unnoticed coding difference from the development system (Yes in theory this should never happen but in the real world it does happen that a patch fails to make it from development to production.) It is also possible that size differences between the development system and the production system can lead to errors (eg if there is a fixed size array that overflows) as in many companies the development/test systems are not as large as the production systems.
"I would question what "testing" data is doing in a production system?"
It wasn't. The software was written to filter out a certain range of codes that was used for testing. This is perfectly normal, eg certain credit & debit card number ranges are for test purposes only.
They got away with the crime for 15 years, probably made many millions out of it, and now only have to pay a piddling fine that amounts to only a few hours of income, with no other negative repercussions.
Could someone explain to me again why the people who run corporations would voluntarily follow laws that reduce profit if this is the only kind of "punishment" they're likely to get? Or am I right in claiming that the laws of this planet are only applicable to those who don't have the money to buy those who make them?
'They got away with the crime for 15 years, probably made many millions out of it, and now only have to pay a piddling fine that amounts to only a few hours of income, with no other negative repercussions.'
A mistake, not a crime, perhaps a mistake that should have been detected earlier, but unexpected consequence of changes to systems are some of the hardest things to test effectively.
There is no evidence that they made any money from this at all.
Embarassing -yes, evidence of incompetence - maybe, criminal - no.
Dear A.C. and the downvoters (there's a band name in there...): Before you suggest that repeated errors aren't criminal, I suggest that some day you make "mistakes" reporting your taxes. For 15 years. Then see how law enforcement deals with you.
I'm starting to wonder if this really was the "mistake" they claim it was.
<quote>They got away with the crime for 15 years,</quote>
I feel the real crime was in not promptly reporting the issue to the SEC, and for that reason alone, they should have faced a higher fine. It would be nice to extract the fine from whoever sat on that decision's paycheck/pension/golden parachute.
Please allow these assumptions (and I do know the ALTERNATE spelling of "assume"):
1) The original coder never expected branch codes to have an alphabetic character in them. (In hindsight, one should not have been allowed to create an alphanumeric branch code.)
2) WHOEVER decided to implement alphanumeric (10A, 10B, etc) branch codes did not run that decision past the IT staff responsible for maintaining the code. They had no clue to what someone in Ops was doing.
3) No one ever bothered to test the code once the alphanumeric codes were put into use because of 1) and 2) above. This is the reason why the time span was measured in years.
4) Most likely (and I am assuming here) when the discrepancy was discovered, the shit hit the fan. The logic mistake was fixed pronto.
What should have happened was CGMI immediately informing the SEC, and CGMI immediately provide corrected reports.
BUT some corporate asshole sat on the decision, and hoped it would go away. Well, it didn't. CGMI may have come out better if the "optics" of the incident were better. The "optics" being:
1) they fucked up royally,
2) they quickly fixed the fuckup,
3) they provided only a partial correction of the misleading reports,
4) they sat on reporting to the SEC for months which imparts the stench of "coverup"
IM(NS)HO, who ever at CGMI 'sat' on the decision NOT to promptly inform the SEC should be sent on a new career path without any "parachute" (severance pay).
NewsFlash :- New Secret Tax Haven found
"CitiGroup revelations show the holiday resort of 10B in South Wales to be a secret hideaway for gazzillions of dollars. This also came as a surprise to the Bank which had told it's clients the money was safe near the warm sunny sub tropical beaches of New South Wales in Oz."
Nearly 40 years ago, I used to do regular preventive maintenance on some of Citibank's VAXes in London. For years they followed a standard backup routine, copying all their (presumably valuable) data onto reels of magtape which were then filed away in a special archive room. There were hundreds of reels in there.
One day it was discovered that the backup procedure had been wrong from Day One, and not a single one of those tapes contained any useful backup data. We didn't hear which heads rolled, but presumably some did. It sounds as if the culture has not changed a lot.
Yup...also been there, this time during an AS/400 upgrade. We needed some stuff off a recent backup. It turned out that the backup was corrupt. It also turned out that no one had ever tested the restore process, and that all the carefully taken backups were unusable!!
Lesson: Do the backups....but test the restore process too.
AS/400. That takes me back. Working at a large UK based insurance company that's only accessible directly and not through any price comparison sites, we had 3 AS/400s that the European breakdown system ran on (this insurance company also has as part of it's group a breakdown recovery company).
They were named after members of the 1980's royal family and doing the backups was a royal PITA as it meant everyone using the system had to log off or be forced to log off. The consoles were perched on top of the cabinets so we had to climb one of those kick stools to get to them. Wouldn't be able to do it now but it wouldn't surprise me if they were still in use.
To those of you frothing at the mouth about an evil corporation making a huge profit out of breaking the rules, I suggest you re-read the article and the report. It doesn't say anywhere that anyone made any profit, they just failed to correctly report on a relatively small proportion of trades completed by their customers. That reporting was just used to help the SEC in monitoring and investigating problems. Not reporting it didn't help that monitoring, but it also didn't directly hinder it.
They've also subsequently identified and reported those trades, and the fact that there's no mention of them being problematic presumably means they were run of the mill trades, and it wasn't hiding anything untoward. If there were more to it, there'd be a fraud investigation, not a paltry fine for incorrect reporting.
> To those of you frothing at the mouth about an evil corporation making a huge profit out of breaking the rules, I suggest you re-read the article and the report. It doesn't say anywhere that anyone made any profit, they just failed to correctly report on a relatively small proportion of trades completed by their customers. That reporting was just used to help the SEC in monitoring and investigating problems. Not reporting it didn't help that monitoring, but it also didn't directly hinder it.
This is all true. But, if the bug affects all reporting from these branches, then any money laundering going on through those branches also would not be reported up for investigation. So the question is: did someone spot the bug but leave it in and sell the knowledge onto money launderers? Probably not, but until the SEC demand the historic reports be re-run we'll never know.
Although it doesn't say it explicitly, the report does say "On April 23, 2015, CGMI finally uploaded to SIAC the data omitted from the 2,382 EBS submissions to the Commission", which to me sounds like the missed data has been submitted. Maybe it's yet to be analysed, and then maybe a fraud case will come out of it, but the implication is that there's currently no suspected naughtiness.
When I was getting close to retirement, I took the time to explain to one of the younger workers just how a rather intricate statistical analysis worked. She didn't pay real attention and I later heard that ultimately, when they had to rerun the analysis on revised data, it took them two weeks to figure it out - about the same amount of time it took me to figure it out in the first place.
The thing that's mildly remarkable - but not surprising - is that this analysis was key evidence in a long, drawn out legal proceeding. Management never even whispered one syllable about the need for knowledge transfer.
I like to think some kids with Raspberry Pi's and a couple of degrees of moral compass will get jobs in places like this and actually know how to do JOINs on the odd table or two and start chucking up anomalies. If someone at the SEC notice that Citibank had noticed and fixed the problem without letting them know the $7M would look like very very very very small change.
That's interesting ! This is why there is QA which is seen as a necessary evil and ignored, certainly at my company. But hopefully this will carry on, because it is very expensive and the developers make very few mistakes normally. This is why my company (Wirecard) is planning on going into business with Citigroup selling prepaid credit cards in the USA, some of my best people are on it. With the money laundry that we are already being accused of, the BAFIN and SEC will not know where to start !
well, this is like Madame Clintons classified email server. "I didnt intend to do that all, im sorry please forgive me!" NO, NO nO! being a simple programming oversight (lost in the tomes of ever present code revisions, tweaks, crashes and change of personell?) a nice hefty FINE of a few million bucks could placate most any USA agencys (seeing as their life blood is essentially the flow of MONEY). as for the security (national defense, financial balance, regulations and other trivial issues), the taxpayers get the general drift that making mistakes will end them in prison, while the fat cats get their fat wads of cash nibbled and shared among the others.
Biting the hand that feeds IT © 1998–2019