Ulster Says “No Money!”

This Ulster Bank debacle is a weird turn of events, isn’t it?  The one remaining bank we thought we could rely on decides to run a software upgrade and promptly goes feral, refusing to credit people their money, but still taking the mortgage payments out of their accounts.

Now look.  I know what they’ve been saying.  This is a glitch.  Shit happens.  Sorry about that.  We’ll probably have everything working fine Friday week, maybe, possible, perhaps.

Ulster Bank, like every other organisation in the world, has systems running on ancient code and from time to time, they need to fix a bit of that code because they found out it doesn’t do something properly.  That something might be very trivial, like not putting your initials in capitals, ot it might be more serious, like sending all your money to a central African dictator by mistake.  One way or another, it needs to be fixed.

This is not unusual.  Any computer program has ragged edges, because when programs get that complex, nobody can say precisely what they’ll do in every conceivable situation.  It’s just not possible.  That’s why big companies are always poking and prodding at their systems, fixing problems as they crop up.  Programs that evolve over decades are going to get flaky from time to time and that’s just reality.

In this case, however, it seems that Royal Bank of Scotland, the owners of Ulster Bank, decided to add a minor patch, but somebody screwed up monumentally, and when they installed the additional code, it made complete shite of their system.

Good.  Take that, geeks.  This is the end of drawling, condescending knobheads trying to make me feel like an idiot.  Go back to your pizzas, your World of Warcraft and your Slayer collection.  We now know that you can screw up just like everyone else.  Everywhere you go these days, some tool who speaks only Unix is trying to make you feel like an idiot for asking a perfectly reasonable question, or at least what would be perfectly reasonable to anyone except a guy with no life outside first-person shoot-em-ups.

At the same time, RBS needs to be looking hard at itself.  Did it have no Plan B?  Did it simply fail to understand that a software collapse could be catastrophic?  When the textbooks are rewritten, I think this will be presented as an example of how not to recover from a disaster.  I’m no management twonk, but even I know that a basic principle when you completely screw your customers is to come out with your hands up, apologise, explain how you plan to fix the problem  and promise never to do it again.

Have RBS or Ulster Bank done this?

No.  They didn’t write to their customers.  They didn’t email them.  They didn’t phone them.  They lied about the extent of the trouble.  They made a feeble effort to help by opening a small number of their branches for a few extra hours.  They set unachievable targets for fixing the problem — targets they won’t be able to meet.

Ulster Bank and their owners RBS, have been a classic example of corporate arrogance, and I think they’ll pay the price as thousands of customers close their accounts in coming weeks and move their money to banks that have operated properly.

I think everybody should move their accounts to, eh, AIB, or eh, Bank of Ireland, or eh, TSB,  or …

Ah fuck it, leave your money in Ulster.  They might say No, but they won’t say No Hope. 

 

 

23 thoughts on “Ulster Says “No Money!”

  1. Could the minor patch update not have been tested first?
    What about a recovery plan in the event of a problem? Reverting back etc.
    Standard stuff really.

    Funnily enough I usually end up doing the drawling at geeks… usually as they’re not answering what I asked, or not doing what’s actually needed.. not understanding the problem. Geeks can be a bit thick like anyone else really.
    As well as that, there’s an awful lot of them that don’t know their stuff and try to fob you off with bullshit. Someone fucked up here monumentally alright

  2. “Good. Take that, geeks. This is the end of drawling, condescending knobheads trying to make me feel like an idiot. Go back to your pizzas, your World of Warcraft and your Slayer collection.”
    Ah, now. As a professional software developer, who makes his living by knowing *exactly* what the software I write will do in any given situation, I have to disagree. Plus I don’t play WOW or listen to Slayer. More of a Trivium man, although I do enjoy the odd pizza. And I am hairy. and have a (kind of ) beard. And beer belly. Hmm, not really making a good case for myself here am I?Anyway, I think that anyone who calls themselves a geek but produces software or designs/maintains a system where there are unknowns is not a geek, they are a fecking eejit.
    The unknown/unquantifiable is anathema to any proper geek (quantum physicists excepted here). If you do not know the answer to a question, you make it your business to find out…you will probably find that the set of eejits and the set of condescending knobheads has quite a large intersection :) Personally, it’s been my experience that the less a person actually understands how something works (here I make the distinctions between someone who can rattle off all Unix commands (knob) vs someone who knows how they work but might not actually remember them all of the top of their head(geek)), the more dickish they get with people who can’t understand their explanation…
    Personally I believe that there is nothing in for example software, no matter how complex, that cannot be explained to in simple english, perhaps with an analogy here and thereI reckon that this is some kind of defence mechanism that triggers (terrified someone will expose their lack of knowledge maybe) – I find this kind of attitude not just in software but in the doctor’s office, schools, churches, etc.

    On topic, The word on the street is that this fuckup was caused because of plain old incompetence, as a result of RBS letting go so many experienced IT staff (who knew how to solve issues like this) and outsourcing the jobs to India….RBS deny this but it’s a fact that they left go many “lifers” in the last couple of years and this particular fuckup was caused by inexperience (the software upgrade having failed, an experienced operator would have known how to “undo” the botched job without deleting the queue of pending transactions which is what happened) So it was not the software per se, it was the cockup of some poor sod who in fairness probably should not have been doing this job without someone more experienced on hand to help.

    Interesting articles here:
    http://www.theregister.co.uk/2012/06/26/rbs_natwest_ca_technologies_outsourcing/

  3. lady in the bank told me yesterday, that payments into and out of peoples accounts are being “bounced back” to who ever is makng or looking for payment.
    seems the problem first occurred with Natwest banks. this is also part of the RBS. think it took place a few weeks ago. what if any action was taken to ensure it would not spread to other parts of the bank i do not know.1000 IT jobs had been transferred to india, and i wonder how big a part this played in the time its taking to resolve.
    the whole thing could be a basis for a movie bank heist plot though. no one knows exactly what their bank balance is, it can not be checked easily, and this mess won’t be sorted out for at least another week.plenty time for a genius geek to transfer large sums of money without it being noticed.

  4. Steve — i wholeheartedly agree with you. Unfortunately though, the insecure knobhead is everywhere. Even dare I say it, in my own hosting service, which I usually find very professional but which has a shit interface and a very talking-down-to-you sort of geek when you contact them with a problem. Still, on balance they’re much better than the rest.

    I also agree that the Ulster Bank / RBS/ NatWest problem is caused by dumbing down the system and putting administrators in charge instead of knowledgeable professionals. This happened in our public sector about ten or twelve years ago, and look what we got.

  5. @Steve – +1 about the Geeks v knobs.

    On topic – I don’t believe it was an issue with a release backout. If it was they would have failed over to COB and there would have been minimum impact. Also – deleting your tnx queue isn’t going to cause a 1 day delay in all payments. Unless they got their systems back to BAU and all the tnx re-input in < 24hr. And if they had done this then the issue would be "fixed" – at least to the client. Nah – there's something else here. No idea what, but it wasn't just a f-ed up backout.

  6. @Ron, as far as I know there was a standard procedure for backing out of the failed upgrade to return to Business As Usual but the person doing so apparently fucked it up in unexpected ways. I imagine they may have been presented with a confusing choice (“Are you sure you want to not cancel; this will maybe not undelete previously deleted data; OK/Cancel/YES”) and made the wrong one, but then again I don’t know if that’s actually what happened..I think the main point is that whoever was doing it was not experienced enough, and didn’t have the support available to them, to correctly recover, this is the real problem
    I reckon the deletion of 24 hours of pending transactions probably then caused a snowball as they struggled to fix the error with more transactions backing up – and it’s also possible that the system ended up in a state where they had no option but to re-commit the original update (example of what can happen – the update changes the database definition but the fucked up backout doesn’t restore it; now your existing transactions can’t even go into the “new” database and the red tape around allowing a manual hack to fix it takes weeks, you have no “correct” backup of the original database because the poor sod doing the upgrade performed it whilst the last backup was in process without realising, that might even be the reason for trying to abort the upgrade in the first place, etc, etc.) and now have to maybe manually or using some older backup system, send the queue through for processing. Then again, I wasn’t there (honest!) so I don’t know…
    @gerryo – now that you mention it, around the same time as this there was also that thing about the massive online bank heist (google it)…hmmm…..

    @Bock, who does your hosting? The other thing about the outsourcing/downsizing has kind of struck a nerve with me (not for personal reasons). I worked for a long time in the call centre industry and I saw the whole cycle of outsource-fuckup-bring it back many times…now we see it in the IT sector and even the medical sector..for a small short-term gain, what we’re actually doing is training other countries to compete with us. And since in software, one of the biggest assets is manpower (very few genius coders actually exist in the industry and what tends to set one app above another is the amount of time/people you have to polish and rework it), we’re starting to get creamed by the Indians/Asians at our own game…yet we have loads of them in this country already, working in Petrol stations or takeaways and willing to settle here, but hardly anyone will give them a break because they’re only here to steal our jobs….pretty insane to me!

  7. The Other Ron — I’d appreciate it if you’d write your comments in language everyone can understand instead of exclusive nerdy shit that impresses nobody. Thanks.

  8. That’s a bit OTT Bock. I have no idea what “BAU” means but the rest of Ron’s comment was clear enough.

  9. You’re right. I am being a bit harsh, but I don’t know what BAU means and I don’t know what COB means. I also don’t know what a tnx queue is.

    Jargon is fine among people who share it, but when it escapes into the wild, it becomes a means of bamboozlling others, which is often the case with geeks.

  10. I think COB stands for close of business Bock.
    I was bamboozled .. but not for long!!

    Steve, that was some hypothesis!
    Even that one has me perplexed –
    “I imagine they may have been presented with a confusing choice (“Are you sure you want to not cancel; this will maybe not undelete previously deleted data; OK/Cancel/YES”

  11. COB=Close of Business, BAU = Business as Usual (had to google it in fairness and pick the most likely). tnx queue I guess means Transaction queue.
    @FF1, the correct answer to that question is the hidden fourth option “Fuck you computer, I’m going home” :)
    To be honest I could be totally off the mark with my theories and was only speaking from personal experience of the kind of things that can cause what should be a simple recovery into a total fuckup…..you’d be amazed sometimes at the way big software packages that are relied on by the likes of banks, suck really badly at giving information to the user about what’s going on – they tend to assume that the person pushing the buttons already knows what’s happening behind the curtain, so the error messages and choices they give can often be really confusing to the uninitiated (example : http://thedailywtf.com/Articles/Unscheduled-Programming.aspx – look at the 4th one about the Oracle installer – this is a message from a piece of database software that costs a *ridiculous* amount of money and is used by more than a few banks)….can you imagine being the poor fucker seeing this message in the wee hours of the morning with your manager breathing down your neck to “just get it done”!!!
    The key point being that had they retained their experienced staff, we’d probably have never heard about the problem.

  12. While UNIX at code level is complex and not a bit intuitive, it is industrial strength & reliable when compared to ropey windows. The standard Bank configuration therefore is a UNIX O/S with Oracle database customized on top of it. Oracle is the preferred bankers choice for two features: The two sided handshake nature of any transaction whereby money exchanges do not take place until the giving and receiving are both completed correctly. This is why an ATM may sometimes go through it’s motions, but then spit out your card and not the money and switch to out-of-order mode. Your account is not changed because the money has not left the machine so the transaction is not complete. The other reason Oracle is favored above all others is its facility to automatically back-track during failures to the last proper transaction and start again. It takes an old hand to reset it, but in the right hands, it should be rolling the same day and working again.

    That is why I tend to agree with the opinions above. Some underpaid moron in sunnier climes went at, made changes, by-passed failsafe and panicked. The machines are neither vindictive, stupid not intelligent. The human input however, can be any of those three !

    Happily however, I’m not an Ulster customer and our Indians have not fucked up yet !

  13. @John, funnily enough, I use Microsoft SQL Server all the time and it also supports distributed transactions whereby a failure in one part of the transaction (i.e across multiple servers, do this, then this, then this then that and that and if any of those steps fail, the whole transaction fails and the information being updated on each server is changed back to what it was before the transaction); the last project I implemented handles nearly 100,000 distributed transactions per day (not financial, but the same principle, add another 20 servers to this mix and you have the exact same ability as an Oracle system). Oracle only has such a foothold in the banking/financials sector because it was one of the first database systems suitable for such large scale deployments, and the banks simply can’t turn it off long enough to change over to anything else…If I was starting up a bank in the morning, I’d be going nowhere near it :)

  14. “The key point being that had they retained their experienced staff, we’d probably have never heard about the problem”.. I agree Steve.
    You see a lot of companies outsourcing to India and it being a disaster.

    It’s greed, greed and more fucking greed as the Irish guy says on the youtube.

    Ulster Bank’s parent company is Royal Bank of Scotland right?
    They’re facing massive fines at the moment for fixing bank rates in the U.K.. I can’t understand the salaries the Bank bosses are compensated with either.

    Another thing, if the banks are in cahoots fixing rates and it’s illegal, why isn’t the punitive action prosecution and prison? Why fines? Especially if the tax payer end up footing the bill, if they are majority state owned. It’s a joke..

  15. Steve, Thank you for that update, but to honest, anything to do with Microsoft Corp, SQL Server included, almost must be unreliable. They have taken years to build up the reputation for ropey software and it is deserved. But, I preface this by admitting I am no longer actively involved at that level, so maybe you can assure us things have changed. As regards Oracle, it has its drawbacks, but it does have a proven track record and is regularly updated.
    But, technology aside, I can’t see beyond human error in the bank fiasco.

  16. Completly agree Fox-Force , this ‘Barclays’ Diamond character forfeiting his huge bonus rather than being locked up for his crimes and we’re all meant to be satisfied with that….what have they learnt ..? Some Pigs are still more equal than others and always will be it seems….fkin sad really .

  17. UB will have to consider offering me a larger deposit interest rate from now on to justify me keeping my ‘spondulicks there! Think I will stomp in there and demand an interest upgrade when this debacle has passed.

  18. @Bock – Sorry i was away for a few days – only seeing your comment now.

    Sorry about all the initialisations / abbreviations in my post. I was just bashing out what was in my head without too much thinking.

    CoB – Continuity of Business – basically a full like-for-like set of servers / applications hosted somewhere that you can direct all your production traffic to if something happens to your production set up.

    “txn” is just a weird abbreviation for “transaction”.

    BAU is Business As Usual.

    Finally, seeing as this is still on-going (according to the Irish Times today) whatever credence I gave their original explanation (not a lot) is further diminished.
    Unless they just didn’t have a Continuity of Business environment which would be mind boggling.

  19. Yeah – exactly. Just fail over to it if something bad happens to your primary set up.

  20. So it’s looking as if they had no backup?

    Could this mean that they simply don’t know what transactions went missing?

  21. Interesting clip here from Channel 4.
    There is mention of ‘disastery recovery exercises’…
    http://www.channel4.com/news/rbs-warned-of-it-outsourcing-risk-in-2010

    With so many transactions, they’d need continuous replication. There’s a lot of articles you can look up to find how that’s done.
    E.g. mirrorview, sancopy, snapview, replication manager, clustering..
    RAID

    Continuity of Business environment is business spiel. Continuity of bullshit.. :)
    “If it was they would have failed over to COB” There’s no such thing as failover to continuity of business Ronald.

    Thorough explaination here also from ‘BrianOz’
    http://www.webhostingtalk.com/showthread.php?t=350268

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.