Events, Alerts, Incidents and Service Ops

A major UK bank with a history of technical issues and outages on both its Mobile and Web banking platforms is in trouble with the regulators, they have been fined and forced to place a large sum of money in escrow with the regulators (£400m) until they can prove that they have a handle on their platforms and more importantly their business services (insert definition here), they have 24 months to do so and any further outages (outside of the usual tolerance of 99.96% uptime) will result in the loss of the escrow head monies.

The Beginning: Year One

When we were approached by the bank, it was quickly apparent that they were in a very difficult place. Not only where they having major platform outages which was affecting their reputation and brand, leading to them being in hot water with the regulators, but they had just annexed off their parent company and were in the process of attempting to stand up their own support teams to pick up/migrate platforms that they had been operating on for the last 15 years. This was, as you can imagine a major challenge full of unknowns, fear and tension.

To make things even more interesting, they had a firm timeline to complete both the migration and satisfy the regulatory bodies of 24 months, any slippage past this timeframe would result in sizeable support/license costs from the recently divorced parent but also the risk of losing the £400 million escrow secured fines to the regulators, enough to almost cripple the bank, this was no longer ‘just the IT project’ but truly a fight for the banks survival.

We must be brave enough to do the hard things!

We started by creating a detailed architecture design for the known ‘end state’, its worth noting at this point that not all the processes and platforms were decided during the early phases of the project so our solution had to remain as flexible as possible while not straying from the golden principle of configuration not customisation (excluding platform application builds) so we didn’t end up in a place where platforms couldn’t be upgraded or became so customised they wouldn’t be fit for purpose in the coming years, this required us to be strong and push back where needed with the bank and understand their requirements and needs on a deep level so we could deliver a fit-for-purpose solution that would stand the test of time.

Next we entered the process mining and enhancement phases, were we analysed the workings of the bank at a sub-atomic level and helped them evolve over a fast paced 4 months with many iterations so they could operate in the new world they were creating, this was no small undertaking and involved upwards of 100 people at one point or another though-out the process across the bank from every department from mainframe to cashier. This was vital to ensure we didn’t build a platform for IT, but one for the whole bank, that everyone could use and understand and that would become pivot for the paradox shift they were after.

By the end of the first year of the project, the bank was almost unrecognisable we had successfully launched the base platform encompassing all the core IT functions, onboarded five of the seven critical vendors, launched build one of the machine learning backed ITOM platform and onboarded twenty of the core forty platforms in time for launch! Departments were starting to use the platform on a daily bases and more than that, they were now trusting the data that was in the platform, sadly though, this is not the full picture.

As good as this sounds, there was however one miss that was going to rear its head and throw a major spanner in the works, that spanner was business change and more directly user adoption and feedback. While it’s true that a number of users were in fact using the tool and were using it well, this was only a small handful of people in ‘fringe’ departments and we needed everyone to come along the journey with us to make the project a success.

This was a critical time in the project and everything that we had spent the last year working on was under threat with the banks board wanting results faster, and in conversations that I had with people around the bank people openly telling me “I still use my excel spreadsheet” something had to be done, and fast. This is where we looked at the goals and priorities that we had set out at the beginning of the project, had we stayed true to them? Well from a technology stand-point it was an overwhelming yes, but we had failed to build a narrative that would inspire the everyday users to drop their old habits and use the new platform. Following this realisation we went hard on the offensive running 20 drop-in sessions a week for people to come by and learn what the project was about, deployed people to ‘walk the floors’ and proactively train users 1-2-1 at their desks, created over 50 training and use-case videos that were hosed online (and buy the end of year two had been viewed over five-thousand times (not all by me!)). Eight weeks in to this concerted effort we noticed that traffic on the platform had dramatically increased as more and more users adopted the platform and embedded it in their daily lives.

The Middle: Year Two

Throughout the project we were working with several “corner stone” providers for Telecoms, Infrastructure, Network and Database support that were all used to operating in their own way, using their tools and providing a “black box” service to their clients. This caused an undercurrent of tension between the vendors, the client and us the implementor as we were fundamentally changing the way that our client, the bank was interacting with them, the norm of the vendor owning the data and exposing only minimal data points to the client was out the window with the bank now owning not only the data but the platform that the vendors were going to be working from with total exposure to the end client, not something that they were used to, or knew how to adopt.

During the project there were a number of times that the tensions bubbled up and we needed to stay strong to our end going and resist the temptation to take the easy way out and allow the vendors to operate in their usual modus operandi as this wouldn’t give the bank the end outcome that they desired or needed to operate at the levels they are required too. One of the major turning points in the project was the completion of the integration between the Telecoms/Network providers toolsets and the ServiceNow platform (in this case the IT Operations Management ITOM Event Management modules which had been configured to use our Context Engines and accompanying rulesets) where we were able to demonstrate significant capabilities, not only in collation of events but the proactive generation of anomaly detection, incident management and major incident management, an area where the bank (and its vendor) had been performing poorly for a number of years. Our product was able to take the timeline for logging a manual incident upon the review of received errors from 6 days to 3ms by using the machine learning capabilities which were native to the product.

We took this demonstration (using real data) to all of the vendors and the transformation in attitude and by-in was palpable, they were able to see the benefits, touch and feel the solution and understand their place in the new world (since then, we have gone on to work with many of the vendors and help them not only with their clients but to improve their own systems, processes and capabilities).