Now that AppSphere 2016 is over, I have been reflecting on the track sessions that I managed and
the ones that I watched at the event. While all of them were very good, there are a couple which really stand out for me. One of these was ‘Standard Bank’s: Agile, DevOps & the Role of AppDynamics’ case study. It was delivered by Andrew Kretschmer, Executive Head, Digital Channel Development and Lenro Meyer, Application Development Manager. What really stood out for me was their story (great slide quality also), honesty, and the benefits derived from AppDynamics’ software. So let me say, thank you guys! Here is a quick breakdown of their session in case you missed the presentation.
A Bit of Background into Standard Bank
Standard Bank is one of South Africa’s largest financial services groups, with 14.9 million retail banking customers, 1,221 branches and 8,000 ATMs. Established 154 years ago, it’s no surprise that today’s IT environment is a mix of new and legacy systems. Andrew explained that their strategy is driven through digitalisation of business processes and a mobile, self-service-first mentality. Ultimately, their enterprise vision is to be the first leading financial services organization in, for, and across Africa, delivering exceptional client experiences and superior value. From the stats highlighted in the slides this vision is certainly being realized with the number of transactions rising significantly in regards to online banking and their SBG mobile banking app (in fact transactions on these platforms are up 387% according to the last company report!).
What Digital Challenges Did They Face?
In short, a number of digital challenges, but in summary:
- IT spend was ~30% of operating costs, when best in class is 15%. This spend made it difficult to allocate budget towards digital innovation. Standard Bank’s unit cost per a feature on an application or service was 3X higher than that of their peers.
- Digital acceleration changing customer expectations. Mobile and web were changing the way customers wanted to bank, but unfortunately it took 300+ days to deliver features into production with many outages. This resulted in low customer satisfaction.
- Developers treated as second-class citizens. This was a pertinent point for me. In February 2015, Lenro explained that their mobile app suffered a severe outage and while his development team were pulled into a crisis room to find the root cause, they were treated as second-class citizens. It actually took them two hours to convince stakeholders that they needed access to the production environment to find the root cause. This coupled with a lack of adequate monitoring capability, led to the outage lasting a day and as Lenro explained to their CIO, “If we continue to treat developers as second-class citizens then this will happen again.”
Was Agile Software Development the Silver Bullet?
In order to solve these digital challenges, the first strategic initiative was to move towards Agile software development practices. But as Andrew explained, while they did try to adopt processes such as having daily standups, etc. this initiative ultimately failed. The reason was simple: You can’t just implement a new process or operating model without changing the culture.
Therefore the solution was to transform to a DevOps operating model, breaking down silos between teams and ramping the number of developers from 12 to 250 over the course of 3 ½ years.
AppDynamics Chosen as a Strategic APM Solution, Key to Driving DevOps Adoption
On the back of the mobile app outage and as part of their DevOps adoption, Lenro began to evaluate their monitoring strategy back in 2015. Initially this was hampered, because as a developer (a second-class citizen) he only was given limited access to existing monitoring tools but by doing research online he identified that a modern, enterprise APM solution was required.
His seven key requirements can be seen below:
After trialing a couple of solutions Lenro and the team at Standard Bank concluded that AppDynamics APM was the only solution on the market to address their key requirements. AppDynamics offered an enterprise, intelligent APM solution showing end-to-end visibility of performance in real-time, automatically and easily.
Three Major AppDynamics APM Benefits
Standard Bank have only been an AppDynamics customer for a year but Lenro outlined three major benefits from their adoption:
- Remove quality impediments in software rapidly. Lenro explained that AppDynamics has been used on Standard Bank’s internet banking platform in Nigeria to remove quality impediments. The situation was that logins were taking 30 seconds and because the internet banking platform follows a layered or tiered architecture approach, it made it extremely difficult to identify the root cause. Within minutes of deploying AppDynamics it gave developers immediate visibility into the layer causing the issue (in this case the core banking platform) and the exact line of code that was the root cause. This means that logins now take less than three seconds. Lenro explained that this rapid visibility means that developers can now focus on building new features rather than hunting and scouring logs to find the culprit of performance problems.
- Prevent outages and reduce the time to fix production issues. In this use case, Lenro explained that they had made changes to the core banking implementation. As soon as the different channels that rely on core banking were started up, AppDynamics started triggering alerts in regards to slow transactions. Using AppDynamics, Lenro and the team were able to immediately identify that it was one node in a four node cluster that was experiencing a 25% call failure. The impact was that potentially 25% of customers using this channel could not check their account balance. But because AppDynamics gave real-time visibility into the cluster, it gave the support team the awareness to remove the node and repair it at a quieter time. This meant that banking services did not experience any issues and helped the support team to maintain their Mean Time Between Failures (MTBF) metrics. Additionally, Lenro talked about an Internet Banking outage in April 2016 when it took only five minutes to identify and the resolve the issue. Before AppDynamics this sort of outage would have resulted in teams being pulled into a war room which would have lasted for two to three hours a minimum before finding the root cause.
- Enable feature teams to do DevOps. One of the main drivers for DevOps adoption was to ensure that teams spent less time on support and more time on feature development and optimization. This is now starting to be achieved with Dev, Test, and Ops professionals all using AppDynamics to give them visibility into how code is performing and to identify areas that need improvement before the business and the customer is impacted. As AppDynamics is used across the software development lifecycle it means that better quality code now goes into production. Today Dev, Test, and Ops are all first-class citizens.
Standard Bank’s Tips for AppDynamics Adoption
Lenro finished the presentation with a number of key tips for successful enterprise adoption:
- Automate as much as you can. At Standard Bank they use Chef to automate the deployment of AppDynamics agents and controllers. This means it’s quick and easy to deploy AppDynamics into new channels or parts of the business.
- Put AppDynamics in the hands of the right people. Don’t centralize the deployment of agents but let dev and ops teams deploy them themselves. This speeds up adoption as dev and ops teams know their application best.
- Prepare for the storm of adoption. As soon as teams see the value that AppDynamics can provide – they will want it! So make sure that you manage demand and can structure your rollout based on business importance.
Standard Bank are continuing to rollout AppDynamics and to ensure adoption in the right way but one thing is certain: AppDynamics has helped Dev, Test and Operations teams to become first-class citizens within the enterprise.
Again, thank you Andrew and Lenro. This was a really good session, highlighting that enterprise adoption of AppDynamics has major benefits for every element of the software delivery lifecycle. I hope you have found this summary useful but please feel free to comment on your experiences below or to try AppDynamics in your enterprise.