5 Years & 600 Developers! What it Takes to Transform a European Bank

Q: What is your official role at KB?

A: My official role is Head of Development, COE (center of expertise) at Komerční banka.

Q: What exactly is your area of responsibility and day-to-day?

A: As the Development COE lead, I consider the expertise, structure and capabilities needed to make software development successful across hundreds of developers. We have roughly 600 people in the broader development team, but I’m not directly responsible for managing them. I’m responsible for defining how they will do their job. So for development best practices, standards unification and so on.

Q: What’s your main focus today at KB?

A: Our transformation project. We have the current bank, which is built on typical legacy banking systems…basically huge, distributed monolithic architecture systems. That is where most of our developers (around 400) are working day-to-day as it generates the bulk of the revenue for the bank. Alongside the legacy applications, we decided to completely modernize to something that we call “Digital Hub” which is replacing the bank’s current infrastructure. The front-end applications are native mobile and web applications. The Digital Hub and new front-end applications are built entirely on microservices so we have to redesign a lot of things and introduce new guidelines for people. All of this together is what we call the “New Digital Bank”. It’s a massive project!

Q: Does this transformation stop at some point?

A: Great question. We think about the transformation in two main parts:

• “The New Digital Bank” initiative is planned for 5 years and we are about 2 ½ years into it already. It started small with a few teams that were ready to adopt the most modern development approaches and it’s growing every year. Today we have about 180 engineers split among 50 teams. Those teams have been hired externally or transferred over from the legacy systems. They are dedicated 100% to “The New Bank” effort.
• The other side is culture transformation. We call this “Agile 2.0” which hits the whole team (both legacy systems and new systems teams). Adopting Agile and DevOps is all about adopting a new way of working. This shift is something that we are driving with our legacy system teams as well. While the building will come to an end at some point, the hope is that the new culture will take hold and be the new way of working at KB.

Q: Beyond agile, have you adopted any new approaches?

A: Yes, there are three new approaches that we adopted:

• Moving to OKRs which aligns well to agile.
• Adoption of Microsoft Teams in all of our meetings which has helped with remote work during the COVID pandemic.
• The last has been adopting ‘the Spotify model’. I don’t know why, but it’s pretty popular in Czech Republic. The biggest thing that resonated with us is that you have functional “tribes” that contain the people and skills that the “tribe” needs to develop and run their own projects or their own applications. This aligns well to the microservice architecture that we are moving towards.

Q: What about tooling, languages and infrastructure? How do you manage that across so many people and teams?

A: As the COE, we are mostly responsible for the unification of how people work. We are pretty rigorous in this. We developed our own KB framework, which we call Speed which is divided into three parts:

• First is a Java SDK, built on top of Spring Boot, which is a framework for building cloud applications.
• CI/CD Pipelines to automate deployment. This is built on Jenkins and Argo CD.
• Kubernetes, where we are deploying and running our applications.

We built our own cloud infrastructure, so everything needs to be on-premise today. If a team needs something, they need to ask for it and it is reviewed by my team. If we have something that solves their problem, then we ask them to use that. If not, I’m open to adopting a new technology.

Q: Is that an evolving system that changes as you go?

A: Yes exactly. We started with the Speed Framework two or three years ago and it is still evolving. The first version was built for monolithic applications and monolithic works. We are on version 4 now. The microservice approach was implemented in version 3. The disadvantage is that we have to follow some product updates and life cycles which is really painful for us. Kubernetes, for example, has three major versions per year. We are now facing the situation that the upgrade pains hurt as it takes a few days each time. Going forward we want to make it more independent of these cycles and automate upgrades in the future. Interestingly, our parent company adopted OpenShift which is something we are keeping an eye on. We’ll see if we unify the frameworks or not in the future.

Q: Sounds like you are using a lot of open source technologies in the new platform, have you always done that?

A: No, we used a lot of the typical products like Oracle, Sun Microsystems and IBM. For the new digital bank we decided to adopt more Open Source technologies like Kafka, Flagsmith and ElasticSearch, which is great because they are free, but we are responsible for keeping it up and running. This is a completely new mindset for teams.

Q: Did you have to fight to use those products internally?

A: No, they sort of just accepted that because they know the reasons. If we show them the contract with Oracle, how expensive it is, there’s a big business case to adopt open source. Even if you have to have a few more people to manage the infrastructure it has a positive impact on our costs.

Q: What about feature flags, was that something that you knew you needed before you started?

A: That’s a good question because, you know, even in current banks (legacy systems), people are using feature flags, but they’ve developed their own systems. They’re very simple systems that manage the configuration file for example. And it was just because the legacy system was only deployed 3 times per year. So you can imagine how difficult it was to keep all of the promises from other teams. So they were really just there to keep the enterprise release management and testing on time and coordinated. It was easy to switch “on” or switch “off” some features that were not ready for the deployment. But for the New Digital Bank we have microservices and continuous integration. We deploy on a daily basis to non-production environments. Interestingly, we only have two non-production environments. One of them is only for testing during the build and the other is the final non-production environment (AKA staging).

Q: Was it a conscious decision to have fewer environments?

A: Yes. This is an approach that I brought to the bank where we completely redesigned some environments and how we deploy to them. The goal is to get to production as fast as possible in order to keep production and non-production as close as possible.

Q: That’s interesting…for some of our customers the holy grail is to only have one environment, just production, but that’s probably impossible in your business, right?

A: Yeah, that’s impossible. There’s a lot of regulations. For example, a lot of developers can’t access the production environment. I have some experience with that where you just test everything in production. For example, one regulation says that you can’t even have a testing account in production because you are operating with real money. Everything must be “real” in production. Despite the regulations, I still wanted to have the smallest number of non-production environments as possible.

Q: So when you were selecting a feature flagging system, did you consider SaaS services or was that something that you immediately ruled?

A: Yeah, we decided to evaluate 3 systems; Flagsmith, your competitor LaunchDarkly and considered building it ourselves. We did three “proof of concepts”. First was with LaunchDarkly, then Flagsmith and then we looked at our own homebuilt system. We decided on Flagsmith not just because of the system’s flexibility, but also the great support, the fact that you guys are open source and the great documentation.

Q: How are things going today with feature flags?

A: We have been using feature flags for one year in our non-production environments which has been working great. As for the production environment, it took a while from a security perspective to implement feature flags in our production environment. The reason it took so long is because the person originally responsible for bringing feature flags to KB left. Then the security team changed their criteria for evaluation and then finally on the third time, we got it into production after passing all of the penetration testing and requirements!

Q: Wow! Making changes at a bank is not for the faint of heart!

A: Yes, because of the security standards the entire banking industry is really sensitive to this. Now we are up and running and focusing on the permission models and logging everything so it is auditable.

Q: Has it changed your workflow at all?

A: Yeah! We decided to start with the front-end, especially mobile teams. People have been waiting for this capability because they’ve used feature flags in previous roles. We have people on the team pushing to adopt because without a feature flag system, everything has to wait until it is fully finished. Now they can deploy to production and more and more teams are onboarding to The New Bank. For those teams in a microservice set-up, it is crucial for teams to be more independent. So far, we have about 85 engineers working with Flagsmith in production. That is almost half of The New Digital Bank already!

Q: Are you using flags on server-side as well or just the front-end?

A: So far just on the front-end, but we are getting ready to roll that out with the backend systems that leverage microservices. That will be very soon!

Q: Any ways that you want to use Flagsmith in the future?

A: Yes. We are looking at integrating the Flagsmith API into JIRA and being able to drive things upstream and be able to keep the QA teams aligned with the product owners.

Q: Oh great! We are actually working on something similar at the moment. We’d love to show you guys and get some feedback.

Well thanks so much for being a customer and for taking time to connect with us today! We really appreciate it.