Bringing DevOps to Mainframe Application Development: A Real-World Story

Author: Tatsiana Ihnashchenka

Sometimes, the old ways are best. Consider a client of ours.

They have operated a massive, central mainframe for over many decades. They do not consider mainframe application development outdated or obsolete, they are increasing the volume and criticality of the workflows that they run on their mainframe every year. It delivers processing power, reliable performance, and a level of security that they believe they could not replicate with more “modern” distributed infrastructure.

But sometimes, the old ways do require an update.

This client continued to follow a legacy approach to developing and deploying applications for their mainframe. They had followed traditional approach since they first launched their mainframe many decades years ago, and it created multiple problems within their development and mainframe application deployment process:

• New applications required long development cycle times that often experienced significant delays.
• Development and delivery processes were manual, fragmented, and varied across their 20+ global teams.
• Once they delivered a new application feature, they often had to roll it back quickly due to quality issues.

This client knew they needed to modernize their approach to mainframe application development and deployment. They also knew they could not complete substantial transformation on their own. That is when they decided to partner with us.

To date, we have worked with client teams for over 18 months. Together we have made significant progress bringing modern DevOps practices to their mainframe world.

Today, we will begin to tell our story.

Our Real-World Story About Modernization Mainframe Processes: Warts and All

This is the first story of this series:
1. Bringing DevOps to Mainframe Application Development: A Real-World Story
2. The Key to Large-Scale Mainframe DevOps Transformation: Building the Right Dedicated Team
3. Choosing the Right First Steps in DevOps Mainframe Transformation
4. Improving the Initial Mainframe Automation and Creating a “Two Button Push” Process
5. How to Select and Implement the Right Tools for Mainframe DevOps Transformation
6. Picking the Right Tools: A Deep Dive Into the Most Popular Options for Mainframe DevOps
7. Completing the DevOps Pipeline for Mainframe: Designing and Implementing Test Automation
8. Making Change Stick: How to Get Mainframe Teams Onboard with New Strategies, Processes, and Tools

We are sharing this client’s story for one big reason, we want to provide teams like yours with a practical understanding of what it takes to bring a modern DevOps approach to mainframe application development and deployment.

That’s why we are going to tell the story as honestly and thoroughly as possible. We will include the highs and the lows, including:

• The Challenges: Modernizing mainframe processes isn’t easy. We want you to understand the problems that we faced as we helped mainframe application development teams modernize decades’ old processes.
• The Solutions: Modernizing mainframe processes is possible. We want you to see exactly how we worked with client’s teams and overcame their challenges. We will include the highly tactical steps you can take yourself.
• The Rewards: Modernizing mainframe processes is worth it. We want you to realize the benefits gained by bringing something new to the world of mainframe, benefits that you may be able to realize yourself.

We have undertaken a long journey, and we will tell this story through a series of articles. Each article will explore a specific step that we took to modernize mainframe application development processes. By the end of this series, you will have a blueprint for how you can undertake a similar journey in your organization.

To begin, we will use the remainder of this introductory article to present a top-level view of the project, including what might be the biggest challenge we encountered to its successful delivery and how we overcame it together.

Project Overview: Biggest Mainframe Development Challenges

While we cannot reveal the name of the client, we can tell you that they are one of the largest multinational organizations in the world. They employ hundreds of thousands of people, operate in over 150 countries, and generate tens of billions of dollars in annual revenue. They are a very tech-driven organization and mainframe is highly entrenched in their day-to-day operations.

This client creates hundreds of new applications for their mainframe every year. Yet despite the importance of these applications to their business, 18 months ago they faced multiple challenges to developing and deploying applications in an efficient, effective manner.

Challenge One

Legacy Processes

They lacked DevOps. They continued to follow a waterfall process they established when first launched their mainframe decades ago, and had been running this project for 20+ years. They utilized legacy development, test, and production systems, as well as a librarian.

Challenge Two

Manual Mainframe Application Deployment

They lacked widespread mainframe application automation. Most testing and promotion was performed manually. Most communications were handled via email. Each step was documented ad hoc and hosted in spaces that were poorly coordinated and not standardized.

Challenge Three

Siloed Teams

They lacked unity. They had 20+ globally distributed teams who were developing their own applications and features. Each team leveraged unique solutions and approaches that could not apply to any other team’s work.

These challenges created significant problems for mainframe application development and deployment capability.

• They experienced poor visibility and control. This led to needlessly long mainframe application development and delivery cycles. It took client weeks or months to deliver any change to their mainframe applications.
• They could only build and enforce selective and siloed automation. This prevented them from producing meaningful process improvement. They were “stuck” doing things the way they always had.
• They could not perform effective quality control. They experienced constant issues with application quality. They had to focus their attention on reactive “firefighting” and not on proactively improving their processes.

They knew they needed to overcome these challenges.

They knew they needed to follow a new approach to mainframe application development and deployment.

They just did not know how to bring new approach to life.

Modernizing Mainframe Processes: First Attempts at Change

Our client had a clear plan about how to turn their situation around.

• They wanted to reduce cycle times, prevent delays, and improve the quality of the applications they delivered by modernizing mainframe processes.
• They wanted to make testing and delivery more efficient by cascading mainframe application automation throughout deployment processes.
• And they wanted to unite 20+ development teams and get everyone to follow the same approach for every one of the hundreds of applications they delivered.

To achieve these outcomes, they decided to bring a modern DevOps approach to mainframe application development across all teams.

Unfortunately, first attempts to bring DevOps to life in their organization did not stick.

We will provide a bit more details on initial DevOps attempts, and what went wrong, in coming articles. But for now, know this. They focused almost entirely on selecting the right tools, and had not considered what it would take to leverage those tools effectively, and to cascade a new culture of DevOps throughout their organization. This is a common mistake that we see. Organizations think that only a technical solution is required, and do not understand the importance of creating cultural change, a point we will explore in-depth later.

In short: They knew where they needed to go. They had an idea of how to get there. But they soon realized they could not do so on their own.

That is when they reached out to the IBA Group Mainframe Center of Excellence.

Our Mainframe DevOps Partnership: Setting the Strategy

When our client reached out to us, they had a good top-level idea of what they wanted to achieve. They had purchased the tools they wanted to use. They had determined what mainframe application automation they would like to develop. How they saw their new architecture. But they knew they needed help to sketch out the details, and to bring their plans to life.

We took their tools and their ideas and designed a new, modernized approach that would bring DevOps to their teams. We built our plan around three core pillars.

1. We Would Create a Dedicated DevOps Team. We knew they needed a team with the right skillset to champion the change, and to focus all of their attention and resources on implementing our new approach. The team would apply the best practices of modern application development to the world of mainframe. They would incorporate scalable pipelines to create mainframe CI/CD, and set the client’s teams up to send smaller, but more frequent, changes.
2. We Would Automate Many of Their Processes. We knew they needed a unified approach to automating their testing and delivery processes. We focused our initial efforts around creating a unified test automation architecture and establishing automated code scans and reviews. Over time, we developed a growing library of automation scripts that would be shared between distributed teams within the organization.
3. We Would Standardize Their Approach Across Teams. Finally, we knew these changes needed to be adopted by all 20+ mainframe development teams across the organization. We created a common repository with our CI/CD approach documented, which included automation library guides. We removed redundant documentation, simplified communications, and performed continuous learning sessions to build the culture across the teams.

We knew how to build DevOps around the tools they had selected, which meant, ultimately, this last challenge, building the new DevOps culture across the organization, proved the largest and the most critical challenge to overcome. 

Overcoming the Biggest Challenge: Building a New Mainframe Culture

 To be clear, we had support from the highest levels of management from the start.

They are the ones who reached out to us. They understood the need to transform their approach to development and mainframe application deployment process. They knew this transformation would give them the ability to deliver new mainframe applications faster and at a higher quality.

It was the actual on-the-ground mainframe application teams that we had to convince to make these changes.

Please note: We do not wish to pick on anyone. We simply wish to present what happened, so you can understand what resistance you might face if you attempt to bring similar changes to your organization. In this example, it was the client’s developers, testers, and business analysts who initially resisted the changes that we proposed.

Overcoming their resistance was the primary project issue we faced. Here’s how we did it.

How We Beat Ground-Level Resistance: Empathy and a Desire for True Buy-In

To overcome this resistance to the changes we proposed, we first sought to understand why these ground-level teams were concerned about our recommendations. And after a little digging, it wasn’t hard to see why they resisted the changes that we proposed.

• The changes would impact these teams’ day-to-day work, and force them to adopt new ways of doing things.
• These teams were very invested in their existing processes. Many of the team members had been following the old ways for 10+ years.
• They believed their existing processes were simpler, more flexible, and faster, especially when they needed to perform a non-standard action.
• At the most fundamental level, these teams felt the changes we proposed would simply add new obstacles to their day-to-day working lives.

After we mapped out these concerns, we set about brainstorming ways to disarm them.

We decided right away that we did not want to just use brute force to push these changes through, even though we could have. We had the support of senior leadership. Our project’s sponsor, the project architect, understood Mainframe DevOps and CI/CD. He was going to make our proposed changes mandatory. We knew that his developers, testers, and analysts would eventually have to adapt to the new ways, whether they wanted to or not.

However, we also knew that the type of mainframe transformation we proposed would benefit from an equal level of “bottom-up” support from the users who would put it to work. At the end of the day, this was their transformation, and it would only proceed optimally if these users adopted it themselves and were not forced to do so.

As such, we worked hard to determine the ideal way to get everyone on board with the changes.

Through trial-and-error, we found three success factors that helped the client’s developers, testers, and analysts relax their concerns, better understand the benefits of the transformation, and happily adopt it.

Success Factor One: We Provided In-Depth Instruction

Many of these teams had one big problem with the changes we recommended, they did not fully understand the change we were proposing, nor how it would (work in detail).

Every time we drove a change, we made sure to break it down into simple steps. We hosted multiple education sessions for each group of specialists impacted. We provided all of the education we could to remove the mystery surrounding the changes that we proposed.

And it worked. Once the teams better understood the changes and how they worked, they felt less skeptical and more open to adopting them.

Success Factor Two: We Provided High-Touch Support

These teams also worried about the transition to the new processes, and how well they would be able to navigate these periods of adjustment.

During each DevOps transition period, we directly supported everyone who would be impacted by the upcoming change. Most important, we answered every question the teams asked (even if the question was unrelated, or written into documentation, or had been asked 100 times already).

Once the teams realized we would fully support them through every step of mainframe DevOps transformation, and that there would always be an answer or a “way out” of any trouble they experienced, they worried less about each new upcoming change.

Success Factor Three: We Provided Real, Tangible Benefits

Finally, these mainframe teams did not understand how our changes would improve their day-to-day working life.

There was no shortcut to overcoming major challenges. We simply had to help them move through each change and experience the superior way of working for themselves.

Experiencing was believing. It took less than a week for each team to get used to each change, to experience the benefits, and to happily integrate them into an ordinary part of their day-to-day work.

Real-World Results: The Outcomes from Mainframe Transformation Journey

By delivering on these three success factors, we have been able to achieve top-level and ground-level buy-in. Over the last 18 months, we have worked with our client to transform how they develop and deploy applications for their mainframe. And the teams are now delivering new applications faster, more reliably, and with increased safety.

Specific outcomes we have delivered include:

• Accelerated Project Delivery: Developed software can now be delivered in 1-2 hours, down from 2-4 weeks.
• Improved Collaboration and Streamlined Onboarding: Global teams now utilize the same toolsets and standardized processes, and work from a common code base and deployment approach, that make it is easy for new members of the team to quickly get up to speed.
• Greater Agility: Engineers are more independent and can deliver new functionality at almost any time, reducing their planning time and increasing their flexibility and innovation.
• Higher-Quality Software Development: They have reduced or eliminated many error-prone manual processes, and improved the speed, scale, and effectiveness of their testing processes.
• More Strategic Mainframe Management: By automating many of their prior time-consuming manual tasks, they have freed their resources to focus on meaningfully improving their platform, and not managing it.

If you are interested in achieving outcomes like this by transforming how you develop and deploy applications for your mainframe, then look out for the remaining articles in this series or subscribe to our newsletter. In the next piece, we will explore how, and why, we built a dedicated DevOps team to drive client’s mainframe transformation.

If you would like to learn more about mainframe DevOps project without waiting, then contact us today and we will talk you through the elements of driving a DevOps transformation for mainframe that are most relevant to your unique context.