In this article, I intend to delve deeper into the topic of risk management, which I have previously touched upon in this prior piece:

Lessons from Climbing: Handling Risks

Handling risks is key both in IT projects and climbing. Using the climbing approach of identifying two types of risks: objective and subjective, I draw some parallels with IT and emphasize the importance of Agility, continuous feedback, and an iterative approach to development and Unit Testing.

Stefano TurriStefano Turri

I will focus on drawing parallels between climbing protection systems and various methodologies employed in Information Technology (IT) testing. This comparative analysis aims to shed light on two crucial yet distinct aspects of effective risk management:

1. Depth represents mastery of the tools and techniques used to manage risks. Just as a seasoned climber must thoroughly understand the function, strengths, and limitations of each piece of climbing protection gear, individuals tasked with IT Testing need to master the tools and techniques available to them. This depth of understanding ensures that you can deploy these tools effectively and efficiently to mitigate risks.
2. Width: This emphasizes the importance of using a variety of tools suitable for the specific context or situation at hand. For instance, a climber will use different kinds of anchors for Trad, Sport, or Ice Climbing routes; similarly, an IT professional would use a combination of Testing approaches that best fit the project context.

Lastly, the goal is to integrate these two dimensions cohesively, creating a seamless approach to Testing. Much like a skilled climber relies on basic tools such as the climbing rope, carabiners, and harness to use the different types of anchors, the IT Professional will combine multiple Test types into seamless DevOps pipelines.

Depth dimensions: Unit testing example

To illustrate the depth dimension in risk management, let's examine the fundamental case of Unit Testing, drawing a parallel with Sport Climbing on routes safeguarded by fixed bolts. Here, each test is likened to a bolt anchor capable of arresting a fall, thereby preventing adverse outcomes.

Both approaches offer significant advantages by minimizing risks, thereby facilitating superior results. In climbing, they enable more challenging ascents and the creation of tougher routes. In IT project delivery, they enhance speed, quality, and predictability. However, expertise and training are essential for safe utilization and awareness of residual risks.   For example:

• Back Clipping:   A dangerous mistake that is too common among beginner climbers is to clip the rope so that it travels from the outside into the cliff.  In a fall, the rope can cause the carabiner gate to open and unclip from the carabiner, leading to longer and more dangerous falls. Analogously, in Unit Testing, incorrect assertions that remain valid even in error conditions render tests ineffective during failure scenarios. "Make it fail, Make it pass," a technique in Test-Driven Development, helps mitigate this issue.
• Z-clipping: closely spaced bolts are misused (e.g., clipped in the wrong order), leading to rope drag that could impede or cause a climber's fall. Likewise, unit tests with superfluous or concealed dependencies may slow down or halt test execution entirely.
• Rusted Bolts: These climbing anchors, left in the rock and exposed to varying weather conditions, gradually deteriorate, losing strength, and need to be evaluated critically by the climber when used. Similarly, older tests may become obsolete, continuing to assess irrelevant code elements or failing to update to newer testing framework versions, thus missing out on advanced features and improvements.

Improving the depth dimension also implies a Growth Mindset approach for continuously learning and adopting new tools and techniques, such as new Belay devices in climbing (e.g., Petzl recently release a new device, the Neox, that while similar to the widely used GriGri 2, has some different features and need specific training to be used correctly for safety ). In IT testing, the same learning approach applies to adopt new testing tools that might also rely on innovative Gen AI approaches.

Width dimension: Beyond Unit testing

To illustrate the width dimension in risk management, let's go beyond sport climbing and look at other contexts.

When climbing traditional routes (Trad Climbing or BigWalls) with no pre-placed fixed bolts in place,  alternative anchors such as Pitons, Nuts, and Friends are used instead. These require specific and more complex skills for the placement and evaluation of the residual risks, making this a specialized premium skill.

Ice Climbing, typically done in winter on waterfalls or steep mountain couloirs, is yet another context with additional equipment and skills. Here placing an ice screw for safety requires specific strength, technique and knowledge of ice quality to minimize the risks.

Other types of climbing, such as Sport Climbing, are more competitive in nature and focus on performances. Even here, specific safety measures are used, like automated top-rope automated belaying devices officially used in Speed Climbing Events (that have become more popular with the latest Tokyo and Paris Olympics).

Similarly, when testing IT Projects, a variety of testing techniques can be used, each one providing benefits in different contexts and requiring specific tools and skills. As an example, refer to the following article that categorizes the many kinds of possible software testing strategies:

21 Types of Software Testing Every Engineer Should Know

Explore 21 crucial software testing types for engineers to achieve better results. Boost quality and reliability through comprehensive testing techniques.

StratoflowArkadiusz Krysik

To effectively navigate between the many types of possible tests, I suggest focusing on the three following axis of evaluation.

The first axis is the type of Requirements that are being tested. Typically, these are split into:

• Functional: Related to the functional behavior of the application, such as service calls, business logic, processes, and UI navigation.
• Non-Functional: Related to quality features of the solution, such as performance (response times, traffic load), scalability, resilience (failover and disaster recovery), and security

Another important axis is the Scope of the tests. The scope can range from a single method or function to an entire component for a unit test, to a set of related interconnected components for an integration test, up to an entire system for end-to-end testing.

As the third axis, I would evaluate the trade-offs between test repeatability (for test regression purpose) and test creation and maintenance cost. The following spectrum of tests is typical:

• Manual tests require the least amount of development effort but are slow and potentially not consistently repeatable
• Automated tests that need to be developed and maintained but can provide a fast way to execute a large number of repeatable tests concurrently in a short amount of time
• Continuous tests are executed at each compilation, build, or deployment. These tests are even more complex and expensive because they need to be optimized to not slow down the software development lifecycle, but in exchange, they can be executed frequently to further accelerate the finding and resolution of problems.

Combining the approaches: The "Swiss Cheese" principle

The width and depth approaches can (and should!) be combined to create the most effective solution to the problem at hand. A good way to describe this approach is the Swiss Cheese Model, a risk management and safety concept that offers a visual representation of how multiple, independent layers of defense must fail simultaneously to result in an accident. Each layer, depicted as a slice of Swiss cheese, has its own specific vulnerabilities or weaknesses, symbolized by holes in the cheese. When the holes align, they create a pathway for the hazard to penetrate all defenses, leading to an incident. The model emphasizes the importance of redundancy to better prevent and identify problems.

Yet again, we can compare the security approach in climbing with IT Testing:

• Redundancy is used by having multiple anchors when needed (e.g., a Bolt every few meters or using a redundant double placement of Friend or Nuts in a sketchy situation), while an IT Project might increase the number of tests in specific test suites.
• Mixing anchor types whenever applicable: A sport route might have a section of crack climbing that can be better protected with additional Trad equipment, or a mixed Ice Route might have a rock climbing section that is protected with fixed bolts. Similarly, an IT project uses a mix of test types  and strategies (Functional vs Non-Functional, Unit vs End-to-End, Manual vs Automated) to minimize the overall risk

Conclusion

The overall goal in both scenarios is maximizing the benefits of risk reduction while minimizing the cost and effort to achieve this.

In both scenarios, we need to combine everything to work seamlessly together:

• In Climbing, we rely on the basic skills for using standard equipment such as ropes and harnesses
• In Software Development and Testing, the DevOps approach provides a way to handle the end-to-end strategy

I hope that this in-depth comparison provides (in a lightweight and funny way) some useful thinking points, even if not 100% accurate and precise.

Observability in managing large complex systems based on cloud and microservices can be challenging due to the dynamic and distributed nature of these systems. The traditional approach to CMDB (Configuration Management Database) which focuses on a slow-changing picture of IT infrastructure based on physical assets is not suitable for modern, continuously changing systems based on cloud and microservices. This is because these systems are highly dynamic and distributed, with numerous components that can be added, removed, or updated at any time. Therefore, it is difficult to keep track of the state of the system at any given point in time.

However, the challenge extends beyond simply having visibility into the underlying infrastructure. It also encompasses understanding how the application components are linked and utilizing that infrastructure. Without this holistic view, it can be difficult to identify and diagnose issues that may be impacting multiple components or services. Additionally, it can be challenging to optimize performance and ensure reliability without a clear understanding of how the application components are utilizing the infrastructure.

Another challenge is the sheer volume of potential data generated by these systems. With numerous components and services, there can be a large volume of code, documentation, reports, logs, metrics, and traces to sift through. Without the right tools and processes in place, it can be difficult to extract meaningful insights from this data. This can make it impossible or lead to delays in preventing, identifying, and resolving issues, which can impact the overall performance and reliability of the system.

Solving the Data Collection problem

Industry or de facto standards in collecting metrics and data on IT systems provide a way to solve the data collection problem from multiple heterogeneous sources by establishing a common format and methodology for data collection, making it easier to aggregate and analyze data from different sources. These standards can include data formats, data collection methods, and data transmission protocols.

Examples of data-collecting standards include:

• Security CVE (Common Vulnerabilities and Exposures) which provides a dictionary of standardized names for vulnerabilities and security exposures
• Static source code scans (such as Sonarqube) provide a standard way to analyze and report on potential security issues in source code
• Application Logs, which provides a standard format for logging application events and errors
• Telemetry data such as Open Telemetry, which provides a standard way to collect and transmit data about application performance and behavior
• Monitoring data from APM (Application Performance Management) systems, which provide a standard way to monitor and report on application performance metrics
• Infrastructure and networking information, which can be collected using standards like SNMP (Simple Network Management Protocol) and NetFlow.

Standardization in data collection is being pushed by regulations like the EU Cyber Resiliency Act (CRA) or by the USA's Cybersecurity and Infrastructure Security Agency (CISA). These regulations aim to improve the visibility of the digital supply chain through an SBOM (Software Bill of Materials) approach, which requires the disclosure of all software components used in a system, including their dependencies and known vulnerabilities. This approach can help organizations better understand the risks associated with their software supply chain and take necessary steps to mitigate those risks.

Structuring Semantic Relationships

A Hub and Spoke platform leveraging the data collection approach described earlier can provide a powerful solution for visualizing and understanding the state of IT infrastructure, applications, and operations. By using the semi-structured information provided by SBOMs and other input data, the platform can correlate and organize information creating a complex graph of relationships that can be utilized and analyzed using semantic web principles and techniques.

This approach can enable the platform to build an end-to-end view of the IT landscape, from network and infrastructure to application code.  Specific use cases can be tailored to be useful at multiple levels of the IT hierarchy, from operatives to managers and executives, by applying different analysis lenses (e.g. Security vs Resiliency vs Compliance focus) on the common data that can help organizations make informed decisions, optimize performance, and ensure reliability.

Leveraging GenAI for Insight Generation

A centralized data platform can serve as an ideal environment for integrating Generative Artificial Intelligence (Gen AI) due to several reasons, particularly in terms of summarization, trend extraction, and action proposal. Here's why:

• Unified Data Access: A centralized platform provides a single source of truth, allowing Gen AI to access all relevant data in one place. This holistic view helps the AI in generating more accurate summaries and trend analyses, as it doesn't miss any vital data points scattered across different systems.
• Data Quality and Security: The data collection integration process comes with data format standardization, quality checks, and robust security measures for sensitive data. Gen AI, operating in this context, can achieve better quality and guarantee privacy while performing its tasks.
• Efficient Processing: By having all data in one place, the computational efficiency of the Gen AI is significantly increased. It can process and analyze data faster, leading to quicker generation of summaries and trend detection.
• Customized Analysis: Applying multiple analysis lenses, smaller, tailored models like Granite can be trained to focus on specific areas of analysis. This specialization allows for more precise summarization and identification of trends in those specific domains.
• Enhanced Decision Making: The AI, after understanding the data trends, can propose actionable insights. It can suggest strategies based on historical data, predict future outcomes, and recommend preventive or corrective measures, thereby aiding in better decision-making processes.
• Automating Audit Trails and Compliance: Centralized platforms maintain comprehensive audit trails, which is crucial for regulatory compliance. Every action taken by the Gen AI can be tracked and reviewed, ensuring transparency and accountability.

In essence, integrating Gen AI into a centralized data platform provides a conducive environment for efficient, secure, and accurate data analysis, paving the way for informed decision-making and strategic planning.

DevOps as a key tool to achieve an application CMDB

To make this approach effective, the key element is a seamless integration into the DevOps chain, that provides key aspects to facilitate the creation of an application-aligned Configuration Management Database (CMDB):

• Data Quality Assurance: Incoming data is standardized and validated before entering the system.
• Real-time Data Processing, since in a cloud-native environment, data is generated at an unprecedented rate.
• Automated Configuration Management: An application-aligned CMDB with accurate, up-to-date information about the components in your IT environment and their relationships can trigger automated remediation actions, positively contributing to the DevOps loop.

Weaving Gen AI into the DevOps chain observability platforms (e.g. such as IBM Concert ) enable organizations to effectively manage the complexity and speed of cloud-native development and operations.

As summer vacation approaches, I've been dedicating more time to training in the evening at the climbing gym to prepare for some climbing routes over the summer. A couple of days ago, as I took a break between attempts at a boulder problem, I did find myself engaged in conversations with fellow climbers about the importance of finding reliable and trustworthy professionals, such as doctors, lawyers, and accountants, who can help us navigate the complexities of our personal and professional lives.

Because of this, I began to think about the similarities between IT consultants and mountain guides. Both are hired to navigate complex and treacherous terrain, guiding their clients through unfamiliar and potentially hazardous activities. Whether it's developing a new system or integrating disparate technologies, IT consultants must be able to anticipate and mitigate risks, just like a mountain guide must assess the risks and challenges of a climbing route.

In this article, I aim to explore the parallels between IT consultants and mountain guides, drawing inspiration and insights from this unique comparison. While we can learn something from this comparison, I must confess that the primary motivation is simply to have fun and explore the fascinating world of IT consulting and mountain guiding.

Project Context

Developing a new system or customizing an existing one is a complex IT journey, much like a climbing ascent or mountaineering trip.

In IT, the terrain to be traversed is the complex web of technologies, systems, and processes that must be integrated and harmonized to achieve the desired outcome. Just as a mountain guide must consider factors such as weather conditions, terrain difficulty, and group dynamics to determine the best route to follow, IT consultants must consider factors such as stakeholder goals, technical feasibility, and budget constraints to choose the best project approach.

What to do also depends on the stakeholder's goals. For example, if the goal is to develop a new strategic platform, the IT consultant must consider the long-term vision and requirements, much like a mountain guide would approach opening a new technical route. On the other hand, if the goal is to adopt a packaged solution, the IT consultant would focus on the ease of implementation and integration, much like a mountain guide would select the normal route to climb a peak faster and safer with less experienced clients.

Also how to do it, or the process of getting there is important. The IT consultant must be able to manage stakeholder expectations, communicate effectively with the team, and ensure that the solution meets the requirements and needs of the organization. Similarly, the mountain guide must know the client's expectations, skills, and fears, to reach the summit in the most appropriate, safe, and efficient way.

Risk Management

IT consultants and mountain guides share a common role in managing risk in complex, high-stakes environments where their clients lack the expertise or resources to mitigate potential dangers on their own.

In these situations, they must proactively identify and neutralize threats, whether it's a software flaw or a rockfall. This requires a keen ability to analyze situations, make swift decisions, and take timely action to prevent calamity. In IT, this might involve pinpointing and addressing security weaknesses, while on the mountain, it might entail recognizing and responding to signs of severe weather.

Effective risk classification and management, coupled with clear communication of these risks to clients, are essential for success in both fields, as they enable informed decision-making and minimize the likelihood of negative outcomes.

Approach: Teacher vs Service Provider

IT consultants and mountain guides can approach their work in two distinct ways. They can either deliver an end-to-end result or experience, taking full responsibility for the project, or they can act as teachers or partners, building client skills for the future.

In IT, this might mean providing a turn-key comprehensive solution, where the consultant takes ownership of the project from start to finish, ensuring that all requirements are met and the system is fully functional. Alternatively, the consultant might work jointly with the client, perhaps through design thinking and shared teams, to help them grow the skills to build and manage their own systems. This approach empowers clients to take ownership of their technology and make informed decisions about its development and maintenance.

On the mountain, this might mean leading clients to the summit, where the guide takes full responsibility for ensuring the group's safety and success. Alternatively, the guide might teach the skills they need to climb independently, empowering them to tackle future climbing endeavors on their own.

Both approaches have their merits, and the best guides and consultants can tailor their approach to meet the client's specific needs to better build trust and deliver greater value.

Both approaches have their merits, and the best guides and consultants can adapt their approach to meet the needs and maturity level of each specific client.

Trust

In both IT and mountaineering, trust is essential, when Clients must be able to rely on their guide or consultant to get them to their destination safely and efficiently. Since Clients usually have a skill gap and are often unable to evaluate the professional quality by themselves, there is a need to certify and validate the professional contributions objectively.

In the case of IT Consultants, this might mean obtaining certifications and completing training to stay up-to-date with the latest technologies and best practices. For Mountain Guides, it might mean joining or obtaining certification from a recognized association or completing rigorous training.

But it's not just about the certification itself - it's about the credibility and reputation in the professional network that comes with it. It's not enough to be only the technical experts, people skills - the ability to communicate effectively, build rapport, and establish trust - are needed for client satisfaction.

Conclusion

While the comparison between IT consultants and mountain guides could be a bit far-fetched, and as a metaphor should not be taken too seriously, I hope this will remind us of the importance of trust, risk management, and choosing the right approach to delivering successful outcomes in each complex project engagement.

Re-use is one of the most iconic "holy grails" in the IT industry. However, as any seasoned software architect knows, achieving successful re-use depends on the context and requires careful consideration of several key factors.

To validate the correct choice, we must evaluate re-use against clearly defined value drivers that take into account the following aspects:

• Economic: What are the cost savings versus the cost of integrating the re-used component? Will re-use really lead to significant reductions in development time and resources, or will it introduce new complexities that offset any potential gains?
• Flexibility: Is the reused logic flexible enough and appropriate for the context? Will it be able to adapt to changing requirements and evolving business needs, or will it become a rigid constraint that hinders innovation?
• Strategic: Is a custom solution strategically differentiating, or can a more standardized approach be used? Will re-use enable us to focus on core competencies and competitive advantages, or will it divert resources away from high-value activities, losing strategic skills?

In this article, I will delve into some aspects of re-use, highlighting key points and important considerations to keep in mind. I'll explore the benefits and challenges of re-use, and discuss how to strike a balance between the desire for efficiency and the need for flexibility and innovation.

By examining the complexities of re-use through the lens of these three critical factors, we can make conscious choices to unlock the full potential of re-use in our software development endeavors.

Components Granularity

Software reuse can occur at various granularity levels, from libraries to frameworks to platforms, each involving different roles and considerations.

• Library: developers can reuse individual components or utilities, such as logging or encryption libraries, to accelerate development and reduce duplication of effort. This type of reuse is often driven by technical leads or senior developers who recognize the benefits of leveraging existing code to solve specific problems but need to be governed to manage potential risks (E.g.: Log4j vulnerabilities).
• Framework: At this level, application architects may reuse entire application frameworks, such as web frameworks or messaging frameworks, to establish a common architecture and reduce the complexity of building new applications, by adhering to the framework conventions (E.g.: Spring, Angular, ...).
• Middleware: Selecting a general-purpose infrastructure, providing a common technical and operational ground. E.g.: Web/Application Servers (IBM WebSphere, Tomcat, JBoss, ...), Databases (DB2, Postgres, Oracle, ...), and others.
• Enabling Platform: At this level, executives and business leaders may reuse entire cross-industry platforms, such as enterprise resource planning (ERP) systems or customer relationship management (CRM) systems, to drive business agility and reduce costs. This level of reuse requires a broad understanding of the organization's business needs and goals, as well as the ability to integrate the platform with existing systems and processes. E.g.: Salesforce, Siebel for CRM,  IBM Maximo for  Asset Management, and  SAP for ERP.
• Business Platform: Industry-focused solutions, providing specialized workflow and data models, that can accelerate the adoption of industry-specific best practices. E.g.: packages in the Travel and Transportation industry, from more general purpose (Sabre) to railway-specific (DELMIA Quintiq, IVU, Sqills) solutions.

Regardless of the granularity level, software reuse may offer numerous benefits, including reduced development time and costs, improved quality and reliability, and increased consistency and standardization across the organization. However, these benefits must be quantified against the adoption cost and risks through an appropriate selection process.

Selecting Components

When it comes to component selection, it's essential to approach it as a Fit-gap exercise, weighting the solution Requirements against the component Features.

On the Fit side, we need to consider the net Actual Value being provided by re-using the component. On the Gap side, we must focus on the misalignment between Requirements and Features, evaluating both the actual Gaps and the impact of Unnecessary Complexity introduced by integrating non-essential features.

Two extreme scenarios, in which the re-use choice is simple to validate are:

• Focused Component Integration: The re-used component provides a clearly defined subset of the required features, which can be integrated with minimal unnecessary complexity. This is a typical re-use scenario for a common, easily consumable library or service that fulfills a specific requirement in isolation, such as Geocoding APIs or libraries that convert coordinates to addresses.
• Large Platform Adoption: The re-used component provides a full platform, including processes that fully cover the requirements. The benefits of full adoption, with minimal gaps to be filled, outweigh the additional costs of eventual unneeded features.

However, in many real-life situations, the choice is not so clear-cut. In such cases, it's worthwhile to evaluate the underlying flexibility of the integration process to determine the best approach for component re-use.

Integrating components

Here I highlight two common reuse patterns, Library and Framework, very much visible at the code level.

The key difference between the two is that a Library provides very specific features as a service, but does not impose a definite structure to the calling solution. The lock-in in the API definition is limited and can be more easily managed.

On the other side, adopting a Framework requires adhering to the framework conventions that impose an external shape and structure on the application being developed. The benefits of this structure (which can greatly simplify development) must be weighted against a stronger lock-in that is more complex to manage.  

When this dichotomy is at the technical level, the adoption of a commonly used framework that provides technical guidance and best practices is easily justifiable by comparing costs and risks.

As the granularity of re-use increases to the Enabling or Business platform level, organizations must exercise caution when adopting industry solutions. While these platforms may offer cost savings, they can also lead to vendor lock-in, which can be detrimental to their business. If the platform doesn't fully align with the organization's business context or strategic goals, there is a big risk of losing differentiating features and skills. Moreover, organizations may become overly dependent on the platform provider, which can introduce significant risks. For instance, what if the provider discontinues critical features or even the entire platform? This could have a devastating impact on business operations.

Conclusion

By involving the right roles and considering the various aspects of reuse, organizations can unlock the full potential of software reuse and achieve significant business value.

Particular care should be taken on adopting large platforms since this decision has a big potential business impact on competition differentiating skills and capabilities. Here I highlight the need for a decision framework like CBM and Product Engineering, to identify and communicate the correct strategy for each component.

Being a professional Engineer, coming from a family background of multiple generations of Engineers (my Grandfather, Great-Grandfather, and Great-Great grandfather all graduated from Politecnico di Milano), I have always been fascinated with engineering techniques and skills that led to complex solutions and achievements standing the proof of time, from the remains of ancient Romand building, aqueducts, and bridges up to modern Telecommunication networks.

Having focused my technical career on software and IT, I soon started wondering how to apply engineering techniques in this field. My initial experiences showed a mix of situations and methods, with some components developed in a quick and dirty way through hurried projects while other components developed with more relaxed timelines showed signs of overengineering and additional complexity.

The more complex my engagements become (such as industry-specific business platforms) the greater the need to find a way to balance effectively these two mindsets.

Product vs Project Mindset Dichotomy

When developing an industry-focused business platform the choice between product and project mindset strongly impacts the success of the platform and the business that relies on it.

Project Mindset: focus on delivering a specific outcome within a defined timeline and budget. Project teams are often formed to deliver a specific project, and once the project is completed, the team is disbanded.

Product Mindset: focus on creating a sustainable, long-term solution that meets customers' needs. Product teams prioritize building a robust, scalable platform that can adapt to changing market conditions and customer preferences.

For a business platform, a project mindset provides time and cost control but can lean too much on short-term goals. Conversely, a product mindset requires a long-term commitment to investing in the platform, continuously improving it, and adapting it to meet the changing needs of the business and its customers.

How can we address these challenges in adopting a product mindset?  

Agile Methods

Agile can be a good starting point for managing a business platform with a product-centric mindset because it emphasizes iterative development, customer feedback, and collaboration between cross-functional teams. Agile methodologies, such as Scrum and Kanban, provide a framework for teams to quickly respond to changing customer needs and market conditions while ensuring that the product meets customer requirements and delivers value.

By breaking down work into smaller, manageable chunks, and regularly delivering functional software, teams can quickly validate assumptions and adapt to changing requirements. Agile fits with a product mindset, with terms and definitions such as "Product Backlog" and "Minimum Viable Product (MVP)", but at the same time includes principles that can mitigate overengineering and keep complexity down (e.g. YAGNI in eXtreme Programming).

But in the end, even if Agile encourages collaboration between teams, it's not always adopted in an end-to-end fashion, especially in organizations that are not purely digital and Software-based.

How can we go over and further systematize the engineering approach?

Product Engineering for Industry Domains

Product engineering is creating and seeing a product through its entire life cycle from conception to end-of-life. Product engineers are responsible for designing, developing, testing, and maintaining products. This is true for both physical and digital software products, going beyond an Agile Software development approach.

Product engineering involves overseeing many aspects of the product, including its quality, cost, performance, usability, reliability, lifespan, and serviceability. Without product engineering, a product would remain an idea, so it plays a crucial role in developing a product.

The industry of product engineering is on the rise, and also IBM is getting involved in this field, having acquired Dialexa company.

Product Engineering can combine Agile, DevOps, and Lean principles with Industry and Domain-specific skills to create a holistic approach to end-to-end product development in customer organizations not limited to IT.

Product Engineering requires a multidisciplinary effort to be adopted and has to be actively managed to be effective.  

How can we find out when and where this approach should be used?

Component Business Modelling (CBM)to narrow the focus

To evaluate where to narrow the focus for adoption, it is key to start with a strategic analysis based on the specific Industry and Business context. At IBM we have a framework to approach this kind of decision, called the Component Business Model (CBM), that can be used to map the company's Business Components.

An analysis can be performed, comparing the specific company model with a reference for the industry sector, and with the support of the model, strategic decisions to focus on some Business Components can be made (E.g., strategic Business Components are marked in Blue in the picture below).  

CBM can be used to select focus areas for Product Engineering in a company belonging to a specific industry in the following ways:

1. Identify industry-specific pain points: using customer feedback, industry trends, and market research to identify the most pressing pain points and opportunities for improvement.

2. Identify areas for differentiation: analyzing competitor data and industry trends to identify areas where the company can differentiate itself from competitors. This can help Product Engineering focus on creating unique and innovative solutions that set the company apart from others in the industry.

3. Improve strengths: finding areas where processes and solutions are already strong and can be further optimized.

This can help Product Engineering streamline their workflows and free up resources to focus on high-impact areas. At the same time, on less strategic and lower priority ones,  more standard packaged solutions or commercially available services can deployed with a Project-minded approach.

Conclusion

Using a combination of these elements as a reference frame to set up an end-to-end Engineering process can lead to creating effective and long-lasting solutions, in appropriate scenarios, and avoid over-engineering in less strategic, lower-value situations.  

With this approach, I have been able to lead the design and implementation of a strategic Commercial Platform for a Railways industry client. The key enabling decision together with the client Business Architecture team was to start everything with a CBM approach. From that, using a Domain-Driven approach to design the system, and an iterative delivery process using DevOps, the business platform was implemented, became a strategic asset for the Client, and was successfully evolved and re-used on several business initiatives.  

As an experienced IT Architect who has worked on complex and risky projects to develop large Business Solutions, I understand the importance of risk management in ensuring the success of such endeavors. However, I believe that the term "risk management" can come across as too clinical and detached, failing to convey the gravity and importance of the concept.

Drawing from my personal experience with climbing and mountaineering, I have come to appreciate the significance of risk management in a more personal and visceral way. In high-stakes situations where lives are on the line, it is essential to have a framework for identifying, assessing, and mitigating risks. This framework must be ingrained in the minds of all team members and practiced regularly, much like the routines and protocols that climbers and mountaineers use to navigate dangerous terrain and handle risks in dangerous situations.

In both IT and climbing, risk management is not just an afterthought or a box to be ticked. It is an integral part of the process, woven into the fabric of every decision and action. By examining the parallels between these two seemingly disparate fields, we can gain a deeper understanding of the importance of risk management and how it can be applied effectively in IT projects.

Objective vs Subjective Hazards

In mountaineering and climbing, risks can be broadly classified into two categories:

• Subjective risks are hazards that are associated with the climber's own abilities, experience, and decision-making. These risks are influenced by factors such as the climber's skill level, physical condition, mental state, and personal choices. Subjective risks can include things like falls due to poor technique, poor judgment, or inadequate training, as well as the risk of overexertion or dehydration. Subjective risks can be mitigated through proper training, experience, and decision-making, as well as by taking steps to manage personal factors such as fatigue, stress, and anxiety.
• Objective risks, on the other hand, are hazards that are inherent in the environment and are independent of the climber's abilities, experience, or decisions. These risks are typically associated with the natural environment, such as rockfall, avalanches, weather conditions, and terrain. Objective risks can be identified and assessed through careful observation, monitoring, and analysis of environmental conditions. Climbers can take steps to mitigate objective risks by choosing routes and itineraries that minimize exposure to these hazards, using appropriate safety equipment, and staying informed about weather and other environmental conditions.

In summary, objective risks are hazards that are inherent in the environment and are independent of the climber's abilities or decisions, while subjective risks are hazards that are associated with the climber's abilities, experience, and decision-making. Both types of risks can be managed and mitigated in different ways through proper preparation, experience, and decision-making.

Let's look at both types of risks and compare them in both Climbing and IT Project situations.

Handling Subjective risks: "Projecting" with Unit Tests

Subjective risks are mainly handled through learning and knowledge. Here a Growth Mindset is key to enable effective learning.

In Climbing, the best way to raise skills is by doing, failing, and trying again until a successful redpoint ascent (Climbing the entire route in one go without intermediate rests). This approach is called "Projecting", and implies choosing a route just outside of your comfort zone and trying it with an open mindset and willingness to fail and fall. To do that safely climbing routes are safely bolted to avoid injuries when falling.      

Projecting a redpoint ascent of a sport route with secure bolts and software development using Unit Testing, TDD, and BDD can be compared in several ways:

1. Preparation: In both cases, preparation is key. Before attempting a redpoint ascent, a climber must first inspect the route, identify the crux moves, and practice them until they feel confident. Similarly, before writing code, a developer must understand the requirements (maybe even with TDD and BDD approaches), identify the critical components, and design a plan.
2. Safety measures: In climbing, secure bolts are essential to ensure the safety of the climber. In software development, Unit Testing, TDD, and BDD serve as safety measures, ensuring that the code functions correctly and meets the requirements.
3. Iterative process: Both climbing and software development involve an iterative process. A climber may attempt a route multiple times, adjusting their technique and strategy until they successfully complete the ascent. Similarly, a developer may write and test code multiple times, refining it until it meets the requirements and functions as intended.
4. Feedback: Both climbing and software development provide immediate feedback. A climber receives feedback from their body and the route, while a developer receives feedback from their tests and code reviews. This feedback helps refine techniques and strategies in both cases and can be very specific (failing a move at a specific bolt vs a single specific unit test failing) to learn effectively.

On top of these, there is a documentation value, with a bolted route remaining available to further climbers to try their skills on and with unit tests providing documentation to the maintenance team joining later into the project, reducing risks and multiplying skills over time.

Handling Objective risks: Navigating dangerous terrains with Agility

Mountaineering objective risks, are inherent in the sport and cannot be eliminated by subject skills, such as rockfall, avalanches, and weather conditions.

With Objective risks, a paradox emerges: Mountain guides have high rates of avalanche accidents and deaths and this highlights a dangerous cognitive bias in experts. Experts are at risk in two ways. First, they may make assumptions without verifying them due to their extensive knowledge. Second, their familiarity with the risk scenario can desensitize them to the potential dangers, leading to a loss of risk sensitivity.

So, to counteract this bias, mountaineers explicitly try to adopt an iterative OODA loop for evaluating Objective risk scenarios:

1. Observe: The first step is to observe the situation and gather information. In mountaineering, this means monitoring weather conditions, snow stability, rock stability, and other environmental factors that could impact the climb.

2. Orient: After observing the situation, the next step is to orient yourself to the environment. This involves analyzing the information you've gathered and identifying any patterns or trends that could impact your climb. In mountaineering, this might involve studying the route, identifying potential hazards, and assessing the risk level.

3. Decide: Once you've oriented yourself, the next step is to make a decision. In mountaineering, this might involve choosing a route, deciding when to start the climb, or determining how to respond to changing conditions.

4. Act: After deciding on a course of action, the next step is to act. In mountaineering, this means executing your plan and accepting the risks associated with the climb.

The Cynefin framework is a decision-making model based on these concepts that help individuals and organizations navigate complex, uncertain situations.  This model is particularly useful for evaluating the applicability of Agile methods in Complex and Chaotic scenarios to contain risk through a continuous feedback loop.

So, Agile methods are useful for handling IT Project Objective risks outside of the control of the project team (Unstable market conditions, Uncertain Requirements, Non-committed Stakeholders, First-of-a-Kind situations), by containing and accepting the risks in minimizable chunks (e.g. MVP, some iterations), and continuously adapting to changing situations. An honest and continuous evaluation of the current situation is required here to avoid the Exper Cognitive bias and other ones (e.g. Sunk cost fallacy).  

"Summit" it up

A Climb is never finished until you are back home or base camp, with statistics showing that about 80%  of incidents are during the descent.

In the same way, on complex Software projects and large Business Platforms, we are not finished when completing design, implementation or even reaching production. Long-term evolution and maintainability of the system is the long view that is absolutely essential for a successful IT career and navigating risks.

In conclusion, risk management is an essential aspect of both IT and climbing, and the parallels between these two fields can provide valuable insights into how to apply it effectively. By embracing a culture of risk awareness and mitigation, teams can ensure the success of their projects and the safety of their members.

The Information Technology field is constantly moving, with individuals and companies requiring constant learning to compete successfully. Fostering a Growth Mindset in an organization is becoming a necessity to catch opportunities provided by new tools, skills, and processes.

But what's exactly a Growth Mindset? Could it become yet another Corporate mantra that everyone talks about, but nobody really knows?

I'll look into it with a different perspective, inspired by some of my mountain-going experiences.

Rock Climbing Technical Skills

As an IT Architect, I'm quite fascinated by technology and activities that require complex skills, to be mastered until eventually achieving competency and enabling flow state execution.

Among sports, for these same reasons, I particularly like mountain-related activities such as Hiking, Skiing, Mountain Biking, and especially Rock Climbing.

As a kid, I was more book-reading than sport focused, so getting into it in my 20ies was a significant challenge. To safely learn I joined a mountaineering training course organized by the Italian Alpine Club (CAI), and here I had several initial setbacks by being unable to handle any but the easiest of climbing routes. However, out of the 20 participants, I was one of the few persisting with the activity and started a long journey of self-improvement and learning that allowed me to do interesting and satisfying climbs, with a responsible and safety-conscious approach.

While doing this I drew support from my local climbing circle, both from more experienced climbers, that provided criticism and suggestions for improvement, and from my regular climbing peers for continuously learning together.  While Rock Climbing for me remains a non-professional activity, I am particularly inspired by the fact that one of my regular climbing partners has been able to become a certified Mountain Guide, filling his technical gaps (e.g. initially not a good skier) to do that.

Growth Mindset Explained - Formal

In some way, my experience maps to the Growth Mindset theory, as defined by Carol Dweck's studies:

First, a mindset is an established set of attitudes of a person or group concerning culture, values, philosophy, frame of mind, outlook, and disposition. A person can have multiple mindsets, whilst the two most common ones are often cited as the Growth and Fixed mindset,

According to Dweck, individuals can be placed on a continuum according to their implicit views of where ability comes from; those believing their success to be based on innate ability are said to have a Fixed mindset, and those believing their success is based on hard work, learning, training, and sheer persistence are said to have a Growth mindset. In Dweck's words:

In a fixed mindset students believe their basic abilities, their intelligence, their talents, are just fixed traits. They have a certain amount and that's that, and then their goal becomes to look smart all the time and never look dumb. In a growth mindset students understand that their talents and abilities can be developed through effort, good teaching and persistence. They don't necessarily think everyone's the same or anyone can be Einstein, but they believe everyone can get smarter if they work at it.^[13]

Growth Mindset Explained - Climbing Fun!

But this is still very much theory, so to keep the Climbing theme I am sharing a very good video explanation provided by a Psychologist that is also a Climbing Coach. Have fun!

Summing it up

Growth Mindset particularly resonates with my extensive experience with Agile and DevOps approaches. Continuous Improvement cycles and Agile retrospectives are habits that showcase this mindset.  

In the end adopting this same approach to build company-specific habits (e.g. for consulting, manufacturing, etc...) is the way to build an adaptable and competitive organization.  

One thing that I find remarkable about current IT Industry best practices is that we daily use the closest thing we have available to a Time Machine. If this was more widely known to students, Computer Science education might appear even more cutting edge than it now is (and "Great Scott!", it could even impress Dr. Emmett Brown, .... maybe 😉).

However, Source Code Management (SCM) practices are not a big part of academic curricula, since are more in support of collaborative development rather than individual skills that are key to achieving an educational degree for the individual.  

But as soon as we are involved in large-scale projects (and Business Platforms based on Cloud and Microservices definitely qualify), these became central in everyday work.

In this post, I will have a look at key concepts that are at the base of these tools, and at effective usage patterns applicable to service platform development.

How SCM Tools support collaboration

Collaboration support is provided by helping to manage file changes through time, with features that have evolved through time with the newer generation tools.

Early tools, such as Revision Control System(RCS), focused on versioning the history of individual files and providing collaboration features such as Locks focused on preventing conflicts due to concurrent modifications of the same file. This approach (akin to DB Pessimistic locking), while guaranteeing consistency, limits efficiencies due to parallel development.

Subsequent generation tools, such as Concurrent Versions System (CVS), focused on allowing concurrent development, enabling a development process that forces conflict resolution before committing (akin to DB Optimistic locking), and storing the changes in the history repository.  CVS did become hugely popular and used for enabling collaboration on OpenSource projects, especially through SourceForge.

However, large projects require tracking time and version dependencies between multiple files. Even if CVS provided features to commit multiple files at the same time, these were limited and not too consistent (e.g. lacking atomicity, and no folder versioning support). Apache Subversion (SVN) was the next step of evolution and provided an atomic changeset of multiple files and folders, creating a consistent history graph of entire projects and not only of single files, with even better capabilities of moving consistently through code versions along the timeline.

Modern tools provide additional features, such as:

• distributed repositories (Git, Mercurial) that allow working offline and scaling to thousands of users.
• cheaper and more manageable branching, that allows committing individual changes before merging across different branches (Rational Team Concert, Git, Mercurial), making development safer by saving and backing up work at any time

Currently, Git-based solutions, especially  GitHub, are extremely popular and easily available online.

However these are just tools, let's have a look at some common collaboration patterns, in order to choose the most effective strategy for our projects.

GitFlow Usage pattern

This pattern is close to the Linux kernel source configuration processes. It's essentially based on a hierarchical communication structure, where developer branches are handled by key lead developers that integrates code developed by teams on different feature branches.  

Key aspects of this model:

• Concentrate responsibilities on the Lead developer controlling the development branch, ensuring review of all work
• All development work is reviewed before integration, reducing the trust required from other contributors
• Contributions are merged into the development branch asynchronously, favoring distributed work between multiple persons at different times

On the flip side:

• A complex process, with constant merge requests to be managed, that slows down the development pace
• Long time spent in Feature branches makes merging and integration complex, with many potential reworks
• The lead developer integration role is central, leading to significant risks in case of unavailability

Trunk-based-development Usage pattern

This approach assumes a completely opposite communication structure, based on a small group of closely collaborating peers like usually found on Agile projects. Code ownership is shared and changes are continuously integrated into the Trank branch. Work on individual features might happen in specific branches, but these are short-lived and integration responsibility remains on the specific person handling the branch.

Key aspects of this model:

• A Simpler, less formal process, that allows much greater delivery speed.
• Greater delivery speed and short-lived feature branches enable a continuous delivery process, that reduces integration risks.
• Not relying on a central authority figure to resolve conflicts

On the flip side:

• Requires trust and accountability of all participants. While often this is associated with developer seniority, with the right people mix it becomes a powerful motivator for up-skilling junior people and making them accountable (e.g. with Agile techniques such as Pair Programming )
• Not safely applicable to scenarios with many "untrusted" contributors working asynchronously

Selection Criteria

While different SCM tools have specific features that support either style, the real selection criteria depend on the context of the specific project, based on required communication patterns.

Open Source projects and commercial product development usually benefit more from a GitFlow approach because of:

• the larger number of involved "untrusted" developers (either Open Source contributors or Support people producing bug fixes) that require a formal change approval process. E.g. merge-request approval by an OSS project maintainer. Each request could come from unknown contributors.
• relative infrequent scheduled releases, that allow periodic integration cycles

Most software development projects on clients, often executed with Agile methods benefit more from a Trunk-based-development approach because of:

• a relatively small team, fully committed to the specific engagement.
• very fast incremental release schedule, using a continuous delivery approach

Conclusion

In the end, the best strategy for a specific context will combine these two approaches. For example, a Business Platform will be developed with an Agile approach, but the release branches will need more formal hierarchical approvals to ensure the segregation of responsibilities.

Cloud-native development with Microservices, given the small team sizes, incremental agile delivery, and speed of release is a good fit for starting with Trunk-based-development.

Business Platforms are characterized by a system context that includes multiple integration points (APIs) with different systems. In complex solutions, the number of integration points increases, and an effective way to document them becomes critical for the program's success.

I will focus here on some key aspects to effectively design, implement, maintain, and reuse platform APIs, such as:

• Choosing a Design Strategy
• Choosing an explicit Target Focus for the documentation
• Handling consistently multiple types of APIs, including both synchronous and asynchronous ones

API Design Strategy

The first key design choice is the strategy for defining the APIs. Two common strategies exist:

• Code First: Operations and data are defined by developers directly in code, with API specifications and further documentation generated via annotations (e.g. with JAX-RS in Java), either exposing documentation endpoints when deployed or generating specification files during the build process. This approach delegates API stability responsibility to developers and is often better suited to internal APIs between closely related agile teams that allow for great amounts of flexibility.  
• Contract First: The API specification is designed first, typically using some kind of modeling technique (e.g. Domain Driven Design - DDD, see also). This allows parallelization of work between API producers and consumers, which will integrate at the end of the development phase.  This approach works best for external APIs and requires effective planning and change control to manage it effectively to guarantee both consistency and a sufficient amount of flexibility in API definition.

A large-scale business platform will use a combination of both approaches but will be mostly characterized by its external APIs that benefit the most from a contract-first approach.

Documentation Target Focus

Another key design choice is to balance the target focus of the API documentation, balancing the following two aspects

• Human-Readable: describing the context and scope of the API, and how to use it, with different levels of complexity covered through examples and tutorials.
• Machine-Readable: enabling several capabilities such as formal version tracking via source control repositories, automated generation of client/server code for different implementation languages, and automated validation of message payloads.

Having formal, machine-readable documentation is a must to manage APIs at scale in a large Business Platform. Most API Gateway solutions provide capabilities to publish this kind of documentation, together with additional human-readable documentation to authorized consumers. Additionally, they provide and document consistent security capabilities to manage the authorization keys and secrets to integrate APIs in production environments.      

Handling multiple API Types  

API Types can be broadly split into two main categories:

• Synchronous:  based on Request/Reply interaction pattern. Examples of this type of API  are SOAP over HTTP,  REST or GraphQL
• Asynchronous:  based on message sending with a Publish/Subscribe interaction pattern. Single or multiple subscribers are possible. Examples of this type are Kafka, AMQ, JMS messaging, use of Websockets or Web Services with asynchronous bindings (e.g. SOAP over JMS)  

Each technology has its own technical features that are reflected in the API specification but also define Business-specific data. Service components might be composed of both synchronous and asynchronous operations that need to operate on a common domain model (eventually defined using approaches like Domain-Driven Design - DDD).  

OpenAPI and AsyncAPI provide this level of compatibility, by allowing the use of JSON Schema references to specify the business-specific data (Request, Response, Messages).  Common Schema definitions can then be shared across multiple APIs of different kinds that participate in business processes using the same entities.  

Other technologies can also use this pattern, such as:

• Both synchronous and asynchronous web services can use WSDL and XSD specification formats
• GraphQL schemas as extensions/combinations of multiple JSON schemas

Conclusion

AsyncAPI and OpenAPI are now both part of the Linux Foundation, so both are well-supported and have a stable future as open standards with community governance and are becoming more and more integrated with open-source tooling and commercial products providing API Gateways features, such as IBM API Connect.

Modernizing applications towards a cloud-native approach, that breaks monolithic components into multiple smaller microservice, leads to a rise in complexity in managing the IT Infrastructure operations. While each microservice becomes simpler and potentially more scalable, the number of moving pieces to be managed drastically increases.

The traditional approach of the Operations teams of managing individual Servers/VMs, monitoring,  and applying security fixes, middleware patches, and upgrades cannot scale effectively.  

A different approach is needed based on large-scale automation, to keep control and scale effectively, and is commonly explained through a common metaphor for this,  "Cattle vs Pets" where:

• Pets are the traditional Servers/VMs, individually and carefully managed by the operation team. This is a labor-intensive activity, that has to be prioritized based on the criticality of components, leading to an increasingly divergent amount of combinations of software components at different levels in each server.
• Cattle are the cloud-native VMs and Containers, managed through large-scale automation. A failing Container is not repaired, but replaced as soon as possible, while at the same time continuously learning how to reduce faults by improving automated procedures.

Two common approaches to automation have emerged, DevOps (or even better, DevSecOps, but in this context, I will not focus on Security) and SRE (Site Reliability Engineering).

Dev Ops and SRE

These two approaches might seem like two sides of the same coin since both focus on comparable tools and practices, rooted in automation and Agile methods. But at the same time, the target focus is slightly different, mostly based on the original sponsors of the approach:

• DevOps approach originates from Development teams, looking for ways to bring their application to production in a faster and safer way. Automating builds, testing, and deployments through a continuous improvement process to bring business value to production, removing operational impediments in doing so.
• SRE, as the name itself implies, focuses on the Reliability aspect, which is a key concern for the Operation teams that strive to keep the infrastructure and server running while the application and software installed might change. The focus is not so much on Business value directly, but rather on improving the manageability of applications, to keep the systems going or eventually restore them as fast as possible.

With this perspective, it seems almost as if the two approaches might be antithetical. So what's the reason for this? It boils down to the consequences of Conway's law, and specifically to the fact that in many organizations the Dev and Ops structures have different success metrics.

The stakeholders

Dev and Ops are driven by two main types of manager profiles, with alternative approaches on some different topics, which I summarize below.

The summary presented here is a kind of polarized reference, with real situations not so black and white. The more the goals of both types of stakeholders are aligned, the simper is collaborating.

Converging on an end-to-end approach

To make this possible, the first step is to better align the Goals of both Development and Operation structures. To achieve a successful Digital Transformation and modernize applications both Business KPIs and Service levels goals must be shared by the organization.  

To implement this, a Platform Engineering approach (see also) is critical to support both Dev and Ops activities, by providing common methods, platforms, and services to boost productivity and collaborations within and across both teams.

Conclusion

Platform Engineering becomes the critical enabler to effectively manage a Hybrid Multi-Cloud at scale. This is the technological foundation to develop effectively any kind of Business platform, that can drive a real Digital Transformation.

Data in applications has a powerful gravitational pull, so transforming and modernizing applications is heavily impacted by existing data. Digital Transformations have focused on enabling new cloud applications and frontends but remained connected to existing transactional backends (see also). This kind of approach highlighted potential issues in handling traffic load shifts in reading vs writing transactional data. I had some direct experience with this on projects:

• Online ticketing: moving from traditional sales at stations to online apps and services leads to a significant reduction in the conversion rate from travel searches to tickets sold (e.g. an online user tries to plan several trip options before committing to a purchase ), increasing resource consumption.  
• Online Banking: The adoption of online banking greatly increased the usage of query operations (e.g. end user can read their account statements online) vs actual financial transactions. To contain the increased load and cost on the backend transactional systems a data replication approach (a.k.a Copy Banking) can be used to offload query traffic from the expensive Mainframe towards a hybrid cloud containerized infrastructure.

To continue the cloud journey, we need to refactor also backend transactional systems, with their persistent data (of mission-critical, financial nature), and make the new services able to scale adequately and cost-effectively.

Breaking down the monolith

Domain- Driven-Design (DDD) is a very good tool to break down the Business domain (see also), to identify parts of the business applications that can be modernized and operated as independent microservice components. Analysis of the data is very helpful in this process, to identify contexts based on different data relationship types:

• Foreign key relationships: Group tables that are related to the same entity
• Transactional relationships: Group tables that are updated in the same transaction

These groups of related tables are indications of candidate DDD aggregates. A good candidate is when the tables of the group are tightly coupled within the group but have few relationships with other tables.  

This is such a good strategy, that the largest chapter of the "Monolith to Microservices" book by Sam Newmann focuses explicitly on patterns for managing data handling.

The identified contexts can then be extracted (code and data together) using variations of the "Strangler Pattern", to create the replacement independent components.

Coexistence architecture

While on the front-end and application components, the strangler pattern can be applied using HTTP proxy/routing techniques to dynamically replace sections of the application, the need of seeing consistent data requires other techniques.

Rolling out a modernized application is rarely a big-bang event, with instead long periods of parallel deployment and testing of the mission-critical applications. This "coexistence" need requires a suitable architecture and in IBM we have a reference cookbook that provides some key fundamental patterns.

These patterns help in moving out the data and managing the transition in either direction:

• Current to Modernized
• Modernized to Current

Let's have a look at these patterns, focusing on the Current to Modernized direction.

Change Data Capture

Current systems events are discovered by consuming changes in the system's existing artifacts (typically database files and/or data sources). This pattern is used when changes to Current Programs are not affordable or strategically desirable, but in this case, additional effort must usually be spent to adapt the event for the destination systems with on-demand transformations that often duplicate existing legacy logic.

Key tools for implementing this pattern are:

• IBM® IBM Change Data Capture (CDC Replication)
• Oracle GoldenGate
• Debezium, for an open source CDC

Application Event Streaming

This Pattern is used to replicate the Current system application's state, through application changes to expose all business events to an event stream(s). Events are sourced to destination systems with on-demand transformation (usually limited to adapting model schemas, but not replicating business logic).

Events are exposed after the business operation is completed to avoid phantom events. To support this it's advised to use a message-passing middleware with transactional guarantees (either local transaction or exactly-once/at-least-once guarantees). Such middleware can be:

• Queue-based systems (JMS, IBM MQ, etc...)
• Topic/Partition based (Kafka)

Further articles in this series will go more in-depth on events/messaging.

Filtering & Transformation engine

This pattern is used to reduce event streams relevant events by applying transformation and filtering rules and caching processed events

Filtering rules are used to:

• Drop irrelevant events (i.e. CDC sends everything)
• Route to different/multiple event destinations

Event cache is used to:

• Filter out duplicate events (i.e. multiple sources)
• Guaranteed idempotent retries of event transformations
• Aggregate low-level events

Transformation can become complex both in CDC and Event Streaming scenarios, especially if transformations need to duplicate the business logic of the Current system.  EAI pattern-aware tools such as Camel/Fuse are particularly useful for these kinds of transformations.

Choosing an approach

Ultimately, CDC is not a silver bullet for modernizing systems. All these patterns are tools for modernizing applications, each with its own strengths and complexity, and each, in the wrong context, can become an anti-pattern.

We need to choose the right balance of complexity based on the solution we want to address.  As a criterion for this Litmus test, I would use the target envisioned architecture, that can either completely replace the old system or partially reuse it:

• Replace: The old system no longer has strategic value, so the goal is moving out of legacy and transitioning completely to a new solution. In this scenario, CDC with Filtering and Transformation logic eases the transition to the new data model, with the goal of removing these steps with duplicated logic as soon as the transition is complete.  This scenario would fit better with a monodirectional data flow.
• Coexist: The old system still has strategic value, so the goal is to get back ownership of the old legacy application and integrate it with the new components. In this scenario, Event Streaming will work best at decoupling the systems and replication data based on a shared semantical model, but CDC might still be effectively used if the two data models do not require significant transformation logic to be maintained. It's also a more appropriate target solution for bidirectional data flows.  

Especially with the second approach, It would be better to avoid a 2-speed architecture organizational model, but rather have the legacy application teams included in the overall IT process, to foster an effective Agile/DevSecOps environment in maintaining the two systems and their data flows.  

Cloud usage has steadily increased in recent years, with most enterprises adopting cloud solutions to deliver new applications and migrate workloads out of proprietary data centers.  However, application modernization towards the cloud has not yet started for a lot of mission-critical systems and workloads. What's slowing this down? reasons for this are in the complexity of interactions of Business Enterprise Platforms, and at the heart of it, we have the transactionality challenge of handling critical business data.

An effective move to the cloud needs to tackle this issue in some way to be effective. As a first step, let's start with defining the problem in the context of existing legacy platforms.

Traditional Transactional applications

Some examples of transactional systems I have worked with include:

• Financial and banking systems that move money between accounts. Every operation must complete atomically, and failures should not cause side effects
• Commercial platforms for Travel and Transportation, which ensure the consistent completion of the ticket purchase process. Either the ticket is issued, the seat reserved and the payment is finalized as a single action or no effect at all should happen.

To achieve this traditional transactional applications are based on some key enabling technologies such as the Mainframe or the Relational Database, which provides facilities (e.g., 2-phase commits, ACID properties) to simplify the widespread usage and creation of transactional logic.

What's the commonality between a DB and Mainframe? it's the presence of a centralized component that ensures the consistency of the system, but what happens if we move toward cloud-native solutions with multiple interacting components on a globally distributed scale?

Why not Distributed Transactions?

Having a distributed and partitioned system introduces new failure possibilities and additional network delays and this makes the scenario more complex:

• Distributed transactions in SOA/WebServices solutions proved to be complex, and brittle with additional pain points created by tightly coupling systems.  
• Middleware such as relational databases become vulnerable to networking issues (e.g: Split Brain)

Like Newtonian physics encountered one of its limits with handling the speed of light, here also the speed and availability of the communication network forces us to rethink the approach in broader terms.

Enter the CAP (or Brewer's) Theorem that tells us that "any distributed system or data store can simultaneously provide only two of three guarantees: consistency, availability, and partition tolerance (CAP)"

A new theory must improve on an old one, so how does CAP Theorem apply to legacy systems?

As seen above, these are based on centralized, high-performing systems, such as Relation DBMSs or Mainframe boxes. The key aspect of these solutions is the very limited use of partitioning and a strong focus on ACID consistency and replication for High Availability and Disaster Recovery.

So, these systems are examples of the Consistent-Available (CA) type of solution allowed by the CAP Theorem.  

Towards Eventual Consistency

Now, with the cloud, we are choosing to increase the partitioning and distribution of our systems.

So we are left with two types of possible systems:

1. Consistent-Partition Tolerant (CP)
2. Available-Partition Tolerant (AP)

While it would be nice to guarantee Consistency, to avoid compromising end-user service in a cloud-distributed solution composed of many moving parts, each of which can fail at any time, forces us to favor Availability.

But not all is lost on the Consistency side. While there is no guarantee that all clients see the same data at the same time, it is possible to quickly converge toward a share data consensus (a.k.a. Eventual Consistency).

Tools to achieve it

Let's see how to combine some techniques to achieve this.

Component internal transactions

Fortunately, like General Relativity did not invalidate Newtonian physics tools when applied in specific contexts, we can often continue to use traditional ACID transactions inside each component, since each component is tightly coupled in its implementation, with limited internal networking and partitioning.

This is a key criterion in Microservice Decomposition, which can be guided by the transactional boundaries of specific business entities to effectively partition a monolithic system.  Domain-Driven-Design (DDD) techniques are based on defining atomic operations on "aggregates" that focus on business entity data that should be updated transactionally (see also).

How to break down a system thus become a critical architectural decision, to evaluate trade-offs in coarse-grained vs fine-grained service decomposition, since services doing transactional coordination might be more effective with the former approach.  

Idempotency

When communicating across components it's necessary to handle and resolve networking errors. Since messages can be lost either before or after the remote systems receive them a mechanism for safe retries is needed. This can be implemented using techniques to ensure idempotency. In short, Idempotency means that duplicate calls are not processed multiple times, preventing errors or unwanted side effects in the system (e.g. the same payment must be processed only once).

This can be implemented with some steps like this:

1. The system receiving a message checks whether the message has been processed before. (e.g. by comparing its unique identifier to a list of previously processed messages).
2. If the message has not been processed before, the system processes the message and adds its outcome indexed by a unique identifier to the list of processed messages.
3. If there's a need for a synchronous response, the outcome (generated by the current or the previous call) is returned.

This pattern is particularly effective for asynchronous notifications since:

• no response is needed (no step 3)
• messaging systems (e.g., JMS, MQ, or Kafka) only-once/at least-once delivery guarantees can be used to simplify retries, limiting impacts on the overall process latency.
• messaging systems usually automatically provides a unique identifier for messages

Compensations

This technique is typical of Business Process Modelling (BPM). BPM solutions target process integrations across heterogeneous systems, and this is a scenario in which distributed transactions are not possible because of the loose coupling f the systems. In instances of a Business Process, compensations occur when the process cannot complete a series of operations after some of them have already been completed, so the process must backtrack and undo the previously completed steps.

The price to pay is the additional complexity of implementing the reversal operations and applying them to execute the backtrack correctly.

SAGA pattern

This pattern combines the three previous techniques to implement a cloud-native BPM implementation, building a sequence of local transactions where each one of these updates the data of its competence (based on the service model decomposition), with each subsequent step activated by the completion of the previous one. Failures and interruptions in the process are identified (e.g., using timeouts) and trigger compensation logic, while external systems are notified of updates via event notification.

SAGA pattern can be implemented with different approaches using:

• a central process Orchestrator service
• a Choreography based on message passing
• a combination of both

Here's an example of how I approach this in designing a solution for selling transportation Tickets:

I based the main transaction loop on a more coarse grain service orchestrating the completion of the transactional process, but I leveraged idempotency and event streaming to trigger notifications to update target systems that could tolerate more delays in reaching an eventually consistent state.

Conclusion: designing applications

In the end, it is a trade-off between requirements and complexity. Different client contexts might require different solutions, even in the same industry (e.g., Banking in Europe tries to achieve ACID consistency, while other countries, such as the US, are more tolerant towards eventual consistency).  

So, a crucial skill for Lead Architects and Designers is being able to work to shape requirements and solutions, together with other business and technical stakeholders.  This can be facilitated by adopting a structured approach such as Enterprise Design Thinking to converge on mutually agreed expectations regarding solutions.

Transactional cloud-native systems based on microservices can be realized effectively, but the complexity increase had to be managed. On the requirement side, Design Thinking and Domain-Driven-Design are good tools to enable this, but implementation complexity should also be managed. I'll look in the future at techniques such as Async API and DevSecOps that enables managing larger and more complex solutions.

Successful business platforms and architectures need to handle change and evolution gracefully. A good problem domain decomposition, documentation, and versioning of Business API and Entities, ideally using Domain-Driven-Design (DDD) principles, help evolve the functional side.

But there is an additional need. The choice of the Database platform and how to use it will impose operational constraints on evolution options and costs.

Handling effectively the evolution of Persistent Data is a crucial aspect, especially when there is a need to keep data records available for a long time. A flexible and cost-effective solution is required for evolving in an Agile and incremental way the persistent data together with the Domain Model updates.

In this post I will touch on an approach I used on very large-scale engagement:

• Choosing a persistency Strategy
• Versioning the persisted entities
• Evolving the entities
• Key Highlights

Choosing a persistency Strategy: is Relational vs Object the issue?  

For this approach, the choice between Relation vs Object (or SQL vs noSQL) is not really the heart of the matter.  What is really required is having a database layer that does not enforce a strict explicit schema, but allows an implicit one.  With an implicit schema, an Aggregate (as a unit of transactional updates related to an entity) can be persisted and be flexibly updated only when needed.

NoSQL databases (such as MongoDB) are particularly effective in working with implicit schemas, but this strategy can be applied also using relational databases with the help of object mapping features (e.g. DB2 with native XML columns, or Postgres JSON support).  I have used this mixed approach to persist and model transportation ticket entities effective, with the ability to choose the most appropriate strategy for each different entity/aggregate.

Versioning Domain Model entities - a cornerstone for architecture evolution

Applying this approach works in two steps, and the first one is the persisting/writing of the entity/aggregate data.

The Domain Model information is serialized using a standard format (e.g XML or JSON) and is always stored with the explicit version being used at the time of writing.  Effective implementation of Domain entities serialization using frameworks (e.g JAXB), code-generation, or templating techniques.

Evolving Entities through chaining model transformations

The second step is reading back the persisted entity/aggregate.

After reading the raw data, version of the persisted entity is compared with the current one. If it's the same, good ! Otherwise the system applies a chain of tranformations to progressively update the information up to current version.

Transformations can be implemented in different ways. When persisting in XML format I did found XSLT a natural way to chain XML transformation, but plain code also works.  

In the examples presented on the above picture, the aggregate with id #2, originally persisted with Version 2.0, is being read twice:

• The first time, when current aggregate Versio  is 3.1, only transformation tB is applied
• The second time, when aggregate Version has increased further to 4.0, thre transformations are chained tB, tC, and tD.

The entity information, now up-to-date, can be deserialized with the same tools used for serializing it.

Key Highlights

Here I focus on some Key Highlights and consequences of adopting this approach.

• On-Demand vs Pre-Emptive evolution

The approach described above updates the entitiy information when reading it, but different trade offs can be applied in  chosing when to write it back to database:

1. Only if the entity is being explicitely modified
2. After reading it
3. With a scheduled, pre-emptive updated

Again, in all the scenarios, the entity information will have the updated Version number persisted together with the data.

The third option might be similar to a traditional schema evolution, but with this approach there is no global schema change that might cause outages, allowing each entity record to be updated in background at different times. These updates can be scheduled via batch processes or triggered with database DevOps automation (e.g. Flyway DB updated tasks on relational databases)

• Maintaining transformations

Pre-emptive updates are important also to avoid an increase in the number of transformations to be managed. After having ensured that all the records have reache a certain versio level, the older transformations are no longer required.

• Encapsulation

This is where the DDD approach and encapsulating the transformation logic into the Domain Services did really shine. These services will always provide outside an up-to-date view of data, with a clearly documented API and semantics, published through standards such as Web Services or REST Services through specific API Gateways.

The lack of encapsulation instead quickly becomes an anti-pattern, unless Data published outside of the Bounded Context is documented really well, with clearly defined API and data exchange formats. I did find examples of this antipattern in legacy data models (e.g. of old mainframe applications), with a combination of structured and relational data, evolved over time and lacking a consistent model description. Exposing this data directly to external consumers (e.g. through Copy Banking projects using Change Data Capture - CDC technologies) is then a very risky and complex proposition.  

Conclusion

This kind of persistency approach is not applicable in every scenario, but focus on handling application specific data over shared/normalized data models. This make it a good fit for encapsulating persistency in microservices implementation.

Business enterprise platforms are complex systems with an ecosystem of many internal and external integrations and dependencies. Modernizing these kinds of applications requires a divide and conquer approach, with a transformation program composed of multiple steps, ideally performed in an Agile way.  But, how to proceed? Let's see how Domain-Driven Design (DDD) addresses this.

Splitting the domain into Bounded Contexts

DDD uses Bounded Contexts as a way to split the overall system into subdomains. It's important to note that each Bounded Context has also its own ubiquitous language that defines the concepts of that domain.  This is important since Business Organizations and IT Systems are linked together in their evolution by Conways's law, so any effort to decompose the business domain should be based also on business criteria, and in different subdomains, concepts and entities might have their own specific and precise meaning.

Since a map is always a useful tool to understand the big picture (see also this article), let's see how the relationship between the Bounded Context could be visualized with a Context Map, such as this one that I have drawn inspired by Travel & Transportation domain experience.

In this example, we see that the same concept might appear in more than one Bounded Context. Is this a problem?

No, this is on purpose, since in different contexts the same entity is used and characterized in its own specific way, based on the ubiquitous language of that specific context.

So in the above example, the Customer of the CRM Context is linked but not the same as the Customer of the Sale Context. However, these are linked via IDs, that allow semantic references across different contexts.

What's inside a Bounded Context? Aggregates to define microservices

We have just seen that each Bounded Context has its own Domain entities, which sometimes can be simple, but often are a composite of multiple data in hierarchical (think object classes) or relational (think table) relationships.  How can we organize the domain models of each context?

In DDD the divide and conquer is applied also at this level, grouping together domain data that is handled together into Aggregates. An aggregate is updated atomically and enforces rules of consistency dependent on the Business domain.

Multiple aggregates can refer to a single domain concept, and these can be related via IDs. For example, in a further drill down of the Context Map shown above, focusing CRM Context we see the main Customer aggregate but also other aggregates related to Customer via the ID, such as:

• PreferredDestinations
• PreferredTravellers
• PaymentOptions

Each of these can be updated independently of the others, and are good candidates for a possible implementation as separate microservices, even in the same domain.

Next step: transactionality

We have used two main DDD tools for Domain decomposition:

• Bounded Contexts
• Aggregates

These are the key to defining the application modernization journey since they drive the microservice solution. Specifically, a microservice cannot be bigger than a bounded context or smaller than an aggregate.

This also provides key elements for managing the transactions of the solution, with each Aggregate as the natural place for using database transaction facilities to ensure their atomic updates, with eventual consistency used between different Bounded Contexts.

But for this, I'll go more in-depth in Part II - "Transaction and Events microservices".

In a complex climb or on a mountaineering expedition the goal and the map (see also) to achieve it are essential but are not enough.  On top of these, a set of trade-offs and decisions need to be handled to successfully and safely get to the goal, such as:

• how much material to have: in this case the advantage of less weight against safety and hunger in case of delays caused by poor weather
• length of pitches and how many anchors to use in rock climbing: trading off speed vs the length of potential falls
• Rope discipline: taking the time to keep the belays organized to avoid creating risks in handling messed up ropes

Likewise, in complex IT systems, like Business Platforms, it's not straightforward how to address a transformation toward a cloud solution. It's essential to manage the complexity of the problem across multiple aspects, decoupling problems and issues to avoid complexity pile-ups and exponentially difficult situations.

Decoupling will not be perfect, but like in the Gordian knot myth, forceful actions and decisions are necessary to get a situation unstuck and develop a new and better design equilibrium.  These decisions will have costs (measured in different ways, such as money, time, effort, or lost features) that make it complex to find the target solution with the optimal trade-offs. However, not deciding is also a choice with its own costs (e.g because of security, maintainability, and obsolescence).

In this series, I plan to go in more depth into a set of different aspects to address and optimize these hard decisions, in the context of decoupling and evolving complex enterprise systems. Specifically, I will look into:

1. DDD to Break down the Business Domain - Here the focus is on functional aspects of the business domain, to identify a better functional partition and decoupling of components. Also exploring techniques to break down independent microservices in a single bounded context.
2. Transaction and Events microservices -  Here the focus is on breaking the monolith by addressing the compute side. Traditional enterprise systems are centralized around a transactional core to guarantee business consistency. Cloud and distributed services require re-thinking of requirements and approaches (e.g. with Event Streaming and CQRS) to move from a traditional ACID approach to an eventually consistent one within the constraints of the CAP Theorem.
3. Unravel the data - An often overlooked aspect, that creates long-lasting complexity in large-scale integrations that need data flowing across multiple systems and domains. This is a critical challenge in breaking monoliths since data encapsulation in old legacy systems is lacking because of ad-hoc batch point-to-point integrations.  
4. DevOps to assemble the pieces - All previous aspects focus on decoupling to handle the complexity of the application modernization initiative. This however increases the number of running pieces in the solution and DevOps is the key to handling this complexity, with a documented and auditable approach. This is not limited to traditional build and deploy pipelines but should address all tools to design, observe, manage and improve the solution.