{"id":15548,"date":"2021-07-09T17:12:43","date_gmt":"2021-07-09T15:12:43","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/?p=15548"},"modified":"2023-06-26T15:07:12","modified_gmt":"2023-06-26T13:07:12","slug":"multisystem-interaction-consistency-with-aws-step-functions","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/backend\/multisystem-interaction-consistency-with-aws-step-functions\/","title":{"rendered":"How to Manage Multisystem Interaction Consistency with AWS Step Functions"},"content":{"rendered":"\n

Logistics <\/em>is one of the most advanced areas in ICT and related technologies today. All recent issues arising – microservices<\/strong>, coexistence with legacy systems<\/strong>, process streamlining<\/strong>, big data cleansing<\/strong>, data streaming<\/strong> – are currently dealt with using code-managed infrastructures. <\/p>\n\n\n\n

New services, such as AI-based software, can easily be introduced in this open architecture, its further development securely in the hands of experienced DevOps-focused engineers who constantly target their activity on increasing company value through continuous, yet secure, delivery of new releases for existing software and brand-new code.<\/p>\n\n\n\n

That makes logistics and large retail\/supermarket chains probably the best places for a developer to gain experience today.<\/p>\n\n\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tTable Of Contents\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Retail and logistics, where microservices rule<\/a>
  2. PHP runtimes for the microservice environment<\/a>
  3. What is Saga?\u00a0<\/a>
  4. Relating unrelated processes<\/a>
  5. Backward recovery<\/a>
  6. A transaction process\u2019 success story<\/a>
  7. Conclusions<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    Retail and logistics, where microservices rule<\/h2>\n\n\n\n

    This is also the train of thought at ShopFully<\/a>, where data sits at the heart of business. ShopFully\u2019s vision is to become the world-leading tech company in drive-to-store and proximity marketing solutions.<\/p>\n\n\n\n

    What is this company\u2019s business and why is it relevant to a developer\u2019s career<\/a>? ShopFully<\/strong> is a tech company, and a leader in drive-to-store solutions. The company\u2019s mission is to make local shopping easier. The company helps more than 40 million consumers globally to save time and money through in-depth information on nearby stores. <\/p>\n\n\n\n

    It is a reliable tech partner to over 700 retailers and brands at an international level. These are inspiring numbers for developers who want to grow in a real-world environment.<\/p>\n\n\n\n

    ShopFully is also a place for retailers and brands to engage with shoppers when they are most open to explore new shopping opportunities. According to their website presentation, ShopFully\u2019s people are the omnichannel shopping experts, measuring and predicting billions of paths from ad exposure to physical store visits.<\/p>\n\n\n\n

    The company offers a full cycle, including software and marketing development. For hundreds of retailers, this platform offers the best way to reach millions of shoppers with promotions and generate measurable store visits. <\/p>\n\n\n\n

    ShopFully offers mobile-to-store advertising and profiling solutions specially designed for brands to influence customer behaviour. <\/p>\n\n\n\n

    The company normally hires qualified developers, currently prioritising full stack competencies (React, Node.js) and classic backend competencies (PHP, Amazon AWS) with a mid- to senior level of expertise and a minimum of 3 years of experience in that role.<\/p>\n\n\n\n

    PHP runtimes for the microservice environment<\/h2>\n\n\n\n

    ShopFully offers high-level retail software solutions for the biggest global chains. Their ICT offering hosts both legacy and modern solutions in a well-orchestrated workflow management system.<\/p>\n\n\n\n

    Today\u2019s companies need to update their existing portfolio, adopting a modern and consistent app model for all phases in the app life-cycle. This demands a continuous evaluation of the overall portfolio, in order to schedule updates for good, but old, software.<\/p>\n\n\n\n

    That leads to a coexistence of both worlds – old and new – in this transformational software paradigm. A migration of classic ICT solutions to modern architectures<\/strong> is underway in ShopFully, together with state-of-the-art, serverless, microservices-based applications. <\/p>\n\n\n\n

    Most of the old software was written with PHP <\/strong>language. PHP applications can be included in the world of microservices thanks to mechanisms provided by third-party applications.<\/p>\n\n\n\n

    The existing PHP-based backend relies on Amazon AWS and Bref <\/strong>to run and deploy serverless PHP applications.<\/p>\n\n\n\n

    \"Bref<\/figure>\n\n\n\n

    Bref<\/a> generates open source, production-ready PHP runtimes for AWS Lambda functions, defining itself as \u201ca higher level stack for a better view of the cloud\u201d.<\/p>\n\n\n\n

    Creating and running modern and scalable PHP applications is not always as simple as it should be. By using serverless technologies<\/strong> like AWS Lambda, the focus turns to development, with servers forgotten.<\/p>\n\n\n\n

    Bref is an open source project that brings full support for PHP and its frameworks to AWS Lambda<\/strong>. It can be overwhelming getting started with the serverless paradigm, if your starting-point is a classic backend approach. <\/p>\n\n\n\n

    Bref integrates with the open source Serverless Framework for configuration and deployment. Powerful, yet simple, it offers the best of both worlds.<\/p>\n\n\n\n

    As well as the Serverless Framework\u2019s documentation and ecosystem, Bref provides complete documentation and examples for PHP applications.<\/p>\n\n\n\n

    This presents a viable method of integrating legacy applications in a more modern execution environment. Serving new needs, or old needs in a new way, is a demanding challenge that requires a novel approach to data management through many services. <\/p>\n\n\n\n

    New elements are developed with a different full-stack programming flux that is based on a classic Javascript-powered Node.js\/React approach. Some stages of this process are commonly referred to as a saga<\/strong>.<\/p>\n\n\n\n

    What is Saga? <\/h2>\n\n\n\n

    Software solutions are very innovative in this large-scale, evolving environment. The overall workflow is the basis on which every piece of software is integrated into the existing portfolio, or modified according to new needs.<\/p>\n\n\n\n

    The workflow used allows sellers to place purchase orders on behalf of customers and automatically associates and creates the bid campaigns that are the core of the business model.<\/p>\n\n\n\n

    Collected data have to be leveraged in the way most useful to subsequent needs, losing no part of their identity.<\/p>\n\n\n\n

    The workflow interacts with different information systems. Communications with separate systems must then be guaranteed while the integrity of the data is maintained. Classic transactional databases are not the reference choice for this task family, that title belongs to distributed systems.<\/p>\n\n\n\n

    In the microservice-based approach, one structural issue is keeping data consistent across multiple services. Transactions often have a long journey to completion, crossing multiple databases and communicating with multiple services that may be temporarily unavailable.<\/p>\n\n\n\n

    Local ACID transactions<\/strong> are not always of help in this context. In fact, according to the AWS page<\/a> on step functions, \u201cbecause these microservices cannot use a single atomicity, consistency, isolation, durability (ACID) transaction, you might end up with partial transactions. In this case, some control logic is needed to undo the transactions that have already been processed\u201d<\/em>.<\/p>\n\n\n\n

    The classic approach is the so-called 2PC<\/em> (two-phase commit). This theoretically solves the problem if everything goes to plan, but this particular case is not the standard scenario in the microservice-based world.<\/p>\n\n\n\n

    How to manage a sequence of local transactions is the issue at hand. The distributed saga pattern is typically used for this purpose. The sequence of events involved is normally referred to as a \u201csaga<\/strong>\u201d. <\/p>\n\n\n\n

    It\u2019s amusing to note that the original meaning of the word refers to long, epic stories written mainly in Iceland between 1120 and 1400. In later times, the term is extended to cover the story of a family and its cross-generational problems, as detailed in a report – and this leads us to an explanation of the modern, technological use!<\/p>\n\n\n\n

    Returning to transaction-solving, the inclusion of a saga gives rise to the need for management. There are two common ways to achieve saga management, choreography and orchestration.<\/p>\n\n\n\n

    Each local transaction publishes domain events that trigger local transactions in other services, dealing with a choreography.<\/p>\n\n\n\n

    An external object known as \u2018the orchestrator<\/em>\u2019 rules all the local transactions to be executed by every called service.<\/p>\n\n\n\n

    This kind of process management<\/strong> leads to great results, as well as ensuring rollback if required. This makes saga-based development look like one of the best ways to ensure the consistency of data in a distributed architecture. <\/p>\n\n\n\n

    ShopFully\u2019s management chose to tackle this issue by using the Saga Pattern<\/strong>. Using AWS as a Cloud provider, the natural choice was to use AWS Step Functions <\/strong>to implement the Saga Pattern.<\/p>\n\n\n\n

    Relating unrelated processes<\/h2>\n\n\n\n

    Originally, the purchase flows of ShopFully\u2019sproducts and their delivery were disconnected from each other, with information being passed manually, without automation.<\/p>\n\n\n\n

    When the two flows were integrated, they involved heterogeneous systems and therefore the problem of maintaining the involved data\u2019s integrity arose. One way to handle this is to use the saga pattern. <\/p>\n\n\n\n

    When you use the saga pattern, \u201cevery service that performs a transaction publishes an event that triggers subsequent services to perform the next transaction in the chain. This continues until the last transaction in the chain is complete. If a business transaction fails, saga orchestrates a series of compensating transactions that undo the changes that were made by the preceding transactions\u201d<\/em>.<\/p>\n\n\n\n

    In brief, then, for each operation performed on a system, there is a compensating transaction that cancels the operation performed if something fails on any of the systems involved.<\/p>\n\n\n\n

    A good solution <\/strong>to the problem involves:<\/p>\n\n\n\n