Event-driven Microservices with Compensating Transactions
An important goal for microservice architecture is loose coupling so that each service can evolve, deploy and scale independently of other services. The database-per-service pattern is an important aspect that enables microservice architecture. It ensures that each microservice encapsulates its own data and makes it available only via its API.
However, a major challenge that emerges from this pattern is coordinating writes as distributed transactions. Unlike monolithic applications that can use ACID transactions to roll back to a consistent state, a failure in any of the microservices involved in a distributed transaction requires each microservice to undo writes to its private database.
In this blog post, we will walk through building microservices on Grainite that communicate with each other and coordinate writes. In case of failure, we use compensating transactions to undo the effects of the writes.
Grainite is a platform that converges the capabilities required to build event-driven microservice applications. These capabilities include an event bus, containers for the execution of user-provided business logic, a NoSQL database, a built-in adaptive cache and more, all in a single executable that runs on a scaleout cluster. And critically, exactly-once guarantees as data moves between the event bus, compute, and database.
When building microservices on Grainite, developers focus on their application logic and leave platform-level concerns of guaranteed event delivery, database storage, caching, scaling, and resiliency to Grainite. Grainite runs on public cloud and private infrastructure. Apart from Kubernetes for cluster management, Grainite has no other dependencies. The Grainite conceptual model explains how to think about Grainite. The rest of this blog assumes knowledge of the Grainite conceptual model.
To follow along, you can request a free Grainite trial (or optionally download a docker image) here and download the Grainite client. The Grainite client includes the command line tool gx and a few samples used to interact with the Grainite server that runs in the trial instance or the docker container.
Example application - Order system in Grainite
We will build an order system to demonstrate compensating transactions between microservices in Grainite. The complete project, along with instructions and prerequisites is in this Gitlab repository.
The application flow is shown in figure below. Order events stream into the Order Service. After some validations, the Order Service sends an event to Inventory Service. The Inventory service also does some validations and if they pass, it sends an event to the account service for payment. If there is a failure at any service, the compensating transactions rollback the changes.
In Grainite, the central artifact is the app.yaml. The app.yaml file contains the definition of the app including its tables, topics, action handlers, and their subscriptions to topics. Here is a snippet of the Order Service app.yaml.
We will create 3 apps - Order Service, Inventory Service, and Account service. The app.yaml files for each of these services are in their respective directories here.
Here is the high-level picture:
Each of the 3 apps can be deployed and updated independently of each other. They do not need to be stopped before updating. Here is the flow along with the relevant code snippets:
Step 1: The client first creates 10 inventory items and accounts
Step 2: It then sends order events to the order events topic
Step 3: Grainite stores these events durably and as specified in the app.yaml, invokes handleIncomingOrderEvent action in OrderHandler, passing the current state of the order in it also. Since this example only sends new orders, the state will be empty initially.
Step 4: The handleIncomingOrderEvent action validates the order. If the validation succeeds, it stores the order in the order table, sends an event to the specific item by invoking the executeItemRequest action that belongs to ItemHandler in Inventory Service, and registers a compensation transaction with the CompensationHandler.
Note: The action executes transactionally. If it succeeds, its state will get updated and the events to InventoryService and CompensationHandler will all be committed. If it fails, none of these side effects will be executed. In the absence of such guarantees, application developers have to resort to complex and inconvenient mechanisms to achieve this outcome.
Step 5: executeItemRequest in turn tries to reduce the quantity of the item in the inventory. If that succeeds, it registers the compensation transaction & sends an event to the deductAmount action in Account Service. If there is insufficient inventory, it triggers compensation.
Step 6: deductAmount action deducts the amount the Account holds by the amount in the order. If successful, it sends an event to the order to change the status from PENDING to SUCCESS. If it has insufficient funds, it triggers compensation.
Step 7: The compensation transactions are registered in the registerCompensations action of the CompensationHandler action handler that is part of Order Service. If any action was unsuccessful, the triggerCompensations will invoke the compensation for each of the registered transactions. Then it is up to the compensation transaction to undo the effects of the successful transactions.
Step 8: The client get the orders from Grainite
Applications built using event-driven microservices require a host of capabilities from the underlying platform. An event bus, database(s) per microservice, ensuring transactional operations in handlers, exactly-once messaging, observability, and the ability to scale compute & storage independently and automatically are just a few of them.
Grainite provides all these capabilities in a developer-friendly Kubernetes environment. As the example above showed, Grainite can host complete applications that are independently developed, deployed, and communicate with each other using asynchronous message passing. Application developers focus on the logic to process the streaming events while rapidly building and deploying these apps to production.
To summarize: In this simple but complete example, you saw how Grainite can
- Ingest events from external clients
- Execute developer-provided processing logic on those incoming events statefully
- Generate events to be sent to other applications asynchronously
- Update the state of entities
- Run compensation transactions across applications in case of failure
- Serve client queries with its database.
This convergence of capabilities is what makes Grainite extremely powerful, simple to use, and an excellent platform for building event-driven microservices.