The Evolution of Event-Driven Architecture: From Disaggregated Products to a Unified Platform

In the world of software development, event-driven architecture has become an increasingly popular way to design and build applications that respond to user actions and system events in real-time. These applications are designed to be reactive and responsive, allowing developers to build applications that are scalable, resilient, and highly available.

Today, the software infrastructure needed to build event-driven applications is dominated by disaggregated products that are designed to handle specific aspects of event processing - such as message queuing, stream processing, or event logging. These products include Apache Kafka, AWS Lambda, Azure Event Hubs, and Google Cloud Pub/Sub, to name a few.

While these point products have proven to be effective at handling specific aspects of event-driven application development, they are often limited in their scope and lack the ability to provide end-to-end event-processing capabilities. As a result, developers are forced to stitch products together in order to build a complete event-driven application, which can lead to complexity, inefficiency, and increased costs.

Gartner acknowledges the same in its report about “Innovation insight for streaming data in motion”. 

Most large organizations already have some systems that process streaming data. However, these systems are fragmented and only partially effective because of their architectural immaturity, incomplete technology infrastructure, and lack of integration with other data assets.

To address these issues, the event-driven application category needs to evolve from a disparate point-product approach to a unified platform approach. A platform approach provides developers with a comprehensive set of tools and services that can handle the entire event processing lifecycle, from ingestion to processing to serving and action.

Similar progressions across other software categories

The evolution of point products to platforms is not a new phenomenon. We have seen similar patterns of evolution in other software categories. CRM (Customer Relationship Management) software category started off as point products designed to manage specific customer interactions. In the early days of the CRM category, point vendors such as Siebel Systems, ACT!, GoldMine, and SalesLogix provided specific solutions for sales force automation and contact management. 

However, as businesses started to focus more on customer-centric approaches, CRM evolved into a platform that could integrate with various other tools such as marketing automation, customer service, and analytics. These point solutions lacked the integration capabilities of modern CRM platforms but paved the way for the comprehensive and integrated CRM platforms we see today, such as Salesforce, Microsoft Dynamics 365, and HubSpot.

Even in software infrastructure space, some examples that stand out are Docker - Containerization Platform, Hashicorp - Infrastructure automation, Twilio - Cloud communications, DataDog - Cloud Monitoring and Analytics etc.

The platformization of event-driven apps is already in flight

This platform approach is already being adopted by some of the major vendors in the event-driven architecture space. For example, AWS EventBridge provides a unified event bus that allows developers to build event-driven architectures that span multiple services and applications. Microsoft Azure Event Grid provides a scalable and event-driven messaging platform that simplifies event processing across Azure services and third-party applications.

However, these products still lack some of the important capabilities such as the ability to transform the event data, look up the history of those events if necessary, and serve the data to end-user applications. They would still have to integrate other point products to achieve this functionality. 

Similarly, Confluent has moved towards extending their platform by the acquisition of Immerok, while Datastax has entered into streaming space by embracing Apache Pulsar. 

These are great moves and create a facade of tightly interoperable wrappers to interact with underlying components. However, they fall short of creating a true integrated platform as the underlying issues of having to manage event logic between the components, handle multiple cluster journals and consensus, take care of weak system level durability & availability standards etc. still remain outstanding

Introduction to Grainite

The benefits of adopting a platform approach to event-driven application development are clear. By using a single unified platform that provides end-to-end event processing capabilities, developers can reduce complexity, improve efficiency, and lower costs. They can also take advantage of advanced features such as event routing, filtering, and enrichment, as well as real-time monitoring and alerting.

This is where Grainite plays a crucial role. Grainite is a Unified Event Streaming Platform that helps with the end-to-end development of event-driven applications. The secret sauce behind Grainite is that it has converged an event queue, a stream processing engine, and a streaming database into a single product.   

‍

Tightly integrating these capabilities and building the product from scratch enables Grainite to offer some unique capabilities

• Guaranteed Message Delivery
• Exactly-once messaging
• Data level observability
• Low latency real-time data sync between sources and destination systems
• Store and serve materialized read/write views to end applications
• Unlimited key-level parallelism that avoids the need to pre-set shards or partitions

In an architecture built with disaggregated products, furnishing all these capabilities will become the headache of the developer. Each application should have its own additional logic to handle message failures, guaranty delivery, add instrumentation code or agents to enable monitoring etc. If you have multiple applications, this effort has to be repeated that many times.

A unified platform like Grainite flips this around. It makes such platform guarantees as part of the infrastructure. Application development becomes a breeze as one has to just focus on their business logic and nothing else.  

We have written more about how Grainite can help with unifying batch and stream processing, simplifying ETL and more here.

Conclusion

While the trend towards Platformization continues it is not always a clear cut choice for every software category. A helpful way to decide is to map out your needs and choose the best option.

In the early stages of a category journey where the use cases are not entirely clear, it is beneficial to go with disparate products that fit your needs. This affords flexibility to swap out products with new ones. In addition to the maturity of use cases, if the needs are very specific or have budget constraints going with disaggregated products makes sense. 

However,  if you are looking for scalability, ability to integrate other tools or if the overall use case is very clear, going with a unified platform is the right choice.

In the case of event-driven applications and streaming data, the use cases are very clear. As businesses continue to demand comprehensive and integrated solutions that can handle end-to-end processes, we can expect to see more vendors in the event-driven applications space adopt a platform approach. In the end, it's the developers and businesses that will benefit from the increased efficiency, lower costs, and advanced features that platform-based solutions can offer. If you are interested in learning more about how Grainite can help, please reach out to us.

About Grainite

Founded in 2019 by leaders from Google and Citrix, Grainite is backed by top-tier VCs. The team has over 200+ years of collective experience in building large-scale enterprise applications at leading data and storage companies. Reach us at info@grainite.com to learn more and to schedule a demo.