Powered by OpenAIRE graph
Found an issue? Give us feedback
ZENODOarrow_drop_down
ZENODO
Article . 2018
License: CC BY
Data sources: Datacite
ZENODO
Article . 2018
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

Architecting Real-Time Systems with Event-Driven Streaming Pipelines: A Unified Log-Centric Approach Using Apache Kafka

Authors: Shekar Vollem;

Architecting Real-Time Systems with Event-Driven Streaming Pipelines: A Unified Log-Centric Approach Using Apache Kafka

Abstract

The rise of web-scale applications, mobile ecosystems, and distributed cloud platforms has created a demand for low-latency, fault-tolerant, and horizontally scalable real-time systems capable of ingesting, processing, and reacting to continuous streams of data. Traditional request/response and batch-oriented architectures struggle to meet modern throughput, elasticity, and responsiveness requirements due to tight coupling, synchronous dependencies, and periodic processing delays. Event-Driven Architecture (EDA), combined with distributed log-based messaging systems such as Apache Kafka, has emerged as a robust paradigm for building resilient real-time streaming pipelines that decouple producers and consumers while ensuring durability and replayability of events. By treating data as an immutable sequence of ordered records rather than transient messages, streaming platforms enable scalable fan-out, state reconstruction, temporal analytics, and independent evolution of microservices. This article maps the architectural evolution toward streaming systems, synthesizes foundational design patterns including Event Sourcing and CQRS as articulated by Martin Fowler and later expanded through distributed systems research by Martin Kleppmann, and presents an evidence-based introduction to streaming pipelines that integrate storage, messaging, and computation into a unified log-centric model. By consolidating theoretical insights with industrial case studies, this study establishes a coherent technical foundation for practitioners and researchers working with early-generation streaming infrastructures, highlighting how distributed logs serve as the backbone for scalable, observable, and fault-tolerant real-time architectures.

Keywords

Apache Kafka, Fault Tolerance, Event Sourcing, Log-Centric Design, Event-Driven Architecture, Streaming Pipelines

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Upload OA version
Are you the author of this publication? Upload your Open Access version to Zenodo!
It’s fast and easy, just two clicks!