
The rise of web-scale applications, mobile ecosystems, and distributed cloud platforms has created a demand for low-latency, fault-tolerant, and horizontally scalable real-time systems capable of ingesting, processing, and reacting to continuous streams of data. Traditional request/response and batch-oriented architectures struggle to meet modern throughput, elasticity, and responsiveness requirements due to tight coupling, synchronous dependencies, and periodic processing delays. Event-Driven Architecture (EDA), combined with distributed log-based messaging systems such as Apache Kafka, has emerged as a robust paradigm for building resilient real-time streaming pipelines that decouple producers and consumers while ensuring durability and replayability of events. By treating data as an immutable sequence of ordered records rather than transient messages, streaming platforms enable scalable fan-out, state reconstruction, temporal analytics, and independent evolution of microservices. This article maps the architectural evolution toward streaming systems, synthesizes foundational design patterns including Event Sourcing and CQRS as articulated by Martin Fowler and later expanded through distributed systems research by Martin Kleppmann, and presents an evidence-based introduction to streaming pipelines that integrate storage, messaging, and computation into a unified log-centric model. By consolidating theoretical insights with industrial case studies, this study establishes a coherent technical foundation for practitioners and researchers working with early-generation streaming infrastructures, highlighting how distributed logs serve as the backbone for scalable, observable, and fault-tolerant real-time architectures.
Apache Kafka, Fault Tolerance, Event Sourcing, Log-Centric Design, Event-Driven Architecture, Streaming Pipelines
Apache Kafka, Fault Tolerance, Event Sourcing, Log-Centric Design, Event-Driven Architecture, Streaming Pipelines
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
