Big Data Patterns | Design Patterns | Fan-out Ingress


Big Data Patterns, Mechanisms > Data Transfer and Transformation Patterns > Fan-out Ingress
Home > Design Patterns > Fan-out Ingress

Fan-out Ingress (Buhler, Erl, Khattak)

How can streaming data arriving from a single source be simultaneously forwarded to multiple destinations without any delay?

Fan-out Ingress

Problem

Capturing and storing streaming data first and then creating multiple copies of data for multiple sinks in support of fulfilling different analysis requirements can significantly impact the speed of data analysis.

Solution

Multiple copies of input stream data are created the moment data is captured and are forwarded to the required sinks in realtime.

Application

An event-based data ingress system with demultiplexing functionality is used so that multiple copies of the incoming stream of data are generated inflight and then sent to the configured sinks.

Mechanisms

Data Transfer Engine, Storage Device

An event data transfer engine with fan-out capability is used to capture an incoming stream of data. The event data transfer engine is configured with multiple sinks. Once configured, the incoming stream of data is automatically replicated and then forwarded to the configured sinks.

Fan-out Ingress: Functionality is added to capture the incoming stream in realtime, create inflight copies of the data stream and forward each stream to the configured data sink. Some basic level of data cleansing is performed before the stream is replicated. The application of the Fan-out Ingress pattern not only removes data provisioning latency but also provides a scalable realtime ingestion functionality.

Functionality is added to capture the incoming stream in realtime, create inflight copies of the data stream and forward each stream to the configured data sink. Some basic level of data cleansing is performed before the stream is replicated. The application of the Fan-out Ingress pattern not only removes data provisioning latency but also provides a scalable realtime ingestion functionality.

In the diagram, a data stream needs to be forwarded to an external system, a processing engine and a storage device. The stream is collected and forwarded as follows:

  1. The stream is captured by an event data transfer engine.
  2. The event data transfer engine creates three copies of the stream as it arrives.
  3. (a,b,c) The copied stream is then forwarded to the external system, processing engine and storage device as a continuous stream.
  4. This results in a low data provisioning latency in terms of the time it takes the data to reach the sink from the source.

BigDataScienceSchool.com Big Data Science Certified Professional (BDSCP) Module 11: Advanced Big Data Architecture.

This pattern is covered in BDSCP Module 11: Advanced Big Data Architecture.

For more information regarding the Big Data Science Certified Professional (BDSCP) curriculum,
visit www.arcitura.com/bdscp.

Big Data Fundamentals

The official textbook for the BDSCP curriculum is:

Big Data Fundamentals: Concepts, Drivers & Techniques
by Paul Buhler, PhD, Thomas Erl, Wajid Khattak
(ISBN: 9780134291079, Paperback, 218 pages)

Please note that this textbook covers fundamental topics only and does not cover design patterns.
For more information about this book, visit www.arcitura.com/books.