Processing Engine

Machine Learning Patterns, Mechanisms > Mechanisms > Processing Engine


Processing Engine (Erl, Khattak)

The processing engine is responsible for processing data, usually retrieved from storage devices, based on pre-defined logic, in order to produce a result. Any data processing that is requested by the machine learning system is fulfilled by the processing engine.

A processing engine utilizes a distributed parallel programming framework that enables it to process very large amounts of data distributed across multiple nodes. It requires processing resources that are requested from the resource manager.










A processing job is submitted to the resource manager (1). The resource manager then allocates an initial set of resources and forwards the job to the processing engine (2), which then requests further resources from the resource manager (3).

Module 12: Fundamental Service API Design & Management

This pattern is covered in Machine  Learning Module 1: Fundamentals Machine  Learning.

For more information regarding the Machine  Learning Specialist curriculum, visit