Machine Learning Patterns, Mechanisms > Mechanisms > Processing Engine

 

Processing Engine (Erl, Khattak)

The processing engine is responsible for processing data, usually retrieved from storage devices, based on pre-defined logic, in order to produce a result. Any data processing that is requested by the machine learning system is fulfilled by the processing engine.

A processing engine utilizes a distributed parallel programming framework that enables it to process very large amounts of data distributed across multiple nodes. It requires processing resources that are requested from the resource manager.

 

 

 

 

 

 

 

 

 

A processing job is submitted to the resource manager (1). The resource manager then allocates an initial set of resources and forwards the job to the processing engine (2), which then requests further resources from the resource manager (3).