Data that is held in storage can be processed in a variety of ways by a given Big Data solution, and all data processing requests require the allocation of processing resources. Users of Big Data solutions can make numerous data processing requests, each of which can have different processing workload requirements.

A resource manager acts as a scheduler that schedules and prioritizes processing requests according to individual processing workload requirements. The resource manager essentially acts as a resource arbitrator that manages and allocates available resources, as shown in the example in Figure 1.

Figure 1 – A realtime processing job (1) and a batch processing job (2) are submitted for execution. The resource manager allocates resources according to the job workload requirements and then schedules the jobs on a realtime processing engine (3) and a batch processing engine (4) respectively.

This pattern is covered in BDSCP Module 2: Big Data Analysis & Technology Concepts.

For more information regarding the Big Data Science Certified Professional (BDSCP) curriculum,
visit www.arcitura.com/bdscp.