Nephele Data Flow Engine
Definition:
Nephele is a massively parallel data flow engine dealing with resource management, work scheduling, communication, and fault tolerance. Nephele can run on top of a cluster and govern the resources itself, or directly connect to an IaaS cloud service to allocate computing resources on demand
NEPHELE Architecture
Figure 5: Architecture of NEPHELE
For details on the architectural components, in includes:
Before submitting a Nephele compute job, a user must start an instance inside the cloud which runs the so called Job Manager. The Job Manager receives the client’s jobs, is responsible for scheduling them and coordinates their execution. It can allocate or deallocate virtual machines according to the current job execution phase. The actual execution of tasks is carried out by a set of instances. Each instance runs a local component of the Nephele framework (Task Manager). A Task Manager receives one or more tasks from the Job Manager at a time, executes them and informs the Job Manager about their completion or possible errors. Unless a job is submitted to the Job Manager, we expect the set of instances (and hence the set of Task Managers) to be empty. Upon job reception, the Job Manager decides depending on the particular tasks inside a job, how many and what kind of instances the job should be executed on. It also decided when respective instances must be allocated or deallocated in order to ensure a continuous but cost-efficient processing
- JobManager
- TaskManager
- InstanceManager
- Scheduler
- MemoryManager
- IOManager
- NepheleGUI
- Client
Before submitting a Nephele compute job, a user must start an instance inside the cloud which runs the so called Job Manager. The Job Manager receives the client’s jobs, is responsible for scheduling them and coordinates their execution. It can allocate or deallocate virtual machines according to the current job execution phase. The actual execution of tasks is carried out by a set of instances. Each instance runs a local component of the Nephele framework (Task Manager). A Task Manager receives one or more tasks from the Job Manager at a time, executes them and informs the Job Manager about their completion or possible errors. Unless a job is submitted to the Job Manager, we expect the set of instances (and hence the set of Task Managers) to be empty. Upon job reception, the Job Manager decides depending on the particular tasks inside a job, how many and what kind of instances the job should be executed on. It also decided when respective instances must be allocated or deallocated in order to ensure a continuous but cost-efficient processing
Advantage
Unlike existing systems (Building a high level language layer that abstracts from underlying massively parallel data processing systems is a quite common concept. For Hadoop there exist several such layers like Apache Pig, Jaql, and Hive.) We chose Nephele for the following two reasons even in principle the presented PACT programming model could be set up to run on top of other DAG-based execution engines as well since the Nephele is benefits in the followings:
- First, Nephele offers a rich set of parameters which allow influencing the physical execution schedule of the received data flow program in a versatile manner.
- Second, Nephele is highlighted by its ability to deal with dynamic resource allocation. The system can independently request new compute nodes from a commercial cloud like Amazon EC2 in order to match the occurring workload. Although this feature is currently only used in a very limited fashion, we plan to leverage it for load balancing and dynamic re-optimization in the future.
Working principle
Nephele supports three different types of communication channels: Network, in-memory, and file channels. While network and in-memory channels allow the PACT compiler to construct low-latency execution pipelines in which one task can immediately consume the output of another, file channels collect the entire output of a task in a temporary file before passing its content on to the next task. Also, the Nephele system executes the compiled PACT programs in a parallel fashion. Nephele considers incoming jobs to be DAGs with vertices being subtasks and edges representing communication channels between these subtasks. Each subtask is a sequential program, which reads data from its input channels and writes to its output channels. The initial DAG representation does not reect parallel execution. Prior execution, Nephele generates the parallel data flow graph by spanning the received DAG.