Cooperative hierarchical based edge-computing approach for resources allocation of distributed mobile and IoT applications

ABSTRACT


INTRODUCTION
First, let us define Internet of Things (IoT), as devices that can receive or transmit data using transceivers, such as wireless sensors, vehicles, actuators, smart grids, or any smart device, in a way that connects devices together. Edge computing paradigm (or similar platforms as Mobile Edge Computing (MEC), fog computing or cloudlet) was introduced, due to the fact that the mobile and IoT applications demands and resources requirements have increased, e.g.; computing platforms [1][2][3][4][5]. Consequently, the necessity of a new platform that supports these applications demands within a close proximity of computational and storage resources, within a geographically distributed area, are really a necessity.
In contrast to the current cloud paradigm, a centralized infrastructure is more suitable for regular personal computer's applications rather than the requirements nature of mobile and IoT devices' applications.  [20] proposed a theoretical-software defined based framework. Their framework integrates mobile-edge servers with software defined controlling system of local and global layers. Both layers are provided with multiple controlling units including network entities, storage, security, computational resources, data aggregation and an IoT unit for controlling and monitoring IoT sensors and actuators. Specifically, local layer serves time sensitive applications in the local domain, while the global layer serves applications that require data aggregation from several mobile edge servers.
Du et al. [21] designed an application-specific MEC platform that optimizes the Mobile Virtual Network Operators (MVNOs) by applying software defined data plane paradigm. Authors of [22] integrate MEC with the emerging technologies of Software Defined Network (SDN) and Network Functions Virtualization (NFV), in order to achieve better edge and IoT resources deployment and management. In the same context, Ravindran et al. [23] merged SDN, NFV and ICN to design an architecture for edgecloud services. Moreover, authors in [24] developed adaptive routing schemes for both ordinary and emergent delay requirement of data transmission in industrial IoT framework that combines edge computing and software defined network. Another work that combines SDN with MEC to support reliability, agility, responsiveness, and application specific requirement of dynamic vehicular network is proposed in [25].
Regarding communication stage, Channel State Information (CSI) is considered by [26] for control and management schemes in MEC systems, taking into consideration environment effects, e.g.; fading, on wireless channel state, which affects transmission data rate, and transmission energy consumption regarding the allowed transmission latency and suitability of channel state for appropriate computation offloading decision by the controller.
Researchers have paid an attention to management and edge resources' provisioning, Hu et al. [27] examine offloading computation impact from the mobile devices to edge's node compared to the exsiting system which offload tasks' processing to the cloud. The results show an improvement in energy consumption for mobile devices and response time depending on the offloading distance. Authors in [28] proposed a dynamic services migration policy in mobile edge-clouds, using a distance-based Markov Decision Process (MDP) framework. This framework approximates user's mobility movement model in 2-D space. They confirmed the superiority of their model analytically and experimentally over various baseline models, such as: no migration, always migrate, and myopic. In [29], a framework is presented for resources management in datacenters, in which resources assignment is enhanced based on users' behavior, type of service, and price.
Resources provisioning approach is proposed in [30], its goal is to maximize fog resources utilization. A hierarchal-fog framework is introduced that provides resource-controlling services in both the cloud and the fog. In this framework, fog colony has a set of fog cells, modules of IoT, and one fogcontrol node. It has three layers, such that the first layer has multiple fog colonies. The second layer has a fog-control node that orchestrating and connects fog colonies together in the first layer. The third layer has the fog-cloud control middleware that controls cloud services. Control nodes provision cloud resources in order to maximize their utilization, also guarantee end-user requested service and forward it to a higher network level.
Kapsalis et al. [31] proposed a fog platform that has four layers: First layer, or the lowest layer, contains IoT devices. The second layer contains gateways named hub layer that connects devices in first layer. The third layer is the fog layer that has a fog broker and fog servers that manages resources. The fourth layer is the cloud layer. Authors proposed a workload-balancer module implemented in the fog broker, its role is to balance fog resources utilization based on the remaining batteries power for mobile devices, latency, and current utilization.
A Message Queuing Telemetry Transport (MQTT) protocol is used for communication in WSNs [32], where exchanged messages contain a code that will be executed, required data, and metadata that specify tasks characteristics. The MQTT protocol is used for communication in the proposed SDN-based fog computing system [33]. A customized integration of edge switches, named fog nodes, with SDN controller and a MQTT broker is developed. Fog node receives MQTT massages from the IoT devices and publishes their data with a certain topic, or type. After that, the fog node transmits the message to devices requesting the same type. Moreover, they proposed conducting analytical analysis in the SDN controller such that it acts as the central node.
Addressing the energy consumption problem, authors of [34] proposed an approach for dynamic tasks assignment in heterogeneous-mobile-embedded systems and cores in mobile clouds that reduces the total energy cost using cyber-enabled applications. Furthermore, for green computing, authors of [35] proposed a mechanism to prevent energy wasting in mobile devices while using dynamic wireless communication. This mechanism is based on cloudlets layer between mobile devices and the cloud, performing a dynamic search to achieve better communication experience between mobile devices and the cloud. Moreover, authors in [36] proposed an energy aware scheme for load balancing and scheduling in smart manufacturing based on fog platform. Regarding security, authors of [37] proposed a solution for security and privacy concerns, in order to maximize security and privacy levels for transmission, while maintaining a successful connection probability within a given timing constraints using multi-channel communication.

OUR PROPOSED APPROACH
The main goal of this work is increasing edges' resources utilization using a cooperativehierarchical based approach; therefore, this work proposes: (a) An edge-computing platform architecture that facilitates communication between edge's entities. (b) An approach with a polynomial-time complexity, in order to distribute the pre-partitioned application modules between edge resources.

System architecture
In this subsection, the platform for edge computing and edge node structure is discussed as follows:

Platform for edge computing
The structure of the proposed platform, as illustrated in Figure 1, consists the core network and three layers: (1)-End layer, (2)-Edge layer, (3)-Cloud layer. The end layer is the lowest layer that has mobile and IoT devices, e.g.; sensors, with poor or limited resources, e.g.; computing platform and memory, where nodes in this layer cannot perform tasks with a heavy computation.
Above the end layer is the edge layer, or the middle layer, that has a geographically distributed edge nodes with computation, communication, and memory resources in a close proximity to the end layer devices. This layer can perform tasks independently, and running pre-defined applications on the behalf of the lower layer, or the end layer, devices in a fast manner to reduce latency, and also to provide efficient streaming without the assistance of the cloud layer.
The upper layer is named cloud layer, which is the farthest layer from users-devices in this platform. Due to this architecture nature, this work goal is to minimize communication between the cloud layer and other layers, namely end and edge layers, in order to reduce number of routing requests through the core network. As a result, the traffic amount through the core network, used bandwidth, and congestion are all reduced. On the other hand, cloud resources could be used as data storage for a long-term to be analyzed in depth. Moreover, if edge-nodes' resources are not capable of executing some tasks, thus, the edge node forwards these heavy processing tasks' requests to the cloud as a final resort.

Edge node structure
Each edge node owns storage, computation, and communication resources. These resources can be heterogeneous and distributed in a hierarchical way, reducing the distance to the cloud to a certain extent, where less powerful resources are located closer to end devices. Moving away from end devices, the more powerful resources become available. In the proposed approach, as discussed in details in our proposed approach description, Subsection 4.2. Tasks are distributed such that light-computation tasks are executed close to the end devices, while tasks with heavy computation requirements are executed by upper level resources, as shown in Figure 2.
Edge nodes might vary in their available resources, and hence depending on the type of task's requirements, the suitable edge node is selected. Edge nodes that provides heavy computational services, such as gaming or augmented reality, should be provided with more resources than nodes that provide lightcomputation services. This design shows the necessity of the proposed cooperative-hierarchical based approach, e.g.; if a node with light resources is flooded with task requests and has a neighbor node that has more resources that currently are underutilized. As a result, some tasks are forwarded to this neighbor node rather than forwarding them far to the cloud layer. Clearly, this reduces traffic in the core network and enhances response time. For example, when deploying systems in wide regions, such as a smart driving assistance, they can benefit from our proposed approach.
When several edge nodes can provide the same requested service, they might be distributed to cover a wide region; however, the amount of traffic is not the same for different areas. When employing our proposed approach between neighboring nodes, the utilization of edge nodes' resources is increased and interacting with the cloud layer is reduced, as demonstrated in Subsection 4.2.2. In this work, it is assumed that each edge node also consists an edge-orchestration node that represents the main component in the edge node. This node buffers information about each running task, such as its running host, the current state of computation and memory resources, in addition to information about the connection state with neighbor-edge nodes and the cloud. As shown in Figure 2, there is a node named orchestration node that is responsible for resources management, making decisions regarding executing tasks on its node's resources, or transferring this task to a neighbor node or even forward it to the cloud layer.

Our proposed approach description
In order to improve edge-node resources utilization, a cooperative-hierarchical approach is proposed in order to distribute and execute application modules between neighbor-edge nodes.

Application
An application to be executed on edge's resources, it must be pre-partitioned into modules; such that each module conducts a specific function. The proposed approach does not partition the application, but some other entity, e.g.; compiler, is responsible for dividing the application into modules. The complexity of each module is defined by number of parameters, such as required RAM size, required CPU time, and transmitted data size. In order to represent the application using a directed graph, a distributed data flow model is used [38], such that each module is represented by a vertex and a directed edge represents the dependency and data flow direction between modules.

Algorithm for modules placement
The pre-partitioned application modules are received by the edge-orchestration node, in order to be distributed to the available resources based on their needs of memory and processing requirements. The baseline module placement method, that used in iFogSim simulator, depends on spanning the constructed graph from the bottom to the top within the same edge node, that is searching for a suitable resource to assign each module. However, if there are no resources satisfy the module needs, therefore, it will be forwarded to the cloud. In [30], a system model is proposed that has utilization and latency metrics for the resources inside the same node, however, the utilization and the delay caused by neighbor nodes are not considered. In this proposed work, the platform is spanned horizontally searching for suitable resources in neighboring nodes, as assigning the module to the cloud is the last resort. The proposed mechanism for module's placement is shown in the below flow chart, shown in Figure 3, and is explained as follows: 1. In the edge node, the orchestration node keeps leaf-to-root paths for the hosts, also it keeps tracking the current CPU utilization (busy time) of each host. The orchestration node represents hosts inside the edge node by a tree structure based on their computing power, such that the least powerful hosts are the leaves and the most powerful host is the root. Apparently, a leaf-to-root path has the possible hosts of executing a module; where the orchestration node visits each path from leaf-to-root. As a result, modules are executed on hosts with less power, if they satisfy modules requirements, that closer to end devices, before moving them to the upper level in order to look for available resources. 2. The orchestration node schedules the order of executing the modules in the edge node based on the dependencies between them, e.g.; a module may need some modules results as input data. We developed a platform for the experiments that contains two edge nodes, where each edge node has one orchestration node, one edge server and a group of mobile devices that varied through experiments from 10 to 100 devices. Figure 5(a) shows that the proposed edge-computing platform with the orchestration nodes and the communication delay between components. It is implemented using iFogSim toolkit.

Performance metrics
To evaluate our proposed approach, we introduced the following performance metrics: 1. Total delay: is the total time delay that required for executing all tuples (tasks) including communication delay, due to the fact that these tuples travel from one module to another on different hosts and/or different edge nodes. Therefore, the total time delay includes network communication, processing, congestion, and collision time. 2. Throughput: is the total number of data tuples executed per time unit. It is evaluated by using (1), where T denotes total number of the executed tuples and D denotes the total delay.
3. Network load: is the amount of data tuples that travels in the network and occupies it for a certain amount of time. It is evaluated by using (2), where Nd i denotes the total delay of the i th tuple and Nw i denotes the i th tuple size.

∑ (2)
Notice that the network load includes congestion and collisions. Therefore, network load reduction leads to reduce collisions and traffic congestion. As above (1) and (2) demonstrate, throughput is inversely proportional to the total delay, while network load is proportional to the network overall delay. The simulation results in Subsection 5.3 prove that the throughput is influenced by reducing total delay for tuples execution more than network load. For consistency, these metrics are measured for the same number of tuples executed in both systems implementation: the proposed approach and the baseline approach, as their architectures are shown in Figure 5(a) and Figure 5(b), respectively. In simulation, when number of devices in edge nodes is increased, clearly, more tuples are produced, and hence, the assessment validity for introduced performance metrics is enhanced.

Simulation results and discussion
In our proposed cooperative-hierarchical placement approach, all mobile devices have same capabilities allows them to execute only the clients' modules, while the edge servers have different capabilities such that some servers can execute the concentration-calculator module while others cannot. Therefore, in the proposed placement approach, a server with enough computing power can execute the entities of this module. On the other hand, regarding the baseline placement method, all entities of this module are executed at the cloud which may cause a higher communication delay.
A preliminary result of this work is published in ICECTA'2017 [40], the initial enhancement results are presented for the proposed approach when 40% of nodes are closer than the cloud. However, in this extended work, more extensive results are presented as follows: IoT devices in the end layer, than the cloud. 2) Extensive experiments are conducted for total delay, throughput, and network load metrics. 3) Moreover, this work demonstrates how different performance metrics are affected by neighbournodes' position or edge nodes proximity to end devices.

Average delay
Delay enhancement results is shown in Figure 6, for example, results show that if 10 devices emanating data, a 60% reduction in latency is achieved when placing modules on neighbor-edge nodes that are 40% closer to devices than the cloud. This reduction in delay is due to reduction of traffic directed through the network core towards the cloud, imposing less network congestion, leading to a lower delay, and that is due to employing our proposed approach rather than the baseline approach.
Delay reduction percentage is going down to 36% when there are 100 devices emanating data, because congestion and collision probability will increase in the network causing more delay. Clearly, delay reduction decreases as number of devices increases. That makes sense, due to the fact that when number of devices increases, available edge nodes resources become not sufficient to satisfy their resources requirements. Therefore, some tuples are forwarded to neighbor-edge nodes or even to the cloud resources through the network core. As a result, the overall communication overhead and data transmission time increase, in other words, average delay increases. Furthermore, Figure 6 shows that when 70% of neighboredge nodes are closer to mobile IoT devices than the cloud, enhancement results are better than when 40% of neighbor-edge nodes are closer. In contrast to the scenario when only 30% of neighbor-edge nodes are closer to devices, delay enhancement is less. Figure 7 proves that our proposed approach improves the throughput significantly. It is interesting to see that the proposed approach enhancement ratio is more than double the enhancement for delay, because both network delay and congestion are reduced. Also, when the number of devices increases, more tuples are generated, and thus, throughput enhancement is degraded. An interesting observation is noticed, as the number of devices increases, the enhancement reaches a saturation level. Even with this saturation, the proposed approach outperforms the baseline approach.

Network load
Network load enhancement means reduction in network congestion and collisions. As predicted, the proposed approach decreases the network load, since fewer packets will be routed through the core network towards the cloud. Figure 8 illustrates that reduction in network load is inversely proportional to number of devices in the platform. A noteworthy observation about network load is as number of devices increases the enhancement decreases, due to the fact that more tuples are generated that cause more network congestion and collision possibility.

CONCLUSIONS AND FUTURE WORK
Edge computing has a crucial role for supporting the operation of mobile and Internet of Things (IoT)-based applications. Edge computing supports applications that requires high computing capabilities, and/or fast response demands; also it plays a major role for applications that based on location awareness. This work goal is to improve edge resources utilization; therefore, we proposed a cooperative-hierarchical based approach for modules' placement in edge node and between neighbor-edge nodes, where this approach has a polynomial-time complexity. Also, this work introduced the architecture that facilitates communication between edge nodes. In simulation experiments, iFogSim toolkit [38] is used. The proposed module placement algorithm is executed only during application deployment phase. Simulation results show that adopting the proposed approach improves the overall of platform performance by reducing the overall latency, reducing network congestion, and improving throughput. Moreover, the end-user experience is improved. As a future work, we plan to study cases with a dynamic environment and study the effect of rapid changes in the environment, and how this influences end-user experience.