Efficient and secure real-time mobile robots cooperation using visual servoing

This paper deals with the challenging problem of navigation in formation of mobiles robots fleet. For that purpose, a secure approach is used based on visual servoing to control velocities (linear and angular) of the multiple robots. To construct our system, we develop the interaction matrix which combines the moments in the image with robots velocities and we estimate the depth between each robot and the targeted object. This is done without any communication between the robots which eliminate the problem of the influence of each robot errors on the whole. For a successful visual servoing, we propose a powerful mechanism to execute safely the robots navigation, exploiting a robot accident reporting system using raspberry Pi3. In addition, in case of problem, a robot accident detection reporting system testbed is used to send an accident notification, in the form of a specifical message. Experimental results are presented using nonholonomic mobiles robots with on-board real time cameras, to show the effectiveness of the proposed method.


INTRODUCTION
Robotics is a complex engineering field because associates deep knowledge of several disciplines such as electronic, mechanic and software engineering. This complexity is compounded with the transition to multi-robot systems. The cooperative Multi Robot Control Systems (MRCS) has grown considerably in the last decade, due to the extensive need to such technology in different fields mainly: biomedical science, rescueing, displacement of heavy items, surveillance for example catching invaders under surveillance areas [1], sensor networks and cooperative transport. The idea is that autonomous, collaborative robots can achieve better results then each robot separately [2].Cooperation means that robots must communicate to exchange information and coordinate their actions in order to accomplish a common task [3,4].
However, building a control system for a group of autonomous robots is a very complex work. Among the advantages of the MRCS, we can cite: 1) The system can react quickly to external interference due to the robots division of labour based on environment sharing, since the limitation of the field of vision of each individual mobile makes almost impossible the perception of the robot entire environment; 2) The training of MRCS maintains a specific attack on the outside that can enhance defense capabilities; 3) It can improve the robustness and efficiency of the system.
In the present study, we want to implement a collaborative control system for a group of nonholonomic mobile robots, using cameras aiming at tracking objects and performing tasks based on visual Table 1. Classification of visual servoing systems with respect to several aspects 1.
The position of the camera as eye-in-hand and eye-to-hand 2.
The feedback representation mode position-based, image-based, and hybrid visual servoing 3.
The combination of vision sensor and controller of the joint-dynamic look-and move system and direct visual servo system 4.
The use of the visual information (control model) distinguishes two types of visual servoing systems: kinematics-based visual sevoing and dynamic visual servoing Great efforts have been devoted to numerous applications of visual servoing in robotics in the last decade. It can be used as soon as a vision sensor is available and a task is assigned to a dynamic system to control its motion. Nowadays, visual servoing is widely used in different fields mainly; gaze control for target tracking, navigation of a mobile robot to follow a wall using an omnidirectional vision sensor, grasping a ball with a humanoid robot, assembly of microelectromechanical system (MEMS) and film of a dialogue within the constraints of a script in animation. Vision-based robot control method uses the visual data to control the motion of dynamic systems. A control law has to be considered that the measurements s(t) reach a desired value s * , defining exact realization of the task. The objective is to minimize the difference between the current and wanted configurations.One of the central questions that must be processed in multi-robot system, regardless of the application domain, is how to cooperate effectively and automatically many robots to execute a common task. Because the stability of the visual servoing system [14,15], we must built optimal visual characteristics. Our proposition should satisfy several criteria mainly; stability of the dynamic system, robustness to calibration, local minima avoidance, non-singularity and maximal decoupling and linear link between the visual characteristics and the degrees of freedom. We developed a control law to calculate the velocity components of the robots providing exponential decay of the error. Two main aspects have great impact on the behavior of any visual servoing scheme; the selection of the visual features used as input of the control law and the form of the control scheme. Visual information obtained from the image processing can be used to extracting 2D features. It can also be used for estimating pose parameters by employing pose estimation algorithm from computer vision. The estimated pose is transformed into the 3D features and the 2D and/or 3D features are then used in the control scheme [16]. Hence, optimization techniques, robot dynamic and robot kinematics are used in the modeling of the control schemes. The aims of the present study are firstly, design and develop a new secure synoptic based on the approach of the visual servoing to control linear and angular velocities of multiple robots. Secondly, to show clearly the interest of the accident reporting system, especially in the outdoor experiments.
The remaining sections are outlined as follows: In section 2 we present a brief survey of related works on visual servoing schemes. Section 3, is devoted to introduce and explain the robotic visual servoing system, an emphasis is put on the implementation of the proposed system encompassing several modules. Section 4, we present an illustration of the results obtained in order to achieve our ultimate goal while conclusions and perspectives are offered in section 5.

RELATED WORKS
The interest in using MRCS is due to their characteristics realized with different types of autonomous vehicles such as ground mobile robots, underwater vehicles [17], unmanned aerial vehicles, and aircraft. Multi-robot coordination purpose is to let robots sharing any information between them. For example, a robot position can be shared with others in order to compute more precise other robot position and avoid collision. In the context of computer vision framework, visual servoingtechniques can be used to adjust the trajectories of the mobile robots fleet. The present work consists of defining a method that combines visual servoing and control, to track a group of mobile robots. The behavior of the loop system depends on the choice of visual information and the associated control law. There is a large number of visual primitiveinformationscategorized into two main groups, that that can be used in visual servoing tasks: The first category uses measurements contained in the image plane and will be referenced under the acronym IBVS (Image-Based Visual Servoing). The control law is calculated from image error, so that the behavior in 3D space is not directly constrained. This type of control is robust to summary calibration of the intrinsic camera parameters. Geometric primitives, such as points, segments, lines and ellipses, were the first to be studied. [18] Proposed a control scheme based on sign detection algorithms for mobile robot platform using Harris corner point detector with eigenvalues decompositions to track the path. According to [19], Mebarki proposed moment's image allowing acceptable results in the 2D and 3D behaviors for binarized images. A method for controlling four degrees of freedom has been proposed by [20], requiring image processing and taking into account the pixel value of the image. Homography was integrated in the control scheme to control the six degrees of freedom of robot [21]. Inspired by this work, planning was proposed to ensure robustness to calibration errors for large camera movements [22]. In order to avoid these image processing steps, such as segmentation or primitive tracking, [23] studied a "direct" visual servoing scheme, taking into account the pixel values of the current and desired images as primitive. Kadhim and Abdulsahib [24] designed a vehicle as robot-mounted sensors capable of carrying the sensors of the metal and obstacle. In order to improve the accuracy of identification of object in different illumination and background conditions, however authors do not study the kinematic models of a robot which help to validate or verify by calculation the mechanical performances of a system. Laterone, mutual information between the current image and the desired image was proposed in order to increase robustness to occultation and changes of illumination [25]. Another criterion based on the difference between the current image and the reference image, updated according to illumination of the current image, was proposed to manage any changes in lighting and mult imodality [26].
The second approach, PBVS (position-based visual servoing) uses pose estimation after localization algorithm [27]. The 3D space is well constrained, however this is no longer the case for the image space, which may lead to output primitive image. Furthermore, to properly do the 3D work, it is important to accurately calibrate the camera. The main problem of the PBVS approach is that the geometric model of the object should be known for pose estimation, which makes it a "model-based" method compared to the image-based method. The camera calibration is required to obtain the unbiased Cartesian positioning to overcome the sensitivity of the camera calibration error.Hybrid approaches combining 2D and 3D primitives association their advantages to benefit from a better 3D behavior [28] or to keep the primitives in the field of view [22]. Selecting good visual characteristic is a crucial aspect of visual servoing as it is necessary for achieving optimal velocities and increasing accuracyand reliability of image measurements, affects performance and robustness of visual servoing [29].
Imaging measurements are either used directly in the control loop or used for relative pose estimation. The number of degrees of freedom (DOF) to be controlled by the employed control scheme determines the minimum number of independent features required.Visual features can be selected in 2D image space as point coordinates, parameters representing straight lines or ellipses, region of interest and contours [30,31]. These features are defined from image measurements [32]. In case of image points, Cartesian coordinates are generally used however, it is possible to use their polar and cylindrical coordinates [33]. In general, all parameters defining the internal camera calibration are required.
Image moments can also be used in visual servoing [34] gives better results compared to the classical visual servoing scheme. Image moments allow generic representation and are able to handle simple geometrical primitives and also complex objects with unknown shapes. It is shown that moment invariants can be used to design decoupled 2D visual servoing scheme and to minimize the nonlinearity of the interaction matrix related to the selected visual features [35]. Generally, object model and image measurements are used to calculate or the relative pose between object and camera frames in the Cartesian space, or to reconstruct the 3D coordinates. Therefore, an advanced knowledge about the camera calibration parameters are required [36].
In this section, we review some visuals aspects and primitives information that can be used in visual servoing task. Also we study some multi robot scenarios design and analysis. We can enumerate the contributions of our paper as follows:  Proposal of a MRCS based on visual servoing, including the design and development of an experimental protocol allowing the collaboration between mobile robots following a master robot that is based at first on the determination of each robot computed position.  The system is able to allow multi robots naviguating in formation using the kinematic model of mobile robots and a camera to construct a command law. As visual primitives in the servoing loop, we use the moments extracted from the current image.  Our work is well-suited for real applications as it is robust, and leads to a fast implementation of the multi robots visual servo control issue.  It shows clearly the interest of a robot accident detection reporting system.

ROBOTIC VISUAL SERVOING SYSTEM
We developed an experimental protocol to control a group of robots based on the information provided by cameras mounted on each robot. A desired value in the image is determined for each robot independently of the other robot. The visual servoing aim is to control the current value by following the desired value in the image by estimating the depth Z between the object and the camera. The following Figure 1 shows the diagram of our visual servoing system.

Visual features
The information collected by the vision sensor decreases the stability problems if the movement carried out by the robot is complex. Therefore, selecting the appropriate information is important to accurately applying the required task. While several choices of s exist, we have chosen in the present work the invariant moments, as visual information. Image moments can be computed from a set of points or a well-segmented region in the image. The geometric moments of a 2D distribution function f(x,y) can be expressed as: with f(x, y) ≥ 0 be real bounded function with support on compact region R.R is the area occupied by the object in the image and p, q is the moment order. In this paper, binary image functions (i.ef(x, y) can only take 0 or 1 value) or image regions, defined by closed contours are considered. Invariance to translation and scaling can be computed from moments (2) as follows:  (4) where: 10 01 00 00 gg mm x and y mm  (5) x g and y g represent the coordinates of the gravity center, and m 00 =a=n is the object area. The main orientation is obtained from the second order centered moments [

Nonholonomic mobile robot
The unicycle mobile robot as shown in Figure 2 is typically a nonholonomic system [38] and can be described by the nonholonomic constraints. There are two types of constraints: the rolling and the sliding constraints.

Figure 2. Unicycle mobile robot
The state vector q=(x,y,) denotes the posture of the robot. (x,y) represents the mass center of the robot. is the orientation of the robot according to the horizontal axis. ν(t) and ω(t) are the translation velocity and the angular velocity, which are used as the control inputs. The nonholonomic constraint for the mobile robots is given by: A general kinematic model of the mobile robot obtained from the nonholonomic constraints is given by the following equation:

Control law
The aim of the control scheme is to eliminate the errors between the initial and the desired positions of features on the image plane. The expression of these errors is defined as: (10) s is the vector of visual feature. Model of the controller is velocity controller and it can be computed as: where represents the velocity vector of the camera including translational and rotational components and is pseudo-inverse of the approximation of image Jacobian matrix.

Interaction matrix
We assume that the object is a continuous surface, therefore, the depth Z of each 3D point is expressed as a function of thex and y coordinates of its projection in the image. More specifically, we have: In the case where the object is planar or has a surface of the planar limbs, its equation in the camera frame is expressed by: using the equations of the perspective projection , we can deduce: with: for each point with coordinates x=(x,y) in the image whose corresponding 3d point has depth Z, we have: where, the interaction matrix Lx is given by: using (15) in (18), (17) can be written as: (1 ).
. .  In the matrix Lx, any control scheme that uses this form of the interaction matrix must estimate or approximate the value of Z. From this equation, we obtain: (20) we have: The interaction matrix associated with the moment m ij can then be determined by: From the general form given in (22), we deduce the interaction matrix of the surface of the object m 00 (we consider i=j=0), and the interaction matrix related to the coordinates xg and y g of the gravity center of an object in the image.The visual servoing algorithm that we present below is based on the ideas proposed by [14,23,35] with modifications made for our application.

Algorithm
Begin 1: Set the parameters used;//λ=0.6 2: Connect to the master robot and Client 1 and Client 2; 3: Determine camera parameters (calibration); 4: Acquiring an image; 5: Image display; 6: Create the surface to follow (the target); 7: Calculate the moments in the image; 8: Determine the coordinates of the desired target to follow (for each robot); 9: Calculate the interaction matrix L x =S xd ; 10: Determine (Z/Z*); //Z*: the desired depth of the form is learned and equal to the initial depth; 11: Calculate //Transformation of the camera frame to the effector of the mobile robot, this transformation makes it possible to calculate the velocities expressed in the end effector frame into the camera frame; 12: For (i=0 ;i<=n ; i++)//n : is the number of images in the video 13: For (x=1; x<=y; x++) // y: is the number of server robots Begin Begin

15:
Calculate the area to follow; (calculating the moments in the image)

16:
Update the current x features;

18:
Update the interaction matrix;

21:
Send the velocities to the robot; End For End For End.

EXPERIMENTALRESULTS
The experiments were performed with two robots Pioneer 3-AT and Pioneer 3-DX equipped with a camera. The image resolution in the experiments was 640x480. All computations, except for the low-level control, were performed on a laptop with 2.5 GHz Intel Core (TM) i5-4200U CPU, with 4-GB RAM. The navigation experiment was performed online. Oursystem is implemented under Visual Studio with the help of Aria and OpenCV library. However, the robot reporting system is implemented with Python 3. The experiments were performed in an indoor environment inside a laboratory and a corridor with λ = 0.6. The pioneer 3DX and 3AT mobiles robots are unicycles non-holonomics with differential drive WMR. P3-DX has 2 independent driving wheels, that canno't be steered on the same axis and free swinging off-center wheel. The whole system summarize in Figure 3. We develope a server application using the C++ programming language which is installed in the robot PC. It performs the several functions, mainly: managing the communication to and from the medial server by sending information about the state of the robot and receiving instructions from the user, controlling the mobile robot movements and managing the mobile robot sensors.
The developed system consists of accident reporting system. Forthe realization of our system, we used the following equipment numbered as shown in Figure 3 and  Based on the RPi 3 which is used to communicate with a GSM and a global positioning system (GPS) module using the UART link, different sensors were used to ensure the detection of the accident. The GPSprovides a three dimensional position as well as current time and date and is available every where. [39,40] Using GPS, GPRS and GSM Technology for simple accident detection but in our case, our system proposed ADRS (Accident Detection Reporting System) allowing the robots navigating safely,thing that remains a challenging task in many research works. The developed system was attached to the top of the robots in order to detect accident in the real time, record the accurate location of the accident and send an automatic alert message to the emergency center [41]. The system was powering up and different sensors were tested. Each of sensor commands the Raspberry Pi3 to generate a specific SMS alert with the type of the accident. The testbed detects two cases of reversal: Right turnover, Left turnover as shown in Figure 5. We used the accelerometer data to calculate the inclination of the robot along the X axis. The value >= 46° indicates that the robot is overturned on the right. If the inclination along the X axis is <= -70°, this implies that the robot is overturned to the left.
We used a lighter to light fire, the flame sensor was checked using norml lighter and the sensor succesfull detected and reported the fire to the Raspberry Pi3 Figure 6. The latter will generate a warning message indicating that a fire has occurred. The shock sensor was tested by creating small obstacle to report the collision (Figure 7). The sensors where able to detect and report the problems in time not exceeding 20 seconds. The robot master P3-AT carries a planar object (Figure 8). After the blob detection, the contour of the interested region is obtained to further acquire the centroid, which will be conducted in each control loop.The moments are computed at the video rate after binarization of the image, the results are shown as Figure 9, we successfully detect the contour. The image acquired at the desired camera position is displayed in Figure 10. The robot trajectory is satisfactory using the visuals features, in our contribution we have to estimate the depth Z for each iteration. The effectiveness of the developedsystem depends on the precision of matching between the current and desired position of the visual information, as shown in Figure 11(a) and Figure 12(a) which confirm that the system is stable and converges to the desired values.The calculated error between the current information and the desired information (for x and y) is relatively unchanging during the first 3ms, this is due to the non-holonomic nature of the robots. However, at 32-37ms the system converged to the right position and the final error was 0.01.In this study, we discussed the motion control problem and visual servoing of non-holonomic wheeled mobiles robots.We presented kinematic model of the robot in addition to their control properties. The simulation results show that the developed control law controls the velocities of all used robots in a robust and precise way. The developed controller is able to drive a set of wheeled mobile robots to a target object until the cameras observe the desired visual characteristics as shown in Figure 11(b)- Figure 12

CONCLUSION
We presented in this paper the visual servoing problem that is able to determine an object tracking, using the kinematic models of mobile robot and a camera, a proportional control law is developed. The derived controller, which is form of image-based eye-in-hand visual servo controller, guarantees asymptotic stability of the closed-loop system. The developed mobile manipulation system is proposed for use in multi-robot cooperative assembly project. Experiments show that the proposed methods improve the overall performance and increase flexibility and precision of our robotic system dynamic channel closed. We propose control strategies that allow mobile robots to perform task of tracking a target linked to a master robot independently.
As a very important step for the autonomy of the visual control, we present the problem of the control and the realization of the tasks based on vision for the navigation of mobile robots and the execution of the complex movements The developed multi-robot cooperative assembly system is intended for use in an unstructured environment with complex terrain and dynamic obstacles distribution, we propose as future works to study the stability analysis while applying the 3D parameters. In the event of failure of the GSM/GPRS type communication network to trace the notifications to the emergency response services, an alternative is to use the vehicular network (VANET) by exploiting the offered communications type V2V(vehicle-to-vehicle) or V2I (Infrastructure vehicule). This is a promising area of research to explore.