Published on: Feb 23, 2023

Prologue

Most of this write-up is based on an old proposal I submitted back circa 2014 for funds. However, I hope to continue the research & development, when I get an opportunity, to extend the edge-AI work in UAV domain in one of the following directions (priority based):

1. Development of autonomous unmanned aerial vehicles (UAVs) that use machine learning and computer vision to perform surveillance, reconnaissance, and target acquisition.
2. Use of big data and machine learning algorithms to identify and predict security threats, such as cyber-attacks and physical breaches, in government-related technology systems.
3. Application of machine learning and computer vision to automate the detection and classification of pests, diseases, and other anomalies in agricultural crops to improve yield and reduce losses.
4. Use of machine learning and NLP to develop natural language interfaces for military personnel to access and process vast amounts of data more efficiently and accurately.
5. Development of machine learning algorithms for autonomous decision-making in defence systems, such as target prioritization, route planning, and resource allocation.
6. Integration of IoT technologies and big data analytics to improve the situational awareness of military operations, including the tracking and management of military assets and personnel.
7. Development of deep learning models to improve the accuracy and reliability of remote sensing data for precision agriculture, enabling more efficient use of resources and increased crop yields.
8. Use of machine learning and computer vision to automate the monitoring and analysis of environmental factors, such as weather patterns and soil moisture levels, to support more informed decision-making in agriculture.
9. Exploration of the potential of machine learning and big data analytics in educational technology to improve student learning outcomes and personalize instruction.
10. Use of IoT technologies and machine learning to develop smart classrooms that can optimize learning environments for individual students based on their preferences and performance.

The theme of these research ideas is in the domains of defence/security, agriculture, and education. These ideas mostly use deep learning (computer vision, NLP) techniques in conjunction with a UAV platform, as such, edge-AI skills/experiences/concerns are the most relevant in this context due to UAVs being payload/energy-constrained.

The rest of the write-up is the original proposal of 3D path planning in $\mu$ UAVs; with minor comments from [Future ME].

Introduction:

$\mu$ Unmanned Aerial Vehicles ($\mu$ UAVs) are small man-portable air vehicles (usually of around 20 cm in size) which are widely used in various contexts; for example:

(a) In military contexts, $\mu$ UAVs are used for missions which involve target and decoy, reconnaissance, combat and logistics operations.

(b) In the civilian context, $\mu$ UAVs are deployed for environmental and agricultural purposes such as weather forecasting, storm, bush fire detection, farm field seeding etc.

However, all $\mu$ UAV use cases have common tasks which require human operator involvement. One such task performed by human operators is Path Planning.

Definition [Path Planing]: is to find an optimal collision-free path, amidst obstacles, for a $\mu$ UAVs from the source point to the destination point.

Planning consists of finding a sequence of actions that transforms some initial state into some desired goal state. In path planning, the states are $\mu$ UAV locations and transitions between states represent actions the $\mu$ UAVs can take, each of which has an associated cost. A path is optimal if the sum of its transition costs (edge costs) is minimal across all possible paths leading from the initial position (start state) to the goal position (goal state). A planning algorithm is complete if it will always find a path in finite time when one exists, and will let us know in finite time if no path exists. Similarly, a planning algorithm is optimal if it will always finds an optimal path.

Statement of Problem

In order to survey small size areas or non-moving scenes the flying robot must be able to perform a stationary flight. In addition, it must be rapidly deployable. The project will therefore concern rotary wing unmanned aerial vehicles (UAV)--e.g. Quadcopters. This machine will be a small-size VTOL (Vertical Take Off and Landing) UAV designed for autonomous inspection and survey tasks in urban areas or more generally in a constrained outside environment (known and unknown environment with moving obstacles). Using the autonomy faculty of the $\mu$ UAV, users will be able to devote their attention to the exploitation of the data coming from the mission sensors and not to the navigation of the vector.

Most often $\mu$ UAVs have a Small onboard computer which is used for various avionics-related tasks, due to weight and limited battery power consideration. However, the functionality provided/executed by these small onboard computers is limited and not suitable for computation-intensive features/algorithms like path planning, object detection & tracking etc. In order to do autonomous path planning on these $\mu$ UAVs we need a computational platform (Hardware and software) which is capable of real-time path planning with robustness— besides being power efficient (due to battery limitations) and low on weight. We propose using a System of Chip (SoC) approach for the development of a robust path planner; wherein we have an SoC with an on-chip energy-efficient processor (like ARM core) and some FPGA fabric for dedicated implementation of resource-intensive logic (object detection, path planning, tracking etc). As such, the central hypothesis of this research project is that the overall performance can be improved by implementing robust Path Planning algorithms on a SOC. This hypothesis is based on the fact that with the FPGA parts, the ARM processor can have a much bigger data throughput capacity and greater parallel computing ability, which is crucial for this type of application.

Objectives and expected outcomes

The heart of the project is the research and development of software and hardware modules for the autonomy of small-size drones in terms of Path Planning for navigation with robustness to unexpected events. To reach this goal, different scientific fields will be addressed: mission planning, collision avoidance, trajectory and localisation algorithms, and drone hardware.

Technical objectives and expected results:

The users will use a dedicated Mission Control System (MCS) with Man Machine Interface (MMI) to define the start and endpoint that the drone will follow autonomously and the target areas to focus on.

This MMI is the main link between the drone system and the user. The objectives and outcome can be summarised as:

• display a map of the area to explore as a 2D or 3D view of the environment. It may be connected to a Geographical Information System.
• enable the user to define the start and end points of the trajectory which will be used by the Path Planner to generate the trajectory to be followed by the drone.
• display the estimated position of the drone and the data coming from the mission sensors (video, IR, ...) and the estimated battery autonomy.
• obstacle avoidance will be achieved based on volumetric data from the Lidar or Kinect sensor, without explicit object recognition.
• the robust planner in 3D using probabilistic and Evolutionary techniques for unseen events.
• the parallel and energy-efficient realisation of hybrid Path Planning algorithms.
• Applicability study of SoC for Avionic applications
• Real-time volumetric analysis of 3D space.

Review of Literature

Hota et al in [4] address optimal path planning in three-dimensional space for an aerial vehicle with a bounded turn radius. However, the objective of their work is to calculate a feasible path of minimum length when the initial and final positions and orientations of the vehicle are given. The proposed method is based on 3D geometry. However, even though the approach has simplicity and low computational requirements this approach cannot be implemented on an $\mu$UAVs with unconstrained turn radius, i.e. particularly rotary-type air vehicles. Carsten et. al [5] present an interpolation-based planning and replanning algorithm that is able to produce direct, low-cost paths through three-dimensional environments. The algorithm builds upon recent advances in 2D grid-based path planning and extends these techniques to 3D grids. Thus, presenting a number of results demonstrating its advantages and real-time capabilities. However, the robustness of this approach to dynamic changes in the environment is unknown. Fichter et. al in [1] presents a path-planning approach for UAVs, however, this approach is applicable to fixed-wing UAVs and not to the rotary type UAVs. Moreover, these approaches have not been realised in the SoC and their performance in resources-constrained computers is to be investigated. Hachour O. in [2] present a Genetic Algorithm based approach for path planning in autonomous robots. The applicability of this approach is not extendible to UAVs which require planning in the 3D space. Moreover, Genetic Algorithms may not necessarily provide an optimal solution like that of A* or D* algorithms. This work will take more of a hybrid approach and will improve upon the work of Kok et. al[3]; wherein issues with optimality and robustness will be incorporated.

Significance of Study:

The results of this research project will augment various emerging realities [Future ME: I don't claim to see/know the future. But, I think the significance of UAVs is mostly realized and recognised after the Nagorno-Karabakh conflict between Armenia and Azerbaijan. However, many strategists I believe were pointing to this reality a long time ago] and national and regional plans. As such, the results of this work could contribute in the following areas:

Scientific Research: can assist science where the desired locations of study are either remote or dangerous. An example of such areas includes the Fukushima Nuclear disaster, hazardous material sites, and volcanic eruption sites.

Disaster Prevention and Management: can be used for monitoring flood situations, and accident sites and mitigating the situation efficiently and effectively. Moreover, we could use this technology for traffic management during traffic jams.

Environmental Protection/monitoring: this can assist in monitoring illegal fishing, illegal logging in our forests, wildlife protection, and water resources protection.


Procedure/Methodology

Flying in a constrained (obstacles etc) environment under out-of-sight scenarios poses new challenges for the control and navigation of $\mu$ UAVs. A collision avoidance system is the first step in the direction of autonomous flight capability. Kinect [Future me: due to funding constraints I have to resort to using Kinect for this POC] is used for real-time dense 3D mapping. It is geared towards games and augmented reality, but could also be of great use for robot/UAV perception. Therefore, the acquired depth details are used to generate a 3D map of the space; i.e a space for which we desire an optimal navigation path. Based on the user-specified start and endpoints, the generated 3D map and user-specified coordinates will be used by a Path Planner to find an optimal path— various path-finding algorithms like A*, D*, Random Graphs, Genetic Algorithm Planner shall be compared for performance evaluation (robustness, energy use, parallelism etc) and a hybrid planning algorithm will be developed. The planner, for a chosen path, will output a sequence of moves for the autopilot of $\mu$ UAVs, (e.g. PX4 board for autopilot used in Parrot AR drones) will be a trajectory of the $\mu$ UAV in the space which it is navigating.

Impact

This research has a direct impact on the applicability of $\mu$ UAVs in day-to-day uses, from courier service to aerial surveying(land, flood, terrain etc)— in the civilian domain. In the recent past, the autonomy of $\mu$ UAVs have become as much of a selling point in the UAV industry. However, today’s systems are still designed with user intervention for Path Planning, to some degree. I am committed to studying leading technologies for Onboard Realtime Path Planning solutions, complete with an evaluation of existing techniques, a hybrid extension of the evolutionary algorithm approach for 3D space, and complete documentation— which meets the increasing demand for open knowledge and makes the technology available for civilian use with greater autonomy.

Financial Details [Future me: redacted the funding numbers]

1. SoC Board (Xilinx Zedboard) ————————————————— around XXXX INR
2. Volumetric Sensor (Microsoft Kinect) —————————————— around XXXX INR
3. Display Panel (HDMI LCD TFT Touch panel) ——————————- around XXXX INR

[Future me: nvidia Jetson is a popular choice and works well. However, I was trying to implement the custom FPGA based HW version of the critical components; this is not available in nvidia jetson that only provides gpu accelaration and can be limited by CPU-GPU memory bandwith]

The reason for choosing ZedBoard is that it is an evaluation and development board based on the Xilinx Zynq-7000 Extensible Processing Platform (EPP). Combining a dual Corex-A9 Processing System (PS) with 85,000 Series-7 Programmable Logic (PL) cells, the Zynq-7000 EPP can be targeted for broad use in many applications. The ZedBoard’s robust mix of onboard peripherals and expansion capabilities make it an ideal platform for both novice and experienced designers. The onboard memories, video and audio I/O, dual-role USB, Ethernet, and SD slot will have our design up-and-ready with no additional hardware needed. Target Applications for Zedboard are Embedded vision, Test & Measurement, Motor control, and Software-defined radio— applicability diversity gives this equipment longevity for use in later projects. Kinect is a low-cost COTS sensor for 3D sensing. Moreover, it also has an integrated RGB camera and a microphone. The display panel is for the display of paths during development and demo.

References

[1]  Fichter, W. Path Planning for Fixed-Wing UAVs with Small Onboard Computers. In SADCO A2CO.

[2]  Hachour, O. (2008). The proposed genetic fpga implementation for path planning of autonomous mobile robot. International Journal of Circuits, Systems and Signal Processing, 2(2), 151-167.

[3]  Kok, J., Gonzalez, L. F., Walker, R. A., Gurnett, T., & Kelson, N. A. (2010). A synthesizable hardware evolutionary algorithm design for unmanned aerial system real-time path planning. In Proceedings of the 2010 Australasian Conference on Robotics & Automation.

[4]  Hota, S.; Ghose, D., "Optimal path planning for an aerial vehicle in 3D space," in Decision and Control (CDC), 2010 49th IEEE Conference on , vol., no., pp.4902-4907, 15-17 Dec. 2010.

[5]  Joseph Carsten, Dave Ferguson, and Anthony Stentz. 3D Field D*: Improved Path Planning and Replanning in Three Dimensions. IEEE International Conference on Intelligent Robots and Systems October 9 - 15, 2006.