Automated Drone Flight: Recording - Systems at Scale

It often makes sense to test large and complex systems using smaller scale models. Automotive, aerospace, and to a lesser extent, construction industries have used wind tunnel models, as an example, to test the aerodynamic behavior of large-scale models using smaller size models. This is a physical scaling test that generates system assurance at reduced costs.

The same principle applies to complex systems at the system level. Unfortunately, it is not applied as often as it should. These so-called "toys" that we will use in this post are generally disregarded in large, complex, and expensive engineering programs.

This post will use one of these "toy" models to test the feasibility of obtaining complex video recordings at very high resolutions and with controlled flight paths without breaking the bank. The final purpose of the model has to remain confidential at this point. We will reveal it as the project unfolds, and it entails the generation of very high-quality, detailed photogrammetry.

Today we explore, as cost-efficiently as possible, the possibility of using an automated flying platform, a drone, to obtain motion imagery of various shapes automatically. That is, given a certain environment, to be later defined, can the drone perform a flight path recording of significant quality automatically and efficiently?

An educational tool is a precious tool for this type of simple system test. If the flight and video recording operations from an educational tool minimally fulfill the requirements of a larger, software-intensive system, we gain assurance that this data gathering will not adversely impact the system's design when more sophisticated systems are introduced. In this case, we will use the Ryzen Robotics Tello EDU educational line for our trial. There is a Python library for remote control, has a decent camera, and is small enough to be flown without restrictions. Even if this model will not fulfill all the requirements of the final system, it is capable though of demonstrating all the functionalities we need. Furthermore, the cost is relatively low, and the learning curve should not be very steep.

To generate our first automated test flight recording, we will need to import the following modules:

import cv2
from threading import Thread
import os
from time import sleep

from djitellopy import Tello

We will need to use multiple threads to keep the video recording and the automated flight working together. The djitellopy module will handle most of the communications to and from the drone. These communications are based on UDP packets with a structure as described in this forum post. We can easily connect to the drone and check its battery status with:

tello = Tello()
tello.connect()
tello.query_battery()

We will not define two separate functions, one to keep the recording on while the drone flies and another to control the pre-programmed flight path. For the recording function:

def videoRec():
    
    # Set video parameters:
    cmd = 'setresolution {}'.format('high')
    tello.send_control_command(cmd)
    cmd = 'setfps {}'.format('high')
    tello.send_control_command(cmd)
    
    # Set text overaly parameters:
    font                   = cv2.FONT_HERSHEY_SIMPLEX
    bottomLeftCornerOfText = (10,500)
    fontScale              = 1
    fontColor              = (255,255,255)
    lineType               = 2
        
    FPS = 30
    codec = 'h264'
    fourcc = cv2.VideoWriter_fourcc(*codec)
    
    # Start the drone stream:
    tello.streamon()
    sleep(0.1)
    
    # Grab a frame to check size:
    img = tello.get_frame_read().frame
    shape = (img.shape[1], img.shape[0])
    
    # Set up recording directory:
    i = len(os.listdir(record_dir))
    video_file = cv2.VideoWriter(f'{record_dir}/test_flight_{i+1}.mp4', 
                                 fourcc, FPS, shape)
    while True:
        img = tello.get_frame_read().frame
        altitude = tello.get_height()
        
        alt_txt = f'Height: {altitude}'
        
        cv2.putText(img, alt_txt, 
        bottomLeftCornerOfText, 
        font, 
        fontScale,
        fontColor,
        lineType)
        
        cv2.imshow ('Drone Video Feed', img)
        
        video_file.write(img)
        sleep(1/FPS)
        
        k = cv2.waitKey(1)
        if k == 27: # wait for ESC key to exit and terminate feed.
            cv2.destroyAllWindows()
            video_file.release()
            tello.streamoff()
            breakreturn 0

We are setting all the video recording parameters to high: highest possible resolution and frames per second. Due to the variable quality of the connection, the video bitrate is left automatic by default. We are displaying the height sensor reading at the bottom left corner of the video image to check how this varies during our flight and the possibility of displaying any other value sent by the drone. Flight recordings will be saved at the following directory:

record_dir = 'mission_recordings'try:
    os.mkdir(record_dir)
except:
    pass

Now we need to program a function that will generate a flight mission. By using the remote control (RC) commands, we can generate any flight pattern we may need. This flight pattern will be fixed, and the drone will not abort or modify its route during flight. This is only a demonstration flight. Any command, passive or interactive, can be inserted into the mission to fulfill any needs we may have:

# Fly around waypoints:
def mission_A():
    tello.send_rc_control(0,0,0,0)
    # Turns motors on:    
    sleep(0.1)
    tello.send_rc_control(-100,-100,-100,100)
    sleep(2)
    tello.send_rc_control(0,0,80,0)
    sleep(2)
    tello.send_rc_control(0,0,0,0)
    sleep(0.1)
    
    for _ in range(4):
        tello.send_rc_control(0,100,0,0)
        sleep(5)
        tello.send_rc_control(0,0,0,0)
        sleep(0.1)
        tello.send_rc_control(0,0,0,-100)
        sleep(1)
        tello.send_rc_control(0,0,0,0)
        sleep(0.1)

    tello.send_rc_control(0,0,0,0)
    sleep(0.1)
    tello.land()

The (0,0,0,0) command returns the flight stick to its center position. The (-100,-100,-100,100) command starts the engines, and then, each of the four numerical positions controls the side movement, forward movement, height, and rotation, respectively. In A flight pattern, we move forward at a speed of 100 cm/s for 5 seconds, using the sleep command, then center the stick and rotate 90 degrees, a 100 turning rate for 1 second. Then we center the stick and autoland wherever we are.

Now the video feed and recording process can be started. It is worth waiting a few seconds as the first frame can take time to arrive:

#This thread runs receiving video feed.
receive_video_thread = Thread(target=videoRec)
receive_video_thread.daemon = True
receive_video_thread.start()

And the flight patter finally started:

#This thread starts the flight pattern.
flight = mission_A
mission_thread = Thread(target=flight)
mission_thread.daemon = True
mission_thread.start()
# Join the threads
mission_thread.join()
receive_video_thread.join()

The resulting recording is this:

The drone is supposed to end close to the starting position, being it an indoors drone and with a little bit of wind, the flight ends a few centimeters off-target. The flight and recording interface is quite simple when using Python, with the flexibility to perform other tasks that it offers. In upcoming posts, we will perform more complex missions that contain interactive elements to control the drone.

Do not hesitate to contact us if you require quantitative model development, deployment, verification, or validation. We will also be glad to help you with your machine learning or artificial intelligence challenges when applied to asset management, automation, or intelligence gathering from satellite, drone, or fixed-point imagery.