Communication protocol

When the onboard service is started, it listens for the commands from a user. The figure below shows how the I-Seed Drone Control Android application communicates with the I-Seed onboard service. The Android application runs on an Android phone connected to the drone’s remote control via a USB cable. The commands are noted in uppercase. If extra parameters for a command are needed, they are sent/received as the follow-up message with data.

When a connection is established between the application and the drone, the application will send the PING command and will wait for a reply with the PONG command back. This one needed to verify that communication is actually working because the API peculiarity is that all the interconnections return successful codes, but the real packets will start receiving later.

There is a service running in a background thread on the drone that periodically sends the current mission states and its geographical coordinates. The drone service sends a DRONE_INFO command to inform the Android app that a coordinate data and mission state message will follow. The mission_path message with the coordinates of the mission path built from the user input polygon will be sent in case of the special drone state called PATH_DATA.

When the drone operator is ready, a mission can be started via the Android application interface. The first step will be to build and approve the mission path by providing an input polygon and sending the BUILD_MISSION command. Coordinates will be taken from the Google Map widget of the Android app. In the second step, user sends a MISSION_START command to the drone service and actually starts the mission. If the mission path doesn't look appropriate for any reason, the user can clear the mission path by sending MISSION_PATH_CANCEL and tweak the input polygon.

During a mission, a drone operator can pause, resume, or abort its execution. This is achieved by sending MISSION_PAUSE, MISSION_CONTINUE, and MISSION_ABORT commands, respectively, to the drone service.

Since the effect of applying mission control command is not immediate, and there is a pool of send/received messages, the event_id counter is introduced to synchronize the state. After each user's mission command, such as MISSION_PAUSE or MISSION_CONTINUE, the event_id counter increased by one. All the mission states received in DRONE_INFO with the less value in event_id will be ignored until the message with updated event_id is received.

The full mission consists of a forward mission (when we take photos and run inference in the background) and a backward mission (when we revisit waypoints where objects were detected and run laser measurements for each object). In both cases, when a waypoint is reached, the mission is paused, and custom code is run in the action_job thread. If a forward mission is finished, the mission type is changed to backward, and a similar cycle continues until ready.

If an I-Seed is detected in a waypoint during a forward mission, the range between the camera and the I-Seed needs to be known. For this, the DJI Zenmuse H20/T laser rangefinder is used in a waypoint during a backward mission. However, there is no API call for the laser_range data available from the Payload SDKs, one needs to use a hack to retrieve this information from the Mobile SDK. The onboard service emits a LASER_RANGE_REQUEST command to the Mobile SDK that runs with the Android app to achieve this. After that, the Mobile SDK sends LASER_RANGE_RESPONSE with the laser_range message back to the onboard service that uses this information to compute the geolocalisation of the I-Seed.

It's not possible to know beforehand the actual height of the drone above the ground. We only know the actual height after laser measurement of the place received. The LASER_RANGE_REQUEST command will be used in this case too. If the actual height is too low or too high, the mission waypoint's height has to be corrected.

I-Seed drone control

There are a UI system thread that receives UI events and three custom threads: reading/writing from pipe (interconnection) and a polling job that checks the state. The executeCommands is a buffer with commands.

UI thread will process the events from the drone operator. E.g., if a user wants to abort the mission, command MISSION_ABORT put into a queue. When writePipelineJob takes control of the queue, it reads MISSION_ABORT and is responsible for sending it to a drone.

Read-from-pipe thread with the readPipelineJob method will process the commands received from the drone. When the PONG command is received, it means communication with a drone is established, and a drone is ready to accept other commands. DRONE_INFO command informs that coordinates with the mission state data are prepared. Drone coordinates are used for drawing drone icons in the Google Map widget. Mission state will be used to draw the controlling UI buttons. E.g., if a mission is stopped, UI is switched to the state when an operator can start a new mission. When readPipelineJob receives a LASER_RANGE_REQUEST request from the drone, it puts LASER_RANGE_RESPONSE into a queue. When writePipelineJob takes control of the queue, it reads LASER_RANGE_RESPONSE and sends it to the drone, but this time command is sent with the data - the value of laser range measurement.

The pollJob method is the main point of calling Mobile SDK API. It is responsible for initialization. It periodically checks the state of a connected smart controller, updates UI widgets, checks the pipeline, checks the GPS signal, enables laser, etc.

The following diagram describes the state of the drone control Android application. The initial state is WAITING since the drone may already be in the progress of mission execution (e.g., the application is restarted). In the READY state, the user can provide a mission input polygon. The cancel button will clear the polygon, and the action button will send the BUILD_MISSION command.

When the mission path is ready, the special state PATH_DATA will be received from the onboard service, and the mission_path message will be the next packet. After this state, the next state will be PATH. In the PATH state user can cancel the mission by sending MISSION_PATH_CANCEL or start the mission by sending MISSION_START. State EXECUTING is a normal state when a mission is in progress. State PAUSED received when a user pauses the mission. In both those states mission can be aborted by sending MISSION_ABORT.

Note

Java, Android

I-Seed onboard service

After the drone is started, there are six threads:

• Payload SDK callback thread, that's where asynchronous callbacks from Payload SDK live. E.g., we can receive an event that the mission is finished or the waypoint reached (see mission and mission_state). Note that there should be no time-consuming calls or blocks here. The callback should be finished as soon as possible
• inference_job checks for new files on SDCard, and if a new JPG file is found, it launches the inference and saves the result back to the waypoint data
• receive_data_job is reading pipe. That's where commands and data (like laser_range::value_received) from the Android app come from
• send_data_job is writing commands to pipe, commands are read from execute_commands_ queue, if needed extra data send as a follow-up
• user_control_job is waiting for the right conditions to execute mission::abort and mission::pause requested by the user. Code moved from receive_data_job to avoid blocking the read pipe
• action_job is executed when we reach a waypoint. Most of the mission it's in the waiting state. That's where we can start shooting a photo, request for laser range, and calculate the ECEF coordinates of detected objects

The following diagram describes the state of the drone onboard service. The initial state is ready. In this state, the service emits READY for drone control until the user receives the input polygon, and the service builds the mission path. When the mission path is ready, the state is switched to a special state, path_data, which emits PATH_DATA and a message mission_path. Immediately after PATH_DATA is sent, the state is switched to the path state, emitting PATH. In this state, the user can cancel the mission or start it. Starting the mission is a two-step process. The first init step will fetch the mission path, switching to forward_wait_start, and the second start step will actually start the mission, switching to forward_wait_update.

In all the states except forward_wait_update, backward_wait_update, forward_executing, backward_executing, the update callbacks received from Payload SDK will be ignored. The forward_wait_update/backward_wait_update will be switched to forward_executing/backward_executing once the Payload SDK update is received. In these states EXECUTING/PAUSED emitted. All other states that were not mentioned so far will emit WAITING. In forward_executing/backward_executing states, the method abort can be called if a user wants to abort the mission (the following method will be stop), or the last fake waypoint reached, or the same mission but with the tweaked height need to be restarted (forward mission only, following method will be start). When the finish event is received from the Payload SDK callback, the state is switched to forward_finished/backward_finished. set_backward will switch the mission to backward_wait_start for the forward mission method. Another way to reach backward_wait_start is from forward_wait_start (when the last fake waypoint is reached and we want to start the backward mission). If no objects are detected in the forward mission, the method stop will move backward_wait_start to ready.

Note

C++, Linux