Eyes, and a first handshake with the real aircraft

Written from the day’s logs.

Giving it eyes

A separate process now owns the camera: a webcam, a network stream, or the simulated aircraft’s own camera. It keeps the latest frame, runs a small YOLO model on it at about twelve milliseconds a frame on the Mac’s GPU, and serves frames and detections over localhost. The pilot never touches video in its own loop, so voice latency does not change.

Two tiers, the same split as flight control. Local and continuous: the detector runs all the time, and with a flag, two consecutive sightings of a person put an airborne aircraft into hold with no model involved. On demand: “what do you see” grabs a frame and sends it with the question. Hawk’s first answer in the simulator, looking at three coloured shapes I had spawned in front of it: a correct description, and the remark that the detector’s “two airplanes” were, in fact, our own propeller blades.

Short words also got a fix today after “land” came through as “Holds like mode” and then as a wall of “send, send, send”. Whisper regurgitates its own vocabulary prompt on a clipped clip. Half a second of silence padded onto each recording, no temperature fallback, and a filter for repetition. And “shut up” no longer parses as “up”.

The Action 4

I have a DJI Osmo Action 4 and no cable to the ground, so the plan was its livestream mode pushing RTMP to the laptop. MediaMTX on the Mac receives it, OpenCV reads the result. Latency on the Mac side measured under a second with a synthetic test stream. Then the real camera did exactly the same thing on the phone’s hotspot. 480p is plenty: the detector resizes everything to 640 wide anyway.

The mount is the camera’s head-mount bracket, zip-tied to the front of the frame. It is a hack, and it has held through every flight since.

The handshake

Telemetry radio in, QGroundControl showing the real aircraft, then Hawk on the same radio in a bench mode that refuses motion. “Status” came back from the real Pixhawk. “Go forward two metres” was refused: not airborne. That refusal was the first thing the validator ever said to a real aircraft, and it was the right thing.

Then the trouble. Arm commands timing out while the RC worked fine. A radio status readout explained it: the ground station heard the aircraft with a 38 dB margin, the aircraft heard the ground station with 14 dB. The aircraft-side noise floor was 22 dB higher than the ground’s, and 1,717 receive errors with zero corrected meant error correction was off.

At home that evening the same radios read a clean 20. The noise was the site, not the airframe. 915 MHz radios in a city whose cellular network sits at 900 MHz is a bad pairing. Error correction went on in both radios through a small tool that talks to their command interface.

Also learned the hard way: QGroundControl grabs the serial port on launch, so the pilot now relays the whole MAVLink stream to it over UDP and both see the aircraft at once. Lost acknowledgements are now judged by telemetry instead of by the radio: if the aircraft is doing the thing, the command got through.