How I connected an LLM to PX4

Draft. The code this describes is the ai-drone repository; numbers are from the actual flights.

This is the architecture that let me say “take off” to a real quadcopter and have it do that, without the model ever being in a position to do anything else. It is not a tutorial on PX4 or on prompting. It is the part in between.

The shape of it

speech  ->  grammar (fast path)  ->  Claude with tools  ->  validator / executor  ->  MAVSDK  ->  PX4
                 |                                                 |
                 +-- stop / hold / land now: direct, no model       +-- every hard limit lives here, in code

Three rules decided everything else.

The model never produces MAVLink. It can only call a short list of tools: status, takeoff, move, yaw, creep, look, hold, land, return_home.
Every tool call goes through one executor, and the limits live there, not in the prompt. A prompt is a suggestion. A function that returns “refused” is a rule.
The transmitter always wins. Any stick input or mode switch takes control away from the software, and the software has to notice.

Speech in

Whisper small.en on Apple MLX, local, about 0.1 s per phrase. The listener records until half a second of silence, pads the clip with another half second so single short words like “land” survive, and transcribes with no temperature fallback. Whisper’s silence hallucinations (“okay okay”, “thank you”, or its own vocabulary prompt echoed back) are filtered before they cost a model call.

A fixed grammar catches the common orders: take off [to N metres], go/move forward/back/left/right/up/down [N], status, land, return home, stop/hold. Those are queued and flown immediately. “Stop”, “hold”, “freeze”, “hover” and “land now” bypass everything and act at once. Only phrasing the grammar cannot parse goes to the model.

The model

Claude, with strict tool schemas and a system prompt that makes it Hawk, a dry British flight engineer. The model’s job is interpretation and conversation: turn “take it up a bit and look around” into a climb and a yaw, ask a clarifying question when the order is ambiguous, say something short while the aircraft moves. After every executed command a one-line flight-log entry is written into the conversation so the model knows what the aircraft actually did. Without that it would start apologising for flights it had flown.

The read-back of direction and distance comes from code, generated from the tool arguments. The model is asked for one remark on top, in parallel, so you hear “Moving three metres forward” exactly once and the model’s line plays while the aircraft is already moving.

The validator

Limit	Value
Max move per axis per command	5 m
Altitude ceiling above home	10 m
Floor for horizontal moves	1 m
Max radius from home	20 m
Takeoff altitude	0.5 to 5 m, default 1.5 m
Min interval between motion commands	2 s

And the rules that are not numbers:

Motion is refused unless the aircraft is armed, airborne, has a valid global position and telemetry fresher than two seconds.
takeoff runs only if the pilot’s own words contained “take off”, “launch” or “lift off”. The model cannot decide to take off on its own.
Moves are single reposition commands in Hold mode. Nothing streams setpoints. If the laptop process dies, the aircraft holds and every PX4 failsafe (RC loss, battery, geofence) stays in force.
No kill, no flight termination, no parameter writes from software. Those stay on the transmitter and in QGroundControl.

What the real aircraft taught me

The simulator passed everything. The real aircraft found two bugs in one afternoon.

Lost acknowledgements. Over a noisy 915 MHz telemetry radio, the arm and takeoff commands were executed but their ACKs never came back, so the executor reported failure for a takeoff that was happening. Now a command is judged by telemetry for up to eight seconds: if the aircraft is armed and climbing, the command got through. If a takeoff never starts, the executor disarms and says so.

The override that was not one. PX4’s takeoff often levels off a little short, so the executor topped up the altitude with a reposition, in a loop of three attempts twelve seconds apart. On the first real session I selected Land on the transmitter during that loop. The second attempt fired on schedule, and the MAVSDK call it uses switches the aircraft to Hold before repositioning. A metre from the ground the aircraft climbed hard and I hit kill.

The executor now subscribes to the flight mode continuously. Any mode it did not command (Land, Return, Position, Altitude, Manual) sets an override latch: every running routine aborts, no further commands are sent, and even a spoken “stop” is refused while the switch says Land, because sending Hold would take the aircraft back. The latch clears only when the next command legitimately expects a new mode. Reproduced in the simulator by selecting Land mid-takeoff and mid-move. Both times Hawk went silent and the aircraft landed under my mode.

Running it

python ai_pilot.py --listen --no-llm                 # grammar only, offline
python ai_pilot.py --listen                          # Claude interprets
python ai_pilot.py --listen --url serial:///dev/cu.usbserial-XXXX:57600 --forward --no-creep