The author details the development of a robotic platform equipped with a camera and a firearm, focusing on the integration of Large Language Models (LLMs) to enable autonomous interaction with the environment. The project utilizes a host computer connected to an Arduino-based chassis via a serial port, with a maximum data transfer speed of 57,600 baud.
To evaluate the robot's perception and decision-making, the developer employed Langfuse, an open-source platform for LLM observability and experiment management. Tests confirmed that models could identify objects, such as heating radiators, and calculate necessary rotation angles based on visual input. The system uses structured JSON responses and tool-calling mechanisms to translate model outputs into physical movements.
The communication protocol between the host and the Arduino is text-based, utilizing specific start and end markers for data packets. A typical telemetry packet includes sequence numbers, motor speeds, sensor data, and battery voltage. At a 100ms interval, the protocol consumes approximately 20% of the available bandwidth, allowing for future expansion of sensor data like rangefinders and gyroscopes.
The project successfully established a functional data exchange loop between the host and the hardware. Initial testing included basic commands such as toggling an onboard LED. This foundation allows the robot to process visual data, interpret spatial maps, and execute movement commands, marking the completion of the core communication infrastructure.
