These instructions are available in less detail from the README of the GitHub repository. There is also more information on the Viam Rover project in my previous blog post, Autonomously Mapping with Visual SLAM.

▶Watch: Setting Up and Calibrating the Autonomous Viam Rover

Installing the Dependencies​

Radxa X4 setup

All instructions should be executed from the Radxa X4, either directly or through an SSH session. I have a video on how to set up the Radxa X4 below. The process is essentially creating a standard USB installation drive with Ubuntu 24.04 and installing it onto the Radxa X4, set up with a heat sink and M.2 SSD. The full robot should also be assembled and ready to use.

▶Watch: Setting Up and Calibrating the Autonomous Viam Rover — supporting video

ROS 2 Installation​

Almost all ROS 2 dependencies can be installed using the helper script provided by Intel in their Getting Started guide. The command is as follows:

wget https://raw.githubusercontent.com/open-edge-platform/edge-ai-suites/refs/tags/v2026.1.0-rc1/robotics-ai-suite/scripts/setup-robotics-jazzy.sh
chmod +x setup-robotics-jazzy.sh
export USE_PROXY=0
./setup-robotics-jazzy.sh

This will set up ROS 2 Jazzy plus some extra dependencies needed for building the code. The rest of the dependencies needed for the ROS 2 packages will be installed using rosdep in the Build the Package section.

RP2040 Setup​

We also need to perform a one-time setup for the RP2040. The RP2040 is visible to the Intel N100 chip as a USB device, and the scripts in my project are configured to automatically mount the USB device and send the latest firmware to it, which is the flashing process. The RP2040 will then automatically restart running the new firmware.

To mount the USB device automatically and without being prompted for sudo access every time, we need to set up some permissions in advance. First, install the system dependencies:

sudo apt install -y \
  cmake \
  gcc-arm-none-eabi \
  libnewlib-arm-none-eabi \
  libstdc++-arm-none-eabi-newlib \
  libgpiod2 \
  gpiod \
  udisks2

Then, to set the permissions up correctly, we execute the following:

sudo sh -c 'echo "SUBSYSTEM==\"gpio\", KERNEL==\"gpiochip[0-9]*\", MODE=\"0660\", GROUP=\"dialout\"" > /etc/udev/rules.d/99-gpio.rules'
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo usermod -aG dialout "$USER"

Log out and back in to enforce the new permissions. This will allow the build script to automatically mount and flash the RP2040.

Build the Package​

We can now check out the code, install dependencies, and build it as follows:

mkdir -p ~/ros2_ws
cd ~/ros2_ws
git clone https://github.com/mikelikesrobots/viam-rover2.git --recurse-submodules src/viam_rover2

# Install ROS dependencies
source /opt/ros/jazzy/setup.bash
rosdep install --from-paths src --ignore-src -r -y

# Build
colcon build
source install/setup.bash

The commands above will:

1. Check out the latest source code from the GitHub repository
2. Build the ROS 2 packages
3. Build the firmware for the RP2040
4. Flash the firmware onto the RP2040

The RP2040 will then be ready to start feeding sensor data back to the main board, as well as accepting motor commands, which is exactly what we need for the next step: calibrating the encoders.

Encoder Calibration​

We need to calibrate the encoders so that the feedback system for the robot works correctly. You can read more about the control system in my previous blog post, Controlling the Viam Rover with ros2_control.

For the control system of the robot to correctly track the robot's movement, it needs to be able to calculate the distance travelled by the wheels based on the encoder ticks. Our task is to move the robot a fixed distance and count the ticks measured while travelling that distance.

Straight Line Movement​

The first step of calibrating the encoders is to make the robot travel in a straight line. Without this step, the robot is very likely to veer off to one side or the other, making it difficult to accurately measure distance travelled for the correct encoder tick counts.

The simplest way to fix this error is to run the full control system for the robot and repeatedly adjust the encoder counts until the robot correctly moves in a straight line. To help with this adjustment, it benefits us to understand how the tick count converts into velocity, so we know which side to accelerate for the robot's movement.

Calculating Distance Travelled from Encoder Ticks

The velocity that the wheel is travelling at is the distance travelled divided by time, and the distance can be calculated as the circumference of the wheel multiplied by the number of revolutions. Hence, the velocity is:

The revolutions is the number of ticks counted during travel divided by the ticks per revolution:

To simplify this equation, we can note that the time and radius are not changing, which means that the velocity is proportional to the revolutions counted:

If the robot is turning left, that means the left motor is not being driven hard enough by the control system. We need to make the measured velocity appear smaller so the controller will drive that motor harder. To do that, we increase the $left\_ticks\_per\_rev$. The same logic works for the right wheel.

Our task is to place the robot down next to a tape measure or a large ruler and keep running the robot, adjusting the $ticks\_per\_rev$ variables, until the robot drives in a straight line. You will need some clear space for this; I'd suggest around 2m in front of the robot and 1m either side of the robot, just in case the robot veers off or overshoots its target. Also, be ready to pick up the robot if it's travelling too far!

To drive the robot, open two terminals on the robot. Use one to execute:

# Terminal 1
ros2 launch viam_rover2 rover2_bringup.launch.py enable_vision:=false

In the other terminal, every time you want to drive the robot, execute this script:

# Terminal 2
python3 tools/forward_velocity_probe.py

The robot should drive a fixed distance, then output the tick counts measured while travelling. Adjust the ticks per revolution variables, then quit the program running in terminal 1, rebuild, and run again. Try adjusting by about 5% if you're not sure how much to increase it, so 1000 ticks becomes 1050.

# Terminal 1
colcon build --packages-select viam_rover2
ros2 launch viam_rover2 rover2_bringup.launch.py enable_vision:=false

Repeat the process until the robot travels in a straight line. Here's a more compact version:

1. Launch the control system.
2. Run forward_velocity_probe.py.
3. Increase the ticks_per_rev value depending on which side the robot veers.
4. Stop the control system and rebuild the package.

# Terminal 2
python3 tools/forward_velocity_probe.py --speed 0.20 --distance 1.0

Scaling the Ticks per Revolution​

Once the robot is moving in a straight line, run the script another time with your chosen parameters. This time, measure the physical distance travelled with the ruler ($actual_distance$), then note the odometry distance reported by the script ($odom_distance$).

Multiply both calibrated ticks_per_rev values by the same ratio. This corrects the overall distance scale without changing the left-to-right relationship that makes the rover travel straight. For example, if the robot travelled 45cm and measured 50cm, you would multiply each ticks per revolution value by $\frac{50}{45}$.

Update the files, rebuild, and run for one last time. The robot should move in a straight line, and the odometry from the robot should be within a few centimetres of the actual value when you measure it.

This step is complete; the next step is to generate a motor lookup table (LUT).

Motor LUT Generation​

The motor LUT is a process that I chose to use for this robot to help output slightly more accurate motor speeds. It's not strictly necessary given that the control system should adjust the motor speed to the commanded value anyway, but the LUT helps it get there with slightly less effort, making the system more responsive and reliable overall. Again, more detail on the LUT and how it works are available in my previous blog post, Controlling the Viam Rover with ros2_control.

The process is very simple. It requires the encoders to already be calibrated, which we accomplished in the previous step. It also requires the robot's wheels to be free-spinning, so place the robot on a box or similar object that allows it to spin the wheels freely.

The script for measuring wheel speeds does not go through the ros2_control-based system, instead directly commanding the motors and reading the encoders over serial. Hence, we need to manually update the values in the script. (If you'd like to update the script so that it takes these values as parameters instead, feel free to raise a PR!) To update the values, edit lines 23 and 24 in the motor_diagnostic.py script directly.

Once updated, run the script:

python3 tools/motor_diagnostic.py

The script will sweep across PWM values and measure the wheel speeds for each PWM value, then print a LUT block. Example output:

motor_lut_[0] = {  // left
    {0.1374, 0.0},
    {0.3, 1.94}, {0.4, 3.39}, ...
};

Take this block of code and paste it into hardware/src/rover2_system.cpp, replacing the existing motor_lut_[0] and motor_lut_[1] entries. Build the code one final time to incorporate the new LUT values.

The encoder calibration and LUT generation are now complete, and the robot is ready to run!

Calibration Complete​

The robot should now be fully ready for mapping. It should:

• Travel in a straight line when commanded with only a linear velocity
• Accurately measure distance travelled with odometry (that is, the velocity probe reports an encoder-measured distance very close to the ruler-measured distance)
• Have newly generated motor LUTs compiled into the system code
• Complete a clean colcon build without errors

At this point, the robot is ready to map, then autonomously navigate an environment. I decided to leave the navigation steps for the next post so I could dive into the navigation system itself. Check back here for future posts if you want to read about how the navigation system works!

This post in particular focuses on the control system of the rover, including motor control, sensor reading, and fusing data together to track the robot's position.

▶Watch: Controlling the Viam Rover with ros2_control

Control System Overview​

The rover uses a closed-loop control system, which means that it uses feedback from sensors on the robot to adjust the robot's speed to more closely match the desired speed. If the sensor shows that the robot is moving too fast, the control system can slightly slow the motors until it reaches the correct velocity. This is a more accurate way of moving the robot than open-loop control systems, which simply drive the motors at a speed without checking what actually happens. Better accuracy is important when the robot is moving autonomously, as it can track its own position and velocity.

The rest of this article details how the control system takes commands from the navigation system, converts that into accurate robot movement, and measures the robot's position to feed back to the navigation system. I'll explain how the motor control and encoder reading work, then discuss how ros2_control interfaces with the hardware through PID controllers, and finish by explaining how the EKF fuses ros2_control's odometry with IMU data to get the best idea of the robot's position.

Units of Velocity​

To understand how the control system is used to move the robot, think about moving the robot from one position on the floor to another. The navigation system calculates a path that the robot should follow, then calculates the linear and angular speeds that the robot should move at to follow this path. The control system is responsible for moving at these speeds as closely as possible.

The standard units for linear and angular speeds are metres per second (m/s) and radians per second (rad/s) respectively. Radians measure angles, similar to degrees, and can easily be converted to and from degrees. As radians are defined in terms of pi, they have some advantages when calculating speeds and rotations, which is why they are used.

The differential drive controller then converts the linear and rotational velocities to desired wheel velocities. To do this, it needs to know the wheel sizes and how far apart they are spaced. If you're interested in the mathematics behind this, take a look at the Robotics Book by Frank Dellaert and Seth Hutchinson. For our purposes, it is enough to understand that the controller converts linear and angular speeds into the speeds that the wheels should move at, also defined in rad/s.

Driving the Motors​

At the lowest level, our hardware controller (later referred to as the hardware interface in ros2_control sections) is responsible for taking the desired wheel velocities and moving the motors at those speeds. The motors on the Viam Rover are driven using Pulse Width Modulation (PWM) signals, so the hardware controller has to convert the wheel speeds to the right PWM values and send PWM signals to the motors.

The PWM signal to send depends on the motors in question. The simplest method is to determine the maximum speed of the motors when driven at full power, which can be measured using the encoders (explained more in the Reading the Encoders section). The hardware controller can then assume that wheel speed is proportional to the PWM duty cycle.

That means if the motors drive at a maximum speed of 1.0 rad/s, and the desired velocity is 0.5 rad/s, it assumes that the correct signal is 0.5 (PWM doesn't have units because it's a fraction of a maximum). If the desired velocity is 0.25 rad/s, it uses a PWM value of 0.25.

With that said, the real system doesn't behave quite that well. I measured the speed of the wheels against different PWM values to obtain the following graph:

The graph is flat around the middle, which is the deadband of the motors. This is where the motors don't have enough power to overcome inertia, so they can't even start moving. The graph also shows that the motors don't exactly match the expected tick rate (the dotted black line) even after overcoming inertia.

With closed-loop feedback, the ros2_control controller can correct for the motors not moving at the correct speed. Still, the more accurately we can drive the motors at the commanded speed, the less work that controller needs to do to correct the speed. For the rover, we use a Look-Up Table (LUT) to get each motor close to its commanded velocity, so that the PID controllers from ros2_control can remove the remaining error more quickly and reliably.

Mapping Wheel Velocity to PWM​

The LUT is used to convert desired wheel velocity to PWM value. The table first has to be built by measuring motor speeds; I have a Motor Diagnostic script that will build the table by sending different PWM signals and measuring the wheel speeds, then using those measured values to build the LUT. The wheels have to be free-spinning for this to work. The user can then copy and paste the output into the rover2_system.cpp source code.

The software looks up values from the LUT by following a simple process.

1. Is the desired velocity in the table? If yes, use the corresponding PWM value.
2. Is the desired velocity below the minimum? If yes, use the minimum PWM value.
3. Is the desired velocity above the maximum? If yes, use the maximum PWM value.
4. Otherwise, the desired velocity is between two entries in the table. Find the two entries and interpolate between them to find the PWM value.

Interpolation calculates an intermediate PWM value by splitting the difference between two adjacent entries. For example, given the following LUT:

If the desired velocity is 1.4, it is 40% of the distance between 1.0 and 2.0, so the result should be 40% of the distance between the PWM values 0.5 and 0.7, which is 0.58. The relevant code is in the rover2_system.cpp file.

This is how the hardware controller will move motors at a commanded speed. Next, we can look at how the controller reads data from the encoders.

Reading the Encoders​

Encoders are devices that can count how fast a shaft spins. The Viam Rover contains encoders integrated with the motors, but these are single-pin encoders, which means they can detect speed but not direction. This section describes how encoders work, how direction can be detected by the sensor, and how we work around the Viam Rover's speed-only sensing by guessing the direction.

Single-Pin​

The encoder mechanism is a light source and detector on either side of a disc with evenly-spaced holes around the edge. The encoder is attached to the shaft of a robot's wheel. When the shaft spins, the disc rotates, and the holes alternately block and transmit the light. The light sensor can then detect the presence or absence of light. Each time the light is detected is called a tick, and the rate of ticks can be used to calculate the speed of the shaft.

However, single-pin encoders don't know the direction that the shaft is spinning in. A clockwise spinning shaft looks the same as an anticlockwise spinning shaft. That's where quadrature encoders come in.

Quadrature​

Quadrature encoders work by having two sets of sensors on the same encoder disc, but spaced at an angle from one another. By looking at which sensor receives the pulses first, it's possible to tell both the direction and the speed of the shaft.

Incremental Encoders

Many higher-end incremental encoders also include a third channel that only produces one pulse per revolution. This provides a known reference position, allowing the controller to re-synchronise its position estimate. These encoders aren't common in hobbyist robotics, but they're still useful to know about!

Interpreting Speed​

ros2_control requires velocity feedback, which means it needs to know both the speed of movement and the direction. Given that the Viam Rover only has single-pin encoders, we need another way to determine the direction. In this case, that way is to assume that the robot is moving in the direction it's been commanded to, and combine that with the measured speed. This works as long as the robot is moving under its own power; if you drag the robot around by hand, it can quickly get confused!

In code, this is done for each motor by checking the last commanded direction, then changing the tick delta to be positive or negative depending on the result.

// Single-pin encoder can't detect direction; infer purely from commanded velocity.
if (hw_commands_[i] < 0.0) {
    last_direction_[i] = -1;
} else if (hw_commands_[i] > 0.0) {
    last_direction_[i] = 1;
}

if (last_direction_[i] < 0) {
    delta = -delta;
}

Now that we've examined how the motors are driven and the encoders are read, it's time to look at how they are combined in the ros2_control system.

Control Using ros2_control​

The whole robot is running ROS 2 for a few reasons (see here for details), including the control system, which is built using ros2_control. I have previously written about ros2_control basics and shown how to use it for a simulated robot arm and open-loop control of a real rover, so take a look at those for a more detailed introduction. In brief, the reason to use ros2_control is that it provides modular controllers for different kinds of robots, which means we only need to build the hardware interface that allows ros2_control to connect its existing Differential Drive Controller (DiffDriveController) to our robot.

Hence, the challenge for this robot was to set up a ros2_control system that incorporates feedback. I chose to build all of the hardware interface code into one System, as opposed to splitting it into Sensors and Actuators. This System is the hardware interface between ros2_control and the motors and encoders.

State and Command Interfaces​

For a robot to be controllable from ros2_control, it must declare the types of commands it can receive and the types of feedback data it produces. These are called command and state interfaces respectively.

For the rover, the motors accept velocity commands, so our System needs to register a velocity command interface. This is the same as for the JetBot, and that robot had no feedback data, so that System only had one command interface. However, our robot also reports sensor feedback data, so we also need to register a state interface.

In our case, we are counting ticks on the encoders. Encoder ticks represent a change in angular position, which tells us the change in position of the wheel. Hence, our System registers a position state interface as well as its velocity command interface.

However, reporting position data alone isn't enough. If we were only using the DiffDriveController, it would accept position data, but wouldn't make any use of it. We need to adjust the output speed based on the feedback data, which means we need to use PID controllers, and PID controllers require velocity data. Hence, every time the encoder positions are read, the velocity is calculated by dividing the change in wheel position by the time since the last read.

In code, this is done by dividing the change in radians by the change in time since last read:

double dt = period.seconds();
// ...
// Convert tick delta to radians and accumulate position.
double delta_rad = (static_cast<double>(delta) / ticks_per_rev_[i]) * 2.0 * M_PI;
hw_positions_[i] += delta_rad;
hw_velocities_[i] = delta_rad / dt;

We report this calculated velocity as another state interface for the PID controllers to use.

Our final list of System interfaces is:

• Velocity (command)
• Velocity (state)
• Position (state)

To set the interfaces, we declare them in the rover2.ros2_control.xacro file:

<joint name="${prefix}left_wheel_joint">
<command_interface name="velocity"/>
<state_interface name="position"/>
<state_interface name="velocity"/>
</joint>
<joint name="${prefix}right_wheel_joint">
<command_interface name="velocity"/>
<state_interface name="position"/>
<state_interface name="velocity"/>
</joint>

We also need to export the interfaces in code. This is done using the export_state_interfaces and export_command_interfaces functions, which report the available interfaces to the ros2_control resource manager.

// Declare position and velocity state interfaces for left and right motors
std::vector<hardware_interface::StateInterface> Rover2SystemHardware::export_state_interfaces() {
  std::vector<hardware_interface::StateInterface> state_interfaces;
  for (auto i = 0u; i < info_.joints.size(); i++) {
    state_interfaces.emplace_back(hardware_interface::StateInterface(
        info_.joints[i].name, hardware_interface::HW_IF_POSITION, &hw_positions_[i]));
    state_interfaces.emplace_back(hardware_interface::StateInterface(
        info_.joints[i].name, hardware_interface::HW_IF_VELOCITY, &hw_velocities_[i]));
  }

  return state_interfaces;
}

// Declare velocity command interfaces for left and right motors
std::vector<hardware_interface::CommandInterface> Rover2SystemHardware::export_command_interfaces() {
  std::vector<hardware_interface::CommandInterface> command_interfaces;
  for (auto i = 0u; i < info_.joints.size(); i++) {
    command_interfaces.emplace_back(hardware_interface::CommandInterface(
        info_.joints[i].name, hardware_interface::HW_IF_VELOCITY, &hw_commands_[i]));
  }

  return command_interfaces;
}

The class has the required write and read functions for ros2_control. In both cases, the code is quite long to include verbatim, so the functions are described and linked to below if you want to look for yourself.

The write function is responsible for taking the commanded speeds and writing them to the motors. In this case, it uses the LUT to determine the best PWM value for the motors, then sends the requested PWM to the motor controller.

The read function is called when the control system needs the latest sensor readings. In open-loop configurations, the simplest action is to report the commanded speed as the measured speed. In our closed-loop case, the function takes the latest encoder readings, guesses the direction of the wheels based on the previous commanded speed, and calculates the velocities of the wheels. It also writes the velocities to a custom sensor publishing node, which runs outside of ros2_control and publishes ROS 2 sensor data; this is helpful as ros2_control needs to be real-time, but it helps to publish ROS 2 data to check what the system is doing.

With position and velocity data being fed back to the controller, the next step is to actually use the data. The default DiffDriveController doesn't do anything with the information directly; we need to add PID controllers.

Chaining PID Controllers​

As stated in the previous section, the DiffDriveController alone does not make use of feedback data - it operates open loop. To use the feedback data, we chain PID controllers with the DiffDriveController.

Adding PID controllers is very easy to do with ros2_control using the concept of chaining controllers (see example 12 of the ros2_control examples repository). Our main controller, the DiffDriveController, sends desired wheel velocities to the left and right PID controllers. Each PID controller has its own parameters, sends velocity commands to the motors, and reads data from the encoders. They then regulate each wheel to the commanded velocity using the feedback data.

The DiffDriveController can drive the robot more accurately using the PID controllers, and it will also publish the odometry of the robot. Odometry is data on the robot's position, orientation, and velocity. It allows higher-level systems like the navigation system to track the pose of the robot.

Declaring Controllers​

The code for declaring these controllers all takes place in the rover2_controllers.yaml file. First, the controller manager is configured to use the two PID controllers and the DiffDriveController:

controller_manager:
    # ...
    pid_controller_left_wheel_joint:
      type: pid_controller/PidController
    pid_controller_right_wheel_joint:
      type: pid_controller/PidController
    rover2_base_controller:
      type: diff_drive_controller/DiffDriveController

We can then configure the PID controllers to interact with the rover's hardware interface. The setup is the same for both controllers; it declares the joint it's going to control, that it sends velocity commands, and that it can reads measured velocities. It also specifies PID parameters with feedforward enabled:

pid_controller_left_wheel_joint:
  ros__parameters:
    dof_names:
      - left_wheel_joint
    command_interface: velocity
    reference_and_state_interfaces:
      - velocity
    gains:
      left_wheel_joint: {"p": 0.5, "i": 2.5, "d": 0.0, "i_clamp_min": -10.0, "i_clamp_max": 10.0, "antiwindup": true, "feedforward_gain": 0.95}
    enable_feedforward: true

The PID controllers also use a feedforward gain, which is separate from the LUT compensation. The feedforward term contributes 95% of the requested wheel velocity directly to the controller's output, while the PID terms correct the difference between the requested and measured velocity. The hardware interface then converts that output velocity into PWM using the LUT, compensating for the motor's deadband and non-linear response.

The DiffDriveController is then configured to control these PID controllers. This is done by giving the left and right wheel names as the PID controller joints:

rover2_base_controller:
  ros__parameters:
    left_wheel_names: ["pid_controller_left_wheel_joint/left_wheel_joint"]
    right_wheel_names: ["pid_controller_right_wheel_joint/right_wheel_joint"]

The DiffDriveController has the standard configuration, such as wheel separation and radius, but the crucial difference from the JetBot is to set it to closed-loop configuration:

open_loop: false
position_feedback: true

The DiffDriveController doesn't change how it commands the motors when in closed-loop configuration, but it does change how it calculates the robot's odometry. This is used by the navigation system to track the robot.

At this point, we have seen how speeds sent by the navigation system are received by the ros2_control DiffDriveController, which calculates required wheel speeds and sends them to the PID controllers. The PID controllers then use the custom hardware interface System to move the motors at those speeds and use encoder feedback to regulate the speeds. The DiffDriveController also publishes odometry, which can be used to track the robot's motion. Still, encoders are not perfect sources of information, so it helps to add another layer above the control system: an EKF, used to filter the encoder data and combine it with IMU measurements.

Extended Kalman Filter​

An Extended Kalman Filter (EKF) is an algorithm that can combine multiple sources of sensor data together. It works by making a prediction, then using sensor data to improve that prediction. In our case, this is predicting the pose of the robot, then using the odometry published by ros2_control and the IMU data to improve that prediction. Each source of data is configured with a covariance, which is effectively how much the EKF trusts that source of data.

The EKF is configured to fuse forward velocity and yaw rate from the wheel odometry, along with yaw rate from the IMU. Each measurement includes a covariance describing its uncertainty, which influences how strongly it affects the filter's estimate. In this configuration, the wheel and IMU yaw rate measurements have the same covariance, while forward velocity is supplied only by the wheel odometry.

Combining these measurements allows the sensors to cover the weaknesses of one another. Encoders alone can provide noisy data, and have physical limitations, such as when one of the wheels slips. In this case, the wheel spins a lot and the encoder reports a large change in position, which the controller interprets as the robot spinning, despite the robot not moving. An IMU provides data about the robot's motion independently of wheel movement, so combining its measured yaw rate with the encoder reading produces a better estimate of the robot's pose overall. Even if the final robot pose is incorrect, it should be closer to the truth, which means other layers can more easily correct the mistake: for example, the depth data from the RealSense camera can be used to correct the robot's position as long as the robot is roughly in the right place.

EKF is included by configuring it and launching it:

ekf_config = LaunchConfiguration("ekf_config_file")
ekf_node = Node(
    package="robot_localization",
    executable="ekf_node",
    name="ekf_filter_node",
    output="screen",
    parameters=[ekf_config, {"use_sim_time": use_sim_time}],
    ros_arguments=["--log-level", "warn"],
)

The EKF is configured to take odometry from /rover2_base_controller/odom, published by ros2_control, and IMU data from /rover2_base_controller/imu, which is published by the sensor node.

The confidence in the IMU is set by the covariance vector in the publish_imu function:

msg.angular_velocity_covariance[8] = 0.02;

The confidence in the encoder data is set by ros2_control configuration:

twist_covariance_diagonal: [0.001, 0.001, 0.001, 0.001, 0.001, 0.02]

It then fuses these two sources together and produces a new odometry topic on /odometry/filtered, which can be used by the localisation node to help keep track of the robot's position.

Conclusion​

This post shows how to close the loop between the Viam Rover's motors and encoders using ros2_control, allowing the rover to regulate its wheel speeds instead of relying on open-loop PWM commands. It also shows how to combine the resulting wheel odometry with IMU measurements using an Extended Kalman Filter, producing a more robust motion estimate for localisation and navigation.

For the Viam Rover in particular, the post showed how wheel speeds are turned into PWM values adjusted for the motor's behaviour using a Look-Up Table, and how encoders are read to provide an estimate of the robot's velocity and how much it has moved. These are built into a ros2_control System, which allows the modular DiffDriveController and PID controllers to correct the remaining error and report the robot's movement as odometry. Finally, the EKF fuses that odometry data with IMU measurements to produce a better idea of the robot's position, which forms a much better foundation for navigation systems than either sensor source could provide by itself.

There are still limitations. The single-pin encoders cannot measure direction directly, so the software has to infer it from the last command, and wheel slip can still make the rover appear to move farther than it really has. Nevertheless, the combination of calibration, feedforward mapping, PID feedback, and sensor fusion is accurate enough to support the rover's autonomous mapping and navigation.

All of the configuration and hardware-interface code is available in the Viam Rover repository. In a future post, I'll show how to tune the PID controllers for a real robot and perform other calibration for accurate motion.

In this post, I walk through how I built an autonomous indoor mapping robot using the Viam Rover as a base, swapping out the webcam for an RealSense Depth Module D421 and using a Radxa X4 as the central compute (instead of the already-supported Raspberry Pi or Jetson Orin). The robot runs a full ROS 2 stack, including Visual SLAM for map building and Nav2 for autonomous navigation.

While building this robot out, I ran into some interesting challenges around encoder noise and calibration, which I cover in detail. If you want to build one yourself, there's a parts list at the bottom, and all the source code is on GitHub.

Autonomous Rover Infographic

Viam Rover​

From their documentation, "Viam is a software platform for building, deploying, and managing robotics applications." To help with being able to deploy their software and try out what they're offering, Viam also sells the Viam Rover, which is an inexpensive kit that runs Viam software out of the box.

All you need to supply is the Single Board Computer (SBC), like a Raspberry Pi, and compatible batteries. Otherwise, everything comes pre-assembled and ready to go. Configure the power, install the Viam software on the SBC, and you should be ready to drive the robot around! Viam provides the control software and makes the interface for managing the robot very simple.

Still, I wanted to develop an inexpensive robot capable of autonomous navigation, and the rover doesn't quite have the hardware to make it happen. It provides encoders, IMU, and webcam, but these sensors alone don't provide enough information for building a map. I decided to swap the webcam for a RealSense camera and use a Radxa X4 board to provide enough computing power to build a map on board. Again, if you're interested in building the robot yourself, I have a parts list at the bottom of this post.

Radxa X4​

The Radxa X4 is a relatively low-cost SBC with a unique selling point: it contains both an Intel N100 chip, for compute-heavy work, and an RP2040 chip, which can be used for real-time embedded software. This works very well for robotics; it's fairly common to use a powerful board to run high-level software like mapping while using a microcontroller for real-time operations. The X4 provides both in one package.

There are a couple of drawbacks of using the X4 for this project. First, it needs its own power supply; the Raspberry Pi can be powered directly from the robot batteries, but the X4 needs its own battery bank with separate connections. Second, it produces quite a bit more heat, so it needs a fairly large heat sink. That expands the size of the robot and means it has to be mounted upside down, which requires extra parts. Thankfully, the rover comes with some parts to expand upwards already, so it's already designed to accommodate larger boards.

The ROS 2 package I've written will build custom firmware for the RP2040, flash it automatically, and then communicate with the RP2040 over a serial link on UART0. This means the RP2040 is responsible for reading sensors and sending PWM signals to the motors, while the Intel N100 operates with a non-realtime Ubuntu installation.

RealSense D421​

As I've previously stated, creating a map from webcam data alone simply isn't enough. We either need lidar or some kind of depth data, which is where the RealSense camera comes in. These sensors are very popular in robotics as they automatically calculate depth data from stereo cameras and can output depth images or point clouds. Depending on the model, they can also provide RGB (colour) data. I chose one of the lowest cost models to keep the total cost of the robot down: the RealSense Depth Module D421. This produces infrared images and a depth image that I use with the rover to map its environment.

ROS 2 Instead of Viam​

One of the great advantages of using the Viam Rover is that the software for controlling the rover can be installed and configured very easily. The interface is excellent. However, the Viam system isn't based on ROS 2 (although it can be bridged to it). I chose to implement ROS 2 Jazzy fully on my robot instead of using the Viam software, and this section is to explain my reasoning.

Mapping Isn't Supported by Viam​

As far as I can tell, Viam doesn't provide modules for indoor mapping and navigation. If you take a look at the Viam navigation documentation, the system requires GPS position and heading to work, which doesn't work with indoor mapping. Hence, my choices were to build the module myself, or use the existing Nav2 framework from ROS 2.

It's possible that I could have used Viam software and separately run a ROS 2 navigation system, then used a Viam module to bridge between them. I wasn't sure how complicated that would have been to do or how efficient the system is at passing data back and forth for high-volume data like images and point clouds, so I didn't go this route.

No Radxa X4 Support​

Viam does offer support for a few different SBCs, most notably the Raspberry Pi boards and Jetson Orin Nano series, but doesn't have support for the Radxa X4 already. In fact, I couldn't find support for any x86 board, and asking for help in the forums didn't yield any responses. I had to choose between building a Viam module for the Radxa X4 or building with ROS 2. I chose ROS 2 as I'm much more familiar with the system, and because I don't know how well the Viam module would have handled spanning across both the Intel N100 and the RP2040 chips.

Custom Encoder Software​

During development, I found that the encoders used for position feedback on the motors behaved very poorly. The signals from the encoders are very noisy when the motors are running, which required me to add extra hardware to fix the noise. I originally had some software filtering to partially compensate for the noise, which I wouldn't have been able to do with the prebuilt Viam module. I have since removed the software filter due to the hardware fix, but this was originally a factor in my decision, hence including it here.

I did ask in the Viam forums if anyone else had experienced this issue or had a solution, but this also yielded no responses.

RealSense Support​

Viam does have a RealSense module which supports:

• D435
• D435i
• D455

Unfortunately, the list does not include my D421 model. I either needed to add support for it myself or use the existing ROS 2 support. Again, I could have used the ROS 2 node and bridged it to the Viam system, but this is adding complexity over a complete ROS 2 system.

Teaching Opportunity​

The reasons above are challenges that I possibly could have overcome with time and effort, but the major reason I went this route was so I could use it as an example for teaching ROS 2 concepts, such as using ros2_control with closed loop feedback (another future post/video!).

ROS 2 Software Architecture​

Given my decision to build the system fully in ROS 2, I chose to build the following architecture (heavily simplified):

Some components, such as the Navigation Stack, have quite a few parts inside. I'll dive deep into these systems in future videos, or you can take a look at the source code for yourself!

Setup and Encoder Calibration​

The majority of the time on this project went into making sure the robot produced reliable odometry. Odometry is the robot measuring its own movement based on its sensors; in this case, its rotary encoders.

These encoders work by attaching a disc with evenly-spaced holes around its circumference onto the same shaft that the wheels are on. The encoder then shines a light through one side of the disc and detects if it passes through the disc. As the disc rotates with the shaft rotation, the disc holes alternately block and allow the light through, measured as pulses/ticks by the encoder. The faster the ticks occur, the faster the shaft is spinning. To calibrate the encoders, I had to find the ticks per revolution of the wheel.

I had a lot of trouble getting the encoders to reliably produce accurate wheel speeds. The rest of this section details the issues I had and how I dealt with them.

Single-pin Vs Quadrature Encoders​

The Viam Rover uses motors with single-pin integrated encoders. The difference between single-pin and quadrature is that quadrature has at least two light sources and sensors set at right angles from one another. By comparing the times between the ticks from the different pins, it's possible to tell which sensor detected a change first, and hence both the speed and direction that the shaft is spinning.

Single pin encoders don't have this ability. They can detect speed, but not direction. My control system requires that the direction is also known, which means I have to infer the direction that wheels are being driven based on how they've been told to move. That generally works, as long as the robot only moves under its own power! As soon as you drag the robot backwards to a new position, the robot believes it's been moved forward because the last commanded direction was forwards, and everything gets out of synchronisation again.

I'll give more detail about single-pin and quadrature encoders in a future post.

Encoder Signal Noise​

The signal from the encoders suffers from a lot of high-frequency noise when the motors are being powered. The only time this doesn't happen is when I'm manually turning the wheels, which is exactly how I was initially trying to calibrate the robot! I would slowly turn the wheel ten times, then divide the number of received pulses by ten to get the ticks (pulses) per revolution. When I ran the robot for real, the motors made the tick frequency incredibly high, throwing off any measurements I took - the robot looked like it was zooming along!

I tried a software filter to reject the high-frequency noise, but the real signal was already lost in the noise, so all I succeeded in doing was capping the tick rate at the frequency cutoff.

Eventually, I reluctantly added RC filters to both signal lines to reduce the motor noise. I wanted to avoid this because it adds build complexity for anyone copying my build, and it requires extra parts. Still, the encoders were all but useless without the filters.

Ticks Vary Non-Linearly With Speed​

The ticks per revolution seems to vary non-linearly with the speed. By that, I mean that doubling the speed did not double the tick rate - it would be slightly off. Honestly, I don't know why. All I know is that measuring the graph got me the following results for speed versus ticks per revolution:

This is an issue because I can't tell if the motors are being driven at the wrong speed or if the encoders measure different speeds as they spin slower/faster. I assume that the encoders are correct because they're the simpler mechanism, so I had to add a new calibration step. I drive the motors at different PWM values and measure the speed according to the encoder ticks, then use this to create a Look-Up Table (LUT). When the system needs to drive at a particular speed, the required PWM is interpolated from the table and sent to the motors so the robot actually drives at the speed it's been commanded to.

Inaccurate Calibration​

Even with the hardware filter in place, calibrating based on manually turning the wheels gives a poor result from the robot. Again, I'm not sure why not! Instead of putting the robot on a box and carefully measuring the ticks per revolution, my current method is to put the robot on the floor, drive it forwards at a steady speed, and measure how far it travelled. Using the distance travelled and the ticks counted over that distance, I can calculate the ticks per revolution.

However, even that isn't perfect! The motors don't perform identically, which means the robot slightly turns as it moves, veering to one side or the other. This should be easily fixed using a feedback loop that measures the speed and changes the commanded speed to compensate. However, we haven't yet calibrated the speed measurement! Instead, I guess initial values for ticks per revolution, then run the full closed-loop control system to drive the robot and adjust the ticks on each side by eye until the robot drives in a straight line. Once it's driving in a straight line, only then can I perform the calibration process.

Still, in the end, I get values of ticks per revolution for each side that drive the robot in a straight line and allow for somewhat accurate measurement of distance travelled and amount the robot has turned. Having good values here is vital, because building a map from bad robot odometry is very difficult. It leads to phantom obstacles and walls in the wrong place. Worse than that, though - inaccurate odometry means navigation drives the robot erratically, so it frequently drives into walls/obstacles as well as building an inaccurate map.

Building a Map​

I started from Intel's Wandering Application on AAEON robot, which was originally meant for the UP Xtreme i11 Robotic Kit using the UP development kits.

The Wandering AAEON example uses the ros2_amr_interface to connect the ROS 2 navigation system to the AMR's controls. As I was starting from scratch, I didn't have AMR controls to interface with, which meant I could directly implement a ros2_control system and connect it directly to the navigation system. The example still gave a great starting point with mapping built from camera data only, and a wandering application to show it in use.

This starting point is especially useful because most autonomous mapping robots use lidar as their source of truth for the environment. Using depth data instead is a process known as Visual SLAM or VSLAM, and it means the robot can only map ahead of itself, where the camera is pointing. The launch file contains a node for converting depth data from the camera into a scan, which looks like it could have come from a lidar, but in a narrow cone in front of the robot; from here, it can be used by the mapping node rtabmap. Hence, the map has to be built by turning the robot to look in every direction.

Another change I made to the application was to have a mapping mode toggle. With mapping on, the robot clears its memory of previous maps, then moves as commanded by a user until a space is completely mapped. With mapping off, the robot loads its existing map and starts wandering. Hence, to build a map, I start by running the wandering application in mapping mode, then using Foxglove teleoperation controls to move the robot around the space until everything is completely mapped out. The command is as follows:

ros2 launch viam_rover2 wandering_mapper.launch.py mapping:=true

With the mapping stage complete, I had a complete map of my office that I could use for the next part, built only from depth and infrared data from the RealSense D421 camera.

Wandering the Map​

Once I had a usable occupancy map, I switched to non-mapping mode (or, autonomous mode). This is the mode that allows the robot to wander its saved map, but first I have to tell the robot where it is! The issue here is that the robot can load the map, but has to guess where it is within the map, and I've yet to see the robot get it right the first time. Hence, the trick is to use Foxglove to send an initial pose, which sets the robot to the position you know it to be. You always know it's correct when the laser scan visualisation matches up with the obstacles in the environment.

The wandering application is set on a delay timer, both to give the navigation system time to start up, and to allow me to set the initial pose before it starts wandering. The command is the same except with the mapping argument removed.

ros2 launch viam_rover2 wandering_mapper.launch.py

The first step is to set the initial pose. This is a bit time sensitive, as the wandering application starts up before too long (this can be adjusted in launch file parameters!). The clip below shows me setting the initial pose three times; first, based on my idea of where the robot is; the latter two were adjustments to line the scan up with the environment correctly.

With the initial pose set and the robot in the right position, it started wandering! I'm not sure if the purpose of the application is to completely map a space or simply wander forever, but my robot seems happy to just keep wandering around. I tend to leave it wandering around (a bit like a pet!) until it gets stuck on an obstacle, like the leg of my desk. The robot is generally pretty good at avoiding obstacles, but obstacles lying flat close to the ground can be hard for it to see and avoid. You can see one example of wandering below:

Can I build this robot?​

Yes! The whole idea behind this project was to build an autonomous mapping robot from easily accessible parts. I tried to keep the cost and build complexity to a minimum. This section contains the required parts and some extra parts that you might already have lying around.

Required Parts​

Note that prices are given based on GBP, which includes VAT, so you might be able to buy these for less!

Parts you may already have​

On top of the required parts above, there are several parts needed that you might already have in stock. In case you haven't, I've given links to compatible hardware. The downside is that they are mostly sold in kits, as shops don't want to sell 1-2 resistors at a time, so you're likely to have a lot of parts left over. Still, maybe that will be good for future projects!

Hopefully, this is enough to get you going building it yourself. I will release more blog posts and videos in the future to show the full process in detail.

Conclusion​

Taking the Viam Rover and converting it to use the Radxa X4 and Realsense D421 took a lot longer than I'd hoped! It was a journey of finding the best hardware modifications, writing custom firmware to communicate between Intel chip and RP2040, getting correct robot odometry from encoder data, and debugging a full ROS 2 navigation stack. It required more hardware changes than I initially wanted, but these turned out to be required because of the encoder noise that made them so unreliable.

In the end, this is a project you should be able to replicate for a relatively low price. It provides a way to study closed-loop feedback mechanisms with ros2_control, explore using the ROS 2 Nav2 stack, and operates using only depth data, where most exploration robots use some kind of lidar.

The source code is all available on GitHub, and I'll be publishing follow-up posts and videos covering the build process and the software in detail. Subscribe to my YouTube channel or check back here (RSS feed) to stay up to date.

This post is a summary of the talks with links and my own thoughts. If you'd rather go direct to the source, go to the FOSDEM 2026 website (specifically, the Robotics & Simulation track page).

Summary​

These are the main trends that I noticed over the whole day of talks:

• ROS 2 remains the backbone of the robotics open source community. It was mentioned in virtually every talk, which is unsurprising given its popularity in the robotics space. Even the talks that presented alternatives to ROS, like Copper-rs, compared those alternatives to ROS.

• A surprising theme of the dev room was the use of Rust. It came up in a surprising number of talks, including those that weren't explicitly about Rust libraries like rclrs and ROS-Z. While Rust tools are still new to the scene, they are no longer untested, and are being integrated in many places in open source software. Examples include use in Gazebo and using pixi as a build tool.

• Speaking of pixi, this was another theme, although explicitly mentioned in fewer talks than Rust. Previously, I only knew of it for dependency management, but the speakers demonstrated it being used for packaging and distributing robotics software as well. It seems like a tool that solves many pain points of building for robots.

• Finally, containerisation seems to be the de facto standard for distributing robotics software. Apptainer offers an interesting way to run containers and is compatible with Docker, while talks like Productionising ROS with Bazel showed mature pipelines using Docker alongside hermetic builds.

Conferences like FOSDEM are excellent for getting a view of what the robotics community is excited about. It seems like ROS 2, Rust, pixi, and distributing software using containers are the current trends. I'll be sure to attend FOSDEM 2027 to see if those trends are continuing - hopefully there will be a larger room available to accommodate all the roboticists!

Special Mentions​

The full talk list is below, but I wanted to highlight a few of the talks from the list.

First, Esteve's talk on rclrs was great to see. I was familiar with a lot of the concepts and arguments, as I've covered rclrs in a couple of posts on this blog already (see Comparing Rust Libraries for ROS Nodes and Building a ROS2 node in Rust!). Still, it was great to hear from its creator about what went into it and how it will be officially supported in the ROS build farm soon. Thank you, Esteve!

Next, the longer talk from Botronics was excellent. Seeing the decisions and mistakes that went into a full robot system was fascinating and eye-opening. Not only that, but Botronics will be hosting ROSCon Belgium this year, so make sure to submit papers and buy tickets if you're interested! I'll update this post with links as soon as they're available.

Another talk with some excellent advice was the "Calibrate good times" talk from Roland and Sam. While I haven't developed data collection systems for robots other than for debugging, I still found the advice here excellent, and anyone looking to collect robotics data should give it a watch.

Finally, a couple of talks inspired me to look deeper into the presented material. In the future, I hope to build the Upkie robot for myself, and I also hope to try copper-rs out on a real robot. In both cases, I'll write up and record my experience to share here. Watch this space or my YouTube channel if you're interested to see that.

Full Talk List​

Each talk is listed below with its duration, a reason you might be interested to watch, and information I could get from the speaker's page about their background.

If any speakers would like their information updated, please contact me and let me know!

• Turning a cheap commercial vacuum cleaner into a useful Open Source mapping tool (25m) - Stef Dillo of Familiar Robotics

  You should watch if:

  • You work with making maps from real sensor data
  • You're interested in open source mapping robots
  • You enjoy reverse engineering products from others!

Calibrate good times​

• Calibrate good times! The tools and methods to get top-quality robot data (25m) - Roland Meertens of Wayve, Sam Pfeiffer of Humanoid

  You should watch if:

  • You want excellent advice, built from experience, about gathering and working with clean data from as early as possible
  • You're gathering data, but not making sure it's clean!

Copper-rs​

• Bridging robotics and systems programming: Why Copper-rs is a game changer (25m) - Guillaume Binet of Copper Robotics

  You should watch if:

  • You're interested in alternatives to ROS for robotics frameworks
  • You want to see determinism in your robots
  • You want higher performance than ROS can offer from your robots
  • You like Rust!

Apptainer​

• Apptainer: Easy Containerization for Robotics (5m) - Malte Schrader

  You should watch if:

  • You like Docker for packaging software, but want a better way to provide system access to containers

• Just1 - An Open-Source Autonomous Mecanum Wheel Robot (5m) - Nicolas Rodriguez

  You should watch if:

  • You want to build a very cheap robot that still has lidar and camera
  • You're interested in mecanum wheels, which allow a 4-wheeled robot to strafe left and right
  • You want a robot with manual and autonomous control out of the box

• Modernizing ROS 2 Skills: Hacking and Orchestrating Cloud Brains, Physical Sensors, and the Network (5m) - Miguel Xochicale of University College London

  You should watch if:

  • You want to provide access to ROS for multiple users (in this case, students)
  • You want to use a mainframe for ROS and have clients connect into it
  • You want to effectively network between multiple ROS containers

• Benchmarking platform for robot localization systems (5m) - Júlia Marsal of Ekumen

  You should watch if:

  • You're doing navigation or localisation with your robot and want to explore the best option
  • You want to autonomously test localisation conditions, including in Continuous Integration
  • You want to view detailed metrics from benchmarks generated with "lambkin", the presented software

• rosidlcpp: A Journey Through ROS2 Build Time Optimization (5m) - Anthony Welte

  You should watch if:

  • You're interested in the core parts of ROS compilation
  • You're interested in optimisation techniques
  • You want to see where improvements to ROS come from!

Gazebo’s future​

• A Core Developer's insights on Gazebo's Future (45m) - Jose Luis Rivero, co-founder of Honu Robotics and Project Management Committee member of the Infrastructure and Gazebo project

  You should watch if:

  • You're interested in the history of Gazebo
  • You want to hear what influences decisions about Gazebo's future, particularly with regards to available technologies
  • You want to know what's in store for Gazebo
  • You want to see Rust being used in more places!

rclrs​

• Introducing rclrs: the official ROS 2 client library for Rust (25m) - Esteve Fernández, an original author of ROS 2

  You should watch if:

  • You're interested in using Rust for robotics (ROS in particular!)
  • You want to hear from a core ROS 2 developer
  • You want to hear news of Rust being better supported in ROS

Upkie​

• Open-Source Robotics in Practice: Lessons from Upkie Wheeled Bipeds (25m) - Stéphane Caron

  You should watch if:

  • You're interested in open source robots in general
    • The speaker maintains awesome-open-source-robots, a list of open source robots
  • You want to build an open source robot of your own
  • You want to learn practical Reinforcement Learning

• Middleware Pain? Meet iceoryx2 (25m) - Michael Poehnl of Ekxide

  You should watch if:

  • You're interested in more ways to move messages around than ROS supports
  • You want to use more advanced events for acting on data
  • You're interested in very high performance robotics, including shared memory transport

• Precision Landing with PX4 and ROS 2 using Aruco Markers (25m) - Ramon Roche, Beniamino Pozzan

  You should watch if:

  • You want to see how to integrate PX4 with ROS 2
  • You want to hear about custom controllers in PX4
  • You want to see a demo run successfully right at the last moment!

• Simple, Safe, Open: Building Your First ROS 2 Rover with Rust and Pixi (10m) - Christophe Simon, Nicolas Daube of Botronics

  You should watch if:

  • You're interested in the cheapest possible rover build
  • You want better ways to manage dependencies and export builds
  • You're interested in Rust

• Vehicle Dynamics Sim: accurately and easily simulate actuation limits (10m) - Arne Baeyens of Intermodalics

  You should watch if:

  • You have issues with vehicle dynamics in commonly-used simulations
  • You want your real robot to behave more closely compared to simulation
  • You want to simulate by only tweaking a few parameters

Productionising ROS​

• Productionising ROS when you have no choice (with Bazel) (10m) - Ricardo Delfin of Humanoid

  You should watch if:

  • You want hermetic builds for your robots
  • You want to see new, reliable ways of building Docker images for your robots
  • You want to use a production stack for building and deploying robot software reliably

• ArduPilot Advanced Integration (10m) - Pierre Kancir

  You should watch if:

  • You're interested in different drone builds
  • You want to see the internals and parts that make up a drone
  • You want to build drones without using ROS

Botronics iXi​

• The Technical Stacks Behind Botronics' iXi Autonomous Golf Trolley (45m) - Antoine Van Malleghem, Enzo Ghisoni, David Moli of Botronics

  You should watch if:

  • You want advice from a company making and selling real robots
  • You want to see the full tech stack of a working robot
  • You want to hear about what went into the tech stack that the company settled on, including the mistakes they made

ROS-Z​

• ROS-Z: A Rust/Zenoh-native stack, fully ROS 2-compliant (25m) - Julien Enoch, Yuyuan Yuan of ZettaScale

  You should watch if:

  • You're interested in Rust in robotics, particularly ROS
  • You want to use a higher performance version of ROS, as ROS-Z replaces many layers of ROS while remaining compatible with ROS
  • You want to see more from the makers of Zenoh!

• EasyNav: An open-source framework for navigating everywhere (25m) - Francisco Martín Rico, Francisco Miguel Moreno of Universidad Rey Juan Carlos

  You should watch if:

  • You have navigation cases that are difficult to solve with Nav2
  • You want to use multiple types of maps to navigate with
  • You need to work with multiple elevations in the same map

• AutoAPMS: Lightweight and versatile integration of behavior trees into the ROS 2 ecosystem (25m) - Robin Müller

  You should watch if:

  • You want to define behaviour trees in a very flexible way (using CMake)
  • You want to see the power of behaviour trees
  • You enjoy seeing how a simple example can explain complex behaviours

• PlotJuggler: the log visualization tool loved by roboticists (25m) - Davide Faconti

  You should watch if:

  • You want to see a more entertaining talk (I enjoyed the James Bond theme!)
  • You're interested in the future of a very widely-used data visualisation tool
    • The audience support for PlotJuggler was incredible!

FOSDEM 2027​

FOSDEM will be back in 2027! If you're interested, you can get updates by checking the FOSDEM website. There will be a page for 2027 as soon as it's available, which is likely to be late in 2026.

This post walks through building a simple ROS 2 node that detects objects in real time using a robot's webcam and the Ultralytics YOLO26 model. The purpose is to show how simple it is to deploy Ultralytics models on a robot using the ultralytics Python package, as well as walking through a simple ROS 2 node for CV.

▶Watch: Deploying Ultralytics YOLO26 Using ROS 2

Ultralytics YOLO26 Infographic

Ultralytics YOLO​

YOLO Models​

Ultralytics provides a suite of CV models called YOLO, or You Only Look Once, designed for efficient and accurate CV tasks. These models are widely used due to their strong real-time performance, simple APIs, and broad hardware support. In my case, it was extremely simple to deploy a model to the OriginBot, although it only runs on the CPU; the RDK X5 on board has hardware acceleration available, but I wanted to keep the ROS 2 node simple so it would be easy to understand. See my article Exploring AI on the RDK X3 Sunrise for more information on the RDK X5's predecessor, the RDK X3.

YOLO models offer the following applications:

• Object Detection: Identify and locate multiple objects within a frame.
• Segmentation: Generates masks, or pixel regions, which belong to a particular object. For example, highlight the areas of an image occupied by ships in a dock.
• Image Classification: Classify an image as belonging to a particular class. For example, an image containing a keyboard may be given the Keyboard class.
• Pose estimation: Estimate the position and orientation of any humans within a frame. Each humans can have a simple skeleton drawn showing the location of their heads, torsos, and limbs.
• Tracking: Tracks an object or person between frames of a video, showing the change in position over time.

Ultralytics YOLO26 is available under two licensing options. The AGPL-3.0 license is an OSI-approved open-source license, perfect for students, researchers, and enthusiasts who want to collaborate openly and share their work. For commercial use or private R&D, the Ultralytics Enterprise License allows seamless integration into proprietary products and services without the open-source requirements of AGPL-3.0. I'm using the models under the AGPL-3.0 License for this demo.

Ultralytics YOLO26​

Ultralytics has released multiple sets of YOLO models over the years, including YOLOv5, YOLOv8, and YOLOv11. The latest model that was released (generally available as of January 2026!) was Ultralytics YOLO26, which is the model we will use for the demo.

Ultralytics YOLO26 is a drop-in replacement for previous YOLO models, so if you're using any other YOLO models (e.g. YOLOv5), it is simple to update your code to use the new model. There are a few advantages to upgrading, as the new architecture of Ultralytics YOLO26 offers faster inference, better performance on large and small objects, and broad hardware compatibility.

The demo from this post only runs on CPU, although it could run much faster with hardware acceleration, and Ultralytics aims to support as many types of hardware acceleration as it can.

Running Ultralytics YOLO26 model on my robot​

To test Ultralytics' claim that the model is easy to deploy, I wrote a simple ROS 2 node that receives images from a camera and performs inference on them. More information on how to build and deploy the node in this sections.

I ran this node on my OriginBot, which uses the RDK X5 as a brain. The RDK X5 has hardware acceleration on board that can run YOLO models extremely quickly, but I chose to run the model using CPU to keep the node simple and easy to understand. Hence, the images come through more slowly than the board is capable of.

I also ran the node on my desktop computer, as it could process the images much faster, so that I could get some examples of the detections from the model. Hence, running the yolo26n.pt model on my desktop gave some of the following results. I tried a series of objects I had nearby to see what would be recognised by the model to select the objects used below.

First, the model performed very well with fruit like apples and bananas, even when the image wasn't properly focused.

The model also performed well for my phone.

The model had more trouble with objects like these scissors and toothbrush, but could still recognise the objects most of the time. I found that the model could recognise scissors more easily when up close and opened up.

There were also some objects that the model didn't recognise well. My examples are a water bottle and a screwdriver.

Finally, as a bonus, my cat was mostly recognised as a cat, but was also labelled as a teddy bear and a dog. I enjoyed that the teddy bear confidence (45%) was higher than the cat confidence (27%), depending on the frame.

The dog label isn't too surprising to me. Depending on the angle, it can be hard to tell cats and dogs apart.

Now that we've seen some examples of objects detected by the network, let's look at how to deploy the model to a real robot.

How to run Ultralytics YOLO26 in a simple ROS 2 node​

I tend to build robotics applications using the ROS 2 framework. ROS 2 comes with a lot of advantages, but the main reason I use it is that it's free and very widely used, which makes it a great teaching tool. For more details on ROS 2, take a look at my previous blog post Getting Started with ROS 2.

ROS 2 works by separating robot tasks into different nodes that communicate by passing messages. In a robot with Computer Vision, we might have one node that reads images from the camera and publishes them, and another node that subscribes to those images and performs inference on them. That's exactly the setup in today's demo.

Let's go through installing dependencies and running the code before taking a closer look at the code in the Code Deep Dive section.

Install ROS 2​

The first step on a robot is to install ROS 2. If your robot is running Ubuntu, it's very simple - just follow the instructions here.

After installing ROS 2, source the setup file. Change humble for the version of ROS 2 that you've installed:

source /opt/ros/humble/setup.bash

This needs to be done in every terminal session that you run ROS 2 commands in. If you ever get the error that ros2 isn't recognised as a command, try running this source command again.

Create a Workspace​

To create the workspace for this example, run the following:

mkdir -p ~/yolo_ws/src

This creates the source folder of the workspace, into which we can clone the example node:

git clone https://github.com/mikelikesrobots/yolo26_ros2/ ~/yolo_ws/src/yolo26_ros2

As can be seen from the line above, all of the code for the demo is kept on my GitHub in the yolo26_ros2 repository.

With the code checked out, we can now install dependencies and build the node. For this, we will use the rosdep tool that comes with ROS 2, which makes it very easy to install extra dependencies for packages as long as they're recognised by the ROS package index. Execute the following:

# Only needs to be executed once per system
sudo rosdep init
# Run these for any new package, including our example node
rosdep update
rosdep install --from-paths ~/yolo_ws/src --ignore-src -r -y

git clone -b humble https://github.com/ros-perception/vision_opencv.git ~/yolo_ws/src/vision_opencv
cd ~/yolo_ws
colcon build --packages-select cv_bridge --cmake-args -DCMAKE_BUILD_TYPE=Release

With all dependencies installed, we can now build the workspace:

cd ~/yolo_ws
colcon build --symlink-install

The code is now fully built and ready to run.

Running the Node​

The package only builds one node called yolo26_ros2. We could run this node by itself, but it's not very useful without being supplied with camera images. To get images from a webcam, we can use a package called usb_cam, which is available for download without us having to build anything. In fact, it's already installed, as it is one of the yolo26_ros2's dependencies - we only need to run it at the same time as our CV node.

To make running both nodes at the same time easier, the package includes a launch file, which is configured to launch the yolo26_ros2 node and the usb_cam node at the same time. The usb_cam node will open the first available camera on the system and publish images from it using ROS 2. By launching these two nodes together, we can open a webcam, move images from the usb_cam node to the yolo26_ros2 node, and perform inference on them.

source ~/yolo_ws/install/setup.bash
ros2 launch yolo26_ros2 yolo_camera.launch.py

If you want to run the demo node by itself, use this ros2 run command instead:

source ~/yolo_ws/install/setup.bash
ros2 run yolo26_ros2 yolo26_ros2

Either way you launch the demo node, it will automatically download the model you've requested, which is yolo26n.pt by default, and save it to the share directory of the ROS 2 workspace.

Viewing the Output​

The yolo26_ros2 publishes 3 outputs, mostly for demonstration purposes. For your own use, you can remove the outputs you don't actively need, or separate them into different nodes. The 3 outputs are:

The easiest way to view the output is to use a tool to visualise the CompressedImage messages. You can either use rqt for this, or my preferred tool, rqt_image_view:

ros2 run rqt_image_view rqt_image_view /image_annotated/compressed

You should be able to see the images streaming live from the robot! You can now try moving it around and seeing what it can detect.

YOLO Model Parameter​

Computer Vision algorithms tend to be divided into classical and Machine Learning. Classical algorithms work by spotting features in images, like edges, whereas Machine Learning versions will build a "model" from training data. This means that it uses a huge set of images with labels of what it should be looking for, and it learns to recognise those labels in those images. Switching out the model will change how the algorithm behaves.

The default model used here is yolo26n.pt, but this isn't the only model you could use. The model is set by a parameter, which is a ROS 2 way to change the configuration of nodes when they start up. In this case, we can set the model_name parameter with a different model name, and the node will attempt to load that model instead. For example, yolo26n.pt and yolo26s.pt are both valid. Take a look at the metrics for the Ultralytics YOLO26 models to select the right model for yourself. For this example, make sure to use a "Detect" model, as the other models work differently.

To set the parameter, we can either edit the launch file:

# Change the default_value argument
model_name_arg = DeclareLaunchArgument(
    "model_name", default_value="yolo26n.pt", description="YOLO model filename"
)

Or we can override the parameter when we launch the node:

ros2 launch yolo26_ros2 yolo_camera.launch.py model_name:=yolo26s.pt

In either case, launching the node will download the new model and run it instead, which gives us the ability to easily change out the model we're using as well as easily deploy it.

Now that we're able to run the node and change the model being used easily, let's take a look at how the code to see how this happens.

Code Deep Dive​

From running the example, we've already seen that the package:

• Allows ROS 2 to automatically install package dependencies
• Copies any models in the models directory to the build output
• Contains a launch file for launching the usb_cam node at the same time as the yolo26_ros2 node
• Contains a node that:
  • Can be configured using parameters to change the model being loaded
  • Loads the model using the ultralytics Python package
  • Subscribes to images from the webcam
  • Performs CV inference on those images using the Ultralytics YOLO26 model
  • Publishes the outputs as a Detection2DArray, an annotated image, and a compressed version of the annotated image

This section looks closely at the code to show how each of those functions are performed.

ROS 2 Package Setup​

This package started with the default Python package template. The template includes package description file package.xml, a Python package build file called setup.py, and the source code of the node in yolo26_ros2/yolo26_ros2.py.

Package Description​

This file contains information about the package, such as the author and the license of the package. It also has a section for package dependencies, which tells ROS 2 which dependencies need to be installed to run the package. These lines state the dependencies for the package:

<depend>rclpy</depend>
<depend>sensor_msgs</depend>
<depend>cv_bridge</depend>
<exec_depend>vision_msgs</exec_depend>
<exec_depend>python3-ultralytics-pip</exec_depend>
<exec_depend>usb_cam</exec_depend>

This means that when rosdep is used to install dependencies for this package, it will look up the right packages to install based on these keys and install them. It also tells the build system that this package needs access to the dependencies to function correctly.

Python Package Build File​

The setup.py file also states the package author, as this is standard for a Python package, but it has some extra parameters describing what should be copied during the package build. These lines state that all files in the models and launch folders should be copied to the output:

data_files = [
    ('share/ament_index/resource_index/packages',
        ['resource/' + package_name]),
    ('share/' + package_name, ['package.xml']),
    ('share/' + package_name + '/launch', glob.glob('launch/*.launch.py')),
    ('share/' + package_name + '/models', glob.glob('models/*')),
]

When colcon is used to build this package, these data files will be copied to the build output. You can use this to download a YOLO model to the models, then copy it to the build output automatically with colcon.

Launch File​

The launch file system is more powerful than a list of nodes to launch at the same time. We are using it for a simple use case, which is to provide the two nodes to launch and the parameters that can be tweaked as we launch them.

The launch file is available at launch/yolo_camera.launch.py.

It starts with imports and a function to generate the launch description, which is used by ROS 2 to launch the nodes. The first lines in the launch description are to set parameters, such as the model name:

model_name_arg = DeclareLaunchArgument(
    "model_name", default_value="yolo26n.pt", description="YOLO model filename"
)

The other parameters are similarly structured.

The launch description then defines the nodes to be launched. For example, the yolo26_ros2 node looks like this:

yolo_node = Node(
    package="yolo26_ros2",
    executable="yolo26_ros2",
    name="yolo_detector",
    parameters=[
        {
            "image_topic": LaunchConfiguration("image_topic"),
            "detections_topic": LaunchConfiguration("detections_topic"),
            "annotated_image_topic": LaunchConfiguration("annotated_image_topic"),
            "model_name": LaunchConfiguration("model_name"),
        }
    ],
    output="screen",
)

This defines that the yolo26_ros2 executable from the yolo26_ros2 package should be run with the given parameters, and the node name should be yolo_detector. The output argument just says that the log statements should be printed to the screen when the launch file is run.

Finally, the parameters and nodes are collected together into one list, which forms the LaunchDescription.

return LaunchDescription(
    [
        model_name_arg,
        image_topic_arg,
        detections_topic_arg,
        annotated_image_topic_arg,
        usb_cam_node,
        yolo_node,
    ]
)

yolo26_ros2 Node​

This node (yolo26_ros2/yolo26_ros2.py) is where most of the work is being done. Let's go through the file section by section to see how it all fits together.

The first section contains imports for the package. This is standard Python, so we will skip past.

The next line creates a class for the detector node, which extends the ROS 2 Node class. It creates a Python constructor and calls the Node's constructor with the node name (source).

class YoloDetectorNode(Node):
    def __init__(self):
        super().__init__("yolo_detector")

The function continues by setting up parameters. These are ways to configure the node's behaviour, such as changing the YOLO mode to be used. Each parameter is declared to ROS 2, then the value of that parameter is retrieved from ROS 2.

# Declare parameters
self.declare_parameter("image_topic", "/image_raw")
self.declare_parameter("detections_topic", "/detections")
self.declare_parameter("annotated_image_topic", "/image_annotated")
self.declare_parameter("model_name", "yolo26n.pt")

# Get parameters
image_topic = self.get_parameter("image_topic").value
detections_topic = self.get_parameter("detections_topic").value
model_name = self.get_parameter("model_name").value
annotated_image_topic = self.get_parameter("annotated_image_topic").value

The node sets up a CvBridge and finds the right location to load the model from, then constructs the model (source). This is the point where the ultralytics library will automatically download the model if it's not already available on disk.

# Load YOLO model from package share directory
package_share_dir = get_package_share_directory("yolo26_ros2")
full_model_path = os.path.join(package_share_dir, "models", model_name)

self.get_logger().info(f"Loading YOLO model from: {full_model_path}")
self.model = YOLO(full_model_path)
self.get_logger().info("YOLO model loaded successfully")

The model is now loaded and ready for use!

The next lines set up the publishers and subscribers for ROS 2. One subscriber subscribes to Image messages from the camera, giving an image_callback function as an argument, which is called whenever an Image message is received. The other lines set up publishers which can publish messages later in the code.

# Create subscriber and publishers
self.img_sub = self.create_subscription(Image, image_topic, self.image_callback, 10)
self.det_pub = self.create_publisher(Detection2DArray, detections_topic, 10)
self.annotated_img_pub = self.create_publisher(Image, annotated_image_topic, 10)
self.compressed_img_pub = self.create_publisher(
    CompressedImage, annotated_image_topic + "/compressed", 10
)

Every time an Image is received by the node, it calls the image_callback function with that message as an argument. The function immediately converts the message from the ROS 2 Image message to the OpenCV format, which can be passed to the loaded model.

def image_callback(self, msg):
    try:
        # Convert ROS Image to OpenCV format
        cv_image = self.bridge.imgmsg_to_cv2(msg, desired_encoding='bgr8')

The model can then perform inference on the image:

# Run YOLO detection
results = self.model(cv_image, verbose=False)

self.model = YOLO(full_model_path)
results = self.model(cv_image, verbose=False)

Now that the node has the results of the inference, it can package it up into the three different outputs. First, it goes over all the results and builds a Detection2DArray message:

# Create Detection2DArray message
detection_array = Detection2DArray()
detection_array.header = msg.header

# Process each detection
for result in results:
    boxes = result.boxes
    for box in boxes:
        detection = Detection2D()
...
# Publish detections and annotated image
self.det_pub.publish(detection_array)

The lines I've skipped are building the individual Detection2D messages from the results of the inference.

The node also takes the original image and adds green rectangles, class labels, and score percentages for each one (source):

cv2.rectangle(cv_image, (x1, y1), (x2, y2), (0, 255, 0), 2)
label = int(box.cls[0])
label_readable = self.model.names[label]
score = float(box.conf[0])
cv2.putText(...)
annotated_img_msg = self.bridge.cv2_to_imgmsg(cv_image, encoding='bgr8')
annotated_img_msg.header = msg.header
self.annotated_img_pub.publish(annotated_img_msg)

One of the key parts of drawing the bounding boxes is labelling the box with what's inside it. However, the model's prediction contains a number rather than a name for the class. That means we need to look up the name from the model. We can look up the name using this code:

label_readable = self.model.names[label]

At this point, the node has published an array of objects it detected in the image, plus the original image annotated with bounding boxes for each detection. The last output is simple - it's the JPEG compressed version of the annotated image. This is purely done so that if you are viewing the images across the network like I do, the performance will be much better.

compressed_image_msg = CompressedImage()
compressed_image_msg.header = msg.header
compressed_image_msg.format = "jpeg"
compressed_image_msg.data = np.array(cv2.imencode(".jpg", cv_image)[1]).tobytes()
self.compressed_img_pub.publish(compressed_image_msg)

After this, there are some log statements to print information to the console and exception catching to make sure errors are handled correctly.

The final part of the node is very simple. It's the main function, which constructs an instance of the YoloDetectorNode class, then runs it until the process is shut down.

def main(args=None):
    rclpy.init(args=args)
    node = YoloDetectorNode()

    try:
        rclpy.spin(node)
    except KeyboardInterrupt:
        pass
    finally:
        node.destroy_node()
        rclpy.shutdown()

And that's the complete file! The three functions that we used from ultralytics were to load a model, perform inference on an image using the loaded model, and look up the name of a class from the model given the label number from a prediction.

# Load the model from disk, or download if not available
self.model = YOLO(full_model_path)
# Perform inference on an image
results = self.model(cv_image, verbose=False)
# Look up the name for a label number from a detection
label_readable = self.model.names[label]

It's clear that Ultralytics has worked hard to make their ultralytics Python package as simple to use as possible. That's great news for robotics engineers, because intelligent robotics almost certainly needs some kind of CV!

Summary​

In this post, we saw a simple ROS 2 node for performing Computer Vision (CV) inference on images from other ROS 2 nodes. We combined this node with the usb_cam node, which can be used to stream images from a webcam. All of this runs directly on a robot using the CPU, although other hardware acceleration is available.

By looking into the structure of the node, we saw how the ultralytics Python package could be used to download, load, and use the Ultralytics YOLO26 model. With just three lines, we could detect objects in an image with a confidence value, and we could also change the model in use by changing a ROS 2 parameter.

The Ultralytics YOLO26 model proved extremely easy to deploy on a robot. The other reasons to use Ultralytics YOLO26, such as faster inference and better performance on large and small objects, are up to you to test! Try the node for yourself and see what kind of performance you can get from various models. Also see if you can enable hardware acceleration, such as enabling cuda on a machine with a graphics card.

This article also includes a video walkthrough.

▶Watch: Comparing Rust Libraries for ROS Nodes

Rust in Robotics​

My previous post on building a ROS 2 Rust node goes into some depth on this - take a look at the post in question.

Essentially, Rust is a systems level language with an emphasis on speed, safety, and concurrency. It is comparable to C++, which is heavily used for robotics code, particularly real-time and high-performance applications like processing high frequency sensor data. As Rust can operate in the same space as C++ and comes with some advantages, such as more user-friendly tooling and helpful compiler errors, it's a great candidate for programming in robotics.

Rust can operate well at other levels too, such as backends for web services; having one language for every level of your robotics stack, right up to the cloud, does bring significant benefits in code reuse and cross-team collaboration.

I also personally find Rust fun to program in, and tend to look for projects where I can use it!

Rust in ROS​

ros2rust​

Until now, I had only considered using ros2rust for writing ROS nodes in Rust. This package provides a client library called rclrs, similar to rclcpp, that follows the standard ROS client library layout.

In the standard ROS client library layout, there are two base layers written in C, called rmw and rcl. rcl is the ROS client library. Any client library in another language, such as rclcpp and rclpy for C++ and Python respectively, then call into rcl to interact with ROS. That means that any changes made to rcl will automatically be used in rclcpp, rclpy, and any other client library - including rclrs.

It also means that nodes built with ros2rust behave like other ROS nodes. They can be built with colcon, they can be launched from launch files, and messages built in their packages can be accessed by other ROS nodes. In short, it looks like a classic ROS node, just like if it had been built in Python or C++.

roslibrust​

The contender to ros2rust in this post is roslibrust, which has a different approach to interacting with ROS. Here are a few reasons to consider using roslibrust over ros2rust:

• The programming interface is built to be async-first. Tasks can be defined and passed to an async framework to run in parallel, which has advantages if the tasks are waiting on data for long periods of time. However, for applications that are processing high volumes of data, the advantage of async is minimal.
• Instead of using the rcl library, roslibrust rebuilds everything from scratch. This means that it has no dependency on the ROS build system except for the message repositories, so you can build it without a ROS installation before deploying it to your ROS-based system. This makes the build process faster and smoother.
• roslibrust supports multiple backends by providing the Ros trait. A trait in Rust is like an interface from other languages, such as Java. Any backend implementation that supports the Ros trait can be used by the same roslibrust code, meaning that ROS1 and ROS2 can be supported by the same application code.
• The same trait abstraction allows roslibrust to support a complete mocking system, allowing for extremely fast unit tests. This is a huge selling point to me!
• The library currently passes data using rosbridge to interact with ROS 2. However, because ROS is soon switching to the Rust-based Zenoh messaging as its default middleware, roslibrust could potentially build a new backend implementation that directly calls into Zenoh. This would bypass the whole rosbridge -> rcl -> rmw -> middleware stack that it currently uses, making it much more efficient.

Other Client Libraries​

There are a few other libraries that offer ROS 2 nodes written in Rust. Two take the same approach as roslibrust, meaning that they build the ROS interaction layer from scratch, while one still provides bindings to rcl (but without relying on the ROS 2 build system). However, these projects seem to be less active than roslibrust based on Github commit history when I checked, and they didn't offer the same async-first approach. Having said that, the biggest reason I didn't include them in this post is because I haven't had time to explore them! If you're interested in taking a look at these alternatives, the links are below. If you know of any others, please leave a comment and I'll happily add them.

• ros2-client: A Rust native client library for ROS 2, using RustDDS for communication.
• r2r: Provides bindings for ROS 2 that do not require hooking in to the ROS 2 build infrastructure by relying on already-generated C code.
• rosrust: A pure Rust implementation of a ROS client library.

Let's take a look at a sample application to see the differences between roslibrust and ros2rust in action.

Sample Application​

This section demonstrates how both libraries handle the ROS fundamentals of publishing, subscribing, and services through a simple test application. The sample application has three nodes:

1. A publisher node, which publishes an incrementing count once per second. This tests the library's ability to publish ROS standard messages.
2. A subscriber node, which subscribes to IMU data and adds up the x values of the angular velocity. This isn't helpful by itself, but does show the ability to receive data and do something with it. I can also test this with a Gazebo simulation to check that the node is integrated into the ROS 2 system correctly, rather than relying on command-line tools alone.
3. A service for multiplying two numbers together. Crucially, this uses a custom ROS message to test that both libraries can handle custom messages.

With this specification in place, the two implementations are available at https://github.com/mikelikesrobots/imu_tracker_roslibrust and https://github.com/mikelikesrobots/imu_tracker_ros2rust for roslibrust and ros2rust respectively.

Installing Dependencies​

To set the repositories up, I used my WSL instance of Ubuntu with ROS 2 Jazzy installed. Each repository got its own ROS 2 workspace. This is where the first caveat comes in: the roslibrust version doesn't need to be in a workspace for its initial build. However, because it has a custom message that I need to access from the ROS 2 system later, it's easier to build it from a ROS 2 workspace to begin with.

Both packages need Rust installed, which on Ubuntu is as simple as:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then restart the terminal. ros2rust does need extra dependencies, so ROS 2 needs to be installed (installation instructions), as well as a couple of extra tools:

pip install git+https://github.com/colcon/colcon-cargo.git
pip install git+https://github.com/colcon/colcon-ros-cargo.git

These extra tools help integrate the build with colcon, such that after the first colcon build, cargo will work correctly with the package.

Finally, roslibrust requires the rosbridge package to interact with ROS, so that needs to be installed:

# Change jazzy for your ROS installation
sudo apt install -y ros-jazzy-rosbridge-server

Cloning the Code​

Setting up the ros2rust version should be familiar for ROS 2 users. The process is to create a workspace, clone the repo, then use the vcs tool to check out the remaining dependencies.

mkdir -p ~/ros2rust_ws/src
git clone https://github.com/mikelikesrobots/imu_tracker_ros2rust ~/ros2rust_ws/src/imu_tracker_ros2rust

vcs import ~/ros2rust_ws/src < ~/ros2rust_ws/src/imu_tracker_ros2rust/imu_tracker_ros2rust.repos
vcs import ~/ros2rust_ws/src < ~/ros2rust_ws/src/ros2_rust/ros2_rust_jazzy.repos

The roslibrust version is similar, but with just two clone commands. Recurse submodules is used to check out the message packages needed for message generation. The mult_msgs package contains a single custom message for the service, and could have been checked out inside imu_tracker_roslibrust. I did it this way to ensure the custom messages are available to other users of the ROS workspace.

mkdir -p ~/roslibrust_ws/src
git clone --recurse-submodules https://github.com/mikelikesrobots/imu_tracker_roslibrust ~/roslibrust_ws/src/imu_tracker_roslibrust
git clone https://github.com/mikelikesrobots/mult_msgs ~/roslibrust_ws/src/mult_msgs

Building the Code​

The ros2rust workspace must initially be built using colcon, as follows:

# Source ROS, if not already done
source /opt/ros/jazzy/setup.bash
cd ~/ros2rust_ws
colcon build
source install/setup.bash

This build will take a significant amount of time, because it's building all of ros2rust alongside the application package. Also, subsequent builds will check all of the ros2rust packages for changes, which still takes quite a bit of time even if you haven't made any changes. Instead, after the first build, you can select only the application package to build or directly use cargo:

colcon build --packages-select imu_tracker_ros2rust
# Alternatively:
cd ~/ros2rust_ws/src/imu_tracker_ros2rust
cargo build

For imu_tracker_roslibrust, the process is just to use cargo:

cd ~/roslibrust_ws/src/imu_tracker_roslibrust
cargo build

This will download roslibrust as a dependency crate and build it. However, it doesn't make our custom messages available to the ROS 2 ecosystem, so we still need to use ROS 2 for that part:

cd ~/roslibrust_ws
colcon build
source install/setup.bash

Testing Each Application​

roslibrust relies on rosbridge to pass messages, so run this as a separate node, making sure to source the roslibrust workspace to get access to its custom messages:

# Get access to mult_msgs package with custom message
source ~/roslibrust_ws/install/setup.bash
# Run rosbridge
ros2 launch rosbridge_server rosbridge_websocket_launch.xml

From here onwards, the instructions are the same, except for changing ros2rust for roslibrust. Hence, I'll only write the details once for ros2rust.

Set the application running:

source ~/ros2rust_ws/install/setup.bash
ros2 run imu_tracker_ros2rust imu_tracker_ros2rust

This gave a startup log:

Counting node has started!
counter: Published 0
IMU summing node has started!
Float multiplying service has started!

I could then subscribe to the /example_counter topic to get the count:

ros2 topic echo /example_counter

Which gave the following output:

---
data: 40
---
data: 41
---
data: 42
---

The count is incrementing, and there's one new message per second, just as expected.

Next is the float multiplication service. This needs the custom messages available to work correctly:

source ~/ros2rust_ws/install/setup.bash
ros2 service call /mult_floats mult_msgs/MultFloats "{x: 2.5, y: 4.0}"

That gave me the following:

waiting for service to become available...
requester: making request: mult_msgs.srv.MultFloats_Request(x=2.5, y=4.0)

response:
mult_msgs.srv.MultFloats_Response(result=10.0)

So far, so good! The node can accept custom messages that I've defined and give the correct response.

Finally, I can test the IMU code. This also tests that the application is able to subscribe to other topics from ROS2 other than messages I've published. To do so, I install the Turtlebot 3 simulation package with:

sudo apt install -y ros-jazzy-turtlebot3-simulations

Then ran the simulation with:

export TURTLEBOT3_MODEL=waffle_pi
ros2 launch turtlebot3_gazebo turtlebot3_world.launch.py

My node then started printing out the data it was receiving:

Got x: 0.00016605295940326384, sum: 0.0013029570806488157
Got x: -0.00010373969315866868, sum: 0.001199217387490147
Got x: -0.00007269928155955913, sum: 0.0011265181059305878
Got x: 0.00002150106849010582, sum: 0.0011480191744206937
Got x: -0.0004081654500772328, sum: 0.0007398537243434609
Got x: 0.000375131405007245, sum: 0.001114985129350706
Got x: 0.00020263544247801625, sum: 0.0013176205718287222
Got x: 0.0002873354535022987, sum: 0.001604956025331021
Got x: -0.00023864045717770534, sum: 0.0013663155681533155
Got x: -0.000059679387952645265, sum: 0.0013066361802006703

Perfect - data received from another node in a ROS 2 network successfully.

I then repeated this full set of tests with roslibrust, and got the same correct behaviour, albeit with printed lines being slightly different. Both libraries have the basics working, so the next step is to dig into the code to see how the API looks.

Comparing Code From Each Library​

Counting Node​

ros2rust

ros2rust commonly uses a pattern where each node has a struct that contains everything it needs. In this case, the struct contains the node itself, the publisher for example_counter topic, and the data that is being incremented. Each of these are wrapped in Arcs, which makes them sharable across threads. Sharing across threads makes a lot of sense in a system that allows callbacks and multi-threading as ROS does.

struct CounterNode {
    _node: Arc<rclrs::NodeState>,
    publisher: Arc<rclrs::PublisherState<Int16Msg>>,
    data: Arc<Mutex<i16>>,
}

Initialisation comes from creating a new struct from a Ros Executor object. This might have issues, so the function returns a Result in case any errors occur. The current count is stored in the struct as the data object so it can keep being incremented between publish calls.

fn new(executor: &rclrs::Executor) -> Result<Self, rclrs::RclrsError> {
    let node = executor.create_node("counter")?;
    let data = Arc::new(Mutex::new(0i16));
    let publisher = node.create_publisher::<Int16Msg>("example_counter")?;
    Ok(Self {
        _node: node,
        publisher,
        data,
    })
}

I then run the publishing node in its own thread, where each iteration of the thread publishes the count, prints out a message, increments the count, then waits for a second.

fn start(self) {
    std::thread::spawn(move || -> Result<(), rclrs::RclrsError> {
        loop {
            self.publisher.publish(&Int16Msg {
                data: *self.data.lock().unwrap(),
            })?;
            println!("counter: Published {}", *self.data.lock().unwrap());
            *self.data.lock().unwrap() += 1;
            std::thread::sleep(std::time::Duration::from_millis(1000));
        }
    });
}

roslibrust

roslibrust's way of operating is to use async tasks instead of full threads. One function creates a publisher, stores data in local variables, and starts looping. Each loop publishes the count variable, then uses tokio time intervals to wait until the next second has elapsed. Tokio is the async framework that handles all the task execution logic, which is how all the tasks can run in parallel within the same thread.

async fn pub_counter(ros: impl roslibrust::Ros) {
    let publisher = ros
        .advertise::<std_msgs::Int16>("example_counter")
        .await
        .unwrap();
    let mut counter = 0;
    let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(1));
    loop {
        interval.tick().await;
        publisher
            .publish(&std_msgs::Int16 { data: counter })
            .await
            .unwrap();
        println!("Published {counter}");
        counter += 1;
    }
}

Note that the publisher is constructed from the ros object passed in to the function, which implements the Ros trait. This is how roslibrust handles different backends; create a backend implementation that supports the Ros trait, and the code inside the function can use that implementation without knowing what it is. That means this code works just as well for ROS 1 as it does for ROS 2 - just pass in a different backend implementation. This is also how roslibrust supports excellent mocking: pass in a backend that uses mock data structures, and you can fully unit test your code. Here's a unit test doing exactly that:

    #[tokio::test(start_paused = true)]
    async fn test_pub_counter() {
        let ros = roslibrust::mock::MockRos::new();
        let mut subscriber = ros
            .subscribe::<std_msgs::Int16>("example_counter")
            .await
            .unwrap();
        tokio::spawn(async move { pub_counter(ros).await });

        // Confirm we get the first message
        let msg = subscriber.next().await.unwrap();
        assert_eq!(msg.data, 0);
        ...
    }

This test shows constructing a mock backend, subscribing to the same topic that the publisher publishes to, then building the counter and passing it to Tokio to manage. This is all the setup needed before checking received messages, and Tokio simulates time passing instead of actually waiting, meaning that all the tests run extremely quickly. The test in the source code also checks the timing of publishing and the counter being incremented, but I left these out for brevity.

IMU Summing Node​

ros2rust

In ros2rust, the subscription takes a function that can be called when a new message is received:

let subscription =
    worker.create_subscription("/imu", |data: &mut SumData, msg: ImuMsg| {
        data.sum += msg.angular_velocity.x;
        println!("Got x: {}, sum: {}", msg.angular_velocity.x, data.sum);
    })?;

The data passed in here is the running sum of data, and ros2rust will pass the updated data to each invocation of the callback function, which allows it to keep a sum of data over time.

roslibrust

In roslibrust, instead of the subscription taking a callback function as a parameter, the task itself waits for messages to be received before acting on them. Once any message is received, it adds the x value to the state variable originally passed into the function and prints out the new value. This is slightly different from the publisher function storing the data locally, but by passing in a reference to an f64 instead of using local data, we can then check the value is added to correctly in unit tests.

async fn imu_summer(ros: impl roslibrust::Ros, state: Arc<Mutex<f64>>) {
    let mut subscriber = ros
        .subscribe::<sensor_msgs::Imu>("/imu")
        .await
        .expect("Could not create subscriber!");

    loop {
        let msg = subscriber.next().await.expect("Failed to get a message!");
        *state.lock().await += msg.angular_velocity.x;
        println!(
            "Got x: {}, sum: {}",
            msg.angular_velocity.x,
            *state.lock().await
        );
    }
}

Float Multiplication Node​

In both versions of the service, the service is created with a callback function to respond to the incoming request.

ros2rust

In ros2rust, the callback function is given a request structure and some information, and is expected to return a response:

fn handle_mult_floats(request: MultFloats_Request, _info: ServiceInfo) -> MultFloats_Response {
    println!("Received request: {:?}", request);
    let result = request.x * request.y;
    println!("Responding with result: {}", result);
    MultFloats_Response { result }
}

Note that the service handler can only return a response, not a Result type.

The service is then constructed and stored in its own MultFloatsNode structure:

fn new(executor: &rclrs::Executor) -> Result<Self, rclrs::RclrsError> {
    let node = executor.create_node("mult_floats")?;
    let service =
        node.create_service::<MultFloats, _>("mult_floats", Self::handle_mult_floats)?;
    Ok(Self { service })
}

roslibrust

In roslibrust, the process is very similar. The callback function is defined first:

fn handle_mult_float(
    req: mult_msgs::MultFloatsRequest,
) -> Result<mult_msgs::MultFloatsResponse, ServiceError> {
    println!("Received request: {:?}", req);
    let result = req.x * req.y;
    println!("Responding with result: {}", result);
    Ok(mult_msgs::MultFloatsResponse { result })
}

In this case, the return type is a Result, meaning that the function can return an error if something went wrong.

The service itself is then built using:

async fn mult_floats(ros: impl roslibrust::Ros) {
    let _handle = ros
        .advertise_service::<mult_msgs::MultFloats, _>("/mult_floats", handle_mult_float)
        .await
        .unwrap();

    tokio::signal::ctrl_c().await.unwrap();
}

Waiting on the Ctrl+C signal allows the service to operate without being cleaned up until the node is terminated.

Main Function​

ros2rust

To start all of the nodes in ros2rust, an executor is the simplest method:

fn main() -> Result<(), rclrs::RclrsError> {
    let mut executor = rclrs::Context::default_from_env()?.create_basic_executor();
    let counter = CounterNode::new(&executor)?;
    counter.start();
    ...

Once all the nodes have started, the main process spins until stopped or an error occurs:

    ...
    executor.spin(rclrs::SpinOptions::default()).first_error()?;
    Ok(())
}

roslibrust

In roslibrust, the main function creates the backend for all of the nodes, then starts each node, and finally waits for the Ctrl+C signal to terminate the node.

#[tokio::main]
async fn main() {
    // Create a rosbridge client we can use
    let ros = roslibrust::rosbridge::ClientHandle::new("ws://localhost:9090")
        .await
        .unwrap();

    tokio::spawn(pub_counter(ros.clone()));
    println!("Counting node has started!");
    tokio::spawn(imu_summer(ros.clone(), Arc::new(Mutex::new(0.0))));
    println!("IMU summing node has started!");
    tokio::spawn(mult_floats(ros.clone()));
    println!("Float multiplying service has started!");

    // Wait for ctrl_c
    tokio::signal::ctrl_c().await.unwrap();
}

Overall Comparison​

In short, my recommendation is to use ros2rust for libraries and roslibrust for applications. That is, if you're looking to provide software support for a sensor or other device, it would work better to build it with ros2rust so it fully integrates in the ROS ecosystem. However, if you're looking to build something to run on your robot, roslibrust is a great choice. Neither library would be a bad choice, of course - it comes down to your personal preferences and what you're trying to achieve with your project.

I would still advise you to try out both libraries (and the other client libraries as well) to find their pros and cons. Here are my preferences between the two libraries:

The reasons given above are very high level. Below, I give more detail to explain my reasoning.

• ROS Integration: ros2rust. All else being equal, I would prefer full ROS integration, such as launching with launch files and building the node alongside other nodes in the same workspace.
• Feature Support: ros2rust. Without giving the full list, ros2rust is more full-featured than roslibrust, such as logging integration. Whether this matters for your use case depends on there features you need.
• Ease of Setup: roslibrust. Not needing a full ROS 2 installation makes the initial setup easier.
• Build Speed: roslibrust. As ros2rust still needs the full library to be compiled alongside the application, the build time is much longer. roslibrust has a much smaller set of code to compile, so it compiles much faster.
• Backend Support: roslibrust. This is a minor point for me as I'm generally going to use the ROS 2 backend, but if you need that flexibility, this may be more of a factor for you. Having said that, it would be a bigger plus once the direct Zenoh support is implemented in roslibrust.
• The Code: roslibrust. I slightly prefer the async syntax of roslibrust because it uses fewer callbacks, although still not none. Fewer callbacks makes it easier to reason about, which is really important when designing a system.
• Unit Testing: roslibrust, by far. The complete mocking system is excellent. It is possible to unit test on ros2rust, but it makes writing the code trickier to do, as you need to pass in your own mock structures. You could also do end-to-end testing with ROS in the loop, but this is much slower than unit tests with mocks.
• Performance: I didn't measure. Measuring performance is tricky to do well, and could change between machines based on factors like how many cores it has. If you know of a good performance comparison, let me know, and I'll happily include it!

Overall, I was surprised at how much I enjoyed using roslibrust. I'm likely to use it for my own ROS 2 code when it's code I'm not planning to share. However, for code that I distribute on Github or this blog for educational purposes, I would still use ros2rust because of that complete ROS integration and extra feature set. The advantages of roslibrust seem numerous above, but are individually minor advantages compared to the ecosystem integration.

I'll always love having a fully mocked backend for unit tests, though.

Summary​

Rust is a language operating at the same level as C++, although it's not as commonly used in the space, so support is not as mature. Still, it offers great performance, strong safety checks, and modern tooling, so it's a great choice for writing robotics code. I also personally find it very fun to work with!

There are a few libraries available for writing ROS 2 nodes in Rust. ros2rust and roslibrust seem to be the most actively maintained, and they take fundamentally different approaches.

• ros2rust integrates with the ROS ecosystem through rclrs, giving full compatibility with launch files, ROS-based logging, and the colcon build system. It's ideal when you need native ROS behaviour, a more feature-complete client library, or plan to distribute your nodes as part of a larger ROS project.
• roslibrust is a pure Rust implementation designed around async execution. It builds faster, doesn't require a full ROS installation, supports ROS1 and ROS2 backends, and enabled extremely quick mock-based unit testing. This makes it great for applications running on your own robots.

For my personal projects, I'll likely reach for roslibrust, but stick with ros2rust for sharing code meant for others to learn from. Both are good choices for writing robotics code. Choose the library that works best for your application, and let me know what you think of them!

By the end of this post, you’ll have a fully automated system to spin up high-end GPU simulation machines for a fraction of the usual cost. What's more, you'll have ROS 2 access, remote desktop viewing, and one-line start/stop scripts to manage the instances.

This article also includes a video walkthrough.

▶Watch: Cost-Effective Robotics Simulation in the Cloud

Running Simulations in the Cloud​

Robotics simulators are highly useful for development, but very resource-intensive to run. One way to get access to computers with huge hard drives and powerful GPUs is to access them from a cloud provider like AWS. Of course, this does cost money; you need to pay for what you use. That means it's a great option if you infrequently run simulations, or if you want to run simulations in the background for testing, for example. If you need to run simulations frequently, you may prefer to buy your own computer and graphics card capable of running simulations.

The system is summarised in this infographic:

I have previously written about running simulations in the cloud. However, that post was about setting up O3DE and uses on-demand instances. O3DE is now much easier to set up thanks to a community AMI from Robotec AI, which means that all the setup necessary for O3DE and remote desktop viewing via DCV is already available on the EC2 instance starting up - it's like resuming a system snapshot with everything already installed. Also, on-demand instances can be off-putting in terms of machine and storage costs. Instead, I'll show how to use Spot Instances to reduce EC2 costs and Amazon S3 to reduce storage costs.

What are Spot Instances?​

Spot Instances are an alternative way to run virtual machines in the cloud using Amazon EC2. Instead of on-demand instances, where you are guaranteed an instance for as long as you need it, Spot Instances can be taken away with only a few minutes' notice. This is because AWS needs to reclaim the instance space for another customer. In exchange, the Spot Instances cost less, often 60-70% less than the equivalent on-demand instance! That means instead of $1 per hour for a powerful GPU instance, you can pay as low as$0.30 per hour.

However, that's not the only downside of Spot Instances. Due to the way that Spot Instances are requested and terminated, we need to set up a system to quickly restore a Spot Instance if removed, including simulator installation and project data. If reducing costs is worth this risk to you, then read on - but even if not, this post and the accompanying CDK code show how to install Ignition Gazebo and Isaac Sim reliably, so you can use it for your own setup. O3DE is already installed by the AMI, so no extra installation is necessary.

In this post, we use an Amazon S3 bucket to store project data. We launch Spot Instances with startup scripts that automatically copy all the project data in the bucket onto the Spot Instance, ready to use. O3DE is installed as part of the base AMI, Gazebo can be optionally installed with startup scripts, and NVIDIA Isaac Sim can be launched from a secondary drive snapshot that we build as a separate one-time setup. When we're ready to save our work and shut down, one command will synchronise all your data with the bucket, ready for your next run.

Simulation Spot Instance Setup​

Let's take a look at the system we'll be setting up. Here are the system components as a diagram, divided into stacks:

The components break down as follows:

1. Amazon S3 Bucket: used to store project data. The contents are copied automatically into the simulator instance on startup.
2. Isaac Sim EBS Snapshot: as Isaac Sim is a large program with a lot of data to download, we set up an EBS snapshot with the simulator on. This means we can create a virtual hard drive with Isaac Sim installed at a moment's notice and attach it to our simulator instance. EBS is Elastic Block Storage, which is AWS's way of providing virtual hard drives.
3. Spot Fleet Request: this requests a Spot Instance for running simulations on. You can scale your Spot Instance fleet up or down by changing its target capacity: 1 to launch an instance, 0 to stop it.
4. AWS Secrets Manager: you can choose whether to set your own password or securely generate one. If you generate one, it is stored in AWS Secrets Manager.
5. O3DE AMI: an Amazon Machine Image (AMI) is the software that the instance will start up with. The O3DE AMI is a free community AMI from Robotec AI that comes with O3DE, ROS 2 Humble, graphics drivers, and the DCV server already installed and configured (DCV allows for remote desktop streaming). This makes O3DE installation incredibly easy: just launch an EC2 instance with this AMI, and you'll have full remote control of an instance with a graphics card running O3DE!

Because there are a few different parts to this setup, I've written CDK code that automates almost all of it. This also makes it easier to specify startup scripts that synchronise project data, install extra dependencies, and so on. CDK stands for Cloud Development Kit, which is an AWS tool that lets you define Infrastructure as Code (IaC), meaning that it's easier to version, reuse, and automate deployment of systems like the one from this post.

I'll explain how to run the setup code and how to use the simulation instance, then dive into the code to explain how the system is built.

Installing Dependencies​

You'll need the following dependencies to run the CDK stack:

• git
• Node Package Manager (npm)
• AWS CDK
• AWS CLI (Command Line Interface)

Once the stack is up, you will also need the NICE DCV Client to view the remote desktop. The files are available on the Amazon DCV site, so follow the installation instructions there.

Linux​

Use apt or your equivalent package manager to install the first two dependencies:

sudo apt install -y git npm

The AWS CLI has its own installation instructions, available from the AWS CLI user guide. I've copied the latest instructions as of time of writing for your convenience:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Finally, the AWS CDK can be installed using npm:

npm install -g aws-cdk

Windows​

On Windows, you can download installers for three of the four tools from these links:

• git: https://git-scm.com/downloads/win
• npm: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
• AWS CLI: https://awscli.amazonaws.com/AWSCLIV2.msi

Once installed, the final dependency can be installed using npm:

npm install -g aws-cdk

Downloading and Configuring the CDK code​

Clone the code:

git clone https://github.com/mikelikesrobots/spot_instance_sims
cd spot_instance_sims

Open the code folder in your preferred editor, then open the bin/spot-instance-sims.ts file. Lines 17 and 27 use an insecure password: either change this to your own password, or remove the lines completely to use a password generated by AWS Secrets Manager.

Line 23 has a TODO where you can paste the snapshot ID from setting up the Isaac Sim EBS Snapshot, if you choose to do so.

Finally, line 26 has a flag to installGazebo. If you want to use Gazebo, leave this as it is, but it does need to be installed on every startup, which takes time and may incur bandwidth costs (AWS charges extra for uploads and downloads).

Once you've set up all the options as you want them, we need to set up AWS credentials to be able to deploy the resources.

AWS Credentials Setup​

There are a few methods to provide CDK with AWS credentials. I'm going to use IAM user long-term credentials as it's one of the quickest to set up, although it's not recommended by AWS. You can pick your preferred method from Authentication and access credentials for the AWS CLI.

Sign in to the AWS Console, then navigate to the IAM service. Access the Users page and create a user if you don't have one already.

To create the user, give it your preferred name, then on the "Set permissions" page, select "Attach policies directly" and select "AdministratorAccess". This will grant all privileges to AWS. I will leave constructing a least permissive policy as an exercise to the user!

Once the user is created, select it and navigate to the "Security credentials" tab. Scroll down to Access keys and click Create access key.

The access key should be for "Command Line Interface (CLI)". Read the confirmation statement at the bottom and check it if you agree, then click Next. Set a description tag if you want, then go to retrieve access keys. Keep this page open until the AWS CLI has been fully configured!

Open a command prompt or terminal, then execute:

aws configure

This will prompt you for details of your access key. Copy the access key ID and secret from the AWS Console. The region and output format are up to you: I use eu-west-2 and json respectively.

Once the configuration is complete, you can close the AWS Console if you want, although it will come in useful in later steps.

Deploy the Dependency Stack​

Open a command prompt or terminal and navigate to the CDK code folder:

cd path/to/spot_instance_sims

The code is divided into a few stacks. The first to deploy is the dependency stack, which only contains the Amazon S3 bucket for storing project data. This is separate so that it can be referenced by other stacks and won't be deleted when other stacks are deleted.

First, we need to initialise the environment, which means creating a few resources that CDK needs to help with deployment:

cdk bootstrap

Then we can deploy the dependency stack using the following command:

cdk deploy spot-sim-deps

Accept the changes, then wait for deployment to complete.

[Optional] Set up the Isaac Sim EBS Snapshot​

This step is completely optional and can be skipped if you don't plan to use Isaac Sim. It's also very time-consuming: doing it takes more than an hour, all told.

If you don't want to use NVIDIA Isaac Sim, make sure to delete the section in bin/spot-instance-sims.ts for isaacSimParams. If the isaacSimParams configuration section doesn't exist, the CDK will not take any actions towards deploying Isaac Sim.

If you do want to use Isaac Sim, you will need to deploy a stack and then go through a few manual steps. First, we will deploy an EC2 Spot Instance with a fast EBS volume for storing Isaac Sim on:

cdk deploy isaac-builder

This instance will have instructions to download and extract Isaac Sim automatically. You can either wait for around thirty minutes for the instance to do its work, or you can monitor it by signing in and monitoring the log. If you wish to do so, the method of accessing the Spot Instance are the same as in Deploying and Using the Simulation Stack; get the instance's IP address, then log in using the DCV Viewer. Once you have desktop access, use the following command in a termianl:

sudo tail -f /var/log/cloud-init-output.log

This will show what the startup script is doing. Once it gets to running the simulator headlessly, it will wait for 10 minutes, then quit the simulator. That's the point where you can access the instance and check that Isaac Sim is working correctly.

Once complete, open a terminal and execute isaac. This should load the simulator up. Wait until it's fully loaded, which takes around five minutes, then click the Play button to run the simulation. It will show an empty world. If you reach this point, congratulations! You have an EBS volume with NVIDIA Isaac Sim installed on it.

With Isaac Sim installed, the next step is to take a snapshot that can be used for future Spot Instances. The work of our current Spot Instance is done, so we can delete it to save on costs - see the stop sim script.

Next, navigate to the Volumes page in the EC2 Console. Once the volume is no longer in use, check the checkbox and select Create snapshot from the Actions menu.

Add a description and tags, if you like, then click Create snapshot. This operation will take a while, roughly 15-20 minutes, so settle in! Once complete, copy the Snapshot ID and replace the placeholder in the sim stack on line 24 of bin/spot-instance-sims.ts.

Once the snapshot is complete, you can delete the Volume used to create it. It won't be needed any more, and it does cost money to keep it around.

That's all the setup you need for Isaac Sim - you can follow the rest of the instructions below.

Deploying and Using the Simulation Stack​

To get your simulation stack running, there's just one more stack to deploy:

cdk deploy spot-sim

Accept the changes to your AWS account, and it will create all of the resources.

At this point, your simulation instance is very likely to have started up already. If you are installing Gazebo, it may take a couple of minutes before it's ready to be used, but either way you can start the remote desktop viewer.

To access the instance, you need to find the IP address. Unfortunately, CDK can't directly give this to us because it creates the fleet request, not the Spot Instance itself, and it can't wait until the Spot Instance is up to find the IP address. One option is to use the AWS Console:

Another option is to use one of the provided scripts:

Get the instance IP address​

# Windows
.\scripts\get_sim_ip.bat
# Linux
./scripts/get_sim_ip.sh

The scripts work by looking up a Spot Instance with a particular tag, created automatically by the CDK stack, and getting its public IP. An example output is:

$ .\scripts\get_sim_ip.bat
-------------------------------------------       
|            DescribeInstances            |       
+----------------------+------------------+       
|  i-08e3bb78d5f8d29ff |  18.171.186.197  |       
+----------------------+------------------+

If you chose to generate a password, now is also a good time to look that password up. You can either use the AWS CLI to obtain it (replace the placeholder ARN with the ARN from the CDK stack output):

aws secretsmanager get-secret-value --secret-id arn:your-arn-here

Or, you can open up the AWS Secrets Manager console and access it there.

Open the DCV Viewer application and paste the IP address in. Click Trust and Connect, then use the username o3de and your chosen or generated password to log in. You'll then need to log in to the Ubuntu desktop itself with the same username and password.

At this point, you should have the full Ubuntu desktop available! You can test by launching each simulator that you've installed in turn, and seeing if that simulator will load and run. Open a terminal and try each of these commands:

# For O3DE
o3de
# For Gazebo
gz sim shapes.sdf
# For NVIDIA Isaac Sim
isaac

Remember, only O3DE is installed by default! For Gazebo, you need to have the flag set to true in the CDK stack, and for NVIDIA Isaac Sim, you need to follow the optional setup steps and update the CDK stack information with the EBS snapshot ID.

If you do try to run Gazebo or Isaac Sim and get an error, try waiting a few more minutes for the setup script to finish working, then use command source ~/.bashrc. If that still isn't enough, you can try the installation steps yourself to see what went wrong, or comment on this article to see if I can help!

Saving Project Information​

The CDK stack will automatically set up a command for syncing project data back to the S3 bucket. The command can't be used automatically by the system, because it isn't guaranteed to have a full shutdown cycle, so instead you will need to remember to save project data back!

The command to use is:

s3sync

This will copy any project files from your spot instance to the S3 bucket created by the CDK stack. The next time a Spot Instance starts up, it will automatically have all data from the bucket copied into it, meaning that your data is preserved between runs. However, only project data created in the /data folder will be copied!

If we open up O3DE and create a new project, you can give that project a folder path of /data/your-project-name. If you do so, running s3sync will copy it to the S3 bucket. You can build the project, run and do any testing, then sync to S3 if you want to keep the changes. On next run, that project will be available on disk, but won't be in the project manager; you'll need to open an existing project to get access to it.

After you've finished working with the project, remember to use s3sync:

o3de@ip-172-31-17-163:~$ s3sync
Completed 381 Bytes/~47.9 KiB (7.3 KiB/s) with ~27 file(s) remaining (calculatinupload: ../../data/MyNewProject/.gitattributes to s3://spot-sim-deps-simbucket31bbb824-m0194qsbkyyz/MyNewProject/.gitattributes
Completed 381 Bytes/~47.9 KiB (7.3 KiB/s) with ~26 file(s) remaining (calculatinCompleted 2.2 KiB/~134.3 KiB (25.0 KiB/s) with ~46 file(s) remaining (calculatinupload: ../../data/MyNewProject/Examples/slam_navigation/launch/navigation.launch.py to s3://spot-sim-deps-simbucket31bbb824-m0194qsbkyyz/MyNewProject/Examples/slam_navigation/launch/navigation.launch.py
... (Removed for brevity) ...
upload: ../../data/MyNewProject/project.json to s3://spot-sim-deps-simbucket31bbb824-m0194qsbkyyz/MyNewProject/project.json
upload: ../../data/MyNewProject/project.json.bak0 to s3://spot-sim-deps-simbucket31bbb824-m0194qsbkyyz/MyNewProject/project.json.bak0

Starting and Stopping the Instance​

The fleet request has a target capacity of 1, which means that it always tries to have 1 running Spot Instance. If you shut that instance down, the fleet request will start another one up. To stop any new instances being created, we have two options:

1. Destroy the CDK stack
2. Set the target capacity to 0

Either option will have the desired effect, but destroying the stack takes a lot longer and would require you to redeploy the stack later for another instance. Instead, we can modify the fleet request to have a target capacity of 0. If we do so, it will stop any running instances in a couple of minutes, then wait to be updated. That means we can leave the fleet request where it is for next time we want a Spot Instance!

To make this easier, I've set up tags on the fleet request using the CDK stack, and written a couple of scripts to update the target capacity. These scripts are as follows:

Stopping the Instance​

This will stop a running instance in a couple of minutes. If the target capacity is already 0, it has no effect.

# Windows
.\scripts\stop_sim.bat
# Linux
./scripts/stop_sim.sh

Starting the Instance​

This will start a new instance. If the target capacity is already 1, it has no effect.

# Windows
.\scripts\start_sim.bat
# Linux
./scripts/start_sim.sh

This gives you the option to start and stop instances at your convenience without having to tear down and set up CDK resources.

Cleaning Up Resources​

If you want to totally destroy all the resources you've used, there are a couple of resources to destroy manually. Access the S3 bucket and empty it, then delete the bucket. Also, if you created an EBS snapshot in the Isaac Sim setup, this will need to be deleted manually, as it was created manually.

All remaining resources will be destroyed by destroying the CDK stacks:

cdk destroy --all

This will make sure there are no outstanding costs on your AWS account. Still, it is up to you to check that there are no outstanding expenses! If you want help tracking costs and usage on AWS, take a look at my video on the subject below:

▶Watch: Cost-Effective Robotics Simulation in the Cloud — supporting video

Popping the CDK stacks​

At this point, you should have a running simulator in the cloud! You can use this for running O3DE, Isaac Sim, or Gazebo simulations as you require. If you're interested to learn how the resources are created, this section goes into more depth of how it works. I won't explain every line of code, but I'll highlight the important parts. The repository is available on Github.

Overall CDK Application​

The main file for CDK to read is in bin/spot-instance-sims.ts. This file is simple; it defines some environment variables that it gets from the calling process, and it declares three stacks: spot-sim-deps, isaac-builder, and spot-sim.

Aside from the variables to change how the stacks behave that I first described in Downloading and Configuring the CDK code, there is also the dependencyStack variable passed into the spot-sim stack. That's because the dependency stack creates the Amazon S3 bucket separately to preserve it in case the sim stack is destroyed. By passing it into the sim stack, that stack is able to link to the bucket, so it can copy data in and out of it.

The Dependency Stack​

This code is contained in lib/sim-dependency-stack.ts. It simply creates an S3 bucket that blocks public access, versions data copied into it, and retains its contents if bucket removal is requested.

Spot Fleet Request​

This is not a stack, but a component used by the Isaac Sim builder stack and the Sim Spot Instance stack. It is contained in lib/sim-spot-fleet-request.ts, and it has quite a few steps. In fact, most of the complexity of the stack is in this file.

To begin with, it looks up the default VPC, which is required for launching EC2 instances.

const vpc = ec2.Vpc.fromLookup(props.callingStack, "DefaultVpc", {
  isDefault: true,
});

It creates two IAM roles. The first is required by the fleet request, which allows it to request Spot Instances. It also gains the AmazonEC2SpotFleetTaggingRole, which allows us to tag the fleet request and any Spot Instances with tags to make script automation easier.

const spotFleetRole = new iam.Role(props.callingStack, "SpotFleetRole", {
  assumedBy: new iam.ServicePrincipal("spotfleet.amazonaws.com"),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName(
      "service-role/AmazonEC2SpotFleetTaggingRole"
    ),
  ],
});

The second IAM role is reserved for the Spot Instances. The fleet request object must have the role defined now so it can pass it to any Spot Instances it starts up. The policies it has attached are defined by the stacks that are calling it: for the Isaac Sim builder stack, no extra policies are required; for the sim spot stack, it must have read and write access to the S3 bucket.

const instanceRole = new iam.Role(props.callingStack, "SimInstanceRole", {
  assumedBy: new iam.ServicePrincipal("ec2.amazonaws.com"),
});
const instanceProfile = new iam.CfnInstanceProfile(
  props.callingStack,
  "SimInstanceProfile",
  {
    roles: [instanceRole.roleName],
  }
);
props?.additionalPolicies?.forEach((policy) =>
  instanceRole.addToPolicy(policy)
);

The next section is about setting a password on the instance. If you defined a password, this will output the password to remind you. If you removed the password argument, it will generate a new password with the Secrets manager, and add permission to the Spot Instance IAM role to read that password. It also defines a command to get the password as an AWS CLI call:

getPasswordCommand = `O3DE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id "${passwordSecret.secretName}" --region ${region} --query 'SecretString' --output text | jq -r .password)`;

The stack creates a security group for the Spot Instances to use. This allows SSH and DCV access.

const sg = new ec2.SecurityGroup(props.callingStack, "SimInstanceSG", {
  vpc,
  description: "Allow SSH and DCV",
  allowAllOutbound: true,
});
sg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(22), "Allow SSH");
sg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8443), "Allow DCV");

Then, the stack defines user data commands. These are commands that the Spot Instance runs automatically on startup. At this point, it retrieves the password that the instance should use, then changes the password on the system and restarts the DCV server. It also adds any extra user data commands, which are defined by the stack using the component. We will see these commands in Isaac Sim Builder Stack and Sim Spot Instance Stack.

const userData = ec2.UserData.forLinux();
userData.addCommands(
  // Change o3de user's password and restart DCV
  getPasswordCommand,
  `echo "o3de:$O3DE_PASSWORD" | chpasswd`,
  "systemctl restart dcvserver"
);
if (props?.additionalUserDataCommands) {
  userData.addCommands(...props.additionalUserDataCommands);
}

The final step of this stack is to make the actual fleet request, which is what everything else has been building up to. It mostly passes in data that was already defined, but there are a few extra parts to highlight.

First, the following image ID is used. This O3DE for 3D Games and Simulations on Ubuntu image already has the DCV server, O3DE, and graphics drivers installed.

imageId: "ami-070cda0e6952fbf24",

Next, the blockDeviceMappings are passed in to the component. If this array is empty, it doesn't add any disks to the Spot Instances. If it has one or more entries, these will be automatically attached to Spot Instances that are launched. This is how we can attach a drive with the Isaac Sim installation: define the EBS drive parameters, including the snapshot ID, and it will be created and attached automatically.

blockDeviceMappings: props?.blockDeviceMappings || [],

Finally, the instance type is specified as g4dn.2xlarge. You could change this to a larger image type if you like. When I tried to change it to a smaller one, it refused to launch, but feel free to try for yourself!

Also, there can only be one instance type specified here. To specify a second instance type, you will need to create another entry in launchSpecifications.

instanceType: 'g4dn.2xlarge',

That's mostly it! There are also a couple of tag definitions, and one last output with the username, which is always o3de from the AMI. Next, we'll see how this component is used by the Isaac Builder stack and the sim spot stack.

Isaac Sim Builder Stack​

This is the simpler of the two stacks that use the SimSpotFleetRequest component from the previous section, defined in lib/isaac-sim-builder-stack.ts. It specifies an additional set of commands to run on startup which mount a drive, make a filesystem on it, then install Isaac Sim there. Finally, it does one headless launch to cache assets, terminating automatically after 10 minutes to allow the user to check the installation.

As well as the script, it creates the EBS volume that Isaac Sim is to be installed on. Crucially, the volume is not deleted on termination, which means you can create snapshots from it even after the Spot Instance has been deleted. However, it also means that destroying the CDK stack won't delete it - it will stay around forever, costing money, unless you delete it manually.

const blockDeviceMappings: DeviceMapping[] = [
  {
    deviceName: "/dev/sdf",
    ebs: {
      volumeType: "gp3",
      deleteOnTermination: false,
      iops: 16000,
      volumeSize: 60,
    },
  },
];

Sim Spot Instance Stack​

This is where the main sim instance is created. It starts by getting the URL and ARN for the S3 bucket created by the dependency stack and adding an inline policy to allow read and write access to it.

const bucketURL = props?.dependencyStack.bucket?.s3UrlForObject();
const bucketArn = props?.dependencyStack.bucket?.bucketArn;
if (bucketArn) {
  additionalPolicies.push(new iam.PolicyStatement({
    ...

Most of the rest of the file define the user data commands to run on startup. First, no matter the simulators used, it installs the AWS CLI and synchronises data from the S3 bucket into /data on the system. It also creates the sync command alias that allows you to run the s3sync command.

Next, it checks whether Isaac Sim is being used, and if it is, adds extra commands and an EBS volume to support it. The commands attach the device and mount it, then create the isaac command to run it. The EBS volume is defined by the snapshot ID passed in to the stack, and is set to delete on termination, so that when the Spot Instance is destroyed, that EBS volume is as well.

blockDeviceMappings.push({
  deviceName: deviceName,
  ebs: {
    snapshotId: props?.isaacSimParams.snapshotId,
    volumeType: "gp3",
    deleteOnTermination: true,
  },
});