Pick and Place with Semantic Instance Segmentation in Omniverse Isaac

In this blog post, we will pick and place simple cubes onto a conveyor belt with the Panda robotic arm in NVIDIA’s Omniverse Isaac simulation environment. We will detect the cubes with an RGB camera and Isaac Sim’s built-in semantic instance segmentation algorithm. In our latest contributions (1, 2), we added the Panda robotic arm to Isaac Sim and controlled it with MoveIt! via ros_bridge. We also attached a custom modeled suction gripper part to the arm. In the resulting simulation, the robotic arm is able to pick a cube that the camera has detected and place it on the conveyor belt for carrying.

Simulation Scene

We will start this post by introducing our new simulation scene. It consists of a conveyor belt, the Panda robotic arm, a shelf surface, a cube and an RGB camera. The scene runs the conveyor belt and simulates an RGB camera that captures the shelf surface for cubes that can be picked by the robotic arm.

We have labeled the important asset of the simulation scene in the image above:

RGB Camera: The RGB camera captures the shelf surface and segments cubes that are semantically labeled.
Conveyor Belt: We built the conveyor with the Conveyor Belt Utility in Isaac Sim. It carries the cube to the shelf surface.
Cube with Semantic Labeling: A cube with semantic labeling falls onto the conveyor belt and is carried to the shelf surface.
Panda Robotic Arm: The robotic arm has a surface gripper attached to its last link. This enables pick and place task execution for cubes that landed on the shelf surface. It will place the cube onto the conveyor belt when it receives the command.
Shelf Surface for Pick-up: The shelf surface serves as an intermediate storage location for the cubes where the camera can detect and segment them from the top.

User Interface

We have extended the user interface of our Isaac Sim extensions to enable the aforementioned workflow of picking a cube form the shelf surface with the robotic arm and placing it onto the conveyor belt.

We added new buttons for added control to our user interface. The robotic arm can now be commanded to the cube and the conveyor belt. It can also be commanded to grasp and release the cube.

The ‘Move to Cube’ button commands the robotic arm to the cube at a grasp pose that is determined at run-time. The position of the cube is determined by the camera via semantic instance segmentation in the background. The scenario class runs the instance segmentation algorithm continuously in the background in order to determine a cube that the Panda robotic arm could approach.

The ‘Move to Conveyor’ button commands the robotic arm to a predetermined position over the conveyor belt. We defined this position to be to the left of the Panda arm. It is specified at ‘/World/cube_dropoff_position’ in our scene.

The ‘Grasp’ and ‘Release’ buttons are implemented with the Surface Gripper API in Omniverse Isaac Sim and hence are not related to the ROS interface.

Semantic Instance Segmentation

The Semantic instance segmentation segmentation is displayed in the gallery below. On the left, you can see the RGB camera’s default view. On the right, we displayed its instance segmentation filter. You can see that the cube is segmented in cyan color.

The camera can segment the semantic class of the cube if it is added to its model definition in the scene as displayed in the figure below. The semantic type is class and the semantic data is

ROS Interface

We will expand the ROS interface package by adding a new node for motion planning via MoveIt! C++ API to our existing package. We will establish the communication between the Isaac Sim extension and the mentioned ROS node with an action message type defined inside a new ROS package called isaac_moveit_msgs. With the new action type, we will send a desired goal pose for the Panda robotic arm to the new ROS node:

geometry_msgs/PoseStamped goal
---
bool success
---

Results

On the ROS side, the simulation can be run by building the ROS workspace and running the launch file:

mkdir -p ~/isaac_moveit_ws/src
cd ~/isaac_moveit_ws/src
git clone https://github.com/erdalpekel/franka_description.git
git clone https://github.com/erdalpekel/isaac_moveit_msgs.git
git clone https://github.com/erdalpekel/isaac_moveit_sample.git
cd ..
colcon build
source install/setup.bash
ros2 launch isaac_moveit_sample simulation.launch.py

The Isaac Sim extension can be run by adding its path to the extensions and opening the extension from the menu bar after enabling autoload. The path ~/isaac_moveit_ws/install/setup.bash needs to be sourced in the ~/.bashrc file.

The custom ROS package as well as the Isaac Sim extension repositories are available in my GitHub profile.