{"id":1293,"date":"2024-09-03T16:51:12","date_gmt":"2024-09-03T14:51:12","guid":{"rendered":"https:\/\/erdalpekel.de\/?p=1293"},"modified":"2024-09-03T17:18:39","modified_gmt":"2024-09-03T15:18:39","slug":"pick-and-place-with-semantic-instance-segmentation-in-omniverse-isaac","status":"publish","type":"post","link":"https:\/\/erdalpekel.de\/?p=1293","title":{"rendered":"Pick and Place with Semantic Instance Segmentation in Omniverse Isaac"},"content":{"rendered":"\n

In this blog post, we will pick and place simple cubes onto a conveyor belt with the Panda robotic arm in NVIDIA’s Omniverse Isaac simulation environment. We will detect the cubes with an RGB camera and Isaac Sim’s built-in semantic instance segmentation algorithm. In our latest contributions (1<\/a>, 2<\/a>), we added the Panda robotic arm to Isaac Sim and controlled it with MoveIt! via ros_bridge<\/code>. We also attached a custom modeled suction gripper part to the arm. In the resulting simulation, the robotic arm is able to pick a cube that the camera has detected and place it on the conveyor belt for carrying.<\/p>\n\n\n\n

Simulation Scene<\/h2>\n\n\n\n

We will start this post by introducing our new simulation scene. It consists of a conveyor belt, the Panda robotic arm, a shelf surface, a cube and an RGB camera. The scene runs the conveyor belt and simulates an RGB camera that captures the shelf surface for cubes that can be picked by the robotic arm. <\/p>\n\n\n\n

\"Simulation\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Simulation Scene with Conveyor Belt and Robotic Arm<\/figcaption><\/figure>\n\n\n\n

We have labeled the important asset of the simulation scene in the image above:<\/p>\n\n\n\n

    \n
  1. RGB Camera: The RGB camera captures the shelf surface and segments cubes that are semantically labeled.<\/li>\n\n\n\n
  2. Conveyor Belt: We built the conveyor with the Conveyor Belt Utility<\/a> in Isaac Sim. It carries the cube to the shelf surface.<\/li>\n\n\n\n
  3. Cube with Semantic Labeling: A cube with semantic labeling falls onto the conveyor belt and is carried to the shelf surface.<\/li>\n\n\n\n
  4. Panda Robotic Arm: The robotic arm has a surface gripper attached to its last link. This enables pick and place task execution for cubes that landed on the shelf surface. It will place the cube onto the conveyor belt when it receives the command.<\/li>\n\n\n\n
  5. Shelf Surface for Pick-up: The shelf surface serves as an intermediate storage location for the cubes where the camera can detect and segment them from the top.<\/li>\n<\/ol>\n\n\n\n

    User Interface<\/h2>\n\n\n\n

    We have extended the user interface of our Isaac Sim extensions to enable the aforementioned workflow of picking a cube form the shelf surface with the robotic arm and placing it onto the conveyor belt.<\/p>\n\n\n\n

    \"Isaac\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
    Isaac Sim Extension User Interface<\/figcaption><\/figure>\n\n\n\n

    We added new buttons for added control to our user interface. The robotic arm can now be commanded to the cube and the conveyor belt. It can also be commanded to grasp and release the cube.<\/p>\n\n\n\n

    The ‘Move to Cube’<\/strong> button commands the robotic arm to the cube at a grasp pose that is determined at run-time. The position of the cube is determined by the camera via semantic instance segmentation in the background. The scenario class runs the instance segmentation algorithm continuously in the background in order to determine a cube that the Panda robotic arm could approach.<\/p>\n\n\n\n

    The ‘Move to Conveyor’<\/strong> button commands the robotic arm to a predetermined position over the conveyor belt. We defined this position to be to the left of the Panda arm. It is specified at ‘\/World\/cube_dropoff_position’<\/strong> in our scene.<\/p>\n\n\n\n

    The ‘Grasp’<\/strong> and ‘Release’<\/strong> buttons are implemented with the Surface Gripper API in Omniverse Isaac Sim and hence are not related to the ROS interface.<\/p>\n\n\n\n

    Semantic Instance Segmentation<\/h2>\n\n\n\n

    The Semantic instance segmentation segmentation is displayed in the gallery below. On the left, you can see the RGB camera’s default view. On the right, we displayed its instance segmentation filter. You can see that the cube is segmented in cyan color.<\/p>\n\n\n\n

    \n
    \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
    Camera View<\/figcaption><\/figure>\n\n\n\n
    \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
    Instance Segmetation<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

    The camera can segment the semantic class of the cube if it is added to its model definition in the scene as displayed in the figure below. The semantic type is class and the semantic data is <\/p>\n\n\n\n

    \"Cube\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
    Cube Semantic Class Type<\/figcaption><\/figure>\n\n\n\n

    ROS Interface<\/h2>\n\n\n\n

    We will expand the ROS interface package<\/a> by adding a new node for motion planning via MoveIt! C++ API to our existing package. We will establish the communication between the Isaac Sim extension and the mentioned ROS node with an action message type defined inside a new ROS package called isaac_moveit_msgs<\/a><\/code>. With the new action type, we will send a desired goal pose for the Panda robotic arm to the new ROS node:<\/p>\n\n\n\n

    geometry_msgs\/PoseStamped goal\n---\nbool success\n---<\/code><\/pre>\n\n\n\n
    Results<\/h2>\n\n\n\n
    On the ROS side, the simulation can be run by building the ROS workspace and running the launch file:<\/p>\n\n\n\n
    mkdir -p ~\/isaac_moveit_ws\/src\ncd ~\/isaac_moveit_ws\/src\ngit clone https:\/\/github.com\/erdalpekel\/franka_description.git\ngit clone https:\/\/github.com\/erdalpekel\/isaac_moveit_msgs.git\ngit clone https:\/\/github.com\/erdalpekel\/isaac_moveit_sample.git\ncd ..\ncolcon build\nsource install\/setup.bash\nros2 launch isaac_moveit_sample simulation.launch.py<\/code><\/pre>\n\n\n\n
    The Isaac Sim extension can be run by adding its path to the extensions and opening the extension from the menu bar after enabling autoload. The path ~\/isaac_moveit_ws\/install\/setup.bash<\/code> needs to be sourced in the ~\/.bashrc<\/code> file. <\/p>\n\n\n\n
    \"Pick\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
    The custom ROS package<\/a> as well as the Isaac Sim extension<\/a> repositories are available in my GitHub profile<\/a>.<\/p>\n\n\n\n
    <\/p>\n","protected":false},"excerpt":{"rendered":"
    In this blog post, we will pick and place simple cubes onto a conveyor belt with the Panda robotic arm in NVIDIA’s […]<\/p>\n","protected":false},"author":1,"featured_media":1301,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10,34,9,33,37,5,3,8,7,4],"tags":[],"class_list":["post-1293","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-c","category-isaac","category-moveit","category-omniverse","category-pick-and-place","category-robot-operating-system-ros","category-robotics","category-ros","category-simulation","category-software-development"],"_links":{"self":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/1293","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1293"}],"version-history":[{"count":27,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/1293\/revisions"}],"predecessor-version":[{"id":1332,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/1293\/revisions\/1332"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/media\/1301"}],"wp:attachment":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1293"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1293"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}