{"id":340,"date":"2021-02-13T20:34:53","date_gmt":"2021-02-13T20:34:53","guid":{"rendered":"https:\/\/erdalpekel.de\/?p=340"},"modified":"2021-02-13T21:04:21","modified_gmt":"2021-02-13T21:04:21","slug":"building-a-web-interface-for-visualizing-panda","status":"publish","type":"post","link":"https:\/\/erdalpekel.de\/?p=340","title":{"rendered":"Building a Web Interface for Visualizing Panda"},"content":{"rendered":"\n<p>In this blog post we are going to construct a minimalist web interface that shows information about the robot&#8217;s current state in numbers and shows it in an embedded 3d visualizer:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-style-default\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"705\" src=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-1024x705.png\" alt=\"\" class=\"wp-image-341\" srcset=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-1024x705.png 1024w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-300x206.png 300w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-768x528.png 768w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-1536x1057.png 1536w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37-1570x1080.png 1570w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-12-55-37.png 1805w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Web Interface for visualizing panda robot&#8217;s state<\/figcaption><\/figure>\n\n\n\n<p>Our intention in this post is to set-up the fundamental components that are needed to communicate with the ROS master from a modern web app. It is not intended to be used as finished work but it should instead provide the basic structure needed for building a modern web interface for the panda robot as it can be improved in numerous ways.<\/p>\n\n\n\n<p>We will need the following building blocks for this web interface:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The existing panda_simulation package<\/li><li>A new ReactJS package for the frontend<\/li><li>A docker container that server panda&#8217;s visual model files to front end<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Preparation &amp; Setup<\/h2>\n\n\n\n<p>We will assume that <a href=\"https:\/\/github.com\/erdalpekel\/panda_simulation\">panda_simulation<\/a> is already installed in the workspace <em>catkin_ws<\/em> and continue with the installation of the other components. We need <a href=\"https:\/\/nodejs.org\/en\/\">NodeJS<\/a>, <a href=\"https:\/\/reactjs.org\/\">ReactJS<\/a>, and <a href=\"https:\/\/yarnpkg.com\/\">Yarn<\/a> for this tutorial. Since there is an unresolved package conflict when using apt to install the node package manager <em>npm<\/em> we will install these components from the <a href=\"https:\/\/github.com\/nodesource\/distributions\/blob\/master\/README.md#deb\">NodeSource<\/a> distribution release:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">cd ~\/catkin_ws # this is the root workspace dir. Change if necessary\ncurl -fsSL https:\/\/deb.nodesource.com\/setup_12.x | sudo -E bash -\nsudo apt-get install -y nodejs\ncurl -sL https:\/\/dl.yarnpkg.com\/debian\/pubkey.gpg | sudo apt-key add -\necho \"deb https:\/\/dl.yarnpkg.com\/debian\/ stable main\" | sudo tee \/etc\/apt\/sources.list.d\/yarn.list\nsudo apt-get update &amp;&amp; apt-get install yarn\ncd \/catkin_ws\/src\/panda_simulation\/\nyarn global add create-react-app\ncreate-react-app panda_web\ncd panda_web\/\nyarn add @material-ui\/core ros3d roslib lodash\nyarn start<\/code><\/pre>\n\n\n\n<p>With the above commands we have created the sub-folder <em>panda_web<\/em> in our source directory <em>panda_simulation<\/em> with the script <em>create-react-app<\/em>. It quickly bootstraps a ready-to-use ReactJS application in the specified folder. After installing the dependencies that are needed for this project we can start the web interface with <em>yarn start<\/em> and it will open a new tab in the browser:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"897\" src=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-1024x897.png\" alt=\"\" class=\"wp-image-350\" srcset=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-1024x897.png 1024w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-300x263.png 300w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-768x673.png 768w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-1536x1345.png 1536w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05-1233x1080.png 1233w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-13-19-05.png 1806w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>create-react-app landing page<\/figcaption><\/figure>\n\n\n\n<p>Now we need to switch to our initial panda_simulation package in order to add the components that will allow our frontend to access information from the ros core. The following changes need to be applied:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"xml\" class=\"language-xml\">&lt;!-- package.xml --&gt;\n&lt;depend&gt;rosbridge_library&lt;\/depend&gt;\n&lt;depend&gt;rosbridge_server&lt;\/depend&gt;\n&lt;depend&gt;rosbridge_msgs&lt;\/depend&gt;\n&lt;depend&gt;rosapi&lt;\/depend&gt;\n&lt;depend&gt;tf2_web_republisher&lt;\/depend&gt;<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"vim\" class=\"language-vim\"># CMakeLists.txt\nfind_package(catkin REQUIRED\n              COMPONENTS \u2026\n                         rosbridge_library\n                         rosbridge_server\n                         rosbridge_msgs\n                         rosapi\n                         tf2_web_republisher)<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"xml\" class=\"language-xml\">&lt;!-- launch\/simulation.launch --&gt;\n&lt;!-- for web interface --&gt;\n&lt;include file=\"$(find rosbridge_server)\/launch\/rosbridge_websocket.launch\"&gt;\n    &lt;arg name=\"port\" value=\"9090\" \/&gt;\n&lt;\/include&gt;\n&lt;node pkg=\"tf2_web_republisher\" type=\"tf2_web_republisher\" name=\"tf2_web_republisher\" \/&gt;<\/code><\/pre>\n\n\n\n<p>Now run the following commands to install these dependencies and then run the simulation:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">cd ~\/catkin_ws\nrosdep install --from-paths src --ignore-src -y --skip-keys libfranka --skip-keys moveit_perception\nsource devel\/setup.bash\nroslaunch panda_simulation simulation.launch # gui:=false optionally without GUI if you're running in a docker container<\/code><\/pre>\n\n\n\n<p>I would like to quickly mention that I have put the necessary scripts for the visualization into the <em>panda_simulation\/panda_web\/public\/scripts<\/em> directory and these scripts are loaded automatically within the <em>public\/index.html<\/em> file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"markup\" class=\"language-markup\">&lt;head&gt;\n...\n&lt;script src=\"%PUBLIC_URL%\/scripts\/three.js\"&gt;&lt;\/script&gt;\n&lt;script src=\"%PUBLIC_URL%\/scripts\/ColladaLoader.js\"&gt;&lt;\/script&gt;\n&lt;script src=\"%PUBLIC_URL%\/scripts\/STLLoader.js\"&gt;&lt;\/script&gt;\n&lt;script src=\"%PUBLIC_URL%\/scripts\/eventemitter2.min.js\"&gt;&lt;\/script&gt;\n&lt;script src=\"%PUBLIC_URL%\/scripts\/custom\/ColladaLoader.js\"&gt;&lt;\/script&gt;\n&lt;\/head&gt;<\/code><\/pre>\n\n\n\n<p>There is one more thing that needs to be done before we can visualize the robot. The model files that represent the robot&#8217;s links are located in the <em>franka_ros<\/em> package and need to be served to the front-end somehow. There are many ways to do this but we are going to run a docker container for this purpose. This has the advantage for the developers that they can just build and start and for the package maintainer this is an advantage as the model files are separated from the <em>panda_web<\/em> project directory. I have added a <em>Dockerfile<\/em> to the project under <em>panda_simulation\/docker\/franka_ros_files<\/em> which downloads the <em>franka_ros<\/em> repository into the serving directory of an <em>nginx<\/em> container:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"vim\" class=\"language-vim\">FROM nginx\n\n# Modify default conf for CORS access and auto file listing\nCOPY default.conf \/etc\/nginx\/conf.d\/default.conf\n \n#install necessary packages\nRUN apt-get update &amp;&amp; apt-get install wget unzip -q -y\n\n# download and unzip repository to html serving dir\nWORKDIR \/tmp\nRUN wget -O franka_ros.zip https:\/\/github.com\/frankaemika\/franka_ros\/archive\/0.6.0.zip\n\nWORKDIR \/usr\/share\/nginx\/html\nRUN unzip \/tmp\/franka_ros.zip\nRUN mv franka_ros-0.6.0\/* . <\/code><\/pre>\n\n\n\n<p>We are going to build and run this image:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">cd ~\/catkins_ws\ncd src\/panda_simulation\/docker\/franka_ros_files\ndocker build -t franka_ros_files .\ndocker run -p 80:80 franka_ros_files<\/code><\/pre>\n\n\n\n<p>You can check if the files are served by accessing <em>localhost<\/em> with the browser:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"251\" src=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32-1024x251.png\" alt=\"\" class=\"wp-image-364\" srcset=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32-1024x251.png 1024w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32-300x74.png 300w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32-768x188.png 768w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32-1536x377.png 1536w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-32.png 1799w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"312\" src=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56-1024x312.png\" alt=\"\" class=\"wp-image-365\" srcset=\"https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56-1024x312.png 1024w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56-300x91.png 300w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56-768x234.png 768w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56-1536x468.png 1536w, https:\/\/erdalpekel.de\/wp-content\/uploads\/2021\/02\/Screenshot-from-2021-02-13-15-27-56.png 1797w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Structure of the ReactJS App<\/h2>\n\n\n\n<p>Now that the basic setup is finished we can modify add the necessary functionality to the bootstrapped ReactJS app. Our app will consist of three different components:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Views<\/li><li>Components<\/li><li>Services<\/li><li>Configuration files<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Views<\/h4>\n\n\n\n<p>This will be a single-view application as it will only be a proof-of-concept. Our view is called <em>DashboardView.js<\/em> and it is contained in the root of the application &#8211; the file <em>App.js<\/em>. DashboardView renders three components:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"jsx\" class=\"language-jsx\">import React from 'react';\nimport { JointStates } from '..\/components\/JointStates';\nimport { TransformClient } from '..\/components\/TransformClient';\nimport { ModelVisualizer } from '..\/components\/ModelVisualizer';\nimport { panda_simulation } from '..\/utils\/constants';\n\nexport default class DashboardView extends React.Component {\n     constructor(props) {\n         super(props);\n <code>        this.state = {}; <\/code>\n    <code>} <\/code>\n\n<code>    render() {<\/code>\n        return (\n        &lt;div&gt;\n            &lt;JointStates topic={panda_simulation.constants.JOINT_STATES_TOPIC} \/&gt;\n            &lt;TransformClient\n                     targetFrame={panda_simulation.constants.WORLD_LINK}\n                     sourceFrame={panda_simulation.constants.PANDA_EE_PARENT_LINK}\n                     tfRate={10}\n                 \/&gt;\n            &lt;ModelVisualizer\n                     urdfPath={\n                         'http:\/\/' + process.env.REACT_APP_FILE_SERVER_URL + ':' + process.env.REACT_APP_FILE_SERVER_PORT\n                     }\n                     targetFrame={panda_simulation.constants.ROBOT_BASE_LINK}\n                     tfRate={panda_simulation.config.tfRate}\n                     width={panda_simulation.config.width}\n                     height={panda_simulation.config.height}\n                     cameraPosition={panda_simulation.config.cameraPosition}\n                 \/&gt;\n        &lt;\/div&gt;\n        );\n    }\n}<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Components<\/h4>\n\n\n\n<p>We have three custom components that each have a different purpose:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>JointStates: Retrieves the joint states of the robot from the given topic and displays the results in a table<\/li><li>TransformClient: Retrieves the transform information between to given frames and displays them as text paragraph<\/li><li>ModelVisualizer: Visualizes the current state of the robot<\/li><\/ul>\n\n\n\n<p>All components have in common that they continuously retrieve data about the robot&#8217;s current state from the ROS master that we started earlier. This is achieved with the service component that will be introduced in the next section.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Services<\/h4>\n\n\n\n<p>Currently the only service that is implemented is the <em>RosService.js<\/em>. It communicates with the ROS master at the address that is provided in the <em>.env<\/em> config file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"javascript\" class=\"language-javascript\">var ROSLIB = require('roslib');\nvar ROS3D = require('ros3d');\n\nvar ros = new ROSLIB.Ros({\n     url: 'ws:\/\/' + process.env.REACT_APP_ROS_BRIDGE_URL + ':' + process.env.REACT_APP_ROS_BRIDGE_PORT\n });\n\nros.on('connection', function () {\n     console.log('Connected to websocket server.');\n });\n\nros.on('error', function (error) {\n     console.log('Error connecting to websocket server: ', error);\n });\n\nros.on('close', function () {\n     console.log('Connection to websocket server closed.');\n });\n\n\/\/ ... implement different ROS client functionality\n\nmodule.exports = {\n     JointStatesListener,\n     tfClientToFrame,\n     viewer3d,\n     markerClient,\n     markerArrayClient,\n     urdfClient\n };<\/code><\/pre>\n\n\n\n<p>The crucial line here is line 4 where a new connection is established with the ROS master. The address is extracted from the <em>.env<\/em> file that is located in the <em>panda_web\/<\/em> directory:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"vim\" class=\"language-vim\">REACT_APP_ROS_BRIDGE_URL=localhost\nREACT_APP_ROS_BRIDGE_PORT=9090\nREACT_APP_FILE_SERVER_URL=localhost\nREACT_APP_FILE_SERVER_PORT=80<\/code><\/pre>\n\n\n\n<p>Port 80 was specified above when we issued the <em>docker run<\/em> command with the <em>-p<\/em> flag. Port 9090 was specified in the <em>simulation.launch<\/em> file within the <em>panda_simulation\/<\/em> directory.<\/p>\n\n\n\n<p>The custom <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/erdalpekel\/panda_simulation\" target=\"_blank\">ROS package<\/a> as well as the modified <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/erdalpekel\/franka_ros\" target=\"_blank\">franka_ros<\/a> repository that is needed for the simulation are available in my <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/erdalpekel\" target=\"_blank\">GitHub profile<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog post we are going to construct a minimalist web interface that shows information about the robot&#8217;s current state in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":391,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,3,8,7,22],"tags":[],"class_list":["post-340","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-robot-operating-system-ros","category-robotics","category-ros","category-simulation","category-web-development"],"_links":{"self":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/340","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=340"}],"version-history":[{"count":45,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/340\/revisions"}],"predecessor-version":[{"id":389,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/posts\/340\/revisions\/389"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=\/wp\/v2\/media\/391"}],"wp:attachment":[{"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=340"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=340"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/erdalpekel.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=340"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}