48-555/ 48-755 Introduction to Architectural Robotics Carnegie Mellon University
Silent Clerk Seeing, scanning, and drawing using an industrial robot with Kinect Humphrey Yang, Hang Wang
1
48-555/48-755 ITAR
Input Skeleton Tracking Robotic Arm Kinect V2 Scene Scanning
Computer with Grasshopper
Output Drawing Marker
Abstract
Fig.01 Diagram of how Movements are recorded and normalized.
In this project we explored and developed a framework of using Kinect and industrail robots for human-machine interactions. With the multifunctional sensor mounted on it, the robot becomes more human-like and possibly mobile, since it does not require fixed-position cameras. In environments where an accurate digital model of the space is difficult to acquire or the immutability of the camera position/orientation can not be guaranteed, our proposed system will out-perform the externally placed sensor configuration. We also speculated that by using the robotic arms, the representation of data, especially 3D geometries, do not require the translation into 2D and loses its depth information.
Introduction Myriads of projects exploring the artistic, practical, and ethical aspect of digital fabrication and human-computer interaction has been administered. The campaign to design novel methods of fabrication using these robot rewarded us with arti2
facts that was previously difficult or even impossible to produce, like free form stone cutting[1][2]. On the other hand, the discussion on human-industrail robot interaction was arduous in a sense that these robots are intrinsically danger-
ous and difficult to operate with. In recent years, designers like Madeline Gannon pioneered this field and reintroduced the robots as companions instead of tools[3] [4]. She analogued them to domesticated animals in her projects, and demonstrated as interactive displays in exhibitions[3]. In these projects, the sensors are detached from the robots. Distributed in space, the sensors granted the robot clairvoyance but also renders it dependent on these still, external eyes and become immobile. The robot companion is chainned. To address this shackle, we propose having hte sensor mounted on the robot itself instead of the environment. The robot sees and locates everything in re-
lation to its position. We can mount the robot on a mobile platform and interact with people without any problem. The camera as an analogy of eyes also imbued the robot with “body languages� that people can observe and make use of. With the newly acquired eyes, the robot can see the object it works with, so the objects need not to be pre-defined. It can be any arbitrary shape as long as the robot’s program sees it fit. The maneuverability of the robots empowered the robot to see 3D objects, and render data onto them, bypassing the inevitable 3D to 2D translation of geometries, as mentioned in Seeing is Doing[5].
3
48-555/48-755 ITAR
Tool Design
Program Workflow The Robot is expected to follow and carry out these commands in sequence:
TCP01
Kinect V2
Marker
TCP02
Fig.02 Design of tool used to capture, scan, and draw.
shape
coding
robot
shape
robot
a.
coding
1. Scan the canvas before engages the main event loop, to identify drawable onjects in the pre-defined draw area. 2. If no player in sight, rotate around to find one. If a player is seen, track his/her hand movement and record it. 3. After capturing, move above the canvas to get ready for drawing actions. 4. Scan the drawable objects again but closer for higher resolution. If the object displaced from its original position, the robot will locate its new position. 5. From the scanned point cloud, construct a mesh and project the movements onto it. Projection method and choice of canvas object depends on the geometry of the movement. 6. Draw the movements as a line on the objects. 7. Housekeeping procedure that cleans all temporary data and moves the robot back to player-seeking position. 8. Go back to procedure one.
b.
drawing
Fig.04 Workflow Diagram. The program will continue to run until manually stopped.
Fig.03 Workflow explained (a) common workflow (b) our interactive workflow
4
5
48-555/48-755 ITAR
Kinect for Tracking The grasshopper definition captures the operator’s hand gesture and maps it to te canvas. Because the sensor is mounted on the robotic arm, and motions captured by the Kinect is positioned in relative space in respect to the Kinect, we have to use forward kinematics to figure out the correct position of the operator in the
environment. Scanned movements are normalized by the operator’s arm length, and using that as reference, we can rescale and project the curve onto canvas ibjects. The optimal distance for tracking is 1500 mm away form the Kinect, and should be no farther than 4000 mm.
Hand movements as drawing path, positions are projected on a plane formed by the torso and shoulder.
Fig.05 An Diagram of the Grasshopper definition. The End-of-Procedure controller, as shown on the right, will pass control to the begining of the code(left) and refire itself.
We implemented our project using Rhino Grasshopper, with HAL to communicate to the robot and Tarsier for Kinect sensor data streaming. The 3D environment of Rhino provides robust representation of and versatile manipulability over 3D geometries, while Grasshopper provides us with parametric computation pipeline. We used GH Python to implement the controller as shown in Fig.05. Note that only one instance of controller was created in the workflow, and it was passed around during the process. The Controller fires
6
at a preset interval and determines which procedure to call. Procedures are implemented with Grasshopper components, and accomplish the task defined . Upon completion of the procedure, the controller at the end of the procedure will record and evaluate the computed data and determine the next procedure to call, and refires itself. As the controller is passed between two ends of the definition repeatedly, we refer to this framework as “Ping-Pong”.
Length from shoulder to wrist is mapped to the size of the canvas
Fig.06 Diagram of how Movements are recorded and normalized.
7
48-555/48-755 ITAR Accurately Captured Materials:
Kinect for Scanning a.
To determine how to reconstruct the canvas mesh, and which material should be used for the canvas in the physical world, we tested several off the shelf softwares to see how well they do in capturing the scene with accuracy on different materials (Fig.07 and Fig.08). The built-in 3D scanning software comes with the Kinect suffers from excessive mesh smoothing. While Rhino Grasshopper with Firefly constructs the mesh, the interpolation of mesh adds additional mesh surfaces to the scene and left us poor maneuverability over the construction process. We ends up using the point cloud instead, and implemented the construction ourselves. While it is not a good method in mesh reconstruction, the algorithm served us well for this proof-of-concept project.
b.
Materials Diffcult to Capture:
c.
d.
Material-wise, by testing several materials with differing properties, we concluded that the material of choice should be non-reflective, plain, and opaque. This limit imposed by material is due to the design of Kinect. An implementation using other hardware for depth-sensing devices like LIDAR may open up opportunity for using reflective and/or transparent materials.
e. Fig.07 Meterials captured with accuracy: (a)Waxed wood, solid, barely reflective. (b)Plain paper, flexible, non-reflective. Materials failed to capture: (c)Bottle with metallic paint, reflective. (d)Glass, transparent and reflective. (e)Coated and painted cardboard, slightly reflective, high contrast. (f)Fabric, frizzy.
8
f. Fig.08 (a)Testing Scene as captured by camera for reference. (b)Mesh constructed with 3D Scan by Microsoft. (c)Mesh constructed withRhino Grasshopper with Firefly.
9
48-555/48-755 ITAR
Mesh Reconstruction
a
a
The scanning and mesh reconstruction of the canvas plays a crucial role in robotic drawing, and such performance is dominated by the intrinsic precision of the sensors. In this case of using Kinect V2, we experimented on the accuracy of several configurations and concluded that the distance between the scanned surface and the sensor should maintain within 500-1500 centimeters. Shorter distance result in larger jittering of point clouds. To construct a digital representation of the environment, we grouped the points by surfaces and constructed a mesh out of each group. The accuracy, as measured and determined by the largest deviation in distance between the projected curve/points to the actual surface, was controlled within 3 mm and within the error mitigation capacity of the tool we designed (Fig.10).
b Fig.09 (a) A captured image from the Kinect with a toolpath overlayed on top. In this image, the sensor was 50cm away from the surface.(b) Scanned scene as point cloud with toolpath projected on the surface, along with grasshopper definition.
10
b Fig.10 Curve projection on the scanned (a) point cloud and (b) mesh
As a proof of concept, we used color coding to carry out this, which will be introduced in the next page. However, a more robust way to do so is to ultilize computer vision and edge detection to group points.
11
48-555/48-755 ITAR
Curve Projection
Preliminary experiment
Conventional robotic drawings are drawn onto a 2D canvas but the depth- information was lost in the projection. Light painting belongs to this family to, as we need cameras to capture the time-lapsed dim and shine of LEDs, which ends up as a 2D image as well. However, since we are using 3D scanning to reconstruct the canvas to project and draw on, we are not confined to using a plannar paper for drawing. The surface can be free-formed and curves are project unidirectionally onto it. A more complex type of projection is 3D projection, similar to cubic or cylindrical texture mapping. Some motions are better described and represented using this method, like spiral movement which will degrade into a zigzaggin polyline and lost its depth information if projected in 2D, as shown in Fig.11.
With components of our concept experimented, we composited them together using the framework explained in Fig.05. As programs may contain sementic errors and trial running is potentially dangerous, we tested our program by live-coomunicating with Robot Studio (Fig.13) first. After making sure the program runs smoothly and accomplishes the job correctly, we tested our workflow on the robot manually by installing the tool we designed and walking through the process with the teach pendent. We moved the robot between tracking and scanning positions, with the Kinect tracking operator movements and scanning for canvas objects, and generate drawing modules which was later uploaded to the robot controller for execution (Fig.14).
a
b
A procedure of the Grasshopper definition will analyze the curve and categorize it into two families- plnnar or 3Dbased on the planarity of the points.
Fig.12 Projection mapping described (a) Projecting to 2D plannar surface, an oil painting canvas for example. (b) Projection on a curvy surface. (c) Projection of a spiral curve onto a cube. Notice how the projection on a surface alone fails to capture the depth change of the the spiral, and the combination of the four vertical surfaces better captures the volution.
12
Fig.13 Trial running with Robot Studio(left), Rhino(top right), and grasshopper+HAL(bottom right). This image shows the scanned canvas object(color coded in blue) proected in the 3D space. The grasshopper panel is showing the controller and monitor for the Ping-Pong framework.
c
13
48-555/48-755 ITAR 8 shape
a
star shape
Fig.15 Canvas objects with movements drawn on them. 8 shape relates to row b in Fig.14, star shape relates to row c in Fig.14
References [1] Zayas, L., McGee, W., Brugmann, D., Durham, J., Clifford, B. “Quarra Cairn: Incremental Stability Through Shifting and Removal of Mass,” in ACADIA 2017: Disciplines & Distruption, Paper Proceedings 2017: 650-659
b
[2] Feringa, J., Terhall, L., “Carrara Robotics“ ROB | ARCH 2014 [3] Gannon, M., Sandoval, J., Mephail, K., & Snell, B. “Mimus- the curious industrial robot” 2016, from https://atonaton.com/mimus [4] Gannon, M. “Quipt- taming industrial robotics” 2015, from https://atonaton.com/ quipt
c
[5] Bard, J., Gannon, M., Jacobson-Weaver, Z., Contreras, M., Jeffers, M., Smith, B. “Seeing is Doing: Synthetic tools for robotically augmented fabrication in highskill domains” ACADIA 2014.
Fig.14 Manually walk through the workflow. (a) experiment setting (b)(c)Each row is an idependent iteration of the program. The operator’s hand moved differently between two shots and the canvas objects are unique. 14
15
48-555/48-755 ITAR
16