THE ROBOT REPORT MAY 2021 by WTWH Media LLC

A Supplement to Design World - May 2021 www.therobotreport.com

Evolution of Boston Dynamics’ box-moving robots page 54

INSIDE: • WTWH Media acquires Mobile Robot Guide .......................................................53 • Designing an omnidirectional mobile manipulator with 7 DoF ...................................62 • 5 critical sensor integration challenges for AMRs ............................................................68

ROBOT_REPORT_COVER_5-21_Vs1.indd 51

4/30/21 3:25 PM

SICK 5-21.indd 52

4/29/21 2:26 PM

The Robot Report

WTWH Media

acquires Mobile Robot Guide Acquisition of industry-leading mobile robotics website strengthens WTWH Media’s robotics brand and serves as a catalyst for future growth. The Robot Report Staff

Business-to-business publisher WTWH Media, parent company of Design World and The Robot Report, acquired Mobile Robot Guide (MRG), the leading online resource for news and information about mobile robots, the applications they enable, and industries they support. Mobile Robot Guide will become incorporated into WTWH Media’s existing network of robotics websites, magazines, newsletters, events, and research. Mike Oitzman, founder of Mobile Robot Guide, will continue as editor of MRG, as well as assist in the development and execution of new robotics media initiatives and lines of business. See mobilerobotguide.com for more information. According to Dan Kara, Vice President, Robotics at WTWH Media, “We are delighted to announce the acquisition of Mobile Robot Guide, and to have someone as high caliber as Mike Oitzman join the WTWH robotics and intelligent systems team. Mike is highly regarded as a leading robotics industry expert and an authority on mobile robots, and the Mobile Robot Guide website he created is recognized as the world’s foremost resource for mobile robotics news and information. I look forward to working with Mike to build and expand WTWH Media’s robotics media offerings, and in doing so, supporting both users and suppliers of robotics and intelligent systems products and services, as well as the robotics sector as a whole.” Scott McCafferty, Co-founder & Managing Director of WTWH Media, notes, “The addition of Mike Oitzman to WTWH Media’s Robotics Group, coupled with the acquisition of Mobile Robot Guide, will further strengthen WTWH Media’s robotics portfolio, allowing us to better serve the entire robotics value chain of end-users, engineers and OEMs, researchers, investors and more. Mike launched Mobile Robot Guide, and as such he is perfectly positioned to integrate this high value asset into THE ROBOT REPORT

Mobile.Robot.Guide.5-21_RR_Vs3.LL.indd 53

www.therobotreport.com

WTWH Media’s existing robotics and intelligent systems business lines.” Mike Oitzman remarked on the acquisition, “I am excited to see Mobile Robot Guide evolve and grow with the resources and reach of WTWH Media. Dan Kara and Steve Crowe are well respected within the robotics community and I look forward to joining the editorial team for the Robotics Group, and I am also anxious to help WTWH deliver world class events again in the coming years.” WTWH Media’s robotics network now includes The Robot Report, Collaborative Robotics Trends, Mobile Robot Guide and Robotics Business Review, online technical, business and investment news and information portals focused on robotics and intelligent systems. WTWH Media also produces leading robotics conferences including the Robotics Summit & Expo, RoboBusiness Conference & Exposition and the Healthcare Robotics Engineering Forum. RR May 2021

4/30/21 2:54 PM

The Robot Report

Evolution The company’s newest robot, Stretch, is a mobile manipulator designed to move boxes out of trucks and build pallets inside warehouses. Steve Crowe Editorial Director The Robot Report

May 2021

Boston-Dynamics-5-21_RR_Vs5.SC.indd 54

of Boston Dynamics’ box-moving robots Using robots to unload eight on receiving docks isn’t a novel idea. A quick YouTube search will return a slew of results. But many of these previous approaches have been stifled by high costs and the inability to handle a growing variety of packages. Of course, robotics companies are still working on the problem, which is one of the major issues for warehouse operators. Dextrous Robotics, a Memphis-based robotics startup, is developing its Chopstick system it hopes can unload eight with more success and at a action of the cost of previous solutions. Co-founder Evan Drumwright discussed the approach in the RoboBusiness Direct session “Advances in Robotic Picking, Grasping and Manipulation.” www.therobotreport.com

THE ROBOT REPORT

4/30/21 2:56 PM

Truck unloading is one application Stretch is targeting. Stretch can lift up to 50 lb, pick 800 cases per hour, and reach boxes up to 10 feet high. | Boston Dynamics

Next-gen Handle Stretch is the next generation of Handle, a robot Boston Dynamics introduced in 2017 that combined wheels

and legs. Stretch doesn’t have legs, but it does have an omnidirection mobile base with four independently controlled wheels, a custom 7-DoF industrial robot arm that can li up to 50 pounds, a custom suction gripper and much more. The air system for the gripper is onboard the mobile base. Stretch comes with an 8-hour battery life, but there will be a 16-hour battery option and the ability to plug Stretch in for continuous power. Stretch uses the Pick vision system, which Boston Dynamics acquired when it bought Kinema Systems in April 2019. Pick uses high-resolution 2D and 3D vision and machine learning algorithms for robotic depalletizing. Kevin Blankespoor, Boston Dynamics’ VP of product engineering, said Stretch can pick up

THE ROBOT REPORT

www.therobotreport.com

Now Boston Dynamics is throwing its innovative hat into the ring. It revealed its newest robot, Stretch, a mobile manipulator designed to move boxes out of trucks and around warehouses. When it goes on sale in 2022, for a yet to be named price, Stretch will initially focus on truck unloading and later add palletizing to its repertoire. Stretch is currently being tested by a few partners. Boston Dynamics is seeking customers to pilot Stretch with truck unloading tasks.

Boston-Dynamics-5-21_RR_Vs5.SC.indd 55

May 2021

5/3/21 7:37 AM

The Robot Report

Building pallets will be a future application for Stretch. Boston Dynamics envisions more autonomy in that task, saying Stretch will be navigating the aisles of the warehouse by itself. | Boston Dynamics

May 2021

Boston-Dynamics-5-21_RR_Vs5.SC.indd 56

to 800 cases per hour. Its arm has a sevenfoot reach and can reach boxes up to 10 feet high. Stretch weighs 2,650 lbs. Stretch is semi-autonomous, Blankespoor said, and the level of autonomy depends on the application. When it comes to unloading a truck, for example, a person still needs to open the truck door, verify the content and move the Stretch robot into position by driving it inside the truck with a joystick. “At that point you hit ‘Go’ and Stretch will fully autonomously do the rest of the job unloading the boxes,” Blankespoor said. Building pallets will be the second application for Stretch, and Blankespoor envisions more autonomy in that task. “Stretch will be navigating the aisles of the warehouse. It’ll be going to different pallets that are singleSKU pallets and grabbing a few boxes building up an outgoing pallet. For that task, Stretch will take on more of the kind of localization and navigation you see in some other AMRs [autonomous mobile robots] that navigate through the warehouse and have more autonomy.” Blankespoor said collaborative robot arms lack the speed and strength to lift the number of heavy cases per hour that Boston Dynamics targeted. So it built a custom industrial robot arm. For safety, Stretch uses speed and separation monitoring. “As people get closer to the robot, it will slow down,” he said. “If they get closer still, it will stop. And that way we can allow the arm to move heavy boxes fast without risk of hurting anyone.”

www.therobotreport.com

Handling variety the key As you might notice in videos of Stretch, all the boxes inside the truck and on the pallet look identical and are perfectly stacked. Most, if not all, trucks that need to be unloaded in the real world won’t look like that. So Stretch’s success, like other attempted solutions before it, could be determined by its ability to handle a variety of packages. “That’s an astute observation,” said Blankespoor. “It’s pretty easy to detect a box sitting by itself. But detecting a box when they’re packed tightly together or when there’s a big variety, is a much harder task. But that’s where our Pick box detection software comes in. And that’s one of the things you’ll see next with Stretch. We’re starting off with more straightforward trucks, but we’re going to get into more complex types of boxes, different types of boxes over the coming year.” FedEx’s Aaron Prather discussed during his recent RoboBusiness Direct keynote that the biggest challenge in solving the truck offload issue is the high level of package diversity a solution would need to be able to handle. Prather started his FedEx career offloading trucks while still in college. “When unloading a bulk loaded truck, you may start with boxes of a certain size, then you hit a layer of bigger or smaller boxes, then a layer of rugs (because people buy rugs online now), and then back to the same size boxes you started with. This is why it is so hard to automate this process.” Stretch robot just the beginning? Boston Dynamics in 2020 acquired a boatload of warehouse automation patents from X Development, LLC, an R&D organization founded by Google. Boston Dynamics was owned by Google from 2013-2017. Certainly some of the IP helped with Stretch, but it likely points to a broader interest in the logistics automation space. Industrial Perception, a company acquired by Google in 2013, partnered with Wynright on truck unloading research. “Well, we’ve definitely had interest in the warehouse space since Atlas. That’s when we figured out what a big market

THE ROBOT REPORT

4/30/21 2:58 PM

Stretch lifts Boston Dynamics popular Spot quadruped. | Boston Dynamics

Stretch features an omni-directional mobile base, four independently controlled wheels, a custom 7-DoF industrial robot arm, a custom suction gripper and a proprietary vision system. | Boston Dynamics

THE ROBOT REPORT

Boston-Dynamics-5-21_RR_Vs5.SC.indd 57

www.therobotreport.com

May 2021

4/30/21 2:59 PM

The Robot Report this is, and how much potential there is for mobile robots in the warehouse space. I don’t want to say too much about the patents, but we have a warehouse robotics team that is growing rapidly, which I’m heading up. And that includes development of the Stretch robot, we have our Pick vision system, we have a fleet management software system that can pull together different robots into doing coordinated activities. So it really just fits into the kind of broader picture that Boston Dynamics is going hard into the warehouse space.”

Electric linear axes ELGC and mini slides EGSC

Learning from Spot Boston Dynamics introduced in February 2021 Spot Arm, a robotic arm for the Spot quadruped that offers a max lift capacity of 24.3 lbs. While Spot Arm looks vastly different from Stretch’s industrial robot arm, Blankespoor said the core technological building blocks are the same. “If you look at the wrist joint of Stretch, it’s actually the same as the hip joint on Spot. So it uses the same electric motors and the same gearboxes, same sensing even the same software,” he said. “We use the same cameras and depth sensors across our different platforms. More importantly, we use the same software underneath to actually understand what those sensors are seeing – everything from box detection to obstacle detection. So even though it looks really different from Atlas or Spot, Stretch is built from most of the same technologies. And that enabled us to get a prototype built really fast.” Atlas remains an R&D platform, while Boston Dynamics has sold more than 400 Spot robots since the quadruped first went on sale last year. Boston Dynamics won a 2020 RBR50 Innovation Award for Spot as it jump-started the commercial quadruped market.

One Integrated Platform for All Your Automation Needs One platform for electric automation means seamless connectivity. From electromechanical actuators when joined with servo motors and servo drives to complete positioning systems, motion control solutions as well as entire handling systems and decentralized control solutions - always with the right software and interface. www.festo.us/ea

Fleet management, WMS integration Boston Dynamics and OTTO Motors released a video in March 2020 that demoed their vision of the future of logistics automation. In the video, Boston Dynamics’ Handle robot picked boxes and built pallets on top of the OTTO 1500 heavy-duty AMR. Blankespoor said the two companies continue to collaborate, and it’s played an important role in the development of a fleet manager for Stretch.

Boston-Dynamics-5-21_RR_Vs5.SC.indd 58

May 2021

THE ROBOT REPORT

4/30/21 3:00 PM

NEW IDEA. NEW GEAR.

BETTER MOTION

Designing a new automation application? You need a new geared solution! The Motus Labs M-DRIVE series of precision geared solutions offer a higher torque density than competing strain wave gearing with no compromise in performance. The patented design utilizes a series of CAM-driven blocks that engage over 80% of the output ring surface area at all times. This design distributes load stresses over a much larger surface area, delivering a much greater torque per unit size than other technologies.

Better Price to Performance HIGH PRECISION | LESS WEIGHT | COMPACT SIZE

MOTUS-LABS.COM Motus Labs 1-21.indd 59

4/29/21 2:27 PM

The Robot Report “Part of the important piece of that demo with OTTO Motors was to get heterogeneous robots working together. There’s a lot of AMR companies that control their own robots, and they might have a fleet manager to do that. But there’s not a whole lot of fleet managers that can control a variety of different types of robots, coordinated to do different tasks. One of the reasons we’re working with OTTO is to coordinate the pallet movement and the case movement with different robots.” Blankespoor also said integration with warehouse management systems is on Boston Dynamics’ roadmap. But he said it’s more crucial later on when Boston Dynamics starts to tackle order building. “Truck unloading is nice because it’s a little more isolated in terms of the integration tasks. Once the robot is in front of the truck and you say ‘Go,’ our box detection system will tell Stretch where all the boxes are, what size they are, we actually even weigh the boxes as we’re first lifting them up, so we can move them as quickly as possible. This means we don’t necessarily have to do as much integration work.” “The second task that we’re going to bite off is order building. And for that you do need warehouse management system integration, but it should be akin to what you do with people, right? Here’s an order, go build me a pallet with five of this box and four of this box, and stack it all up. We’ll need to convey that information to our robots so they can go do those tasks.” Evolution of a box-moving robot You might think a logistics robot is out of left field for Boston Dynamics, but it’s shown interest in the space for a while. In 2016, one of its first videos of the next-gen Atlas robot showed the humanoid bend down, pick up a box, and place it onto a cart. Then in 2017, Boston Dynamics introduced Handle, a hybrid robot that combined wheels and legs. Stretch is the next generation of Handle. “We got a lot of interest [for Atlas] from warehouses,” Blankespoor said. “We knew we could design a simpler robot, and that’s where Handle really came from. Handle was an offshoot of Atlas.

May 2021

Boston-Dynamics-5-21_RR_Vs5.SC.indd 60

“We branched off and designed Handle for a couple of reasons. One was that we wanted to do something more purpose-built for the warehouse. And another reason was that we always wanted to combine wheels and legs. Handle was an opportunity to explore both of those things. So we built a couple versions of Handle and started doing warehouse tasks.” “First we were doing pallet building, and that was working pretty well. Then we started doing truck unloading with Handle. And at that point, it was an eye-opening experience – Handle could do the job. It could grasp the boxes and move them, but it took too long. And to make a cost-efficient robot for customers, you need to move cases pretty fast. Stretch can move cases about five times faster than Handle, so that’s one of the reasons we made the next jump.” Hyundai an early customer? Hyundai Motor acquired a majority stake in Boston Dynamics from Softbank in December 2020 for about $880 million. A Softbank affiliate retained the other 20%. Spot Arm was an important announcement since it effectively turned Spot into a mobile manipulator, not just a data collection platform. But the introduction of Stretch is the biggest announcement from Boston Dynamics since the acquisition. So will Hyundai be an early test customer for Stretch? “They absolutely could be,” said Blankespoor. “We’re excited to work with Hyundai. Obviously they’re a worldclass manufacturer. Hyundai has its own logistics companies in Korea, and they definitely could be a customer for us in the future.” RR

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:00 PM

Obstacle Detection Solution Starter Kit

Shipping Now. Order yours at

www.dreamvu.com/shop/ Use coupon code

Starter Kit Outputs:

palministarterkit

By DreamVu

This Starter Kit allows you to quickly evaluate the DreamVu obstacle detection solution. This solution detects any obstacle within a wide vertical ﬁeld and across a horizontal ﬁeld of view up to 360°. To be compatible with mapping solutions such as Cartographer and Gmapping, it simulates a 2D laser scan output. The Starter Kit includes the DreamVu Explorer GUI for quickly visualizing the RGB-D and point-cloud generated by the PAL stereo camera. It also includes ROS so maps can be generated, viewed and saved.

Ultra-wide ﬁeld of view visual obstacle detection in a warehouse detecting pallet stacks and humans from head to toe Explorer GUI for quick visualization of RGB-D and point cloud

Simulated 2D laser scan output overlayed on 360° RGB

Experience the advantages of the Obstacle Detection Solution Starter Kit

DreamVu 5-21_RR.indd 61

Real-time 360° mapping output in ROS

Unparalleled ﬁeld-of-view

Uses stereo vision to detect

Greatly simpliﬁes the

Generate and view

provides 3D obstacle

objects from ﬂoor to the

sensor stack to save

real-time maps within

detection in an output

ceiling with 0 false positives

compute, time and cost.

ROS Cartographer and

compatible with 2D

even in challenging

mapping solutions

real-world environments.

Gmapping

4/29/21 2:29 PM

The Robot Report

Designing

an omnidirectional mobile manipulator with 7 DoF ROS integration, reduced power consumption, ease of use, and flexibility are key to new mobile manipulator. Steve Crowe | Editorial Director, The Robot Report

Over the past few years, mobile manipulation has often been mentioned as a robotics trend to watch. Perhaps advances being made in a variety of enabling technologies – sensors, controls, power – are finally making mobile manipulation a reality. There have been at least three mobile manipulators introduced within the past couple of months. Boston Dynamics certainly had the most interesting take with its Stretch robot, while Desmasa launched a more traditional mobile manipulator with its MCR+. The third on that list is a new collaborative mobile manipulator from Waypoint Robotics and Productive Robotics, a Calif-based developer of cobot arms. The system integrates Productive’s OB7 cobots onto Waypoint’s Vector omnidirectional autonomous mobile robot. I visited Waypoint to get a sneak peek at the system. The prototype I saw featured a Vector 3DHD and the basic OB7 cobot, but the mobile manipulator can also integrate Productive’s other 7 degree-of-freedom cobot arms – OB7-Stretch, OB7-Max 8, or OB7-Max 12 – depending on the application. The system uses Productive’s standard teach pendant.

May 2021

Designing.a.Mobile.Manipulator_RR_5-21_Vs4SC.indd 62

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:02 PM

“If you have a machine shop that has a bunch of CNC machines with a relatively high cycle time that don’t need to be tended very often, it’s really hard to justify putting an arm at every station,” said Waypoint Robotics CEO and co-founder Jason Walker. “If you’ve got one device that can serve multiple stations, then that makes sense.”

THE ROBOT REPORT

Designing.a.Mobile.Manipulator_RR_5-21_Vs4SC.indd 63

www.therobotreport.com

May 2021

4/30/21 3:02 PM

The Robot Report Productive Robotics OB7 Cobot Lineup Cobot

Payload (kg)

Horizontal Reach (mm)

Repeatability (mm)

DOF

OB7

1000

0.1

OB7-Max8

1700

0.1

OB7-Max12

1300

0.1

OB7-Stretch

1250

0.1

A standard product offering Waypoint Robotics, a spinout of Stanley Innovation, has sold mobile manipulators for years. But they’ve been one-off projects. This mobile manipulator will become part of the product portfolio for both Productive and Waypoint. “There’s a difference between us saying we have a mobile manipulator as a standard product compared to we have an option to have a mobile manipulator,” said Waypoint Robotics CEO and cofounder Jason Walker. “The mobile manipulator [with Productive Robotics] is something we’re integrating a lot more extensively and tightly.” Walker and Productive Robotics president Zac Bogart first met at ATX West 2019. Both companies will be selling the mobile manipulator. Walker said this will benefit customers and integrators who are accustomed to working in particular environments. “If they already know the language of the OB7, they can add a coding block to the teach pendant that tells the Vector to drive to a certain location and do whatever it’s supposed to do,” said Walker. “It’s a control paradigm that lives in the OB7. People who are familiar with that ecosystem don’t have to go learn a whole new ecosystem. It can also work the other way around, too, for those who are familiar with Waypoint and the Dispatcher software.”

Workers can use this mobile manipulator for a variety of tasks, including machine tending, quality assurance sampling, material replenishment, and many others.

ROS integration key Walker talked at length about how Productive’s 7-DoF arms and Vector’s omnidirectional capabilities will enable the mobile manipulator to work in tight spaces and make unique movements.

May 2021

Designing.a.Mobile.Manipulator_RR_5-21_Vs4SC.indd 64

| Waypoint Robotics

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:02 PM

While demoing the system, Walker set up separate waypoints for Vector in its software and the OB7 by using its onboard buttons and moving it through space. However, Walker said there’s no reason why one button can’t make both waypoints, highlighting another way ROS can help take interoperability to the next level.

Productive’s 7-DoF arms and Vector’s omnidirectional capabilities will enable the mobile manipulator to work in tight spaces and make unique movements. | Waypoint Robotics

But there are plenty of other 7-DoF cobot arms. So why did Walker choose to partner with Productive? “We’ve integrated a bunch of different arms, but the OB7 is energy efficient, it’s ROS-native, it’s made in the USA. [Productive] is a scrappy startup about the same age as us,” said Walker. “And it’s a company capable of understanding the opportunity and the unique benefits of mobile manipulation.” The mobile manipulator doesn’t rely heavily on ROS at the moment, but

Walker said it offers a great path forward for even tighter integration. “We can combine setup tasks for each device into one task. That will lower the effort required for somebody to set it up or reconfigure it or move it around from one location to another to do a different job,” he said. “The system also has two computers – one for the arm and one for the mobile robot. The more we can get the two things merging closer together, we can remove one of the computers to further reduce cost.”

Reducing power consumption Walker said one of the biggest problems with mobile manipulators is power and energy consumption. He said nearly all the mobile manipulators he’s built as one-offs in the past required a spare battery and additional integrated battery kit. “Since most of the robots we were building were bespoke robots going into R&D Labs, it was tolerable to plug in the mobile robot into one cord and plug in the batteries for the arm into another cord,” he said. “But for a commercial product, that’s never acceptable.” Walker said Waypoint approached the power problem from a number of angles. One of them is by using Waypoint’s EnZone wireless charging system. The mobile manipulator is capable of supporting two EnZone receivers, meaning double the energy can be delivered. He also said that while OB7 operates at 48 volts, it can be re-tuned to operate at 24 volts – the same as Vector. Walker said his mobile manipulator can get at least six hours of run time.

Waypoint built this older, one-off mobile manipulator for the Verizon 5G Challenge in 2019. | Steve Crowe

THE ROBOT REPORT

Designing.a.Mobile.Manipulator_RR_5-21_Vs4SC.indd 65

www.therobotreport.com

May 2021

4/30/21 3:06 PM

The Robot Report “We’re committed to, over time, cost reducing this thing,” said Walker, who isn’t ready to share the price tag. “A big part of that is taking out all of the extra power conversion stuff, and doing a version of it that’s natively 24 volts. Instead of going through converters, we’ll feed battery power straight into the arm. And that’s going to significantly reduce the cost and size.” Walker claimed Productive’s cobot arms are also more energy-efficient than the competition. “When we do a UR5 integration, we usually use a 1000 watt inverter. We leave the AC-to-DC converter that’s built into the box and then add an inverter so that you can use either DC power from the robot or AC power from the wall. [In this new mobile manipulator], we’re doing it with a 400 watt inverter. The fact that it can be done with that small amount of power is huge. We’re lucky to get 4 hours out of UR5 mobile manipulator, so it’s just more efficient.”

is moving. For example, if there are a bunch of parts that need to go into a CNC machine, they can be sorted or oriented a specific way while the mobile manipulator is in transit. “This is also where the seventh axis is helpful,” Walker said. “The arm can spin around and operate safely within the footprint of the robot.” Are customers ready for mobile manipulation? I’ve seen many mobile manipulators at trade shows that combine a mobile base from Mobile Industrial Robots (MiR) and a cobot arm from Universal Robots. Of course, both companies are owned by Teradyne. I recently asked Greg Smith, president of Teradyne’s industrial automation group, if we’d see an official mobile manipulator from the group in the near future. “There’s an interesting economic challenge around arms on mobile bases. When people look at doing automation,

“If they already know the language of the OB7, they can add a coding block to the teach pendant that tells the Vector to drive to a certain location and do whatever it’s supposed to do,” said Walker.

manipulation and shared a story about one of the first companies interested in this new mobile manipulator. “If you have a machine shop that has a bunch of CNC machines with a relatively high cycle time that don’t need to be tended very often, it’s really hard to justify putting an arm at every station,” he said. “If you’ve got one device that can serve multiple stations, then that makes sense.” He added, “the things that have held customers back in the past has been how easy it is to set up, how much it costs and what it can do.” “The reason we’re doing this product now is because we’ve helped solve the cost problem by having an arm and a robot that, over time, can get less and less expensive. We can integrate the systems more, we’ve massively lowered the entry price because we don’t have to have additional batteries.” “This is an omnidirectional AMR with a 7-DoF arm, so you get something that’s actually better than the other solutions out there. Most of the other solutions are differential drive and a 6-DoF arm. So for me those are the key things – the energy and power, the degrees of freedom, the overall footprint capabilities, ROS integration, and ease of use. If this isn’t easy for everybody to use, then it’s a non-starter. Everything else falls apart.” RR

Dragging items like Spot Workers can use this mobile manipulator for a variety of tasks, including machine tending, quality assurance sampling, material replenishment, and many others. We recently saw Boston Dynamics’ Spot quadruped grab and drag a concrete block. Walker said they’re currently working on similar functionality. “An immediate use would be the ability to drag a tote off of something and onto the robot,” he said. “One of our targets are companies that don’t want to revise their environment, or want to as little as possible. If this robot can drive up to a counter and pull a tote off onto it, that’s less work than setting up a conveyor or a pickup and drop off station.” They’re also working on the ability to perform tasks while the Vector robot

May 2021

Designing.a.Mobile.Manipulator_RR_5-21_Vs4SC.indd 66

they look at it from an ROI perspective. How long is payback going to take? How much utilization does the automation get? If you take a cobot arm and stick it on top of a mobile robot, then you’re in a situation where when you’re using the arm, the base isn’t adding any value. And when you’re using the base, the arm isn’t adding any value.” “It has to be a much higher value application to justify the spend because you’re only using half the robot at any moment. There’s some challenges in terms of getting the right price point to be able to do that. But we also see some interesting niches where even with today’s technology, It makes a whole ton of sense to do.” Walker brought up these comments during my visit, saying he disagreed. He said there’s a thirst for mobile

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:06 PM

Honeywell - Digital 5-21.indd 67

4/30/21 3:54 PM

5 The Robot Report

critical sensor integration challenges for AMRs

Sensor integration challenges can lengthen product development roadmaps and create unexpected downtime during deployments. Adam Rodnitzky | COO & Co-Founder, Tangram Vision

Sensing arrays on autonomous mobile robots (AMRs) are robust and redundant by design; beyond their critical role in operational safety, they build the data foundation for differentiating features like machine learning capabilities and high-speed operation. Yet engineering teams are o en unaware of how difficult it can be to quickly integrate and reliably operate the multi-sensor arrays upon which AMRs rely. These sensing systems are subject to internal constraints (resource-limited hosts, for instance) and external constraints (hazard-filled operating environments) that present unique engineering challenges that must be addressed for safe, reliable operation. Here are five of the most critical engineering tasks to solve when integrating multiple modes of sensing on an AMR platform.

May 2021

5.critical.sensor.integration.challenges_RR_5-21_Vs4.LL.indd 68

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:09 PM

The impact of too much data can be serious for an AMR. While the most likely outcome is excessive latency, the worst outcome is data blackouts.

|Tangram Vision

THE ROBOT REPORT

5.critical.sensor.integration.challenges_RR_5-21_Vs4.LL.indd 69

www.therobotreport.com

May 2021

4/30/21 3:10 PM

The Robot Report

When attaching a sensor to an AMR, pay attention to fastener torque. Over torquing one or more fasteners can twist a sensor’s chassis.

|Tangram Vision

1 2 Time synchronization Let’s start with one of the most important: time synchronization between sensors and the host. Your host has a clock. Your sensors have clocks. But do they agree? Without intervention, they won’t — and that is a problem. When a sensor sends data with a different timestamp than the host, the host interprets that data according to its own clock, not the sensor’s. The host may then reach the conclusion the sensor is miscalibrated, and send the AMR into a failure mode. Worse yet, the host may believe it is interpreting the data correctly, and continue to operate with faulty sensor data. That can lead to an unsafe robot, as its actions will be out of sync with what it observes. To prevent this failure, all sensor clocks need to be time synchronized. This is a process that must be done on a continual basis, as minor differences in clock speeds can add up to big differences over time. Doing this constant correction requires knowledge of the sensors and host om a hardware perspective to understand how the clock is integrated into the hardware. Typically, sensor clock correction isn’t addressed in sensor SDKs or documentation. In fact, sensors om most manufacturers rarely consider or even offer solutions to the problem of understanding the clock offsets and how they might interact in a larger system. A er all, why should a LiDAR provider care about every possible IMU on the market, and try to accommodate a finite but large list of clocks and different methodologies

May 2021

5.critical.sensor.integration.challenges_RR_5-21_Vs4.LL.indd 70

for getting timestamps? This is le up to the developer to figure out, whether they know how to or not. Sensor calibration Most vision sensors used by AMRs leave the factory pre-calibrated. This factory calibration creates a so warereadable model of a set of key physical ccharacteristics haracteristics of the sensor. This iincludes ncludes where the optical elements are a re in three-dimensional physical space in relation to the sensor chassis. It also includes a lens distortion model to account for manufacturing defects that are an unavoidable part of sensor production. By using this data, a perception engineer can tune an individual sensor to achieve the greatest precision and accuracy. But these intrinsic calibrations aren’t permanent. Physical impacts to an AMR (very common in busy environments like warehouses) can throw off these precise sensor calibration settings, requiring recalibration. Sensor Tip: When attaching a sensor to an AMR, pay attention to fastener torque. Over torquing one or more fasteners can twist a sensor’s chassis. Even small torsional twists can severely throw off factory calibration. The current standard for in-field recalibration is the checkerboard process. With this process, a checkerboard (or similar) pattern is placed in different locations and orientations in ont of www.therobotreport.com

the sensor, and a so ware process is run to interpret the sensor’s new spatial coordinates and distortions based on measurements it constructs om the pattern. The most popular checkerboard process comes om OpenCV. However, more advanced processes have arrived more recently, including Kalibr om the ETH-Zurich computer vision lab, and Vicalib om the University of Colorado Boulder. Both are better suited to multisensor calibration processes, and both promise a more rigorous calibration process that produces better results in terms of accuracy and precision of the final intrinsics.

Spatial registration When an AMR is equipped with multiple sensors, each sensor must understand not just where it is in three-dimensional space, but where every other sensor is in three-dimensional space. Along with clock synchronization, spatial registration is critical to ensure that sensor fusion can be accomplished for an AMR. Sensor Tip: What is sensor fusion? This is taking data streams om two or more sensors and combining them in both space and time. This can help achieve much higher accuracy than what is possible with a single sensor, and can also unlock capabilities (3D reconstruction, for instance) that aren’t possible (or easily possible) with a single sensor.

THE ROBOT REPORT

4/30/21 3:10 PM

2 Conceptual rendering of the multi-jointed robotic arm of a surgical system.

3 1 4

Giving robots a sense of touch FUTEK's miniaturized sensor technology allows surgeons to perform as if they had virtual fingertips. The sensors’ precise measurement and feedback allow the machine to emulate the dexterity and haptics of human hands.

QTA143

Micro Reaction Torque Sensor

Dimensions: 14 mm × 10 mm × 26 mm Provides closed-loop feedback on torque measurement.

LSB205

Miniature S-Beam Jr. Load Cell

Dimensions: 19 mm × 18 mm × 6.6 mm Provides critical force feedback.

QLA401

Load Cell Built for Autoclave

Dimensions: Ø 14 mm × 3.28 mm Designed to withstand the autoclave sterilization process.

go.futek.com/medtech

QLA414

ANSI

ISO

Z540-1

17025

9001

13485

FUTEK 11-20_RR.indd 71

U.S. Manufacturer

Nano Force Sensor

Dimensions: 4mm × 5mm Enables direct measurement that eliminates any drift in the output.

4/29/21 2:31 PM

The Robot Report

Much like single sensor calibration, spatial registration can degrade when sensors or the AMRs they are attached to move due to physical impacts. Even tiny, unexpected shi s will require that all sensors undergo extrinsic recalibration. Similar to a single sensor intrinsic recalibration process, multi-sensor extrinsic calibration is currently achieved with the same hand-eye process using a checkerboard pattern. Unfortunately, this process can take up to 45 minutes or more for multi-sensor arrays. Tangram Vision is working on sensor autocalibration so ware to automate this process on a continual basis while an AMR is in operation.

Real-time data streaming A single sensor can generate hostcrushing amounts of data. Multiple sensors? This requires carefully designed data structures and data management. The impact of too much data can be serious for an AMR. While the most likely outcome is excessive latency, the worst outcome is data blackouts, where entire packets are dropped. Imagine a fastmoving robot that suddenly receives late data or no data. Depending on what is in the path of that robot, the result can be disastrous.

Connection reliability “USB is the worst.” While there is some truth to this statement, the reality is that all connection types have inherent weaknesses that can stop an AMR in its tracks. USB is singled out more o en because it is ubiquitous, and because it is capable of performing many tasks simultaneously (for instance, both data and power delivery). Sensor Tip: Many USB connection issues are the result of using a poorly designed hub or the incorrect cable. Make sure that your hub and cable choices support the data rates your platform will need for operation. Keep cable runs as short as possible to minimize the possibility of data and power interruptions. Ethernet can provide more reliable data transfer, but it requires a separate power connection, which can then introduce an additional point of failure. No matter what connection type is specified, it pays to understand the different standards that connector and cable manufacturers can use, and to find cables and connectors that meet a standard that supports your use case.

Sensor Tip: High dynamic range (HDR) cameras are becoming more widely used for robots that work in environments with very dark or very bright lighting conditions. However, HDR cameras can take up huge amounts of bandwidth and host compute because of the sheer amount of data they generate. Depending on the host, they may consume an entire core just to process their datastream. When speci ing an HDR camera, consider your host’s processing capacity and ensure it can handle the added strain.

DevOps for Perception The team at Tangram Vision previously built the popular Structure Sensor & SDK depth sensing platform. We saw firsthand all of these sensor struggles and more as we worked with robotics companies to integrate both the Structure Sensor and other perception sensors onto their platforms. These so ware and sensor challenges typically added a year or more of additional engineering time to product development roadmaps, and created equent, unexpected downtime during deployments. We launched Tangram Vision to eliminate these development and deployment challenges with a continually evolving, best-in-class sensor management platform. The Tangram Vision SDK automatically manages time synchronization, sensor calibration, spatial registration and data management for multi-modal, multisensor platforms like AMRs and other robots. It also provides early notification of sensor faults, including connection errors, and can assist with automated restarts when connections temporarily fail. The Tangram Vision SDK is launching in public beta in Summer 2021. Early access to our private Alpha is available by request. RR

This Vicalib sensor calibration pattern was developed by the Autonomous Robots and Perception Group at University of Colorado Boulder.

|Tangram Vision

To prevent data blackouts and latency, your data model needs to consider a concurrent approach that does not block the host om continuing to consume new data as it comes in. This approach not only ensures that just enough sensor data is always available for safe operation, but also preserves host compute for other critical AMR tasks.

May 2021

5.critical.sensor.integration.challenges_RR_5-21_Vs4.LL.indd 72

www.therobotreport.com

THE ROBOT REPORT

4/30/21 3:11 PM

Robotics Robotics

DreamVu Obstacle Detection Starter Kit Shipping Now If a robot can’t sense it, it can’t accommodate it. And yet, today’s depth sensors fall short of meeting designers’ needs for obstacle detection. This means that the sensor stack is complex and compromised. And complex sensor stacks are a challenge to design, characterize, calibrate, deploy and support. This is where DreamVu’s solutions are having the biggest impact. This Starter Kit allows you to quickly evaluate the DreamVu obstacle detection solution. All you need is a keyboard, mouse and display. This solution detects any obstacle within a wide vertical field and across a horizontal field of view up to 360°. To be compatible with mapping solutions such as Cartographer and Gmapping, it simulates a 2D laser scan output. Ready to get started? www.dreamvu.com/shop to order now. Don’t forget to use the coupon code “palministarterkit”.

THE ROBOT REPORT

Robotic Tips 5-21_Vs1.indd 73

DreamVu www.dreamvu.com +1.484.612.5816 3675 Market Street Suite 200 Philadelphia, PA 19104

Search Quickly, Find Quickly This smart software brings all of Festo’s online tools and Window tools under one roof. You’ll receive online access to all product information together with CAD data, marketing of the core product range components, and more. The shopping cart and our tools for selection and sizing are just a click away.

Festo Corporation 1377 Motor Parkway Suite 310 Islandia, NY 11749 Phone: 1-800-993-3786 Web: www.festo.com e-mail: customer.service.us@festo.com

www.therobotreport.com

May 2021

4/30/21 3:39 PM

Robotics Robotics

FUTEK We make innovation possible FUTEK Advanced Sensor Technology specializes in creating inventive sensor solutions for today’s leading tech innovators: • Load cells • Torque sensors • Pressure sensors • Multi-axis sensors • Instruments • Software Our end-to-end measurement products and services include sensors, amplifiers, and calibration, allowing you to streamline and optimize your system and achieve better results at a lower cost than legacy solutions. All our products are made in the USA. To learn more, visit www.futek.com.

FUTEK Advanced Sensor Technology, Inc. 10 Thomas Irvine, CA 92618 USA www.futek.com +1 (949) 465-0900

Honeywell Intelligrated® Robotic Solutions By leveraging advanced robotic technology with extensive material handling experience, warehouse automation solutions from Honeywell Robotics provide the speed, accuracy and efficiency to satisfy a broad and growing range of operational requirements. Innovative designs, simulation tools that predict performance before installation, application expertise and committed support ensure maximum dependability and round-the-clock productivity. Robotic solutions also relieve workers of some of the most arduous, repetitive and injury-prone tasks, freeing up limited labor for more rewarding, higher-value jobs. Best of all, Honeywell Robotics solutions can be leveraged as part of larger integrated systems, backed by proven integration and support capabilities. Honeywell Intelligrated is recognized by the Robotic Industries Association (RIA) as a Certified Robot Integrator, with more than a quarter-century of experience providing single-source robotic solutions for high-performance distribution and manufacturing operations. From system concepting, simulation, fabrication and integration to installation and commissioning, training and ongoing support, each solution is approached with a comprehensive lifecycle view to maximize the value of your system.

May 2021

Robotic Tips 5-21_Vs1.indd 74

www.therobotreport.com

Honeywell Intelligrated 1.866.936.7300

https://sps.honeywell.com/

THE ROBOT REPORT

4/30/21 3:39 PM

Robotics

JUST RELEASED: Revolutionary - Motus Labs ML1000 Series precision M-DRIVES offer a higher torque density than competing strain wave gearing with no compromise in performance. The ML1000 family of hollow shaft drives includes standard gear drive sizes ranging from 17-40. Motus Labs’ patented design utilizes a series of cam-driven blocks, instead of traditional gear teeth, that engage over 80% of the output ring surface area at all times resulting in a more rigid drive at a lower weight. The design distributes load stresses over a much larger surface area, permitting the M-DRIVE transmission to deliver up to twice the torque per unit size and volume and 15% greater efficiencies compared to strain wave drives.

Motus Labs 17815 Davenport Road Suite 130 Dallas, Texas 75252 www.motus-labs.com

Fostering Innovation, Expanding Opportunities, Building a Community

5.20.21 Construction Robotics: Critical Needs, Functional

Session Sponsors:

Gaps and New Opportunities in a $10T Market

5.27.21 Edge, Fog and Cloud Robotics: Computation,

Storage and AI Solutions for Robotics Developers

6.03.21 Robotics-Assisted Surgery: Critical Challenges

robobusinessdirect.com

and Robotics Opportunities

6.24.21 3D Printed End-of-Arm Tools: Advantages,

Sponsorship opportunities are available for future ROBOBusinessDirect programs.

Applications, & Use Cases

For more information, contact

COURTNEY NAGLE

cseel@wtwhmedia.com | 440.523.1685

THE ROBOT REPORT

Robotic Tips 5-21_Vs1.indd 75

www.therobotreport.com

May 2021

4/30/21 3:40 PM