Robotic bricklaying, Towards adaptive robotic programing by Khaled ElAshry

M Sc Ada p tiv e Arch i tecture and c om pu ta t ion Bartlett school of graduate Studies Unive rsity Colle ge London September 2013

Robotic Bricklaying Towards adaptive robotic programing Khaled ElAshry

1 `

Word Count: 10891 This dissertation is submitted in partial fulfilment of the requirements for the degree of Master of Science in Adaptive Architecture & Computation from Bartlett School of Graduate Studies, University College London. I, Khaled ElAsrhy confirm that the work presented in this thesis is my own. Where information has been derived from other sources, I confirm that this has been indicated in the thesis. September 2013 2 `

Abstract

Robotic Bricklaying

ABSTRACT After the recent move of the field of architecture towards fabrication, the use of industrial robots surfaced as a replacement of custom fabrication machines. Robots are typically controlled with strict offline programing to produce controlled results similar to how robots are used in manufacturing. However, in construction robots did not manage to replace humans in on site operations. In the bricklaying profession, the craftsman uses his tuition to adapt to the material he is working with and to the changing work environment. This raised the question of the ability to build a system that can respond to changes in order to replicate the craft using robots. The research focuses on two main aspects, of bricklaying: The craft of laying brick and mortar itself and the ability to keep track of the work on a larger scale. In order to replicate these aspects an adaptive program had to be built with reliance on sensors and feedback to control the robotic arms. The robotic arm managed to achieve the tasks in question using two software: A new plugin to control the movements of the robots and a server that is capable of dealing with feedback. Key words: Robotic Arms, adaptive programing, bricklaying, construction, architecture, fabrication, feedback, computer vision, unpredictable material, Mortar, Brick.

Contents

Robotic Bricklaying

CONTENTS Abstract ............................................................................................................... 3 Contents .............................................................................................................. 4 Acknowledgment ................................................................................................ 6 1

Introduction ............................................................................................... 9

Background .............................................................................................. 12 2.1 The craft of bricklaying ......................................................................... 12 2.2 Robots in architecture .......................................................................... 15 2.2.1 Early adoptions ............................................................................. 15 2.2.2 The current state of robotic research in architecture. ................. 16 2.3 Observations during the Robotic foaming Workshop, SG2013 ........... 17 2.3.1 Robotic brick work ........................................................................ 19

Methodology ........................................................................................... 21 3.1 The robotic arm .................................................................................... 21 3.2 Onix Grasshopper plugin ...................................................................... 23 3.2.1 Kinematics of the UR robotic arm. ............................................... 24 3.2.2 Forward kinematics ...................................................................... 25 3.2.3 Inverse kinematics ........................................................................ 26 3.2.4 Solution toggles ............................................................................ 27 3.2.5 Angles calculation ......................................................................... 30 3.2.6 Execution path generation ........................................................... 31 3.2.7 Robot program and program uploader ........................................ 32 3.3 The end‐effectors ................................................................................. 33 3.4 Robot execution paths ......................................................................... 34 3.4.1 Pick and place of bricks. ................................................................ 35 4

Contents

Robotic Bricklaying

3.4.2 Accompanying tasks. .................................................................... 35 3.4.3 Laying mortar ................................................................................ 36 3.5 Brick laying server ................................................................................ 38 3.5.1 Program operation sequence ....................................................... 39 3.5.2 Interface ........................................................................................ 40 3.5.3 Computer vision and sensing ........................................................ 41 3.5.4 TCP/IP Feedback. .......................................................................... 43 3.5.5 Bricklaying server to Onix communication ................................... 44 3.6 Wall building tests ................................................................................ 44 4

Discussion and results .............................................................................. 47 4.1 Testing Results ..................................................................................... 47 4.2 Critical assessment ............................................................................... 48

Conclusion ................................................................................................ 51

Appendix .................................................................................................. 53

References ............................................................................................... 59

Acknowledgment

Robotic Bricklaying

ACKNOWLEDGMENT I take this opportunity to express my profound gratitude and deep regards to my supervisors Sean Hanna and Ruairi Glynn for their endless support and guidance, Angelos Chronis for his valuable feedback, the Bartlett CADCAM team for their great help, my gurus Mohamed Zaghloul and Ebtissam Farid for their inspiration, my friends for always being there for me, my father Mohamed ElAmir ElAshry and sisters Kholoud and Tayseer for their endless support and encouragement, and last but not least my late mother Neveen ElMaghloub for her love and endless belief in me.

List of figures

Robotic Bricklaying

LIST OF FIGURES Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003) ............. 12 Figure 2 Bricklaying on‐site ........................................................................................ 14 Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm, Famulus 1973 ............................................................................................................. 15 Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994) .......................................................................................................................... 16 Figure 5 Experiments during the Robotic foaming workshop SG2013 ...................... 18 Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57) .......................... 19 Figure 7 DimRob system (Helm et al. 2012, p.1) ....................................................... 20 Figure 8 : To the left Six degrees of freedom around a point .................................... 22 Figure 9: To the right a sample of the solutions of rotations around a point using a UR10 robot and Onix .................................................................................................. 22 Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm ....... 22 Figure 11 Onix grasshopper plugin example file ........................................................ 24 Figure 12 Universal Robot UR10 servo axis diagram ................................................. 25 Figure 13 Forward kinematics component in Onix .................................................... 25 Figure 14 Inverse kinematics component in Onix ...................................................... 26 Figure 15 First IK solution toggle diagram ................................................................. 27 Figure 16 Second IK solution toggle diagram ............................................................. 28 Figure 17 Third IK solution toggle diagram ................................................................ 29 Figure 18 Angle calculation digram ............................................................................ 30 Figure 19 Move, I/O and program uploader components in Onix ........................... 32 Figure 20 Components of the gripper end‐effector .................................................. 33 Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached to the handle .............................................................................................................. 34 Figure 22 Pick and place sequence ............................................................................ 35 Figure 23 Gripper holding the trowel for mortar operation ...................................... 36 Figure 24 laying mortar execution path sequence .................................................... 37 Figure 25 robot executing mortar laying ................................................................... 37 Figure 26 robot removing excess material using the trowel ..................................... 38

List of figures

Robotic Bricklaying

Figure 27 the bricklaying server in operation. ........................................................... 38 Figure 28 Operation sequence of the bricklaying program ....................................... 39 Figure 29 Detailed operation sequence of the bricklaying program ......................... 40 Figure 30 Bricklaying server interface ........................................................................ 40 Figure 31 Intel procedural computing camera .......................................................... 41 Figure 32 Intel camera point cloud used for surface approximation ........................ 42 Figure 33 Limit switch testing brick hieght ................................................................ 43 Figure 34 right: Sequence of communication between the bricklaying server and Onix for a single operation left: program dispatcher in Grasshopper ...................... 44 Figure 35 first test to check mortar laying movements and server operation .......... 45 Figure 36 Second test to check error handling and feedback ................................... 46 Figure 37 third test to check program adaptation to different rotations ................. 46 Figure 38 operation sequence ................................................................................... 53 Figure 39 IK solver (Left Square) and uploading component (Right Square) location in the Onix plugin example file ...................................................................................... 56 Figure 40 IK solver component parts cluster ............................................................. 56 Figure 41 Point A calculation ..................................................................................... 56 Figure 42 Point B calculation ...................................................................................... 56 Figure 43 Point C calculation ...................................................................................... 57 Figure 44 Angle sorting algorithm cluster .................................................................. 57 Figure 45 Robot uploading component cluster ......................................................... 57 Figure 46 the eight possible solutions for a single tip plane using Onix IK solver ..... 58

Introduction

Robotic Bricklaying

1 INTRODUCTION Robotics is a fairly new field in technology; The term robot dates to the 1920s (Spong et al. 2005, p.1). Robots have come a long way since then, growing more complex and capable. Robots are now a key part in manufacturing industries such as the automotive industry. However, in construction, they did not play an important role. Human labour remained the main asset for many reasons. e.g., the work environment. Construction is a very complex work field; tasks that are very easy for human to achieve (like avoiding an obstacle or moving from one work location to the other) are big challenges for robots. In the 1980s (Bechthold 2010, p.119) numerous trials to integrate autonomous machines in construction surfaced especially in japan, these trials were faced by another major drawback; the economical visibility of the integration of such technology. These machines were huge and expensive which meant that they weren’t a viable solution economically to manual labour. Also, construction now might take the advantage of machinery; they are most of the times directly controlled by humans and very far from being autonomous. Recently, the interest in robotic in construction has surfaced again as robots are getting cheaper while labour in western societies is getting more expensive. Robotic fabrication has been traditionally associated with high‐tech industrial environments, where fixed positioning and constant conditions determine the role that the robot may undertake in the fabrication process. Unlike in stationary facilities, a construction site is a spatially complex and heterogeneous environment in which a robotic system may be exposed to continuous change and unpredictable events. (Helm et al. 2012, p.1) Bricklaying is a construction profession that depends mainly on manual labour. The craft of bricklaying hasn’t changed dramatically through history. The tool used in the modern construction site is as old as the profession itself(Campbell & Pryce 2003, p.303). The craftsman throughout history played a huge role in the realization of several great structures that are still standing till now. Architects had a huge influence on bricklaying as they themselves were sometimes considered master craftsmen (Hill 2005, p.14). Nowadays, the craft is becoming more of a profession of filling the gaps between prefabricated parts of the buildings in the modern western society. However, Architects are returning to their historic close relation to the 9

Introduction

Robotic Bricklaying

realization of their designs in the real world, researching new techniques in fabrication and construction, retaking their role as master builders (Kolarevic 2005, p.65). In Architecture, robotics has become a major research arena over the past eight years. The introduction of robotics in architecture research has started with the move of the field towards research in fabrication. As research grew more complex in the virtual form finding field, Architects were getting more and more interested in the art of making. Industries like ship construction(Kolarevic 2005, p.9) had a great impact on research in fabrication. Industrial robotic arms which were a crucial part of such industries for years, are getting much cheaper. Which made them a viable replacement to getting huge specialized milling or routing machines. The versatility of the robots meant there are several uses that go beyond just that and new research by pioneers in the field started to explore their potential. Robotic arms are very versatile machines, their use is fundamentally determined by the program that operates them and the tool attached to it. They were able to work with composite materials for example, and were as well able to do jobs like pick and place very efficiently. Architects were also researching in the software side of robotics, linking robotic operation to CAD software for easier generation of the controlling program for instance. In fabrication, Material behaviours plays a big role, the complexity of the integration of material behaviour in the process is determent by the unpredictability of the material itself. The exploration of the use of the robots with messy materials and composite fabrication projects required more advanced programing for the robots. This time using senses and feedback to produce an autonomous program that can adapt and change to external stimuli, making them more efficient and productive. This research into sensing techniques and adaptable programing is getting us closer to achieve robotic integration in heterogeneous environments like construction sites. Bricklaying is a craft that require practice, tuition and adaptable technique with the need to work with relatively unpredictable materials. Robots, also are powerful and accurate machines, are generally very inadequate at the execution of such set of 10

Introduction

Robotic Bricklaying

qualities. This raised the question of the ability of an autonomous robot to replicate this craft using the same low tech tools used by a bricklayer. In order to achieve that the research the primary focus of this research is to develop an adaptive operating workflow that can address two specific qualities of the bricklaying workmanship using robots: 

The craft of working with messy materials like mortar in combination with other components like bricks with the ability to shift between them



The ability to keep track of the larger plan and execute error correction.

Following the introduction, the background chapter focuses on the bricklaying craft, discussing the role of the bricklayer, followed by the recent related research in the fields of robotics and fabrication in architecture. Next chapter is the methodology, this chapter focuses on the creation of the software platforms and the process that allowed the testing of the craft reproduction using robotic arms, and also the tests conducted to build wall samples. Upon that the discussion chapter discuss the results and outcomes of the tests. And the research then ends up with conclusion.

Background

Robotic Bricklaying

2 BACKGROUND 2.1 The craft of bricklaying 2.1.1 The evolution of bricklaying The Hanging Gardens of Babylon, one of the seven wonders; the Great Wall of China, the largest manmade object in the planet; the hag Sofia, one of the most beautiful churches ever built …, they were built out of bricks. Brick is at once the simplest and most versatile of materials, the most ubiquitous and the least regarded, all too familiar yet strangely neglected (Campbell & Pryce 2003, p.13) Brick is one of the oldest building materials. Moulded brick is dated back to the 5000 BC (Campbell & Pryce 2003, p.15). Brick building evolved significantly through history with inventions like fired bricks and the use of concrete in Greece. Historic Brick building varies according to the type of brick, type of bonding material used and the technique used by the bricklayer. Bricklaying craftsmanship played a big role in the construction of the masterpieces all around the world. Throughout the history the relation between the Architect and the craftsman in this case the Bricklayer was strong. Architects were considered master masons in the middle ages. However, In the Italian Renaissance architects started distancing themselves from the craftsmen focusing more on the building representation (Hill 2005). Like many labour dependent workmanship, the bricklaying skill level required in ancient times is no longer in demand no today and skills that are not in demand diminish by time (Campbell & Pryce 2003).

Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003)

Background

Robotic Bricklaying

The Bricklaying craft was always accompanied through history by the profession of brick–making .In the western world nowadays, brick‐making is almost extinct. Bricks now are manufactured mechanically and transported to site. On the other hand, bricklaying seems to be fairly untouched by modern technology (Campbell & Pryce 2003, p.303). Machines that take bricklayer place are still not seen. This is the very distinct difference between the evolution of manufacture and construction in recent history. The brick can be mass produced cheaply in a factory in a relatively controlled automated manner. While a bricklayer has to deal with the changing work environment, adapting to it with each different job on site. 2.1.2 Observations of the bricklaying craft on-site Bricklaying is a profession that requires years of experience. The work of a master worker can be easily differentiated from a novice one especially when dealing with facing bricks. When the research started there was a necessity to watch a bricklayer craftsman and also try brick laying to get to know the profession. The observation and personal tries of bricklaying on‐site resulted in the Focus on two main aspects that defines the craft. 1. the act of brick and mortar laying itself 2. The focus on larger scheme of things. A perfect technique will cut time and waste material, and also result in a much better end product. Technique and material greatly affect the result. Personal tries revealed how the craft is a combination of skill that can be taught and expertise that is compiled over time. The bricklayer almost does his craft unconsciously. The tool is a major part of the process. The trowel itself is as old as the profession of brick laying and haven’t changed significantly along its history. Using such low tech tools, human tuition plays a huge rule in deterring the movement according to the different features of the material itself like weight and consistency.

Background

Robotic Bricklaying

Figure 2 Bricklaying on‐site

A bricklayer might seems to only focus on laying a brick but he also keeps an eye on the larger plan. This is vital in order for the brick rows and columns to end up matching in the whole building. Bricklaying is accompanied by other work on‐site like building openings and reinforced concrete works for example. A simple error in calculation can easily propagate to produce huge problems in the end that may require major error fixing or even the destruction of wall portions resulting in badly finished product. Bricklaying is still very far from mechanization but the quality of the craft is declining in western high tech world. Prefabricated parts is being used when possible. This lean towards prefabrication wasn’t only for machined parts that may require special manufacturing, but even the parts of the brick building that required most of the skill like the arches for example are shipped fully assembled and put in place in construction site. This is due to the higher expenses required to hire a quality craftsman to work on‐site. And at the same time ability to better control construction time and increases efficiency as much as possible. Brick laying is still a relatively untackled territory when it comes to automation. This gives a large space for what can be achieved. The innovation in the field brick design require new tools that can reproduce the technique and quality of ancient brick laying and go beyond that. At the same time increase efficiency and decrease time of construction, while also filling the need for such craft in unhuman environments for instance.

Background

Robotic Bricklaying

2.2 Robots in architecture 2.2.1 Early adoptions Industrial robotic arms have been offered commercially since the 1970s (international fedration of robotics 2012, p.3) Machinery has been a core topic in manufacturing for many years pushing the limits of research. Productivity on such fields have increased significantly in the last decade as opposed to the construction industry (Balaguer & Abderrahim 2008, p.2). Robots however seems to be always constrained to work in highly controlled facilities. Many robots barely have any sensing systems of their environment, perhaps a few sensors to detect errors like motors overloading but little else. When the robotic on‐site construction idea was explored in the 1980s and 1990s (Bechthold 2010, p.119) the formula that succeeded in the manufacturing industry didn’t work here. Robots were never developed to adapt to changing environments or to deal with a messy type of data. In fact they were developed to do very specific tasks with very specific programs that may take years to develop and test and then run for the life time of the machine (Keating & Oxman 2013, p.1).

Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm, Famulus 1973

In 1980s, during the Japanese booming era, up to 200 systems of automated construction were developed (Bechthold 2010, p.3). During this era the Japanese economy was very strong and the construction rate was very high. The profession of a construction worker on the other hand was very alienating for its high risk and low payment. The scarcity of construction workers in opposition of high demand required searching for new possibilities. Robots seemed to be the solution, and numerous 15

Background

Robotic Bricklaying

automated systems were developed. These were huge custom machines that required large amount of money to build and operate. By the end of the 1990s, the economy of Japan was hit and the research in automated construction stopped. The Idea continued with a slower pace. In 1994 some papers were published during the 11th ISARC conference. One of the papers was discussing the autonomous machine to build brick walls (G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994). Another system under the name of ROCCO was also proposed as a solution for robotic masonry work on‐site (Andres et al. 1994). The system was referred to as a “Fault Tolerant Assembly Tool”. As inferred from the title it tried to rely on sensors and feedback for fault correction .It was designed as a purpose specific robotic system. This time it was smaller and more approachable than the systems that were developed in japan. It also proposed a link between CAD and construction directly. The research in the platform continued with two more papers in 1995 and 1996, but it wasn’t put in practical use.

Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994)

2.2.2 The current state of robotic research in architecture. In recent years. The architecture practice has been changing and branching with more research into material, construction and fabrication, forming new modes of collaboration between disciplines. Architects this time w developing their own design culture (Glynn & Sheil 2011, p.117). In this new turn architects are searching into means to represent their digitally created work in the real world. This turn towards fabrication made research in robotics surface again with huge momentum. This time using the same robots that has been used by the industry for years instead 16

Background

Robotic Bricklaying

of using huge custom robots (Braumann J. and S. Brell‐Cokcan. 2013, p.8). Robots like these are getting more affordable thus becoming more and more accessible for both architectural research and even practices. Pioneering offices like Snøhetta and Foster+Parteners are getting their own robotic arms in order to experiment with fabrication. For the major part of the research robots are usually used as either a sophisticated 3D printing or milling machine executing specific preconfigured paths in a controlled manner similar to the way it is used in the industry. However, this time architects are finding better much faster ways of planning these fabrication programs that is more approachable and simpler. This advanced the research of robotics in architecture significantly in the field of fabrication. It allowed a faster learning curve and ability to easily link to geometry directly from CAD software. The real world is very complex which make modelling it in a virtual environment is usually quite a challenge. Another side of research in robotic fabrication explores a more sophisticated operational programing. This requires more knowledge of the field robotics and links to other fields like computer vision, mechanical engineering and mathematics. . Research in this field rely on using feedback loops and more sophisticated sensing methods to read from the surrounding environment to allow a more dynamic approach that can adapt with the material in use. 2.2.3 Observations during the Robotic foaming Workshop, SG2013 During the participation in the “Robotic foaming workshop” (Colletti et al. 2013) in the 2013 smart geometry conference, the problem with direct blind execution from digital to physical was visible. The workshop was built on the idea of working with spray foam (a very unpredictable material), using the precisely controlled robots. This subject matched perfectly the conference theme “Constructing for Uncertainty” in theory. Even though very interesting forms merged form the workshop, it was far from perfect in the process front. During the workshop, foam was mixed manually with other ingredients like hardener and water. The process produced a very elastic fibrous mixture. It was then attached to plates on the three robotic arms. The arms then took the part of stretching the mixture. During the testing the foam mix 17

Background

Robotic Bricklaying

produced was very inconsistent, and even mixtures that were produced in a very similar and timed process still behaved differently.

Figure 5 Experiments during the Robotic foaming workshop SG2013

The material was delicate and required the constant change of speeds while executing the program. Due to that, running the preconfigured program directly always resulted in a complete tear of the material and the failure of the test. On the other hand, the tests run by pulling it manually by the participants were very successful. In order to achieve the same result using the machines, the robots had to be controlled almost in a total non‐autonomous manual manner using the control joystick, this defied the purpose of the controlled offline designed program. The work on this project put into perspective the importance of having an online feedback loops and vision systems that can deal with such materials while keeping the automation. The importance of using sensors and adaptable programing was investigated in other similar projects recently like The magnetic architecture (Dubor & Diaz 2012). The project was dealing with form finding using iron filling and magnets. The same result was concluded from the research; Digital representation only can’t achieve accurate results with such messy material. This deemed the linear process of offline program generation not reliable.

Background

Robotic Bricklaying

2.2.4 Robotic brick work ETH Zurich is considered one of the major pioneer institutes that work on the integration of robotics in architecture. One of their major areas of research is the construction related manufacturing, especially with bricks. Gramazio and Kohler succeeded in creating a practice that can combine research and architectural work with many projects designed and constructed in the past years. Many of the brick walls manufacture were usually produced in a lab environment and then shipped to the construction site like the Gantenbein vineyard façade project for example (Gramazio & Kohler 2008, p.95). Although these walls were very innovative visually, the product was never structural and resembled a cladding system more. This was a great potential worth exploring, building structural Brick walls that can also benefit from the innovative visual effect.

Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57)

Other construction related robotic research at ETH Zurich considers the use of robotic arms outside the laboratory setting. In 2007 R‐O‐B was introduced (Gramazio & Kohler 2008, p.57), it is a mobile manufacturing facility that included an ABB industrial robotic arm at its core (Figure 6). This allowed the exploration of the production of brick structures on‐site. The unit was moved around the world in locations in Europe and the USA and produced brick walls on‐site. Another recent project had more emphasis on feedback. DimRob is mobile construction system designed also in ETH Zurich. The system relied of a mobile robotic arm. The size of 19

Background

Robotic Bricklaying

the proposed system allowed the movement of the robots trough standard openings in site (Figure 7). It heavily relied on sensors to respond to the uniqueness of each construction site(Helm et al. 2012, p.1) along with more collaboration with humans.

Figure 7 DimRob system (Helm et al. 2012, p.1)

The research in the field of on‐site construction and dealing with changing environments and materials is still relatively young in the architecture field. The Pioneers in robotics and fabrication are researching more in this field of adaptive robot programing and the use of sensing capabilities. Such field have several applicable potentials. Bricklaying is considered a great candidate for the integration for such process.

Methodology

Robotic Bricklaying

3 METHODOLOGY In this research, the robotic arm was an important part along the software that runs it. The understanding of such tool was crucial. However, this was not a straight forward process the novelty of the robots used during the research and the lack of any technical support meant it was generally a process of trial and error. This also required building new platforms from scratch to explore the controlling possibilities of these robots before the research can proceed with the testing of its hypothesis.

3.1 The robotic arm Robots are fascinating electromechanical machines. It is defined as “a machine capable of carrying out a complex series of actions automatically, especially one programmable by a computer”(Oxford dictionaries n.d.). As the definition implies, robots are usually defined by two main aspects: their physical mechanical aspect and the software that runs them .In this essence the robotic revolution went hand in hand with the finding and explorations in the field of computer science. The most common multifunctional robots are robot Manipulators “Robot Manipulators are composed of links connected by joints to form a kinematic chain. Joints are typically rotary (revolute) or linear (prismatic).”(Spong et al. 2005, p.3). This study has focused on the most common type which is the articulated rotary robotic arms. Robotic arms are considered rigid body systems and usually are defined by its degrees of freedom. They are the possibilities of movement of the end effector tip. The most common number of degrees in industrial robotic arms is six degrees where the end is able to translate in three dimensions and also rotate around three axes (Figure 8); this allow all the possible motions around a point, which renders more degrees in most cases useless. The degree number is also directly related to the number of servos in the robot.

Methodology

Robotic Bricklaying

Figure 8 : To the left Six degrees of freedom around a point Figure 9: To the right a sample of the solutions of rotations around a point using a UR10 robot and Onix

The setting of the robotic arm’s servos affects directly how the robot is controlled. The most common type of robotic arms manufactured by major manufacturers is the spherical wrist robotic arms. In this type the three wrist axes of rotations intersect at one point. The non‐spherical wrist type which is relatively uncommon was introduced recently by the Danish company universal robots. The latter type has advantages over the spherical wrist type but needs a more complex inverse solution for its kinematic chain.

Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm

In 2012 The Bartlett School of architecture received two universal robots for the sake of expanding its fabrication facilities. The UR10s are six degree of freedom non‐spherical axis robots with 10 kg pay load. The problem the research faced along other initial users of the robots is the lack of any controlling software or technical 22

Methodology

Robotic Bricklaying

support for these robots as opposed to others manufactured by the other companies like ABB and KUKA. The robots initially were only controlled via its control pendent and could only be programed for simple pick and place programs. The uncommon joint configuration design meant the available tools for other robots cannot be reused to control the URs at the same time they lacked any sort of publicly available control software either by the manufacturing company itself or third party developers. In the search for tools to control the robot many suggestions were tested like using the Blender modelling software to control the robots via bones and animation (developed by Tom savilans). The initial blender model relied on the animation functions in blender and required to send the instructions to the robot one by one. The tool however was unreliable and a new tool was needed to control the robots

3.2 Onix Grasshopper plugin In the beginning of the research, it was very crucial to build a control software that will enable the researcher to work with the robots in a meaningful way. It was decided to build tool from scratch that will allow planning robot jobs easily with the ability to directly upload to the program. This also allows having access and the ability to fiddle with all its components according to the needed task. Grasshopper was chosen as the base platform to develop the software. Grasshopper is a .net plugin that runs on top of the 3D NURBS modelling software rhino. It combines ease of use through its VPL interface and also the ability to program in VB, C# or python languages directly inside it. The adoption of several of practices and universities to using grasshopper made it a very good candidate to build a controller software. Grasshopper also has several built‐in tools to create plans. The planes contain all the required data (three translations and the rotation) in order to calculate the kinematic chain of the robot. These tools become very useful to build paths that the robot can execute along points, curves or even surfaces. Other solutions for controlling robotic arms that were built on it for other manufactures succeed in the architecture related research environment. In a comparison between the use of KUKA|PRC (a grasshopper plugin for offline controlling of KUKA robots) 23

Methodology

Robotic Bricklaying

against the company supplied path planning software was found to increase productivity and reduce time to move from design to fabrication (Braumann & Brell‐ Cokcan 2011) .

Figure 11 Onix grasshopper plugin example file

The result definitions during the research lead to the creation of the Onix grasshopper plugin. The plugin allowed the control of the robot directly through streaming from the grasshopper interface. It included Inverse and forward Kinematic solver for the universal robots in addition of upload, communication and end– effector tools which will be detailed further in the upcoming part (Figure 11). The plugin was packaged and released as an open source plugin and is used as a controlling platform for other research projects. 3.2.1 Kinematics of the UR robotic arm. The robotic arms are considered rigid bodies. When using robotic arms the user usually is majorly concerned mainly by the position and orientation of the tip of the end effector (The end‐effector refer to the tool used at the end of the robot that may have extra transformations). In order to represent this tip point, six number are needed which represent transformation location and rotation of the point, this set of data will be referred to as planes during the research. The set of servo rotation angles that achieves this tip plane position is called configuration(Spong et al. 2005, p.4). The universal robots have six servos referred to by the manufacturer as (base, shoulder, elbow, wrist 1, wrist 2, wrist 3) (Figure 12) The calculation of kinematic 24

Methodology

Robotic Bricklaying

chain for any configuration require two major function to be computed: Forward and inverse kinematics.

Figure 12 Universal Robot UR10 servo axis diagram

3.2.2 Forward kinematics

Figure 13 Forward kinematics component in Onix

Forward kinematics deal with the relationship between the joints positions the position and orientation of the end‐effector(Spong et al. 2005, p.68) . The forward kinematic fundamentally solves the location of the tip according to the joint position. The related component in the Onix plugin relied on an input of the six rotation angles (the configuration) of the robots servos (Figure 13). The plugin then applies a set of transformation and rotations to the model of the robot until achieving the tip position. It was also used also to show the representation of the robot along with the solution of the tip plane. 25

Methodology

Robotic Bricklaying

3.2.3 Inverse kinematics For the Inverse kinematics solution, the function is presented with the tip plane as the input and calculates the set of the joint positions needed to achieve this location. Inverse kinematics is generally nonlinear meaning it results in multi solutions for a single input. In this part the difference between the universal robots and their counterpart are the most noticeable. The less common new servo setting of the universal robots meant fewer singularities (Singularities in robotics refer to a tip plane that don’t have one unique solution in the joint solution space. This either refers to having infinite solutions or no solution at all) that may occur in the middle of the configuration space. The UR robots are capable of eight different solutions for most of their reach space (Appendix III) as opposed to four in the spherical wrist type. In order to calculate those solutions, the inverse kinematic solver in Onix was built from scratch directly in grasshopper using a geometric approach rather than using a solely mathematical one. This allowed it to be easier to debug and edit in grasshopper. The input of the inverse kinematic solver in Onix consist is the plane of the tip point. Starting with this input, the kinematic chain is computed then the angles are calculated accordingly. The inverse kinematic chain is calculated by solving the location of three key points along the chain. Each of these points have multiple solutions for each tip plane input, thus three more toggles were added to the input of the inverse kinematic solver in order to switch between the eight different solutions.

Figure 14 Inverse kinematics component in Onix

Methodology

Robotic Bricklaying

3.2.4 Solution toggles Due to the novelty of the universal robots, there were no information about how to solve their inverse kinematic chain and the initial trials with using the same methods for spherical robots directly was unsuccessful. The new proposed solution however came from the observation of the robots themselves. During their operation it was noted that some configurations can have unlimited solutions in the joint space, where the fifth axis can rotate continuously altering the rest for the chain while maintain the tip poison. The solver was built initially to solve this configuration as it allowed the second point in the chain to be disregarded (point B as referred to latter on) as any solution for it is legitimate. After calculating the geometrical solutions for the first and third points, finding the solution for the second point for the rest of the configurations was much easier and the kinematic chain was finally constructed.

Figure 15 First IK solution toggle diagram

The solutions calculation starts with the inputs of the tip end plane. The first point along the chains (point D) (Figure 15) is determent as the offset of the tip plan in the direction of its normal; this determines the location of the fifth axis. Afterwards the different solutions are calculated. First solution toggle results from the calculation internal tangents between two circles. 1.

One with the point D as its centre, and radius equal to the distance between axes five and six. (These distances are constant and comesfrom the robot manufacturing dimensions). 27

Methodology

Robotic Bricklaying

The second circle with the base origin coordinates as its centre, projected on the first circle corresponding plane, with radius is equal the distance between the origin and the second axis.

These two circles represent the unlimited solution that the fifth and second axis servo rotations. The intersection between the tangents and the resulting circles represents two unique solutions that achieve the desired tip position. Point A on the chain is determent according to the toggle choice.

Figure 16 Second IK solution toggle diagram

The Second solution toggle calculation depends on the solution of two intersecting planes. One representing the tip plane and the other is the plane normal to the perpendicular vector on the chosen tangent (from the first solution)(Figure 15). The two solution for this toggle result from intersection between the resulting straight line and the circle representing all the possible solutions of the forth servo rotation around the six. The point B Figure 16) represents the selected solution point. This solution toggle is the major difference between the URs and the spherical wrist robots.

Methodology

Robotic Bricklaying

Figure 17 Third IK solution toggle diagram

The third solution toggle is determined by the calculation of the intersection between two circles with centres A and B (Figure 17) (determined from the first two solutions). A similar approach for the calculation of this solution was published by Daniel piker for the calculation of the angles of the KUKA robots (Piker 2013). 

The radius of circle with centre point A is the distance between the second and third axis.



The radius of the circle with centre point B is the distance between the third and fourth axis.



The solution results from intersection between the two circles. The chosen solution is point C.

Methodology

Robotic Bricklaying

3.2.5 Angles calculation

Figure 18 Angle calculation digram

The angles for the joint position are calculatedaccording to the kinematic chain determined from the solution toggles. 1. The first servo angle (base) has is the angle between the X direction vector of the origin plane and the vector represented by the direction between the origin and the projection of point A (the base rotation vector) on the origin plane. 2. The second servo angle (Shoulder) is the angle between the perpendicular vector to the base rotation vector and vector in the direction A‐C. 3. The third axis (elbow) angle is the angle between the vector C‐A and C‐ B. 4. The forth angle (wrist 1) is the angle between vector B‐C and the vector B‐D. 5. The Fifth angle (wrist 2) is the angle between the normal direction vector of the tip plane and the base rotation vector 6. The sixth angle (wrist 3) is the angle between vector B‐D and the x axis of the tip plane.

Methodology

Robotic Bricklaying

3.2.6 Execution path generation The universal robots servos are capable of moving 720 degrees, from positive 360 degrees to negative 360 degrees for all its joints. This resulted in having 2 legitimate joint solutions in any servo that can produce the same solution. For example if the axis angle result from the inverse kinematic algorithm is 50° this means that ‐310° is also a legitimate angle to active the same location. This increases the total solution possible for any tip plane to 512 and the solutions that can achieve one of the eight kinematic solutions to 64. This became a challenge when there is several inputs for the inverse kinematic algorithm (when dealing with a continuous path for example), the IK solver calculates the solution for the inputs individually only in the angle space from 0° to positive 360°. In the initial tests, this resulted in some major errors in the robot program when executed that may occur randomly when moving from one configuration to the next. This might have catastrophic results duo collision for the robot with the surround environment or the model it is working on. The solution for this was building a sorting algorithm that allows the grouping of the closest solutions when dealing with a continuous program. It was crucial for the program when dealing with paths to put the timeline and previous positions into consideration. To achieve that the IK algorithm calculate the First configuration for the first tip position (referred to as home position usually) which the user can override manually using angle toggles (Figure 14), and accordingly he selects from two sets of automatically generated angles list for the whole path for each servo. One list starts with the angle that is the closest to zero and one starts with the angle that is the furthest. For example The IK for a single servo generated numbers like (100, 10, 330 and 20) for the points along a hypothetical path. If this list is executed without sorting, the servo will rotate 320° form the second state to the third state and back 310° degrees. After the sorting of the solutions, the two generated lists are (100, 10, ‐30 and 20) or (‐260,‐350, ‐30 and ‐340). These two lists have both evade this problem resulting in a much smaller changes between states. The inverse kinematic solver and path calculations are all grouped into one component in the Onix plugin, allowing easy conversion from a set of planes to a set 31

Methodology

Robotic Bricklaying

of corresponding configurations in one step. An error system was also added to the component to stop program from execution if a problem with angle lists is still present in one of the lists or if the Inver kinematics fails to produce a viable solution for the configuration. 3.2.7 Robot program and program uploader

Figure 19 Move, I/O and program uploader components in Onix

After the calculation of the set of joint locations that represent the path. The “Move” component is responsible of converting these values to the URscript language. The URscript is a set of instruction that the universal robots can execute (universal robots n.d.). For each of the joint positions, the robot required one line of code starting with either “movej” or “movel” followed by the eight numbers , six of them refer to the joint positions of the servos then either the speed and velocity, or time and rounding. The difference between the two is related to the intermediate movements between states. The movej allow the movement to happen in the joint space while the “movel” required the movement to be calculated according to the tip position to maintain a linear movement of the tip. The set of moves are then combined in the uploading component. The upload component is responsible for the conversion of the lines of “Move” commands along with the I/O sates to a working program. The program is then uploaded via TCP/IP connection to the robot and is executed instantly.

Methodology

Robotic Bricklaying

3.3 The end-effectors End‐effectors are an important part of any robot system. They define the rule and use of the robot. In the beginning of the research, the end effectors were designed to test different stages of the process which will requires either using multi robots or changing the tools for each task manually. In the end a single system was designed to transfer from one task to the other seamlessly. The three major tasks needed to be covered by the end‐effector were: brick pick and place, laying mortar and sensing.

Figure 20 Components of the gripper end‐effector

The end effector was designed around an industrial pneumatic gripper. The chosen gripper had 3 cm stock and gripping force of 600 newton while keeping a weight of 1 kg. Adjustable holding fingers was then machined and attached to the gripper. The pneumatic air flow to the gripper was controlled by a 24 volt five way solenoid valve. The valve allowed direct control over it from the robots control box through the robot program itself. This was crucial to achieve synchronized robot movements and end effector actions

Methodology

Robotic Bricklaying

Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached to the handle

Achieving the second task required using a traditional trowel. A special handle was created in the same width as the bricks in use (100 mm) to allow the gripper to pick the trowel and use it. The force of the pneumatic gripper in addition to using rubber eliminated the problem of the trowel slipping while in use. The sensing was covered by a depth camera a limit switch. Due to wiring, these two needed to be attached directly to the end effector. The end‐effector design was tested virtually to achieve the best configuration that would decrees the incidents of collision and also keep the camera and the limit switch out of the mortar. The camera was controlled by the server computer, while the limit switch was controlled directly by the robot control box, connected in this case to a digital input.

3.4 Robot execution paths Onix allowed an easy way to create, edit and test the execution paths of the robots. Tasks needed to fulfil the brick laying process were divided in relatively small job specific paths for each task. Each of these execution paths are constructed in a parametric manner, changing and updating according to its inputs. Some of these inputs were determined once in the first calibration phase, while other change each time they are executed.

Methodology

Robotic Bricklaying

3.4.1 Pick and place of bricks.

Figure 22 Pick and place sequence

A simple path was generated to pick one block from one location to the other. The program has three inputs which are the brick stack location, brick number in the stack and the placing location and rotation. The brick stack location is defined once while the brick stack number and placing location change according to the order of the program. It was noted that during the placing the bricks on top of the mortar layer, the forces required to place the brick in its location increase dramatically if there is excess amount of mortar under. This is due the fact that the robot tries to push lots of material out and is faced with the opposite force that and results in the emergency stop of the robot. This led to the failure of the placement of the brick and the need of re‐initialization of the robot. After some tests the movement was altered to have rotational and side movements while placing the brick. This allowed the material to move and level easily and decreased the perpendicular forces facing the movement of the brick before. 3.4.2 Accompanying tasks. A set of simple paths were designed to allow the shift between tools. First path is the one which picks and leaves the trowel form a known location. The second path was moving the camera to a perpendicular position to the mortar location. The 35

Methodology

Robotic Bricklaying

location of the mortar working board is also fixed and given as a constant. The third program is picking the mortar. The input for this program is the point determined by the depth camera, indicating how much the trowel need to move to get the right amount of mortar.

Figure 23 Gripper holding the trowel for mortar operation

3.4.3 Laying mortar Laying mortar is the most interesting part of the process. After the gripper pick the trowel, the end‐effector is adjusted in grasshopper and the working plane is shifted from the middle of the gripper to the middle of the trowel surface. The degree of success of the laying moment was determent by two factors; First, the ability to move all the mortar from the trowel to the brick surface. Second is the creation of a consistent layer of mortar on top of the brick. A professional bricklayer usually uses one continues movement to lay the mortar on the brick. In order to mimic that with the robots, the tool needed to move very quickly to achieve a good result in one movement. After observing bricklayers do their work, it was noted that there are numerous factors which the movements of the trowel is adjusted accordingly. During the first tests the teaching option of the universal robots was used. In this mode the robots can be moved very easily by hand while streaming the joint positions through the TCP/IP connection. In grasshopper the movement are then recorded in real time creating the path. This made it possible to try to mimic the basic movement that the bricklayer does when using his own tool. 36

Methodology

Robotic Bricklaying

Figure 24 laying mortar execution path sequence

Figure 25 robot executing mortar laying

In order to achieve a more universally successful results, the laying process was broken into smaller movements that were tested individually with different mortar consistencies. First was the movement to place the mortar on the surface of the brick. The location of the placing of the mortar was very important to achieve a consistent layer latter on. The next movements to be tested was the set of movements that ensure that the all material left the trowel. The trowel paused and moved up and down a number of times to ensure all the material left it and then the trowel moves linearly to achieve a consistent layer of the mortar on the surface and remove any excess material. The mortar program only requires one input which is the location of the brick surface it is supposed to lay mortar on top. 37

Methodology

Robotic Bricklaying

Figure 26 robot removing excess material using the trowel

After applying the mortar and laying the next set of Bricks. The robot picks the trowel again and remove the excess material resulting from pushing the mortar out by the bricks. The process is one continouse movement from one side of the brick row to the other side. The input of this path is the start and end location of the brick row and the row hight.

3.5 Brick laying server

Figure 27 the bricklaying server in operation.

In the beginning ONIX was used on its own to control the robot. The major challenges were the limitation of grasshopper as real‐time controller. Also the real time visualization of the robot was integrated in grasshopper and allowed some features like path recording, problems with speed and inability to work with computer vision came between using grasshopper alone as the brick server. The 38

Methodology

Robotic Bricklaying

major problem with grasshopper by far was the way it is built in the first place. Grasshopper is very linear in the way it executes its programs. Even with direct programing it still faced lots of problems dealing with things like loops and retaining variable data. This required building another program that can be the server controlling the bricklying process. The program was built using processing programing language version 2.0 and was used to control the second major aspect of the research which is keeping track of the major plan and fixing general errors. It was also in control of the computer vision part of the process.

Figure 28 Operation sequence of the bricklaying program

3.5.1 Program operation sequence The processing server works as the controller of the operation. It collects all the data and control the execution of the robot paths. It allowed the shift between different programs and tools. After the user selects the operations he needs to be executed, the server initiates a logical sequence of operations to be processed. Execution commands are then sent one by one to be executed in the Onix plugin. The server always keeps track of the execution state of the program using feedback from the robot, and sends the next program as soon as the job is completed. It shows visual feedback in the interface about the progress of the work, and also allows the user to overwrite the program in the case of errors.

Methodology

Robotic Bricklaying

In order to achieve that, the program is built on a set of If statements each one representing a task. The program won’t go in the execution of one until it ensures the one before is executed. Each statement also keeps track of the locations of the tasks allowing the server to go from one location to the next. The server also automatically compensate for errors.

Figure 29 Detailed operation sequence of the bricklaying program

3.5.2 Interface The simple user interface of the program is built using the ContolP5 (Schlegel 2012) plugin for processing. The user interface is used to input the basic information like the width and height of the brick, the number of brick rows and columns of bricks, the spacing between the rows and columns of the bricks and the brick stack amount for both full size and half bricks. The interface also is used to choose the programs tasks needed to be executed by the robot, like brick pick and place, laying mortar, pausing the program and emergency stop. The interface helps monitor progress and errors and accept edition during execution.

Figure 30 Bricklaying server interface

Methodology

Robotic Bricklaying

3.5.3 Computer vision and sensing Mortar in the right consistency is very close to liquid but still solid enough to keep its shape. This makes mortar a very unpredictable material specially when dealing with using low tech tools. When adding or subtracting from the material, it will not level and would keep its shape and change slowly with time. The challenge was getting the right amount of mortar on the trowel each time. This required having a way to determine the arbitrary surface of the mortar each time before digging in with the trowel.

Figure 31 Intel procedural computing camera

Depth cameras came as a solution for this problem. The focus was to have all the tools needed on the robot itself without having a general vision system that overlooks the whole process. This made the Kinect camera unsuitable for the task as it is bulky in comparison to the size of the universal robots. Nevertheless, the new Intel Perceptual computing camera prototype many advantage. It is much smaller which made it possible to attach easily to the rest of the end‐effector and is unlike the Kinect camera it was able to work on a range from 0 to 1.5 meters. This made it possible for the robot to move and peek at the objects at close range getting a higher density point cloud of the scanned object.

Methodology

Robotic Bricklaying

Figure 32 Intel camera point cloud used for surface approximation

The point cloud you get from the camera suffers from the same noise problem as the Kinect camera. But because it can work at a very close range meant that the noise reduction would have less effect on the accuracy of the scanned surface. For the noise reduction, the scanning algorithm would store the point cloud data five times in a frame of a second and then the average of each of its point location is used to resemble the surface. The surface if the mortar is then determent and the volume is calculated. Each point of the point cloud is considered a rectangular box with fixed width and depth and variable height. The height data are calibrated according to 4 points that represent the base working surface. These points are determent on the first run of the program. The algorithm then moves on the surface of the mortar from right to left row by row determining how far the towel should go until it have the amount of mortar needed for the application.

Methodology

Robotic Bricklaying

Figure 33 Limit switch testing brick hieght

In order to check the height of the placement of the bricks for errors, the first plan was to use the depth camera to check the surface of the brick after laying it on top of the mortar surface. Due to the noisy data of the camera this was deemed unreliable. Instead a limit switch was added to the end effector. The limit switch was connected to the digital input of the robot control box. During operation the robot would move to a point which would make the limit switch tip 5mm away from the supposed surface of the brick. Then it will move at very slow speed until the limit switch is pressed and the location of the new point is recorded at the server program. 3.5.4 TCP/IP Feedback. The universal robots are transmitting their state continually through their TCP/IP connection. In initial tests it was difficult to decrypt the message that was sent back form the robot due to the lack of support from the manufactures. Instead the server relied on the python‐urx plugin (Roulet n.d.). The plugin was used to read the message back from the robot and decrypt it. The message was then sent to both the processing server and the onix plugin via the User Datagram Protocol (UDP) connection. For the processing server the message included the state of the program running. This allowed the server to send the next command to the Onix plugin as soon as the previous task was executed. The location of the tip was also sent to the server in order to determine the height of the point acquired after the limit switch is 43

Methodology

Robotic Bricklaying

pressed. This allowed to check the deviation of the bricks surface form the original design. For the Onix plugin however the message included used the joint locations. This allowed the plugin to show the robot virtual representation in real tam while excursing the program while also record movements if needed. 3.5.5 Bricklaying server to Onix communication The communication between the two programs was achieved using UDP connection. Using the protocol the processing server sends an execution command to grasshopper starting with a specific identifier. This ID allowed the program to be distributed to its corresponding robot parametric path in onyx. The message also included the information needed of the parametric grasshopper definition. For example the message sent for a pick and place job would be “B%10,50,0%‐ 300,0%3,True” In grasshopper the message will be broken intro 

Program ID : B (Brick pick and place)



Location of the brick to be placed is ( 10 mm , 50 mm , 0 mm ) from origin



Brick Stack location ( ‐300mm , 0 mm ) from origin



Number of the brick in stack is 3

The numbers are then dispatched to it corresponding parametric input and the program is altered and sent to the robot to execute.

Figure 34 right: Sequence of communication between the bricklaying server and Onix for a single operation left: program dispatcher in Grasshopper

3.6 The building tests The First bricklaying experiment was conducted to test the mortar laying movements. The aim was to build two rows of bricks with the least excess material. 44

Methodology

Robotic Bricklaying

It was repeated a number of times with alternations to the program until the desired result was achieved. The flow of the program was also tested. The server program shifted from the tasks of brick laying with the gripper directly to gripping the trowel and changing the program to deal with mortar and back to laying bricks and removing the excess amount of mortar on the side after.

Figure 35 first test to check mortar laying movements and server operation

The second test was conducted to build a higher wall. This allowed the testing of the sensor data. The location of the brick wall needed to be shifted from the table surface to the level of the ground to allow the robot to build higher and safer. One of the first problems was the adaptation of the movements that is built in grasshopper to the new position. The movements were built initially to work in the space in front of the robot and relative to the fixed world coordinates. This created a problem that was generally visible when dealing with trowel. The programs were then rebuilt for this task to work according to the location relative to the robot. This allowed the program to be executed in any location around the robot easily without the need to alter the program. The test also was done using bricks and half bricks. The bricks were cut manually and stacked in a secondary location. The server shifted between the locations of the half bricks and the full ones according to the need. The rows were built one by one and the height was tested for each row before laying the new mortar layer.

Methodology

Robotic Bricklaying

Figure 36 Second test to check error handling and feedback

The Third test was conducted to test the ability to use the program to build different design formations. This time the rotation angles for each brick changed in order to create the desired pattern both for the brick pick and place program and mortar laying. Another challenge accrued due to the use of the gripper. The new formation of the bricks meant that the gripper will collide with the previous brick when exciting pick and place. This required altering the picking location of the bricks according to the rotation.

Figure 37 third test to check program adaptation to different rotations

Discussion and results

Robotic Bricklaying

4 DISCUSSION AND RESULTS 4.1 Testing Results The experimentation for the program started with testing of the success of task specific operations. Tasks like pick and place and camera movement were straight forward and high success rate was achieved form initial tests.in the during the test of laying mortar on the other hand, trying to replicate the bricklayer movements directly were unsatisfying and had low success rate. This was due to the inconsistency of the mortar material itself. When a bricklayer usually lays the mortar with the trowel it is easy for him to determine the consistency of the mortar and adjusts the movement accordingly, most of the time subconsciously. The robot on the other hand at this point has simpler inputs to its program and is unable to determine the consistency of the material it is holding. Several of other challenges occurred in the creation of the mortar laying path. When the trowel is left for some time while the robot is laying bricks for example the mortar layer on the trowel starts to harden as it loses moisture. This makes the trowel lose its smooth surface which makes it harder for the mortar to leave the trowel after picking it up. The successful set of movements to lay mortar however was different from the ones used by a brick layer. This time the process was much longer and included several movements that might not be needed at all times, but is needed to achieve a global success rate with different scenarios. After several tests the brick laying movement had a high success rate to achieve a consist layer of mortar between bricks. The Onix plugin managed to be a successful tool to produce and control these paths, test them and stream them to the robot with the robots. With regards to computer vision, the depth camera was successful to determine the arbitrary surface of the mortar. The program managed to determine depth needed for the trowel to go into the mortar to get the desired amount. During the tests the process required a large amount of mortar to be placed on the working board. When dealing with a small amount, the mortar usually moved along with the movement of the trowel, resulting in an unreliable outcome. 47

Discussion and results

Robotic Bricklaying

The next tests were conducted to test the ability of the bricklaying server to control the process autonomously. The server relied on the data coming back from the robots like the statues of the execution of the programme and the location of the tip to alter and proceed with its sequence of operations. The tests showed that building an adaptable program for the robots to perform a multi‐task craft as bricklaying is achievable. The server program succeeded in the shift between different tasks while keeping a record of execution progress. The test also exposed how the error factor due to the complexity of the real world can accumulate. During the test of the height of each of the layers of bricks was shifted from the original virtual drawing. This was mostly due to the mortar settling under pressure from the layers on top. The brick itself was a part of this issue, even though the brick used were mechanically machined engineering bricks. There were small differences in the dimension of the brick in comparison to its advertised size. This resulted in up to three mm difference by the end the fourth row of bricks. The server was able to keep track of the error and try to compensate it in the mortar horizontal spacing. The last test revealed the possibility to use the same process to produce different brick designs. The parametric inputs of the execution paths allowed easy manipulation of the bricks rotation angles and locations for each row while keeping the automation of the program. The pick and place and the mortar laying movements both adapted to the changes in design.

4.2 Critical assessment For years the digital field was the major breakthrough arena in research and practice, the move from digital drawing to the physical creation was left for other disciplines without the control of the designer. However, there is more focus recently on physical realisation of ideas. During the robotic brick laying research the emphasis was on building a robotic program that can adapt and change. This nevertheless was not a case of direct execution digital to physical. The reliance on an adapting program meant that a dialogue can be maintained to some extent between the real environment and the virtual one. 48

Discussion and results

Robotic Bricklaying

To achieve the goals of this study, three main aspects needed to be addressed: the tool, the software that runs it and the task. In this case it was the robotic arm, the controlling software and the craft of bricklaying. Because of the novelty of the tool, a part of the research has been conducted to understand how it works, how to communicate to it and how to control it. This nevertheless resulted in a new platform that can be used by other users to easily control the robots in future research. The two software platforms used in the process are still in their early stages and are still in a constant development. The Onix grasshopper plugin can get more and more research on board to work with the robots with its simple interface and fast learning curve. It contained a set of easy tools that can be used directly to run the robots, or can be altered for more advanced users. Although this platform is capable to work on its own in the field of fabrication for example, it is still not able to work with feedback in a way that is more significant than the visualization of real time robot movement. This made the data flow in one direction from digital to physical and back but without a way to readapt the program automatically. These drawbacks are related to the grasshopper platform itself and how the programs is executed inside it and also other problems with speed for example as grasshopper wasn’t built for real‐time execution in the first place. The brick laying server program is the other software developed for the research .on the contrary was able to work with the feedback with great success. The use of programing for this tool allowed the program to easily adapt, keeping record of errors and building up data during the execution. In the same time signalling Onix to execute its paths with ease. Working with coding allowed the designing of a task specific interface that can keep the execution and data gathered easily visible for the user. This puts focus on the importance of programing as a language for controlling design. The server software is very far from being used with unspecialized architects in a way that is more significant that dealing with the simple interface. Combining the two platforms in a one working workflow benefited from both, allowing an easy familiar way to build robotic paths while maintaining the more specialized adaptable program.

Discussion and results

Robotic Bricklaying

Bricklaying as task was a major challenge, Even though several researches were inducted dealing with bricks. The more interesting challenges emerged from using mortar rather than the simple task of precise pick and place. Pick and place is a task taken for granted in robotics for years. On the other side, working with mortar put emphasis on the problem of dealing with a relatively unpredictable material. Adding this to the equation of pick and place rendered the precision of it on its own less efficient to predict the precision of the final result. Although the test was conducted in a small scale, it showed how the errors can build but quickly which may lead to a very unstable structure in the end if it wasn’t addressed during the execution. Mortar also exposed the importance of intuition in such craft as brick laying. While a brick layer might do one movement to lay mortar with high success. The initial tests that were conducted to directly copy these movements to the robots weren’t very satisfying. Achieving a good rate of success with the same tool required tests and numerous adjustments. Quite a few of the moves in the executed path might not be required all the time but is there to produce a more general success rate. The performance of the program was measured by the ability of the robot to achieve a mortar layer that cover all the surface without having a large amount of excess material that can come in between laying the next brick. The success rate in doing that was high in the last test but still was far from producing a very immaculate end result as produced by a skilled worker. Another drawback was the inability of the program to fix the errors directly, for example pushing a brick that is higher than the rest of the row to level it up. This was due to the limitation of the payload of the universal robots. The robots usually went into emergency stop very easily when bushing against mortar. This kept the error correction only on a larger scale dealing with the general execution of the program rather than on the scale of a single brick.

Conclusion

Robotic Bricklaying

5 CONCLUSION The research intended to shed the light on the potential of adaptive programing as a mean to control robotics in the field of architecture. Using such method robots can become more responsive to the task they are programed to execute. This potentially can lead to better results working with messy materials and heterogeneous environments. It can also enhance the design process with more integration of material behaviour and help overcome the difficulties facing the introduction of robotics in construction. The choice of bricklaying as the field of research was successful. It had all the potentials required to test the hypotheses. It is an important part of construction industry, with a large space for innovation, at the same time it is a craft that heavily dependent on human labour. Working on such craft allowed the testing of the ability of the robots to work with multi material using different tools and the ability to shift between them seamlessly. Mortar was the most interesting material during the process. The movement executed by the bricklayer to produce a consistent layer of mortar is highly based on skill, technique and tuition. Achieving high success rates with robots needed several tests. The resulting process is still not as immaculate as the work of a skilled bricklayer. However, the process achieved high success rate in the end. On the software side of the process, using such novel tool as the universal robot resulted in the creation of the Onix plugin, a new plugin specially designed to control universal robots directly from grasshopper. The potential of the plugin goes beyond this research and hopefully will continue as potential controlling software for other researches as well. The plugin allowed easy creation and edit of execution paths for the robots. This was very useful during the testing and editing of such paths for the tasks needed to fulfil the process, like brick pick and place and mortar laying. The process needed another piece of software that is capable to achieve the adaptive features of the process. The Brick laying server worked as the controller for the whole process, keeping track of the progress of the program and also dealing with the

Conclusion

Robotic Bricklaying

feedback from the robot, form the sensors and vision and also controlling the onix plugin itself. The small scale tests of the program succeeded in achieving its goals, however the process is still very far from being used in a construction environment. Humans still have vastly more sophisticated senses and capabilities and are still also the economical choice. Nevertheless, it is a step closer to achieving such integration. The field nevertheless still have great potential of further exploration for example in the design side of the process, in the mobility of the robots or the integration of machine learning. The programing aspect of the research still has great space for further exploration. The server program was only dealing with a part of the data available for it. The end‐effector and the robot itself offer large amount of data that is still to be utilized. Like live video feed from the Intel camera and torque data from the robot. The increased amount of information available can result in a more sophisticated program. And with more data, Finding patterns and making sense of this data can sometimes be more than a human can handle. Research in machine learning can be a logical next step for robotic adaptive programing. Producing a program that is not only adaptable but can grow more complex on its own learning from its mistakes and taking program beyond the designer’s scope.

Appendix

Robotic Bricklaying

6 APPENDIX 6.1 Appendix I: bricklaying server program operation sequence persuade code

Figure 38 operation sequence

User selects desired operations and bricks. Pick and place = True Mortar = True Check and adjust Height = True Read robot execution statues While (X < Number of operations) { If (communication established && Pick and place && Robot is Idle) { Send message “B + Brick location + stack number + True” to Onix X+=1 Brick +=1 If (Brick == column number) { Pick and place = False Brick = 0 } } If (communication established && Mortar && Row > 1 && ! Pick and place && Robot is Idle) { Check Height sequence While (Brick < column) { Send “H+ Brick location” To Onix Brick + 1 Read Tip location Calculate difference Shift locations of bricks If (Brick == column number) { Brick = 0 Height checked = True Mortar = Ture } } } If (communication established && Mortar && ! Pick and place && Robot is Idle)

Appendix

Robotic Bricklaying

{ Pick trowel sequence Send message “T + True” to Onix Trowel picked = true Mortar = false } If (communication established && Trowel picked && Row > 1 && ! Pick and place && Robot is Idle) { Remove excess mortar sequence Send “r+ row height” X+=1 Trowel picked = false Surface checked = true Excess removed = True } While ( Brick < column) { If (communication established && excess removed || Row=1 && && ! Pick and place && Robot is Idle) { Run Check motor surface sequence Send “C + True” to Onix Run depth calculation algorithm Trowel picked = false Surface checked = true } If (communication established && surface checked && ! Pick and place && execution not in progress) { Pick mortar from result location Send “P+ result location from depth” to Onix Surface checked=false Mortar picked = true } If (communication established && mortar picked && ! Pick and place && Robot is Idle) { Lay mortar from result location Send “m+ Brick location” Brick +=1 X+=1 Mortar picked = false If (Brick == column number) { Brick = 0 Mortar laid = True } } If (communication established && mortar picked && ! Pick and place && Robot is Idle) {

Appendix

Robotic Bricklaying

Lay mortar from result location Send “M+ Brick location” to Onix Brick +=1 X+=1 Mortar picked = false If (Brick == column number) { Brick = 0 Mortar laid = True } } } If (communication established && Mortar laid = && ! Pick and place && Robot is Idle) { Leave Trowel sequence Send “L” to Onix Mortar =false Pick and place = True Trowel left = false } } If X = Number of operations { STOP PROGRAME } }

Appendix

6.2

Robotic Bricklaying

Appendix II: Inverse Kinematic and uploading components

Figure 39 IK solver (Left Square) and uploading component (Right Square) location in the Onix plugin example file

Figure 40 IK solver component parts cluster

Figure 41 Point A calculation

Figure 42 Point B calculation

Appendix

Robotic Bricklaying

Figure 43 Point C calculation

Figure 44 Angle sorting algorithm cluster

Figure 45 Robot uploading component cluster

Appendix

Robotic Bricklaying

6.3 Appendix III : universal robots possible solutions via Onix IK solver

Figure 46 the eight possible solutions for a single tip plane using Onix IK solver

References

Robotic Bricklaying

7 REFERENCES Andres, J. et al., 1994. First Results of the Development of the Masonry Robot System ROCCO: a Fault Tolerant Assembly Tool. Automation and Robotics in Construction XI, 11, pp.87–93. Balaguer, C. & Abderrahim, M., 2008. Trends in Robotics and Automation in Construction. , pp.1–22. Bechthold, M., 2010. The Return of the Future: A Second Go at Robotic Construction. Architectural Design, 80(4), pp.116–121. Braumann J. and S. Brell‐Cokcan., 2013. Rob|Arch 2012: Robotic Fabrication in Architecture, Art and Design, Vienna: Springer Vienna Architecture. Braumann, J. & Brell‐Cokcan, S., 2011. Parametric Robot Control: Integrated CAD/CAM for Architectural Design. In ACADIA. Campbell, J.W.P. & Pryce, W., 2003. Brick: A World History, London, UK: Thames & Hudson. Colletti, M. et al., 2013. Robotic foaming cluster SG2013. Available at: http://smartgeometry.org/index.php?option=com_content&view=article&id=219:rob otic‐foaming&catid=48:sg2012‐london [Accessed April 20, 2013]. Dubor, A. & Diaz, G.‐B., 2012. Magnetic Architecure. In Rob\Arch. p. 7. G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle, J.H., 1994. Application specific realisation of a mobile robot for on‐site construction of masonry. ISARC Proceedings, 11, pp.95 –102. Glynn, R. & Sheil, B., 2011. Fabricate: Making Digital Architecture, Riverside Architectural Press. Gramazio, F. & Kohler, M., 2008. Digital Materiality in Architecture: Gramazio & Kohler, Boden: Lars Muller Publishers. Helm, V. et al., 2012. Mobile robotic fabrication on construction sites: DimRob. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 4335– 4341. Hill, J., 2005. Building the Drawing. Architectural Design, 75(4), pp.13–21. international fedration of robotics, 2012. History of Industrial Robots, From the first installation until today, Frankfurt. Available at: http://www.ifr.org/uploads/media/History_of_Industrial_Robots_online_brochure_b y_IFR_2012.pdf. Keating, S. & Oxman, N., 2013. Compound fabrication: A multi‐functional robotic platform for digital design and fabrication. Robotics and Computer‐Integrated Manufacturing, 29(6), pp.439–448.

References

Robotic Bricklaying

Kolarevic, B., 2005. Architecture in the Digital Age: Design and Manufacturing, Taylor & Francis. Oxford dictionaries, robot: definition of robot in Oxford dictionary (British & World English). Available at: http://oxforddictionaries.com/definition/english/robot [Accessed August 28, 2013]. Piker, D., 2013. Lobster grasshopper group. Available http://www.grasshopper3d.com/group/lobster [Accessed April 1, 2013].

at:

Roulet, O., python‐urx. Available at: https://github.com/oroulet/python‐urx [Accessed August 1, 2013]. Schlegel, A., 2012. ControlP5 plugin. Available http://www.sojamo.de/libraries/controlP5/ [Accessed April 1, 2013].

at:

Spong, M.W., Hutchinson, S. & Vidyasagar, M., 2005. Robot Modeling and Control, John Wiley & Sons. universal robots, The URScript Programming Language. , p.30. Available at: http://www.wmv‐robotics.de/home_htm_files/scriptmanual_en_1.5.pdf [Accessed June 1, 2013].