LOOK INSIDE: Neural Architecture by ORO Editions

#matias del campo

ORO Editions Novato, CA

Acknowledgments 10

Preface, Mario Carpo

Prologue, Matias del Campo

Chapter I – This Building Does Not Exist —An Attempt at a Theory of Neural Architecture What does a Melange have to do with Artificial Intelligence? Familiar but Strange: Bits, Pieces, Features & Neurons With or Without you - Dependency between Concrete and Abstract Objects Neural Networks are Abstract Objects. Properties It’s complicated – about a Relationship About Wild Features and How to Capture Them Things, Facts, and the Ontology of Neural Networks What About Aesthetics, Agency, and Authorship? Aesthetics of Neural Architecture The Sensibility of Neural Architecture Agency in Neural Architecture Neural Architecture is a New Paradigm References

Chapter II – The Robot Garden The Robot Garden How to Test a Robot Posthuman Design is here. Big Data, AI and Architecture Design References

Chapter III – In the Valley of the Hallucinating Machines. Computational Vision as Design Method What Would Turing Do? Architecture’s Empathy Towards Images and Representation How to Recognize a Gothic Column The Nature of Neural Networks Learning Architectural Features Fountains, Figures, and Features – or How to Confuse an AI What it Means to Be a Pixel Machines Hallucinating Architecture References

Contents

Chapter IV – Not a Question of Style—Style, Artificial Intelligence and Architecture A Closer Look into the Suspicious Noun Style When Aesthetics Collide with Technology References

110

Chapter V – Talking Architecture Talking Architecture Materials and Methods Initial Modeling Attentional Generative Adversarial Network The Urban Context of the Design Signs, Scripts, and Codes—A Theory of the Artificial References

130

Chapter VI – How Machines Learn to Plan—A Critical Interrogation of Machine Vision Techniques in Architecture How machines learn to see A Posthuman Trajectory for Plan Formation Estranged - but in a good way Neural Networks and Learning the 2-D Visual World Modeling the Style of the Real World The Defamiliarization of the City, or An Alternative Utopia References

156

Chapter VII – Space: The Final Frontier (How to Wrangle a Neural Network to Deal with 3D Models) Terms of Engagement: Aesthetics, Agency, Sensibility, and Other Nasty Problems Aesthetics Sensibility Agency Authorship Database Construction Neural Optimization Framework Experiments and Results So, Can a Neural Network Learn a Sensibility? References

184

Chapter VIII – The Politics of Neural Architecture and Artificial Intelligence The Problem with Low-Skill Labor The Accelerationist Project The Return of Ornament References

198

Epilogue, Matias del Campo

208

Image Credits

212

Bibliography

214

About the Author

216

Index

Preface

by Mario Carpo

Composuit Zeuxes Iunonem e quinque puellis - Engraving by J. J. von Sandrart after J. von Sandrart. The conflict between realism and idealism in the arts has found different solutions throughout the ages, but from a more practical point of view, the technicalities of Zeuxis's mode of artistic operation—the parsing, selection, and the reassembly of parts coming from many models— have equally invited and prompted a never-ending stream of theories and speculations.

Preface

The ancients did not have theories on what we now call “the fine arts.” Where the moderns formulated lofty aesthetic principles on the modes and functions of mimesis in painting, sculpture, and architecture, the Greeks only had some fancy tales (aka anecdotes, myths, topoi, or parables) telling how some deft craftsmen on some noted occasions managed to make stuff that looked peculiarly natural—i.e., that looked as if made by nature. So we know for example that the famed Zeuxis once painted grapes that were so lifelike that birds came down to peck at them, and Apelles painted a horse so perfect that its image fooled even other horses. My cat, today, won’t be moved by any photographic picture nor even by moving pictures of other cats, let alone of dogs; but back then the archrival of Zeuxis, called Parrhasius, once won first prize in a painters’ contest by drawing a curtain that Zeuxis himself was tricked into attempting to pull aside (on that occasion Zeuxis admitted defeat and came in second). The same Zeuxis also figures in another, quite different story. Having been invited to a town in what would now be southern Italy, to paint a picture of a goddess, in search of inspiration he asked to see some examples of local beauties. The town elders sent him a selected group of handsome young men. Zeuxis protested. He was then allowed to see some girls; but finding none of them quite to his taste, he retained five of them as models. His painting was a fusion, a blend, or an assemblage of features taken from all five, and it was met with great success—

hence the lasting popularity of the anecdote. From the point of view of art theory however, and even of the theory of human knowledge, this seemingly innocent tale conceals a number of major theoretical conundrums. If Zeuxis already had an idea of feminine beauty in his mind, why did he need to imitate any real-life model? And if, on the contrary, he did not have an innate idea of beauty, how could he choose among so many incomplete manifestations of the ideal? The conflict between realism and idealism in the arts has found different solutions throughout the ages, but from a more practical point of view, the technicalities of Zeuxis’s mode of artistic operation–the parsing, selection, and the reassembly of parts coming from many models—have equally invited and prompted a never-ending stream of theories and speculations. Evidently, the artist would not have limited himself in that instance to just cutting and pasting a number of pieces—as in a jigsaw puzzle—he would most likely have had to rework, modify, and adapt some of the parts, not only to chip off the edges but also more generally to make them blend with one another and merge in a single, harmonious composition. Hence the question: how much of Zeuxis’s operation was what we could call today a collage, and how much of it would have been some looser and more creative form of imitation—the work of a talented artist only vaguely and distantly inspired by some of his models, or sources? Could people look at his finished painting and tell: see, these are Emily’s eyes, Peggy’s nose, and Nancy’s lips? Or did he blend all of his

sources in one transfigured, truly supernatural composition, where one would say: see, there is a certain undefinable something in this portrait that reminds me of Emily, and of Peggy, and of Nancy, but it’s hard to tell what, precisely, comes from each? In classical art theory, this is where art would have equaled nature, because this is the way nature works: this is the way a daughter may look like her biological mother. This is also where Zeuxis could have vastly profited from today’s GAN technologies, of the kind that Matias del Campo and Sandra Manninger describe in this book. In fact, hard to say if by chance or by design, GAN technologies today appear to do exactly what Zeuxis—as well as all painters working in the classical mode for the last twenty-five centuries—always strived to do: first, extracting a certain number of common features (or attributes or predicates) from a set of carefully chosen, compatible visual samples (in the case of Zeuxis, the dataset consisted of Zeuxis’s own pick of Sicilian beauties); then, using this list of common features to generate a theoretically unlimited number of copies that will all be similar, to some extent, to each of the models, but identical to none (Zeuxis appeared to have produced just one copy in the story as told, but the principle is the same). Each copy will remind us, somehow, of some or all of the

models, but no one will ever be able to say in what precisely, or why, or where, the models and the copies look alike. The mystery of creative imitation has been at the core of the classical theory of mimesis in the visual arts since the beginning of time. And this is what GAN technologies (i.e., one the latest avatars to date of data-driven machine learning) are now starting to tackle. Just like print and photography, good old mechanical technologies, mastered the art and science of making identical copies, and just like identicality (the making of identical reproductions) was a trope and tenet of modernism in the arts, today’s digital tools are inaugurating a new trend of mass-produced similarities: the making of copies that are never the same, but all have a certain something in common—thus they resemble one another and their models. In short, today’s AI is about to automate imitation. And in true AI fashion, it is doing so without telling us how that happens: creative imitation remains a mystery even when carried out inside the black box of a recursive algorithm. Which common features are being extracted from the dataset we feed to the system? And how are these common traits embedded in the new images we can derive from the original models?

Preface

In other terms, what do the items in the original dataset, and the newly created ones, have in common? Why do they look similar? Nobody knows. In that, we are not an iota more advanced than any of our classical predecessors. When dealing with what we would now call highly structured datasets, classical artists soon found another, equally generic way to summarize the loose ambit of visual resemblances among works of art that have something—a certain undefinable something—in common: since Vasari, it was understood that works made in similar ways (showing the “hand” of the same master, for example, or the distinctive traits of the same school of painting) might be said to be in the same “manner.” In the course of the nineteenth century, the term “style” was largely adopted with a similar meaning. In the classical tradition, all art was to some extent imitative; all artists had to learn the art of copying; some ineffable similarity between the model and the copy was the sign of art well done; artists that produced creative copies in equally ineffably similar ways were said to be working in the same style. That was then. The theory and practice of creative imitation, and the related critical category of style, were equally obliterated by the modernist

leviathan. Seen from the vantage point of lateromantic sensibility, all copy is plagiarism; from the point of view of modernist morality, all imitation is sinful; from the point of view of twentieth-century iconoclasm, all discussion of style is wasteful. Did we ever stop copying? Of course not, because that’s to some extent inevitable in all we do. Did we ever stop noticing that some artworks happen to be in the same style as some others? Of course not—we just found other, more opaque, hypocritical and labyrinthine ways to say so. Matias del Campo and Sandra Manninger’s work, as documented in this book, powerfully emphasizes to what extent the use of AI today obliges us to reassess some aspect of our natural intelligence that twentieth-century industrial modernism had made us forget. It was time. Imitation and style are back to where they should be—not as judgments of value, but as indispensable creative and interpretive tools. The work of Matias del Campo and Sandra Manninger also powerfully demonstrates to what extent our post-industrial, computational future is bound to be closer in spirit to our pre-industrial past than to our late industrial present. Given the deliquescent state of our late industrial present, that gives us more than a glimmer of hope. Mario Carpo

Prologue

Artificial Intelligence (AI) is a general term used to describe several varying approaches. In computer science, artificial intelligence is defined as the study and development of intelligent agents, which includes any device that perceives its environment and takes actions to maximize its chance of successfully achieving its goals. In general, the term artificial intelligence is applied when a machine mimics the cognitive functions that humans associate with other humans, such as learning and problem-solving. The general trajectory of this line of inquiry is preoccupied with aspects of optimization, such as ideas for optimizing floorplans, material consumption, and time schedules of construction sites, which cover the tame problems of disciplinary considerations. The other end of the spectrum is the inquiry into the wicked problems of designing architecture: creativity, intuition, and talent. This opens up questions about the nature of the creativity1 that an AI can possess, its role in the inception of architectural projects, the methods to evaluate this, and the nature of creativity at large. Can an AI create a novel sensibility? If so, can we, as humans, perceive, and understand it? The architecture project within this frame of consideration not only tackles the problem from the aesthetic point of view—the idea that AI can creatively generate a sensibility—but also from a series of profoundly ethical points of view. For example, there is the question of whether an AI can develop a sense of agency? Does that frame of thinking materialize in some way? Do robots dream of perfect cathedrals?

Posing questions about authorship, the nature of ingenuity, imagination, and creativity, this book discusses a posthuman2 world operating within a frame of considerations that include the role of a human actor in an AI-assisted design universe. How do we fit in a world that debates the alien ontologies that these new technologies produce? When suddenly architecture projects emerge from gargantuan amounts of data crushed through GPU’s running, sophisticated deep-learning processes? Surpassing the ability of humans to process data in unprecedented numbers. On the other end is the human ability to extrapolate possible new design methods out of these processes, the ability to find inspiration in mistakes, and the nuanced sensory to infuse space with additional meaning than the bare materiality would suggest. Questions on the nature of architecture and AI is gaining enormous momentum, as public interest in the methodology of using AI as a design tool is growing steadily. It is the right moment for a discussion about the impact of AI on the world of architecture. Some of the most surprising aspects emerge when these techniques learn from inherently different architectural styles, such as using datasets of Baroque architecture in combination with modern architecture, resulting in weird mash-ups that challenge disciplinary positions and provoke novel architectural trajectories. Familiar, but different; comprehensive, but exotic; alien, but beautiful.

And The Beat Goes On – Siri, Nest, & Alexa The year is 2021, and the first digital turn in architecture has come full circle. The use of computers in the office is commonplace. Within a generation, the design activity of the architectural discipline changed from an occupation centered around manual labor— ink pen, triangle, and Letraset—to a primarily digital activity—mouse, screen, and software. Mario Carpo, architecture historian at The Bartlett UCL smugly described it in his book The Digital Turn as follows: Building a multi-story car park these days typically involves more digital technologies than were available to Frank Gehry’s office for the design of the Guggenheim Bilbao in the early 1990s. 3 After this first phase of consolidating research and application, a new player is currently entering the field: artificial intelligence. How does architecture take part in this conversation? How will this novel protagonist participate within the plateau of architectural discourse and practice? How are habits of the built environment changed by the fact that intelligent machines capable of learning are becoming part of the process of construction, maintenance, and everyday life? Based on the idea that architecture experienced its first digital turn between 1992 and 2012, the follow-up inquiry pertains to

the cultural meaning of these novel toolsets. It almost seems that through miniaturization and cloud services, large chunks of technology start to dissipate from the visual focus, the center of the home, and move into the periphery of the household. For architecture, questions emerge regarding its phenomenological qualities. How can AI contribute to the experience of architecture in this environment? Is it part of the materialization of architecture? The ability to learn neural networks allows large amounts of data to be processed in a short time. This property of using big data as a tool is already being used on a massive scale today. The online Google software Deep Dream allows, for example, photos and their inherent stylistic qualities to be combined. Astonishingly, the application of an artificial intelligence algorithm is triggering a renaissance of the question of style. Considering the longoverdue unshelving of the problem of style in architecture, it seems only natural that the current AI research provides prompts for the critical interrogation of aspects of style (e.g., Style Transfer, StyleGAN). This contemporary conversation can contribute profoundly to a novel understanding of style as an expression of our current age. But is AI a style? I would argue that it is not—at least not an architectural style. Too much AI is already involved in our everyday lives.

Prologue

It’s All the Fault of Lady Lovelace 19

The intimate relationship between computational processes and human imagination emerged right there, at the very beginning, when the first programmer and the first machine converged. In 1842, the ink on Lady Lovelace’s seminal notes on Turinese mathematician and scholar Luigi Menabrea’s sketch on the Analytical Engine 4 was not even dry yet when she proposed that the machine could also be used to compose music.5 This marriage of computational tools and cultural exploration has not lost its edge to this very day. On the contrary, the emergence of artificial neural networks has infused new life into the conversation around the culture of machines capable of synthetic reason.6 Before this potential area of cultural inquiry could be explored, a technological framework had to be created. As in so many cases, necessity was the mother of invention—and in this case, the necessity was war. In case the Cold War turned hot, scientists were tasked with exploring the potential of neural networks to perform feature recognition—ultimately, the goal was to manage an entire array of cognitive tasks. The translation of basic neural patterns allowed us to explore tasks such as solving mathematical problems, rudimentary translation, text creation, and feature recognition in images. These early experiments grew into what we see today: artificial intelligence is deeply embedded in our everyday life. Neural networks help diagnose medical cases, make decisions on bank loans, decide parole cases, read your bank checks, filter job applications, drive cars, fly drones, detect fraud, translate text online, and

recommend to you which book to read next. They are part of our households in the form of quasi-intelligent thermostats and security systems, help us find our way in our cars by using adaptive maps, and are close to the human body in the shape of smartphones and watches—monitoring your moving patterns by day, your breathing patterns by night, and your heartbeat all day long—to estimate your health status (health insurance benefits, anyone?). They recognize your face and how you type and help you photograph like a pro, among many other things. The data generated by all these activities aid in per fecting the training of all these algorithms. In other words, AI is everywhere. The result of this application is an architecture that is both familiar and strange, as well as a design language that is partially accessible and alienating at the same time. Perhaps this is the first genuinely new architecture of the 21st century. It’s hardly a question of whether artificial intelligence changes our social behavior. It’s a question of how. Does this also change architecture? How does the use of artificial intelligence affect social (residential) construction? Is it used purely to optimize residential construction, or does it influence social behavior, social norms, and architectural design? In this book, you will find a series of attempts to explore some of the ideas, possibilities, and problems that arise at the intersection of artificial intelligence and architecture.

Prologue

References

1. Margaret A. Boden, “Creativity and Artificial Intelligence,” Artificial Intelligence 103, no. 1–2 (1998): pp. 347–356. 2. Francesca Ferrando, “Towards a Posthumanist Methodology: A Statement,” Frame, Journal of Literary Studies 25 (2012): pp. 9–18. 3. Mario Carpo, “The Digital Turn in Architecture 1992-2012,” AD Reader, John Wiley & Sons, West Sussex, (UK, 2013): pp. 8–14 4. Luigi Federico Menabrea, “Sketch of the Analytical Engine Invented by Charles Babbage ... with Notes by the Translator [Augusta Ada King, Countess of Lovelace],” in Scientific Memoirs III, ed. Richard and John E. Taylor (London, 1843). 5. Betty A. Toole, Ada, The Enchantress of Numbers (Sausalito, California: Strawberry Press, 1992). 6. Manuel DeLanda, Philosophy and Simulation - The Emergence of Synthetic Reason (Bloomsbury Academic, London, 2015).

◀ In July 1958, Frank Rosenblat t demonstrated

the “Perceptron” to the U.S. Office of Naval Research. The IBM 704, a behemoth of a machine weighing five tons and filling an entire room, read a set of punch cards that were explicitly marked on the lef t or the right side. Af ter 50 trials, the perceptron had learned to differentiate successfully between these two types of cards. This event represents the origin of the neural networks based on connectionism that we see today.

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

chapter I

This Building Does Not Exist An Attempt at a Theory of Neural Architecture

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

214

About the Author

215

Matias del Campo. PhD is a registered architect, designer, and educator. He is an Associate Professor at Taubman College for Architecture and Urban Planning (University of Michigan) where he serves as director of the AR2IL—The Architecture and Artificial Intelligence Laboratory. He is co-founder of the architecture practice SPAN together with Sandra Manninger, PHD Computational methodologies and philo-sophical inquiry inform their architectural designs. SPAN gained wide recognition for the award-winning Austrian Pavilion at the 2010 Shanghai World Expo and the new Brancusi Museum in Paris. He was awarded the Accelerate@CERN fellowship, the AIA Studio Prize, and elected to the board of directors of ACADIA. SPAN’s work is in the permanent collection of the FRAC, the MAK, the Benetton Collection, the Albertina, and several private collections.