Neural Architecture by ACC Art Books

Acknowledgments 10

Preface, Mario Carpo

Prologue, Matias del Campo

Chapter I – This Building Does Not Exist —An Attempt at a Theory of Neural Architecture What does a Melange have to do with Artificial Intelligence? Familiar but Strange: Bits, Pieces, Features & Neurons With or Without you - Dependency between Concrete and Abstract Objects Neural Networks are Abstract Objects. Properties It’s complicated – about a Relationship About Wild Features and How to Capture Them Things, Facts, and the Ontology of Neural Networks What About Aesthetics, Agency, and Authorship? Aesthetics of Neural Architecture The Sensibility of Neural Architecture Agency in Neural Architecture Neural Architecture is a New Paradigm References

Chapter II – The Robot Garden The Robot Garden How to Test a Robot Posthuman Design is here. Big Data, AI and Architecture Design References

Chapter III – In the Valley of the Hallucinating Machines. Computational Vision as Design Method What Would Turing Do? Architecture’s Empathy Towards Images and Representation How to Recognize a Gothic Column The Nature of Neural Networks Learning Architectural Features Fountains, Figures, and Features – or How to Confuse an AI What it Means to Be a Pixel Machines Hallucinating Architecture References

AR+D

Contents

Chapter IV – Not a Question of Style—Style, Artificial Intelligence and Architecture A Closer Look into the Suspicious Noun Style When Aesthetics Collide with Technology References

110

Chapter V – Talking Architecture Talking Architecture Materials and Methods Initial Modeling Attentional Generative Adversarial Network The Urban Context of the Design Signs, Scripts, and Codes—A Theory of the Artificial References

130

Chapter VI – How Machines Learn to Plan—A Critical Interrogation of Machine Vision Techniques in Architecture How machines learn to see A Posthuman Trajectory for Plan Formation Estranged - but in a good way Neural Networks and Learning the 2-D Visual World Modeling the Style of the Real World The Defamiliarization of the City, or An Alternative Utopia References

156

Chapter VII – Space: The Final Frontier (How to Wrangle a Neural Network to Deal with 3D Models) Terms of Engagement: Aesthetics, Agency, Sensibility, and Other Nasty Problems Aesthetics Sensibility Agency Authorship Database Construction Neural Optimization Framework Experiments and Results So, Can a Neural Network Learn a Sensibility? References

AR+D

Composuit Zeuxes Iunonem e quinque puellis - Engraving by J. J. von Sandrart after J. von Sandrart. The conflict between realism and idealism in the arts has found different solutions throughout the ages, but from a more practical point of view, the technicalities of Zeuxis's mode of artistic operation—the parsing, selection, and the reassembly of parts coming from many models— have equally invited and prompted a never-ending stream of theories and speculations.

AR+D

Preface

The ancients did not have theories on what we now call “the fine arts.” Where the moderns formulated lofty aesthetic principles on the modes and functions of mimesis in painting, sculpture, and architecture, the Greeks only had some fancy tales (aka anecdotes, myths, topoi, or parables) telling how some deft craftsmen on some noted occasions managed to make stuff that looked peculiarly natural—i.e., that looked as if made by nature. So we know for example that the famed Zeuxis once painted grapes that were so lifelike that birds came down to peck at them, and Apelles painted a horse so perfect that its image fooled even other horses. My cat, today, won’t be moved by any photographic picture nor even by moving pictures of other cats, let alone of dogs; but back then the archrival of Zeuxis, called Parrhasius, once won first prize in a painters’ contest by drawing a curtain that Zeuxis himself was tricked into attempting to pull aside (on that occasion Zeuxis admitted defeat and came in second). The same Zeuxis also figures in another, quite different story. Having been invited to a town in what would now be southern Italy, to paint a picture of a goddess, in search of inspiration he asked to see some examples of local beauties. The town elders sent him a selected group of handsome young men. Zeuxis protested. He was then allowed to see some girls; but finding none of them quite to his taste, he retained five of them as models. His painting was a fusion, a blend, or an assemblage of features taken from all five, and it was met with great success—

hence the lasting popularity of the anecdote. From the point of view of art theory however, and even of the theory of human knowledge, this seemingly innocent tale conceals a number of major theoretical conundrums. If Zeuxis already had an idea of feminine beauty in his mind, why did he need to imitate any real-life model? And if, on the contrary, he did not have an innate idea of beauty, how could he choose among so many incomplete manifestations of the ideal? The conflict between realism and idealism in the arts has found different solutions throughout the ages, but from a more practical point of view, the technicalities of Zeuxis’s mode of artistic operation–the parsing, selection, and the reassembly of parts coming from many models—have equally invited and prompted a never-ending stream of theories and speculations. Evidently, the artist would not have limited himself in that instance to just cutting and pasting a number of pieces—as in a jigsaw puzzle—he would most likely have had to rework, modify, and adapt some of the parts, not only to chip off the edges but also more generally to make them blend with one another and merge in a single, harmonious composition. Hence the question: how much of Zeuxis’s operation was what we could call today a collage, and how much of it would have been some looser and more creative form of imitation—the work of a talented artist only vaguely and distantly inspired by some of his models, or sources? Could people look at his finished painting and tell: see, these are Emily’s eyes, Peggy’s nose, and Nancy’s lips? Or did he blend all of his

AR+D

The people in this grid of faces do not exist. They are an example of the ability of Generative Adversarial Networks to create realistic images based on large datasets of faces.

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

This Building Does Not Exist An Attempt at a Theory of Neural Architecture What does a Melange have to do with Artificial Intelligence? 25

Vienna is home to one of the world’s oldest clusters of research on artificial intelligence: the Austrian Institute of Artificial Intelligence (OFAI). The OFAI is an offspring of the Österreichische Studiengesell schaf f t für Kybernetik, a registered scientific society founded in 1969. About 30 years after its founding, in the summer of 1998, Sandra Manninger and I were sitting around in a Schanigarten, a typical Viennese coffee house guest garden, in the court of the Baroque ensemble that houses the OFAI in the heart of Vienna. We were discussing the possibilities of integrating artificial intelligence into an architectural design with Professor Robert Trappl, the director of the OFAI, and Dr. Arthur Flexer. Of course, this was a purely hypothetical conversation since neural networks based on computational processes were in their infancy—while sipping Melange coffees, they told us with excitement about a recent success: the simulation of one neuron to another neuron interaction. A few years later, in 2006, SPAN, in collaboration with the OFAI, conducted the first machine learning workshop at the Angewandte in Vienna. After a move to the University of Michigan in 2014, a collaboration with Michigan Robotics and Computer Science paved the way for a series of new design techniques. In particular, the director of Michigan Robotics, Jessy Grizzle, and PhD student Alexandra Carlson have been crucial to this collaboration. Neural architecture is the field of architecture that is primarily preoccupied with interrogating the emergent field of Artificial Neural Networks (ANNs) as a method of designing architecture. ANNs can be described, in short, as a sequence of mathematical algorithms that are capable of

registering latent correlations in a set of data. The applied algorithms mimic certain aspects of how the human brain operates. This position, however, is still highly disputed. For the sake of this book, I would suggest agreeing that the algorithms at work in Neural Networks (NNs) were inspired by the current knowledge about certain processes in the human brain, such as hallucinating1 and dreaming.2 In this sense, neural networks can be described as a system of neurons that can be either organic or artificial. 2015 was a turning point in the application of techniques derived from AI research to the arts. That year saw the introduction of the Generative Adversarial Network (GAN) by Ian Goodfellow, 3 as well as the publication of the paper A Neural Algorithm of Artistic Style, by Leon Gatys.4 In recent years, these novel methods have taken hold in the arts5 and music.6 The newly emerging art form is fittingly named neural art7. The source of this term can be found in the title of a paper by Gatys, 8 which forms the base for the work of several of the most prolific neural artists, such as Mario Klingemann (who describes himself as a neurographer) and Sofia Crespo, whose series of works, Neural Zoo, reflects a keen interest in the estrangement and defamiliarization of deep-sea creatures. There are many more artists in this genre.9 How is the work of these artists related to architecture? Maybe an example will help clarify how provocative this novel method is for architecture. Mario Klingemann uses databases of Western art, particularly portraits, as the base for his StyleGAN10 applications. Thousands of images from the Renaissance to the 19th century were fed through a StyleGAN algorithm. In a conventional case, StyleGAN would be used

AR+D

to create images that convincingly represent a known object to the observer. Most famously demonstrated with examples like This Person Does Not Exist, a website that generates highly convincing images of persons based on a large dataset of portraits. It might be important here to understand that neural networks are basically function approximation algorithms;11 they will always strive to achieve an approximation of, for example, the number one. Function approximation can also be described by a curve, or as computer scientist and philosopher Judea Pearl pointed out: “Machine Learning is just glorified ‘Curve Fitting.’”12 As much as this approach can produce convincing images of objects, it is the area outside the perfect

Based on a large image dataset of Gothic churches, this GAN identified common features in the image set and interpolated them in order to create variations of the given image set. The adversarial network, in this case, was trained intentionally sloppy to avoid completely realistic images of Gothic churches, but rather to emphasize the possibility of an architecture that is strange but familiar at the same time.

fit of the curve that really produces the more interesting results in that they maintain a certain familiarity, despite their alien appearance. Back to the work of Mario Klingemann: His images and animations maintain elements of the dataset informing the StyleGAN. This results in images that show contorted bodies and distorted faces that have a surreal quality to them—bizarre Janus heads with multiple faces, strange cyclops, and monstrous chimeras between humans and animals. The trained eye will still recognize features of historic paintings and drawings. A bit of Francisco Goya here, some James Whistler there, glimpses of Jeanne-Etienne Liotard, Eduard Magnus, Franz von Lenbach, and Franx Xaver Winterhalter.

As in the example of the Gothic churches, Mario Klingemann approaches the application of GAN’s in his work Memories of Passersby I by avoiding the ability of GAN’s to create realistic variations of classic portraits. He instead emphasizes the estrangement provoked by avoiding perfect curve fitting in the algorithm.

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

But none of the images are designed to approximate the particular artists, as This Person Does Not Exist would do. Rather, a StyleGAN renders the features that were recognized by the neural network and re-combines the pixels into a new image outside the conventions through which we, as humans, understand the depicted object. In doing so, the emergent pieces of art provoke questions of authorship and agency. In addition, the question is raised of the value of a sensibility that was created somewhere between human input and machinic output. Is this the art of the posthuman age? To illustrate an example in the field of architecture, we collected Gothic architecture images and used them as a dataset

for a StyleGAN. Ironically, using the ever-sopopular scraping method to collect “Goth” images results in, let’s say, interesting results. Does this development take into consideration the possibility of evaluating the role of humans in a world where the boundaries between human and non-human creativity are blurred? In architecture, we can observe a similar tendency, with architects increasingly picking up on novel techniques in machine learning and machine vision. I would describe this new tendency in architecture as neural architecture, borrowing from computer science and neuroscience as much as from the language used in the arts and music (neural art, neural music).

This dataset with several thousand images of Baroque plans forms the base for a process that reveals how a neural style transfer imprints the learned features from the Baroque plans to a modern plan.

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

AR+D

III

In the Valley of the Hallucinating Machines—Computational Vision as Design Method

design. Or they could be used in projects of cultural preservation, in that they dream of how to complete the restoration of historic buildings. Or they can be used to optimize the planning of housing projects by learning to compare thousands and thousands of housing plans.19 The opportunities are remarkable and possibly will generate a completely new paradigm as to how to approach architecture design. In terms of disciplinary implementation, the project contributes to the discussion of style in the 21st century. Borrowing from the conversations of Gottfried Semper20 about the nature of style, it can be stated that style has turned into a posthuman quality, where artificial players contribute to the discussion by analyzing and proposing ideas for a cultural discussion with an ever-increasing speed. If robots can dream of Gothic cathedrals, humans need to renegotiate their position in a contemporary design ecology.

DeepDreaming weird animals onto the Robot Garden. The DeepDreaming algorithm can only make sense of the world based on the dataset that it possesses. In a feed-forward process, this concept would be used to train a network to understand features around it - a crucial tool in machine vision for cars, for example. In Deepdreaming, the flow of information is reversed; thus, the tool is not used analy tically but generatively. This results in weird images full of strange creatures as the algorithm tries to make sense of the world it is seeing.

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

AR+D

Current top view of the Robot Garden, with the current version of the features dreamed on the site. Minor things were implemented manually, such as the position of the poles holding sensors for tests with robots and a plat form (lef t of image) to hold a control desk.

136

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

137

AR+D

Machines can do things that resemble plans. Edge detection, corner detection, and detecting pixel blobs allow neural networks to create something that a human can identify as a plan. However, it is missing any inherent semantic information as long as the data is not labeled correctly. This image was one of the first attempts to train a neural network to generate plans. In order to be able to do this, a dataset with several thousand plans has to be created first.

148

Style transfer evokes memories of the discussion on style in architecture. In this image, you can see a stylistic mash-up between Baroque features (represented by a Collegio di Propaganda Fide window designed by Francesco Borromini in 1646) and the features of several of SPAN’s 3D pattern renderings generated for various projects. It is indeed amusing that the term style returns into conversations about architecture and planning via neuroscience and computer sciences as if it comes back to haunt the discipline and reminds them of the importance of its tradition.

AR+D

How Machines Learn to Plan - A Critical Interrogation of Machine Vision Techniques in Architecture

The Defamiliarization of the City, or An Alternative Utopia 149

In architecture discourse, the line, the plan, and the abstract representation of materiality have played a major role, and they have always been interpreted as the result of human cognition. This can be illustrated as a core idea in the architectural theory of, for example, Leon Battista Alberti, as expressed in the De re Aedificatoria, pertaining to the distinction between “lineament,” the line in the mind of the architect, and “matter,” the material presence of the building. 22 This particular distinction plays a key role in architectural design and the conceptualization of the architectural project throughout the history of western architecture. Le Corbusier described this at the heyday of modernism in the 20th century as follows: “Architecture is a product of the mind.”23 The distinction between mind and matter can be found in Vitruvius, in the distinction between “that which signifies and that which is signified”; at the Accademia di San Luca in Rome, between disegno interno and disegno esterno; or in Peter Eisenman’s distinction between deep aspect and surface aspect in architecture, to name just three examples24 that profoundly describe the planning process as a particular ability of the human mind. What position does the discipline take when it comes to understanding the potentialities of applications, such as neural networks, that are able to produce results that question the sole authorship of human ingenuity? Well, therein lies the chicken or the egg conundrum the neural network’s origin in the human mind. That they are able to autonomously generate plan solutions is in itself not yet proof of thinking or even intelligence. However, if we take the philosophical standpoint of materialism, it would allow creating an even field between these two thinking processes. In a materialist tradition, though, thinking is just the result of material processes in our brain, neurochemical reactions able to form thought.

If this position is taken, then the conclusion is that AIs can think as much, and form original language or shape as humans can, the only difference being that their neural processes are not based on neurochemical processes, but computational processes within another material paradigm. In this book, we present the possibility of utilizing AI applications for the generation of planning processes. In particular the application of style transfers with neural networks. This approach, on the one hand, critically interrogates the unique position of the human mind when it comes to creative processes and in addition questions aspects of creativity in planning processes. In a design ecology where the boundaries between human and computational cognition are increasingly blurred, the presented process harvests the multiplicious solutions found by architects throughout the ages and employs mining big data to create possible novel solutions to planning problems. From this outlook it can be stated that this is only a first attempt in the area of the critical interrogation of planning in architecture in the age of AI. In fact, there is still much to be done. The first alien results achieved in this paper can only be seen as a first tapping into the potentialities of this approach, from a novel design direction that rather talks about how machines see our world—with all its wonderfully strange results in terms of morphologies, chromatics, and possible theories—to profoundly pragmatic approaches.

AR+D

Going back to the initial question of whether machines can learn to plan, we can state that machines can certainly model and/or recognize styles, but a planning process needs far more semantic information in order to successfully fulfill the task. This means that the recognition

150

AR+D

This Building Does Not Exist—An Attempt at a Theory of Neural Architecture

151

AR+D