Research about Architectural Video Mapping by Ahmad Ehab

OUTDOOR LIGHTING 2

VIDEO MAPPING Group Research

Submitted By Abdelrahman Hisham Ahmad Ehab Hend Ihab Iman Ahmed Karim Tarek Samy Mahmoud Nour El-Din

Submitted to Dr. Ashraf Nessim Eng Ahmed Sayed Eng Lamis Mohamed Cairo, 2018

Table of Contents Table of Contents .........................................................................................................................................2 1 Abstract ...................................................................................................................................................7 2 Introduction .............................................................................................................................................7 3 What is Video Mapping? ........................................................................................................................8 Idea ..............................................................................................................................................8 HISTORY ....................................................................................................................................9 Key Players................................................................................................................................10 Possibilities ................................................................................................................................11 Technical ...................................................................................................................................12 Production Chain .......................................................................................................................14 Concerns ....................................................................................................................................15 4 Projector Attributes ...............................................................................................................................16 5 Techniques used for Video Mapping ....................................................................................................16 Laser shows ...............................................................................................................................16 5.1.1 Laser beam categories .....................................................................................................17 Laser graphics and effects .........................................................................................................18 Laser mapping and architectural laser lighting .........................................................................20 Interactive laser shows ..............................................................................................................20 Laser billboards .........................................................................................................................20 6 Methods of projections..........................................................................................................................21 Geometric Warping ...................................................................................................................23 Multifocal Projection .................................................................................................................24 Box â&#x20AC;&#x201C; Bot & Dolly ....................................................................................................................25 Set-up â&#x20AC;&#x201C; Projector & Table ........................................................................................................26 Shoot Footage ............................................................................................................................26 7 Projector attributes ................................................................................................................................26 8 Recent advancement in projection mapping .........................................................................................27 Projection mapping in a brief: ...................................................................................................27 2

8.1.1 How these calibrations occur? ........................................................................................27 Main factors affecting advanced video mapping production: ...................................................28 8.2.1 Algorithms: .....................................................................................................................28 8.2.2 Hardware: ........................................................................................................................30 9 Video mapping applications..................................................................................................................32 10 Case study: Digital video expression using Projection Mapping techniques at Mojiko station ...........33 Abstract .....................................................................................................................................33 Research background ................................................................................................................33 10.2.1 Purpose of research .........................................................................................................33 10.2.2 Summary of the projection object ...................................................................................33 Projection Mapping ...................................................................................................................34 10.3.1 Projection Mapping cases ...............................................................................................35 Work Production .......................................................................................................................35 10.4.1 Production process ..........................................................................................................36 10.4.2 Contents .....................................................................................................................37 10.4.3 Correction process of projection target ...........................................................................38 10.4.4 Video production ............................................................................................................38 10.4.5 Structure of the System ...................................................................................................38 Experiment ................................................................................................................................39 Performance...............................................................................................................................40 Conclusion .................................................................................................................................40 11 Case Study: Dynamic Urban Projection Mapping ................................................................................41 Abstract .....................................................................................................................................41 Concept and objectives ..............................................................................................................41 Vehicle Flows and Light Hours in the Selected Area ...............................................................42 Execution ...................................................................................................................................45 Conclusions ...............................................................................................................................49 12 Case Study: Urbanscreen, The Hamburger Kunsthalle Museum Show ...............................................50 Background and location: ..........................................................................................................50 Analysing of the location: .........................................................................................................50 Lighting design concept ............................................................................................................51 The Performance and its mechanism .........................................................................................51 The production chain of the performance: ................................................................................52 3

Fixtures and hardware: ..............................................................................................................52 The software by MXWendler: ...................................................................................................53 Conclusion: ................................................................................................................................53 13 Case Study: Yekpare (Monolithic) .......................................................................................................54 Lighting objective......................................................................................................................54 The main lighting scenes: ..........................................................................................................54 Lighting Fixtures Used ..............................................................................................................57 Lighting Techniques Used.........................................................................................................57 Conclusion .................................................................................................................................58 14 Case Study: Zaha Hadid Architects casts Digital Light Mapping Projection on a Baroque Castle Façade ...................................................................................................................................................59 Background. ..............................................................................................................................59 Details ........................................................................................................................................59 15 Case Study: IMAPP 2016 – INTERCONNECTION ...........................................................................61 About the Building ....................................................................................................................61 iMapp Bucharest........................................................................................................................62 Interconnection ..........................................................................................................................62 Technical information ...............................................................................................................63 16 References .............................................................................................................................................64

List of Figures and Tables Figure 1: laser beams and effects .............................................................................17 Figure 2: Laser fans. sheets and tunnels ..................................................................17 Figure 3: Liquid sky laser effects.............................................................................17 Figure 4 :Audience scanning laser effects ...............................................................18 Figure 5: Laser graphics and images .......................................................................18 Figure 6: Laser logos and projections ......................................................................18 Figure 7: Laser abstracts ..........................................................................................19 Figure 8: 3D Laser animations .................................................................................19 Figure 9: Laser mapping effects ..............................................................................20 Figure 10 Sidney Laser Billboard ............................................................................20 Figure 11 This Projection Mapping at the Ralph Lauren building in London ........21 Figure 12: Ralph Lauren. .........................................................................................21 Figure 13 <<Living Room>> Projection Mapping..................................................21 Figure 14: geometric calibration ..............................................................................27 Figure 15: photometric calibration...........................................................................28 Figure 16: geometric calibration ..............................................................................29 Figure 17 ( a full detailed human face captured by several projectors moving around it) ..................................................................................................................29 Figure 18 ( a picture taken by different photometric compensation) ......................30 Figure 20 ..............................................................................................................30 Figure 19 ..................................................................................................................30 Figure 21 ..................................................................................................................31 Figure 22 ..................................................................................................................31 Figure 23 (a 3D image caused by light field projector and angle expending screen) ..................................................................................................................................31 Figure 24 (different display methods) .....................................................................32 Figure 25 ( figure shows the different between multi spectral display and a conventional LCD display) ......................................................................................32 Figure 26 .Dynamic urban projection mapping framework ....................................41 Figure 27. Analysis of the site according to Lynch studies .....................................42 Figure 28. Perceptual influence of the “landmark” according with Cullen’s studies ..................................................................................................................................43 Figure 29. Traffic flow tool in the area of interest available through Google Maps application ................................................................................................................44 Figure 30. Traffic peak hours in the area of interest................................................44 5

Figure 31. Front façade on GRA outer carriageway................................................45 Figure 32. Back façade on GRA inner carriageway ................................................46 Figure 33. System management scheme ..................................................................47 Figure 34. Application interface ..............................................................................47 Figure 35. Project plan .............................................................................................48 Figure 36. Dynamic urban projection mapping application ....................................49 Figure 37( a google earth picture of Hamburg Kunsthalle) .....................................50 Figure 38 (Galerie der Gegenwart (Kindl, 2016)) ...................................................51 Figure 39 (Different scenes from the kubik show (Kindl, 2016)) ...........................51 Figure 40 (production chain of the VM show (Kindl, 2016)) .................................52 Figure 41 (Sanyo XT35 projectors (Know-How, 2018)) ........................................52 Figure 42( MXWendler logo (online, 2011)) ..........................................................53 Figure 43the light installation takes shape on the 170-meter-long façade of the karlsruhe’s baroque castle ........................................................................................59 Figure 44 Zaha hadid architects worked closely with digital artists, computer science researchers, and musicians ..........................................................................59 Figure 45spaces show the circulation and congregation of digital actors ...............60 Figure 46 Dynamic virtual spaces use real-time data to interprets the virtual characters’ interactions ............................................................................................60

1 Abstract 3D architectural mapping is a video projection technique that can be done with a survey of a chosen building in order to realize a perfect correspondence between its shapes and the images in projection. As a performative kind of audiovisual artifact, the real event of the 3D mapping is a combination of a registered video animation file with a real architecture. This new kind of visual art is becoming very popular and its big audience success testifies new expressive chances in the field of urban design. In this report, we will talk about video mapping and discuss the techniques used their effects on the architecture industry.

2 Introduction This report introduces 3D architectural mapping in a Visual Studies perspective; as a new kind of art form, and as a product of an historical path. In the realm of digital technologies for performing arts and humanities, this report focuses on the description of a video projection technique. As a performative kind of audiovisual artifact, the real event of the 3D mapping is a combination of a registered video animation file with a real architecture. This new kind of visual art is becoming very popular and its big audience success testifies new expressive chances in the field of urban design.

3 What is Video Mapping? Projection mapping has had several different names over the years as it was being developed and experimented with. Some of you may know it as “spatial augmented reality” or “video mapping”. Whatever you know it as, it’s the use of projectors to shine onto a 3 D surface opposed to a typical flat rectangle that’s often seen in classrooms and conference rooms. When a projector shines a mirror image of your computer display for example, it’s typical ly shining on a standard projector surface. While this is cool tech, and has definitely revolutionized the way that we teach and present ideas, it’s become so standard that it wouldn’t really be a mind blowing sight no matter where you see it. A decent amo unt of people even has these installed in their homes instead of using a typical TV. By this level of tech becoming so standardized, it’s only normal that it’s being taken to the next level in commercial and creative situations with projection mapping.

Idea The term “Projection Mapping” (PM) is relatively new. The older, academic name is “Spatial Augmented Reality” or “Video Mapping”, but all terms have the same meaning. Light is mapped onto any complex surface to create a 3D display out of everyday objects, as illustrated in figure 1 (Jones, 2012).

figure 1 – the idea of projection mapping (Jones, 2012) PM requires one or more controllable light sources, a surface and a viewer. The idea is to control the emitted light of an object’s surface. Therefore, the surface properties such as texture and color can be changed by time. By these changes the illusion of motion can be created (Oliver & Ramesh, 2005). Because everyday objects have a more complex shape and radiosity than classic projector screens, the projected light has to be adapted in a few ways. Most of the time projectors are used as light sources and are controlled with computers. The computer is responsible for content delivery and image adaption. For a small setup everyday video projectors and computers can be used. For bigger installations on buildings for example highperformance projectors and computers are needed (Hronek & Dieni, 2011). 8

In 1969 light was projected the first time on non-planar surfaces, with the intention to create an illusion where the real and the virtual objects merge together.

HISTORY The first known instance of projection onto a non-flat surface dates to the 1969 opening of the Haunted Mansion ride in Disneyland. The dark ride featured a number of interesting optical illusions, including a disembodied head, Madame Leota, and 5 singing busts, the ‘Grim Grinning Ghosts‘, singing the theme song of the ride. These were accomplished by filming head-shots of the singers (with 16 mm film) and then projecting this film onto busts of their faces. The next projection mapping instance comes in 1980, with the immersive film installation Displacements by Michael Naimark. In this art installation a living room with two performers were filmed with a rotating camera, then the camera was replaced with a projector. The result is rotating projection mapping. Creating illusions as if the people interacting with the objects were really there. In 1991, Disney not only pioneered the technology of projection mapping, they also have the earliest patent (that I can find) in the space. Entitled “Apparatus and method for projection upon a threedimensional object“. It essentially describes a system for digitally painting an image onto “a countoured, threedimensional object”. GE also has an early patent for a “A system and method for precisely superimposing images of computer models in three-dimensional space to a corresponding physical object in physical space.” In 1998, projection mapping really started to get traction when it was pursued in academia. “Spatial Augmented Reality” was born out of the work by at UNC Chapel Hill . It all got started with a paper The Office of the Future. The Office of the Future envisioned a world where projectors could cover any surface 9

connecting offices from different locations by projecting people into the office space as if they were there. Instead of staring at a small computer monitor, we would be able to experience augmented reality right from our desk. In 1999, John Underkoffler as the designer who invented the Minority Report interface. But before that, he pioneered some of the early work in interactive projection mapping. He introduced the concept of the I/O Bulb (Input/ Output Bulb), namely a projector coupled with a camera that could one be as universal as a traditional light bulb. In early 2000s, researchers follow on Shader Lamps work. And more artists began using projection mapping in artwork, and groups such as Microsoft began experimenting with it as a means of technological advancement. From there researchers went on to explore moveable projectors (predicting the Pico projectors of the future). These hand-held smart-projectors are aware of their position and orientation through a variety of sensors. They demonstrated using smart projectors to aid in warehouse inventory and maintenance.

Key Players There are many players who are involved in conceptualizing, creating and appreciating stunning projection mapping displays. The following are some of the groups that come together for a typical installation, but this may vary depending on the project.

Clients: They initiate the request for a projection mapping installation and define the core requirements for the project. They can include anyone from consumer brands, museums and government to nightclub and theme park operators. Stagers: The specialist companies that focus heavily on the event business, usually renting the projection and related hardware needed to â&#x20AC;&#x153;stageâ&#x20AC;? events, and providing the assortment of skills and experience needed to design, manage and execute projection mapping spectacles. Content Developers: While many staging companies have in-house creative capabilities, production studios focus specifically on the output of the projection systems. They conceive and produce the video and motion graphics that transform objects and structures. Systems Integrators: Professional AV and IT systems companies have the necessary experience and broad understanding of all the requirements - from optimal technology choices to light and noise restrictions - needed to execute successful projects. 10

Audience: No amount of investment in technology or creative matters unless it has the desired effect - from pure wonder to calculated brand awareness - on the viewing audience.

Possibilities The field of PM opens a wide range of new and different styles of narration. Because PM is not a finalized subject, technologies from other departments are used and are combined with the classic PM setup to create new ones. Only the possibility to control the texture and color of an object over the time offers enormous freedom of design. In this way huge three-dimensional screens for storytelling are created. Especially impressing results emerge, if the projection interacts with the object. A great example is the project “Searchlight” from Urbanscreen. They installed a moving head projector on the ceiling of a room. The projector can be controlled in pan and tilt rotation and projects additional information onto the walls and pillars. So the projector beam moves through the room and enables the view behind the surface or creates objects on the surface (Urbanscreen, 2015b). “Amadeus – Interactive Projection Mapped Piano” is an installation, where the player can interact with a projected piano on the floor to create music and visuals on a façade (Jones, 2015b). It is a big advantage for a PM project to know the viewer’s position, because the awareness makes it possible to create specular and transparency effects for the projected object. Either, the position of the viewer is fixed, for example in some cases of façade projection the viewers are bounded by barriers, or the position of the viewer is tracked and the tracking information is sent to the software (Nayar et al., 2003).

As a result, the basic PM setup enables a lot of possibilities to narrate with creative content. But these possibilities can be extended by the combination with other systems. In this way setups can be created where for example content is projected on moving objects or the viewer influences the content.

Technical Display Components: Scaling, warping & blending systems Production and playback systems organize and choreograph motion graphics, video, stills, sound and live camera feeds across a broad canvas illuminated by multiple projectors and sources. Often, one visual is too large to be driven by one projector, so edge blending and related tools stitch multiple displays together to create one image with no perceptible seams. In other cases, technology does the work to create a mosaic of visuals. This scaling, warping and blending is achieved with software or with hardware management and switching devices supported by software that can layer, mix, define, shape and blend sources. Playback devices Any PC or video source can send a display signal to a projector or management and control device but several companies have developed PC-based playback systems that are specifically tuned to the high-performance demands of projection mapping projects. These devices are capable of sending out resource-intensive, uncompressed video and doing real-time video processing. Some incorporate the show controls of scaling and blending systems.

Software Pure-play software companies focused on demanding projection work development tools make projects easier to set up and optimize for the viewing audience. Many companies offer tools that manage soft-edge blending and image warping and make what can be a long, painstaking process of aligning projections and stacking projectors relatively fast and easy. There are also software applications designed to make real-time projection - for users such as live performance vee-jays and live theater designers - both flexible and simple to use. Displays Many display manufacturers offer consumer and office-grade projectors, but very few of those companies have the engineering, experience, installed footprint and support to drive super-bright visuals for mapped surfaces of any size. The same technology foundation that drives the most advanced digital cinemas around the globe is used for a variety of powerful commercial projectors applied to mapping. Projector choices are driven primarily by the amount of lighting power needed by the project. Vision systems A handful of companies have developed solutions that take much of the complexity out of managing and unifying multiple displays. These systems can blend the outputs of multiple projectors - even commodity, non-professional projectors - and deliver them as unified, malleable displays.

Production Chain To create a successful PM you have to follow some steps during the production. Depending on the field of application these steps can vary. As mentioned in (Pauser, 2010) there exist several implementation options for different fields of application: Pre-adjusted content Subsequent video-mapping Relief projection Distortion as a stylistic device The focus in this research paper is on subsequent video-mapping, because it is useful for the majority of the projects. This method is used when the whole projection cannot be planned previously. For example, it isn’t clear where the projector can be mounted. The production chain for this method can be separated into three big steps as shown in figure 5 (Pauser, 2010). The first step is the creation of a virtual model of the projection surface. The goal is to create a virtual model, which has the exact geometry as the projection surface. The more exact the model the better the result of the work. In the simplest case, the model can be created from a picture with high resolution, or it can be created with infrared sensors. If the creation of the virtual model was successful, you can start creating the content for the PM. The quality of the result benefits, if the artist works close together with the model to get a good feeling what the projection surface looks like. This means for example to separate layers like pillars and windows in a façade projection from each other to merge the real and the virtual world together. To create such content software like “Adobe After Effects” or “Autodesk Maya” can be used. If you know the approximate position of the projector in relation to the projection surface, you can create a virtual camera in the composition and render the scene with a black background through this camera to compensate first distortions. With the finalized render you can move to the next step. The aim of the third step is to install the projector and warp the content onto the projection surface to compensate the position offset. For this task there is a lot of different mapping software available. Software like “MXWendler” and “Resolume” allow manually warping with different elements. This means on the computer are different points visible, which can be repositioned on

the surface to warp it. Other software like “TouchDesigner” allows the import of the virtual model and it calculates the position offset. The user just has to specify six points on the projection surface.

Concerns Audience Many questions need to be asked and explored about the audience for a projection mapping project. These include:       

How large is the crowd? Are they seated or standing? How far back will they be? How wide is the viewing zone and will those at the sides see the visuals properly? Does the performance include audio and if so, what’s the audio delivery technology? Is this a scheduled performance that people come to watch with a hard start and finish or is it intended as continuous ambient material? Or something else? Do the content plans and objectives make sense for the composition of the expected audience and the tone of the event? (For example: Is what’s planned appropriate?) Are there any aspirations to make the event interactive with audience participation through gestures, sound or other means?

Site conditions Technology and creativity can overcome the characteristics of most structures and objects targeted for projection mapping projects, but a thorough site inspection at the start of any project is essential. Both the technical and creative leads need to inspect the site and the environmental and physical conditions that can affect the presentation and technical operations. Those considerations include:       

The amount of ambient, surrounding light Obstructions on the projection path, such as trees, streetlights and power poles For live performances, the movement of people potentially in the projection path The distance, location and height of projector positions, which informs decisions on the brightness and number of projectors needed, as well as the lenses and media devices Power availability Weather conditions, not just temperatures and moisture, but wind as well Rooftop access or line of sight window access

4 Projector Attributes “Projector attributes are the most important in delivering the opti mum projection mapping experience.” 1. 2. 3. 4. 5. 6. 7. 8.

A high resolution projector, Lens quality, Reliability and ease of fail- over, Brightness of the image, Contrast ratio, Color accuracy, Image quality, Moving image quality.

5 Techniques used for Video Mapping Laser shows Lasers the strongest light source known to man. Laser show audience members are not just spectators, but an integral part of the experience. Laser light shows and effects can be used as stand-alone entertainment, synchronized with music, or as part of c omplex multimedia entertainment. Laser Shows are highly effective in punctuating presentations and special events; adding drama and impact to openings. Laser can be used for the indoor and the outdoor events. At times, you might also see laser shows projected into an audience area, where people can actually touch the laser light. This is called “audience scanning” and it is one of the most beautiful effects that can be created with laser. However, it also requires a deeper understanding of laser safety to ensure no one in your audience is harmed from laser coming in contact with their eyes.

5.1.1 Laser beam categories Laser beams and aerials are considerably the most common type of “effect” created with laser.

 Beam shows and effects (Figure 1)

Figure 1: laser beams and effects

 Laser fans, sheets and tunnels (Figure 2)

Figure 2: Laser fans. sheets and tunnels

 Liquid sky laser effects (Figure 3)

Figure 3: Liquid sky laser effects



Audience scanning laser effects (lasers come in contact with people) (Figure 4)

Figure 4 :Audience scanning laser effects

Laser graphics and effects Laser graphics are another fun type of laser effect that can be crated. These also take multiple forms and may include animated laser graphics, laser logos, laser text, laser abstracts, 3D laser images, and even fully animated graphical laser shows.

 Laser graphics and images(Figure 5)

Figure 5: Laser graphics and images

 Laser logos and projections (Figure 6)

Figure 6: Laser logos and projections



Laser text

 Laser abstracts (Figure 7)

Figure 7: Laser abstracts

 3D Laser animations (notice how you can layer images) (Figure 8)

Figure 8: 3D Laser animations

Laser mapping and architectural laser lighting One of the newest and fastest growing laser effects, is called “laser mapping”. When you perform a laser mapping effect, you take a laser projection system and use it to map out the edges of a building, object or surface (such as a city building or structure, car, stage, etc.). This is very popular among architects and designers, looking to highlight and accent a structure or object.

 Laser mapping effects (Figure 9)

Figure 9: Laser mapping effects

Interactive laser shows Interactive laser shows are where people in an audience control the movement of lasers, with their own body movement. This new technology has only been possible in recent years. (pangolin laser system, 2017)

Laser billboards Unobtainable and exotic sites can be turned into animated Laser Billboards. Large buildings, sides of mountains and even entire stadiums have been turned into canvases for Laser Billboard projections. It is possible to project logos, texts and abstracts over very long distances. The Figure 10 Sidney Laser Billboard brightness of the laser beams let the advert "shine" and gives an additional emotional effect. Laser Billboards are not very common yet, so the effect of the "new" way of advertisement will catch additional attraction. Laser Billboard Advertising is a special way of putting extra attraction to a certain brand, product or event. As there is no dependance of certain projection surfaces, it is possible to use existing infrastructure. (laservision, n.d.)

6 Methods of projections Method to project onto buildings This Projection Mapping at the Ralph Lauren building in London. The images were projected onto Figure 12: Ralph Lauren, the large Ralph Lauren buildings (. It shows off the images effectively with the color of the building and characteristics of the shape. Projection Mapping onto buildings varies from large buildings like those of Ralph Figure 11 This Projection Mapping at the Ralph Lauren building in London Lauren to relatively small buildings. We create a CG model with the same scale based on the blueprint of a building, and produce the images for the projection surface that has been selected in advance. As with outdoors, it depends on ambient lighting, weather, positions of the building, and the situation around them. A high number of high-intensity projectors need to be prepared depending on the size of the building. This can be extensive work in many cases. This large -scale Projection Mapping has been used by advertising companies.  Method to project onto rooms and passages In 2011, the German Projection Mapping expert group Mr. beam announced 360 degree three-dimensional Projection Mapping new technology as a testing form. This test video “living room” is directed as the furniture and the carpet placed in the room change as time goes by projection. The living room is different from Projection Mapping methods of projecting images on the surface of buildings that they used to project onto and now it focuses on small objects such as furniture. It points out Projection Mapping as a space direction of projecting the images onto an area and an entire passage. Viewers can experience the images as a space, it makes them feel as if the virtual and real were actually real.

Figure 13 <<Living Room>> Projection Mapping

 Method to project as stage direction Projection Mapping as stage direction is used on a stage like a live performance and fashion show. It has benefits such as the radical change of the stage and atmosphere with one image as well as decreasing spending except screen installation and projectors. In the music video of American artist Nosaj Thing’s song “Eclipse blue” by Manabe, Projection Mapping has been done interactively applying the people by image processing the approximate shape of a human with a depth sensor. As it receives the shape of the human in real time, the images move to the dance. It brings on a strange impression that cannot be brought with video synthesis by post-production as it combines the human and the images on the spot.

 Method to project onto Sculptures “Polygon playground” is an interactive video installation produced by WHITE void as a “large scale interactive lounge” at the SMUK fest music festival in Denmark. According to their website, they made a large-scale polygon object glow that offers room for up to 40 people at a time using 360 degree Projection Mapping. An additional sensory system detects people’s positions.

Video Mapping Installation Geometric Warping The projected image has to be warped to fit the object. For example, if the object has a cubical shape, it can be illuminated with one projector on three faces. The image has to be adapted, so that each face has its own plane. If this warping was successful, the viewer can move around and perceives anywhere three individual illuminated and distortion-free faces. The projected image should therefore plate the object with a new color or texture . To build such planes and layers for objects, photographs of the objects or manual adjustable grid points can be used. Usually the objects have a more complex shape than a cube. To be able to fit these shapes, models of the objects are created. These models can be imported into any 3D computer graphic software and serve as planning guides. There exist different techniques to create such models:  Stereoscopy: Two cameras are located side by side and are oriented to the object. The position and the camera settings are calibrated, so that each camera provides a different view. Via triangulation the coordinates of the object can be created. The dependence of ambient light is a disadvantage of this method. 

Structured light scanning: This method is similar to stereoscopy. One camera is replaced by a projector, which projects defined gray code sequences. Because of the object’s shape, this grey code looks in the view of the camera deformed. Positions of camera, projector and deformation points allow the calculation of the object’s coordinates.



Shape from focus: This method uses the depth of field of cameras. Pictures with different focus values are created. The depth of field gives information about the distance. These pictures can be put together and can create the coordinates.



Infrared sensor: Devices using infrared like the “Structure Sensor” 4 allow the creation of a model in combination with an “iPad”.

Multifocal Projection The easiest way to realize projection mapping projects is to use a single projector, but in some cases more than one projector is needed, because most projectors have just one focal plane. In the following examples multiple projectors are required. The shape of the used object is too complex and has more focal planes, for example, spherical objects or stairslike objects as shown in figure 4 (a) and (c). The depth of field of one projector is not enough for such uses. The size and the shape of an object require projections from different directions. If something has to be projected onto a big cube, its geometry blocks the view on other faces. Another reason for using multiple projectors is, if more light intensity is neede d. Several projectors can be overlaid to generate more light per area. Projectors are limited by their maximum distance to an object, these results in a maximum projection area. If the object is bigger than the maximum projection area, multiple projectors are formed into projector arrays to cover the whole object. If the possibility of peoplesâ&#x20AC;&#x2122; presence between projector and object exist, multiple projectors can be used to eliminate the shadows produced by persons. To create a credible illusion with the usage of multiple projectors a lot of adjustments have to be done. The colors and intensities of each projector have to be matched. But the most difficult part of using multiple projectors is the area of overlapping. The intensity and color of each overlapping pixel has to be adjusted to create a smooth and invisible transition. In (Bimber & Emmerling, 2006) a multifocal projection concept is described. This concept uses a camera to evaluate the value of defocus for each pixel and for each projector shown in figure 4 (b). After creating a defocused map for each projector a composition of a final image with minimal defocus can be calculated. To improve the crossover area, the edges are filtered with a low-pass. Thereby, the individual projector images can be overlaid.

Box – Bot & Dolly The experimental project “Box” by Bot & Dolly uses industrial robots, an actor, illusion techniques and PM to create an astonishing performance, which was captured in-camera. All effects of the five minutes video are created by the complex installation without adding effects in the postproduction. The design of the whole video is kept simple but it is all the more impressing. The design concept of “Box” is based on five principles of illusion: transformation, levitat ion, intersection, teleportation and escape. Each principle is built upon the previous one. The performance takes place in Bot & Dolly’s hall in San Francisco, California. Three industrial robots are used. Two of them are holding and controlling 4’ x 8’ white canvases, which function as projector screens. One of them is static; the other one is mounted on a dolly system to cover a bigger range. The third robot is responsible for the camera movement and takes care of the “viewer’s” right position, to enable the illusions. Behind the canvases background lighting is installed to expand the perceived projection area. In figure 6 you can see the performance during the intersection part (Munkowitz, 2013). All illusions of the performance work because of the int eraction between camera, canvases and projected images. So the crucial point for working illusions is the common coordinate system of the components. The projected images are created in “Autodesk Maya”. The canvases and the camera are also controllable through the robot software in the 3D scene. To synchronize all different 3D scenes, Bot & Dolly developed their own software, called “BD Move”. In this way a preprogrammed performance was created, which looks each take the same, so that the actor has enough freedom to interact with the system to get the best out of it (Cone, 2013). To create the perception of the viewer more credible, the camera movements of the robot have been humanized. A camera operator filmed the whole performance with a shoulder rig. The movements of the shoulder rig have been tracked and this motion path has been applied to the camera robot (The Creators Project, 2013).

Set-up – Projector & Table A small table set with plates, cups and cutlery served as projection surface. The plane of the table was positioned parallel to ground. To provide sufficient lighting through the projector without casting shadows on the table through the dishes, the projector has to be mounted directly above the table. So this was the most challenging part of this task. The correct position was also necessary to have a homogenous plane of focus on the table. The height of the dishes caused different distances from the projector to the projection surface, which results usually in blurriness because some areas are out of focus. This phenomenon caused in this set-up no problems because the differences of the distances were too small. To mount a projector in such an unusual way the “OMB Monoprojector” ceiling mount was bolted onto a wooden cube. This cube was mounted through screw clamps onto the wardrobe as shown in figure 12.

Shoot Footage To create photorealistic content the majority was shot beforehand. It was important to keep the angle between the camera and the object the same as the angle between the projector and the object, this means shooting straight down on the top of the objects. Soft light was used to illuminate the scene to avoid strong shadows, which could limit the creative possibilities during the compositing. The theme of this project is breakfast, so the goal of the shooting was to capture as many meals as possible. The filmed footage gave a good basis to compose with, as shown in figure 13.

7 Projector attributes ‘’ Projector attributes are the most important in delivering the optimum projecti on mapping experience.’’ 1) 2) 3) 4) 5) 6) 7) 8)

A high resolution projector, Lens quality, Reliability and ease of fail- over, Brightness of the image, Contrast ratio, Color accuracy, Image quality, Moving image quality. (Panasonic)

8 Recent advancement in projection mapping In the emerging application scenarios, there are strong demands for displaying desired appearances on non-planar, textured, and/or dynamically moving surfaces under environmental lightings. To meet the demands, researchers developed computational algorithms to project geometrically and photometrically correct images by applying projector-camera systems (procams). Those pro-cams use cameras to observe the projection onto the surface and to estimate how to adapt the projection image to display the desired augmentation.

Projection mapping in a brief: The fundamental tasks which need to be carried out in most projection mapping applications can be classified into two categories: (1) ones with the desire to project geometrically undistorted content onto complex geometry; (2) ones which are able to flexibly control the colour appearance of the projection, or a mixture of both. To achieve such goals, several calibration steps have to be carried out which can be mainly classified into: (1) geometric calibration tasks dealing with the exact modelling of the shape of the projection surface as well as the internal and external parameters of the used projectors and cameras; (2) photometric calibration tasks which deal with the estimation of the internal colour processing of the used input and output devices as well as the reflectance properties of the surfaces to project on.

8.1.1 How these calibrations occur?

Figure 14: geometric calibration

ď&#x192;&#x2DC; Cameras are used to capture a series of projected structured light patterns to generate pixel- or even sub-pixel-accurate mappings between projector and camera pixels. ď&#x192;&#x2DC; In combination with a geometric multi-camera calibration procedure, estimating their optical properties as well as global orientation with respect to each other, the surface geometry can be reconstructed and the projectors allows to be geometrically calibrated to generate a consistence projection image. (Figure 14)

Figure 15: photometric calibration

ď&#x192;&#x2DC; A photometric uniformity is also achieved, several further colour and intensity patterns are projected and analysed to estimate the deviceâ&#x20AC;&#x2122;s internal colour processing as well as the light modulation on the surface. Figure 15

Main factors affecting advanced video mapping production: Factors of video mapping production

Algorithms Geometric calibration of projector- camera systems

Hardware

Dynamic projection mapping

High dynamic range projection Radiometric control of projector camera systems

High speed projection High resolution projection Increasing focal depth High field projection Multispectral projection

8.2.1 Algorithms: The algorithms are mainly subdivided into methods related to the geometric calibration of procams. Secondly, the dynamic projection mapping. third area is the calibrating and compensating the radiometric properties of procams. 28

8.2.1.1 Geometric Calibration of Projector-Camera Systems: This is the task to estimate the three-dimensional properties of the devices as well as their relationship with respect to each other, as well as to the surface in contrast to a projector-camera pixel correspondence estimation and mapping into a two-dimensional space, which we call geometric registration. Figure 16

Figure 16: geometric calibration

8.2.1.2 Dynamic Projection Mapping With the evolution of CPUs and GPUs power, and high-speed cameras and projectors are now be-coming commercially available, more and more dynamic projection mapping systems have been published. In this technology, the projectors or the cameras are allowed to move to help receiving more details and forming a 3D detailed imaged. Figure 17

Figure 17 ( a full detailed human face captured by several projectors moving around it)

8.2.1.3 Radio-metrical Control of Projector-Camera Systems: The radiometric control is responsible of accurately measuring, controlling, and compensating the light which is emitted by the projectors and reflected from the surface. Figure 18. Figure 18 ( a picture taken by different photometric compensation)

8.2.2 Hardware: technologies of hardwares is responsible of going beyond algorithmic improvements to further improve the projected image quality for arbitrary surfaces by applying an emerging approach, Computational Projection Displays, optics and computational algorithm. Haredwares features are divided into 6 main features which are mainly the responsible of the high-quality image projected.

1) High Dynamic Range Projection: A high dynamic range (HDR) representation would be required to realistically render and display both natural and computer-generated images.

2) High Speed Projection: High speed projection systems enabling a much higher frame ratethan a normal video rate. And this by reflecting a light from a light source to the objective lens at thousands of frames per second.

3) High Resolution Projection: It is highly demanded to realize a high-resolution projection since, due to the high spatial resolution of the human eye. There are two main approaches to accomplish this goal:(1) using multiple projectors to generate a higher resolution, or (2) realizing this task with a single projector. Figure 7 shows examples of high resolution projection systems Figure 19 Figure 20

Figure 20

Figure 19

4) Increasing focal depth: Projectors are designed with a large aperture to minimize the loss of light emitted from the light source. This optical design leads to a shallow depth of focus (DOF). Consequently, an image projected on a surface with large depth variance can become blurred quickly. Single-projector approaches digitally sharpen original images before projection so that an optically defocused projection closely approximates the original image. one of the single projectors approach is the focal sweep projection. Figure 22 and Figure 21

Figure 21

Figure 22

5) Light Field Projection: Glasses-free 3D or light field projection systems, are systems which provide physically correct views for a wide range of perspectives where observers do not need to wear special glasses. The number of projectors roughly matches the number of view directions. Figure 23

Figure 23 (a 3D image caused by light field projector and angle expending screen)

6) Multispectral projection: Multispectral projection is responsible of colour richness. Current three colour channel (RGB) projectors can only reproduce limited colour spaces. Innovated multi-primary designs increase the number of colour primaries either using grating (7 primaries) for a larger colour gamut. Various adaptive colour primaries have been investigated such as LEDs (a pair of prism) and DMD (a pair of diffraction grating), and a programmable spectral light source. (Iwai, 2018)

Figure 24 ( figure shows the different between multi spectral display and a conventional LCD display)

Figure 25 (different display methods)

9 Video mapping applications 1) Video mapping in Buildings facades. 2) Video mapping dances 3) Live face Video mapping

10 Case study: Digital video expression using Projection Mapping techniques at Mojiko station Abstract This research is about creating contents about the origin and history of Mojiko station using Projection Mapping one of the new digital media techniques, and designing video expression and evaluation in order to develop a new genre of Projection Mapping.

Research background Architecture is a subject with complex meanings. Sometimes, it is evaluated as public art beyond one for human convenience. Currently, there are cul tural assets with the traces of history at Mojiko Retro in Kitakyushu City. Architectural heritages, in particular, among cultural properties are ones that show cultures and histories best, which reflect the images of each period well. Architecture has characteristics of each country, but the flow of the recent world architecture homogenized in external aspect. In addition, the present world shows complex multi-cultural characteristics along with each country's own culture, and among them, young generations are losing the meaning of historical and cultural identities. Based on this phenomenon, using the contents of cultural heritages with cultural and historic values is considered as a nice opportunity of increasing people's interests in and values of the cultural heritages that are being forgotten. In addition, it is expected that a secondary effect by which the meaning of architectural heritages through works become more educative and corresponding values.

10.2.1 Purpose of research This research is about creating contents based on the origin, history and image of cultural architecture heritage using Projection Mapping one of the digital media techniques.

10.2.2 Summary of projection object

the

We selected Mojiko station located in Mojiko, Northeastern Kitakyushu as the cultural asset for which contents will be produced. Mojiko station is a bisymmetrical, NeoRenaissance style, two-story wooden building constructed in 1914. Fig.7 "Mojiko station" important cultural properties 33

Remaining intact as it had been built, it was designated as an important national cultural asset first as a station building. In the station, there are various historic assets such as basins which have been used since before the war and a fountain where people can wash their hands, etc. In this study we selected Mojiko station as the object of projection because of its historic value designated as one of Japan's important cultural assets and because the station will enter a preservation and maintenance project from September 2012 to March 2018, it would be the very building from which residents or tourists could reflect on memories. In addition, the contents of the work were produced using subject matters related to the history of Mojiko station [7].

Projection Mapping Projection Mapping was born in the process of the developments of projection techniques. It has the exact same form of â&#x20AC;&#x153;projecting images onto somethingâ&#x20AC;? as the former projection techniques. However, the major difference is to project images onto three-dimensional and textured surfaces. The definition of Projection Mapping by Projection Mapping Association Japan (PMAJ) is an imaging technique synchronizing objects (real) with images (virtual), which is not simply projecting images or computer graphics on the screen or a plane, but projecting them on a building, 3D structure or artifact with a projector. Images created with former projection techniques have only the virtual side of images. The screen is the only real object and projection medium to capture images. It enables us to enjoy not only images projected but also objects that are being projected onto. The fantastic and illusive visual expressions can make audiences feel like the objects move, change shape and glow by itself with the moves and changes of the images [3]. Figure 1 shows a schematic diagram of Projection Mapping. Projection Mapping enables complex expressions including the representation of the surrounding environment combining seeing a real object and watching images. Fig.1 Projection Mapping conceptual diagram

10.3.1 Projection Mapping cases We investigated the case of existing Projection Mapping works while we created our work. Expression methods of Projection Mapping have been diversified. It is categorized into several types such as the methods of projecting onto buildings, projecting onto rooms and passages, projecting as stage direction, and projecting onto shaped objects. It is important to have a clear vision before starting to work on a project. Mechanism of Projection Mapping is so simple that the object is changed from a screen to a three-dimensional object, but as projectors emit light to the object theoretically it can project any three-dimensional object. Therefore, when an object changes it can widen the range of expression.

Work Production Existing Projection Mapping can be classified by whether the object of projection is â&#x20AC;&#x153;static or dynamic, and the video is either interactive or non -interactive.â&#x20AC;? This study will produce a work with the static technique which is most often used among the existing Projection Mapping methods. Since a static object is one that is not moving such as a building or furniture, an interactive device is not specially necessary for the production of projection images. The characteristic of the Projection Mapping of a static object is a method of prerendering1 for the object is not moving. Figure 6 shows a brief diagram of the flow of the projection images: This is to trace one object at a time and produce images fit with the traced data. Fig.6 Static target Projection Mapping flow.

10.4.1 Production process As briefly described above, the Projection Mapping of a static object is described with an example of Projection Mapping at Mojiko station was carried out on December 24, 2011. It is a Projection Mapping by making contents with the value and historic materials of the station building which is just as it was in the old days. Generally, Projection Mapping uses pre-rendering movies. The flow of production is as follows in Figure 8. Fig.8 Production process to static target

10.4.2 Contents Use Mojiko's specialties such as rickshaws, banana characters, a train and historic materials as the contents, then produce them so that they can make the audience who visit Mojiko station feel the charms of the station. The concept of the production is Projection Mapping digital image expression using attractive images such as Mojiko's unique retro images which induce the audience's emotional changes or romantic Mojiko's night view images. The flow of the stories of the image expression is as follows in Figure 9.

10.4.3 Correction process of projection target In Projection Mapping, we construct a 3D virtual space a computer and carried out a simulation test in the same environment as reality to deliver stereoscopic visual effects to the audience. We then grasp the shape of the projected object and the positions of the projector and audience's viewpoint in advance and distribute each of them in the virtual space on the computer. Then we carry out projection texture mapping of the rendering images obtained from the audience's viewpoint on the object in the virtual space in the same condition and obtain the position of the projector and the rendered image. Projecting the image obtained this way in the actual environment as the simulation makes it possible for 3D Projection Mapping. (Figure 10) 10.4.4 Video production The method of production is first to select an object and accurately measure the shapes of the selected object or sculpture. And then, produce an image to project based on the data. To construct a 3D virtual space, we use mental-ray software that is broadly used to produce video contents such as films or games to construct the scenes of the necessary modeling and lighting. We also use a mental-ray rendering engine for rendering. 10.4.5 Structure of the System In this study we installed a system like Figure 12. Panasonic PT-D7700 was used, which is a projector supporting Full HD resolution with brightness of 7000 ANSI and contrast of 1300:1. The projection image and sound effect (BGM) was controlled by portable switcher â&#x20AC;&#x153;Any Cast Station.â&#x20AC;? The system in this study is characterized by installing two workstations and projecting images simultaneously to prevent the images from halting in their projection. Fig.12 Structure of the System

In addition, two overlapped projectors were used so that the system can respond to the brightness of the surroundings at the place of projection.

Experiment In the first experiment, the distance was checked between a projector and the projection object , the size (height x width) of the surface of the projection object (Mojiko station) , brightness of a projector installed with the number and width of video. The equipment was installed based on the blueprint of the projection object and the size investigated in advance. The distance between the projector and projection object is 66.8m. The height and width of the projection object is 13m and 20.6m. This is one of the important initial steps for accurate matching of the projection video and projection object in the process of projection mapping. A high-definition (1920*1080) still photograph images and short videos was prepared and had an experiment to test performance brightness of the projector. As a result, no problem was found with the size of the surface of the projection subject but found out that the brightness of the projector is not enough. The

reasons could be the brightness of the surrounding environment, the performance of the projector, the image of the created contents etc but in this case it was discovered it is because the brightness performance of the projector is not sufficient. As a solution to this problem, using a high performance projector is the fastest solution, but this study was able to solve the problem of the brightness of a projector at a low price using two projectors currently held (Panasonic PTD7700). This method is suitable for smallscale Projection Mapping research without expensive equipment.

Performance On December 24th 2011, we performed Projection Mapping methods at Kyushu Mojiko station as the projection object in Mojiko, Kitakyushucity. The video was about 10 minutes long. In this performance we started projecting at 18:30 and we played it 7 times for 1.5 hours. Because of the media report the day before, we were able to have a large audience. We published the entire video of this performance on the internet. With this work, we won a special award in “Kitakyushu Digital creator contest 2012” presented by Human Media Creation Center / KYUSHU and a prize in “2012 ASIA DIGITAL ART AWARD” we were able to obtain a certain level of achievement.

Conclusion The expression combining images with real objects was made for cultural heritages as the objects of projection, and the methods of expressing contents based on the cultural heritages could be obtained. By presenting works, it is considered that social evaluation was obtained as even people who did not know the histories and stories about th e cultural heritages could enjoy and watch them and they could look back on the history of Mojiko with many others. Moreover, it is considered that by making experiments of Projection Mapping, it is necessary to express the materials of cultural heritages with image contents and to study the problems that occur during the process of production more in the future. In the near future, in order to perfect the techniques based on this research we think it is important to clarify how to develop the contents of Projection Mapping and techniques by testing them and experimenting under various conditions. We especially need to identify specific issues such as light source issues by using Projection Mapping and the reproducibility of the contents and solve them in future research.

11 Case Study: Dynamic Urban Projection Mapping Abstract “Dynamic projection mapping” is a variation of the best-known “projection mapping”. It considers the perceptual analysis of the urban landscape in which the video projection and the observer’s displacement speed are hypothesized. This latter, in particular, is variable and may depend on factors not directly controllable by the driver (slowdowns due to accidents, rallies, etc.). This speed can be supported and controlled by a number of traffic flow measurement systems.

Concept and objectives A case in point is the promotion of a showroom on the Grande Raccordo Anulare (GRA, Ring road) in Rome, based on perception from moving vehicles. The showroom location is particularly interesting since it exhibits the following features: • the area is located in a place delimited on both sides by the GRA following the completion of the 3rd lane in 2000. Due to the massive presence of craft workshops that could not be demolished or expropriated, ANAS was forced to split the GRA. They placed the lane of the outer ring in the old site whereas the craft workshop area is located next to the inner lane limit, toward the city; • the presence of several workshops and plants along this section of GRA often causes a slowdown in the vehicle flow both in the inner and external lanes, with some congestion times; • it allows a visual design that focuses on a visual interaction with architectural structures or settings through audio, video, and site-specific installations. The case explores a number of previously analyzed aspects within the framework of urban landscape perception, “projection mapping” and “interactive design”. Its purpose is to create an attractive identity of a building or urban area to the observer’s eye. The observer has intrinsic features of movement and mobility. Likewise, the application is smooth and dynamic. “Projection mapping” (Figure 26 .Dynamic urban projection mapping framework ) is made on an industrial architecture, with an adaptive (interactive) design according to the various space-time conditions around it.

Figure 26 .Dynamic urban projection mapping framework

Vehicle Flows and Light Hours in the Selected Area After identifying the site for a methodological exemplification, the urban landscape is analyzed in an effort to contextualize the selected building. The preliminary survey aims at evaluating the effects that the landscape exerts on the observer’s perception. In particular, reference is made to the studies conducted by Kevin Lynch and Gordon Cullen. Following the classification proposed by Lynch, satellite views of the area around the showroom are used to identify the “main” and “secondary” road arteries. In addition to accommodating a continuous flow of cars, they act as barriers and define some agglomerations of houses and commercial buildings, called “districts”. At their intersections, they identify “directional nodes”, key point for road traveling. According to the legend, inspired by the one developed by Lynch, one can observe how the showroom (landmark) is located at the center of the bifurcation of the Grande Raccordo Anulare, between two nodes (Figure 27. Analysis of the site according to Lynch studies).

Figure 27. Analysis of the site according to Lynch studies

In line with Cullen’s studies, the perceptual influence is analyzed in a more specific way and from the eye of an observer inside a car (Figure 28). The analysis focuses on identifying the predominant characters of the visual field and, consequently, on their perception, marking them with different colors. The front side of the showroom (external carriageway of GRA) is a contemporary architecture that can be clearly distinguished from the surrounding landscape. The back of the showroom (inner carriage of GRA), on the other hand, has a simpler and more compact architectural shape.

Figure 28. Perceptual influence of the “landmark” according with Cullen’s studies

Figure 29. Traffic flow tool in the area of interest available through Google Maps application

Figure 30. Traffic peak hours in the area of interest.

By comparing the hourly traffic data on this GRA section, the daily hours of darkness, sunrise and sunset time in Rome, it can be inferred that the best design choice is to perform the projection in the busy traffic afternoon hours, also because of the environmental darkness. An environmentally friendly design includes a control system, which checks the presence of vehicles after 10 pm.

Execution The front facade can be covered by 2 projectors (Figure 31). A critical feature is the glass portion of the faรงade. This material allows the projector light flows to cross it. A good fix is a special transparent film to set and lock an image even when the indoor lights are on. It can be applied on any transparent glass, acrylic or plexiglas surface.

Figure 31. Front faรงade on GRA outer carriageway

On the back of the building, a single projector positioned frontally is sufficient (Figure 32). The projection surface consists of a series of horizontal metal strips, arranged in regular bands with an empty space between them. Taking into account the principle of proximity of Gestalt and the distance from the observation point, motorists perceive the facade as a single surface. For a proper viewing, however, it is necessary to avoid multimedia content with small-sized details. These would be poorly visible due to the shape of the coating structure (Figure 35). 45

Figure 32. Back façade on GRA inner carriageway

The projection system is designed to work autonomously by capturing data from the external environment and from web applications through an internet connection. A computer with a powerful graphics card locally handles the video stream while a node-based program synchronizes the different projectors. In particular, the system consists (Figure 33) of: • Brightness sensor: within the established nighttime operating range, it allows to spot the low light conditions best suited for “projection mapping”. In the event of a bright light, the system does not start the projection. • API Google Maps: Google’s traffic data allow a dynamic and automated content management of contents uploaded on media servers. Data is checked every 5 min to possibly update the projected content. • Speed sensor: it controls the road data obtained by Google. In addition to its feedback function, it acts as an emergency system in case of network malfunction. 46

â&#x20AC;˘ Media server: it is a cloud service, where the contents to be displayed in the video projection are loaded. It allows a remote control of the system and an update of the video projection contents.

Figure 33. System management scheme

The system management scheme requires a computer for data analysis and video content management purposes. Its function is to convert data from physical sensors and online applications (API Google Maps) into video-projection run commands. As noted above, the system needs a physical control device for scanned data and a system for switching on/off the video projection stream. Two different types of sensors are connected to an ArduinoÂŽ card, a small hardware to convert signals from sensors into computer-usable data (Figure 34).

Figure 34. Application interface

The content of “dynamic urban projection mapping” (Figure 36. Dynamic urban projection mapping application) can be of a diverse nature, ranging from the story of the evolution of the industrial/crafts area, to the marketing of the showroom products or the promotion of the company’s history and, therefore, of its brand. The key aspect is that content is in-sync with the speed of the vehicle’s flow and is perceived by travelers accordingly. It is perceived faster or more superficially when the vehicle flow is fast, in greater detail when traffic is congested with severe slowdowns.

Figure 35. Project plan

Figure 36. Dynamic urban projection mapping application

Conclusions “Dynamic urban projection mapping” well fits marketing and visual graphics applications and allows a new and different form of business promotion no longer based on the use of static (archigraphic) or dynamic images (chromatic variation of lighting) but rather on the observer’s moving speed. One aspect to be checked is the New Traffic Laws and, possibly, public land occupation or public lighting interruption. However, these latter are not applicable when “projection mapping” is carried out on a private property.

12 Case Study: Urbanscreen, The Hamburger Kunsthalle Museum Show Background and location: The Hamburger Kunsthalle, is an art museum located in Hamburg, Germany. It is the largest museum in the country and has a great importance. The museum represents an architectural highlight thanks to its central location between the central station and the Alster. (Tourism, 2018). The video mapping show – titled 555 kubik- in the ‘’Galerie der Gegenwart’’ was designed and excuted by Urbanscreen company.

Figure 37( a google earth picture of Hamburg Kunsthalle)

Analysing of the location: By looking at the map (Figure 37), the cubic building of the museum is seen by cars and pedestrians from 2 directions: GlockengieBerwall road, and Ferdinanstor road. The main entrance of the museum, and its longest side is from GlockengieBerwall road, and it has the heaviest traffic. That’s why the video mapping show was on the GlockengieBerwall road façade.

Lighting design concept The basic idea – as mentioned by Urbanscreen company – was to dissolve and break through the strict architecture of the ‘’Galerie der Gegenwart’’ (the cubic building of the museum). Figure 38 (Galerie der Gegenwart (Kindl, 2016)) the basic shape of the “Galerie der Gegenwart” is the cube as shown in Figure 38. The whole building is cubical and is built-up of single cubes. The Urbanscreen team took advantage of this fact and laid the focus of design on it. Each square is interpreted as an independent cube. By movements in depth of the façade plane, these cubes form different structures.

The Performance and its mechanism

Figure 39 (Different scenes from the kubik show (Kindl, 2016)) The performance starts with two hands interacting with the façade. They break through the building and introduce the show (Figure 3 a). The main part includes construction sketches and plays with the perspective of several components of the building like stairs or windows (Figure 39 b). The colours yellow, blue and violet are used to connect the components and they lead through the show. To complete the show, at the end the shape of the hands is recognizable, and they disappear behind the wall. The sound design of the performance is kept minimalistic. The main hearable sounds are the movements of the bricks to enhance the illusion. To make the movements of the bricks visible, a virtual light source was added to the scene. It was positioned near the plane of the wall, to create long shadows. The combination of shadows, sound effects and the real building creates a credible illusion.

The production chain of the performance:

Figure 40 (production chain of the VM show (Kindl, 2016))

1) The first step is the creation of a virtual model of the projection surface. The goal is to create a virtual model, which has the exact geometry as the projection surface. the model was created with infrared sensors and high-resolution image. This step is necessary because the artist, who creates the content for the PM needs something to work on. 2) After finishing the creation of the virtual model, the team started creating the content for the PM. (To create such content software like “Adobe After Effects”5 or “Autodesk Maya” would be used). By knowing the approximate position of the projector in relation to the projection surface, the team created a virtual camera in the composition and rendered the scene with a black background through this camera to compensate first distortions. 3) The aim of the third step was to install the projector and warp the content onto the projection surface to compensate the position offset. For this task, software like “MXWendler” allow manually warping with different elements.

Fixtures and hardware: To create the ‘’ 555 KUBIK ‘’ show, four Sanyo XT35 projectors have been used. To provide four projectors with synchronized content, a main controller is needed. Urbanscreen worked with the “FXServer” from “MXWendler” as a software controller. To generate higher luminance, two projectors deliver the same image and are overlaid to each other. To cover the whole façade two of this Figure 41 (Sanyo XT35 projectors (Know-How, 2018)) projector pairs are positioned next to each other. This 3-LCD projector creates 5.000 ANSI-lumen with a 330 watt projection lamp and delivers a contrast of 1.000:1. With the Sanyo XT35 it is possible to project images with 76.2 cm to 762 cm in the diagonal and it has a native XGA resolution (1024x768 pixels) with an aspect of 4:3. If the projector is mounted angularly to the plain of the object, it provides a vertical digital keystone correction of plus/minus 40 degrees. 52

The software by MXWendler: “MXWendler” is a company located in Germany. They develop software and hardware for PMs and lightings in the field of theaters, events and installations. Their software is called “Stage Designer” and it is optimized for their “FXServer”. “Stage Designer” is a real time video software and it can work with a lot of different industrial standards. One of these standards is “Digital Multiplex” (DMX). This digital protocol can be used for choosing media files or for remote keystone control. Another compatible standard is “Open Sound Control” (OSC). The “FXServer” utilizes OSC to communicate program flows to other systems or for the synchronization with other “FXServers”, if more are required. It is also possible to integrate devices with “Musical Instrument Digital Interface” (MIDI) to control different Figure 42( MXWendler logo (online, 2011)) properties. The simplest workflow for “MXWendler Stage Designer” is to import different media, like images, video or audio, create a playlist out of them and warp the output image to the shape of the object.

Conclusion: The Video mapping of the ’Galerie der Gegenwart’’ was successful and attracted the visitors. Using the new technology of VM shows is a good way of tourism attraction, but it should be limited by duration and time to avoid boarding the viewer. Also, the vehicles circulation must have been putted into consideration when designing this video mapping show to avoid vehicles destruction or glare.

13 Case Study: Yekpare (Monolithic) “Yekpare” is a story teller to narrate 8500 years' story of Istanbul. The story has symbols from Pagans to Roman Empire, from Byzantine Empire to Latin Empire, and finally from Ottoman Empire to today’s Istanbul. A building, which we play on, is Haydarpasa train station has brilliant architectural forms. Connection between Middle East to west as Istanbul since 1906 was formed by Haydarpasa. Internal migration triggered chaotic order of Istanbul dialectical daily life scenes. Either its conceptual and political and geographical position or its location’s depth of field to watch the entire show from Kadıkoy coast; Yekpare is dramatic as a representation.

Lighting objective Installation’s objective is to display historical view of Istanbul by using symbolic and narrative images, through the creation of artistic and dramatic space. It aims to reach more viewers and touch their feeling, creating this emotional ambience dramatically.

The main lighting scenes: 1. Scene 1: Sunrise (01:18) Istanbul is the only place, where joins literally the East meets the West side over the Bosphorus. The story develops over this theme. The Sun is used as a symbol to emphasize the proximity of East and West. Fort this reason, it is located on the scene as an observer. As an appropriate function of the Haydarpasa Train Station, the sun rises from Anatolia and sets at Europe. This scene is supported by light and elegant music to create a mystical overture.

2. Scene 2: Byzantine – New Rome – Istanbul (02:47) B.C. about 700 years ago, Istanbul was mentioned as Byzantine, having started to use the advantages of being a port city. This location, developing by trade, has been taken the names of the Byzantine - Augusta Antonina - New Rome – Constantinapole - Konstantiniye-Istanbul respectively. This scene begins with shadows of the ships, boats supported by audio elements, which identifies the polyphonicism of the period. The Byzantine walls are created, destroying Haydarpasa Train Station’s architectural elements. The religions existing in Istanbul are visualized with regards to symbolism. By transforming Haydarpasa Train Station’s windows into colorful stained glasses, frescos are reflected to the facade. This process of reflecting advances in a linear time. The images of the period of Istanbul’s conquer such as ships going into the Bosphorus; destroyed walls are visualized with sound elements. On the other hand, the aesthetics of Ottoman are shown by “Calligraphic art” to the audiences

3. Scene 3: Watering and Marbling (04:08) Art is also the element, which has come with history apart from religion and language. The visuals of Marbling on the facade interact with the audiences at the same time with the surface of the sea that composes a strong process.

4. Scene 4: Bridge, Migration and Wagons (08:38) There is an animation of the Bosphorus Bridge that connects the European and Asian sides to each other, symbolizes east and west with colors. The east side, with the string instrument and the west side, with the violin instrument. The importance of the train station is improved with the conjunction of two contents by narrating with the movements of wagons that appears in the middle of the facade. The existing identity of Haydarpasa is visualized with the movements of the wagons. This movements support with the sound of the train and narrate the migration.

5. Scene 5: Windows and Crowded (11:10) Increasing migration from the year 1950 the city of Istanbul in different cultures, life values, requirements are changed. Creating people silhouettes to emphasize the stationâ&#x20AC;&#x2122;s windows state this description. Istanbulâ&#x20AC;&#x2122;s complex structure is edited with auditory sounds such as traffic sounds and human sounds.

6. Scene 6: Related Structures; People (12:28) Istanbul, is a growing city with its communication around its surroundings every minute and ever day. The established ties between the station windows describe the communication process of the city. At the same time, describes this aesthetic layout and the Internet era of computing with the development of culture in economic systems Networks are changing the structure that created among the people to tell the technology.

7. Scene 7: Whirling Dervish (14:33) Towards the end of the show, Haydarpasa Station tower is resembled a whirling dervishes and rotated in a slow sound effects. Istanbul transforms so far calm place after its complicated and exuberant moments.

Lighting Fixtures Used Watchout server system distributes the video equally to the projectors. It provides to transfer the images to the surface. 1. Projection: 5 (right) + 5 (left) SANYO15.000 Ansi Lumen The distance between the train station and the audiences is 400 m. and 500 m. therefore, the sound links to the image at 330 m. in 1.3 seconds. 2. 24 speakers with 600 and 800 w.

Lighting Techniques Used Analogue visuals: scene 4. Software: 1. 3D simulation: 3D Max (animation and modeling software). 2. Motion graphics: After Effects, Cinema 4D (motion graphics and visual effects software, 3d video animator). 3. Sound design: Abelton (Live is about making music: for composition, songwriting, recording, production, and live performance).

4. Real-time Video: MxWendler (The image transfers to the surface and then ‘key stone’ setting is done, where the artist would like to interfere various part of the images such as warming, squeezing etc.) For instance, it is used in scene 6. The distribution of the images on Haydarpasa has a mathematical system. An image is divided into 2 parts as left and right. Each part is divided into 4 sub-parts as 1,2,3,4,5,6,7,8,9. Number 10 and 5 are the whole image itself.

Conclusion Over the past decade, the growth of digital technologies plays a vital role to create computerbased artworks to display. Digital installations, branches of video arts, are started to apply not only on a flat screen but also on public spaces. These kind installations are called “Video Projection Mapping” and also can be called as “Urban screening”. Urban screenings, which are integrated with architecture and technology, give information about the content by bringing together audio-visual elements and presenting them by the help of a projector. These elements emphasize the concept of the artwork that builds relationship between images, perceptions and space. So, it is an outstanding point to apply conceptual design stages - brainstorming, defining scenarios, sketching, composing the techniques - in an appropriate and understandable way. A performed urban screening project within 2010 Istanbul European Capital of Culture events, Yekpare (Monolithic), is produced by video projection mapping technique based on a concept Istanbul - a datacity to inform society. In this paper, over the theme, Yekpare’s scenarios are reflected the form of images, symbols and datas that are related to the context and key words. The images are produced by motion graphics applications, sound design, realtime video applications. Composed images are shown on 3d dummy model of Haydarpasa Train Station’s facade to observe the ambience of visual effects, creating dynamic forms in an aesthetic way before live performance. The essential point of Yekpare project is to explore how such displays can create the emotional links between the space and the audiences. Therefore, audiences’ feedbacks were gathered by live interviews during and at the end of the performance and also from questionnaires, which were available in the social media. As a result of feedbacks, most of the The Turkish Online Journal of Design, Art and Communication - TOJDAC July 2011 Volume 1 Issue 1 Copyright © The Turkish Online Journal of Design, Art and Communication 18 audiences were affected and excited by the harmony and the unity of audio-visual elements, which narrate the story in a visual show.

14 Case Study: Zaha Hadid Architects casts Digital Light Mapping Projection on a Baroque Castle Façade Background. An 18th century baroque palace in Karlsruhe, Germany has become the canvas for a dazzling light mapping projection in the latest project by Zaha Hadid Architects.

Figure 43the light installation takes shape on the 170-meter-long façade of the karlsruhe’s baroque castle

It was named “Behaviour Morphe,” the projection display was created in collaboration with composer Max Cooper and leading digital artists Andy Lomas and Mubbasir Kapadia for the city’s 2017 Schlosslichtspiele Festival, exploring how digital spatial concepts could shape the living spaces of the future.

Details On the 170-meter-long façade of the karlsruhe’s baroque castle in germany, zaha hadid architects (ZHA) has collaborated with digital artists/computer science researchers andy lomas and mubbasir kapadia, and musician max cooper to create an immersive light mapping projection. ‘behaviour morphe’ highlights the digital spatial simulation tools ZHA applies in the comprehensive analysis 44 Zaha hadid architects worked closely with and planning of its architecture, and seeks to explore Figure digital artists, computer science researchers, and how living spaces can be designed in the future. musicians 59

Presented for the 2017 schlosslichtspiele festival, ‘behaviour morphe’ reveals the interior of the castle as a series of digital laboratories. these spaces show the circulation and congregation of digital actors programmed with AI that interact with the architecture of the building, as well as each other. zaha hadid architects worked together with andy lomas, mubbasir kapadia, and max cooper to create dynamic virtual spaces using real-time data that interprets the interactions of virtual characters. “Dynamic new virtual spaces are defined by using real-time data to Figure 45spaces show the circulation and congregation interpret the virtual actors’ interactions. These of digital actors digital spaces are then explored with iterative growth systems that emulate the evolutionary process of nature, demonstrating the potential of metamorphic simulations and digital morphogenetics.” The project was produced by ZHA’s Computational Design research group (ZH CODE), which is exploring how to “apply exciting and contemporary research to address locally relevant issues and to understand and assimilate traditional wisdom.” “ZHA's ideas and work fit well with my musical approach, being based on human/machine boundaries, and emergence of biological forms from simulated systems of behaviour,” said Cooper. “I drew on a Figure 46 Dynamic virtual spaces use real-time data to couple of tracks which fit the visual style and interprets the virtual characters’ interactions the development of the piece, which goes from a playful exploration of human interactions with an architectural space to a barrage of cellular forms and audio-visual intensity. “I spent a lot of time working on the sound design to score each tiny event and ripple of visual processing to sync with the mapping across the front of the castle. From my perspective, the crux of the piece is in this synchronisation, bringing the sound to life through the painstaking and indepth simulation and generative computational approaches of the visuals.”

15 Case Study: IMAPP 2016 – INTERCONNECTION

Date – September 2016 Location – Bucharest, Romania Designer – LimeLight Projection mapping & light installation

About the Building

The Palace of the Parliament is the seat of the Parliament of Romania. Located in the national capital city of central Bucharest . it is the second largest administrative building in the world after The Pentagon. The Palace has a height of 84 metres, an area of 35,000 square metres A colossal building, designed and supervised by chief architect Anca Petrescu with a team of approximately 700 architects, constructed over a period of 13 years (1984 –97), it was built as a monument for a totalitarian kitsch style of architecture, in Totalitarian and modernist Neoclassical architectural forms and styles. 61

iMapp Bucharest It is one of the biggest annual 3D multimedia & projection mapping events in the world organized by the Municipality of Bucharest. Video mapping uses a complex 3D image, that brings to life the surface of the building by forms and colours, using each architectonic element in order to personalize and reshape the gigantic area of the construction. Thus, the “outline” of the building becomes the main protagonist of a dynamic show of sound, color, light, a true optical illusion. The creative concept of 2016’s contest was dialogues.

Interconnection

Installation art collective Limelight has transformed the Parliament Building of Romania into a eye-popping, psychedelic light show for the iMapp Bucharest International Video Mapping Competition. Titled “Interconnection,” the video utilized proje ction mapping (also known as spatial augmented reality) techniques to render the world’s third largest building in a blaze of shape-shifting, technicolor graphics and animations. Taking home top honors at the event, The piece focuses on the interconnectedness of all things, the micro and the macro, the outer and the inner universe. Connecting logical and emotional, individuals and society, art and audience, planet and galaxies, architecture and animation etc. Even though we often lose touch with the bigger picture, we are all part of a cosmic network. It is only the human mind’s false perception that inner and outer are separated, causing confusion and misunderstanding in the world sometimes. The piece is aiming to reopen the dialogue between the internal and external through a cinematic journey from the state of separation to the state of eternal openness. The artist is aiming to draw from contemporary visionary art and architecture, sacred geometry, even physics and chemistry to turn the Palace of Parliament into a vibrant, endlessly beautiful scenery of our symbolic dialogue. Integrating the facade’s architectural characteristics into our artwork is also a key element in the concept adding an extra value to this conversation between artist and audience, the city and its citizens 62

Technical information The projection required the use of 104 video projectors to cast the 23,000 square meter surface of the Parliamentâ&#x20AC;&#x2122;s front facade in over 2.000.000 ANSI lumens . Panasonic PT-DZ21K2, PT-RZ970 projectors were used of 10,000:1 contrast and 20,000 lm. Cinema 4d & Adobe After effects were used for modelling & cg production, And MxWendler for creating real time video to be projected on building.

16 References Baker, J. F. (Fri,April 14, 2017). ‘Happily Ever After’ To Feature The Most Advanced Projection Mapping Tech Yet. Disneypark .com. Kindl, F. (2016). An Effective Setup for Realizing Projects in the Area of Projection Mapping as a Small Group. Pölten : Sankt Pölten. laservision. (n.d.). Retrieved from laservision: http://www.laservision.com.au/technology/ Panasonic, b. (n.d.). projection mapping spectacular opportunities. installation. pangolin laser system. (2017). Retrieved from pangolin laser system: https://pangolin.com/types-laser-shows/ Tourism, H. (2018). HAMBURGER KUNSTHALLE. Baker, J. F. (Fri,April 14, 2017). ‘Happily Ever After’ To Feature The Most Advanced Projection Mapping Tech Yet. Disneypark .com. Panasonic, b. (n.d.). projection mapping spectacular opportunities. installation. Lynch, K. The Image of the City; The MIT Press: Cambridge, MA, USA, 1960; ISBN 9780262120043. Cullen, G. The Concise Townscape; Architectural Press: New York, NY, USA, 1961; ISBN 9781136020902. Hall, E.T. The Hidden Dimension; Anchor Books Editions: New York, NY, USA, 1966; ISBN 9780385084765. Gehl, J. Life Between Buildings: Using Public Space; Island Press: Washington, DC, USA, 1971; ISBN 9781597268271. Gehl, J.; Gemzøe, L. Public Spaces Public Life; The Danish Architectural Press and Royal Danish Academy of Fine Arts; School of Architectural Publishers: Copenhagen, Denmark, 1996; ISBN 9788774071877. O’donovan, S. 2012: Movement in Urban Space: Drake’s Place Garden Case Study. Available online: https://suzanneodonovan.wordpress.com/research-theory/2012-drakes-placegarden-_-m-arch-year-1/ (accessed on 3 September 2017). Maniello, D. Realtà Aumentata in Spazi Pubblici: Tecniche Base di Video Mapping; Edizioni Le Penseur: Brienza, Italia, 2014; ISBN 9788895315256 Dempsey, A. (2002). Styles, Schools & Movements. London: Thames and Hudson. Forum Nokia. Conceptual Design. (2011) [interactive]. Internet link: http://www.forum.nokia.com/Design/Design_process/Getting_started/Conceptual_design .xht ml Jonson, B. (2002). Sketching Now. International Journal of Art & Design Education 21 (3): 246253. Marcos, F., Pedro Segio Branco and Nelson Troca Zagalo. (2009). The Creation Process in Digital Art. Handbook of Multimedia For Digital Entertainment & Arts. Ed. Borko Fuhrt. NY: Springer Science and Business Media. 64

Nerdworking. (2011)[interactive]. Internet link: http://nerdworking.org/ Paul, C.2002. Digital Art. London: Thames and Hudson. Saglamtimur, Z. (2010). Digital Art. Anadolu University Journal of Social Sciences 10 (3): 213-238. Struppek, M. (2006). Urban Screens â&#x20AC;&#x201C; The Urbans Potential of Public Screens for Interaction. Intelligent Agent 6 (2). Special Issue: Papers presented at the ISEA2006 Symposium, August 2006. Udart, Video Projection and Motion Graph. The Projection mapping on the rise. (2011)[interactive]. Internet link: http://www.udart.dk/2010/02/17/projection-mappingon-the-rise/. http://www.imapp.ro https://www.theartistation.com/2016/09/imapp-2016/