VIRTUAL PRODUCTION: LED VOLUMES

INTO THE VOLUME: FIVE INNOVATIVE FIRMS WITH LED STAGES EYE YEAR OF GROWTH

By CHRIS McGOWAN

TOP: The set of the ship’s deck in 1899 with actors Emily Beecham and Andreas Pietschmann to the right, in front of DARK BAY’s LED Volume. The setup included input from Framestore, ARRI Solutions Group, FABER AV, ROE Visual, Vicon Motion Systems, Epic Games, Helios Megapixel VR and Netflix’s Product Innovation team. (Image courtesy of DARK WAYS and Netflix)

OPPOSITE TOP TO BOTTOM: The deck of the ship in front of the LED wall seen from above during the filming of the 1899 series at the DARK BAY LED studio in Potsdam-Babelsberg, Germany. (Image courtesy of DARK WAYS and Netflix)

A car commercial at the MELS LED Studio, which had over 20 productions in 2021, including Transformers 7 and Disappointment Boulevard. (Image courtesy of MELS Studios)

At Studio Lab using an LED wall for a remote location and accurate reflections. (Image courtesy of Studio Lab) These are the formative years of virtual production, full of challenges and discoveries. Here, executives of five prominent firms with LED stages – DARK BAY, MELS Studios, Virtual Production Studios (80six), Studio Lab and Pixomondo – discuss the arc of their growth and some of their latest developments and innovations.

DARK BAY

The genesis of the DARK BAY LED stage in Potsdam-Babelsberg, Germany, was driven by Baran bo Odar and Jantje Friese, the creators of Netflix’s Dark, who were working on ideas for their next series. 1899 was originally imagined as a pan-European show with vast backlot builds and locations spread over four countries in Europe. Then came the pandemic. Odar and Friese came up with the idea to shoot 1899 in a LED volume, which seemed the best solution to realize their vision while keeping the cast and crew safe. In addition, building their own stage made sense considering the scale of the series, according to DARK BAY Managing Director Philipp Klausing. Thus came about the DARK BAY Virtual Production Stage, owned in a majority share by Odar and Friese’s production company Dark Ways and in a minority share by Studio Babelsberg. DARK BAY was financially supported by funding from the Ministry of Economic Affairs, Labour and Energy of the State of Brandenburg (MWAE) and backed up by a long-term booking commitment from Netflix. The facility, with an LED wall 180 feet long and 23 feet high and an active shooting area of 4,843 square feet, is located on the lot of Studio Babelsberg and came online in May 2021.

ARRI was involved in the DARK BAY planning and installation, and offered its know-how throughout the setup. The ARRI Solutions Group specializes in setting up mixed reality studios

around the globe and was a crucial support, according to Klausing, who says, “ARRI was a big help in finding the look of the project. One of the biggest supports we received was the customization work on the ARRI anamorphic lenses.”

Framestore also played an active role in setting up the volume, which happened in parallel with the pre-production for 1899. Framestore took responsibility for the virtual content that played on the LED wall, according to On Set VFX Supervisor Andy Scrase. “Artists, technicians and myself, all from Framestore, were involved very early in pre-production with the decision-making process for the different sets,” remarks Scrase. “Our VAD team in London then started building our digital environments several months in advance, which we began to review and adjust as soon as we were set up on the DARK BAY stage in Babelsberg. Once we’d completed this, there was the important task of pre-light and making sure our virtual sets matched the physical ones [which was done at different stages during the schedule]. When shooting started, the Framestore team made sure our content played as it should on the wall and that it tracked correctly with cameras on set.”

Netflix’s Product Innovation team, along with its VP division, also provided assistance in setting up the stage. “Andy Fowler, Vice President of Product Innovation, and Girish Balakrishnan, Director of Virtual Production, saw the potential and guided us in the right direction,” notes Klausing. “Without their trust and passion to enable virtual production, we would not have been able to get the stage up and running in such a short time [three months].” Other contributors to the DARK BAY setup included FABER AV (delivery and construction of all LED elements [ROE Ruby 2.3, CB3 and CB5] ), plus Vicon Motion Systems (Delivery of Tracking System), Epic Games (support for any Unreal-related

TOP: The MELS LED stage in Montreal has a diameter of 65 feet, height of 20 feet and a fully motorized and computerized ceiling. The volume is composed of ROE BP2V2 and is powered by Brompton processors, Unreal Engine and disguise technology. (Image courtesy of MELS Studios)

BOTTOM: Behind the scenes of Star Trek: Discovery, Season 4, filmed on a Pixomondo Toronto stage, that shows the physical lighting hanging from the ceiling. The Discovery DP decided that he wanted to use physical soft boxes for more traditional control of his lighting. The perspective on the wall is skewed and stretched due to the high angle of the camera (seen at the end of the Technocrane). (Image courtesy of Paramount+ and CBS Interactive) topics), and Helios Megapixel VR (support during processor implementation).

For productions utilizing DARK BAY’s and other LED stages, Klausing explains, “The biggest underestimated benefit is that you are able to finish a day of shooting and have an almost finalized shot, which has a big impact on the precision of the edit.”

For 1899, creating the content for a virtual production studio was a collaboration between the art director and VFX supervisors, all under the supervision of the director, the DP and the production designer. Klausing notes, “All departments need to talk a lot to each other. [It is] a collaboration which was in the past always separated by production and post work. Working with an LED studio like DARK BAY enhances creative freedom and control. It allows the merging and definition of every aspect of a production, all at the same time. Beyond that, it’s very satisfying to see actors performing in an environment that comes closer to reality than classic greenscreen setups.”

One of DARK BAY’s innovations is a revolving stage inside the LED volume. Almost the entire interior of the volume consists of a motor-driven turntable with a diameter of 69 feet and a load capacity of 25 tons, which allows filming the environment from all possible angles. This removes the common physical limitations of shooting in an LED volume and can reduce or remove the time needed for a set rebuild, according to Klausing. “Complete sets can be rotated 360° in just three minutes,” he says, “and entrances and exits can be simulated through a trapdoor built into the stage. We are convinced that we will see this revolving floor in all studios in the future.”

VIRTUAL PRODUCTION STUDIOS (80SIX)

80six’s Virtual Production Studios stage is located in Slough, U.K. and debuted in March 2021. The facility has expanded and now has a footprint over 10,000 square feet, according to Aura Popa, 80six Marketing Manager, “which enables us to build to spec large-scale LED volumes of any shape and form, up to eight meters high.” She continues, “Rather than having a pre-configured LED stage of a standard shape, we’ve discovered that this is the only viable approach to make virtual production with LED screens accessible to productions of all budgets. We wouldn’t be able to do this if we didn’t have one of the largest ROE Visual Diamond 2.6mm inventories in the U.K., a leading LED panel currently used for in-camera VFX with incredible cinematic results.”

Popa adds, “Creatively, cinematographers and the VFX teams especially embrace our flexible offering as it gives them the chance to work alongside our tech team to decide the specs of the LED volume, depending on the size of the shot, the nature of the physical props required and the camera movement. We’ve installed a multitude of LED volumes for best in-camera VFX results that match the interactive light from the LED panels to the lighting in the location, capturing the reflections and shadows inherent to it.”

As noted above, virtual production creates new schedules. Popa explains, “With in-camera VFX, post-production starts in pre-production. You need to come to the set ready with a solid shooting and lighting plan, after a considerable testing period. Usually, the director, cinematographer, VAD team and the Unreal Engine Artist [if Unreal Engine is used] come together early in pre-production to create the virtual set and pre-light it, so all the elements of the set, “Probably the most critical area in the pipeline is the color workflow. We have spent a large amount of time to ensure the color remains the same as it flows through Unreal onto the LED panels, through the camera and finally to the DIT for the final look. The calibration of the LED panels is checked frequently and we develop custom LUTs for each camera and also for each show, which is finessed with the DP and DIT.” —Phil Jones, VFX Supervisor - Virtual Production, Pixomondo

TOP: Behind the scenes of a “100% Shutterstock” ad shoot at Virtual Production Studios by 80six, in Slough, U.K. (Image courtesy of Virtual Production Studios)

TOP: The Ni’var Science Institute, from Star Trek: Discovery Season 4, Episode 3, was shot on a Pixomondo LED stage in Toronto. (Image courtesy of Paramount+ and CBS Interactive)

BOTTOM: The Studio Lab team dials in camera settings for a multi-cam LED virtual production shoot of a forest setting. (Image courtesy of Studio Lab)

OPPOSITE TOP: Shooting Netflix’s live-action Avatar: The Last Airbender on a Vancouver Pixomondo stage. Unreal Engine was the primary software, and the setup includes ROE’s Black Pearl II V2 panels, Brompton processors, and OptiTrack or Vicon Tracking systems. (Image courtesy of Netflix) including physical and virtual, are known and have been tested.”

Popa thinks LED volumes offer incredible flexibility in terms of simulated settings. “You are now inside an LED studio, a highly controllable space, and with the help of the LED screens and realistic graphics you can bring to life any location you can imagine – from locations that don’t even exist to difficult locations to reach. During a single day, you can shoot a variety of different environments with the exact lighting conditions of that location. [And] inside an LED volume, you can have the ‘golden hour’ for seven hours. This level of control on the shooting conditions has never been achieved before.”

Virtual Production Studios seeks to design a “seamless VP workflow with Unreal Engine, aided by disguise media servers which speed up the virtual production workflow.” Popa comments, “It is a different approach to the VP workflow which helps streamline it by facilitating a stable communication between Unreal Engine and the camera tracking system, Mo-Sys StarTracker, through disguise’s innovative RenderStreams.”

STUDIO LAB

Studio Lab was born from a desire to provide world-class studios and gear to creatives across a variety of disciplines, according to Studio Lab Director Benjamin Davis. It officially opened to the public in April 2020. Located in Derry, New Hampshire, the stage has a 14-ft.-high x 52-ft.-long LED wall.

Davis decided early on that there are three main reasons to use an XR LED stage as opposed to using traditional sets or shooting on location. “The first is if the project is created for an environment that does not exist. Think stylized environments or an alien planet. Then, of course, XR is the perfect creative solution for the imagined world.”

He continues, “The second reason for an XR [LED volume] project, would be if the shoot requires a visit to an actual recognizable location that would otherwise be prohibitive due to cost or

“The first thing [when working with virtual production and LED stages] is to get the proper gear. Not all video walls, drivers and cameras work well for virtual production, and one can spend millions on a solution that seems like it should work, only to come up short because one piece of hardware doesn’t perform the way you expected it to. Do the research and spend the money to get the correct solution.” —Benjamin Davis, Director, Studio Lab

scheduling. This would be things like sports venues, landmarks, or specific buildings. And thirdly, if the project requires shooting in multiple locations in a short amount of time, XR can save time and money by having all locations in one place while allowing you to move between them quickly.”

In general, Davis notes, “We decided to put our focus on making the XR process and workflow as easy as possible. This equates to days of testing before a shoot and ensuring color, spatial and lens calibration is correct before the start of the shoot. We also provide each project team with easy-to-use tools that allow for changes to be made on the fly with XR Stage App Integration. Teams are able to directly find exactly the settings they need using controls via an iPad. Providing DMX controlled lighting and environmental controls, such as weather and time of day, are good examples of this.”

One of the biggest breakthroughs for Studio Lab came fairly early on with the ability to support live camera switching inside of virtual environments, according to Davis. “In addition to aiding traditional shooting, especially for video/stills in the same shoot, this makes the technology available to us for live broadcasting and opens a host of possibilities. None of these options would be possible without the use of disguise and they have been an excellent partner throughout.”

To have success with virtual productions with LED stages, “the first thing is to get the proper gear,” says Davis. “Not all video walls, drivers and cameras work well for virtual production, and one can spend millions on a solution that seems like it should work, only to come up short because one piece of hardware doesn’t perform the way you expected it to. Do the research and spend the money to get the correct solution.”

PIXOMONDO

Pixomondo has two LED volumes in Toronto and two in Vancouver. “We are currently planning to build more stages in North America as well as Europe,” says Phil Jones, Pixomondo VFX Supervisor - Virtual Production. Avatar: The Last Airbender, Star Trek: Discovery and Star Trek: Strange New Worlds are among the major productions that used Pixomondo stages last year. Its first such stage in Toronto launched in January 2021 with an LED volume internal stage that is 72 feet in diameter, 90 feet in length and 24 feet high.

TOP: An Open Cube LED volume built to specs at Virtual Production Studios by 80six. (Image courtesy of Virtual Production Studios)

BOTTOM: The Studio Lab team showcases a Ferrari in an XR virtual production using LED wall, ceiling, and moveable/trackable wall sections. (Image courtesy of Studio Lab)

OPPOSITE TOP: A virtual production takes place in front of a ROE LED wall in VA Studio Hanam, near Seoul. South Korea took a big step forward in VP in late 2021 with VA Corporation’s establishment of VA Studio Hanam, a virtual production facility located in Hanam, created in partnership with South Korea’s ARK Ventures. VA Studio Hanam has three main studios with large walls featuring ROE LED panels, powered by Brompton Technology’s Tessera processing. Studio C is Asia’s largest virtual studio, according to VA Corp., with a floor area of 1,088 square meters and the biggest oval LED wall in Korea, with ROE Black Pearl 2 panels and ROE Black Marble on the floor. (Image courtesy of ROE Visuals)

In terms of the new scheduling required for productions utilizing LED stages, “We are now coordinating with the client creative team even earlier in production, as compared to traditional VFX,” says Jones. “In the early script stage, our virtual art department works with the lead creatives on the production side, building 3D concepts for each environment. Starting the environment builds in the script concept stage helps to ensure the creative and aesthetic needs are met well before the shoot day. These concepts are then used as the base for the remainder of the work required to finesse and finalize the environment.”

Jones notes that review sessions have changed somewhat, particularly in that many of the notes can be handled in real-time, “which obviously speeds up the process. In some of the client review sessions, after creative notes are given, we spend some time blocking out shots with the director and DP in real-time. This is very valuable for all involved, and it also identifies the areas of the environment that will require that little bit of extra love!”

For Jones, the disciplines required to create the environments for the wall are very similar to those required in a normal VFX pipeline, with a few additions. “We are using many of the same tools from our VFX toolset and have added Unreal to the artist’s toolkit.” He adds, “There are some areas that differ from our regular VFX pipeline, and we have some specialized pipeline and coding artists who have added new tools and processes to blend Unreal in to our workflow. They have created specialized tools that we use throughout the process, including such areas as asset creation, element transfer, environment reviews and Unreal

real-time optimization tools.”

He explains, “If a shot does require some post-VFX work, such as the addition of a CG character, we have developed tools and processes to ensure the data from the selected take in editorial makes its way to the appropriate VFX shot in the post pipeline. Due to our close relationship with Epic, some of the tools we have developed are being wrapped into future releases in Unreal Engine.”

In working with the LED volume, Jones notes, “Probably the most critical area in the pipeline is the color workflow. We have spent a large amount of time to ensure the color remains the same as it flows through Unreal onto the LED panels, through the camera and finally to the DIT for the final look. The calibration of the LED panels is checked frequently and we develop custom LUTs for each camera and also for each show, which is finessed with the DP and DIT.”

For Jones, the most gratifying part of shooting on the LED stage “is seeing everyone’s reaction when they first step into the volume. Instead of having to imagine what will be placed beyond the practical sets and props within a massive green screen, everyone can see and react to the filmmakers’ intent right there on the day. This excitement has also been seen in all the artists who came to the stage and saw their work on an unusually large screen. It is always fun to watch them walk around inside the work they created.”

MELS STUDIOS

The MELS Studios LED volume bowed in February 2021, and had over 20 different productions last year, including Transformers 7 and Disappointment Blvd. The volume is situated in a 10,000sq.-ft. stage in Montreal, where MELS can offer a turnkey service and additional services, like grip, camera, lighting and insert stage on demand, according to Richard Cormier, MELS Studios Vice President Creative Services and Executive Producer – Virtual Production. “We have built and designed on specs a second volume for specific production in addition to our permanent stage.”

Cormier notes that the permanent MELS LED stage has a diameter of 65 feet and a height of 20 feet, but “what makes ours special is the ceiling that is fully motorized and computerized. It is separated into two 20 ft. x 20 ft. panels that can be programmed within a millimeter. It gives production full and precise control over reflection and dynamic lighting, and it can host as many SkyPanels as production wishes.”

Work in the volume caused two main collaborations to emerge for MELS. “One is our involvement with the set design team from the onset all the way through shoot day,” says Cormier. “The other has to do with our involvement with the VFX team on the asset preparation so we can have the ultimate flexibility on shoot day.”

On the innovation front, he adds, “The one I can share is that we have the possibility of recording a greenscreen and our digital set at the same time, which can give compositing the perfect greenscreen and the perfect scene lighting.”