More Creativity in Post-production

Fraunhofer IIS, the world’s renowned source for audio and multimedia technologies, announces its Department of Moving Picture Technologies will unveil new, innovative camera and acquisition technologies for lightfield technology at IBC. Fraunhofer IIS is introducing a camera array that makes it possible to capture more than one perspective in a single shot.

In the throng of the film set, camera operators have to determine the camera angle, the aperture, and depth of field of the camera. In the future, they will be able to change these parameters in post-production thanks to a new camera technology, lightfield.

And – Action! Today’s movie sets resemble an anthill with all of the actors, extras, cameras, and in between all of this, the director calling out his instructions. Amidst this chaos, the camera operator must make sure to have the correct settings, pay attention to the flow of the scene, and instruct the camera assistants simultaneously. This includes which camera angle should be assigned to which camera? Which part of the image should be sharp, and which should retreat, diffuse and be out of focus? Once the recordings are “in the can”, these parameters can no longer be corrected. At least, not until now.

An algorithm combined with a new type of camera array will enable these changes to be made retroactively in the future, thereby allowing more creativity in post-production.  Filmmakers can decide which area of the scene should be portrayed sharply or can virtually move the scene around for the “Matrix effect.” To create this effect, the actor is frozen in the scene, hanging motionless in the air, while the lightfield camera array moves around, capturing the scene from all sides.

Many perspectives instead of just one

Researchers at the Fraunhofer Institute for Integrated Circuits IIS in Erlangen, Germany, have developed a new camera array that makes that makes lightfield acquisition possible and will be exhibit it at this year’s IBC in Amsterdam.

“The array consists of 16 cameras in total, arranged in four rows and columns”, explains Frederik Zilly, Group Manager at IIS. “Instead of having just one, single camera as usual,

which records the scene from just one position, the 16 cameras collects light rays at various points in the plane.”

As noted, the array captures part of the lightfield from the scene, instead of only one specialized perspective. Although the array consists of 16 cameras, its cross section is only 30 cm by 30 cm (12 feet x 12 feet), making it convenient and easily employed on the set and in the studio.

But how does the lightfield array work? Why can we edit so much better retroactively?

The algorithm software estimates a depth value for every pixel recorded by the cameras. It determines how far from the camera array the object portrayed is located. Intermediate images can be calculated in post-production from this depth information. Therefore, we have virtual data not from just four columns and four rows of cameras, but from the equivalent of a 100 x 100 camera array instead. The outer cameras of the array have a different angle of view than the cameras located in the middle. As the camera operator films the subject, each of the outer cameras is able to look a bit behind the subject. After recording is complete, this allows the filmmakers to virtually “drive around” the subject and to change the camera angles and depth of field.

Fraunhofer IIS scientists have already developed the software for processing the recordings from the camera array. The graphical user interface (GUI) is also ready for recording on set, but is still being developed for the post-production. This should be finished in about six months. Fraunhofer IIS next plans to produce a stop-motion film as a test run of the software. This will be later used it as a demonstrational film to show the possibilities and opportunities offered by employing this camera array.

About Fraunhofer IIS

Founded in 1985 the Fraunhofer Institute for Integrated Circuits IIS in Erlangen, ­ today with more than 750 staff members, ranks first among the Fraunhofer Institutes concerning headcount and revenues. As the main inventor of mp3 and universally credited with the co-development of AAC audio coding standard, Fraunhofer IIS has reached worldwide recognition. It provides research services on contract basis and technology licensing. The Fraunhofer IIS organization is part of Fraunhofer-Gesellschaft, based in Munich, Germany. Fraunhofer-Gesellschaft is Europe’s largest applied research organization and is partly funded by the German government. With 20,000 employees worldwide, Fraunhofer-Gesellschaft is composed of 60 Institutes conducting research in a broad range of research areas.

For more information, contact Angela Raguse, angela.raguse@iis.fraunhofer.de, or visit http://www.iis.fraunhofer.de/en/abt/bewegt/.

About the Department Moving Picture Technologies

The Department Moving Picture Technologies develops new innovative imaging systems and procedures based on High Dynamic Range (HDR), Lightfield and 3D capturing methods. Main application areas are the motion picture and TV industry, but also other areas will be covered. The algorithms will be used to extend technical and creative opportunities on the set and in the postproduction. To achieve practical use specific components like image processing ASICs, software tools or complete prototypes and devices will be developed.

Well known software developments will be used, e.g. easyDCP for creation, play back and control of Digital Cinema Packages. Actual and future extensions will work for enhanced 3D distribution packages, multi-format mastering or archiving of media content. The department is well connected to other organizations and associations and is working in several international standardization organizations.