Virtualisation Chemnitz Opera House Part 1: Photogrammetry
To realise live events in virtual reality, 3D models of event spaces have to be created. In order for smaller cultural institutions to be able to benefit from this in the future, the effort required to create virtual event locations must be as low as possible. We have experimented with different methods for virtualizing a large opera hall and present our experiences to you.
There are various possibilities for creating virtual images of real event spaces. Usually, virtual worlds and rooms are modeled with programs such as Autodesk 3D Studio Max or Blender. Depending on the size, nature of the room, and the required degree of realism, this process can be very time-consuming. This applies particularly to complex event spaces, such as opera houses, which are very large and characterized by many details. The correct spatial geometries are difficult to determine, for example, on the basis of technical plans or 2D image material and the creation of replicas can be faulty. However, correct geometries are particularly important for our planned setting, the transfer of video recordings of the stage content into virtual space. Alternatives are methods that automatically generate the real environment. These include 3D scans and photogrammetry methods. Photogrammetry methods are characterized by a particularly efficient and simple implementation, but they also have disadvantages.
View into the Chemnitz Opera House © Die Theater Chemnitz
Using photogrammetry to automatically create 3D models from simple photographs.
Photogrammetry uses the parallax shift between several images taken from different viewpoints. This allows capturing complex 2D and 3D motion fields, from which a 3D model can be generated. Simultaneously, the programme creates a spatial scatter plot by evaluating the parallax shifts of images taken from different angles. While photogrammetry is mainly used to virtualize objects and smaller spaces with good results, the question arises as to what result can be achieved with very large and complex environments, such as an opera house. We have tested the use of current photogrammetry solutions using the Chemnitz Opera House as an example.
First 3D model of the Chemnitz Opera House, with many artifacts and unrecorded areas, based on photogrammetry
Limited accessibility of recording positions leads to inadequate results
Since opera houses and theatres in particular are almost constantly occupied, not only for events but also for rehearsals and technical tests, recordings must be completed in a relatively short time. For our first attempt, we took 1194 photos of all three levels of the hall of the Chemnitz Opera House using different camera positions. For the photogrammetrisation, we used the software „Reality Capture“ from Epic Games, which is a good represantation of the current technological state. For this purpose, so-called control points had to be marked manually in the image material so that the programme could assemble the different perspectives. The result reproduces the basic geometry of the opera house well and shows some good textures. However, the result contains many errors and artefacts. Many parts of the hall are not captured correctly despite the high number of images due to the accessibility of the positions. A disproportionate amount of additional effort is required to improve the result by using the available tools and further post-processing.
Optimised recording strategy using drones
In the second experiment, we chose an adapted shooting strategy. For this purpose, shots from the long shot at different heights were to be combined with detailed shots of the walls, floors, and ceilings in circular rounds. To facilitate and speed up the accessibility of the necessary perspectives, we used a drone. This allowed us to convert the shots from the open perspectives in a spiral shooting path from the bottom to the top, all the way to the ceiling. For the detail shots, the drone flew straight lines parallel to the walls in order to document parallax shifts as good as possible. With the use of a drone, accurate position changes can be made very quickly, which would not be possible by hand. In total, 1500 photos were taken.
3D point cloud of the Chemnitz Opera House (shading in the background) based on photogrammetry using the drone imagery; you can see the combined recording strategy of spiral long shots and rectilinear detail shots (white squares show camera position)
Here, too, manual re-encoding of the image material was partly necessary. However, the result was much better and offered a comparatively good overall optical impression. Most of the relevant geometries could therefore be captured correctly. The effort was reasonable with half a working day for the recordings on site and about two working days for the post-processing and conversion using the software. Nevertheless, the result contains many errors, especially in the area of occlusions such as the rows of seats. Deviations in the depth information of the scatter plot also lead to visual noise, which appears strongly negative, especially with flat surfaces. The high number of polygons of 1 million surfaces is also not performant enough for use in VR applications. Therefore, the model has to be remodelled manually in a further step.
