2D Color and Shape from 3D data
Full Resolution: Color Layer over Surface Shape Layer for lower panel 5
Diego Rivera, The Marriage of the Artistic Expression of the North and of the South on This Continent, also known as Pan American Unity, 1940, 6.7 meters tall by 22.47 meters wide (22 X 74 feet); courtesy City College of San Francisco; © 2020 Banco de México Diego Rivera and Frida Kahlo Museums Trust, Mexico City / Artists Rights Society (ARS), New York; Color and Shape Images - © Cultural Heritage Imaging.
The Color Layer
The color layer shows the colors of Pan American Unity exactly where Diego Rivera painted them. This color layer, called an orthomosaic, is made with information derived from the 8.1 billion colored points in the mural’s 3D model. (learn more below)
The Shape Layer
The shape layer uses changing color to show the working imprint of Rivera’s brushwork in the fresco’s wet plaster surface. As the depth of the textured plaster changes, the color of the surface changes. The shape layer, called a Digital Elevation Model (DEM), is made from the shape information acquired in the mural’s 3D model.
These 2D outputs are a result of 3D capture of the mural surface using photogrammetry. CHI used 2469 50-megapixel images to create the 8.1 billion point 3D model.
The 3D Photogrammetric Data
Pan American Unity is a big mural. Documenting it required a big 3D model. The 3D documentation was made from 2469 50-megapixel photographs using the photogrammetry technique. The photographs were used to construct a scaled 3D model made up of 8.1 billion shape and color measurements. A wealth of information about the surface shape and color of the mural can be seen in the model. However, the extent of this information poses a problem: the 3D digital representation requires a huge amount of computer resources and bandwidth to display on the internet.
To make this information available to the general public Cultural Heritage Imaging (CHI) used the 3D data to make two special types of high-resolution 2D images; orthomosaics and Digital Elevation Models (DEMs). Together, this pair of orthomosaic and DEM high-resolution images show both the surface color and shape of the mural. This image pair conveys most of the information contained in the massive 3D mural model, but requires fewer computer resources to view and can be experienced by many more people.
Each of these 2D images is 163,861 pixels long and 49,177 pixels high or 8.06 billion pixels (8.06-gigapixels) in total. On the mural, the tiny square area each of these pixels represent is 0.139mm in length: for context, a thick human hair is .100 mm wide. This means that every square millimeter of the 6.7 meter by 22.5 meter (22’ by 74’) mural is covered by 51.75 pixels. These, have the resolution and precision to show the shape of an individual bristle stroke of Rivera’s paint brush and the path it took through the fresco’s wet plaster. Additionally, condition issues, such as cracks in the surface are also clearly visible in the 3D data.
By its nature, photogrammetry produces 3D models with perfect alignment between the imaging subject’s color and surface shape. This is because, in photogrammetry, each 3D point in the surface is built by triangulating multiple pixels matched from multiple photos taken with different viewpoints on the subject. Photogrammetry software keeps track of these pixels –– and their colors –– and assigns these colors to the corresponding surface point in the 3D model. The color and shape representations inherit this exact registration.
The Orthomosaic (Color)
CHI built the color orthomosaic of Pan American Unity using the same set of color managed, high-resolution digital photographs that generated the 3D model. The 3D model was built from these images using a technology called photogrammetry. Information generated during the photogrammetric process that was used to build the mural’s 3D model was also used to build the orthomosaic.
The most common use of orthomosaics is in the construction of maps of the earth’s surface. Orthomosaic maps are usually made from multiple orthophotos. Each pixel in an orthophoto represents each feature on the ground as if it was directly overhead, perpendicular to the feature it represents. Also, in an orthophoto, each pixel covers the same distance on the ground. Because each pixel represents the same distance, maps can be used to measure distance. In other words, each pixel in each orthophoto used to make an orthomosaic map is exactly where it should be and represents the features on the ground with their true size and position.
Orthophotos cannot be captured directly by a camera. Orthophotos used to make maps are made from aerial photos, which undergo a metamorphosis –– much as a caterpillar becomes a butterfly. Caterpillars use information in their DNA to become butterflies: aerial photos use information generated by photogrammetry to become orthophotos. When a caterpillar is inside its cocoon, DNA instructions rearrange its molecules, transforming a crawling creature into an insect that can fly. When an aerial photo is transformed by photogrammetry, knowledge gained by building a 3D model of the terrain shown in the photo is used to rearrange the photo’s pixels so they look as though all the features on the terrain were viewed simultaneously from directly above; the aerial photo becomes an orthophoto.
When the locations of the pixels in the aerial photo are transformed, the photogrammetric process removes a variety of distorting errors that come from:
- Perspective –– the convergence of parallel lines, like railroad tracks, to a point in the distance
- Optical distortions –– the various effects produced by light passing through the camera lens
- View angle distortions –– the keystone-like effects of taking a photograph from an angle –– the tilt of an airplane taking an aerial photo for example
- Surface topology near/far scale differences –– the effect that makes closer objects seem bigger and farther objects seem smaller
- The curvature of the earth
Once these errors are removed, the final act of the metamorphosis is to place the pixels on a virtual 2D plane oriented parallel to the earth’s surface. To make a larger map, more orthophotos can be added to the same virtual plane. The result is a map that shows features on the earth in their true locations and at their true size and shape.
How CHI Built the Pan American Unity Orthomosaic:
The orthomosaic for Pan American Unity was built in a similar way. CHI built the 3D model from 2469 50-megapixel photographs, using photogrammetry. CHI used 1970 of these photos of the mural to build the orthomosaic. Each of the 1970 photos was transformed into an orthophoto using the knowledge gained from building the 3D model.
The orthophotos needed to be built on a virtual plane running parallel to the plane of the mural panels. However, doing this was complicated because the installation of Pan American Unity’s 10 panels in City College of San Francisco’s Diego Rivera Theater was curved. In the theater, the curve in the mural was made from five pairs of panels, each pair composed of a lower and upper panel. Each of these panel pairs were on the same plane, forming a flat surface. CHI created five separate orthomosaics, each one oriented parallel to one of the five pairs of installed panels. CHI then aligned the five orthomosaics next to each other in 2D space and joined the images into a single orthomosaic of the entire mural. This operation digitally “flattened” the 3D generated orthomosaics. CHI created the 3D generated Digital Elevation Model images of Pan American Unity in the same way; to show the mural as Rivera had painted it and intended it to be seen.
Orthomosaics vs. stitched images: Orthomosaics are not like other types of high-resolution images that are “stitched” together from many smaller photos; they are free from the distortions inherent in photography.
Because they are made from 3D information in a 3D environment, orthomosaics can provide very high-resolution images, free of near/far, optical, perspective, or point-of-view induced photographic distortions of the imaging subject. Just like orthomosaic maps of the earth made using photogrammetry and aerial photography, in the orthomosaic of Pan American Unity each pixel in the orthomosaic is exactly where it should be and represents the features on the mural exactly as they are.
In contrast, high-resolution stitched images, no matter how carefully made, “pull and push” pixels to make the constituent images “line up.” This is because stitching operations lack the optical camera calibration, camera position and orientation, and surface relief information generated by building photogrammetric 3D representations of the imaging subject. Even if some optical distortion correction tools exist in the stitching software and the image “looks” fine, the errors from the other sources produce a result with many incorrectly located pixels in the image and the failure to accurately represent the surface of the imaging subject.
Accurate Measurement: The orthomosaic of Pan American Unity can be directly measured in a 3D environment.
The Digital Elevation Model (DEM) (Shape)
What are DEMs? Digital Elevation Models (DEMs) were originally developed to show terrain elevations in 2D representations made from 3D maps of the earth’s surface. These 3D maps of the earth were made using aerial photogrammetry. CHI used DEMs in a similar way to show the 3D shape of Pan American Unity’s surface. DEMs use changing color to visualize the height and depth of each point of the worked plaster texture on each fresco panel. This shape information is derived directly from the 3D point cloud of the mural built using photogrammetry. The DEM shows each fresco panel’s ~ 2.5cm (1 inch) range of height difference represented by a color scale with deep blue showing the lowest points and bright red showing the highest. Light blue, green, yellow, and orange show the relative intermediate heights. The distance between the highest and the lowest point on each panel pair varies slightly. Here is a DEM color key that shows the exact depth to color relationship for each panel pair.
The DEM shows the construction history of the mural. By zooming in close, individual brush strokes and even individual brush bristle paths can be seen carved into the wet plaster. Rivera’s brush action often made fine, bas-relief sculptural portraits of the people in the mural. This sculptural quality can be seen in the face of Samuel Morse shown below:
On a larger scale, the DEM clearly shows the boundary regions of the area of wet plaster (the giornata) that was applied for each day’s painting. Outlines of each giornata in comparison to the same area in the DEM show how Rivera wove the giornata boundaries into the mural composition. (Learn more about giornata at Through the Eyes of a Conservator)
The mural is covered with complex patterns of surface relief. Almost all of this relief is invisible to the naked eye when the fresco is observed in person due to the overriding prevalence of the mural’s color information in human perception. When the mural’s color is digitally removed, the remaining shape information becomes visible. The DEM and the colored orthomosaic in this project are perfectly aligned. This makes it possible to compare an area in the orthomosaic image with realistic color to the same area’s shape relief in the DEM image. By moving the sliders that control transparency in the image’s layer panel, the user can explore the mural with varying amounts of shape and color information. (See the video embedded above which describes this in more detail)
How CHI Built the Digital Elevation Models (DEMs)
CHI built the DEMs first, followed by the orthomosaics, because the surface relief height information carried by the DEMs is used in the construction of the orthomosaics. The DEM is used to correct the near/far scale distortion during the construction of the orthophotos used in the orthomosaic. (See the orthomosaic section above)
For the DEMs and the orthomosaics to successfully show Pan American Unity’s integrated shape and color information, exact spatial registration between the images is necessary. The DEMs and the orthomosaics are both built from the mural’s 3D data. Each of the 3D model’s colored points in virtual space becomes a pixel when the 3D data generates a DEM. To ensure exact pixel-to-pixel registration between the images, CHI carefully selected the set of 3D points that would be used to build each image set by drawing a boundary along the edges of the mural panels in the 3D model. Only 3D points within this boundary line were used to generate pixels in the corresponding DEM and orthomosaic. This resulted in the DEM and orthomosaic having exactly the same number of pixels in the same relative spatial location in each 2D image. When the user compares the combined DEM and the orthomosaic layers using the transparency sliders in the control panel, a feature on the mural will always appear in the same place in both images.