Cultural heritage 3D representation by using image manipulation and sketching
Target datasets
Cherish is a tool that aids archaeologists and designers to represent historical, urban and archaeological structures and sites in 3D by means of image manipulation and sketching. Cherish creation was inspired by the Yale’s Dura Europos dataset - a large image dataset of an ancient site which is located in modern Syria. One of the main characteristic of the data set is its spacial sparsity and featureless natures of photographs which does not allow us to utilize common computer vision methods in order to obtain 3D representation of the site.
The figure above displays a photo sample of the Dura-Europos dataset from Yale Universiry collection. Note the featureless nature of the surface textures due to the photo quality and the state of the remains - combined with sparse sampling of the site, we cannot simply feed the dataset to the well-developed algorithms like Bundle Adjustment in order to obtain 3D model of the site automatically.
The figure below demonstrates another “challenging” dataset - Horace Walpole’s Strawberry Hill Collection:
This dataset is composed of XVIIIth century views of Strawberry Hill following the sequence established by Walpole’s description of the Villa. Each room is located by a dot on an accompanying ground plan adapted from the published description. For the dataset like Strawberry Hill, there are no means to create a 3D representation of the site by automatic methods. Moreover, some of the paintings were not created by using strict laws of perspective, and therefore are not correct, perspective-wise, and cannot be used within methods like Single View Metrology, Automatic Photo Popup or Make3D to perform a single view-based 3D reconstruction.
The single-view based methods would not even work for the Dura Europos photographs due to the nature of the photographs. E.g., they have too much noise to obtain superpixels data, and do not contain any color information which is utilized within Automatic Photo Popup and Make3D methods.
For the above reasons, we are not able to use an automatic approach when trying to organize a dataset into a coherent 3D space. Therefore, it has to rely on user experience and interaction when performing the formation of the scene. For starers, let’s consider types of data we will be dealing with:
- Sparse datasets of featureless nature, e.g., old photographs, soil-like textures.
- Datasets consisting of non-photographics images, e.g., paintings, diagrams, drawings.
- Text descriptions and annotations that allow to draw some conclusions about the site.
- Expert knowledge and deduction based on the presented data.
Goal
The main idea behind Cherish is to use the knowledge and expertise of a specialist (e.g., archaeologist) in order to perform manual data organization, e.g. photos and sketching strokes within a 3D space. In other words, given a problem of disparate and disperse data types, Cherish is a tool that helps to combine the data and the expert knowledge into a comprehensive 3D space, i.e., it aids to make 3D sense of the data.
One of the main challenges when creating Cherish was to figure out what the final product looks like and what features and functionality it should include. So, the development process was iterative and was based on constant user feedback and testing that helps to refine the final look and functionality of the software.
Another challenge was figuring out what is the most intuitive and efficient step-by-step site reconstruction process that is easy to understand for a user with non-technical background. That also involves finding a right vocal way to communicate the main idea behind the Cherish and its terms to the user. In order to solve this, we tried to visualize the built sites by means of 3D printing and communicate the main technical terminology by using the realistic 3D printed demo examples.
Screenshots
An example of a 3D reconstructed Strawberry Hill room using sketching:
Features
Canvas
The primary means of representing any given site is through performing sketching and other 2D drawing procedures within a virtual plane. We define a canvas to represent such plane in 3D space. Since each canvas is only means to define a plane to perform 2D drawing and, in theory, is infinite in its extent, we created its representation as a finite bounding rectangle which embraces all the containing elements (e.g., strokes, polygons, etc.). Such representation makes it easy to see the canvas orientation in 3D. Moreover, the canvas can be positioned and rotated in 3D space using CAD-like tools for rotation and offset.
In order to denote what canvas the user is about to draw on, each canvas has a state which can be: current, previous or the rest. The information on the state is transferred through the color of canvas bounding rectangle, e.g., magenta - for the current, purple - for the previous and gray for the rest of canvases. Each canvas has a pickable area -i.e., when clicking on it with the mouse, the canvas gets selected and becomes current, while the previously selected canvas changes its state to previous, and the previously previous canvas takes a status of the rest.
2D entity
Any element that can be drawn within a given canvas is called a 2D entity, and these are the types of the entities: stroke, line segment, photo (textured rectangle) and polygon. Depending on the type of the structure, the entities should be used in combination, e.g., stroke would suit the best to define weathered and partial structures while line segment would work the best when doing the reconstruction of the rigid structures with clearly defined features, the photo entity serves exclusively as documentation piece while polygon can help to create occlusion to avoid having the look of wired structure that may becomes confusing once the complexity grows.
Bookmark
A bookmark is defined as a saved camera position with fixed camera parameters (e.g., focal length) with an attached state of the scene graph (e.g., visibility of each of the canvas and transparency levels of each of the photographs, etc.). Like any type of CAD system, the software supports the change of the current position by means of zoom, rotation and pan. The bookmarks come to be useful in cases when the user depends on the photographic documentation in order to introduce changes by means of drawing into the 3D model. For example, when user is working with the scene using two photographs, it is possible to change views from one photo to another in order to verify the scene structure from the both views.
User Manual
Basic functionality
- Camera navigation: zoom, rotation, drag.
- Canvas widget: state change, renaming, hiding, deletion, photo transparency.
- Bookmark widget: navigation between the bookmarks.
- Photobase widget: photo drag and drop.
- Camera panel: previous and next views, look at the current canvas, field of view.
- Model export: export to
*.objfile. - Canvas: creation, selection by using selection tool, editing canvas location.
- Strokes and 2D entities: drawing of strokes, selection, scale, rotation, positioning, deletion, cut-copy-paste.
- Line segments: drawing freely, snapping to canvas local coordinate axis.
- Photographs: insertion by using Photo widget, file types (
bmponly), photo manipulation - scale, rotation and positioning, deletion. - Polygons: color selection and drawing.
- Bookmarks: creation by manual positioning.
Beginner guide
Step-by-step reconstruction of Strawberry Hill exterior (pdf file).
Samples
Installation
Windows
For Windows platform, use the latest installer.
Linux
Unfortunately, there are no pre-compiled binaries for Linux, so the only way is to follow README instruction and install the necessary dependencies and compile Cherish by yourself.