Evaluation Of Marker Based Reference System For Head Pose Detection

Motivation and Overview

Computer vision-based head and complete human motion capture is a well-established area of research that has seen a lot of activity over the past two decades as a survey shows (Adelson & Bergen, 1991). The activity is motivated by a wide range of applications for robust motion capture, and the high complexity of the task gives rise to many still unsolved problems. The best-known approach which is also very convenient in terms of processing due to the coded identification of the landmarks, for capturing head motion is the use of markers as random landmarks. Cameras get the images of the markers in real-time and identify the markers, and a three-dimensional point cloud is generated. This approach has several drawbacks: the markers are invasive and can cause experimental artifacts, and recreating motion from the point cloud usually involves time- consuming manual post-processing (Kato & Billinghurst, 1999; Klein & Murray, 2006; “A.R.T. Advanced real time tracking.” n.d.).

However, in practice robust object tracking in real-time is quite challenging and remains an open problem. Reasons for that are manifold. First of all tracking systems require an initialization, i.e. providing the system with the initial poses of the object or camera. Once tracking is started, rapid motions causes’ image features to undergo large motion between frames which can cause visual tracking systems to fail. Furthermore, it can cause motion blurred images where extraction of feature correspondences may fail. The lighting during a shot can change significantly; specular reflections may confuse the tracker. Finally, complete or partial occlusions as well as large amounts of background clutter may result in tracking failure. Once tracking fails the system must be re-initialized in the same manner. Therefore, one of the main challenges in tracking remains automated initialization and recovery. Systems that are initialized by hand or require the camera to be very close to a specified position are not desirable options for automotive head tracking solutions.

The lack of a fast, reliable and automated initialization method is one of the main crucial problems that make many current tracking systems useless in the industry as well as in computer aided minimally invasive surgery. This has led to increased popularity of fiducial- or marker-based tracking system such as ARToolKit, or A.R.T. system, where the 3D tracking task is simplified to overcome the tracking limitations by detecting predefined artificial markers in every frame independently without constrains on camera pose. Using markers increases robustness and reduces computational requirements. However, it requires engineering the environment, which is not accepted in many industrial applications by the end-users and is sometimes even impossible, e.g. in outdoor environments. This has led to the development of systems in hybrid configurations involving expensive magnetic or inertial trackers. The difficulty of overcoming the vision based tracking limitations as described above stems from the need for fast and robust detection and pose estimation of objects in the scene from a single image without priors on the pose.

Rolling Shutter (Bundle Adjustment)

In bundle adjustment, a key modelling assumption is that each image is associated with one single camera pose. For a moving camera to satisfy the assumption, the whole image has to be captured at single time instance, referred to as the camera having a global shutter. This is the case for the CCD sensor which has been the dominant sensor in digital imaging. Due to cost and energy efficiency the CCDs are gradually being replaced by the CMOS sensor.

The rolling shutter camera on the iPhone takes 30ms to capture a complete image. Whereas in video mode the lens has a horizontal angle of view of 46.7◦ and the video format is 720p. For a horizontal panning motion where the camera is turned 90◦, the following table of rotation speeds and pixel displacements can be composed:

See Full Size >

See Full Size >

Q1 Current Situation:

• What is the state of the art in the field of computer vision-based Head-Pose Estimation?

Q2 Data acquisition:

• How can three dimensional information about the subject, be recovered from single camera video/images in real time?

Q3 Estimation:

• How can an underlying Head-Model be recovered more accurately from the three dimensional reconstructed data?

• How can the pose estimation algorithm be applied over the available data?

• How to modify the system in case of undesirable accuracy and precision?

Q4 Performance:

• How should the performance of the whole system be evaluated?

• How can the proposed pose estimation approach be used in a practical application?

Awareness of problem

The research effort originates from the awareness of an interesting problem. This awareness can stem from new developments in the field, or by studying related disciplines from which existing methods can be applied in novel ways. The result of this phase is a proposal for a research paper.

Suggestion

Existing knowledge or the theoretical base of the problem area is used to suggest possible solutions to the problem. This knowledge was acquired through a literature review of the field, and used to formulate a tentative design for a solution to the problem. This also involves making necessary assumptions about the solution and limiting the scope of the task to be manageable within the constraints of the project. Research questions Q1 and Q2 were answered in this phase.

Development

During the development phase the tentative design evolves into a complete design of the solution. An artifact is implemented based on the design. Algorithm was modified to answer Q3 .

Evaluation

The proposal defines a number of criteria for expected behaviour of the artifact. In the evaluation phase the performance of the artefact is assessed and compared against those criteria. Deviations from the expected behaviour is analysed, and hypotheses are formed about the shortcomings of the current solution. As the circumscription arrows in figure 02 shows these hypotheses lead to new awareness of problems, and a new iteration of the design cycle begins and an improved design is created. Concretely, for this project, the algorithms developed in the previous phase were evaluated using Robert Bosch datasets, and their accuracy, robustness, and processing time assessed, thus answering Q4a.

Conclusion

Eventually the research effort is concluded, at which point the artifact still might not fully satisfy all expected behavior, but is considered good enough. The results are consolidated and the new knowledge gained is documented. Anomalous behavior can serve as starting points for new research efforts. This phase resulted in a tentative answer to Q4b .

Contribution

The main contributions of this research effort are:

C1 A set of scripts for evaluating accuracy and precision

• Evaluation of the Calibration procedures (both Intrinsic and Extrinsic)

• Evaluation of Head-Tracker System‘s accuracy and precision

C2 Designing and Construction of Mechanical part of the project

• Hardware setup to perform tests for evaluating Position and Orientation accuracies and precisions

• Designing a Super-Marker by using multiple MCMXT markers to attain desired results

• Structure of the Helmet and Head-Model, allowing the feature points of the driver to keep visible for other camera systems.

C3 A novel marker-based Head-Pose estimation method:

• A General-Purpose computation on Graphics Processing Units (GPGPU) implementation of a fully parallel thinning algorithm that is well suited for use in Head-Pose estimation, and achieves real-time performance.

• A pose estimation method based on constructing a Head-Model by using a tool named Photomodeler based on the markers placement over the Helmet. The configuration of the Head-Model is independently structured in an offline tool, which overcomes the limitation of processing time, geometrical structure of various drivers’ heads and provides a pretty accurate 3D model. The method achieves real-time performance on any possible model.

C4 Transferring the data to the central database (ADTF) via UDP protocol

C5 MATLAB scripts for extracting the desired data from ADTF in a structured form

C6 Evaluation and Authentication of the modified Head-Pose Estimation Algorithm

See Full Size >

Intrinsic Calibration Pattern.

Extrinsic Calibration Pattern.

3D Head Model

Generating a 3D Head-Model by using multiple markers over the helmet of the driver is really important in terms of knowing the exact relative positions of the markers. Coordinate values of this model will be further used by the pose estimation algorithm as 3D correspondence points of the markers and perspective matching of the these points will be done with the image points visualized by the camera.

The most vital point at this stage was to identify a way to elaborate the head model that could either have been done manually by measuring the relative distances and orientations of the markers or by using an independent offline tool to visualize and generate a head-model. This task was first done manually for some simple structures or formations of markers placement. But unfortunately to achieve the same level of accuracy for more complex or real head structure, the manual approach failed. So, a tool called “PhotoModeler” was employed up to generate an accurate Head-Model in an automated manner.

Camera Calibration Using PhotoModeler

PhotoModeler Scanner is a 3d scanner that provides results similar to a 3d laser scanner. Scale-independent object modeling - model small objects or big scenes (“PhotoModeler Scanner,” n.d.). Usual calibration algorithms are used by PhotoModeler for the cameras, not very different from the techniques used by SpACiAL software. Various numbers of specified patterns containing dot-markers are used from several different views, for the intrinsic calibration of the camera. These markers are automatically detected, marked and referenced by the software, whereas in case of other markers it has to be done manually or by setting up a target-code.

See Full Size >

See Full Size >

Pose Estimation

Head pose estimation is an important part of the human perception and is therefore also relevant to make interaction with computer systems more natural. However, accurate estimation of the pose in a wide range is a challenging computer vision problem. It basically gives the orientation and position of an object in the relative coordinate system.

The requirement was to achieve a goal of ±1degree and ±1mm of accuracy in orientation and position within the given Head-Box, respectively (Martin, Camp & Stiefelhagen; 2014).

See Full Size >

MXT MT Multicam

MXT_MT_Multicam implements the Fraunhofer IOSB MXT library. It detects badges in the provided image streams and forwards the image streams to the next plug-in. In addition it adds an output image per input image (DevEnviroViewerPluginInterface::ImageType::IMG_T_LIST_DOUBLE) at the end of the image data list, which contains a list of lists with all detected badges in the stream. It contains the badge ids, radius in pixel, center points and the position of satellites as well. The final look of super-marker and helmet by using techniques.

Like point to point matching and triangulation.

See Full Size >

Recommendation

According to my analysis and experience, I would recommend using this modified system if the required accuracy level is ±4mm and ±4.5° along with high precision of ±0.6mm and ±0.2°, also where the robustness is not a highly required parameter. In case of more accurate applications try different system with high accuracy, precision and robustness.

Future Perspectives

To further increase the level of accuracy, two plugins were designed in order to merge/split the images coming from two cameras and tried to implement the epipolar technique, which was not in the end implemented and tested. Whereas trying this approach along with calculating highly accurate 3D model data might make it possible to achieve the required accuracy.

The 3D model of the helmet also has a great influence over the accuracy and hence several methods were used to get the accurate model-structure of the helmet. A test was constructed to evaluate the accuracy of the 3D head-model by using the orthogonal target, which was designed over CAD and printed out with an accurate 3D printer as shown in figure 03 , where the relative distances and angles of all the points are known and the data of the Photomodeler was needed to evaluate with the known-one. This test was not accomplished due to the time limitations but in future this could be a beginning point to proceed from.

See Full Size >