• TestF3F - Results

TestF3F Photometric Function Sensitivity Test Results

Definitions

Key Findings

• The Lommel-Seeliger Photometric Function subtests (F3F1 and F3F2) performed well with small differences in measures of accuracy and correlation.

• The Clark and Takir Photometric Function subtest (F3F3) performed poorly throughout testing, with pervasive degradation of the digital terrain with every processing step conducted, eventually failing to align/correlate images for a subset of landmarks at the 5cm tiling step.

• CompareOBJ RMS with and without optimal translation and rotation shows an inability to distinguish between subtests which perform well and subtests which perform poorly. Indeed, there is no indication from the CompareOBJ RMS that the poorly performing F3F3 subtest failed to complete the 5cm tiling processing step.

• There is also no indication of performance from an inspection of the RESIDUALS RMSs.
• The normalized cross correlation scores are clearly distinct for the well-performing and poor-performing subtests.
• A single North-South transit through the center of the evaluation region immediately indicates good and poor performance at all resolutions tiled.

Results and Discussion

Results from testing the three photometric functions split into two groups characterized by differing digital terrain accuracy and model behavior. Subtests F3F1 and F3F2 (Lommel-Seeliger photometric function without the 2 and with the 2 respectively) performed well with minor differences in the measurements of accuracy, whereas subtest F3F3 (Clark and Takir photometric function) performed poorly with pervasive degradation of the digital terrain with every processing step conducted. A detailed analysis of the behavior of F3F3 is reported here:

Test F3F3 - Analysis.

CompareOBJ RMS

Three CompareOBJ RMS values for the final 5cm resolution 20m x 20m evaluation bigmap are presented for each subtest and each S/C position and camera pointing uncertainty:

• The largest CompareOBJ RMS (approx. 57cm across subtests) is obtained by running CompareOBJ on the untranslated and unrotated evaluation model.
• The second smallest CompareOBJ RMS (approx. 15cm across subtests) is obtained by running CompareOBJ with its optimal translation and rotation option.
• The smallest CompareOBJ RMS (approx. 6cm across subtests F3F1/2) is obtained by manually translating the evaluation model and searching for a local CompareOBJ RMS minimum.

The CompareOBJ optimal translation routine is not optimized for the evaluation model scale (5cm pix/line resolution). Manual translations of the bigmap were therefore conducted in an attempt to find a minimum CompareOBJ RMS. The manually translated evaluation models gave the smallest CompareOBJ RMSs.

The CompareOBJ RMS without translation or rotation is similar across subtests showing an inability to distinguish performance differences apparent from visual inspection of the evaluation maps, the normalized cross correlation scores, and the failure of F3F3 subtest at the 5cm tiling step. The CompareOBJ RMS with optimal translation and rotation is little better at distinguishing performance with some decrease in RMS of the poor-performing F3F3 subtest when compared with the well-performing F3F1 and F3F2 subtests. CompareOBJ with manual translation shows the most ability to distinguish between good- and poor-performing subtests, but the RMS of the poorly performing F3F3 subtest is still unexpectedly low.

CompareOBJ RMSs do not change with iteration.

CompareOBJ with Manual Translation - RMS:

CompareOBJ with Manual Translation - Translation:

RESIDUALS RMS

Again, there is very little difference in RESIDUALS RMS across the subtests. At the 10cm iteration steps, the RESIDUALS RMS decreases once GEOMETRY is performed, conversely at the 5cm iteration steps, the RESIDUALS RMS increases once GEOMETRY is performed. RESIDUALS RMSs do not change with iteration.

RESIDUALS RMSs:

Normalized Cross Correlation Scores

The evaluation maps were compared with a truth map via a cross-correlation routine which derives a correlation score. As a guide the following scores show perfect and excellent correlations:

• A map cross-correlated with itself will give a correlation score of approx. 1.0;
• Different sized maps sampled from the same truth (for example a 1,100 x 1,100 5cm sample map and a 1,000 x 1,000 5cm sample map) give a correlation score of approx. 0.8.

There is very little difference between the normalized cross correlation scores for the Lommel-Seeliger Photometric Function subtests (F3F1 and F3F2), both exhibiting very good correlation between the evaluation map and the truth map. The data however shows a poor correlation between the evaluation map and the truth map for the Clark and Takir Photometric Function subtest (F3F3).

Correlation Scores:

Transits

The following charts show North-South transits through the center of the evaluation region. The entire set of tests show a displacement from the truth. It is clear from inspection of the transits that subtest F3F3 is performing poorly, failing to represent features smaller than 3m.