TestF3F Photometric Function Sensitivity Test Results
Definitions
CompareOBJ RMS: 
The root mean square of the distance from each bigmap pixel/line location to the nearest facet of the truth OBJ. 
RESIDUALS RMS 
The root mean square residual error reported by RESIDUALS. 
Key Findings
The LommelSeeliger Photometric Function subtests (F3F1 and F3F2) performed well with small differences in measures of accuracy and correlation.
The Clark and Takir Photometric Function subtest (F3F3) performed poorly throughout testing, with pervasive degradation of the digital terrain with every processing step conducted, eventually failing to align/correlate images for a subset of landmarks at the 5cm tiling step.
CompareOBJ RMS with and without optimal translation and rotation shows an inability to distinguish between subtests which perform well and subtests which perform poorly. Indeed, there is no indication from the CompareOBJ RMS that the poorly performing F3F3 subtest failed to complete the 5cm tiling processing step.
 There is also no indication of performance from an inspection of the RESIDUALS RMSs.
 The normalized cross correlation scores are clearly distinct for the wellperforming and poorperforming subtests.
 A single NorthSouth transit through the center of the evaluation region immediately indicates good and poor performance at all resolutions tiled.
Results and Discussion
Results from testing the three photometric functions split into two groups characterized by differing digital terrain accuracy and model behavior. Subtests F3F1 and F3F2 (LommelSeeliger photometric function without the 2 and with the 2 respectively) performed well with minor differences in the measurements of accuracy, whereas subtest F3F3 (Clark and Takir photometric function) performed poorly with pervasive degradation of the digital terrain with every processing step conducted. A detailed analysis of the behavior of F3F3 is reported here:
CompareOBJ RMS
Three CompareOBJ RMS values for the final 5cm resolution 20m x 20m evaluation bigmap are presented for each subtest and each S/C position and camera pointing uncertainty:
 The largest CompareOBJ RMS (approx. 57cm across subtests) is obtained by running CompareOBJ on the untranslated and unrotated evaluation model.
 The second smallest CompareOBJ RMS (approx. 15cm across subtests) is obtained by running CompareOBJ with its optimal translation and rotation option.
 The smallest CompareOBJ RMS (approx. 6cm across subtests F3F1/2) is obtained by manually translating the evaluation model and searching for a local CompareOBJ RMS minimum.
The CompareOBJ optimal translation routine is not optimized for the evaluation model scale (5cm pix/line resolution). Manual translations of the bigmap were therefore conducted in an attempt to find a minimum CompareOBJ RMS. The manually translated evaluation models gave the smallest CompareOBJ RMSs.
The CompareOBJ RMS without translation or rotation is similar across subtests showing an inability to distinguish performance differences apparent from visual inspection of the evaluation maps, the normalized cross correlation scores, and the failure of F3F3 subtest at the 5cm tiling step. The CompareOBJ RMS with optimal translation and rotation is little better at distinguishing performance with some decrease in RMS of the poorperforming F3F3 subtest when compared with the wellperforming F3F1 and F3F2 subtests. CompareOBJ with manual translation shows the most ability to distinguish between good and poorperforming subtests, but the RMS of the poorly performing F3F3 subtest is still unexpectedly low.
CompareOBJ RMSs do not change with iteration.
CompareOBJ with Manual Translation  RMS:

CompareOBJ RMS (cm) 

Processing Step 
F3F1 (LommelSeeliger without the 2) 
F3F2 (LommelSeeliger with the 2) 
F3F3 (Clark and Takir) 
20cm Iteration 00 
9.0284 
8.6072 
13.7263 
10cm Iteration 00 
7.2155 
6.6544 
13.0804 
5cm Tiling (incomplete) 


10.7716 
5cm Iteration 00 
6.1890 
5.8275 

5cm Iteration 20 
5.4468 
5.7187 

CompareOBJ with Manual Translation  Translation:



Translation 

Subtest 
Photometric Function 
Processing Step 
x (cm) 
y (cm) 
z (cm) 
Distance (cm) 
LommelSeeliger without the 2 
5cm Iteration 20 
175.7 
41 
40 
184.80 

LommelSeeliger with the 2 
5cm Iteration 20 
175.7 
41 
40 
184.80 

Clark and Takir 
5cm Tiling (incomplete) 
180.9 
41.2 
40 
189.80 
RESIDUALS RMS
Again, there is very little difference in RESIDUALS RMS across the subtests. At the 10cm iteration steps, the RESIDUALS RMS decreases once GEOMETRY is performed, conversely at the 5cm iteration steps, the RESIDUALS RMS increases once GEOMETRY is performed. RESIDUALS RMSs do not change with iteration.
RESIDUALS RMSs:

RESIDUALS RMS (cm) 

Processing Step 
F3F1 (LommelSeeliger without the 2) 
F3F2 (LommelSeeliger with the 2) 
F3F3 (Clark and Takir) 
20cm Iteration 00 
42.5852 
42.6027 
42.6358 
10cm Iteration 00 (pre Geometry) 
42.3146 
42.3362 
42.4303 
10cm Iteration 00 (post Geometry) 
41.3606 
41.3900 
41.4881 
5cm Tiling (Incomplete) 


41.0550 
5cm Iteration 00 (pre Geometry) 
40.8840 
40.8434 

5cm Iteration 00 (post Geometry) 
41.6120 
41.4276 

5cm Iteration 20 
41.6355 
41.4529 

Normalized Cross Correlation Scores
The evaluation maps were compared with a truth map via a crosscorrelation routine which derives a correlation score. As a guide the following scores show perfect and excellent correlations:
 A map crosscorrelated with itself will give a correlation score of approx. 1.0;
 Different sized maps sampled from the same truth (for example a 1,100 x 1,100 5cm sample map and a 1,000 x 1,000 5cm sample map) give a correlation score of approx. 0.8.
There is very little difference between the normalized cross correlation scores for the LommelSeeliger Photometric Function subtests (F3F1 and F3F2), both exhibiting very good correlation between the evaluation map and the truth map. The data however shows a poor correlation between the evaluation map and the truth map for the Clark and Takir Photometric Function subtest (F3F3).
Correlation Scores:

Correlation Score 

Processing Step 
F3F1 (LommelSeeliger without the 2) 
F3F2 (LommelSeeliger with the 2) 
F3F3 (Clark and Takir) 
20cm Iteration 00 
0.6141 
0.6133 
0.4572 
10cm Iteration 00 (post Geometry) 
0.7143 
0.7168 
0.4506 
5cm Iteration 00 (post Geometry) 
0.7679 
0.7756 

5cm Iteration 10 
0.7839 
0.7564 

5cm Iteration 20 
0.7872 
0.7884 

Transits
The following charts show NorthSouth transits through the center of the evaluation region. The entire set of tests show a displacement from the truth. It is clear from inspection of the transits that subtest F3F3 is performing poorly, failing to represent features smaller than 3m.