OptimalF2892016-05-23 15:45:37DianeLambert882016-05-23 15:45:23DianeLambert872016-05-23 15:45:02DianeLambert862016-05-23 15:44:46DianeLambert852016-05-23 15:41:16DianeLambert842016-05-23 10:47:44DianeLambert832016-05-23 10:46:36DianeLambert822016-05-23 10:45:58DianeLambert812016-05-23 10:45:21DianeLambert802016-05-23 10:39:27DianeLambert792016-05-23 09:32:33DianeLambert782016-05-23 09:32:10DianeLambert772016-05-23 09:31:33DianeLambert762016-05-23 09:30:43DianeLambert752016-04-28 15:38:34DianeLambert742016-04-28 15:35:41DianeLambert732016-04-28 15:35:26DianeLambert722016-04-28 15:33:36DianeLambert712016-04-28 15:32:22DianeLambert702016-04-28 15:30:04DianeLambert692016-04-28 15:29:38DianeLambert682016-04-28 15:28:50DianeLambert672016-04-28 15:19:36DianeLambert662016-04-28 15:19:08DianeLambert652016-04-28 15:18:44DianeLambert642016-04-28 15:03:42DianeLambert632016-04-28 15:01:28DianeLambert622016-04-28 14:40:36DianeLambert612016-04-28 14:40:18DianeLambert602016-04-28 14:40:01DianeLambert592016-04-28 14:39:35DianeLambert582016-04-15 17:16:12KristoferDrozd572016-01-06 12:01:33DianeLambert562016-01-06 11:44:46DianeLambert552016-01-06 11:42:03DianeLambert542016-01-06 11:39:15DianeLambert532016-01-06 11:36:46DianeLambert522016-01-06 11:21:15DianeLambert512016-01-06 11:18:49DianeLambert502016-01-06 11:17:12DianeLambert492016-01-06 11:10:30DianeLambert482016-01-06 11:09:54DianeLambert472016-01-06 11:01:22DianeLambert462016-01-06 10:54:18DianeLambert452016-01-06 10:39:10DianeLambert442016-01-06 10:35:48DianeLambert432016-01-06 10:30:40DianeLambert422016-01-06 10:29:36DianeLambert412016-01-06 10:24:37DianeLambert402016-01-06 10:11:09DianeLambert392016-01-06 10:09:04DianeLambert382016-01-06 10:07:03DianeLambert372016-01-06 10:02:04DianeLambert362016-01-06 10:00:46DianeLambert352016-01-05 18:03:37DianeLambert342016-01-05 17:53:09DianeLambert332016-01-05 17:47:51DianeLambert322016-01-05 17:40:03DianeLambert312016-01-05 17:33:00DianeLambert302016-01-05 17:23:57DianeLambert292016-01-05 17:00:13DianeLambert282016-01-05 16:57:32DianeLambert272015-11-19 13:25:06KristoferDrozd262015-11-19 13:24:47KristoferDrozd252015-11-19 13:23:36KristoferDrozd242015-11-19 13:22:16KristoferDrozd232015-11-13 17:36:55DianeLambert222015-11-13 17:36:21DianeLambert212015-11-13 17:34:03DianeLambert202015-11-13 17:32:16DianeLambert192015-11-13 17:28:08DianeLambert182015-11-13 17:23:03DianeLambert172015-11-13 14:24:06DianeLambert162015-10-18 20:36:09DianeLambert152015-10-18 20:24:38DianeLambert142015-10-18 20:18:01DianeLambert132015-10-18 20:13:41DianeLambert122015-10-18 20:13:15DianeLambert112015-10-18 20:04:31DianeLambert102015-10-18 19:24:11DianeLambert92015-10-18 19:23:52DianeLambert82015-10-18 19:22:39DianeLambert72015-10-18 18:43:28DianeLambert62015-10-18 18:41:25DianeLambert52015-10-18 18:39:56DianeLambert42015-10-18 18:39:38DianeLambert32015-10-18 18:35:25DianeLambert22015-10-09 08:21:06EricPalmer12015-10-05 17:24:35DianeLambertTest 05: OptimalF2Aim and ObjectivesPurpose of test: To obtain highest possible topographic accuracy and resolution for TAG site 1, given perfect camera position and pointing and optimal SPC image suite. Additional Objectives To obtain user-in-the-loop and server-in-the-loop RMS error variations. To obtain user-in-the-loop procedural variation. To obtain beginner/expert user variation. AcronymsDTMDigital Terrain Model

GSDGround Sample Distance

RMSRoot Mean Square

SPCStereophotoclinometry

TAGTouch And Go

Definitions and DisambiguationFormal UncertaintyThe RMS landmark position uncertainty of the bigmap as output by the utility program RESIDUALS.

CompareOBJ RMSRMS per-sample distance to nearest truth surface point as output by the utility program CompareOBJ.

MethodologyData Data was generated by Imager_mg using shape3.8 on ormacsrv1. PT10A - Centered at the middle of TAG site one. Image Suite: Optimal flight path, image resolutions: 50cm, 20cm, 10cm, 5cm. S/C: Every 20° azimuth angle, 45° zenith angle Sun: 0, 90°, 180°, 270° azimuth angle, 45 zenith angle 135° azimuth angle, 30° zenith angle Spacecraft: SC_perfectF2_POLARS.jpg Sun: SUN_perfectF2_POLARS.jpg User/Server-in-the-loop: UserExperienceServerNumber of Core Processors

TCampbellIntermediate?Ormacsrv2?8

KDrozdIntermediateOrmacsrv18

DLambertBeginnerOrmacsrv36

EPalmerExperienced??

JWeirichIntermediateDD12

Bigmaps The following TAG1 bigmaps were tiled/iterated and evaluated: Starting topography defined from: START1.MAP: GSD = 25cm; Q size = 150; width = 75m; center lat/wlong = -8.027, 262.768. DTM Bigmap parameters: StepGSD(cm)Maplet Overlap FactorApprox. Maplet OverlapQ SizeWidth

35cm-Tiling351/1.360%11260m

18cm-Tiling181/1.360%21761m

9cm-Tiling91/1.360%43461m

5cm-Tiling51/1.360%78062m

Evaluation Bigmap parameters: GSD = 5cm; Q size = 500; width = 50m; Tiling Parameters Tiling parameters may have varied between users: ParameterTCampbellKDrozdDLambertEPalmerJWeirich

Image elimination: INVLIM000?0

Image elimination: SLIM60 (@35cm), 50 (o.w.)5060 (@35cm), 50 (o.w.)?50

Image elimination: CLIM.5.5.5?.5

Image elimination: ILIM.5.5.5?.5

Image elimination: RSMIN0 (@35cm), .25 (o.w.).250 (@35cm), .25 (o.w.)?.25

Image elimination: RSMAX333?3

Calculate Central Vector (v, 1)YESYESYES?YES

Differential Stereo (2, 6)YESYESYES?YES

Shadows (2, 7)NONONO?NO

alpha-numerics in brackets refer to Lithos/LithosP menu options. Iteration Parameters Iteration parameters varied between users: ParameterTCampbellKDrozdDLambertEPalmerJWeirich

Reset albedo/slopes (a, y, y)NOYESYES?YES

Calculate Central Vector (v, 1)YESYESYES?YES

Differential Stereo (2, 6)NONONO?NO

Shadows (2, 7)NOYESNO?YES

alpha-numerics in brackets refer to Lithos/LithosP menu options. ResultsPlease see Tables and Figures. DiscussionUser Differences in Tiling and Iteration ParametersThe only differences in tiling parameters pertain to the auto-elimination of images during the initial 35cm tiling. The maximum permissible emission angle varied between 50 and 60 degrees. The minimum allowable ration between image scale and misplace varied between 0 and 0.25. All other tiling step parameters were the same between users. Iteration parameters however varied greatly between users, with no two users applying an identical set of parameters. All users calculated the central vector (v1), however not all users reset albedo/slopes, conditioned with differential stereo and/or conditioned with shadows. User Differences in Processing StepsSince users had different criteria for triggering the cessation of iteration and continuing to the next tiling step (such as 2 iterations per tiling step, or, no further change in evaluation statistics), users iterated solutions a varied number of times. Number of iterations conducted at each GSD step: Processing StepTCampbellKDrozdDLambertEPalmerJWeirich

35cm Iterations26562

18cm Iterations22612

9cm Iterations212-2

5cm Iterations2---1

User ErrorsAdditionally, two users applied incorrect or non-standard processing procedures for some portion of the test related to their level of experience and training: DLambert calculated statistics on an incorrectly sized evaluation bigmap during the 35cm tiling and first iteration leading to loss of statistics data for these processing steps; DLambert failed to eliminate low correlating images during the first three 35cm iteration steps, leading to initial problems in the topography which are reflected in the formal uncertainties and RMS errors. The errors receded at the 18cm tiling step. KDrozd used different parameters in the utility program FITS2OBJ, upstream of obtaining the CompareOBJ RMS error, as other team members, leading to differing RMS error statistics which do not necessarily reflect differences in the evaluation bigmap compared to other users' evaluation bigmaps. User Differences in Evaluation StatisticsCompareOBJ RMS errors show large variation from user to user. Some of these variations can be directly linked to user procedural differences, for example the high RMS errors through the 35cm iterations for DLambert and KDrozd are directly related to incorrect and non-standard processing (see above section). Other differences reflect some combination of: variation in iteration parameters (see above section); variation in number of iterations conducted; variation in user skill and experience in manually registering/eliminating problem images; server-based variation; number-of-core-processors-based variation: SPC uncertainty due to random seeds utilized in maplet slope and height calculations. Formal uncertainties were more similar between users, differences can be directly linked to user procedural differences, specifically incorrect and non-standard processing covered in the previous section. Overall SPC PerformanceAll users obtain formal uncertainties below the highest image resolution (< 5cm). Final CompareOBJ RMS errors vary. The two final solutions which reached a GSD of 5cm obtained an RMS error of 12.3cm and 4.5cm. Slight improvements in CompareOBJ RMS can be gained by using the CompareOBJ optimal translation/rotation utilities. Conclusions and RecommendationsFormal uncertainties performed well in respect to both between-user variation and final performance measures. However, the CompareOBJ RMS error did not, showing high variability user-to-user and high variability in final performance measures. User-to-user differences can be minimized in future testing through a combination of standardization of tiling/iteration parameters and iteration cessation criteria, and increase in user skill and experience. Tiling and iteration parameters should be investigated to find the best-practice for standardized seed file sets. This recommendation is addressed in Test06 - FindHeights. To further investigate the performance of SPC with perfect camera position and pointing and optimal viewing conditions, a suite of higher-resolution images will be added and the best performing 5cm DTM will be tiled and iterated at higher ground sample distances. This recommendation led to the implementation of Test08 - OptimalUltraF2. Tables and FiguresRMS Distance (cm) - Compare OBJ (with no translation/rotation) Resolution (cm)StepDianeEricJohnKrisTanner

3535cm-Tiling7.06 7.068.35

3535cm-Iteration 16.957.158.49

3535cm-Iteration 211.517.157.137.448.58

3535cm-Iteration 311.767.67.44

3535cm-Iteration 411.858.049.10

3535cm-Iteration 511.317.5711.05

3535cm-Iteration 68.8113.39

1818cm-Tiling8.785.438.796.49

1818cm-Iteration 18.628.126.299.376.50

1818cm-Iteration 28.577.4510.606.58

1818cm-Iteration 38.57

1818cm-Iteration 48.56

1818cm-Iteration 58.55

1818cm-Iteration 68.54

99cm-Tiling5.476.008.494.98

99cm-Iteration 15.287.399.804.92

99cm-Iteration 25.279.154.92

55cm-Tiling 7.684.47

55cm-Iteration 112.314.44

55cm-Iteration 24.45

OptimalRMS-resized.png RMS Distance (cm) - Compare OBJ (with optimal translation/rotation) Resolution (cm)StepDianeKris

RMS without Translation
or Rotation (cm)RMS with Optimal
Translation (cm)RMS with Optimal
Translation and Rotation (cm)RMS with Manual
Translation (cm)RMS without Translation
or Rotation (cm)RMS with Optimal
Translation (cm)

3535cm-Tiling7.06

3535cm-Iteration 17.15

3535cm-Iteration 211.517.44

3535cm-Iteration 311.7611.4010.897.44

3535cm-Iteration 411.8511.4010.929.10

3535cm-Iteration 511.3110.9810.3511.05

3535cm-Iteration 613.39

1818cm-Tiling8.628.788.158.79

1818cm-Iteration 18.628.448.199.37

1818cm-Iteration 28.578.388.0910.60

1818cm-Iteration 38.578.448.06

1818cm-Iteration 48.568.408.02

1818cm-Iteration 58.558.417.96

1818cm-Iteration 68.548.447.96

99cm-Tiling5.476.065.496.578.49

99cm-Iteration 15.285.405.269.80

99cm-Iteration 25.275.455.326.13

55cm-Tiling

55cm-Iteration 1

55cm-Iteration 2

CompareOBJRMS_wwo_opt_trans_rot_resized_60pct.png Formal Uncertainty (cm) - RESIDUALS Resolution (cm)StepDianeEricJohnKrisTanner

3535cm-Tiling5.485.49

3535cm-Iteration 16.326.91

3535cm-Iteration 229.784.765.365.72

3535cm-Iteration 339.124.715.45

3535cm-Iteration 414.744.875.98

3535cm-Iteration 512.695.046.72

3535cm-Iteration 65.237.51

1818cm-Tiling8.789.173.125.114.34

1818cm-Iteration 17.902.973.984.45

1818cm-Iteration 27.233.333.984.22

1818cm-Iteration 36.84

1818cm-Iteration 46.61

1818cm-Iteration 56.47

1818cm-Iteration 66.40

99cm-Tiling3.822.242.862.54

99cm-Iteration 13.532.572.992.46

99cm-Iteration 23.183.012.26

55cm-Tiling2.991.59

55cm-Iteration 13.971.43

55cm-Iteration 21.35

OptimalResidual-resized.png Normalized Cross Correlation Score Resolution (cm)StepDianeEricJohnKrisTanner

3535cm-Tiling

3535cm-Iteration 1

3535cm-Iteration 20.6433

3535cm-Iteration 30.6424

3535cm-Iteration 40.6405

3535cm-Iteration 50.6423

3535cm-Iteration 6

1818cm-Tiling0.7360

1818cm-Iteration 10.7373

1818cm-Iteration 20.7377

1818cm-Iteration 30.7378

1818cm-Iteration 40.7380

1818cm-Iteration 50.7381

1818cm-Iteration 60.7381

99cm-Tiling0.8118

99cm-Iteration 10.8113

99cm-Iteration 20.8113

55cm-Tiling

55cm-Iteration 1

55cm-Iteration 2

35cm GSD Maplets - 50cm Resolution Images MTAG15-35cmMaplets.jpg 18cm GSD Maplets - 20cm Resolution Images MTAG15-18cmMaplets.jpg 9cm GSD Maplets - 10cm Resolution Images MTAG15-9cmMaplets.jpg