make_sumfiles

Description

This program ingests images into SPC, building the sumfiles and nominals.

make_sumfiles reads metadata about the images from make_sumfiles.in, the position and pointing data from the kernels, and camera metadata from make_sumfiles.txt to generate body-fixed spacecraft position and pointing. It does the following:

• Generates <filename>.SUM in SUMFILES

• Generates <filename>.NOM in NOMINALS

• Appends the image name onto PICTLIST.TXT

User Warnings

• All the vectors in the sumfiles and nominals are in a body fixed frame with the origin at the center of the target body. The origin might be off a little bit, which means you may have to use shift eventually.

• The ck and spk kernel names in make_sumfiles.txt must cover the time span of all the images being ingested. Otherwise, make_sumfiles will not work.

• This program is non-standard. It frequently requires different versions for different missions to get the data in a form usable by SPC. Although the goal is to make a single version that does it all, each mission has a different image format.

• If the image name is already listed in PICTLIST.TXT, make_sumfiles will not make NOMINALS or SUMFILES when you select option "0". (See below)

Changes from the ECR for 3.0.1

• Put in COM offset for the spacecraft -- See 15 June 2017 document from Brishen Hawkins and Coralie Jackman -
• Requirement SOR.5013 - account for the offset from the spacecraft center of mass of each camera

SCOBJ_SHIFT.TXT maintains the current required shift between the center of mass of the OREx spacecraft and the camera focal plane. There is one line per camera consisting of camera number, delta X, delta Y, delta Z. Units are in meters. The coordinate frame is _. If the file isn't present, then this processing step will just be skipped.

•    1  0  0  0
     2  0  0  0
     3  2000  3000  4000
     4  1  1  30000
     5  1  1  30000

TMP2FOCLEN.TXT

• 1     0.d0    0.14351D-02    0.d0     mapcam
  2     0.d0    0.64241D-04    0.d0     samcam
  3     0.d0   -0.79251D-02    0.d0     polcam
  4     0.d0    0.14540D-03    0.d0     navcam
  5     0.d0    0.14508D-03    0.d0     nftcam

BIAS.TXT

Input Files

• Spice kernel file (SPICE)

• make_sumfiles.txt - list of SPICE kernels

• make_sumfiles.in - list of files from process_fits

• SCOBJ_SHIFT.TXT - adjustment for spacecraft center of mass to focal plane

• TMP2FOCLEN.TXT - adjustment of focal length based on temperature

• BIAS.TXT - a simple shift of the spacecraft position.

Output Files

• IFRAME.TXT - picture properties

• PICTLIST.TXT - list of pictures

• NOMINALS - nominal navigation information and uncertainties

• SUMFILES - camera properties and navigation information

• RandV.TXT - File comtaining s/c inertial position and velocity ordered by time.

Using make_sumfiles

When new images have been run through process_fits, then you can finish the ingestion by running make_sumfiles. Option 0 brings in new images. Typically, you will have to update make_script.txt to include SPK and CK kernels that cover these new images. Typically, you want to wait until the reconstructed kernels have been published (although you can use predicted kernels and then update them using make_sumfiles, option 1).

Option 1 is used when there is an update to the kernels (typically updated CK or SPK). It can also be used to update that starting sigma for the NOMINALS, but dynamics is a better tool for that because updating the NOMINALS will update everything to the same value.

Option 2 must be used with care. This changes the existing SUMFILES, and it allows you to tweak specific parts. Care is needed because you don't want to overwrite anything that had been done by hand and is believed to be correct (most specifically, geometry).

The following sample shows the standard inputs and the list of options for make_sumfiles:

 ~/bin/e.make_sumfiles
 Use B-Frame? (y/n)
n
 0.  Create SUMFILES & NOMINALS and ..
 1.  Update old NOMINALS and ..
 2.  Update old SUMFILES
0

• Usually, the B-Frame is not selected. The author has never selected it.

Enter the appropriate option:

• 0 Create SUMFILES & NOMINALS and .. - Completes both operations and ends.

• 1 Update old NOMINALS and .. - Completes NOMINALS operation and ends.

• 2 Update old SUMFILES - Issues the following prompts:

  Update UTC? (y/n)
  Update image thresholds? (y/n)
  Update camera parameters? (y/n)
  Reset geometry? (y/n)
  Reset sigmas? (y/n)

• Update UTC - Updates time a picture was taken.

• Update image thresholds - Changes image constraints.

• Update camera parameters - Changes the camera focal length, K-matrix, etc.

• Reset geometry - Resets position and pointing back to nominal.

• Reset sigmas - Resets uncertainty back to nominal.

Additional Reference

Alternative Description

Combines the converted image file with the image number, spacecraft state, spacecraft position & velocity and pointing information along with the associated calculated errors (Sigmas).

[from MAKE_SUMFILES.f]

Option 0 creates .SUM and .NOM files for any new images but leaves all others

C     unchanged.  Additionally, option 1 updates all .NOM files.  Option 2 dies all of the
C     above and makes changes in all .SUM files.  The possible changes, specified in new
C     entries in make_sumfiles.txt, are:
C
C           Update UTC? (y/n)
C           Update image thresholds? (y/n)
C           Update camera parameters? (y/n)
C           Reset geometry? (y/n)
C           Reset sigmas? (y/n)
C
C     The procedure first uses SPKEZ to calculate the spcaecraft - object vector SCOBJ and
C     the unit vector to the Sun in inertial space at the ephemeris time ET of an image.
C     It also determines the light time LT from the s/c to the target.  It then determines
C     the transformation matrix from the camera frame (INST) to inertial space (TTVI) at
C     time ET using the SPICELIB routine PXFORM.  Now all relevant quantities are available in
C     inertial space.  In SPC all these quantities are used in the body-fixed frame, so we again
C     use PXFORM to translate from inertial to body-fixed (TIPM).  We do this not at ET but at
C     ET-LT, since this is when the light forming the image left the target body.

(Compiled by KD)

make_sumfiles-3.0A2

CategoryThreeOhOne

PROCESS_FITS

Description

Process_fits is the main ingestion program for SPC, specifically adjusted to support OREx's metadata and file format. It does the following:

• reads a FITS image file
• extracts relevant information from the header

• adds a record to the make_sumfiles.in file

• adds a raw 16- bit unsigned integer image to the IMAGEFILES directory

SPC-OREx Naming Convention - FITS filenames are shortened to 12 characters. The 1st char and last two chars have embedded metadata in them. This explains how Bob encoded it. In general, it is camera symbol (P,M,N,T), the ET and then two char for filter number and processing status.

Required Files

• FITS image file - You provide the name as part of the standard input.

• INIT_LITHOS.TXT - Contains SPICE kernels. Each line entry that contains PCK= will have the spice command furnish run on it. SPC expects the leap second kernel to be defined in this list (which is also defined in make_sumfiles.txt).

• IMAGEFILES/

Outputs

• a row for the image is appended to make_sumfiles.in

• 16-bit unsigned integer image(s) are stored to IMAGEFILES directory. Stretched to a maximum 65536.

make_sumfiles.in is generated from process_fits. FITS keywords are used to provide the data to construct make_sumfiles.in. The key elements new in SPC v3.0.1 is the addition of temperature and MTR_POSITION.

• Example:

• Image Name	UTC or ET	Camera Num	Spacecraft Num	Binning	Temp [c]	Motor Position or Filter Position	Original filename (partial)
  P596924821J3	3/0596924726.30615	3	1	1	-25.50	270	20181201T082552S75100Z_pol_L0_V002.fits

  • Temperature is calculated (see below) using DN2TEMP.TXT and written out in make_sumfiles.in

  • Motor Position is taken directly from keyword MTR_POS regardless of PolyCam or MapCam. make_sumfiles handles the differences between cameras.

User Warning

• The program display will put the 0,0 position of the image in the top left, but OREx wants 0,0 to be in the bottom left.

Level 0

The data stored in each image file contained in the IMAGEFILES directory is a scaled unsigned integer (as 14 bits folded into 16 bits), with the exclusion of values less than 0. Acceptable values are from 0 to 16,383 (14 bit). The files will contain a bias -- i.e. an offset to ensure that the system can record no-photons detected without needing to use a negative value (in all flight conditions and temperatures)

ECR Changes from 3.0A2

• Support OREx level 0 images
  • Trim the second channel from the image to make it 1024x1024
  • Remove dark/bias -- This comes from the shielded overscan region
  • Correct the flat field
  • Desmear the image

Additional Required Files

• DATA/DN2TEMP.TXT

• DATA/FLATFILES.TXT

Level 0 Processing Steps

The level 0 OCAMS and NavCam images need to have a basic level of calibration done to them, which occurs in process_fits. The details of the calibration comes from the following documents

Output to the terminal

P634735130B7
 OKAY
2020 FEB 11 23:17:46.167
EXPOSURE(ms) =      3.225
 MSB SIGNED
    152.62    155.00     27.18    279.19  16383.00
   2866.0033510981912

            WRITE(6,FMT='(5F10.2)') DARKL, DARKR, SDL, SDR, FLOAT(K)

Dark for the left and right side of the chip. The standard dev for the left and right. The K, which is the max DN (I think).

OCAMS

Level 0 OCAMS images have the images stored as two FITS images. The first FITS image is the standard 1024x1024 FITS image that most of the science team will use (if they want level 0). The second FITS image array is 1111 x 1043, which is the full sensor for OCAMS and contains the overscan region used for calibration. This format is fully described in the OCAMS SIS.

These keywords are read by SPC. FITS Keywords describes what is done with each.

1. DN2TEMP.TXT

   1. OCAMS thermoresitors DNs are converted into physical units using the OCAMS Flight Software Engineer Dictionary (OREX-DOC-05.01-00259, Rev_C, Dated 12/04/2015) for SPC v3.0.1.

   2. SPC uses the 4 weighting parameters in DN2TEMP.TXT to calculate a linear weighted value for the effective temperature of the camera optics. For testing, we are weighting each thermoresistor by .25, providing each one with equal weight.

   3. The OCAMS Instrument Team will provide the final weighting during operations. There is no ICD or SIS for the information, so it will have to be done via email and documented as a memo.

2. Reduce Size - SPC will read the second image of the file, which includes the overscan regions. It will take the main 1024x1024 pixels for the image DN, loading the other portions into Flat and Dark variables for calibration. If the label doesn't match the image, it will output "INCONSISTENCY"; otherwise, it will show "OKAY".

3. Dark bias - The chip has four overscan regions that are averaged to calculate a day. The chip is broken into a left and a right half, and the darks are calculated accordingly. SPC will average all of the overscan pixels in each half, then print: Dark Left, Dark Right, Std Dev Left, Std Dev Right, Max.

   •     DN(i,j) -= DK(i,j)

4. Frame Transfer correction - OCAMS does not have a shutter, so as the pixels are read, they are shifted one column at a time to be read, creating a smear or frame transfer artifact. As long as the image is not saturated, the bias can be removed by the same routine used for New Frontiers LORRI camera.

   •            Readout smear from NEW HORIZONS LORRI camera. Tf is average of frame scrub
                time T1 and frame storage time T2.  Parameter alpha = (T2-T1)/(2*Tf) so
                T1=(1-alpha)*Tf and T2=(1+alpha)*Tf.  Te is exposure time, n=number of lines,
                P is smeared image column and Q is unsmeared image column.

5. Flat field correction - FLATFILES.TXT contains a list of flat files for each camera and filter. process_fits will read in the relevant file for the camera/filter and apply it to the image's DN. If FLATFILES.TXT does not exist, process_fits will error and quite. If the referenced file doesn't exist, SPC will just throw a warning (NO FLATFILE) and continue without applying a flat.

   • The code has a part that calculates and prints min and max, as well as scaling the flat: flat(i,j) /= max. However, it does not appear to be run because a boundary test ends the loop early.

     •     DN (i,j) *= FLAT(i,j)

6. Scale - Normal SPC prefers to keep the data in unsigned integers; however, to do these corrections, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^{16 / 2}14 (65535/16384 otherwise known as 4). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact on SPC's performance.

TAGCAMS (NavCam / NFTCam)

The following FITS Keywords are read by SPC. This describes what is done with each.

1. Temperature Calculation - Using DN2TEMP.TXT, apply the polynomial coefficients to convert from DN to a temperature in C. Once that is done, then use the last four columns of DN2TEMP.TXT to calculate an effective temperature for the optical path. This is implemented just as for OCAMS (see above).

   •    temp = C1 * DN^3 + C2 * DN^2 + C3 * DN + C4

2. Reduce Size - Remove the black borders on the NavCam/NFTCam images. The TAGGACS suite (NavCam and TAGCam) contain overscan regions on all portions. SPC removes these to output a single array (stored in IMAGEFILES) that is 2592 x 1944.

   • Top	50
     Top boundary	4
     left	134
     left boundary	6
     bottom	2
     bottom boundary	4
     right	10
     right boundary	10

3. Darks - NavCam and TAGCam darks are calculated from region 4 of the CCD. They are grouped into 8 bins, four on one row and four on the next. SPC calculates the dark (average) and the standard deviation for each pixel bin. It prints out each of the eight bins and the sum of all eight standard deviations. Then SPC removes the darks by simple subtraction. For TAGGAMS, this value is typically 150-180 DN. SPC checks, and if needed sets to zero, any negative values. Then it prints the minimum DN value after dark correction.

   •     DN(i,j) -= DK(i,j)

4. Flat field correction - SPC will multiple every DN by a flat field correction that was read in from the FLATFILES.TXT. Flat files are implemented the same way as OCAMS (see above).

   •     DN (i,j) *= FLAT(i,j)

5. Scale - Normal SPC prefers to keep the data in unsigned integers; however, to do the calibration, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^{16 / 2}12 (65535/4096 otherwise known as 16). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact of SPC's performance.

Level 1

If the source is a real (or 32-bit float) value process_fits takes the original images' data float and calculates the maximum value. Then it scales the data to spread from 0 to 65536 to maximize the accuracy of the stored data.

Using process_fits

process_fits can be run in two modes:

• Single file - just give the FITS filename at the prompt
• Batch mode

  • If the file newpix.txt exists, then process_fits will search through ../OSIRIS_REx/NEW_IMAGES_YYYY_DDD to find the image and process.

  • The built-in search only goes from 2015 to 2022 directories. The files don't actually have to be in the correct directories, it is just good practice for organization. SPC will search all directories from 2015/001 to 2022/365.

Here is a sample of the standard input:

 Input filename
2019-01-20T22-47-05.541_PCAM_L0b_V004.fits
 OKAY
2019 JAN 20 22:47:05.050
EXPOSURE(ms) =    100.000
 MSB SIGNED
      0.00      0.00      0.00      0.00      0.00
   5.0000000000000000
P601296494J3

Old - process_fits-3.0A2

(Compiled by EP)

CategoryThreeOhOne

regres

Description

This program generates a set of ascii interface files, one for each image, containing optical navigation observables, partials, and relevant ancillary information for use in the FDS navigation software suite(s). Both MIRAGE (KinetX) and GEODYN (GSFC) are currently ingesting these files. A file is created for each image in the restricted picture list, PICTLISTR.TXT, and only landmarks from a restricted landmark list, LMRKLISTR.TXT, are used. If the PICTLISTR.TXT file does not exist, the software defaults to PICTLIST.TXT.

Each interface file contains the following information:

• SPICE IDs, image epoch, camera parameters
• TITV matrix (inertial to camera frame transformation)
• TPMI matrix (body-fixed to inertial transformation)
• SPC solution for:
  • Spacecraft-to-object vector
  • Camera pointing (RA, Dec, Twist)
  • Spacecraft position sigma
  • Camera pointing and twist sigmas
• Nominal (a priori):
  • Spacecraft-to-object vector
  • Camera pointing (RA, Dec, Twist)
  • Solar unit vector
• For each landmark identified in the image from SPC process:
  • Landmark name
  • Body-fixed control point vector
  • Body-fixed control point sigma
  • Observed landmark center in (pixel, line) image coordinates
  • Predicted landmark center in (pixel, line) image coordinates based on nominal S/C position and SPC solution for camera pointing
  • Partial derivatives for (pixel, line) w.r.t. spacecraft-body position vector (SCOBJ)
  • Partial derivatives for (pixel, line) w.r.t. camera pointing (RA, Dec, Twist)
• List of all SPICE kernels used

regres requires the directory "REGRES_FILES" in the working directory. regres exports files for each picture (<PICTURE NAME>.TXT) to the "REGRES_FILES" directory.

regres uses all pictures in PICTLIST.TXT and landmarks in LMRKLISTR.TXT to make the exported regres files. However, if you have created a PICTLISTR.TXT, regres will only use the pictures from that file.

regres uses data from NOMINAL

Input Files

• PICTLIST.TXT or PICTLISTR.TXT

• LMRKLISTR.TXT

• NOMINALS/

• SUMFILES/

• INIT_LITHOS.TXT

• make_sumfiles.txt

Output Files

• <path to working directory>/REGRES_FILES/<picture name>.TXT

Using regres

If all required inputs are available in the working directory, simply execute regres to produce the output files.

Here is a sample output file from regres:

REGRES FILE.  CREATED Tue Oct  3 14:47:34 2017
PARTIALS UNITS: PX/KM, PX/DEG

N599562509F0                                                   IMAGE ID
2018-12-31T21:07:21.000_NCAM_L0b_V005.fits                     ORIGINAL NAME
ORX_NAVCAM1                                                    CAMERA ID
       -64                                                     SPACECRAFT ID
   2101955                                                     TARGET ID
LT+S                                                           ABCORR

2018 DEC 31 21:07:21.000                                       UTC
    0.5995625102D+09                                           ET SEC PAST J2000

  2592  1944  1000 65535                                       NPX, NLN, THRSH
    0.7680000000D+01    0.1296500000D+04    0.9725000000D+03   MMFL, CTR
  454.5455    0.0000    0.0000    0.0000 -454.5455    0.0000   K-MATRIX

Camera temperature (deg C) =    N/A                            CAMERA TEMP

Distortion type =   OPENCV                                     DISTORT BEGIN
Parameters:                                                    DISTORT CONTINUE
P( 0) =   0.0000000D+00                                        DISTORT CONTINUE
P( 1) =  -0.5357300D+00                                        DISTORT CONTINUE
P( 2) =   0.3611700D+00                                        DISTORT CONTINUE
P( 3) =  -0.1534700D-06                                        DISTORT CONTINUE
P( 4) =   0.0000000D+00                                        DISTORT CONTINUE
P( 5) =   0.0000000D+00                                        DISTORT CONTINUE
P( 6) =   0.0000000D+00                                        DISTORT CONTINUE
P( 7) =   0.0000000D+00                                        DISTORT CONTINUE
P( 8) =   0.0000000D+00                                        DISTORT END

    0.5815432282D+00   -0.6754376338D+00   -0.4534219630D+00   ROW 1 TITV MATRIX
   -0.8014742143D+00   -0.5712327735D+00   -0.1770090457D+00   ROW 2 TITV MATRIX
   -0.1394509146D+00    0.4663444234D+00   -0.8735423980D+00   ROW 3 TITV MATRIX

    0.8583303375D+00   -0.5130525373D+00    0.6791585157D-02   ROW 1 TPMI MATRIX
   -0.4619122250D+00   -0.7668748603D+00    0.4455783264D+00   ROW 2 TPMI MATRIX
   -0.2233967950D+00   -0.3855905115D+00   -0.8952171968D+00   ROW 3 TPMI MATRIX

LITHOS SOLUTION:

   -0.1877903947D+00    0.1184174267D+01   -0.1797326337D+01   SC - OBJ VECTOR
    0.1066482361D+03   -0.6087293551D+02   -0.2132491821D+02   CAMERA RA, DC, TW
    0.1732050808D-01                                           S/C POSITION SIG
    0.8102846845D-08                                           PNT SIGMA (DEG)
    0.5729577951D-08                                           TWIST SIGMA (DEG)

NOMINAL STATE:

   -0.1877903947D+00    0.1184174267D+01   -0.1797326337D+01   SC - OBJ VECTOR
    0.1066482361D+03   -0.6087293551D+02   -0.2132491821D+02   CAMERA RA, DC, TW
   -0.5629404130D+00   -0.7205839705D+00   -0.4047923329D+00   SOLAR UNIT VECTOR

LANDMARKS:

END LANDMARKS

KERNEL LIST

att_nte2_m3a-m4a_truth.bc
ORX_SCLKSCET.00000.example.tsc
naif0012.tls
orx_ocams_v05.ti
orx_181215_190112_190108_od012_v1.bsp
orx_v06rwg.tf
de424.bsp
bennu_nte2_truth.tpc
pck00010.tpc

END FILE

(Compiled by KD)

regres-3.0A2

CategoryThreeOhOne

DN2TEMP.TXT

Description

This input file contains the parameters to convert DN values (level 0 images) into temperatures and the weighting to apply to the final solution.

Coefficients

• Cameras 1, 2, 3 do not use the coefficients.
• Cameras 4 and 5 (TAGCAMs) use only the first two.
• Coefficients 3 and 4 are never used.

The camera's thermoresister is calculated as

• T_i = DN * C1³ + DN * C2² + DN * C3 + C4

Weighting

There are four thermoresisters on each camera. The weights 1-4 provide how much impact each thermoresistor should have. While this example has it totaling 1.0, SPC normalizes the weighting, so it isn't needed.

• N = W1 + W2 + W3 + W4
• T_final = T1 * W1/N + T2 * W2/N + T3 * W3/N + T4 * W4/N

Columns

Here is a sample DN2TEMP.TXT file:

1     0.80168E+02   -0.69285E-01    0.23237E-04   -0.37331E-08  0.25  0.25  0.25  0.25
2     0.80168E+02   -0.69285E-01    0.23237E-04   -0.37331E-08  0.25  0.25  0.25  0.25
3     0.80168E+02   -0.69285E-01    0.23237E-04   -0.37331E-08  0.25  0.25  0.25  0.25
4    -263.15    0.15259  0  0  1.D0  0.D0  0.D0  0.D0
5    -263.15    0.15259  0  0  1.D0  0.D0  0.D0  0.D0

C    NAV & NFT CAMERA HEAD TEMPERATURE PARAMETERS FROM
C    UA-SIS-9.4.4-322_OSIRIS-REx-NavCam-Data-Product-SIS_Rev_2.0_CCv0005.pdf
C
C    OCAMS CAMERA HEAD TEMPERATURE PARAMETERS CALCULATED FROM
C    OREX-DOC-05.01-00259_OCAMS Flight SW Engineering Dictionary_Rev_C_CCv0018.pdf
C    Fit to DN 1200 - 3400 (TC from ~ 25 to -35) has RMS = 0.18C, MAXDEV = 0.30C

C    Focal length temperatures for OCAMS are linear combinations of housekeeping temperatures:
C      TMC =  F1*MCFWHTMP+F2*MCFWMTMP+F3*MCLENTMP+F4*MCROETMP
C      TSC =  F1*SCFWHTMP+F2*SCFWMTMP+F3*SCLENTMP+F4*SCROETMP
C      TPC =  F1*PCFOHTMP+F2*PCFOMTMP+F3*PCMR1TMP+F4*PCROETMP
C

CategoryThreeOhOne

FITS Keywords

Description

This list shows the keyword that process-fits reads, the SPC variable it sets and a description explaining the setting.

• INSTRUME	NC	Comment
  MAP		N/A Because INSTRUME will be OCAMS
  SAM		N/A Because INSTRUME will be OCAMS
  POL		N/A Because INSTRUME will be OCAMS
  NCM	4
  NFT	5
  OCAMS	CAMERAID+1

• The order of this is important because the camera number is the reference that SPC uses for camera ID within the configuration files make_sumfiles.txt, DN2TEMP, and FLATFIELD.TXT.

CategoryThreeOhOne