• process_fits

PROCESS_FITS

Description

Process_fits is the main ingestion program for SPC, specifically adjusted to support OREx's metadata and file format. It does the following:

• reads a FITS image file
• extracts relevant information from the header

• adds a record to the make_sumfiles.in file

• adds a raw 16- bit unsigned integer image to the IMAGEFILES directory

SPC-OREx Naming Convention - FITS filenames are shortened to 12 characters. The 1st char and last two chars have embedded metadata in them. This explains how Bob encoded it. In general, it is camera symbol (P,M,N,T), the ET and then two char for filter number and processing status.

Required Files

• FITS image file - You provide the name as part of the standard input.

• INIT_LITHOS.TXT - Contains SPICE kernels. Each line entry that contains PCK= will have the spice command furnish run on it. SPC expects the leap second kernel to be defined in this list (which is also defined in make_sumfiles.txt).

• IMAGEFILES/

Outputs

• a row for the image is appended to make_sumfiles.in

• 16-bit unsigned integer image(s) are stored to IMAGEFILES directory. Stretched to a maximum 65536.

make_sumfiles.in is generated from process_fits. FITS keywords are used to provide the data to construct make_sumfiles.in. The key elements new in SPC v3.0.1 is the addition of temperature and MTR_POSITION.

• Example:

• Image Name	UTC or ET	Camera Num	Spacecraft Num	Binning	Temp [c]	Motor Position or Filter Position	Original filename (partial)
  P596924821J3	3/0596924726.30615	3	1	1	-25.50	270	20181201T082552S75100Z_pol_L0_V002.fits

  • Temperature is calculated (see below) using DN2TEMP.TXT and written out in make_sumfiles.in

  • Motor Position is taken directly from keyword MTR_POS regardless of PolyCam or MapCam. make_sumfiles handles the differences between cameras.

User Warning

• The program display will put the 0,0 position of the image in the top left, but OREx wants 0,0 to be in the bottom left.

Level 0

The data stored in each image file contained in the IMAGEFILES directory is a scaled unsigned integer (as 14 bits folded into 16 bits), with the exclusion of values less than 0. Acceptable values are from 0 to 16,383 (14 bit). The files will contain a bias -- i.e. an offset to ensure that the system can record no-photons detected without needing to use a negative value (in all flight conditions and temperatures)

ECR Changes from 3.0A2

• Support OREx level 0 images
  • Trim the second channel from the image to make it 1024x1024
  • Remove dark/bias -- This comes from the shielded overscan region
  • Correct the flat field
  • Desmear the image

Additional Required Files

• DATA/DN2TEMP.TXT

• DATA/FLATFILES.TXT

Level 0 Processing Steps

The level 0 OCAMS and NavCam images need to have a basic level of calibration done to them, which occurs in process_fits. The details of the calibration comes from the following documents

Output to the terminal

P634735130B7
 OKAY
2020 FEB 11 23:17:46.167
EXPOSURE(ms) =      3.225
 MSB SIGNED
    152.62    155.00     27.18    279.19  16383.00
   2866.0033510981912

            WRITE(6,FMT='(5F10.2)') DARKL, DARKR, SDL, SDR, FLOAT(K)

Dark for the left and right side of the chip. The standard dev for the left and right. The K, which is the max DN (I think).

OCAMS

Level 0 OCAMS images have the images stored as two FITS images. The first FITS image is the standard 1024x1024 FITS image that most of the science team will use (if they want level 0). The second FITS image array is 1111 x 1043, which is the full sensor for OCAMS and contains the overscan region used for calibration. This format is fully described in the OCAMS SIS.

These keywords are read by SPC. FITS Keywords describes what is done with each.

1. DN2TEMP.TXT

   1. OCAMS thermoresitors DNs are converted into physical units using the OCAMS Flight Software Engineer Dictionary (OREX-DOC-05.01-00259, Rev_C, Dated 12/04/2015) for SPC v3.0.1.

   2. SPC uses the 4 weighting parameters in DN2TEMP.TXT to calculate a linear weighted value for the effective temperature of the camera optics. For testing, we are weighting each thermoresistor by .25, providing each one with equal weight.

   3. The OCAMS Instrument Team will provide the final weighting during operations. There is no ICD or SIS for the information, so it will have to be done via email and documented as a memo.

2. Reduce Size - SPC will read the second image of the file, which includes the overscan regions. It will take the main 1024x1024 pixels for the image DN, loading the other portions into Flat and Dark variables for calibration. If the label doesn't match the image, it will output "INCONSISTENCY"; otherwise, it will show "OKAY".

3. Dark bias - The chip has four overscan regions that are averaged to calculate a day. The chip is broken into a left and a right half, and the darks are calculated accordingly. SPC will average all of the overscan pixels in each half, then print: Dark Left, Dark Right, Std Dev Left, Std Dev Right, Max.

   •     DN(i,j) -= DK(i,j)

4. Frame Transfer correction - OCAMS does not have a shutter, so as the pixels are read, they are shifted one column at a time to be read, creating a smear or frame transfer artifact. As long as the image is not saturated, the bias can be removed by the same routine used for New Frontiers LORRI camera.

   •            Readout smear from NEW HORIZONS LORRI camera. Tf is average of frame scrub
                time T1 and frame storage time T2.  Parameter alpha = (T2-T1)/(2*Tf) so
                T1=(1-alpha)*Tf and T2=(1+alpha)*Tf.  Te is exposure time, n=number of lines,
                P is smeared image column and Q is unsmeared image column.

5. Flat field correction - FLATFILES.TXT contains a list of flat files for each camera and filter. process_fits will read in the relevant file for the camera/filter and apply it to the image's DN. If FLATFILES.TXT does not exist, process_fits will error and quite. If the referenced file doesn't exist, SPC will just throw a warning (NO FLATFILE) and continue without applying a flat.

   • The code has a part that calculates and prints min and max, as well as scaling the flat: flat(i,j) /= max. However, it does not appear to be run because a boundary test ends the loop early.

     •     DN (i,j) *= FLAT(i,j)

6. Scale - Normal SPC prefers to keep the data in unsigned integers; however, to do these corrections, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^{16 / 2}14 (65535/16384 otherwise known as 4). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact on SPC's performance.

TAGCAMS (NavCam / NFTCam)

The following FITS Keywords are read by SPC. This describes what is done with each.

1. Temperature Calculation - Using DN2TEMP.TXT, apply the polynomial coefficients to convert from DN to a temperature in C. Once that is done, then use the last four columns of DN2TEMP.TXT to calculate an effective temperature for the optical path. This is implemented just as for OCAMS (see above).

   •    temp = C1 * DN^3 + C2 * DN^2 + C3 * DN + C4

2. Reduce Size - Remove the black borders on the NavCam/NFTCam images. The TAGGACS suite (NavCam and TAGCam) contain overscan regions on all portions. SPC removes these to output a single array (stored in IMAGEFILES) that is 2592 x 1944.

   • Top	50
     Top boundary	4
     left	134
     left boundary	6
     bottom	2
     bottom boundary	4
     right	10
     right boundary	10

3. Darks - NavCam and TAGCam darks are calculated from region 4 of the CCD. They are grouped into 8 bins, four on one row and four on the next. SPC calculates the dark (average) and the standard deviation for each pixel bin. It prints out each of the eight bins and the sum of all eight standard deviations. Then SPC removes the darks by simple subtraction. For TAGGAMS, this value is typically 150-180 DN. SPC checks, and if needed sets to zero, any negative values. Then it prints the minimum DN value after dark correction.

   •     DN(i,j) -= DK(i,j)

4. Flat field correction - SPC will multiple every DN by a flat field correction that was read in from the FLATFILES.TXT. Flat files are implemented the same way as OCAMS (see above).

   •     DN (i,j) *= FLAT(i,j)

5. Scale - Normal SPC prefers to keep the data in unsigned integers; however, to do the calibration, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^{16 / 2}12 (65535/4096 otherwise known as 16). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact of SPC's performance.

Level 1

If the source is a real (or 32-bit float) value process_fits takes the original images' data float and calculates the maximum value. Then it scales the data to spread from 0 to 65536 to maximize the accuracy of the stored data.

Using process_fits

process_fits can be run in two modes:

• Single file - just give the FITS filename at the prompt
• Batch mode

  • If the file newpix.txt exists, then process_fits will search through ../OSIRIS_REx/NEW_IMAGES_YYYY_DDD to find the image and process.

  • The built-in search only goes from 2015 to 2022 directories. The files don't actually have to be in the correct directories, it is just good practice for organization. SPC will search all directories from 2015/001 to 2022/365.

Here is a sample of the standard input:

 Input filename
2019-01-20T22-47-05.541_PCAM_L0b_V004.fits
 OKAY
2019 JAN 20 22:47:05.050
EXPOSURE(ms) =    100.000
 MSB SIGNED
      0.00      0.00      0.00      0.00      0.00
   5.0000000000000000
P601296494J3

Old - process_fits-3.0A2

(Compiled by EP)

CategoryThreeOhOne