= PROCESS_FITS = ||Category B ||Version 3.0.1 || == Description == Process_fits is the main ingestion program for SPC, specifically adjusted to support OREx's metadata and file format. It does the following: * reads a FITS image file * extracts relevant information from the header * adds a record to the [[make_sumfiles.in]] file * adds a raw 16- bit unsigned integer image to the [[IMAGEFILES]] directory [[SPC-OREx Naming Convention]] - FITS filenames are shortened to 12 characters. The 1st char and last two chars have embedded metadata in them. This explains how Bob encoded it. In general, it is camera symbol (P,M,N,T), the ET and then two char for filter number and processing status. === Required Files === * FITS image file - You provide the name as part of the standard input. * [[INIT_LITHOS.TXT]] - Contains SPICE kernels. Each line entry that contains ''PCK='' will have the spice command ''furnish'' run on it. SPC expects the leap second kernel to be defined in this list (which is also defined in [[make_sumfiles.txt]]). * [[IMAGEFILES]]/ === Outputs === * a row for the image is appended to [[make_sumfiles.in]] * 16-bit unsigned integer image(s) are stored to [[IMAGEFILES]] directory. Stretched to a maximum 65536. [[make_sumfiles.in]] is generated from [[process_fits]]. FITS keywords are used to provide the data to construct [[make_sumfiles.in]]. The key elements new in SPC v3.0.1 is the addition of temperature and MTR_POSITION. Example: || Image Name || UTC or ET || Camera Num || Spacecraft Num || Binning || Temp [c] || Motor Position or Filter Position || Original filename (partial) || || P596924821J3 || 3/0596924726.30615 || 3 || 1 || 1 || -25.50 || 270 || 20181201T082552S75100Z_pol_L0_V002.fits || * Temperature is calculated (see below) using [[DN2TEMP.TXT]] and written out in [[make_sumfiles.in]] * Motor Position is taken directly from keyword ''MTR_POS'' regardless of PolyCam or MapCam. [[make_sumfiles]] handles the differences between cameras. === User Warning === . /!\ The program [[display]] will put the 0,0 position of the image in the top left, but OREx wants 0,0 to be in the bottom left. -------- == Level 0 == The data stored in each image file contained in the [[IMAGEFILES]] directory is a scaled unsigned integer (as 14 bits folded into 16 bits), with the exclusion of values less than 0. Acceptable values are from 0 to 16,383 (14 bit). The files will contain a bias -- i.e. an offset to ensure that the system can record no-photons detected without needing to use a negative value (in all flight conditions and temperatures) === ECR Changes from 3.0A2 === * Support OREx level 0 images * Trim the second channel from the image to make it 1024x1024 * Remove dark/bias -- This comes from the shielded overscan region * Correct the flat field * Desmear the image === Additional Required Files === * DATA/[[DN2TEMP.TXT]] * DATA/[[FLATFILES.TXT]] == Level 0 Processing Steps == The level 0 OCAMS and NavCam images need to have a basic level of calibration done to them, which occurs in process_fits. The details of the calibration comes from the following documents ||software version || Document || || v3.0.1 and v3.0.2 || KinetX Interoffice Memorandum SNAFD.B / 023-16, Dated 25 Sept 2017 || Output to the terminal {{{ P634735130B7 OKAY 2020 FEB 11 23:17:46.167 EXPOSURE(ms) = 3.225 MSB SIGNED 152.62 155.00 27.18 279.19 16383.00 2866.0033510981912 }}} {{{ WRITE(6,FMT='(5F10.2)') DARKL, DARKR, SDL, SDR, FLOAT(K) }}} Dark for the left and right side of the chip. The standard dev for the left and right. The K, which is the max DN (I think). === OCAMS === Level 0 OCAMS images have the images stored as two FITS images. The first FITS image is the standard 1024x1024 FITS image that most of the science team will use (if they want level 0). The second FITS image array is 1111 x 1043, which is the full sensor for OCAMS and contains the overscan region used for calibration. This format is fully described in the OCAMS SIS. These keywords are read by SPC. [[FITS Keywords]] describes what is done with each. 1. [[DN2TEMP.TXT]] a. OCAMS thermoresitors DNs are converted into physical units using the OCAMS Flight Software Engineer Dictionary (OREX-DOC-05.01-00259, Rev_C, Dated 12/04/2015) for SPC v3.0.1. a. SPC uses the 4 weighting parameters in [[DN2TEMP.TXT]] to calculate a linear weighted value for the effective temperature of the camera optics. For testing, we are weighting each thermoresistor by .25, providing each one with equal weight. a. The OCAMS Instrument Team will provide the final weighting during operations. There is no ICD or SIS for the information, so it will have to be done via email and documented as a memo. 1. '''Reduce Size''' - SPC will read the second image of the file, which includes the overscan regions. It will take the main 1024x1024 pixels for the image DN, loading the other portions into Flat and Dark variables for calibration. If the label doesn't match the image, it will output "INCONSISTENCY"; otherwise, it will show "OKAY". 1. '''Dark bias''' - The chip has four overscan regions that are averaged to calculate a day. The chip is broken into a left and a right half, and the darks are calculated accordingly. SPC will average all of the overscan pixels in each half, then print: Dark Left, Dark Right, Std Dev Left, Std Dev Right, Max. {{{ DN(i,j) -= DK(i,j) }}} 1. '''Frame Transfer correction''' - OCAMS does not have a shutter, so as the pixels are read, they are shifted one column at a time to be read, creating a smear or frame transfer artifact. As long as the image is not saturated, the bias can be removed by the same routine used for New Frontiers LORRI camera. {{{ Readout smear from NEW HORIZONS LORRI camera. Tf is average of frame scrub time T1 and frame storage time T2. Parameter alpha = (T2-T1)/(2*Tf) so T1=(1-alpha)*Tf and T2=(1+alpha)*Tf. Te is exposure time, n=number of lines, P is smeared image column and Q is unsmeared image column. }}} 1. '''Flat field correction''' - [[FLATFILES.TXT]] contains a list of flat files for each camera and filter. '''process_fits''' will read in the relevant file for the camera/filter and apply it to the image's DN. If [[FLATFILES.TXT]] does not exist, '''process_fits''' will error and quite. If the referenced file doesn't exist, SPC will just throw a warning (NO FLATFILE) and continue without applying a flat. . /!\ The code has a part that calculates and prints min and max, as well as scaling the flat: flat(i,j) /= max. However, it does not appear to be run because a boundary test ends the loop early. {{{ DN (i,j) *= FLAT(i,j) }}} 1. '''Scale''' - Normal SPC prefers to keep the data in unsigned integers; however, to do these corrections, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^16 / 2^14 (65535/16384 otherwise known as 4). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact on SPC's performance. === TAGCAMS (NavCam / NFTCam) === The following [[FITS Keywords]] are read by SPC. This describes what is done with each. 1. '''Temperature Calculation''' - Using [[DN2TEMP.TXT]], apply the polynomial coefficients to convert from DN to a temperature in C. Once that is done, then use the last four columns of [[DN2TEMP.TXT]] to calculate an effective temperature for the optical path. This is implemented just as for OCAMS (see above). {{{ temp = C1 * DN^3 + C2 * DN^2 + C3 * DN + C4 }}} 1. '''Reduce Size''' - Remove the black borders on the NavCam/NFTCam images. The TAGGACS suite (NavCam and [[TAGCam]]) contain overscan regions on all portions. SPC removes these to output a single array (stored in [[IMAGEFILES]]) that is 2592 x 1944. || Top || 50 || || Top boundary || 4|| || left || 134 || || left boundary || 6 || || bottom || 2 || || bottom boundary || 4|| || right || 10 || || right boundary || 10 || 1. '''Darks''' - NavCam and [[TAGCam]] darks are calculated from region 4 of the CCD. They are grouped into 8 bins, four on one row and four on the next. SPC calculates the dark (average) and the standard deviation for each pixel bin. It prints out each of the eight bins and the sum of all eight standard deviations. Then SPC removes the darks by simple subtraction. For TAGGAMS, this value is typically 150-180 DN. SPC checks, and if needed sets to zero, any negative values. Then it prints the minimum DN value after dark correction. {{{ DN(i,j) -= DK(i,j) }}} 1. '''Flat field correction''' - SPC will multiple every DN by a flat field correction that was read in from the [[FLATFILES.TXT]]. Flat files are implemented the same way as OCAMS (see above). {{{ DN (i,j) *= FLAT(i,j) }}} 1. '''Scale''' - Normal SPC prefers to keep the data in unsigned integers; however, to do the calibration, the data had to be converted into real (float) values. Because SPC stores the image data as unsigned integers, the real values must be rounded into integers. Because this results in a 1 DN addition of noise for no value, for this project, the integer values are scaled using 2^16 / 2^12 (65535/4096 otherwise known as 16). This reduces the impact of rounding floating point numbers into integers, and because SPC does a normalized cross correlation, it has no impact of SPC's performance. ----- == Level 1 == If the source is a real (or 32-bit float) value '''process_fits''' takes the original images' data float and calculates the maximum value. Then it scales the data to spread from 0 to 65536 to maximize the accuracy of the stored data. ---------- == Using process_fits == '''process_fits''' can be run in two modes: * Single file - just give the FITS filename at the prompt * Batch mode * If the file [[newpix.txt]] exists, then process_fits will search through ../OSIRIS_REx/NEW_IMAGES_YYYY_DDD to find the image and process. . /!\ The built-in search only goes from 2015 to 2022 directories. The files don't actually have to be in the correct directories, it is just good practice for organization. SPC will search all directories from 2015/001 to 2022/365. Here is a sample of the standard input: {{{ Input filename 2019-01-20T22-47-05.541_PCAM_L0b_V004.fits OKAY 2019 JAN 20 22:47:05.050 EXPOSURE(ms) = 100.000 MSB SIGNED 0.00 0.00 0.00 0.00 0.00 5.0000000000000000 P601296494J3 }}} ---------- Old - [[process_fits-3.0A2]] ''(Compiled by EP)'' ---- CategoryThreeOhOne