Block 2 - Ingest V2
This block additionally ingests images from FITS files into the SPC world.
- Copy FITS images
- Make a list of images
- Update kernels
Set up process_fits
Set up make_sumfiles
Copy FITS images
Install a copy of the FITS images for each day into the NEW_IMAGES_YYYY_DDD location. Use 3 digits for the id (i.e. 012)
ID=339 cp <files> /[path]/OSIRIS_REx/NEW_IMAGES_YYYY_ID
Make a list of images
SPC can ingest images one at a time via process_fits, but we prefer to run it in batch. If newpix.txt exists, then process_fits will use that as input. It will search through all directories in the form of ../OSIRIS_REx/NEW_IMAGES_YYY_DDD .
For tracking purposes, we will name the list of images via data shortName and put it into lsupport. The shortName is the short hand for mission name (Ap, PS, OA, DS, OB, Re) and the sequence number of the images set, i.e. the first delivery is 1, the next is 2, etc.
cd /path/ops/OSIRIS_REx/DATA/NEW_IMAGES_YYYY_$ID ls > ../../newpix.$ID cd ../Approch150B.__ mv ../newpix.$ID lsupport/ relink.sh lsupport/newpix.$ID newpix.txt ls -l newpix.txt | tee -a notes
Note: You may want to hand edit newpix.txt to remove images that you don't want (e.g. color filter images, overexposed for star fields)
Typically, you will need to update the spacecraft (SPK or file extension bsp) and pointing (CK or file extension bc). It may be necessary to update the spacecraft leap second clock also (lsp). Finally, if you get a new POLE, the planetary constants kernel for Bennu needs to be updated (PCK or file extension tpc).
- Remember that many spacecraft kernels have both a reconstructed past history and a future predicted plan. As such, if you forget to update the SPKs, you could have images that are ingested using the predicted part of an outdated kernel, which won't throw an error.
- Remember that SPICE will load kernels in order and use the last loaded kernel for a given time, so the newest kernels should always be at the end of a list.
Edit make_sumfiles.txt. We want to use chmod to avoid mistakenly over writing such an important file.
chmod 664 make_sumfiles.txt vi make_sumfiles.txt # add the new kernels chmod 444 make_sumfiles.txt
If the PCK has changed, update INIT_LITHOS.TXT also.
Review the following
PROCESS_FITS -v | tee ingest ls -l DATA/DN2TEMP.TXT | tee -a ingest ls -l DATA/FLATFILES.TXT | tee -a ingest ls -l newpix.txt | tee -a ingest
Review the following
ls -l DATA/BIAS.TXT | tee -a ingest # seldom used ls -l DATA/SCOBJ_SHIFT.TXT | tee -a ingest ls -l DATA/TMP2FOCLEN.TXT | tee -a ingest
Run process_fits, make_sumfiles
For this step, we want to log the output to ensure flat field etc are properly applied. We want to capture the output from make_sumfiles to get the list of new image name, which we put in lsupport/imageList.<shortName>
chmod 664 make_sumfiles.in PROCESS_FITS | tee -a ingest chmod 444 make_sumfiles.in echo 0 | MAKE_SUMFILES | tee out cut -c 1-13 out | sed -e "s/^/ /" | sed '1,3d' > lsupport/imageList.$ID
Note: You can run evalReg.sh with lsupport/imageList.Ap\ to see if there are images that you don't want (here you should focus on overexposed starfield images). If so, you can make a list of them and batchTuck.sh them. imageList won't skip them, so you might want to edit that file (as well as make a backup original copy)
Note: Review the log for PROCESS_FITS to ensure that the threshold amounts make sense. If you receive an 8-bit image, the lower threshold may be wrong and need to be updated for those images (look at the max DN value). You can use batchThreshold.sh to fix that
Before running dynamics, you need to make sure that DYNAMICS.TXT has been updated with the new images. Take the image list (lsupport/imageList.<shortName>) and add it into DYNAMICS.TXT before the "END". Update the uncertainties as required (for the new images and any update for the old images if needed. Note, if an image is not listed DYNAMICS.TXT, then it will not have it's uncertainties updated.
echo "# Day $ID" >> DYNAMICS.TXT vi DYNAMICS.TXT Gkddpkk :r lsupport/imageList.\
Evaluate the images
At this point, we need to evaluate the images. One needed step is to make thumbnails
These will be copied to ~/send (the place we dump stuff because of running on an external server
- Check the DN range. When making thumbnails, review the DN range. If the average value is below our threshold, then we need to update them (batchThreshold.sh)
- Are they blank. If so, they can be tucked away (batchTuck.sh)
- Are they over exposed. Again, tucked
Finish the block
We use blockFinish to store all needed artifacts to the log directory.
i.e. blockFinish Ap4
echo 1 | blockFinish ps$ID
(Compiled by EEP)