Small Footprint Processing

Summary

In order to avoid overwriting files, we create separate directories for each batch tiling job that we do. However, with the landmarks and mapfiles growing so much (50,000 files each), it is becoming unwieldily.

File Structure

There are two directories that are needed for this system to work.

1. run-master. This directory contains a full set of what is needed to do all SPC tasks (either actual or symbolic links). Basically, I will make symbolic links to the ~/Dropbox/Dawn-shape/Vesta directory for images, seed files, etc. Then for items that get changed, such as landmarks and mapfiles, I have my own copy. I use an rsync script, ["update.sh"] to keep my copy of these files up to date which what is in the Dropbox.

• There are two tasks that we use this directory for. Both of these functions are executed within the script ["setupRun.sh"]
• Generation of the big map. We use the command ["bigmap.e"] to figure out which files need to be copied into a working directory that are required for tiling. It is important to do this is its own directory, otherwise, you'd be changing stuff within Dropbox itself, which effects everyone else. b. A source for the required landmarks and mapfiles. Once a bigmap is created, then I take the USED_MAPS.TXT and copy it to LMRKLIST1.TXT. Then I use ["export.e"] to create a tar ball of (almost) all the files that I need do the tiling. The script ["setupRun.sh"] will copy this tar ball into a directory.

2. run - This is the working directory that will actually do the processing. I like to number them run1, run2, run3, etc.

• This directory is much smaller (about 300 MB - mostly because it keeps its own copies of SUMFILES and NOMINALS). I use the ["setup.sh"] script to build the symbolic links needed to "static" resources (images, seedfiles, shapefile, blemishes, etc.

Running things

Right now, the script ["setupRun.sh"] has the bigmap name as a variable at the top of the file that you must change (some day I will change it into an argument).

Tasks

1. Create a working (or run) directory.

Basically, you need a directory similar to ~/Dropbox/Dawn-shape/work_generic, but without the landmarks and mapfiles. You can create it from scratch by taking a blank directory and running the ["setup.sh"] script. Everything else should be generated by the ["setupRun.sh"] script.

• Example

   mkdir run3
   ~/Dropbox/Dawn-shape/script-limited-copy/setup.sh

2. Change the bigmap name in the ["setupRun.sh"]. You should have your own copy of this file somewhere on your filesystem (pick where ever you want this file - only one copy is needed)

• Example

   cd run3
   cp ~/Dropbox/Dawn-shape/script-limited-copy/setupRun.sh /Volumes/Media/run-master/
   vi /Volumes/Media/run-master/setupRun.sh
  
   ...
   which=ZS0224
   left=50
   right=950
   top=200
   bottom=800
   ...

3. Change the left, right, top and bottom to set the boundaries for the ["make_scriptT.in"] file. The default for a empty region (no previous overlap) would be left and top = 50, and right and bottom = 950. If there is overlap, you can change those value to limit the generation of landmarks (such as left=200, right=800).

4. In the working directory, run the ["setupRun.sh"]. You must be in the working directory because it will copy everything into that one. Note - their are path variables at the top of the file incase you want a different path for your Dropbox or the run-master directory.

• Example

   /Volumes/Media/run-master/setupRun.sh

5. ["setupRun.sh"] will figure out which maps are needed, copy them and setup everything (hopefully) you need to do a tile sequence. It uses ["bigmap.e"], so it takes about 5 minutes to run.

6. Run the generated script, ["run_script.b"] which was also created by ["make_scriptT.e"].

• Example

   ./run_script &

7. Watch things run. You can use ["find_nofitT.e"] to see the progress