Small_Footprint_Processing

Small Footprint Processing

Summary

In order to avoid overwriting files, we create separate directories for each batch tiling job that we do. However, with the landmarks and mapfiles growing so much (50,000 files each), it is becoming unwieldily.

File Structure

There are two directories that are needed for this system to work.

1. run-master. This directory contains a full set of what is needed to do all SPC tasks (either actual or symbolic links). Basically, I will make symbolic links to the ~/Dropbox/Dawn-shape/Vesta directory for images, seed files, etc. Then for items that get changed, such as landmarks and mapfiles, I have my own copy. I use an rsync script, update.sh to keep my copy of these files up to date which what is in the Dropbox.

There are two tasks that we use this directory for. Both of these functions are executed within the script setupRun.sh
Generation of the big map. We use the command bigmap to figure out which files need to be copied into a working directory that are required for tiling. It is important to do this is its own directory, otherwise, you'd be changing stuff within Dropbox itself, which effects everyone else.
b. A source for the required landmarks and mapfiles. Once a bigmap is created, then I take the USED_MAPS.TXT and copy it to LMRKLIST1.TXT. Then I use export.e to create a tar ball of (almost) all the files that I need do the tiling. The script setupRun.sh will copy this tar ball into a directory.

2. run - This is the working directory that will actually do the processing. I like to number them run1, run2, run3, etc.

This directory is much smaller (about 300 MB - mostly because it keeps its own copies of SUMFILES and NOMINALS).
I use the setup.sh script to build the symbolic links needed to "static" resources (images, seedfiles, shapefile, blemishes, etc.

Running things

There are two major things that are needed to be done. First, setup the file structure, which is done only once (tasks 1 and 2). Second, setup and run a tiling operation for a big map, something that is done numerous times (tasks 3, 4, 5 and 6)

Tasks

1. Copy scripts. Get your own copies of the scripts and change the paths as required. Note, an absolute path is needed for symbolic links when used for

Example

 cd ~/Work
 cp ~/Dropbox/Dawn-shape/scripts-limited-copy/setup.sh .        
 cp ~/Dropbox/Dawn-shape/scripts-limited-copy/update.sh .
 cp ~/Dropbox/Dawn-shape/scripts-limited-copy/setupRun.sh .

 vi setup.sh

  ...
  # You need to change this to match your path
  # You cannot use ~ if you have things on different volumes
  static="/Users/epalmer/Dropbox/Dawn-shape/Vesta"    <- change this path
  ...

 vi setupRun.sh

  ...
  dropPath=~/Dropbox/Dawn-shape          # path to dropbox     <- change this path
  localMain=~/Work/run-master    # path to master directory    <- change this path
  ...

2. Setup the run-master directory. The run-master has everything in it that you need to do almost everything (tiling, coverage, bigmaps, shape, etc.) I assume that you have a directory structure with a work directory for your files. Once both setup.sh and update.sh have been run, the run-master directory will be a full and complete directory that can run about any program.

Example

 cd ~/Work
 mkdir run-master
 cd run-master
 ../setup.sh
 ../update.sh       <- this copies 100,000 files (3+GB) and can take a while

3. Create a working (or run) directory.

Basically, you need a directory similar to ~/Dropbox/Dawn-shape/work_generic, but without the landmarks and mapfiles. You can create it from scratch by taking a blank directory and running just the setup.sh script. Everything else should be generated by the setupRun.sh script.

Example

 cd ~/Work
 mkdir /run3
 cd /run3
 ../setup.sh       <- or wherever you put this file - you may need to change the path in this file.

4. Configure (or just check) the parameters that you will be using for the run. You will do then whenever you need to deal with different overlap of landmarks (usually when you finish a swatch of the surface, such as when you finish the set from ZS0200 to ZS0270).

setupRun.sh will generate a new make_scriptT.in based on parameters in the file. Change the left, right, top and bottom to set the boundaries for the make_scriptT.in file. The default for a empty region (no previous overlap) would be left and top = 50, and right and bottom = 950. If there is overlap, you can change those value to limit the generation of landmarks (such as left=200, right=800). If the autogenerated make_scriptT.in doesn't work for you, just replace it with something that you tailored by hand and run make_scriptT.e

Example

 vi ~/Work/setupRun.sh

  ...
  left=50     # Makescript left boundaries
  right=950   # Makescript right boundaries
  top=200     # Makescript top boundaries
  bottom=800  # Makescript bottom boundaries
  ...

5. In the working directory, run the setupRun.sh with the bigmap name that you want to tile. You must be in your working directory because it will copy everything into that one. It will update the run-master directory and do processing within that file structure, to include changing files.

setupRun.sh will remove all old files, create directories and symbolic links. Then it will run bigmap which is a "cheat" to figure out the overlapping mapfiles that must be exported. bigmap uses the <mapname>.IN file that are located in the Dropbox/Dawn-shape/Vesta directory. Then it exports the needed files and configures them in the working directory. Finally, it will auto-generate make_scriptT.in and run make_scriptT.e. The whole process takes about 5 minutes.

You will have to explicitly start the tiling process (and I suggest you check things to ensure they make sense).

Example

 cd ~/Work/run3
 ../setupRun.sh ZS0216      <- This will prepare everything that is needed to tile bigmap ZS0216

6. Run the generated script, run_script.b which was also created by make_scriptT.e. I suggest checking make_scriptT.in to see if it makes sense. Also, an ls *INN will show all the parameter files that lithos will be using.

Example
```
 ./run_script &
```

7. Watch things run. You can use find_nofitT.e to see the progress