Running GMT on the PSI Hawk Cluster

To begin making projected geoTiffs for ArcGIS and ISIS, start with the txt files generated when making the photometric cubes. The scripts used produce the following txt files:

The above files need to be uploaded to the Hawk cluster, while keeping their parent directory structure.

Go to the photocube directory for each bigmap 

rsync -hapvP --exclude=*.cub --exclude=*.prt N1* [email protected]:/home/jweirich/Tethys/geoTiffTethys/LEADHL/

Also copy the output from backplanesGMT and config.txt

Ex. config.txt 

BIGMAP=LEADHL
ARGR=-R226.209259/261.399750/3.66536975/33.9945526
ARGI=-I801+n
DEGPX=0.0378641483770287
PROJECT=60300
BODY=Tethys
RUNALL=No
VERSION=1

The basic command being run is similar to the Topography step above: 

echo "Blockmean"
gmt blockmean $var $argR $argI  > ${product}out
echo "sphereinterpolate"
cat ${product}out |  gmt sphinterpolate $argR $argI -G${product}.nc

... where $product is going to be IoverF, a, e, or i. Everything else is just a wrapper to deconflict all the files since we're running in parallel.

I have the GMT commands above in a script called makeaGeoBig.sh which reads most of the inputs from config.txt. makeGeoBig.sh is called by makeIoverFGeoBig.sh, makePhaseGeoBig.sh, makeEmissGeoBig.sh, and makeIncGeoBig.sh

The key lines in these scripts are: (Ex. below is for Incidence) 

echo "getting ready for GMT ..."
~/bin/bustGrid $bigmap-i.TXT > bustedi-$bigmap-$img.txt
paste ../$bigmap-lonlat.txt bustedi-$bigmap-$img.txt > $var

echo "calling makeGeoTiffBig.sh"
sh ~/bin/makeGeoTiffBig.sh $var $product

$product is as before, while $var is IoverF.ll, phase.ll, emiss.ll, or inc.ll

The above four scripts are called by run4.sh, which is essentially a copy/paste script with different images. Note the use of & and "wait". & is to get multiple processors running on each machine, and "wait" is to keep the script running until all four processes complete.

The key lines from run4.sh look like this: 

cd $img
sh ~/bin/makeIoverFGeoBig.sh $img &
sh ~/bin/makePhaseGeoBig.sh $img &
sh ~/bin/makeEmissGeoBig.sh $img &
sh ~/bin/makeIncGeoBig.sh $img &
cd ..

wait

Since there are 8 processors per Hawk node, we want to call 2 images per node. To do this we have a script that uses sbatch commands. Here's what one of those looks like: 

#\!/bin/bash
#
#SBATCH -J Call1.sh              # Job name
#SBATCH -t 2-0:00 # time (D-HH:MM)
#SBATCH --partition=cpu
#SBATCH -o slurm.%N.%j.out # STDOUT
#SBATCH -e slurm.%N.%j.err # STDERR
 
sh ~/bin/run4.sh N1563651588 &
sh ~/bin/run4.sh N1563651648 &
 
wait

Rather than type each by hand, I instead have a script (makeScript.sh) to make a bunch of these: 

# 29 Sep 2021 - John R. Weirich
# Script to call different photo scripts

imgList=$1

if [ -z $imgList ]; then
        echo "Please select a list of images"
        echo "Usage: <program> <Image List File>"
        exit
fi

list=`cat $imgList`

top=1
count=0

rm -f Call*.sh

for i in $list
do
 if [ $top = "1" ]
 then
  let count=$count+1
  out="Call$count.sh" 
  echo "#!/bin/bash" > $out
  echo "#" >> $out
  echo "#SBATCH -J $out              # Job name" >> $out
  echo "#SBATCH -t 2-0:00 # time (D-HH:MM)" >> $out
  echo "#SBATCH --partition=cpu" >> $out
  echo "#SBATCH -o slurm.%N.%j.out # STDOUT" >> $out
  echo "#SBATCH -e slurm.%N.%j.err # STDERR" >> $out
  echo " " >> $out
  echo "sh ~/bin/run4.sh $i &" >> $out

  top=2
 else
  echo "sh ~/bin/run4.sh $i &" >> $out
  echo " " >> $out
  echo "wait" >> $out

  top=1
 fi

done

if [ $top = "2" ]
then
 echo " " >> $out
 echo "wait" >> $out
fi

To fire everything off, you'll make a copy/paste that will look like this: 

sbatch Call1.sh
sbatch Call2.sh
etc.

Once all those processes are finished, you'll have converted each *.txt file into a *.nc file. Now pull that back down to your local machine to run through GDAL. 

rsync -hapvP --prune-empty-dirs --include="*/" --include="*.nc" --exclude="*" [email protected]:/home/jweirich/Tethys/geoTiffTethys/LEADHL/ .

Once downloaded, run 

sh ../bin/wrapperAfterClusterImage.sh imgListLEADHL

wrapperAfterClusterImages.sh is this: 

# 30 Sep 2021 - John R. Weirich
# Wrapper to turn all the *.nc into geoTIFFS

imgList=$1

if [ -z $imgList ]; then
        echo "Please select a list of images"
        echo "Usage: <program> <Image List File>"
        exit
fi

list=`cat $imgList`

for i in $list
do
 cd $i
 sh /usr/local/spc/bin/afterClusterImage.sh IoverF $i
 sh /usr/local/spc/bin/afterClusterImage.sh a $i
 sh /usr/local/spc/bin/afterClusterImage.sh e $i
 sh /usr/local/spc/bin/afterClusterImage.sh i $i
 cd ..
done

And afterClusterImage.sh is this: 

# 28 Sep 2021 - John R. Weirich
# Make the various geoTiffs using GDAL
# Run from geoTiff[directory]/
# 30 Sep 2021: Began modifying, then undid changes (hopefully they are undone correctly!)

# Usage in geoTiff directory : sh /usr/local/spc/bin/afterClusterBigmap.sh <product/type name> <image>

type=$1
img=$2

if [ -z $type ]; then
        echo "Please select the type"
        echo "Usage: <program> <type> <Image>"
        exit
fi

if [ -z $img ]; then
        echo "Please select the image"
        echo "Usage: <program> <type> <Image>"
        exit
fi



 if [ ! -e ../config.txt ]
 then
  echo "Make config.txt!"
  exit
 fi

 source ../config.txt

 if [ ! -e ./${type}.nc ]
 then
  echo "Make ${type}.nc!"
  exit
 fi



bigmap=$BIGMAP
argR=$ARGR
argI=$ARGI
degPx=$DEGPX
proj=$PROJECT
body=$BODY
runAll=$RUNALL
ver=$VERSION

echo "$bigmap"
echo "$argR"
echo "$argI"
echo "$degPx"
echo "$proj"
echo "$body"
echo "$runAll"


# Make normal GeoTiff

echo "gdal translate"

gdal_translate -of GTiff -b 1 -a_srs ../support/$proj.prj  NETCDF:${type}.nc ${bigmap}-${img}-${type}-v${ver}.tif

#Make GeoTiff readable by ISIS
gdal_translate -of ISIS3 -tr $degPx $degPx -r bilinear -b 1 -co TARGET_NAME=$body -co DATA_LOCATION=GEOTIFF -a_srs ../support/$proj.prj NETCDF:${type}.nc ${bigmap}-${img}-${type}-ISIS-v${ver}.lbl

Note: See "Grid to GeoTIF" for a better program than gdal_translate, since gdal_translate doesn't always write the projection.