Managing BTC Search Runs in Berkeley, 1998/03/28

...is done via three Perl scripts.

Step 1: Choose a directory

Actually, choose two. A working directory where all of the compressed files from Chile will land, and a final directory where all of the surfaced files (loaded into the database) will go. The working directory should be a subdirectory of one of the /home/astro*/deepsearch directories. It's probably best to have a new working directory each night. The final directory is one of /home/astro*/deepsearch. The working directory needs to have something like .8gig free. The final directory needs to have something like <=2gig free.

For example, for mar31, I would suggest the following directories:

Step 2: Get the scripts

You can find them, for now, in /home/astro20/deepsearch/ctio98mar27. Copy the following files to your working directory:

Step 2.5: Edit the scripts

You will probably need to edit all four of these scripts to tell them where the working directory is. hotrcp.perl additionally needs to know the "Compression Machine" in chile and the directory where to find the compressed files. The folks at the telescope can give you this information.

Step 3: Run the scripts

This is what they do:

hotrcp.perl
Looks down to Chile to see what is compressed. Pulls up any files which it hasn't pulled up yet. Run this in its own window. Note that whoever is running this should run it on panisse, and should be in the .rhosts file on ctioa7.ctio.noao.edu.

btcsurfload.perl
Uncompresses, surfaces, and loads images into the database. Probably run this one on cha-am, because it tends to be slow. (It's convenient to run all of hte scripts on windows open on the same workstation, for ease of monitoring, but each using a different CPU.)

monitor.perl
Reduces, finds the seeing and S/N, and writes those to a log file.

sortmonitor
For a good time, in the directory where all the data is showing up, run "perl -w ./sortmonitor". That will sort the logfile of monitor.perl by grid and seeing.

The Log Files

Each script reads the log file written by the previous script to find out what is avilable for it to process. It writes out its own log file to keep track of what it has already done. All of these log files are relative to the working directory -- which must be your current directory when you run each script!

Not all the scripts have the same format, but typically they list the image filename (obj*.imh or similar), the priority (a non-negative integer), and the title (which is the title put in at the telescope with the amplifier nubmer and filter number appended to it).

compressed
This log file is actually the last one from Chile. hotrcp brings it up regularly to check what those yahoos down there have finished. It lists which files have been processed and compressed down there. hotrcp chooses one among those with the highest priority which it hasn't done yet.

This file is sometimes empty if hotncftp is in the process of getting a new one.

transferred
The log file that hotncftp writes. These are the files which have made it up to Berkeley. "wc transferred" will tell you how many have been copied up so far.

uncompressed
The log file that btcsurfload writes. These files are uncompressed and ready for action. This log file includes the name of the file as it appears in the database in addition to the information found in the other log files.

00monitorlog
The log file that monitor writes. Has all kinds of neat information.

Error Recovery

Errors won't happen.

I also happen to have this very nice bridge available. Governor Wilson says they're going to build a new one, and I can sell you this old one -- still in servicable shape! -- at an excellent price.

To first order, if the script fails, you can just start it again. They tend not to write log files until they have succesfully completed an operation, so if something didn't succesfully compelte, it will just start over. One exception is btcsurfload.perl; if the decompression failed, it will rename the .H compressed image file so that it doesn't keep trying the same file over and over and over again. You have to rename it back to just its obj*.fits.H name if you want btcsurfload.perl to try again. In practice, I've not seen the decompression fail. (And I might be able to get you an additional discount on that bridge.)