Reference/Search Runs with the BTC

December 26, 1997

At any time if you have questions on any of this, particularly the laptop configuration, contact Rob Knop at one (prefereable of the first two) of:

All of this is a little easier than it used to be because everything can be done on the BTC UltraSparc named ctioa7. There is no longer a need to bring the laptop to Chile.

Table of Contents:

Before You Leave Berkeley

Make sure you know the fields.
Bring finding charts and a list of fields and coordinates. Make sure you understand where on the BTC footprint the listed coordinates are. Our standard is that the list is the center of chip #1, the northwestern chip.

Make sure you have DLT tapes.
One tape per night of observing should be just sufficient, in that you can just fit two nights on one tape. Three tapes per every two nights might be safer.

Setting up Directories

You should do or check all of these steps each night. First, make sure there is enough space on the giant BTC drive (probably /ua72) for all of a couple nights' worth of data. 10gig should be sufficient. If there isn't bug the CTIO folks to find out what of other peoples' stuff you can delete.

Make yourself a "parent" directory, such as:


This assumes that /ua72 is the giant BTC 23 or 37 or whatever the hell it is gig drive. If it's not, then you will need to change /ua72 to something else all throughout these instructions, and this will be your last reminder of that.

IF, horrors, this directory already exists, you might want to do "rm -r *" in it to clean it out so that you are sure that you've got everything set up right when you are done with all this.

Then, make the following subdirectories in that parent directory:

Of course, put the correct evening date in place of "97dec27" for the latter three. Make a new set of those directories each night.

Get the hcompress software. This can be a challenge. It might actually already be present on ctioa7 in "~btc/bin". Give it a look. If not, then get the file "~rknop/hcompress.tar.Z" from, build the software, and copy the requisite executables into "~btc/bin".

Finally, go back to your home directory ("cd ~") and edit your .rhosts file. Make sure it has the line: deep

Setting Up the BTC Software

This is mostly covered in the BTC documentation. The only note is that you should make sure that the raw FITS files obtained by the BTC go to the proper directory. If you don't do this, then nothing will work, we won't find any supernovae, we won't have anything to send to HST, and we will be the laughing stock of the astronomical community.

In the example above, the directory where you should tell the BTC to write its files is:


Substitue your current evening date for 97dec27.

Important Note: When taking data with the BTC, the scripts will be much happier if you observe the following guidelines:

Set Up IRAF for the Automated Scripts

IRAF will already be set up on the BTC for general use. You need to do an additional setup of IRAF for the automated scripts. To do this, issue the following two Unix commands:

That will create a file in the directory /ua72/btc/deepsearch/iraf. Next, copy all of the following files (which may be found on in ~rknop/iraf) to this IRAF directory:

Issue the command "which perl". If you get anything other than "/usr/local/combin/perl", then you need to edit the first line of the following scripts to point to the correct location of perl:

Issue the following command to make all the perl scripts executable:

chmod 755 listmefits postbtcprocess makefitsfilter getfitsname ctiocompress.perl

Finally, edit your file. Find the block where there are about ten or so lines of "task" statements in a row. Below that, add the following lines:

Note the dollar signs! Put one in the wrong place, or leave one out, and Western Civilization will end.

Closer to the top of the file, locate the "set imdir" command. Change the definition there to "hdr$pix/" (or maybe "HDR$pix/", I'm not sure which one actually works.)

Taking Zeros and Flats

Do this during the day at CTIO. Use the normal procedure. Take many, but not too many, and take an odd number of each. Between nine and fifteen should be a good number for each filter. Take dome flats in every filter you anticipate observing during the night.

To combine the flats, you may use your own IRAF savvy. However, I do not yet fully trust the FITS kernel in IRAF, so I would recommend the following procedure:

Do this from eiter the BTC home directory or from "/ua72/btc/deepsearch/iraf". Type "cl" to run iraf.

Type "mosred".

Type "cd /ua72/btc/deepsearch/97dec27_work/".
As with all of these instructions, substitue the correct date.

Set up the instrument.
Do "epar setinstrument". Set "site" to "ctio", and "instrument" to (probably) "btc". Ask the local BTC support folks if this is still correct. Type ":go" and hit RETURN when done. Scan the options of the next two parameter files you see, doing ":wq" with each one when you are satisifed either that you are happy with the options, or that you are clueless enough that you have no idea if you should be happy with the options.

Read your Zero Frames into IRAF
OK. Maybe you don't have to do this. However, there are two reasons to do it. One, I don't fully trust the FITS kernal in IRAF. Two, if you do, you don't risk accidentally processing your only copy of the raw file straight off of the telescope. (You definitely want to keep an original copy of that.) Use the procedure "rfits" to read FITS files out of "/ua72/btc/deepsearch/97dec27/" into IRAF files in "/ua72/btc/deepsearch/97dec27_work/".

run ccdlist
This is an IRAF command that will show you what you've got. It also creates the crucial "subsets" file. If you don't understand that, don't worry much about it, and run ccdlist a lot.

run zerocombine
In IRAF, type "epar zerocombine". Set "process" to "no". Give a wildcard or list file (with @) to specify all the zero (aka bias) frames you've just taken. Name the combined Zero file something like "Zero97dec27". When you are done setting up your parameters, hit the colon key and type "go". Hit return. Wait a long time.

Process the combined Zero file
In IRAF, type "epar ccdproc". Make sure you are editing the parameters for the mosred version of ccdproc. Set the following options to the following values:
When you are done, use ":go" to run the ccdproc.

Clean up
Keep the combined "Zero97dec27*" files in the work directory. However, you can use "imdelete" to blow away the individual zero files in "97dec27_work/". NEVER delete anything out of the raw "97dec27/" directory.

Read your raw Flat frames into IRAF
See "Read your Zero Frames into IRAF" above, only now do it to your domeflat frames.

run ccdlist
Again issue the command "ccdlist" to IRAF. Make sure to enjoy it.

run flatcombine
In IRAF, type "epar flatcombine". Set "process" to "no". Specify all of your flatfields. Name the combined something like "Flat97feb9R". Make sure "subsets" is set to "yes". Type ":go" to do it. Wait.

Process the combined flatfield
Do "epar ccdproc". Set all of the options the same as what you did for zerocombine with the two following exceptions:
zeroWhatever is the name of your combined Zero.
When you are done, use ":go" to run the ccdproc.

Setting Up the Reduction Scripts


It's less complicated than it used to be, but that doesn't mean it's not still complicated. There are three things going on:

This used to convert raw BTC IRAF files into FITS files. Now they come out as FITS. All this program does is monitor the data coming off of the BTC, and when it thinks the data is there and stable, logs it in the file "/ua72/btc/deepsearch/97dec27/obtained" (where the current date is substituted for "97dec27"). It also assigns the internet transfer priorities on the fields based on the title which of course the person operating the BTC set right.

This watches the "obtained" file written by the previous process. It flatfields all of those files and sticks the result in the directory "/ua72/btc/deepsearch/97dec27_clean/". It writes the logfile "processed" in that clean directory to let the world know what it did.

This watches the "processed" log file, and when something is there, runs fcompress on it to produce a .H compressed file in the clean directory. It logs what it has done to "compressed". The latter is the logfile that the internet transfer routines running in Berkeley watch.

Set up makefitsloop

Open a shell on ctioa7.
Separate from the place where you plan to aquire or do "quick look" evaluation of the data. You may want to telnet from one of the other workstations in the control room to ctioa7.

cd to "/ua72/btc/deepsearch/iraf/" and type "cl". Type the following commands to set up IRAF: As always, substitute the current evening date for "97dec27".

Set your instrument
Run "setinstrument" as you did for combining zeros and flats.

epar makefitsloop
Set the following parameters to the following values:
matchA pattern of raw IRAF files of interest written by the BTC aquisition software. Probably "obj*fits".
Remember trailing slashes on all IRAF directory parameters!!!!! Otherwise Western Civilization will fall. Also, remember to substitute the current evening date for "97dec27". Type ":wq" when you are done editing the parameters for makefitsloop.

Just type "makefitsloop" into the "mo>" prompt of IRAF, and all should be well.

In Case of Error
Panic. Then call Rob. If you have to restart the script, give the command "flpr" to IRAF a couple of times before you do so.

Set up btcprocess

This assumes that you made your Zeros and Flats as specified above, and that you left your combined Zero and Flat frames in your "97dec27_work" directory (where, as always, you substitue the current evening date for "97dec27").

Get yoursef a new shell
Do this just like you did it for makefitsloop above... but, obviously, it has to be a new shell! It must be running on ctioa7, although you may want to telnet in to ctioa7 from a different workstation for convenience.

On ctioa7, issue these commands: As always, substitute... well, you know.

Set your instrument
Do "epar setinstrument" much as you did it on the BTC computer during combining Zeros and Flats.

Make sure all your zero and flatfield frames come out correctly. In particular, make sure that ccdlist lists zeros as zeros and flats as flats, that it gets the filters right, and that it gets the chip number right. If it doesn't, Panic. Then consult the BTC support people and/or call Rob.

epar ccdproc
Set it up for your cleaning. Some of the key parameters which should be set are:
zeroThe name of your combined zero (e.g. "Zero97dec27*.imh")
flatA wildcarded expression which includes all your combined flats (e.g. "Flat97dec27*.imh")
Hopefully, everything else should already be set to some sane value. If they aren't, then even Panic won't help.

You may want to also "epar mosred" to make sure that you do not have a backup directory set. Don't use that feature, because the scripts already take care of that kind of thing.

Excellent, Dude!
Just thought I would insert that.

epar btcprocess
Set parameters as follows:

Remember trailing slashes on all directories in IRAF paremeters!!

As always, substitute your current evening date for "97dec27".

(I.e. type "btcproces" at the "mo>" prompt in IRAF.) At this point, everything should work. When it doesn't, be sad. Go over all the steps above, and make sure everything is right. When it is, be sadder. Cry, even. Finally, Panic. Then call Rob.

Watch the Script!
In case it dies or does something wrong, you don't want to waste any time before being sad about it. If it dies, issue the command "flpr" to IRAF a couple of times before restarting the script. You may also want to make sure all your parmeters are set correctly.

Set up ctiocompress.perl

Open a shell on ctioa7
Use the tcsh. ksh is for wimps. Again, this must be running on ctioa7, but you may find it convenient to telnet from a different computer. Obviously, this shell is a third shell different from both of the last two shells you opened up.

Edit /ua72/btc/deepsearch/iraf/ctiocompress.perl
Change the first line to point to the proper location of perl on the Compression Machine. It may already be correct, especially if you followed all of the earlier instructions.

ctiocompress.perl requires some configuration. There are some variables at the top which should be set to:
$bindirThe directory where the hcompress software is.
$maxtapesize(doesn't matter, keep the default)
$tapeOKstring(doesn't matter, keep the default)

As always, substitude your current evening date for "97dec27".

Issue the command "/ua72/btc/deepsearch/iraf/ctiocompress.perl". This should start the compression script going.

Give it a look. You may have to restart it every so often... hopefully not.

All Those Damned Log Files

Vast quantities of log files are written by the scripts at each stage of the execution. This serves two purposes:

The log files all share a common format. Each line of the log file represents one file which has been processed by the script that write the log file. Each line looks like:

    filename   priority   title

The name of the file in question, without the ".fits".

A priority which lets the next stage of the whole process know which files are more important. Higher numbers mean higher priority. These priorites are assigned at the first step through some mysterious process that even the writer of the scripts does not understand. After that, each script just keeps the priority the file had in the previous script.

The default priority is 5.

This is wholly for human consumption. It's so you can hope to have a clue of what a file is when its filename is obtuse.

The specific log files

This is probably in /ua72/btc/deepsearch/97dec27/ (where you substitute your current evening date blah blah blah). These files have been noted and logged as available. "makefitsloop". "btcprocess" looks here to decide which files it needs to grab and clean.

Probably in /ua72/btc/deepsearch/97dec27_clean/. These files have been cleaned by "btcprocess". "ctiocompress.perl" looks here to decide what to compress next.
Probably in /ua72/btc/deepsearch/97dec27_clean/> These files have been compressed by ctiocompress.perl. The LBL script "hotncftp" should look here to decide what to transfer up to Berkeley next.

It is probably worth starting all of the log files over with each night of observing. This will happen automatically if you put the date in all of your sundry directory names, and make new directories each night. (And, of course, make sure to set all of your IRAF parameters up correctly. Yes, all ten million of them.)

Editing Log Files

In practise, the only log file you will ever need to edit is "compressed" in the Clean FITS directory on the Compression Machine. This is what hotrcp up at Berkeley uses to decide its priorities, and hotncftp is the biggest bottleneck.

PROCEED WITH EXTREME CAUTION. Here's the scenario: you want to change a priority, so you read a log file into an editor. You futz with it a bit. Meanwhile, the script that writes that log file has added another line to it. You write out your edited file-- thereby removing the line that the script had just written to it. Suddenly your log file is incorrect. This is very bad.

Of course, if you are enlightened and you use a recent version of emacs to edit the log file, when you try to write it out it will detect if the file has been changed since you've last read it, and will prompt you rather than merrily overwriting the log file. This leads to the cardinal rule of editing log files: NEVER, EVER USE VI OR PICO TO EDIT LOG FILES. Use emacs. If you don't know emacs, then tough, you don't get to edit log files. And, when you do edit log files, you will save yourself frustration if you do it quickly, and write out what you've done.

Maybe in the future there will be a more user friendly way to do this, but for now this is how it's gotta be done.

There are two reasons why you might want to edit a log file:

Changing the priority of an image
You have to do this in the latest log file the image has been listed in, otherwise it won't make one whit of difference. In practice, you will probably only need to edit the "compressed" log file to change the priorities for hotrcp, since that will be the biggest bottleneck.

Getting a script to redo something.
This is scary. However, for example, if you remove a line from the "processed" log file, the btcprocess script on the Reduction Machine will grab it from the Aquisition Machine and process it again. You would also have to remove it from the "compressed" log file, and any log files up at Berkeley, to get it the rest of the way through the system.

In general, unless you've set something up horribly wrong (don't do that), and unless the scripts are all terribly broken (in which case it's all pretty hopeless anyway) you shouldn't ever have to do this with the CTIO stuff. If you think you do, Rob might be happier if you just called him.

Backing Up and Other End-of-Night Activites

The first thing you should do at the end of the night is wait until all of the "obj*.imh" files have made their way through the makefitsloop, btcprocess, and ctiocompress.perl scripts. These scripts really shouldn't fall that far behind, so this doesn't entail that much waiting.

Once these are all done, and all of them are doing nothing but sleeping over and over again waiting for more stuff to do, it is safe to kill each of them.

At this point, you may wish to write one or more DLT tape. This may be done from ctioa7. Note that the BTC documentation is not entirely correct about either the device name of the DLT drive or about whether the btc account has the proper default tape drive. Pay very close attention to everything you do.

Last time Greg and I were there, at the end of the first night we used "tar" in "/ua72/btc/deepsearch" to copy everything from the first night (about 4gigs worth) to DLT tape #1. The second night, we put that tape back in, did a "mt eom" to jump to the end of the tape, and then used tar to write _just_ the files from the second night (another 4-5 gig) as a second tar file on DLT tape #1. Then, we put in a fresh new DLT tape #2 and wrote the whole schmear (everything underneat /ua72/btc/deepsearch) to one gigantic tar file.

If memory serves (and it doesn't, really), it took .... well, hell, I can't remember, but I think it was something like 2 or 3 hours to write all 9-10 gigs of everything to tape #2.

Whatever you do, make sure you've got at least two copies of everything, especially the raw FITS files, before you leave. If you get back and it turns out you boffed up the tape writing, we will be so sad that it will be just a little bit extraordinary.

Last modified: Fri Dec 26 11:13:49 1997