Reference/Search Runs with the BTC

December 26, 1997

At any time if you have questions on any of this, particularly the laptop configuration, contact Rob Knop at one (prefereable of the first two) of:

• rknop@lbl.gov
• (510)486-5878
• (510)234-6494

All of this is a little easier than it used to be because everything can be done on the BTC UltraSparc named ctioa7. There is no longer a need to bring the laptop to Chile.

Table of Contents:

• Before you leave Berkeley
• Setting up Directories
• Setting Up the BTC Software
• Set Up IRAF for the Automated Scripts
• Taking Zeros and Flats
• Setting Up the Reduction Scripts
• All Those Damned Log Files
• All Those Bloody Directories
• Backing Up and Other End-Of-Night Activities

Make sure you know the fields.

Bring finding charts and a list of fields and coordinates. Make sure you understand where on the BTC footprint the listed coordinates are. Our standard is that the list is the center of chip #1, the northwestern chip.

Make sure you have DLT tapes.

One tape per night of observing should be just sufficient, in that you can just fit two nights on one tape. Three tapes per every two nights might be safer.

You should do or check all of these steps each night. First, make sure there is enough space on the giant BTC drive (probably /ua72) for all of a couple nights' worth of data. 10gig should be sufficient. If there isn't bug the CTIO folks to find out what of other peoples' stuff you can delete.

Make yourself a "parent" directory, such as:

/ua72/btc/deepsearch/

This assumes that /ua72 is the giant BTC 23 or 37 or whatever the hell it is gig drive. If it's not, then you will need to change /ua72 to something else all throughout these instructions, and this will be your last reminder of that.

IF, horrors, this directory already exists, you might want to do "rm -r *" in it to clean it out so that you are sure that you've got everything set up right when you are done with all this.

Then, make the following subdirectories in that parent directory:

• /ua72/btc/deepsearch/iraf/
• /ua72/btc/deepsearch/97dec27/
• /ua72/btc/deepsearch/97dec27_work/
• /ua72/btc/deepsearch/97dec27_work/pix/
• /ua72/btc/deepsearch/97dec27_clean/

Of course, put the correct evening date in place of "97dec27" for the latter three. Make a new set of those directories each night.

Get the hcompress software. This can be a challenge. It might actually already be present on ctioa7 in "~btc/bin". Give it a look. If not, then get the file "~rknop/hcompress.tar.Z" from panisse.lbl.gov, build the software, and copy the requisite executables into "~btc/bin".

Finally, go back to your home directory ("cd ~") and edit your .rhosts file. Make sure it has the line:

panisse.lbl.gov deep

This is mostly covered in the BTC documentation. The only note is that you should make sure that the raw FITS files obtained by the BTC go to the proper directory. If you don't do this, then nothing will work, we won't find any supernovae, we won't have anything to send to HST, and we will be the laughing stock of the astronomical community.

In the example above, the directory where you should tell the BTC to write its files is:

/ua72/btc/deepsearch/97dec27/

Substitue your current evening date for 97dec27.

Important Note: When taking data with the BTC, the scripts will be much happier if you observe the following guidelines:

• Always make sure you've set the title of the image right.
• Never reset the run number of an image to something lower. At the beginning of the night, you might want to bump it up to the next 100 boundary. (Up to you.)
• Always make sure the base name of your images is correct. Otherwise, junk we don't want sent will use up internet bandwidth, or crucial search frames will never make it to Berkeley. The suggested convention is to use "obj." as the basename for search fields, "std." for standard star fields, "twil." for twilight flats, "bias." for zero (bias) frames, and "dome." for dome flats.

Do this during the day at CTIO. Use the normal procedure. Take many, but not too many, and take an odd number of each. Between nine and fifteen should be a good number for each filter. Take dome flats in every filter you anticipate observing during the night.

To combine the flats, you may use your own IRAF savvy. However, I do not yet fully trust the FITS kernel in IRAF, so I would recommend the following procedure:

Run IRAF.

Do this from eiter the BTC home directory or from "/ua72/btc/deepsearch/iraf". Type "cl" to run iraf.

Type "mosred".

Type "cd /ua72/btc/deepsearch/97dec27_work/".

As with all of these instructions, substitue the correct date.

Set up the instrument.

Do "epar setinstrument". Set "site" to "ctio", and "instrument" to (probably) "btc". Ask the local BTC support folks if this is still correct. Type ":go" and hit RETURN when done. Scan the options of the next two parameter files you see, doing ":wq" with each one when you are satisifed either that you are happy with the options, or that you are clueless enough that you have no idea if you should be happy with the options.

Read your Zero Frames into IRAF

OK. Maybe you don't have to do this. However, there are two reasons to do it. One, I don't fully trust the FITS kernal in IRAF. Two, if you do, you don't risk accidentally processing your only copy of the raw file straight off of the telescope. (You definitely want to keep an original copy of that.) Use the procedure "rfits" to read FITS files out of "/ua72/btc/deepsearch/97dec27/" into IRAF files in "/ua72/btc/deepsearch/97dec27_work/".

run ccdlist

This is an IRAF command that will show you what you've got. It also creates the crucial "subsets" file. If you don't understand that, don't worry much about it, and run ccdlist a lot.

run zerocombine

In IRAF, type "epar zerocombine". Set "process" to "no". Give a wildcard or list file (with @) to specify all the zero (aka bias) frames you've just taken. Name the combined Zero file something like "Zero97dec27". When you are done setting up your parameters, hit the colon key and type "go". Hit return. Wait a long time.

Process the combined Zero file

In IRAF, type "epar ccdproc". Make sure you are editing the parameters for the mosred version of ccdproc. Set the following options to the following values:

Option	Value
fixpix	no
oversca	yes
trim	yes
zerocor	no
darkcor	no
flatcor	no
illumco	no
fringec	no
readcor	no
scancor	no

When you are done, use ":go" to run the ccdproc.

Clean up

Keep the combined "Zero97dec27*" files in the work directory. However, you can use "imdelete" to blow away the individual zero files in "97dec27_work/". NEVER delete anything out of the raw "97dec27/" directory.

Read your raw Flat frames into IRAF

See "Read your Zero Frames into IRAF" above, only now do it to your domeflat frames.

run ccdlist

Again issue the command "ccdlist" to IRAF. Make sure to enjoy it.

run flatcombine

In IRAF, type "epar flatcombine". Set "process" to "no". Specify all of your flatfields. Name the combined something like "Flat97feb9R". Make sure "subsets" is set to "yes". Type ":go" to do it. Wait.

Process the combined flatfield

Do "epar ccdproc". Set all of the options the same as what you did for zerocombine with the two following exceptions:

Option	Value
zerocor	yes
zero	Whatever is the name of your combined Zero.

When you are done, use ":go" to run the ccdproc.

Overview

It's less complicated than it used to be, but that doesn't mean it's not still complicated. There are three things going on:

makefitsloop

This used to convert raw BTC IRAF files into FITS files. Now they come out as FITS. All this program does is monitor the data coming off of the BTC, and when it thinks the data is there and stable, logs it in the file "/ua72/btc/deepsearch/97dec27/obtained" (where the current date is substituted for "97dec27"). It also assigns the internet transfer priorities on the fields based on the title which of course the person operating the BTC set right.

btcprocess

This watches the "obtained" file written by the previous process. It flatfields all of those files and sticks the result in the directory "/ua72/btc/deepsearch/97dec27_clean/". It writes the logfile "processed" in that clean directory to let the world know what it did.

ctiocompress.perl

This watches the "processed" log file, and when something is there, runs fcompress on it to produce a .H compressed file in the clean directory. It logs what it has done to "compressed". The latter is the logfile that the internet transfer routines running in Berkeley watch.

Set up makefitsloop

Open a shell on ctioa7.

Separate from the place where you plan to aquire or do "quick look" evaluation of the data. You may want to telnet from one of the other workstations in the control room to ctioa7.

Run IRAF

cd to "/ua72/btc/deepsearch/iraf/" and type "cl". Type the following commands to set up IRAF:

• mosred
• cd /ua72/btc/deepsearch/97dec27_work/

As always, substitute the current evening date for "97dec27".

Set your instrument

Run "setinstrument" as you did for combining zeros and flats.

epar makefitsloop

Set the following parameters to the following values:

Parameter	Value
datadir	/ua72/btc/deepsearch/97dec27/
fitsdir	/ua72/btc/deepsearch/97dec27/
match	A pattern of raw IRAF files of interest written by the BTC aquisition software. Probably "obj*fits".
priofile	/ua72/btc/deepsearch/iraf/prioriites
obtained	/ua72/btc/deepsarch/97dec27/obtained

Remember trailing slashes on all IRAF directory parameters!!!!! Otherwise Western Civilization will fall. Also, remember to substitute the current evening date for "97dec27". Type ":wq" when you are done editing the parameters for makefitsloop.

makefitsloop

Just type "makefitsloop" into the "mo>" prompt of IRAF, and all should be well.

In Case of Error

Panic. Then call Rob. If you have to restart the script, give the command "flpr" to IRAF a couple of times before you do so.

Set up btcprocess

This assumes that you made your Zeros and Flats as specified above, and that you left your combined Zero and Flat frames in your "97dec27_work" directory (where, as always, you substitue the current evening date for "97dec27").

Get yoursef a new shell

Do this just like you did it for makefitsloop above... but, obviously, it has to be a new shell! It must be running on ctioa7, although you may want to telnet in to ctioa7 from a different workstation for convenience.

Run IRAF

On ctioa7, issue these commands:

• % cd /ua72/btc/deepsearch/iraf/
• % cl
• cl> mosred
• mo> cd /ua72/btc/deepsearch/97nov27_work/

As always, substitute... well, you know.

Set your instrument

Do "epar setinstrument" much as you did it on the BTC computer during combining Zeros and Flats.

ccdlist

Make sure all your zero and flatfield frames come out correctly. In particular, make sure that ccdlist lists zeros as zeros and flats as flats, that it gets the filters right, and that it gets the chip number right. If it doesn't, Panic. Then consult the BTC support people and/or call Rob.

epar ccdproc

Set it up for your cleaning. Some of the key parameters which should be set are:

Paremeter	Setting
ccdtype	object
noproc	no
fixpix	no
oversca	yes
trim	yes
zerocor	yes
darkcor	no
flatcor	yes
illumco	no
fringec	no
readcor	no
scancor	no
baissec	image
trimsec	image
zero	The name of your combined zero (e.g. "Zero97dec27*.imh")
flat	A wildcarded expression which includes all your combined flats (e.g. "Flat97dec27*.imh")

Hopefully, everything else should already be set to some sane value. If they aren't, then even Panic won't help.

You may want to also "epar mosred" to make sure that you do not have a backup directory set. Don't use that feature, because the scripts already take care of that kind of thing.

Excellent, Dude!

Just thought I would insert that.

epar btcprocess

Set parameters as follows:

Paremeter	Setting
cookeddir	/ua72/btc/deepsearch/97dec27_clean/
workdir	/ua72/btc/deepsearch/97dec27_work/
processed	/ua72/btc/deepsearch/97dec27_clean/processed
writerawtape	no
remotehost	localhost
remoteuser	btc
remsrcdir	/ua72/btc/deepsearch/97dec27/
obtained	/ua72/btc/deepsearch/97dec27/obtained
desthost	localhost
remdstuser	btc
remdstdir	/ua72/btc/deepsearch/97dec27_clean
writecookedtape	no
firstjazdisk	1
dojazraw	no
dirjazraw	/dev/null/
dojazcooked	no
dirjazcooked	/dev/null/

Remember trailing slashes on all directories in IRAF paremeters!!

As always, substitute your current evening date for "97dec27".

btcprocess

(I.e. type "btcproces" at the "mo>" prompt in IRAF.) At this point, everything should work. When it doesn't, be sad. Go over all the steps above, and make sure everything is right. When it is, be sadder. Cry, even. Finally, Panic. Then call Rob.

Watch the Script!

In case it dies or does something wrong, you don't want to waste any time before being sad about it. If it dies, issue the command "flpr" to IRAF a couple of times before restarting the script. You may also want to make sure all your parmeters are set correctly.

Set up ctiocompress.perl

Open a shell on ctioa7

Use the tcsh. ksh is for wimps. Again, this must be running on ctioa7, but you may find it convenient to telnet from a different computer. Obviously, this shell is a third shell different from both of the last two shells you opened up.

Edit /ua72/btc/deepsearch/iraf/ctiocompress.perl

Change the first line to point to the proper location of perl on the Compression Machine. It may already be correct, especially if you followed all of the earlier instructions.

ctiocompress.perl requires some configuration. There are some variables at the top which should be set to:

Variable	Value
$cleandir	/ua72/btc/deepsearch/97dec27_clean
$bindir	The directory where the hcompress software is.
$tapedrive	/dev/null/
$maxtapesize	(doesn't matter, keep the default)
$writetape	0
$tapeOKstring	(doesn't matter, keep the default)

As always, substitude your current evening date for "97dec27".

Do

Issue the command "/ua72/btc/deepsearch/iraf/ctiocompress.perl". This should start the compression script going.

Watch

Give it a look. You may have to restart it every so often... hopefully not.

Vast quantities of log files are written by the scripts at each stage of the execution. This serves two purposes:

• So you the user can track what has been done.
• So the script can keep track of what's been done at the previous stage, what it has already done, and what remains to be done.

The log files all share a common format. Each line of the log file represents one file which has been processed by the script that write the log file. Each line looks like:

    filename   priority   title

filename

The name of the file in question, without the ".fits".

priority

A priority which lets the next stage of the whole process know which files are more important. Higher numbers mean higher priority. These priorites are assigned at the first step through some mysterious process that even the writer of the scripts does not understand. After that, each script just keeps the priority the file had in the previous script.

The default priority is 5.

title

This is wholly for human consumption. It's so you can hope to have a clue of what a file is when its filename is obtuse.

The specific log files

obtained

This is probably in /ua72/btc/deepsearch/97dec27/ (where you substitute your current evening date blah blah blah). These files have been noted and logged as available. "makefitsloop". "btcprocess" looks here to decide which files it needs to grab and clean.

processed

Probably in /ua72/btc/deepsearch/97dec27_clean/. These files have been cleaned by "btcprocess". "ctiocompress.perl" looks here to decide what to compress next.

compressed

Probably in /ua72/btc/deepsearch/97dec27_clean/> These files have been compressed by ctiocompress.perl. The LBL script "hotncftp" should look here to decide what to transfer up to Berkeley next.

It is probably worth starting all of the log files over with each night of observing. This will happen automatically if you put the date in all of your sundry directory names, and make new directories each night. (And, of course, make sure to set all of your IRAF parameters up correctly. Yes, all ten million of them.)

Editing Log Files

In practise, the only log file you will ever need to edit is "compressed" in the Clean FITS directory on the Compression Machine. This is what hotrcp up at Berkeley uses to decide its priorities, and hotncftp is the biggest bottleneck.

PROCEED WITH EXTREME CAUTION. Here's the scenario: you want to change a priority, so you read a log file into an editor. You futz with it a bit. Meanwhile, the script that writes that log file has added another line to it. You write out your edited file-- thereby removing the line that the script had just written to it. Suddenly your log file is incorrect. This is very bad.

Of course, if you are enlightened and you use a recent version of emacs to edit the log file, when you try to write it out it will detect if the file has been changed since you've last read it, and will prompt you rather than merrily overwriting the log file. This leads to the cardinal rule of editing log files: NEVER, EVER USE VI OR PICO TO EDIT LOG FILES. Use emacs. If you don't know emacs, then tough, you don't get to edit log files. And, when you do edit log files, you will save yourself frustration if you do it quickly, and write out what you've done.

Maybe in the future there will be a more user friendly way to do this, but for now this is how it's gotta be done.

There are two reasons why you might want to edit a log file:

Changing the priority of an image

You have to do this in the latest log file the image has been listed in, otherwise it won't make one whit of difference. In practice, you will probably only need to edit the "compressed" log file to change the priorities for hotrcp, since that will be the biggest bottleneck.

Getting a script to redo something.

This is scary. However, for example, if you remove a line from the "processed" log file, the btcprocess script on the Reduction Machine will grab it from the Aquisition Machine and process it again. You would also have to remove it from the "compressed" log file, and any log files up at Berkeley, to get it the rest of the way through the system.

In general, unless you've set something up horribly wrong (don't do that), and unless the scripts are all terribly broken (in which case it's all pretty hopeless anyway) you shouldn't ever have to do this with the CTIO stuff. If you think you do, Rob might be happier if you just called him.

The first thing you should do at the end of the night is wait until all of the "obj*.imh" files have made their way through the makefitsloop, btcprocess, and ctiocompress.perl scripts. These scripts really shouldn't fall that far behind, so this doesn't entail that much waiting.

Once these are all done, and all of them are doing nothing but sleeping over and over again waiting for more stuff to do, it is safe to kill each of them.

At this point, you may wish to write one or more DLT tape. This may be done from ctioa7. Note that the BTC documentation is not entirely correct about either the device name of the DLT drive or about whether the btc account has the proper default tape drive. Pay very close attention to everything you do.

Last time Greg and I were there, at the end of the first night we used "tar" in "/ua72/btc/deepsearch" to copy everything from the first night (about 4gigs worth) to DLT tape #1. The second night, we put that tape back in, did a "mt eom" to jump to the end of the tape, and then used tar to write _just_ the files from the second night (another 4-5 gig) as a second tar file on DLT tape #1. Then, we put in a fresh new DLT tape #2 and wrote the whole schmear (everything underneat /ua72/btc/deepsearch) to one gigantic tar file.

If memory serves (and it doesn't, really), it took .... well, hell, I can't remember, but I think it was something like 2 or 3 hours to write all 9-10 gigs of everything to tape #2.

Whatever you do, make sure you've got at least two copies of everything, especially the raw FITS files, before you leave. If you get back and it turns out you boffed up the tape writing, we will be so sad that it will be just a little bit extraordinary.