Reference/Search Runs with the BTC
December 26, 1997
At any time if you have questions on any of this, particularly the laptop
configuration, contact Rob Knop at one (prefereable of the first two) of:
All of this is a little easier than it used to be because everything can
be done on the BTC UltraSparc named ctioa7. There is no longer a need to
bring the laptop to Chile.
Table of Contents:
Before You Leave Berkeley
- Make sure you know the fields.
- Bring finding charts and a list of fields and coordinates. Make sure
you understand where on the BTC footprint the listed coordinates
are. Our standard is that the list is the center of chip #1, the
- Make sure you have DLT tapes.
- One tape per night of observing should
be just sufficient, in that you can just fit two nights on one tape.
Three tapes per every two nights might be safer.
Setting up Directories
You should do or check all of these steps each night. First, make sure
there is enough space on the giant BTC drive (probably /ua72) for all of a
couple nights' worth of data. 10gig should be sufficient. If there isn't
bug the CTIO folks to find out what of other peoples' stuff you can
Make yourself a "parent" directory, such as:
This assumes that /ua72 is the giant BTC 23 or 37 or whatever the hell it
is gig drive. If it's not, then you will need to change /ua72 to something
else all throughout these instructions, and this will be your last reminder
IF, horrors, this directory already exists, you might want to do "rm -r
*" in it to clean it out so that you are sure that you've got everything set
up right when you are done with all this.
Then, make the following subdirectories in that parent directory:
Of course, put the correct evening date in place of "97dec27" for the
latter three. Make a new set of those directories each night.
Get the hcompress software. This can be a challenge.
It might actually already be present on ctioa7 in "~btc/bin". Give it a
look. If not, then get the file "~rknop/hcompress.tar.Z" from
panisse.lbl.gov, build the software, and copy the requisite executables into
Finally, go back to your home directory ("cd ~") and edit your
.rhosts file. Make sure it has the line:
Setting Up the BTC Software
This is mostly covered in the BTC documentation. The only note is that
you should make sure that the raw FITS files obtained by the BTC go to the
proper directory. If you don't do this, then nothing will work, we won't
find any supernovae, we won't have anything to send to HST, and we will be
the laughing stock of the astronomical community.
In the example above, the directory where you should tell the BTC to
write its files is:
Substitue your current evening date for 97dec27.
Important Note: When taking data with the BTC, the scripts will be
much happier if you observe the following guidelines:
- Always make sure you've set the title of the image right.
- Never reset the run number of an image to something lower. At the
beginning of the night, you might want to bump it up to the next 100
boundary. (Up to you.)
- Always make sure the base name of your images is correct. Otherwise,
junk we don't want sent will use up internet bandwidth, or crucial
search frames will never make it to Berkeley. The suggested
convention is to use "obj." as the basename for search fields, "std."
for standard star fields, "twil." for twilight flats, "bias." for
zero (bias) frames, and "dome." for dome flats.
Set Up IRAF for the Automated Scripts
IRAF will already be set up on the BTC for general use. You need to do
an additional setup of IRAF for the automated scripts. To do this, issue
the following two Unix commands:
- % cd /ua72/btc/deepsearch/iraf
- % mkiraf
That will create a login.cl file in the directory
/ua72/btc/deepsearch/iraf. Next, copy all of the following files (which
may be found on panisse.lbl.gov in ~rknop/iraf) to this IRAF directory:
Issue the command "which perl". If you get anything other than
"/usr/local/combin/perl", then you need to edit the first line of the
following scripts to point to the correct location of perl:
Issue the following command to make all the perl scripts executable:
chmod 755 listmefits postbtcprocess makefitsfilter
Finally, edit your login.cl file. Find the block where there are about
ten or so lines of "task" statements in a row. Below that, add the
- task btcprocess = /ua72/btc/deepsearch/iraf/btcprocess.cl
- task makefitsloop = /ua72/btc/deepsearch/iraf/makefitsloop.cl
- task $listmefits = $/ua72/btc/deepsearch/iraf/listmefits
- task $postbtcprocess = $/ua72/btc/deepsearch/iraf/postbtcprocess
- task $makefitsfilter = $/ua72/btc/deepsearch/iraf/makefitsfilter
- task $getfitsname = $ua72/btc/deepsearch/iraf/getfitsname
Note the dollar signs! Put one in the wrong place, or leave one
out, and Western Civilization will end.
Closer to the top of the login.cl file, locate the "set imdir" command.
Change the definition there to "hdr$pix/" (or maybe "HDR$pix/", I'm not sure
which one actually works.)
Taking Zeros and Flats
Do this during the day at CTIO. Use the normal procedure. Take many,
but not too many, and take an odd number of each. Between nine and fifteen
should be a good number for each filter. Take dome flats in every
filter you anticipate observing during the night.
To combine the flats, you may use your own IRAF savvy. However, I do not
yet fully trust the FITS kernel in IRAF, so I would recommend the following
- Run IRAF.
- Do this from eiter the BTC home directory or from
"/ua72/btc/deepsearch/iraf". Type "cl" to run iraf.
- Type "mosred".
- Type "cd /ua72/btc/deepsearch/97dec27_work/".
- As with all of these instructions, substitue the correct date.
- Set up the instrument.
- Do "epar setinstrument". Set "site" to "ctio", and "instrument" to
(probably) "btc". Ask the local BTC support folks if this is still
correct. Type ":go" and hit RETURN when done. Scan the options of
the next two
parameter files you see, doing ":wq" with each one when you are
satisifed either that you are happy with the options, or that you are
clueless enough that you have no idea if you should be happy with the
- Read your Zero Frames into IRAF
- OK. Maybe you don't have to do this. However, there are two reasons
to do it. One, I don't fully trust the FITS kernal in IRAF. Two, if
you do, you don't risk accidentally processing your only copy of the
raw file straight off of the telescope. (You definitely want to keep
an original copy of that.) Use the procedure "rfits" to read FITS
files out of "/ua72/btc/deepsearch/97dec27/" into IRAF files in
- run ccdlist
- This is an IRAF command that will show you what you've got. It also
creates the crucial "subsets" file. If you don't understand that,
don't worry much about it, and run ccdlist a lot.
- run zerocombine
- In IRAF, type "epar zerocombine". Set "process" to "no". Give a
wildcard or list file (with @) to
specify all the zero (aka bias) frames you've just taken. Name the
combined Zero file something like "Zero97dec27". When you are done
setting up your parameters, hit the colon key and type "go". Hit
return. Wait a long time.
- Process the combined Zero file
- In IRAF, type "epar ccdproc". Make sure you are editing the
parameters for the mosred version of ccdproc. Set the following
options to the following values:
When you are done, use ":go" to run the ccdproc.
- Clean up
- Keep the combined "Zero97dec27*" files in the work directory.
However, you can use "imdelete" to blow away the individual zero
files in "97dec27_work/". NEVER delete anything out of the
raw "97dec27/" directory.
- Read your raw Flat frames into IRAF
- See "Read your Zero Frames into IRAF" above, only now do it to your
- run ccdlist
- Again issue the command "ccdlist" to IRAF. Make sure to enjoy it.
- run flatcombine
- In IRAF, type "epar flatcombine". Set "process" to "no". Specify all
of your flatfields. Name the combined something like "Flat97feb9R".
Make sure "subsets" is set to "yes". Type ":go" to do it. Wait.
- Process the combined flatfield
- Do "epar ccdproc". Set all of the options the same as what you did
for zerocombine with the two following exceptions:
When you are done, use ":go" to run the ccdproc.
|zero||Whatever is the name of your combined
Setting Up the Reduction Scripts
It's less complicated than it used to be, but that doesn't mean it's not
still complicated. There are three things going on:
- This used to convert raw BTC IRAF files into FITS files. Now they
come out as FITS. All this program does is monitor the data coming
off of the BTC, and when it thinks the data is there and stable, logs
it in the file "/ua72/btc/deepsearch/97dec27/obtained" (where the
current date is substituted for "97dec27"). It also assigns the
internet transfer priorities on the fields based on the title which
of course the person operating the BTC set right.
- This watches the "obtained" file written by the previous process. It
flatfields all of those files and sticks the result in the directory
"/ua72/btc/deepsearch/97dec27_clean/". It writes the logfile
"processed" in that clean directory to let the world know what it
- This watches the "processed" log file, and when something is there,
runs fcompress on it to produce a .H compressed file in the clean
directory. It logs what
it has done to "compressed". The latter is the logfile that the
internet transfer routines running in Berkeley watch.
Set up makefitsloop
- Open a shell on ctioa7.
- Separate from the place where you plan to aquire or do "quick look"
evaluation of the data. You may want to telnet from one of the other
workstations in the control room to ctioa7.
- Run IRAF
- cd to "/ua72/btc/deepsearch/iraf/" and type "cl". Type the following
commands to set up
As always, substitute the current evening date for "97dec27".
- cd /ua72/btc/deepsearch/97dec27_work/
- Set your instrument
- Run "setinstrument" as you did for combining
zeros and flats.
- epar makefitsloop
- Set the following parameters to the following values:
Remember trailing slashes on all IRAF directory
parameters!!!!! Otherwise Western Civilization will fall.
Also, remember to substitute the current evening date for "97dec27".
Type ":wq" when you are done editing the parameters for makefitsloop.
|match||A pattern of raw IRAF files of interest written
by the BTC aquisition software. Probably
- Just type "makefitsloop" into the "mo>" prompt of IRAF, and all
should be well.
- In Case of Error
- Panic. Then call Rob. If you have to restart the script, give the
command "flpr" to IRAF a couple of times before you do so.
Set up btcprocess
This assumes that you made your Zeros and Flats as specified above, and
that you left your combined Zero and Flat frames in your "97dec27_work"
directory (where, as always, you substitue the current evening date for
- Get yoursef a new shell
- Do this just like you did it for makefitsloop above... but, obviously,
it has to be a new shell! It must be running on ctioa7, although you
may want to telnet in to ctioa7 from a different workstation for
- Run IRAF
- On ctioa7, issue these commands:
As always, substitute... well, you know.
- % cd /ua72/btc/deepsearch/iraf/
- % cl
- cl> mosred
- mo> cd /ua72/btc/deepsearch/97nov27_work/
- Set your instrument
- Do "epar setinstrument" much as you did it on the BTC computer during
combining Zeros and Flats.
- Make sure all your zero and flatfield frames come out correctly. In
particular, make sure that ccdlist lists zeros as zeros and flats as
flats, that it gets the filters right, and that it gets the chip
number right. If it doesn't, Panic. Then consult the BTC support
people and/or call Rob.
- epar ccdproc
- Set it up for your cleaning. Some of the key parameters which should
be set are:
Hopefully, everything else should already be set to some sane value.
If they aren't, then even Panic won't help.
|zero||The name of your combined zero
|flat||A wildcarded expression which includes all your
combined flats (e.g. "Flat97dec27*.imh")|
You may want to also "epar mosred" to make sure that you do
not have a backup directory set. Don't use that
feature, because the scripts already take care of that kind of
- Excellent, Dude!
- Just thought I would insert that.
- epar btcprocess
- Set parameters as follows:
Remember trailing slashes on all directories in IRAF
As always, substitute your current evening date for "97dec27".
- (I.e. type "btcproces" at the "mo>" prompt in IRAF.)
At this point, everything should work. When it doesn't, be sad. Go
over all the steps above, and make sure everything is right. When it
is, be sadder. Cry, even. Finally, Panic. Then call Rob.
- Watch the Script!
- In case it dies or does something wrong, you don't want to waste any
time before being sad about it. If it dies, issue the command "flpr"
to IRAF a couple of times before restarting the script. You may also
want to make sure all your parmeters are set correctly.
Set up ctiocompress.perl
- Open a shell on ctioa7
- Use the tcsh. ksh is for wimps. Again, this must be running on
ctioa7, but you may find it convenient to telnet from a different
computer. Obviously, this shell is a third shell different from both
of the last two shells you opened up.
- Edit /ua72/btc/deepsearch/iraf/ctiocompress.perl
- Change the first line to point to the proper location
of perl on the Compression Machine. It may already be correct,
especially if you followed all of the earlier instructions.
ctiocompress.perl requires some configuration. There are some
variables at the top which should be set to:
|$bindir||The directory where the hcompress software
keep the default)|
keep the default)|
As always, substitude your current evening date for "97dec27".
- Issue the command "/ua72/btc/deepsearch/iraf/ctiocompress.perl". This
should start the compression script going.
- Give it a look. You may have to restart it every so
often... hopefully not.
All Those Damned Log Files
Vast quantities of log files are written by the scripts at each stage of
the execution. This serves two purposes:
- So you the user can track what has been done.
- So the script can keep track of what's been done at the previous
stage, what it has already done, and what remains to be done.
The log files all share a common format. Each line of the log file
represents one file which has been processed by the script that write the
log file. Each line looks like:
filename priority title
- The name of the file in question, without the ".fits".
- A priority which lets the next stage of the whole process know which
files are more important. Higher numbers mean higher priority.
These priorites are assigned at the first step through some
mysterious process that even the writer of the scripts does not
understand. After that, each script just keeps the priority the file
had in the previous script.
The default priority is 5.
- This is wholly for human consumption. It's so you can hope to have a
clue of what a file is when its filename is obtuse.
The specific log files
- This is probably in /ua72/btc/deepsearch/97dec27/ (where you
substitute your current evening date blah blah blah).
These files have been noted and logged as available.
"makefitsloop". "btcprocess" looks here to decide which
files it needs to grab and clean.
- Probably in /ua72/btc/deepsearch/97dec27_clean/.
These files have been cleaned by "btcprocess". "ctiocompress.perl"
looks here to decide what to compress next.
- Probably in /ua72/btc/deepsearch/97dec27_clean/>
These files have been compressed by ctiocompress.perl. The LBL
script "hotncftp" should look here to decide what to transfer up to
It is probably worth starting all of the log files over with each night
of observing. This will happen automatically if you put the date in all of
your sundry directory names, and make new directories each night. (And, of
course, make sure to set all of your IRAF parameters up correctly. Yes, all
ten million of them.)
Editing Log Files
In practise, the only log file you will ever need to edit is "compressed"
in the Clean FITS directory on the Compression Machine. This is
what hotrcp up at Berkeley uses to decide its priorities, and hotncftp is the
PROCEED WITH EXTREME CAUTION. Here's the scenario: you
want to change a priority, so you read a log file into an editor. You futz
with it a bit. Meanwhile, the script that writes that log file has added
another line to it. You write out your edited file-- thereby removing the
line that the script had just written to it. Suddenly your log file is
incorrect. This is very bad.
Of course, if you are enlightened and you use a recent version of emacs
to edit the log file, when you try to write it out it will detect if the
file has been changed since you've last read it, and will prompt you rather
than merrily overwriting the log file. This leads to the cardinal rule of
editing log files: NEVER, EVER USE VI OR PICO TO EDIT LOG
FILES. Use emacs. If you don't know emacs, then tough, you don't
get to edit log files. And, when you do edit log files, you will save
yourself frustration if you do it quickly, and write out what you've
Maybe in the future there will be a more user friendly way to do this,
but for now this is how it's gotta be done.
There are two reasons why you might want to edit a log file:
- Changing the priority of an image
- You have to do this in the latest log file the image has been
listed in, otherwise it won't make one whit of difference. In
practice, you will probably only need to edit the "compressed" log
file to change the priorities for hotrcp, since that will be the
- Getting a script to redo something.
- This is scary. However, for example, if you remove a line from the
"processed" log file, the btcprocess script on the Reduction Machine
will grab it from the Aquisition Machine and process it again. You
would also have to remove it from the "compressed" log file, and any
log files up at Berkeley, to get it the rest of the way through the
In general, unless you've set something up horribly wrong (don't
do that), and unless the scripts are all terribly broken (in
which case it's all pretty hopeless anyway)
you shouldn't ever have to do this with the CTIO
stuff. If you think you do, Rob might be happier if you just called
Backing Up and Other End-of-Night Activites
The first thing you should do at the end of the night is wait until all
of the "obj*.imh" files have made their way through the makefitsloop,
btcprocess, and ctiocompress.perl scripts. These scripts really shouldn't
fall that far behind, so this doesn't entail that much waiting.
Once these are all done, and all of them are doing nothing but sleeping
over and over again waiting for more stuff to do, it is safe to kill each of
At this point, you may wish to write one or more DLT tape. This may be
done from ctioa7. Note that the BTC documentation is not entirely correct
about either the device name of the DLT drive or about whether the btc
account has the proper default tape drive. Pay very close attention to
everything you do.
Last time Greg and I were there, at the end of the first night we used
"tar" in "/ua72/btc/deepsearch" to copy everything from the first night
(about 4gigs worth) to DLT tape #1. The second night, we put that tape back
in, did a "mt eom" to jump to the end of the tape, and then used tar to
write _just_ the files from the second night (another 4-5 gig) as a second
tar file on DLT tape #1. Then, we put in a fresh new DLT tape #2 and wrote
the whole schmear (everything underneat /ua72/btc/deepsearch) to one
gigantic tar file.
If memory serves (and it doesn't, really), it took .... well, hell, I
can't remember, but I think it was something like 2 or 3 hours to write all
9-10 gigs of everything to tape #2.
Whatever you do, make sure you've got at least two copies of everything,
especially the raw FITS files, before you leave. If you get back and it
turns out you boffed up the tape writing, we will be so sad that it will be
just a little bit extraordinary.
Last modified: Fri Dec 26 11:13:49 1997