This is an overview of how CCD images from telescopes are used and stored for the Supernova Cosmology Project here in L.B.L.
Images taken at the telescope are send to us either over the computer network or on tapes by mail. During our "search runs", we prefer fast electronic transfer because of the urgency of finding SNe in real time. Because of the immence size of data (2-32 Mbytes/exposure, ~50 exposures/night), we usually compress the data at some loss. Since cleaned (flatfielded) images comppress better and we need to search for SNe as soon as possible, we flatfield and compress images at the telescope and ftp them over here. Because of software availability, we have to backround "surface" them here. These images are later marked as "***com.fts". When we get original images which did not suffer any loss due to comppress/decompress process, we tag these images "***raw.fts". They are flatfielded here and cleaned images are named "****csg.fts" (used to be "***ebs.fts" "***dbs.fts"). As soon as these csg images are available we stop using "com" images.
No matter where it is flatfielded, a cleaned image we use has to be :
Flatfielding here means the usual Bias-subtraction and QE frame division.
Surfacing means to subtract out the gradient in sky background level by a fitted surface. We surface our images mainly because our SNe and host galaxies are very "dingky" objects, while bright nearby stars can cause significant gradients in background level. For calibration field images, which are short exposure images of bright "standard" stars, we do not perform surface fitting and these are tagged as "***cln.fts". Surface fitted or not, background level is subtracted so that our cleaned images should have zero background level. (The original sky level is save in the header & database, of course)
Electronical signal gathered in the CCD basically have Poison noise (plus some other minor noises), e.i.,
Each telescope and each instrumentation can give arbitrary image orientation. We have to rotate and/or flip images to have uniform image orientation. Since we started this project using INT images, orientation then became the standard. When image(x's, y's) are displayed in the usual x-y plane, North is decreasing y, and East is decreasing x direction.
Images are renamed and registered to our image database. Our image naming convention (is using lowcase letters) :
Our imagedatabase holds an entry for each image (seperate ones for "raw" and "csg"). That entry has many vital informations about that image like, date-obs, UT, RA, Dec, exposure time, Filter, SKY level, sky noise, and more plus a copy of the entire FITS header, from which those informations are extracted.
Renamed and registered (or "loaded") images have to be in one of our image directorys, which are specified by UNIX environmental variable DEEPIMAGEPATH. (At UNIX promt, try : echo $DEEPIMAGEPATH) They are /home/astro#/deepsearch/ like /home/astro30/deepsearch/. In other words, it does not matter may597eso81csg.fts is at /home/astro30/deepsearch/ or /home/astro8/deepsearch/. However, if it is in any of the subdirectories from there, like /home/astro8/deepsearch/esodata/, it is NOT to be found by our software.
Obsolete images like "***com.fts" and raw images like "***raw.fts" are deleted from disks after being backed-up to tapes. All of ***.fts files in our image directories are backed-up semi-regularly. /home/astro9/deephome/imagesontape/imagesontape is the list of all the images copied to tapes.