LPipe - Detailed usage manual

Table of Contents

1. Goals and Scope
2. General Workflow
3. Display, Validation, and Logging
4. Descriptons of Processing Steps
5. Additional Tools
6. Examples
7. Installation
8. Troubleshooting

Goals and Scope

LPipe is a package for fast and entirely automated reduction of longslit and imaging data acquired with LRIS. It is designed to function in a wide range of circumstances: for bright and faint objects and using all available gratings, grisms, and dichroics. (It does not yet support polarimetry or multi-slit modes.) It supports all four versions of the red CCD. It prioritizes robust reductions over obtaining theoretically optimized S/N, and is able to identify and ignore (or take steps to mitigate) most forms of bad data resulting from instrument problems or observer error.

It is well-suited for quicklook reductions at the telescope, for time-critical observations (e.g. TOO's), and for exploration of archival data. It is less well-suited for specific projects where e.g. extremely accurate calibration is necessary. For spectra, the typical relative flux calibration accuracy is about 5-10 percent and the typical wavelength calibration accuracy is about 1 pixel. Imaging astrometry is good to 0.5 arcsec across most of the field (but worse at the edges, and not suitable for mosaicing); flat-fielding accuracy is completely depending on the data-gathering procedure. In rare cases, during periods of instrument/telescope problems, or on nights when poor observing/calibration procedures are used, the performance may be worse. The imaging pipeline was developed earlier than the spectroscopic pipeline and has comparatively effort expended to modernize it or make it flexible/extensible.

In default mode the pipeline offers one-line reductions, and it is anticipated that this is the mode that will be employed by most users. In other words, typical usage requires nothing more sophisticated than this command to reduce an entire night of data:

IDL> lpipe

However, the pipeline also permits finer control by specifying additional options at the command lines, and via a GUI interface. Additionally, a wide variety of quality-assurance (QA) check-plots are produced which may be helpful in diagnosing pipeline issues. Some information about these options (and general information on pipeline operations to help demystify the procedures) is provided below. For more information, users can also consult the in-code documentation or the associated publication.

Note that you can always type:

IDL> lpipe, /help

For some basic information (including a list and descriptions of all processing steps and some additional tools.)

General Workflow

The pipeline employs a series of self-contained steps in series. Some steps are shared between imaging and spectroscopy (although most have separate implementations internally). The steps, in the order they are normally carried out in pipeline processing, are:

Deprecated steps that will be restored in future pipeline versions are in italics.

Within each step, blue imaging, then blue spectroscopy, then red imaging, then red spectroscopy, are processed. Within that sequence the order usually follows that of the associated filenames, even if the configuration changed back and forth during a night.

The steps are designed to be entirely self-contained: almost all information is stored in file headers and re-read with each new step. If the pipeline is interrupted it can thus continue at a later date without having to repeat any earlier steps. By default, if the pipeline is restarted it will retry any failed operations but will not redo any operation that would overwrite a file that already exists. This behavior can be altered using the options below.

Workflow control command-line options:

Step control:

• mode - Set this to 'i' to process only imaging or 's' to process only spectroscopy (or join both together to specify what order to process both, e.g.: 'si' or 'is'.) Steps and files not associated with the active mode will not be processed.
• step - Specify an individual single step to carry out. This must be one of the recognized steps in the above table, written exactly as it is above, e.g. 'fluxcal'.
• start - Specify which step to begin with, in case the user wants to carry out multiple (but not all) steps. Helpful for restarting after processing was stopped or for redoing observations of a particular object after changing options (see redo, below).
• stop - Specify which step to end with. Helpful when the user wants to pause after a particular step to inspect the outputs before continuing.
• /redo - Tell the pipeline to overwrite existing files, and thus redo all operations for the steps. Generally used if the first run causes results that are judged to be suboptimal and something is changed, e.g. a bad standard is deleted or an aperture is moved. Often used in concert with the file-filter options below.

Higher-level control:

• /continuous - When the pipeline finishes, start over again from the beginning. For hands-off reduction of an observing run: set up an rsync script and run in this mode to automatically assimilate new data as it appears. The pipeline will not stop until a Ctrl+C command is issued.
• directories - Batch-mode operation for reducing many different nights: this tells the pipeline to run once in each of the subdirectories specified in this option. These different directories do not share any calibration files: if this is desired the files should be placed in a common single directory (or through careful use of altflatdir and altarcdir).

File-based filtering:

• camera - Specify 'red' or 'blue' (or 'r' or 'b') to process only one of LRIS's two cameras. (Or specify 'rb' or 'br' to do both cameras in a specific order.)
• chip - For steps beginning with split, specify 'r' or 'l' to process only one chip or 'rl', 'lr', or 'both' to process both chips. Pre-split steps always process both chips. Note: The spectroscopic pipeline generally assumes all spectra during a run were taken on one side or another (this is almost always the right side, but recent problems with the LRIS-red amplifiers have recently required centering single-objects on the left chip.). The use of the pipeline to reduce observations that use both chips during the same night has not been tested.
• files - Specify an individual file number or range of file numbers to process: this excludes the run date and simply refers the four-digit (original format) or five-digit (KOA format) file specifier. e.g., 56 or '04121'
• filters - Specify a specific filter (or filters, if given as an array) to process: e.g. 'R' or ['G','V'].
• gratings - Specify a specific grating/grism (or gratings/grisms if an array) to process.
• targets - Specify a specific target (e.g. TARGNAME) to process. The TARGNAME in a file header comes from the starlist uploaded by the observer at the start of the night (and is distinct from the OBJECT name, which the pipeline does not use). Note that this is the full target name, not the abbreviated form that appears in filenames.
• date - Process only files from a specific UT date (in cases where multiple nights of a run are being reduced together.) e.g., 150101 or 20150101.

Display, Validation, and Logging

Almost every message printed to the screen during pipeline operations is also saved to a permanent logfile (lpipelog.txt). If the pipeline is re-run the logfile is appended from the bottom; it is cleared only if deleted by the user.

Additionally, most of the 1D spectral reduction steps will produce check plots for user quality assurance. These may be flashed up on the screen, written to disk as a postscript file, or both. These are generated by specific pipeline processing steps as they run (see the individual step sections for more details).

The most powerful validation option by far, though, is the lrisvalidate tool. This will allow you to visually Step through final spectra and some of their associated calibration files, such as sky line plots (to confirm wavelength calibration), response plots (to confirm flux calibration), and 2D spectra (to confirm tracing, object selection/extraction, and cosmic ray rejection).

Display/logging command-line options:

• /nodisplay - Don't pop up any display windows to the terminal (postscript plots are still generated). Useful for remove/server reductions for which X-windows forwarding either doesn't work or slows down the reductions too much.
• /nofailuresum- Normally, any time a file cannot be processed at any time, a warning message is displayed at the time of processing and also at the end of the pipeline run. Set this tohide the summary of unprocessed files printed at the end.
• /debug - Prints out some more information during some steps, and also turns off most exception handling. This means that when a task within a step fails, the program simply crashes at that line and does not attempt to recover.
• /timer - Log and print how long it takes to run each step.
• /configure - This writes a configuration file to disk that allows the user to change some additional aspects of the pipeline, like the default data subdirectories.

Summary of All Steps

prepare

This step does several things together: the overscan (bias) is subtracted, the amplifiers and left/right chips are joined together into a simple extension-free FITS file, and the array is transposed so that (for spectra) the wavelength dispersion axis is horizontal and the spatial axis is vertical, and a large amount of header information is added.

Bias subtraction and amplifier combination uses a modified version of readmhdu.fits (written by Marc Kassis), with the /linebias option set. There is no gain correction applied (here or at any other stage in the pipeline). Spectra or images taken in full-frame mode are subsequently cropped down to the standard cropping regions and transposed. Additionally, a bad pixel map is employed to flag pixels and columns that are known to be problematic (they are set to NaN, which all subsequent steps interpret as missing data.)

The amount of header information added is substantial and includes solar/lunar ephemerides, unique identifiers corresponding to the first file in the same configuration sequence, and many other details. Particularly notably, the pipeline adds information about binning cropping via the LTM and LTV keywords, which allow the 'physical' coordinates as seen in DS9 to always self-consistently map to the same detector pixels regardless of binning or cropping. For existing header fields, interpretive comments are added. The header is also 'corrected' for missing information: frequently LRIS fails to write some critical keywords to the header, which the pipeline recovers by copying those keywords from the matching file in the opposite camera.

Associated options:

• data - Specifies where to look for the raw data. (If left blank the pipeline will look in the current working directory.)
• /prepareall - Normally the pipeline skips all processing for multi-slit and polarimetric images and for bias frames. Set this flag to run the prepare step on these, too. This may be useful if you want to write your own code to processes these frames but still want to take advantage of the added header metadata, or to inspect overscan subtraction on bias frames.
• /skipfocus - When running on non-KOA data, focus frames have a different filename convention (e.g., bfoc0123.fits) and are always ignored unless they are explicitly renamed. For KOA-formatted data, focus frames share the same filename convention as others and *will* be processed (at least in the case of spectra - the imaging focus procedure uses a special slitmask and is still ignored for that reason). These are used by default, but if you want them to be excluded, set the /skipfocus option.
• reducer - Add your name to the file headers (REDUCER keyword).

Output:

• p[b|r]*.fits - Prepared images.

Diagnostic output:

None.

makeflat

Produces flat-fields by identifying and combining images of a uniform source. For imaging this is either the dome screen or the twilight sky; for spectroscopy it can also be an internal flat such as a halogen or deuterium flat. A median combination is used in all cases but additional filtering is done as necessary: for twilight-flat imaging bright sources are excluded and removed, and for spectroscopy a variety of methods are used to filter out spectral lines or remove spatial banding/gradients associated with the lamps. These methods are not perfect and leave some residuals.

Associated options:

• /forceflat - Generally only 'necessary' flats are obtained, and if multiple types of flat were acquired the pipeline will only produce what is likely to be the best for the data. Set this option to produce all types of flat-fields for which enough individual flat-field images were acquired.
• flatrequire - Change/specify the minimum number of individual exposures necessary to construct a flatfield. Set this if you didn't take many flats and want allow processing to occur even with a lower-quality flat-field.
• /rightchipdead - Tell the pipeline that the right chip, which is usually used to normalize the spectroscopic flat field, is dead/malfunctioning and should not be used for this purpose. (The effects of a dead chip on imaging have not yet been tested.)

Output:

• l[b|r]*_[flattype].fits - Processed and combined flat fields.

Diagnostic output:

None.

flatten

Uses processed flat-fields to correct the data. Ideally the flat-fields are in the exact configuration as the data, so this is straightforward, but there can be a variety of complications. For example the wavelength solution for spectra might be slightly different, a dichroic might be different, an order-blocking filter might be present or absent, etc.

Even when the configuration is identical the flat-fielded image is unlikely to exactly match the data because of flexure and because components, once moved, may not always return to their original configuration.

• flatrequire - Specify the minimum number of flats needed to construct a spectroscopic flat-field. The default is a (very minimal) 3.
• /norebin - It is very frequent that observers accidentally acquire flats in the 'wrong' binning mode compared to what they actually employ later in the night for a certain configuration. The pipeline will rebin flats from one binning mode to another when this happens. You can turn off this behavior with this option, in which case those data will simply not be processed.
• altflatdir - Give an alternative flat-field directory or list of directories, such as a directory with processed flats from a previous run.
• /ignoredichroic - Allow a flat with one dichroic to be flat-field using flats constructed using another dichroic. (For imaging only.)

Output:

• fp[b|r]*[l|r].fits - Flattened images.

Diagnostic output:

None.

reflatten

Currently this step is for imaging only. It has been observed that when LRIS is switched in and out of imaging mode the dust spots and overall vignetting pattern of the filter itself remain at a constant location move by many pixels. However, this pattern remains fixed within a block of imaging observations even when the filter is changed. Thus the pipeline constructs super-sky flats, based on all data in each filter during a given imaging block. It then, as part of the same step, corrects the data using these flats. (This is performed only if there are a large number of frames and different fields for a particular filter. Otherwise, no super-sky flattening is performed.)

Associated options:

• /noreflat - Skip this step. On by default (the user must specify noreflat=0 to activate reflattening.)

Output:

• ifp[b|r]*[l|r].fits - Re-flattened images.

Diagnostic output:

None.

split

This is a straightforward step that breaks apart the joined frame into right (and left, if specified in the chip option) amplifiers. Some additional cropping is also applied.

Associated options:

• /distort - Some distortion keywords are written to the header of images. These are SIP keywords and while the pipeline tries to convert them to the convention recognized by sextractor and swarp as used in astrometric matching, it is not clear that this works correctly, so this shouldn't be relied upon.

Output:

• fp[b|r]*[l|r].fits - Split images / 2D spectra.

Diagnostic output:

None.

crclean

Identifies and removes cosmic rays from the data. Given the challenging nature of cosmic rays in deep-depletion data a custom algorithm is used. Similar algorithms are used for imaging and spectroscopy although they are coded separately.

Associated options:

• /nocrclean - Skip this step.
• crzeal - Change the aggressiveness of cleaning from its default value of 1. If you are more or less concerned about false positives (lines being zapped) versus false negatives (cosmic rays being missed and generating false lines) you can change this up or down. This is a multiplicative factor so it must be positive; factors of 0.5 or 2.0 are large changes.

Output:

• zfp[b|r]*[l|r].fits - Cosmic-ray-cleaned images or 2D spectra.
• zfp[b|r]*[l|r].mask.fits - Cosmic ray bad pixel mask.

Diagnostic output:

None.

skysubtract

Subtract an estimate of the night-sky emission flux from the 2D spectrum and place it in an extension. It is important to note that this procedure is not used directly in final spectrum construction because the sky is 'readded' at the type of extraction, but temporary removal of the confusing sky lines is needed for intermediate steps including (in particular) source identification and tracing of faint objects.

Associated options:

None.

Output:

• szfp[b|r]*[l|r].fits - Sky-subtracted 2D spectra. The first extension contains the subtracted spectrum, the second extension contains the sky lines.

Diagnostic output:

None.

sum

Coadd the individual 2D spectra of a common source (at a common slit orientation) to produce a single 2D spectrum. If the telescope was dithered along the slit the dithered images (i.e. those after the first) are shifted before stacking using the header positional keywords. (If the shift was shifted laterally these are not included and are instead used for a separate stack.)

Associated options:

• /nosum - Treat every individual exposure as an individual observation and do not perform any coadding. (Can be used in conjunction with the target option to make this specific to a specific object/objects)

Output:

• ls[b|r]*.fits - Combined spectra. The first two extensions are as above; a third extension contains the summed cosmic ray mask.

Diagnostic output:

None.

trace

A multi-stage step that does several, related tasks. First, each (summed) 2D spectrum is loaded. If a moderately bright object is located anywhere on the trace, a section of the 2D spectrum around that object is then used to calculate a generic tracing function for use in all extractions involving that file. This is saved as a '.trace' file, which contains the polynomial fit terms. The polynomial order is generally low (3rd order by default and even less if the trace is lost for a significant fraction of the spectrum), and as a result small inaccuracies in the tracing - of order 1-2 pixels - are not uncommon, but large tracing errors are now very rare. (If there is no bright object on the trace, the pipeline instead selects another object observed in the same configuration and close in time and uses its trace function as a substitute.)

Next, a median-filtered sum along the trace (i.e, over all wavelengths) is used to produce a 1D spatial profile along the slit. This is saved as a file ending in '.profile' and is used in source detection.

Next, two object detection procedures are run on the profiles described previously. The first run is only for profiles that contain very bright, single objects (generally, standard stars): it is used to determine where the primary target tends to fall on the detector. The second run is for all science observations: all profile peaks are measured and the algorithm selects the source that is closest to the position where the bright objects tend to be located. (The brightness of the source is used as a secondary criterion if there are several sources close to the nominal position.) All object positional data (for all files) is saved in a single text file, objectpos.txt.

Finally (after both cameras have been run), the red and blue trace center positions are compared for all spectra that overlap in time to see if the same source is being extracted on both sides and if the aperture diameters are the same (or at least similar). If not, one aperture is shifted to avoid the production of 'chimeric' spectra in later stages.

Associated options:

• /norbalign - Skip the red/blue alignment correction at the end of this step.

Output:

• ls[b|r]*.trace - Trace polynomials
• ls[b|r]*.profile - Spatial profiles
• objectpos.dat - (Default) object position database

Diagnostic output:

• ls[b|r]yymmdd_nnnn_object.trace.eps - A two-part plot, generated for every trace spectrum. The top plots the spatial profile (counts versus y) at various locations along the slit, before and after correcting for the trace function. The bottom part shows a median-averaged image of the 2D spectrum (y versus x; intensity indicates counts) with trace centers and fit overplotted.
• Red-blue alignment plots are plotted to the screen (not yet to postscript.)

extract

Following tracing the objects are extracted to produce 1D spectra. This uses the trace and positional data generated above and any user modifications to it (see below). The extraction is done using a custom procedure, which uses a basic "top hat" extraction with two parallel background bands on either side of the object to measure (and subtract) the sky background. (Note: the sky is 're-added' to the sky-subtracted spectrum because the sky subtraction procedure can sometimes remove some source flux.)

Note that this is not an optimal extraction. Indeed, to guard against tracing errors the default aperture radius is larger than is optimal even for a simple extraction.

Associated options:

• /noreaddsky - Do not readd the sky as above, but instead rely on the 2D sky-subtracted spectrum.

Additionally, users can exert fine control on the extraction using the GUI system lrisapertures. This is a separate routine that must be run after the trace step is complete, and allows the user to change the positions and widths of the object extraction apertures. If any object apertures are changed those targets will need to be re-reduced starting with this step. The lrisapertures routine will be separately documented.

Output:

• xls[b|r]*.spec

Diagnostic output:

• lristraces.html - An HTML file including the 2D spectra of all extracted sources with the traces indicated. Also plotted to the screen. (Associated with a group of .png files).
• profiles.ps - A postscript plot of all spatial profiles, with traces indicated. Combines and matches red and blue profiles.
• profiles[b|r].ps - A postscript plot of all red/blue spatial profiles, with traces indicated.
• xls[b|r]*.reg - DS9 region files giving the extraction centers and boundaries; can be loaded into the corresponding fits files to interactively check extraction.

Note that the above plots are generated after extraction, not after tracing. This ensures that they match what is actually extracted (as opposed to what is going to be extracted next time the pipeline is run.

wavcal

Wavelength-calibrate arc spectra. This does its own basic median extraction to produce a 1D spectrum, then runs a line detection routine on the result and uses a custom pattern-matching routine to match the resulting line list against a reference line list and use it to determine a wavelength solution.

The Cd and Zn arcs take about 5 minutes to fully warm up and produce lines with the expected strengths and line ratios, which is about 4 minutes longer than the available patience of a typical observer. Unwarmed arcs are missing expected lines and often confuse the pattern matcher, leading to a bad solution. Two steps are taken to mitigate this. If multiple identical-configuration arcs were taken in sequence, all except the last are ignored (under the assumption that earlier ones might not be fully warmed). Second and more critically, all solutions are validated to make sure that all lines that expected to be present actually are present - and if not, the solution is not written.

Solutions are stored as a list of polynomial fit terms. These are relative to the 'middle' of the array in array coordinates which makes them vulnerable to changes in cropping or binning, though the pipeline does know how to translate binning changes and thus arcs can be taken in binning modes different from the science data. (Note: it is not a bad idea to always take 1x1 binning arcs to avoid line saturation and help ensure accurate centroid measurements even if science observations are binned.)

Associated options:

• /forcewavcal - Solve wavelength solution even for marginal lamp combinations
• /novalidatewavcal - Do not validate wavelength solution.

Output:

• fp[b|r]*r.sol - Wavelength solution polynomials.

Diagnostic output:

• fp[b|r]yymmdd_????r_arc.wavsol.ps - Wavelength solution check plot, showing the 1D spectrum with identified lines marked with triangles and putative sky line matches marked with red (strong lines) or gray (weak lines) lines. A second page zooms in on the strong lines.
• wavsol[b|r].eps - A combined plot showing all wavelength solutions.

wavapply

Does two things: first, it matches arc solutions to science data to determine the actual expected wavelength of each pixel; next, applies a linear flexure correction to these wavelengths using the sky lines.

For consistency, each base configration (grating+dichroic) has its own associated master arc solution that is used for all observations with that configuration during a run, even if multiple arc solutions exist. If arcs were taken during the night (i.e., as well as during the afternoon/morning), the arc closest in time to an observation is used to correct the linear (central wavelength) term only, but all higher-order terms originate from the master arc.

Flexure adjustment is usually performed by matching the wavelengths of detected sky lines in the sky-spectrum column. Failing this (for very short spectra or spectra in twilight) telluric absorption is used instead. This is much easier for the red camera than for the blue camera, since the latter generally has only one or even zero strong sky lines, and as a result the blue wavelength solution is much less likely to be accurate.

Associated options:

• altarcdir - Additional directory/directories to check for arc solution files (e.g. to consider arcs from other nights or runs.)

Output:

• els[b|r]*.spec - Wavelength-calibrated 1D spectra

Diagnostic output:

None.

response

Determine the response function, a translation function from (spatially summed) DN's to actual physical f_lambda units as a function of wavelength using observations of known standard stars.

Standard star reference are taken from a variety of sources. Some of these (e.g. HST CALSPEC standards) are excellent but others are quite poor; many are missing spectral features and some contain telluric lines. The program interpolates both these standard reference spectra and the observed counts spectra over known absorption and telluric lines, then determines an overall response function which is then median-filtered (to remove outliers) and smoothed (to suppress noise). The telluric absorption profile is then measured separately by comparison of the actual counts spectrum to the interpolated spectrum. These are stored as separate columns in a .response file.

Associated options:

None.

Output:

• cels[b|r]*.response - Response files (for standards).

Diagnostic output:

• els[b|r]yymmdd_nnnnr_object_response.eps - A plot showing the standard star reference spectrum, observed spectrum, and response solutions. Also flashed to screen.

fluxcal

Uses the response function to flux-calibrate observations. The program tries to find two standard stars, one at lower airmass and one at higher airmass, and averages them (after adjusting each using a model Mauna Kea atmospheric attenuation function). 'Good' standards with few spectral features and very good reference spectra are chosen in preference to 'poor' standards. The overall response function is applied (again with an airmass adjustment based a Mauna Kea atmospheric adttenuation curve), and then subsequently the telluric absorption is corrected for using observations of the same standard.

Telluric-specific standards are not specifically recognized or applied at this stage; the overall flux standard is always the same star that is used for telluric correction.

Associated options:

• /notelluric - Skip telluric correction.
• altresponsedir - Give an alternative standard-star response curve directory or list of directories, such as a directory with processed standard-star response curves from a previous run.

Output:

• cels[b|r]*.spec - Flux-calibrated 1D spectra.

Diagnostic output:

None.

connect

Combines the flux-calibrated red and blue files to produce a final output spectrum.

Blue and red spectra are first matched to identify which spectra to 'pair up'. Next, the pipeline determines the region of wavelength overlap with which to calculate a rescaling factor (to correct the relative flux calibrations for small offsets).

The pipeline must first (for each object) calculate a rescaling adjustment factor because the flux calibration procedure can cause small (or on nonphotometric nights, large) absolute errors in the flux calibration. This is based on the median flux ratio over the overlap region. If the S/N is low or if there is no (or little) overlap, no rescaling is performed.

Additionally it must decide at what wavelength to join the spectra. (A 'hard' junction is used: every individual row in the output spectrum comes either from the blue or the red camera, not a coaddition of both.) This is based on the maximum of the products of the red and blue response over the overlap. For some grisms (e.g. 600/4000) this ends up being the edge of the blue CCD.

Associated options:

• /requirepair - Do not write out final spectra unless they contain both a red and a blue file. (If unset, the pipeline will copy any unpaired single-camera spectra to the final output directory.)
• shortnames - Omit the run date from the output names.

Output:

• lrisyyyymmdd_[object].spec - Final processed spectra.

Diagnostic output:

• rescale_object.eps - A plot showing the pre- and post- scaling red and blue spectra (top panels separate this into the object and the sky; bottom panel overplots the response curves as well). Also flashed to screen.

astrometry

Solve for the WCS of an image. Uses a custom python routine, autoastrometry.py. This is done in three stages: first, a set of reference star catalogs are downloaded from SDSS or Pan-STARRS for each field (in order to avoid having to repeatedly query the web for each individual image and if the pipeline is run again). Second, each image is aligned to this master catalog. Finally, each image is aligned again: beginning with a reference image (ideally a short exposure) and then 'building up' with each subsequent image aligned either to the reference image or another twice-solved image. This second step increases (relative) astrometric accuracy by allowing a much deeper catalog than SDSS/PS1 to be used.

Associated options:

None. (See the split option, however.)

Output:

• azfp[b|r]yymmdd_nnnn[l|r].fits - Image solved directly against a catalog
• eazfp[b|r]yymmdd_nnnn[l|r].fits - Image solved against another deep image
• eazfp[b|r]yymmdd_nnnn[l|r].tca[l|z|t] - Star catalogs calibrated against another image in the same field.

Diagnostic output:

None.

photometry

Solve for the zeropoint of each image. This uses an 'absolute and relative' combined algorithm similar to what is used for astrometry (i.e., a short exposure is solved first and then the relative zeropoints of longer exposures are solved relative to that short exposure). Generates a catalog file of star locations and magnitudes for each image; its file suffix indicates the nature of the absolute calibration of that image.

Note that solving absolute zeropoints with LRIS is difficult because the images are deep: it is challenging to find good PSF stars that are not saturated, except in short images (1 minute), and the unsaturated stars are likely to have significant catalog uncertainties. (This problem is most acute in redder filters.) Color terms are not corrected for. Partial zeropoints in a standard aperture (e.g. 1" radius) are generally more reliable because the risk of contamination by neighboring objects is much less: it is better to solve photometry directly against a catalog, rather than rely on a zeropoint solution.

Associated options:

• /nophotometry - Skip this step.
• /photometric - Ordinarily no attempt is made to establish an absolute calibration for fields without any secondary (SDSS/PS1) standards. If the night was photometric, set this flag and an airmass solution will be measured and used to extend the calibrations of known fields (including any Landolt fields) to uncalibrated fields. Mainly needed for u-band since all other filters can be calibrated more-or-less directly to PS1.

Output:

• stars[object].cat - Combined photometric catalog for a target.
• [e]azfp[b|r]yymmdd_nnnn[l|r].ca[l|z|t] - Star catalogs calibrated against against a catalog, or not at all (if .cat).
• [e]azfp[b|r]yymmdd_nnnn[l|r].tca[l|z|t] - Star catalogs (further) calibrated against each other, for monitoring transmission changes during an observation (even in the absence of an absolute calibration).

Additionally, the headers of the images produced in the astrometry step are edited to add the new values.

Diagnostic output:

• lris[b|r]cal.eps - Plot of image zeropoint versus airmass for each filter. Points are color-coded by calibration origin.
• lris[b|r]transmission.eps - Plot of image zeropoint versus time for all filters combined, as well as an extinction graph (airmass-dependent extinction is removed, so only cloud-based attenuation is shown: ideally this is close to zero for all images.). Points are color-coded by filter.

stack

Combines individual images together into a stacked mosaic. If both chips are being processed, all r images and all l images are combined separately, and then the coadds are combined together. The zeropoints calculated from photometry are used to adjust the photometric scaling of each image prior to stacking.

Associated options:

• shortnames - Omit the run date from the output names.

Output:

• coadd[object]_[filter].[l|r].fits - Reduced, stacked image, one chip only
• lris[b|r]yymmdd[object]_[filter].fits - Final reduced, stacked image. (Will be the same as the coadd file above unless both chips are used.)

Diagnostic output:

None.

Examples

1. Reduce data stored in the subdirectory raw/:

   IDL> lpipe, data='raw'



2. Fully reduce only spectroscopy, only the red camera:

   IDL> lpipe, mode='s', camera='red'



3. Run only the "prepare" step (bias subtraction/formatting), on raw data stored in a separate directory:

   IDL> lpipe, step='prepare', data='/scr3/user/lris/20120501/'



4. Run all the 2D spectroscopy reduction steps, but don't do any later (1D) reductions.

   IDL> lpipe, mode='s', stop='skysubtract'

   
   
5. Reduce all the imaging, include both left and chips (full field):

   IDL> lpipe, mode='i', chips='rl'



6. Reprocess the extraction and all subsequent steps of a target:

   IDL> lpipe, mode='s', start='extract', target='J1910+1234', /redo



7. Display some information at the command line:

   IDL> lpipe, /help

Additional Tools

Beyond the pipeline command-line options, a number of tools exist for data acquisition and exploration, and for checking and changing the pipeline results. These are run separately from the usual "lpipe" command. A brief summary of these is provided below; the code headers themselves can be checked for further information.

Checking, editing, and validation tools:

• lriscat - Produces a catalog table of many LRIS files. Works on raw data, intermediate-processed data, and spectra.
• lrisapertures - Aperture control GUI. Visually move, expand/contract, add, or delete object apertures and their corresponding sky-subtraction regions.
• lrisvalidate - Check a final spectrum against various intermediate data products.
• lriscompletioncheck - Compare minimally processed data to final output data to check which files were processed and which were not. Can help identify which exposures were problematic without having to re-run the pipeline.

Archive convenience tools:

• downloadkoadata - Convenience tool for direct downloading a run or runs from the KOA archive. Together with the the pipeline, this allows entire nights or even entire months of data to be processed in just 2-3 lines of code input.
• downloadkeckdata - Wrapper for rsync downloads directly from the Keck copymon scratch disks. Facilitates automatic acquisition of data during a run.
• unwrapkoa / sortkoa - Sorts downloaded data into nights. unwrapkoa un-tars a series of KOA tarballs before sorting. sortkoa sorts data that was downloaded directly or which has already ben un-tarred.

Installation

Create the subdirectory 'lpipe' somewhere on your hard drive (probably in your IDL directory), and unpack the contents of the pipeline tarball there (e.g., tar -xvf lpipe.tar.gz). You will need to tell IDL about the existence of this new directory by editing the IDL_PATH system variable: add the string ":+/path/to/lpipe:+/path/to/lpipe/dependencies/" to whatever paths are stored there currently, replacing "/path/to/" with the actual path. (The variable will need to be edited in your .bashrc, .cshrc, or .idlenv file to be available for future use.) The GSFC IDLastro routines must also be installed (and visible within IDL_PATH); see http://idlastro.gsfc.nasa.gov/ for programs and instructions.

In order to fully process imaging observations, you will also need to have autoastrometry, swarp and sextractor installed (this requirement will eventually be removed via a simplification of the astrometric solver method). If the latter two cannot simply be called via "swarp" and "sex", you may need to edit the file lpipe.par AND also edit the dependencies/autoastrometry.py file to indicate the actual commands to call these routines in the global variables at the top of the code. The standard UNIX routine wget is also used to download star catalogs. These are not necessary for spectrosopic reductions.

See the installation guide for more info.

Troubleshooting

While this pipeline is designed to deal with all possible observing circumstances (including many common mistakes), much more testing and development will be required before this ideal is fully reached. Despite best efforts, the program may crash if it encounters an unanticipated situation or has problems accomplishing its goals and is unable to proceed. If you encounter problems, try e-mailing Daniel Perley (d-a-perley[at]ljmu-ac-uk; replace the dashes with dots) for assistance, after checking the below.

If the pipeline does not crash, but does not process any files:

• Is the pipeline unable to find a necessary function or subroutine?
  Double-check that you have set up IDL_PATH correctly and that the GSFC IDLAstro libraries are installed (see installation).
• Are the data files in a different directory?
  If the raw data files are not in the current working directory, you need to point the pipeline to the directory containing raw data. Set data='/path/to/data/' at runtime or edit the data keyword in lpipe.par.
• Are you using old LRIS data or data downloaded from the KOA archive?
  The pipeline assumes files are named according to one of the standard filename conventions, e.g. [b|r]yymmdd_nnnn.fits. If your files are named using a different convention you can rename them by executing the following command (assuming only raw data is in the current directory):

  IDL> lrisrename, '*.fits'

  If important files have the same sequence number this will cause them to be skipped because they don't have unique ID's. Instead use:

  IDL> lrisreseq, '*.fits'

• Are you using new LRIS red data (since May 2021?) Make sure you are using the latest version of the pipeline. If still having problems, you may need to edit the header data to correct errors in the CCD parameters written by the telescope software. See README_lrisr4.txt for more information.
• Did you already run the pipeline once on these data, but want to re-run?
  If you want to re-do a step, it will not overwrite existing files by default (the pipeline is designed to be able to automatically resume from wherever it left off if interrupted or if new data is added.) To override this behavior, set the /redo keyword, which will overwrite files from any previously-attempted steps. Be sure to set the "start" or "step" keywords to the step you want to start at unless you want to restart the pipeline from the beginning for all files.
• Did you already run the pipeline once on these data, and it did not complete?
  See the below section.

If the pipeline crashes, halts, or processes no files beyond a certain step:

• Did it report a 'file not found' or 'variable is undefined' error?
  If so, a subroutine, configuration file, or catalog it needed was not found at the expected location. Generally this is an installation problem, but you can inspect the paths in lpipe.par to make sure that the directories and files specified there actually exist.
• Did an external call (autoastrometry, swarp, sex) fail?
  Check that you have specified the paths correctly and actually have the required software installed for astrometry/coadding (see "installation", above.) Alternatively if you don't care about reducing images, specify mode='s' at runtime to process only the spectroscopy, which uses only IDL routines and requires no external packages (except IDLastro).
• Did it fail to generate an appropriate flatfield?
  If science frames are not being flat-fielded, setting the /forceflat option and rerunning starting from makeflat may help. Otherwise, you may be able to obtain calibrations from a different night and copy them to this directory. If desparate, you may be able to trick the pipeline into processing a file using a calibration taken in some other (closely related) setting by editing header keywords, though obviously this should be used with caution.
• Did wavelength calibration fail?
  It is common for some arc wavelength solution attempts to fail, usually because the observations were taken before the Cd and Zn lamps warmed up fully, although the arc might also be saturated because of too much binning (red 2x2 arcs are problematic) or too long an exposure time. This is fine as long as other, good arcs exist and solve correctly. If all arc calibrations fail for a given setting, the pipeline cannot perform most of the 1D spectroscopic reductions. If there are no good calibrations in the required setting, you can try to find an archival arc somewhere in the same setting. If good arcs exist and aren't being solved, possibly the central wavelength is off by enough to prevent automatic line matching from succeeding. Editing the CWAVE parameter in the file headers may fix things.
• Is it failing to perform flux calibration?
  It is also possible that no useable flux calibration standard is available. Check that the data directory contains a flux standard in the correct configuration. If one exists, it is possible that its processing was stopped at an earlier stage (e.g. lack of appropriately binned flat, wavelength calibration failure) - rerunning the pipeline might highlight this if so, and the solutions above may apply here as well. If the response solution of the standard was successfully produced (see response/ directory) it is possible that its configuration was not the same as your science data: double-check this. If so, y ou may need to obtain a standard from elsewhere.

If processing completes, but the results are problematic:

• Did it extract the wrong object, or no object at all?
  The pipeline does its best to determine which source was yours based on the location and brightness of its trace, relative to that of standards. If it guesses wrong or if you need more control over the placement of the aperture or the background window, the lrisapertures tool provides a visual way to inspect and modify your object apertures. Run:

  IDL> lrisapertures

  This offers an intuitive visual interface to change, add, or subtract extraction apertures. (Aperture settings are stored in the file userobjectpos.txt, which can also be edited by hand.) Be sure to edit the corresponding red file to match the blue, and vice versa. Then, re-run the pipeline starting with start='extract' (and /redo). You can also inspect (although not edit) the apertures/traces within DS9 (open the FITS 2D spectrum and then load [filename].tracepos.reg) or view every file at once by opening "lristraces.html" in a web browser or loading profiles[b|r].ps. You can update these plots as follows:

  IDL> lrisplotprofiles, camera='b'  ; (or camera='r')

• Does one (or a few) particular output spectra look bad?
  Some possible causes are:
  • Aperture placement issues: If just a few spectra show problems (in particular, square-wave-like "oscillations" or other jumps) this probably originates from the fact that the background aperture was placed over another object. The pipeline may also have selected the wrong object for extraction or extracted different objects on the red and blue sides. In either case, the solution is to simply type "lrisapertures" and reposition the apertures and/or background locations (see above item). In a future release the background placement will automatically avoid contaminating objects to reduce the incidence of this problem.
  • Tracing issues: On rarer occasions (very faint objects) the tracing may have failed although this problem has essentially gone away in current LPIPE versions due to a major upgrade in the tracing algorithm. You can check the trace placement using the tracing .html files, by loading the appropriate extraction region file on top of the 2D spectrum in DS9, or by re-running the profile plotter directly (lrisplotprofiles). There is no simple solution to tracing issues if encountered.

If not one of these, it might also be one of the issues below.
• Do many spectra, or all spectra in a particular configuration, look bad?
  If all your spectra show similar issues the issue is more likely to be connected to the calibrations.
  
  • Wavelength calibration: Issues with wavelength calibration will propagate to the flux calibration. Very poor red-blue matching or telluric corrections are tell-tale signs of wavelength calibration problem. You can also look or signs of issues in the wavelength calibration in the output logs, in the wavelength solution plots (spredux1d/*wavsol.ps), or in the sky line plots (skyplots*.ps). It is common for some individual arc frames to have poor wavelength solutions (because of non-warmed lamps) but the pipeline avoids using these and they don't cause any problems. However, if the wavelength calibration for all arcs shows issues this is a major problem. There is no easy solution for this, but you can delete any bad arcs (and acquire new ones or use archival arcs), or override the wavelength solution by writing your own polynomial terms into the *arc.sol files. On rare occasions, the instrument itself can be the source or problems: an individal arc lamp can malfunction and not actually turn on when the header keywords say that it is on, or the grating angle can not be what is expected. Workarounds exist (edit brokenarcs.txt or the WAVELEN keywords in the file headers.)
  • Flux calibration: The flux calibration itself is usually robust except when the wavelength calibration fails in a key region, e.g. around the dichroic. If the wavelength calibration is good but flux calibration is not, you can inspect individual standard star's flux solutions and delete any bad ones. If your target was observed at a very different airmass from your standards or in very different seeing conditions, this is difficult to calibrate well due to differential slit loss and extinction. You may just have to live with a sub-optimal flux calibration unless you can acquire a new standard-star observation. If your flux calibration star was poor (faint, too many lines, no bright and no sky lines for flexure correction) this will also lead to smaller issues.
  • Telluric calibration: If the flux standard is at a different airmass from the standards the telluric correction will inevitably be poor as the model is too simplistic. Ideally you should obtain better standards, or you can set /notelluric and perform a separate correction (or excision) yourself.
  • Red-blue combination: The red-blue scaling may be off, particularly if there are also issues with flux and/or wavelength calibration, or the S/N is very low, or there are issues with background subtraction. It is possible to manually combine red and blue with a fixed scaling offset to reduce the impact of these issues by running lrisconnect directly.
• Is the output spectrum noisier than you expected, or does it contain narrow artifacts?
  
  • Possibly the aperture is too large or the sky aperture contains another object that's being subtracted from yours (use lrisapertures). If the slit drifted during observations (loss of guiding) this will also cause difficulties.
  • Poor sky subtraction may be a culprit in some gratings. You can try turning off 2D sky subtraction within lrisapertures.
  • Residual cosmic rays not removed by the pipeline may remain in your final object spectrum. You can try filtering these more aggressively by adjusting the nsigma parameter in pzapspec, but watch out to be sure narrow features aren't zapped.
  • Especially if the 0.7" slit was used, flexure in the spatial direction can cause significant noise due to residual flat-field variations associated with tiny slit defects. A tool exists to remove these (reflatslit) but it is not incorporated in the pipeline or documented yet.
• Is the cosmic ray removal malfunctioning?
  In some circumstances bright nebular emission lines can be zapped, especially in conditions of good seeing and binned data. The cosmic ray cleaner settings can be modified internally if this is a problem. Check both the 1D reduction and the 2D subtracted frames to be sure cosmic ray cleaning is not causing problems if nebular emission lines are important to your science. By default, the clean threshold is set relatively high to avoid zapping real features at the expense of some cosmic rays getting through (especially in imaging mode) but very bright emission lines or images in exceptional seeing can still cause problems.
• Do the reduced images look bad?
  The most common source of this is bad flat-fielding. Especially if twilight flats were not taken, the routine may have attempted to construct a supersky flat, which is difficult to do with Keck due to bright stellar halos and (especially) galaxies. There is a way to specify fields not to use in supersky flats (badflattargets.txt). Fields with bright galaxies are also susceptible to sky over- subtraction during stacking (within swarp). Sky subtraction can be turned off for these fields (currently manually by modifying the swarp file, but a semi-automated method will be coming soon.) Other times, a bad image can result from unavoidable instrument issues such as scattered light from a bright star outside the field.
• Were certain files skipped and not reduced?
  If the pipeline encountered a non-fatal problem processing an individual image or spectrum (such as a failed wavelength calibration, inability to flatfield or calibrate, or no trace to extract) then it will simply not process that file any further. For most such cases a summary of the affected files and the problems will be printed out at the end of processing. If a file is not being processed and you do not see it in the final summary, you can simply rerun the pipeline (without deleting any files and without setting the redo flag) and it will try to repeat any failed steps of this nature, so you can see the issues during runtime without them getting lost in other notifications from successful steps.

Users are encouraged to report all major bugs (especially crashes) by e-mailing Daniel Perley (d-a-perley[at]ljmu-ac-uk).