;+
; NAME:
;   lpipe                                         (version 2022.06)
;
; PURPOSE:
;   Fully-automated reduction of LRIS imaging and spectroscopy.
;
; CALLING SEQUENCE:
;   lpipe, [settings]
;
; INPUTS AND KEYWORDS (all optional):
;   directories  - List or search string of subdirectories. (Usually blank.)
; General:
;   data         - Location of raw data (current directory if unspecified)
;  /help         - Print some basic help information
;   reducer      - Specify name of user to add to the file headers
; Step control:
;   mode         - Mode to process (imaging or spectroscopy)
;   start        - Start with this step, skipping previous ones
;   stop         - End with this step, skipping subsequent ones
;   step         - Do only this step
;  /redo         - Repeat actions, overwriting any existing files
; File control:
;   camera       - Camera to process (red or blue)
;   chip         - Chip to process (left or right)
;   files        - Specific file numbers to process 
;   filters      - Specific filters to process 
;   gratings     - Specific gratings to process 
;   targets      - Specific targets to process 
;   date         - Specific night or month to process (e.g. '150423')
; Options:
;  preparation/formatting
;   /prepareall  - Bias-subtract multislit and dark files (still ignores specpol)
;   /skipfocus   - Skip frames associated with focusing
;  flat-fielding
;    flatrequire - Specify the number of images needed to make a flat-field
;   /norebin     - Don't rebin if no flats exist in same binning as data
;   /noreflat    - Do not perform super sky flat correction for images
;    altflatdir  - Give an alternative flat-field directory (archival flats, etc)
;   /forceflat   - Produce all flatfields even if not needed for science data
;   /ignoredichroic- Allow dichroic differences when flat-fielding images
;   /rightchipdead - Normalize flats using left CCD
;  cosmic ray cleaning
;   /nocrclean   - Do not zap cosmic rays at all
;   crzeal       - Adjust cosmic ray cleaning (default=1.0, >1.0:more aggressive)
;  sky subtraction
;   skyalgorithm - Use a different sky-subtraction algorithm ('kelson')
;  summing
;   /nosum       - Do not coadd any repeat spectra (e.g., for time series)
;  tracing
;   /norbalign   - Don't attempt to align red and blue traces of the same target
;  spectrum extraction
;   /noreaddsky  - Rely on 2D sky subtraction for sky-line removal.
;  wavelength calibration
;    altarcdir   - Give an alternative arc-lamp wavelength solution directory
;   /forcewavcal - Solve wavelength solution even for marginal lamp combinations
;   /novalidatewavcal - Do not validate wavelength solution.
;  flux-calibration
;    altresponsedir - Give an alternative flux-calibration directory
;   /notelluric  - Skip telluric correction
;  red-blue connection
;   /requirepair - Do not write out final spectrum without both red+blue
;  astrometry
;   /distort     - Apply distortion solution (not well-tested, may not work)
;  photometry
;   /nophotometry- Skip zeropoint calculation and attenuation adjustment
;   /photometric - Use zeropoint solution to calibrate non-SDSS/PS1 fields
;  visualization/output
;   /nodisplay   - Don't pop up display windows (for remove/server reductions)
;   /nofailuresum- Hide the summary of unprocessed files printed at the end
;   /debug       - Print out more info; turn off some exception handling
;   /timer       - Print out processing times
;   /shortnames  - Exclude the six-digit date string from output files
;  Other
;   /continuous  - Keep running, assimilating new data as it appears
;   /configure   - Write configuration file to disk for even more control
;   
;
; ADDITIONAL OPTIONS:
;   If any of the following files are found in the directory where lpipe
;   is run, it will change the default behavior.  (These have not been tested
;   recently.)
;
;   lpipe.par - Contains various defaults (mainly directory paths)
;   catalog.txt - Source positional catalog.  Use to auto-correct the object
;        name in the header based on position (independent of TARGNAME.)
;   brokenlamps.txt - List of arc lamps that are not functioning.
;   userobjectpos.txt - List of files and extraction positions (y-coordinates).
;        Enables the user to specify which object(s) to extract.
;        Now handled visually by lrisapertures, or see below to set manually
;   seeing.txt - A file with only two numbers, the minimum and maximum seeing
;        experienced in arcseconds.  (obsolete?)
;   extendedsources.txt - List of imaging fields (TARGNAMEs) containing large 
;        galaxies or other sources for which median-filtered sky subtraction is
;        not appropriate. Scalar background subtraction is used instead. 
;        These will also be excluded from constructing supersky flats.  
;   badflattargets.txt - List of additional targets not to use in making a
;        supersky flat (e.g., fields with lots of scattered light.)
;   badfiles.txt - List of files not to process at all.
;
;
;
; OUTPUTS:
;   The following files are written to disk (from most to least processed):
;
;   Imaging -
;   * lris[object]_[filter].fits         - Reduced, stacked image
;     stars[object].cat                  - Photometric catalog for a target
;     coadd[object]_[filter].[l|r].fits  - Reduced, stacked image, one chip only
;     eazfp[b|r]yymmdd_nnnn[l|r].fits    - Individual aligned image
;                                          (one exposure)
;     eazfp[b|r]yymmdd_nnnn[l|r].tca[t|l|z]-Calibrated list of stars in image
;                                          using another exposure of this field
;     eazfp[b|r]yymmdd_nnnn[l|r].cal     - Calibrated list of stars in image
;                                           using SDSS imaging of this field
;     eazfp[b|r]yymmdd_nnnn[l|r].caz     - Calibrated list of stars in image
;                                          using a zeropoint/airmass solution
;     eazfp[b|r]yymmdd_nnnn[l|r].cat     - Uncalibrated list of stars in image
;     azfp[b|r]yymmdd_nnnn[l|r].fits     - Image solved directly against an
;                                          astrometric catalog catalog

;
;   Spectroscopy -
;   * lrisyymmdd[object].spec            - Flux-calibrated spectrum of an object, 
;                                          combining blue/red and all exposures.
;                                          ASCII table (read with readspec.pro)
;     cels[b|r]yymmdd_nnnn[l|r].[obj].spec - Flux/wave-calibrated spectrum
;     els[b|r]yymmdd_nnnn[l|r].[obj].spec  - Wave-calibrated 1D spectrum
;                                              (not flux-calibrated)  
;     xls[b|r]yymmdd_nnnn[l|r].[obj].spec - Extracted, uncalibrated 1D spectrum
;     ls[b|r]yymmdd_nnnn[l|r].[obj].fits - Summed 2D spectrum
;     szfp[b|r]yymmdd_nnnn[l|r].fits    - Sky-background subtracted 2D spectrum
;
;
;   Both -
;     zfp[b|r]yymmdd_nnnn[l|r].spec     - Cosmic ray-cleaned image
;     fp[b|r]yymmdd_nnnn[l|r].spec      - Flat-fielded image, split between the
;                                          disconnected left/right chips
;     fp[b|r]yymmdd_nnnn.spec           - Flat-fielded image
;     p[b|r]yymmdd_nnnn.spec            - Overscan-subtracted, bad-pixel 
;                                          marked, coverted to simple format,
;                                          updated header (otherwise raw) image
;
;   Only the starred (*) files are written to the current directory.  Other 
;   files are written to the imredux/ or spredux*/ directories.  Processed
;   calibrations are written to the flats/, arcs/, or response/ directories.
;   If using KOA file name conventions, yymmdd_nnnn becomes yymmdd.nnnnn.
;
;         
;
; COMMENTS:
;   This code is meant to be fully automated, producing science-quality
;   output with a single command line with zero intervention.  Reduces
;   both LRIS-B and LRIS-R data (including LRIS-R1 with some limitations) in
;   either imaging or spectroscopy mode.
; 
;   The production of images is quite high-level and includes photometric 
;   calibration of the field (although the accuracy of this has not been
;   robustly tested).  Astrometric distortion corrections are currently
;   in development and may not always work correctly: please report any
;   difficulties with the astrometry.
;
;   Full spectroscopic extraction and reduction (including flux calibration)
;   is also available.
;
;   The program works in a series of steps following standard CCD reduction
;   techniques, automatically recognizing calibration files, matching
;   calibrations together and science frames with appropriate calibrations,
;   etc.  Users looking for more oversight over the reductions can run each
;   step individually with the "step" command.
;
;   If the reduction is interrupted, the program will resume without
;   incident (although any failed steps may be repeated); each task is 
;   run independently of all others.
;   
;   The program tries to correct for observer mistakes (such as inconsistent
;   windowing or poor-quality calibrations) or instrument problems where
;   possible.  Some situations require user attention to obtain a reliable
;   reduction - see the troubleshooting page if the pipeline does not return
;   the results you expect.
;
;   Filenames for the input raw images are expected to be in one of the 
;   standard formats output by the telescope or KOA archive.
;   Compressed (.gz) files are also acceptable.  They can be either in the 
;   current IDL working directory or in a different directory as specified
;   by the 'data' keyword at the IDL prompt at runtime.
; 
;   The code runs in a series of steps in the following order:
;
;   1. "Prepare" the data by converting the multi-header extension FITS files
;      to a standard single frame, correcting/flagging known pixel defects,
;      cropping unused area of the chip, bias-subtracting using the overscan,
;      and adding extra information to the header.   The X and Y axes
;      for spectroscopic frames are transposed for easier viewing of traces on 
;      standard monitors and to match 1D spectrum plots (wavelength=x-axis).
;      Output: p[b|r]*.fits (in imredux/ and spredux2d/ subdirectories)
;
;   2. Create flat-fields.  The program searches through all science exposures
;      and determines what flat fields are needed, then attempts to make the
;      best possible flat for each case.  In the case of imaging flats,
;      stars are identified and masked out.  For spectroscopic flats, a response
;      correction is applied (these are pixel flats only).  Regions of a 
;      spectroscopic flat with low signal (common for LRIS-B in the UV) are 
;      not flat-fielded in the wavelength direction (slit correction only).
;      Output:  flats/l[b|r]*flat*.fits
;
;   3. Flat-correct data.  Divide each image by the flatfield.  A more
;      refined cropping is also done at this stage, depending on the placement
;      of the image area during the run (which is variable.)
;      Output:  fp[b|r]*.fits
;   
;   4. "Re"-flatten imaging data by constructing a super-sky flat using a 
;      median of the science images in a given configuration.   This is 
;      customized for each exposure (the exposure itself is excluded from the
;      flat-field to avoid self-division, which can create peculiar image noise
;      statistics). For LRIS-R1 (pre-2009) I- or RG850-band data this process 
;      will be subtractive rather than divisive (fringe correction), but this 
;      is not implemented yet.  (This can be slow and isn't helpful unless a
;      great deal of imaging is acquired during the night, and is off by 
;      default.)
;      Output:  ifp[b|r]*.fits    (imaging only)
;
;   5. Split the data into separate right and left chips.  (Because of the
;      physical gap and rotation between the LRIS chips, they must be treated
;      independently after this point.)  A standard astrometric distortion is
;      added at this point for imaging exposures.
;      Output:  [i]fp[b|r]*[l|r].fits
;
;   6. Removes cosmic rays, using the independent routines pzap.pro and
;      pzapspec.pro.  See those programs for more information.  
;      This can be a time-consuming process.
;      Output: zfp[b|r]*[l|r].fits     (the "i" is removed from imaging.)
;
;   At this point different procedures are applied for imaging and spectroscopy.
;   For imaging:
;  
;   7. Solve astrometry of the field against the best available online catalog
;      (SDSS or USNO-B1.0), using the independent autoastrometry.py code.
;      (Also requires sextractor to be installed.)
;      All images are solved against a common star catlog, then each image is 
;      solved in a relative sense (optionally including distortion correction) 
;      against the image nearest to it, starting from the central image.
;      Output:  azfp[b|r]*[l|r].fits   (catalog alignment)
;              eazfp[b|r]*[l|r].fits   (relative alignment)
;   
;   8. Produce photometric solution of the field.  The GSFC IDLastro aper.pro
;      program is used to measure photometry of point sources in each field
;      for comparison of different exposures (on the same target) to establish 
;      the relative zeropoints.  
;      Output: eazfp[b|r]*[l|r].[t|][cat,cal,caz] and other text files
;  
;   9. Stack exposures.  A weighted, masked median is performed for each field.
;      Requires swarp to be installed and properly linked.
;      Output: coadd[object].[filter].[l|r|]fits
;              lris[b|r][object]_[filter].fits
;
;   For spectroscopy:
;
;   7. Subtract night-sky emission lines by fitting low-order polynomials in 
;      vertical blocks passing across the data.   Can introduce artifacts in 
;      some situations (e.g. poor flatfielding or 0.7" slit).  Slow.
;      Output: szfp[b|r]*[l|r].fits
;
;   8. Sum exposures.  This uses only the header information: in the event of
;      guiding loss this will not be desired!
;      Output: ls[b|r]*.fits    (in spredux1d/ subdirectory)
;
;   9. Determines the approximate trace position of all objects, starting with
;      standard-star observations.
;      Output:  ls[b|r]*.trace, ls[b|r]*.profile, objectpos.dat
;
;   10. Extract trace to produce a 1D spectrum of the object of interest.
;      Output:  xls[b|r]*.spec
;
;   11. Wavelength-calibrate arc frames.
;      Output:  fp[b|r]*[l_r].sol
;
;   12. Applies and adjusts wavelength calibration.
;      Output:  els[b|r]*.spec
;
;   13. Flux-calibrate the 1D spectrum using observations of standard stars.
;      Output:  cels[b|r]*.spec
;
;   14. Combine all spectra of a single object in a single grating.
;      (This is an optional step and is now skipped by default.)
;
;   15. Connect red and blue sides of an object to produce final spectrum.
;      Output: lrisyymmdd_[object].spec
;
;
; 
; EXAMPLES:
;   1.  In a directory containing a full night worth of LRIS data, enter:
;
;       IDL> lpipe
;
;       This will automatically execute all of the steps above.
;       Depending on the quantity and binning of data from your run and the 
;       speed of your computer, the reduction of data from an entire night
;       takes 20 minutes up to about three hours.
;       This defaults to reducing only the right chip, ignoring the left.
;
;
;   2.  Fully reduce only spectroscopy, only the red camera:
;
;       IDL> lpipe, mode='s', camera='red'
;
;
;   3.  Run only the "prepare" step (bias subtraction/formatting), on raw data
;       stored in a separate directory:
;
;       IDL> lpipe, step='prepare', data='/scr3/user/lris/20120501/'
;
;
;   4.  Run all the 2D spectroscopy reduction steps, but don't do any later
;       (1D) reductions.
;
;       IDL> lpipe, mode='s', stop='skysubtract'
;
;
;   5.  Reduce all the imaging, include both left and chips (full field):
;
;       IDL> lpipe, mode='i', chips='rl'
;
;
;   6.  Convert KOA-style filenames, 
;       move raw data to another directory, then run lpipe, and plot all
;       the spectra for quick viewing:
;
;       IDL> lrisrename, 'L?.*.fits'
;       IDL> $mv [b|r]*_????.fits raw/
;       IDL> lpipe, data='raw/'
;       IDL> lrisplotall
;
;   7.  Reprocess the extraction and all subsequent steps of a target:
;
;       IDL> lpipe, mode='s', start='extract', target='J1910+1234', /redo
;
;
;   8.  Display some information at the command line:
;      
;       IDL> lpipe, /help
;
;-
;
;
;   INSTALLATION:
;
;   Create the subdirectory 'lpipe' somewhere on your hard drive (probably
;   in your IDL directory), and unpack the contents of this installation file
;   there (e.g.,  tar -xvf lpipe.tar.gz).  You will need to tell IDL about
;   the existence of this new directory by editing the IDL_PATH system variable:
;   add the string ":+/path/to/lpipe:+/path/to/lpipe/dependencies/" to whatever
;   paths are stored there currently, replacing "/path/to/" with the actual path.
;   (The variable will need to be edited in your .bashrc, .cshrc, or .idlenv file
;   to be available for future use.)  The GSFC IDLastro routines must also be
;   installed (and visible within IDL_PATH); see http://idlastro.gsfc.nasa.gov/
;   for programs and instructions.
;
;   In order to process images, you will also need to have autoastrometry,
;   swarp and sextractor installed (this requirement will eventually be removed
;   via a simplification of the astrometric solver method).  If the latter two
;   cannot simply be called via "swarp" and "sex", you may need to edit the 
;   file lpipe.par AND also edit the dependencies/autoastrometry.py file
;   to indicate the actual commands to call these routines in the global
;   variables at the top of the code.  The standard UNIX routine wget is also 
;   used to download star catalogs.  These are not necessary for spectrosopic
;   reductions.
;
;   For a Caltech astronomy cluster matchine, it is recommended to simply point
;   your IDL path variable to the latest release version in Daniel Perley's 
;   home directory (see citsetup.txt for more info.)  This will ensure you 
;   always have the most up-to-date version.
;
;   More detailed installation instructions are given in installation.txt.
;
;
;
;
;   TROUBLESHOOTING:
;
;   While this pipeline is designed to deal with all possible observing
;   circumstances (including many common mistakes), much more testing and 
;   development will be required before this ideal is fully reached.
;   Despite best efforts, the program may crash if it encounters an 
;   unanticipated situation or has problems accomplishing its goals and is
;   unable to proceed.  If you encounter problems, try e-mailing Daniel Perley
;   (d.a.perley@ljmu.ac.uk) for assistance, after checking the below.
;
;   If the pipeline does not crash, but does not process any files:
;
;   * Is the pipeline unable to find a necessary function or subroutine?
;       Double-check that you have set up IDL_PATH correctly and that the GSFC
;       IDLAstro libraries are installed (see installation).
;  
;   * Are the data files in a different directory?
;       If the raw data files are not in the current working directory, you
;       need to point the pipeline to the directory containing raw data with 
;       the "data" keyword in lpipe.par.
;
;   * Are you using old LRIS data?
;       The pipeline assumes files follow the standard convention as written
;       at the telescope (since ~2010) or the KOA filename convention.   If 
;       your files are named using a different convention (e.g. older data)
;       then you can rename them by executing the following command (assuming
;       only raw data is in the current directory):
;         IDL> lrisrename, '*.fits'
;       If important files have the same sequence number this will cause them
;       to be skipped because they don't have unique ID's.  Instead use:
;         IDL> lrisresequence, '*.fits'
;
;   * Did you already run the pipeline once on these data, but want to re-run?
;       If you want to re-do a step, it will not overwrite existing
;       files by default (the pipeline is designed to be able to automatically
;       resume from wherever it left off if interrupted or if new data is
;       added.)  To override this behavior, set the /redo keyword, which will
;       overwrite files from any previously-attempted steps.  Be sure to set 
;       the "start" or "step" keywords to the step you want to start at unless
;       you want to restart the pipeline from the beginning for all files.
;
;   * Is the pipeline not progressing beyond a certain stage of operation?
;       Likely, this is because of the lack of an appropriate calibration file
;       (e.g. a flat-field, arc, or flux-standard taken in the same setting.)
;       You may be able to trick the pipeline into processing a file using
;       a calibration taken in some other setting by editing header keywords,
;       though obviously this should be used with caution.
;
;       
;
;   If the pipeline halts or crashes during operation:
;
;   * Did it report a 'file not found' or 'variable is undefined' error?  
;       If so, a subroutine, configuration file, or catalog it needed was not 
;       found at the expected location.  Generally this is an installation 
;       problem, but you can inspect the paths in lpipe.par to make sure
;       that the directories and files specified there actually exist.  
;  
;   * Did an external call (autoastrometry, swarp, sex) fail?
;       Check that you have specified the paths correctly and actually have the
;       required software installed for astrometry/coadding (see "installation", 
;       above.)  Alternatively if you don't care about reducing images, specify
;       mode='s' at runtime to process only the spectroscopy, which uses only
;       IDL routines and requires no external packages (except IDLastro).
;   
;   * Did it encounter some other error while processing a file?  
;       Check which file the program was working on when it crashed.  If the
;       file is not essential, try deleting it and re-running the pipeline
;       starting with the current step (or delete ALL of the file's precursors
;       with the same file number and rerun the pipeline.)  Please report the
;       error (and perhaps include a copy of the deleted file) so the problem
;       can be identified and fixed in future releases.
;   
;   * Did it fail to generate an appropriate flatfield?
;       If science frames are not being flat-fielded, setting the /forceflat
;       option and rerunning starting from makeflat may help.  Otherwise, 
;       you may be able to obtain calibrations from a different night and 
;       copy them to this directory.  If desparate, you may be able to trick 
;       the pipeline into processing a file using a calibration taken in 
;       some other (closely related) setting by editing header
;       keywords, though obviously this should be used with caution.
;
;   * Did wavelength calibration fail?
;       It is common for some arc wavelength solution attempts to fail, usually
;       because the observations were taken before the Cd and Zn lamps warmed 
;       up fully, although the arc might also be saturated because of too much
;       binning (red 2x2 arcs are problematic) or too long an exposure time.
;       This is fine as long as other, good arcs exist and solve
;       correctly.  If all arc calibrations fail for a given setting, the 
;       pipeline cannot perform most of the 1D spectroscopic reductions.  If
;       there are no good calibrations in the required setting, you can try to
;       find an archival arc somewheresomewhere in the same setting.
;       If good arcs exist and aren't being solved, possibly the central
;       wavelength is off by enough to prevent automatic line matching from 
;       succeeding.  Editing the CWAVE parameter in the file headers may fix
;       things.
;
;   * Is it failing to perform flux calibration?
;       It is also possible that no useable flux calibration standard is available.
;       Check that the data directory contains a flux standard in the 
;       correct configuration.  If one exists, it is possible that its processing
;       was stopped at an earlier stage (e.g. lack of appropriately binned flat,
;       wavelength calibration failure) - rerunning the pipeline might highlight
;       this if so, and the solutions above may apply here as well.  If the response
;       solution of the standard was successfully produced (see response/ directory)
;       it is possible that its configuration was not the same as your science data:
;       double-check this.  If so, you may need to obtain a standard from elsewhere.
;
;
;   If processing completes, but the results are problematic:
;
;   * Did it extract the wrong object, or no object at all?
;       The pipeline does its best to determine which source was yours based on
;       the location and brightness of its trace, relative to that of standards.
;       If it guesses wrong or if you need more control over the placement of
;       the aperture or the background window, the lrisapertures tool provides
;       a visual way to inspect and modify your object apertures.  Run:
;          IDL> lrisapertures
;       This offers an intuitive visual interface to change, add, or subtract
;       extraction apertures.  (Aperture settings are stored in the file
;       userobjectpos.txt, which can also be edited by hand.)  Be sure to edit
;       the corresponding red file to match the blue, and vice versa.
;       Then, re-run the pipeline starting with start='extract' (and /redo).
;       You can also inspect (although not edit) the apertures/traces within
;       DS9 (open the FITS 2D spectrum and then load [filename].tracepos.reg)
;       or view see every file at once by opening "lristraces.html" in a web
;       browser or loading profiles[b|r].ps.
;       You can update these plots as follows:
;          IDL> lrisplotprofiles, camera='b'  ; (or camera='r')
;
;   * Does one (or a few) particular output spectra look bad?
;          Aperture placement issues:
;       If just a few spectra show problems (in particular, square-wave-like
;       "oscillations" or other jumps) this probably originates from the fact
;       that the background aperture was placed over another object.  
;       The pipeline may also have selected the wrong object for extraction or
;       extracted different objects on the red and blue sides.  In either case,
;       the solution is to simply type "lrisapertures" and reposition the
;       apertures and/or background locations (see above item).
;       In a future release the background placement will automatically avoid
;       contaminating objects to reduce the incidence of this problem.
;           Tracing issues:
;       For very faint objects the tracing may have failed.   You can
;       check the trace placement using the tracing .html files, by loading
;       the appropriate extraction region file on top of the 2D spectrum in 
;       DS9, by re-running the profile plotter directly (lrisplotprofiles),
;       using the 2D view within lrisapertures, or using lrisvalidate.
;       There is no simple solution to tracing issues if encountered.
;
;   * Do many spectra, or all spectra in a particular configuration, look bad?
;       If all your spectra show similar issues the issue is more likely to be
;       connected to the calibrations.
;          Wavelength calibration:
;       Issues with wavelength calibration will propagate to the flux 
;       calibration.  Very poor red-blue matching or telluric corrections are
;       tell-tale signs of wavelength calibration problem.  You can also look
;       or signs of issues in the wavelength calibration in the output logs,
;       or in the wavelength solution plots (spredux1d/*wavsol.ps).
;       It is common for some individual arc frames to have poor wavelength 
;       solutions (because of non-warmed lamps) but the pipeline avoids using 
;       these and they don't cause any problems.  However, if the wavelength 
;       calibration for all arcs shows issues this is a major problem.  
;       There is no easy solution for this, but you can delete any bad arcs 
;       (and acquire new ones or use  archival arcs), or override the
;       wavelength solution by writing your own polynomial terms into the
;       *arc.sol files.
;       On rare occasions, the instrument itself can be the source or problems: 
;       an individal arc lamp can malfunction and not actually turn on when the
;       header keywords say that it is on, or the grating angle can not be what is expected.  
;       Workarounds exist (edit brokenarcs.txt or the WAVELEN keywords in the
;       file headers.)
;          Flux calibration:
;       The flux calibration itself is usually robust except when the 
;       wavelength calibration fails in a key region, e.g. around the dichroic.
;       If the wavelength calibration is good but flux calibration is not, you
;       can inspect individual standard star's flux solutions and delete any
;       bad ones.  If your target was observed at a very different airmass from
;       your standards or in very different seeing conditions, this is 
;       difficult to calibrate well due to differential slit loss and 
;       extinction.  You may just have to live with a sub-optimal flux 
;       calibration unless you can acquire a new standard-star observation.
;       If your flux calibration star was poor (faint, too many lines, no bright
;       and no sky lines for flexure correction) this will also lead to
;       smaller issues.
;
;   * Is the output spectrum noisier than you expected, or does it contain
;     narrow artifacts?
;       - Possibly the aperture is too large or the sky aperture contains
;       another object that's being subtracted from yours (use lrisapertures).
;       If the slit drifted during observations (loss of guiding) this will
;       also cause difficulties.  
;       - Poor sky subtraction may be a culprit in some gratings.  You can
;       try turning off 2D sky subtraction within lrisapertures.
;       - Residual cosmic rays not removed by the pipeline may remain in your
;       final object spectrum. You can try filtering these more aggressively
;       by adjusting the nsigma parameter in pzapspec, but watch out to be 
;       sure narrow features aren't zapped.
;       - Especially if the 0.7" slit was used, flexure in the spatial
;       direction can cause significant noise due to residual flat-field 
;       variations associated with tiny slit defects.  A tool exists to remove
;       these (reflatslit) but it is not incorporated in the pipeline or 
;       documented yet.
;
;   * Is the cosmic ray removal malfunctioning?
;       In some circumstances bright nebular emission lines can be zapped, 
;       especially in conditions of good seeing and binned data.  Real cosmic
;       rays can also be missed.  If you are having problems with an image or
;       set of images, you can adjust the cosmic ray cleaning aggressiveness
;       with the crzeal parameter (default 1.0; increase to make it more
;       aggressive or decrease to make it less aggressive).  Some cosmic rays
;       (those that streak purely vertically or horizontally) are not well
;       cleaned by the algorithm and difficult to remove in any setting.
;
;   * Do the reduced images look bad?
;       The most common source of this is bad flat-fielding.  Especially if
;       twilight flats were not taken, the routine may have attempted to
;       construct a supersky flat, which is difficult to do with Keck due
;       to bright stellar halos and (especially) galaxies.  There is a way 
;       to specify fields not to use in supersky flats (badflattargets.txt).
;       Fields with bright galaxies are also susceptible to sky over-
;       subtraction during stacking (within swarp).  Sky subtraction can be
;       turned off for these fields (currently manually by modifying the swarp
;       file, but a semi-automated method will be coming soon.)  Other times, 
;       a bad image can result from unavoidable instrument issues such as 
;       scattered light from a bright star outside the field.
;
;   * Were certain files skipped and not reduced?
;       If the pipeline encountered a non-fatal problem processing an
;       individual image or spectrum (such as a failed wavelength calibration,
;       inability to flatfield or calibrate, or no trace to extract) then it 
;       will simply not process that file any further.  For most such cases a 
;       summary of the affected files and the problems will be printed out at 
;       the end of processing.   If a file is not being
;       processed and you do not see it in the final summary, you can simply
;       rerun the pipeline (without deleting any files and without setting the
;       redo flag) and it will try to repeat any failed steps of this nature,
;       so you can see the issues during runtime without them getting lost in
;       other notifications from successful steps.
;
;
;   The risk of encountering pipeline issues can be significantly reduced
;       by following standard observing procedures, below.  
;
;
;
;   OBSERVING TIPS:
;
;   Spectroscopy -
;
;   Slit: The 0.7" slit has many defects, and because this slit moves around
;      due to instrument flexure this inevitably results in extremely poor
;      flat-fielding and sky subtraction.  Use the 1.0" or 1.5" slit instead.
;      (However, a new routine is now available to correct 0.7" slit data
;      using the sky lines.  For now it needs to be run manually.)
;
;   Binning:  Do not bin the red side by more than 2x1.  Further binning
;      (e.g. 2x2) causes severe saturation issues on bright arc lines needed 
;      for automated wavelength calibration, and greatly increases the 
;      likelihood of false-positive cosmic-ray removal.
;
;   Arcs:  Please turn the mercury lamp on during red-side arc exposures (in
;      addition to the standard neon and argon lamps).  If using a setup with
;      the D560 dichroic where continuous wavelength coverage is desired, tune
;      the central wavelength during setup to be sure that the bright Hg
;      line appears on the red side as well as the blue - this is needed for an
;      accurate wavelength solution across the D560 dichroic.  Always make sure
;      the lamps are adequately warmed, and try to obtain calibrations not at
;      the horizon location.  If you change the configuration during the night,
;      it is also a good idea to take at least one new arc before changing it 
;      again: it is possible that a component may not be where you think it is.
;
;   Flats:  Halogen flats are poor (uneven illumination, variable spectrum, 
;      significant scattered red light present on the blue side) and should not
;      be relied on.  Spectroscopic dome flats are superior:  turn on the 
;      spectral dome lamp and get flats as you would for imaging.  Note that 
;      the UV cannot be pixel-calibrated with either the halogen OR with dome 
;      flats - the lamp is not hot enough.  Twilight flats and deuterium-lamp 
;      flats should provide sufficient UV signal.  Preliminary support for 
;      these has now been added to the pipeline, but they leave some line 
;      residuals so please use with caution.
;
;   Standards:  The list of recognized spectrophotometric flux-calibration 
;      standards can be viewed in the 'refdata' subdirectory as the file
;      specstandards.dat, or online at:
;      http://astro.caltech.edu/~dperley/programs/lpipe/specstandards.txt
;      Of these standards, the hot white dwarfs essentially always make much
;      better choices than features with strong absopriton lines.  If no
;      clean standard is available the flux calibration will be worse.
;      Any standards not on the list above will not be recognized as standards
;      (if you have a favorite standard not on this list, please suggest it.)
;      Ideally the same standard should be observed twice at significantly 
;      different airmasses in each setup.
;      Additionally, it is recommended to take arcs alongside observations of 
;      bright standards where a short integration time (<20 sec) is employed 
;      or for standards observed in bright twilight.  This will ensure the 
;      standard has a good wavelength solution if refinement based on 
;      night-sky OH emission is impossible, which in turn will help ensure 
;      accurate flux-calibration.
;
;
;   Imaging -
;
;   Windowing:  You can window the red side to Image Area on the XLRIS CONFIG
;      menu to save readout and processing time with essentially no compromise.
;      One can window it even more to save more readout time at the expense
;      of FOV (use the ETC menu).  Windowing is fully integrated into the 
;      pipeline, and images of different windowing settings can be processed
;      without difficulty even if the windowing of the flats differs from that
;      of the target.
;
;   Flats:  In general, twilight flats produce the most reliable results, so
;      try to get them if possible.  Observing in the U-band REQUIRES twilight
;      flats since there are not enough dark-sky counts and the dome lamp is
;      not hot enough to emit in the UV.  (Twilight flats do seem to create
;      artifacts with the glue defects in some filters, however.)  Also, note
;      that the mirror does not return precisely to its original state after
;      changes in and out of spectroscopy mode, changing the vignetting and
;      dust pattern.  Try to avoid changing the configuration between flats
;      and science frames, or take sufficient science frames that super-sky
;      correction flats ('reflat' step) can be constructed reliably.  Of
;      course, this is not always possible.
;
;   Standards:  If you are imaging any uncalibrated fields and conditions are 
;      photometric, try to get at least two measurements of 
;      Landolt photometric standard fields (preferably two observations of 
;      the same standard field) per filter at significantly different 
;      airmasses.  This can be done during twilight.  While the program now
;      uses imaging of science fields in SDSS to help calculate the zerpoint
;      solution, these can be susceptible to crowding when calculating the
;      aperture correction and Landolt fields are useful to verify them.
;
;   Science fields:  Always specify that you want the source placed at the 
;      "LRIS-B imaging position" to avoid the chip gap.  A test exposure is
;      usually not necessary if the operator is trustworthy, but a short
;      exposure (< ~30s) will help significantly with calibration later if a 
;      moderately deep (e.g. SDSS) calibration of the field is not available,
;      especially in the redder filters (z- / RG850- band in particular).
;      When dithering, try to make sure that every image in the
;      dither sequence is at a different vertical position (as seen on the
;      live DS9 display) by e.g. dithering 1" vertically to accompany each
;      otherwise horizontal dither. LRIS-R3 has some bad columns (horizontal
;      lines in the viewer orientation) which one should avoid keeping over 
;      the same horizontal location (helpful, since these columns actually 
;      fluctuate in intensity and do not always flat-field out.)
;
;   Organization -
; 
;   Use save_state and restore_state (NOT the GUI) to store your LRIS 
;      configurations.  This includes the CCD parameters and will ensure that
;      you do not forget to change the binning and windowing which are NOT
;      included in the XLRIS save/load options.  
;
;
;
;   RELATED PROGRAMS:
;
;   To check what calibrations (etc.) you have or need, you can type:
;         IDL> lriscat
;      This will print out a catalog table listing all relevant LRIS settings
;      for each file in the current directory.  You can specify particular 
;      files (including extracted spectra or intermediate-reduction files)
;      by specifying the file wildcard string, e.g., 'spredux1d/ls*.fits'.
;      A python version also exists that can be run outside IDL: download it
;      at http://www.astro.caltech.edu/~dperley/programs/lriscat.py
;
;   To re-create output/diagnostic plots produced by the pipeline, run e.g.:
;         IDL> lrisplotall           ; final plots of reduced spectra
;         IDL> lrisplottraces        ; 2D spectra showing extraction apertures
;         IDL> lrisplotprofiles      ; profile plots
;
;   To change trace extraction or background settings for specific objects:
;         IDL> lrisapertures
;
;   To inspect and validate final output spectra:
;         IDL> lrisvalidate
;
;   Some steps can be run manually on single images, e.g.: 
;         IDL> lrisfluxcal, 'science.spec', 'fluxcalstar.spec' ; flux-calibrate
;         IDL> lrisconnect, 'red.spec', 'blue.spec'            ; R-B connect
;
;   Detailed pipeline reference information will soon be moved online.
;
; -----------------------------------------------------------------------------

