Index

Accessing the archive from python

This page documents a module that can be used in python scripts to read archived or live time-streams of telescope and receiver registers. The python module is called readarc.

Telling python where to find the readarc module

If the readarc module has been installed in the system directory that python looks in for standard modules, then you can skip this section. If the module has been installed in a non-standard directory, then before python is run, it is necessary to tell python where to find the module. This is done using the PYTHONPATH environment variable. For example, if the module were installed as /scr/mcs/40m/lib/python/readarc, then this environment variable would need to be set as follows:

Under csh or tcsh:

  setenv PYTHONPATH /scr/glast/ovro40m/lib/python

Under sh or bash:

  export PYTHONPATH="/scr/glast/ovro40m/lib/python"

For convenience, the appropriate one of the above lines should be placed in your shell startup file, such as .cshrc or .bashrc. Note that if the PYTHONPATH environment variable is already set to something, then just append a colon, followed by the location of the readarc module, to the existing value of the variable. Another way to tell python where to find the readarc module, is to add its path within the python script before importing it. This can be done as follows:

  import sys
  sys.path.append("/scr/glast/ovro40m/lib/python")
  import readarc

Loading the `readarc` module

The readarc module depends on the NumPy numeric python module. So it is necessary to also load this. The following two python statements do this.

 import numpy
 import readarc

Opening a stream of archive files.

To read registers from the on-disk archive files, the open_file_stream method of readarc must be used.

  ms = readarc.open_file_stream(start=55362.5, end=56000.0,
                                arc_dir="/scr/glast/ovro40m/arc",
                                cal_dir="/scr/glast/ovro40m/cal")

Note that all of the arguments of this function are optional, although the arc_dir and cal_dir arguments are only optional if their defaults are provided via environment variables. Their meanings of the arguments are as follows:

start The earliest date and time of interest, expressed as a Modified Julian Date (MJD). The first frame of data that the stream will subsequently return, will be the oldest frame in the archive whose timestamp matches or follows this MJD.
If this argument is omitted, then the stream will start with the oldest frame in the archive.

end The latest date and time of interest, expressed as a Modified Julian Date (MJD). The last frame of data that the stream will subsequently return, will be the newest frame in the archive whose timestamp matches or precedes this MJD.
If this argument is omitted, then the stream will end at the newest frame in the archive.

arc_dir The directory where the archive files are stored. If this argument is omitted, then the method looks for an enviroment variable called TCS_ARC_DIR and uses its value instead. If this variable is also not found, then an exception is raised.
So for convenience it is recommended that you set the TCS_ARC_DIR environment variable in your shell startup script. For example:

Under csh or tcsh:

setenv TCS_ARC_DIR /scr/glast/ovro40m/arc

Under sh or bash:

export TCS_ARC_DIR="/scr/glast/ovro40m/arc"

cal_dir The directory where the register calibration files are stored. Note that these are files that tell the archive library how to convert from the integer values in the archive files, to floating point values with appropriate units, such as degrees for angles etc... The calibration file for the mount registers is called mount.cal, and the calibration file for the kuband receiver registers, is called kuband.cal. There are text files that you can examine to see what units a given register is given by them.
If the cal_dir argument is omitted, then the method looks for an enviroment variable called TCS_CAL_DIR and uses its value instead. If this variable is also not found, then an exception is raised.
So for convenience it is recommended that you set the TCS_CAL_DIR environment variable in your shell startup script. For example:

Under csh or tcsh:

setenv TCS_CAL_DIR /scr/glast/ovro40m/cal

Under sh or bash:

export TCS_CAL_DIR="/scr/glast/ovro40m/cal"

The return value of the readarc.open_file_stream() method is an opaque handle to the opened stream, and must be passed to the other methods that read the stream. In principle, you can have multiple streams open, each identified by such a handle.

Opening a live stream to the control program.

To read frames of registers in real-time from the control program, it is necessary to open a network stream to the control program, using the readarc.open_live_stream() method.

  ms = readarc.open_live_stream("obs40m", cal_dir="/scr/glast/ovro40m/cal")

The first argument is mandatory, and specifies the name or IP address of the computer where the control program is running. For the 40m telescope, this computer is called "obs40m", or "obs40m.cm.pvt" from the main OVRO network. The cal_dir has the same meaning as described for the open_file_stream method above, and the return value is a handle to the opened stream.

Selecting the registers that are to be read.

After opening a register stream to either the control program or the disk-based archive, as described in the previous two sections, you must tell the opened stream which registers you wish to read from this stream. This is done by invoking the readarc.add_reg() method, once for each register. This method is used as follows.

  register_handle = readarc.add_reg(ms, register_name)

The first argument of the method is the stream handle, as returned by the preceding call to either readarc.open_file_stream() or readarc.open_live_stream(). The second is the full name of the register that you wish to subsequently read, and the return value is the opaque handle that you must later pass to one of the register reading functions, when you wish to read the latest value of this register. At the time of writing, the lists of available archive registers can be found at:

The following is an example of using the readarc.add_reg() method, to select 3 registers for later reading.

# Select the two element UTC register that timestamps each frame.
# These two elements contain the date as a MJD day number, and the
# time of day in milliseonds.

utc_reg = readarc.add_reg(ms, "mount.frame.utc")

# Also select the register that indicates how many 1ms kuband receiver
# samples were recorded in the current frame.

nsample_reg = readarc.add_reg(ms, "kuband.rx.nsample")

# Select the registers that contain the 1ms receiver samples of the
# latest second.

samples_reg = readarc.add_reg(ms, "kuband.rx.samples")

Preparing to read array registers.

Registers can either be read one element at a time, using standard Python scalar datatypes, or they can be read into Numeric Python arrays. When reading into an array, the caller of the reading method must provide the array. This allows the same array to be reused for each frame of the archive, instead of the reading function having to allocate potentially very large arrays for each frame. The following example shows how one might allocate the array that would be used to read the radiometer samples register, which was selected for reading in the example of the previous section. This register can hold up to 1001 samples. So we tell the numeric Python numpy.zeros() method to allocate an array of this many double precision elements (known as float in python), and assign zero to each element.

 samples_array = numpy.zeros(1001, dtype=float) # ADC counts.

Reading register frames, and the values of their registers

The archive stream consists of a sequence of frames of registers. Each frame contains the values of all registers during a given one second time-span. Reading archive registers thus consists of reading one frame of registers at a time from the stream, then extracting the values of the previously selected registers from this frame. The job of reading the next frame of registers, is performed by the readarc.next_frame() method. This takes the handle of a stream returned by either either readarc.open_file_stream() or readarc.open_live_stream(), and returns 1 each time that it manages to read the next frame from that stream, or 0 once it reaches the end of the stream. The following example shows how this is used, and how each time that it is called, register values can be acquired from the returned frame.

# Read one frame at a time until the stream ends.

while readarc.next_frame(ms):

    # Extract the MJD UTC of the frame.

    utc = readarc.get_scalar_float(utc_reg, 0) + \
          readarc.get_scalar_float(utc_reg, 1) / 86400000.0 # ms -> days

    # Determine the number of radiometer samples that are archived in
    # this frame.

    nsample = readarc.get_scalar_int(nsample_reg, 0)

    print "At UTC %g, there are %d radiometer samples." % (utc, nsample)

    # Extract the nsample 1ms sample values into the samples_array that
    # was allocated in the example of the previous section.

    readarc.get_array(samples_reg, samples_array)

    i = 0
    while i < nsample:
      print " Sample %d has value %g" % (i,samples_array[i])
      i = i+1

# End of the loop over the frames.

Extracting register values

The of the preceding section shows examples of a subset of the methods that can be used to extract the values of the selected registers, each time that a new frame is read by the readarc.next_frame() method. This section documents these and the other extraction functions. To understand how these are used, note that all registers in the archive are stored as numbers, even including registers that hold string values. The methods that are called upon to extract these values, convert the numbers in the registers to the data type associated with the chosen method. So, for example, readarc.get_scalar_float(), returns the specified register value in a python float variable, whereas readarc.get_scalar_int() would return the same value as a Python integer, after truncating the original floating point value to an integer.

`readarc.get_scalar_int(register_handle [,index=0])`	This returns the value of a specified single register element, as an integer. The register is specified via its register handle, as previously returned by `readarc.add_reg()`. By default, the first element (element 0), of the register is returned. To specify a different element, indicate the index of this element using the optional index argument.
`readarc.get_scalar_float(register_handle [,index=0])`	This returns the value of a specified single register element, as a floating point number. The register is specified via its register handle, as previously returned by `readarc.add_reg()`. By default, the first element (element 0), of the register is returned. To specify a different element, indicate the index of this element using the optional index argument.
`readarc.get_string(register_handle)`	This returns the value of a specified single register element, as an ASCII string. The register is specified via its register handle, as previously returned by `readarc.add_reg()`. Note that this method assumes that the individual bytes of the register contain ASCII character codes. This is the case for registers like the `mount.tracker.source` register, which contains source names. If `readarc.get_string()` is applied to a register whose purpose is to record numbers, the returned string will likely be an empty string, since the most significant byte of most registers is zero, which is a string terminator. At worst it will be a string of random printable ASCII characters, since any unprintable ASCII characters found in a register are replaced by spaces.
`readarc.get_array(register_handle, array)`	This returns the values of all of the elements of a specified register, in a specified numeric Python array. The register is specified via its register handle, as previously returned by `readarc.add_reg()`. The second argument must be the numeric python array in which the register values should be stored. If the register contains more elements than can be stored in the specified array, then only the number that will fit in the array, from the corresponding initial elements of the register array, are returned. Alternatively, if the register contains fewer elements than can be stored in the specified array, then the extra elements at the end of the specified array are left unchanged.

Miscellaneous methods

The following methods may be useful in some cases.

`readarc.rewind_stream(ms)`	If the sole argument is a stream opened by `readarc.open_file_stream()`, then after this call, the next call to `readarc.next_frame()`, will return to reading the first frame of the originally specified time range.
`readarc.close_stream(ms)`	This method can be used to reclaim all resources taken by the specified stream, including closing archive files etc. Any further method calls that take `ms` as an argument, will raise an exception. Note that the resources of a stream will also be reclaimed when all references to the stream, and to register selections that refer to it, are no longer referenced by any python code.

A more complete example

#!/usr/bin/env python

# Import the numeric Python module and the archive reading module.

import numpy
import readarc

# Create an iterator that will read through all archived data between two
# MJD date/time limits.

ms = readarc.open_file_stream(start=55362.0, end=55362.00003,
                              arc_dir="/scr/glast/ovro40m/arc",
                              cal_dir="/scr/glast/ovro40m/cal")

# Select the two element UTC register that timestamps each frame.
# These two elements contain the date as a MJD day number, and the
# time of day in milliseonds.

mjd_reg = readarc.add_reg(ms, "mount.frame.utc")

# Also select the register that tells you how many 1ms samples were recorded
# in the current frame.

nsample_reg = readarc.add_reg(ms, "kuband.rx.nsample")

# Select the registers that contain the samples, and the dicke-switch
# off/on states (0/1).

samples_reg = readarc.add_reg(ms, "kuband.rx.samples")
states_reg = readarc.add_reg(ms, "kuband.rx.states")

# Select the source-name register.

source_reg = readarc.add_reg(ms, "mount.tracker.source")

# Select the register that contains the string label that procedures
# use to indicate what they are doing (eg. it might contain "flux:A").

label_reg = readarc.add_reg(ms, "mount.frame.label")

# Select the register that indicates the tracking status. This will have
# the value 3 when the telescope has acquired a source. The other numbers
# are listed in the online register documentation.

track_status_reg = readarc.add_reg(ms, "mount.tracker.state")

# Select the register that indicates the tracking mode. This will have
# the value 1 when the telescope is attempting to track a source. The
# other numbers are listed in the online register documentation.

track_mode_reg = readarc.add_reg(ms, "mount.tracker.track_mode")

# Create a couple of arrays of 1001 samples each. The radiometer samples
# and dicke-switch states will be read into these arrays. Note that 1001
# elements are used, because if the radiometer readout clock is running a
# bit faster than 1KHz, there could be an extra sample in some 1-second
# frames.

sample_counts = numpy.zeros(1001, dtype=int) # Integer ADC counts.
sample_states = numpy.zeros(1001, dtype=int) # Dicke-switch states (0=off,1=on)

# Read one frame at a time until the stream ends.

while readarc.next_frame(ms):

  # Extract the tracking status and tracking mode. Note that the
  # second argument of get_scalar_int() is the index of the element of
  # the specified register. For registers with only one element, this
  # should always be 0.

  tracking_status = readarc.get_scalar_int(track_status_reg, 0)
  tracking_mode = readarc.get_scalar_int(track_mode_reg, 0)

  # Only read the rest of the registers of this frame if the telescope
  # has acquired its current target and that target is a source (rather
  # than say stow).

  if tracking_status == 3 or tracking_mode == 1:

    # Get the name of the current source

    source = readarc.get_string(source_reg)

    # Extract the MJD of the frame. Note that the radiometer data will
    # have been taken during the previous second.

    mjd = readarc.get_scalar_float(mjd_reg, 0) + \
          readarc.get_scalar_float(mjd_reg, 1) / 86400000.0 # ms -> days

    # Determine the number of radiometer samples that were archived in
    # this frame.

    nsample = readarc.get_scalar_int(nsample_reg, 0)

    # Extract the arrays of nsample 1ms sample counts and dicke-switch states.

    readarc.get_array(samples_reg, sample_counts)
    readarc.get_array(states_reg, sample_states)

    # Get the label string that indicates what the procedure is doing.

    label = readarc.get_string(label_reg)

    # Process the samples according to what the procedure was doing.

    print "At MJD=%.15g, read a frame of %d samples from source %s (procedure=%s)" % (mjd, nsample, source, label) 

    # Print the samples.

    i = 0
    while i < nsample:
      if sample_states[i] & 1:
        print "Sample %d is REF value %d" % (i,sample_counts[i])
      else:
        print "Sample %d is ANT value %d" % (i,sample_counts[i])
      i = i+1

# End of the loop over the frames.

Martin Shepherd (26-Jun-2010)

`start`	The earliest date and time of interest, expressed as a Modified Julian Date (MJD). The first frame of data that the stream will subsequently return, will be the oldest frame in the archive whose timestamp matches or follows this MJD. If this argument is omitted, then the stream will start with the oldest frame in the archive.
`end`	The latest date and time of interest, expressed as a Modified Julian Date (MJD). The last frame of data that the stream will subsequently return, will be the newest frame in the archive whose timestamp matches or precedes this MJD. If this argument is omitted, then the stream will end at the newest frame in the archive.
`arc_dir`	The directory where the archive files are stored. If this argument is omitted, then the method looks for an enviroment variable called `TCS_ARC_DIR` and uses its value instead. If this variable is also not found, then an exception is raised. So for convenience it is recommended that you set the `TCS_ARC_DIR` environment variable in your shell startup script. For example: Under csh or tcsh: setenv TCS_ARC_DIR /scr/glast/ovro40m/arc Under sh or bash: export TCS_ARC_DIR="/scr/glast/ovro40m/arc"
`cal_dir`	The directory where the register calibration files are stored. Note that these are files that tell the archive library how to convert from the integer values in the archive files, to floating point values with appropriate units, such as degrees for angles etc... The calibration file for the mount registers is called `mount.cal`, and the calibration file for the kuband receiver registers, is called `kuband.cal`. There are text files that you can examine to see what units a given register is given by them. If the `cal_dir` argument is omitted, then the method looks for an enviroment variable called `TCS_CAL_DIR` and uses its value instead. If this variable is also not found, then an exception is raised. So for convenience it is recommended that you set the `TCS_CAL_DIR` environment variable in your shell startup script. For example: Under csh or tcsh: setenv TCS_CAL_DIR /scr/glast/ovro40m/cal Under sh or bash: export TCS_CAL_DIR="/scr/glast/ovro40m/cal"