The CARMA Data Quality Analysis Script

The purpose of the quality script is to give both the observer and the project PI an idea of how good the observed data are. Note that CARMA data prior to March 21, 2007 were run through an older version of the script, which was originally designed for BIMA.

Downloading the script

Assuming you have MIRIAD installed, you can now run the script on your own machine.  You can download the script here.  In addition to the proj keyword, the following keywords are required for offline use:
  1. DATADIR: absolute pathname for directory where the miriad file resides.  Do not use a relative pathname here! Default is the current directory.
  2. SCRATCH: absolute pathname for directory where the quality output and scratch files will appear.  A relative pathname will probably work too. Default is to create a tree in the current directory.
  3. meanmed: absolute or relative pathname to the meanmed program, if it's not in your PATH.  This is a small utility written in C++ which can be downloaded here: meanmed.c, meanmed.h.   Compile it with g++.
A sample wrapper script is available.  Note that some features may be disabled for offsite use, for example the archiving of quality reports.  The script also uses the psselect utility to eliminate some pages from the Tsys plots, but will run fine if that program is unavailable.

Running the script

The script can be run from any area with the command: "quality proj=<datafile>" where <datafile> is the name of the MIRIAD data set (e.g. cx062.I20319_cs.1.mir). Other command line inputs can be used but are not required and the most useful are outlined below. They are given as keyword=value (e.g. prn=y).

For fully non-interactive operation, the keywords sources, gaincals, and passcals should give comma-separated lists of each type of object, and fluxcal should be a single flux calibrator. One object can appear on multiple lists. For example, it's usually OK to use a gain calibrator for bandpass, or vice versa, if everything has been observed in the same correlator mode. However, one should avoid choosing as a gain calibrator a source that has been observed in multiple correlator modes (e.g. "hybrid" modes). Only sources and gaincals are actually required by the script. If you don't give these keywords then the script will ask you for them.

When running on site, the data outputs from quality are put in 2 locations:

  1. The SCRATCH area, /tmp/quality/c####/obsblock.#/, contains the processed visibility data and plots; these are automatically deleted unless you hit Ctrl-C near the end of execution.
  2. The REPORTS area, /opt/sdp/quality/c####/obsblock.#/, contains the archived logfile and plots; these can be viewed on the web.

Interpreting the output

The output of quality is a single text file (*.qq.log), a collection of plots bundled in a PDF file, and a tarfile containing a MIRIAD gains dataset and a README file explaining how to apply it once you have downloaded your data from the archive.

The first part of the log is a scan list from listobs with system temperatures for all available antennas. Note that the Tsys values are averaged over all bands, so if one band is bad (for example, outside the usable part of the IF) it will drag up all the numbers. Also note the LO frequency given in the header; this tells you if it was a 1mm or 3mm project. The date in the header corresponds to the UT date at the project start.

PROJECT=c0104I.8D_115NGC4254.2.miriad  DATE=08JUN08  Q_RUN_TIME(cedarflat3.carma.pvt,v.16dec08)=Sat Dec 20 10:17:28 CST 2008
------------------------------------------------------------------------------------------
Source              UT      LST     Dur Elev       Sys Temps (K) near 112.1865 GHz (LO1)
                  hhmmss   hhmmss    min deg     1    2    3    4    5    6    7    8    9   10   11   12   13   14   15
MARS              001918.0 093346.7  5.0  70.  282  264  248  261  257  265  231  192  255  245  259  237  227  261  247
3C273             002614.5 094044.4  2.0  38.  375  349  328  346  346  351  320  255  351  354  351  331  310  351  328
NGC4254           002844.5 094314.8 10.0  49.  310  294  275  289  287  294  266  216  294  289  298  271  260  297  278
NGC4254           003949.0 095421.1  9.0  51.  309  291  273  288  285  292  263  214  290  282  296  270  255  293  277
3C274             005000.5 100434.3  2.0  49.  312  291  273  288  287  291  263  215  291  284  294  272  257  293  274
3C273             005251.5 100725.8  2.0  42.  343  322  301  319  313  323  293  235  323  318  324  299  284  322  302
NGC4254           005522.5 100957.2 10.0  53.  292  279  263  274  274  281  252  206  279  265  281  257  243  281  264
NGC4254           010627.0 102103.5  9.0  55.  289  271  254  270  267  271  244  200  270  258  271  248  235  272  257
3C274             011640.0 103118.2  2.0  53.  291  274  257  272  269  275  242  203  272  265  275  250  237  274  260
3C273             011931.5 103410.1  2.0  46.  313  296  279  297  292  298  266  219  297  294  300  274  261  298  278
NGC4254           012204.0 103643.1 10.0  58.  283  266  250  265  262  268  236  197  266  248  269  243  231  266  258
NGC4254           013309.0 104749.9  9.0  59.  281  265  248  264  260  265  236  195  262  247  265  241  227  265  246
3C274             014322.0 105804.6  2.0  58.  288  270  253  270  264  272  241  199  265  251  269  245  233  268  253
3C273             014615.0 110058.0  2.0  49.  304  289  271  288  285  291  259  212  287  279  293  266  251  290  272
NGC4254           014849.0 110332.5 10.0  62.  277  263  247  264  257  264  234  193  259  240  263  238  228  263  249
NGC4254           015953.5 111438.8  9.0  63.  278  264  248  264  259  266  235  194  260  254  265  240  228  263  260
3C274             021006.5 112453.5  2.0  61.  283  269  251  269  262  270  238  197  264  259  269  245  231  269  253
3C273             021302.0 112749.4  2.0  52.  305  288  269  290  281  288  259  212  285  276  288  264  250  288  270
NGC4254           021541.0 113028.9 10.0  65.  276  263  246  265  255  264  231  194  258  250  263  236  225  261  245
NGC4254           022646.0 114135.7  9.0  66.  275  258  242  255  254  259  230  191  252  246  258  234  223  259  249
3C274             023700.5 115151.9  2.0  64.  272  257  240  255  253  258  232  192  254  251  261  237  222  260  251
3C273             023957.0 115448.9  2.0  54.  293  276  259  274  270  276  249  206  273  261  281  258  242  281  270
3C273             025014.5 120508.0  2.0  54.  293  273  257  272  269  274  249  204  271  263  279  254  239  275  262
NGC4254           025251.0 120745.0 10.0  67.  266  251  235  248  247  253  225  186  247  239  255  231  219  253  247
NGC4254           030355.5 121851.3  9.0  67.  267  251  236  247  246  252  224  186  246  234  254  230  217  251  250
3C274             031409.5 122907.0  2.0  65.  267  253  238  251  247  256  227  188  249  238  260  233  219  254  252
3C273             031707.5 123205.5  2.0  55.  285  272  256  269  267  273  246  201  268  265  273  249  236  272  260
NGC4254           031947.0 123445.4 10.0  67.  270  252  236  249  248  253  224  187  247  238  253  229  218  252  250
NGC4254           033051.5 124551.7  9.0  66.  262  250  234  246  245  252  225  185  244  242  252  227  217  250  249
3C274             034106.5 125608.4  2.0  64.  270  255  239  253  247  256  227  188  250  244  259  230  221  257  253
3C273             034403.5 125905.9  2.0  54.  289  273  256  271  267  273  245  202  268  261  276  251  239  275  274
NGC4254           034637.5 130140.3 10.0  65.  268  253  237  250  249  254  225  187  248  238  256  229  219  253  251
NGC4254           035742.0 131246.6  9.0  64.  271  254  238  251  248  255  226  188  246  242  254  231  220  256  255
3C274             040757.0 132303.3  2.0  63.  269  255  239  253  250  257  227  188  248  237  257  232  219  256  254
3C273             041053.5 132600.3  2.0  52.  294  274  257  272  268  276  248  202  270  256  277  250  240  276  270
NGC4254           041329.0 132836.2 10.0  62.  268  254  239  251  250  255  226  188  250  235  259  229  221  255  249
NGC4254           042434.0 133943.0  9.0  61.  274  253  237  251  249  254  228  187  248  232  255  229  219  253  252
3C274             043447.5 134958.2  2.0  59.  276  255  239  252  250  256  227  189  252  237  259  231  220  257  255
3C273             043742.5 135253.7  2.0  50.  297  276  258  272  270  277  248  203  270  264  282  252  242  276  276
NGC4254           044012.0 135523.6 10.0  59.  276  258  241  255  251  258  231  191  253  239  260  233  223  258  255
NGC4254           045116.0 140629.4  9.0  57.  280  259  243  256  255  260  232  192  256  240  261  236  223  259  258
3C274             050131.0 141646.1  2.0  56.  282  262  245  258  259  263  234  194  259  245  263  238  228  263  261
3C273             050424.0 141939.6  2.0  47.  310  285  268  282  280  286  257  210  280  275  290  264  250  286  285
3C273             051419.5 142936.7  2.0  45.  312  290  272  287  286  293  263  214  287  284  294  268  254  290  289
NGC4254           051647.5 143205.1 10.0  53.  291  267  250  264  262  269  241  198  263  254  270  245  233  267  268
NGC4254           052752.0 144311.4  9.0  51.  294  270  254  267  265  273  242  200  268  257  273  249  234  272  268
3C274             053803.5 145324.6  2.0  50.  295  272  256  270  270  275  246  202  270  260  277  251  237  273  271
3C273             054056.0 145617.6  2.0  41.  323  298  280  296  291  300  273  220  297  289  299  279  262  299  296
NGC4254           054324.0 145846.0 10.0  48.  301  275  258  271  268  277  246  205  274  271  278  256 8682  275  277
NGC4254           055428.0 150951.8  9.0  46.  307  282  264  279  276  283  257  209  278  268  286  262  244  282  281
3C274             060440.0 152005.5  2.0  45.  310  285  267  281  279  287  258  210  281  277  289  264  247  285  274
3C273             060730.5 152256.4  2.0  37.  345  321  302  318  316  323  295  237  321  316  325  305  285  322  314
NGC4254           061004.5 152530.9 10.0  43.  313  292  273  288  287  293  267  216  291  278  296  273  255  293  287
NGC4254           062109.5 153637.7  5.0  41.  326  301  282  297  292  302  275  222  301  290  305  283  264  301  293
3C273             062722.5 154251.7  2.0  33.  366  343  321  340  331  344  314  252  346  341  345  328  307  344  330

Then you get some information about the purpose of each source (as recorded in the MIRIAD file), and the roles that the person running quality actually assigned to each source (usually, but not always, the same). Also reported are the number of NOISE integrations (since these are discarded from the listobs output for clarity, but you should make sure there are some) and the chosen reference antenna for selfcal. Then, the csflag task is run to flag shadowed data; the number of flagged visibilities is reported.

Source MARS has purpose BF
Source 3C273 has purpose G
Source NGC4254 has purpose S
Source 3C274 has purpose O

sources : NGC4254,3C274
gaincals: 3C273
passcals: 3C273
fluxcal: MARS
There were 18 NOISE integrations
Using 9 as the reference antenna
csflag: Processed  191520 records, flagged  0 O/H/C:  0 0 0

A summary of project times and correlator setups is then given. The total project time is basically the time elapsed during the project (actually an underestimate, since initial setup and tuning occurs before the MIRIAD file starts being written). The total observe time is the total time spent integrating on the sources listed. Note that if a source has purpose 'O' and is not included as a source or calibrator, it is not included in the total observe time. Also, if the frequency setup changes during the observation, this will not be reflected in the correlator setup listing.

Total project time 6.2 hrs from start to finish
Total observe time 0.08 hrs for MARS
Total observe time 0.53 hrs for 3C273
Total observe time 0.40 hrs for 3C274
Total observe time 4.05 hrs for NGC4254
Total observe time 5.06 hrs for entire track

Correlator setup for gain calibrators:
      1     110.170   -0.468750  GHz
      2     110.062   -0.061523  GHz
      3     110.010   -0.061523  GHz
      4     114.203    0.468750  GHz
      5     114.311    0.061523  GHz
      6     114.363    0.061523  GHz
Correlator setup for sources:
      1     110.170   -0.468750  GHz
      2     110.062   -0.061523  GHz
      3     110.010   -0.061523  GHz
      4     114.203    0.468750  GHz
      5     114.311    0.061523  GHz
      6     114.363    0.061523  GHz
Using window 1, BW 468 MHz for gain calibration

Next come decorrelation estimates for the main calibrators. These are based on comparing the scalar and vector averaged amplitudes after a phase-only selfcal with an interval of 5 minutes. The interval length can be changed to another value using the solint parameter on the command line. Coherence values close to 1 indicate excellent coherence; small values suggest poor phase stability or just a weak calibrator. To help decide between these, the vector averaged amplitude is also printed (note these are in raw units, not scaled using the flux calibrator, but should be within a factor of 1.5 or so of Jy units). Values for planets or other resolved sources should be treated with caution.

****************************************************
decorrelation estimated from global scalar to vector
average ratio using a selfcal interval of 5 mins.

  source       vec_avg    phase_coherence
----------     -------    ---------------
     3C273      15.89            0.92
      MARS      18.89            0.89

Next, for both the flux calibrator (if available) and phase calibrator, the time-averaged amplitude gains after selfcal on a 5-minute interval are listed. These should be "absolute" in the sense that they have been scaled to give the expected flux in Jy from MIRIAD's flux table (or planet models, if appropriate). If they are close to 1, the system default values of Jy/K are probably pretty good. Large values indicate bad pointing or decorrelation within the 5-minute interval. Beware that fluxes are not always known for calibrators at both 1mm and 3mm, which might cause selfcal to assume a flux of 1 Jy.

Calibrated antenna gains from MARS (soln interval 5 min)
 0 00:21:48       0.959      1.405      0.943      0.992      0.974      0.965
                  1.007      1.227      1.094      1.022      1.098      1.035
                  0.994      0.936      1.072

Calibrated antenna gains from 3C273 (soln interval 5 min)
 0 03:33:02       1.028      1.576      0.938      1.045      1.071      1.031
                  1.040      1.170      1.065      1.054      1.076      1.085
                  1.002      0.964      1.029

Then the theoretical rms in Jy is given for each spectral window of the source data. These are based on the system temperatures and antenna gains, and should be reasonable estimates for the center of the map. They are used to set the contour levels for imaging the source(s). If the "hybrid" correlator mode was used, you should see an additional message here about the bandwidth of the gains which were used.

Passband gains copied to NGC4254
Window 1 has theoretical rms 1.3e-03 Jy at map center
Window 2 has theoretical rms 3.5e-03 Jy at map center
Window 3 has theoretical rms 3.6e-03 Jy at map center
Window 4 has theoretical rms 1.5e-03 Jy at map center
Window 5 has theoretical rms 4.2e-03 Jy at map center
Window 6 has theoretical rms 4.2e-03 Jy at map center

Finally the script reads several variables in the MIRIAD dataset that indicate observing conditions, and calculates their statistics. "rmspath" is the RMS path in microns on a 100m baseline measured at an elevation close to 45 degrees over a period of 10 minutes. "precipmm" is the precipitable water vapor (PWV), based on humidity, temperature and pressure. It is probably not as reliable an indicator of sky opacity as doing skydips. "tau230" is an estimate of the opacity at 230 GHz, based on skydips performed every 10 minutes by the tipper. The script converts it to a nominal PWV value using a linear scaling based on the MIRIAD task obstau.

Track statistics for rmspath:
  Npts       Median        Mean         Rms          Min          Max
   690       174.24       178.09        44.62       103.79       278.99
 
Track statistics for precipmm:
  Npts       Median        Mean         Rms          Min          Max
   690         5.80         5.89         0.33         5.24         6.76
 
Track statistics for tau230:
  Npts       Median        Mean         Rms          Min          Max
   690         0.62         0.62         0.06         0.52         0.70

The script then tries to assign a grade to the track.  This is hardly a perfect science, but it tries to take into account both an "effective" opacity due to phase noise and the "actual" opacity due to atmospheric absorption.  Of course it's not that simple (there's atmospheric emission for one thing) but it should be indicative.  The phase opacity is calculated for the maximum baseline length and assumes Kolmogorov-like scaling.  The atmospheric opacity is determined from the median PWV (derived from "tau230" not "precipmm") using MIRIAD's obstau task and should be appropriate to the observing band (112 GHz is assumed for all 3mm projects, and 230 GHz for all 1mm projects).  Thus, a 1mm project will usually get a lower grade than a 3mm project under the same atmospheric conditions - this is intentional, and is a change from the old grading system.

Using median values and a nominal elevation of 45 deg
The maximum baseline length for this track is 362.66 m
Effective opacity at 112 GHz due to phase noise: 0.24

Atmospheric opacity based on tau230 in the datafile
Estimating a median precipmm value of 10.25
Opacity at 112 GHz due to atmospheric absorption: 0.33
 
Total opacity: 0.57
COMPOSITE SCORE = 100-(25*tau) = 86
COMBINED GRADE: B 
 
This is based on weather conditions only.
The plots produced by quality are now provided in a single PDF file. The plots are described individually below; click on a figure to see it in full size.
  1. Weather conditions - 3 pages of output, plotting the weather-related variables rmspath, precipmm, and tau230.  These come from the seeing monitor, weather station, and tipper repectively.


  2. Systemps - 2 pages of output, giving the system temperatures vs. time for all 15 antennas in the first spectral window. Each source is coded with a different color. The y-axis range is autoscaled in the first plot, and has fixed scaling from 0 to 1000 K in the second plot.


  3. Line-length correction - plot of linelength phases. These should be tight and track smoothly with time.


  4. Phase vs. uv-distance - single plot (all baselines together) of visibility phase vs. baseline length. The phases have been selfcal'ed with a 5-minute solution interval (can be changed with the solint parameter) to bring them to 0; nonetheless, they will tend to decorrelate on longer baselines due to atmospheric phase fluctuations within that time, so more scatter on the right of the plot is expected.


  5. Antenna-based gains - plots of gains vs. time, derived from the gain calibrators, with a 6m antenna as the reference. A 5 minute solution interval (can be set with the ampgainsolint parameter) is used, so there is generally one point per calibrator observation. A third plot shows the phase gains with a 10m antenna as the reference (the amplitude gains are generally the same). The reference antennas can be changed by setting the refant and ovrorefant parameters in the script.


  6. Bandpass gains - plot of bandpass gains vs. channel, using by default C9 as reference. Windows are coded by color, and narrower bandwidth windows generally have more channels and noisier gains. Overall the amplitude gains should be close to 1, with a smooth dropoff at the window edges. The phase gains will generally show offsets between windows, but should have little scatter within a window; the LSB and USB windows should look more or less like mirror images. NOTE: You may see more than the expected number of windows in hybrid mode, or when Doppler tracking has shifted the frequency of the first channel of a window by more than half a channel.


  7. Flux calibrator - plot of visibility amplitude vs. uv-distance, averaged in bins of equal uv-distance, for the flux calibrator. The data have been self-calibrated using the default model in selfcal (a point source for quasars, uniform disk for planets). For a source which is resolved, or suffers from phase decorrelation, the amplitude will drop on longer baselines. For quasars observed in good conditions the amplitude should be constant at a value close to the adopted flux (from the flux table). The gains from the calibration are time-averaged and reported in the log.


  8. Source maps - Images of source data, with bandpass and time-dependent gains applied, but no absolute flux calibration (other than assuming an a priori flux for the gain calibrator). Images are made for each spectral window, averaging all channels but excluding two at the edge. If there are several sources, each is imaged independently. The noise in each image is estimated for the map center using the theoretical rms calculated by mossen. Maps have been normalized by a sensitivity image from mossen in order to avoid noise amplification at the edges; they are strictly signal-to-noise images. A 4-sigma contour is drawn. These maps may or may not show signal (dependent on the strength of the emission, and whether it is line or continuum) but are useful to check the theoretical noise estimates and the overall quality of the source data and/or gain solutions.

Additional plots may be produced in the future as the system develops.

 

Tony Wong, 3 May 2009 (original: Douglas Friedel)