P-AIRCARS documentation¶

This is the official documentation of the P-AIRCARS. Source code can be found at : https://github.com/devojyoti96/P-AIRCARS.git

Introduction¶

P-AIRCARS (Polarimetry using Automated Imaging Routine for Compact Arrays of the Radio Sun)¶

P-AIRCARS is an automated calibration and imaging routine for polarization calibration and imaging of solar observations done with Murchison Widefield Array (MWA) https://www.mwatelescope.org. P-AIRCARS has been developed and is maintained by solar physics group at the National Centre for Radio Astrophysics, Tata Institute of Fundamental Research (NCRA-TIFR), Pune, India https://www.ncra.tifr.res.in.

P-AIRCARS uses CASA (Common Astronomy Software Application) https://casa.nrao.edu for intensity and bandpass self-calibration and polarisation calibration is performed by our own implementation of the full Jones calibration algorithm described by Mitchell et al. 2008 https://doi.org/10.1109/JSTSP.2008.2005327.

Imaging at P-AIRCARS is performed by WSClean https://wsclean.readthedocs.io. If WSClean can not be not installed, P-AIRCARS performs imaging by CASA. When CASA is used for imaging the computation speed is slow.

Basic philosophy of P-AIRCARS is self-calibration and use the instrumental model to perform the precise calibration. Details of the algorithm and implementation can be found in the follwoing papers

1.AIRCARS (Mondal et al. 2019) https://doi.org/10.3847/1538-4357/ab0a01

2.Kansabanik et al. 2021a, in preparation

3.Kansabanik et al. 2021b, in preparation

If you use P-AIRCARS for any publication or scientific work, please cite these three papers.

P-AIRCARS has several modules and scripts. Modules can be used for self-calibration of astronomical observations also. Users who want to use the modules please read the documentation of the module details.

Instructions for installing P-AIRCARS in local machines, laptops and work stations can be found in the installation section. Implementation for high performance computing (HPC) environment is in development and will be available shortly. A basic tutorial is also included for new users.

For any queries and issues reach us at or raise an issue in github:

dkansabanik@ncra.tifr.res.in, paircarsnotification@gmail.com

System requirements¶

Minimum hardware requirements¶

CPUs: 8 cores

RAM : 8 GB

Installation space : 700 MB

Disk space : 6 times of the data volume

Workstation configuration¶

P-AIRCARS has been tested on two types of workstations with following configurations.

Intel Xeon Gold 6138 2 GHz 40 CPUs - 256 GB RAM

4 x 10TB (RAID-0) 40TB , 2 sets of 3 x 10 TB (RAID-5) , 2 x 20 TB
Boston Intel E5-2699 v3 2.30GHz, 72 CPUs

256 GB (8 x 32) RAM - 2 x 300 GB SSD System

Data 41 TB (12 x 4 TB SATA with RAID-5, 256KB stripaize)

Data1 81 TB (12 x 8 TB SATA with RAID-5, 512KB stripaize)

960 GFLOPS

Software environment¶

P-AIRACRS has been tested in the follwoing linux environments

CentOS 7
Ubuntu 20.0.4

Testing in other linux environments is in progress.

Details of other software requirements is available in installation section.

Installation¶

P-AIRCARS is a self-contained python package. It is build on python3. Python version greater than 3.6 is required for P-AIRACRS. It is recommended to use python virtual environment to install P-AIRCARS, but it is not a necessary requirement. Installation steps are as follows.

Creating virtual environment¶

Install python virtual environment:

>> python3 -m pip install –upgrade pip

>> python3 -m pip install virtualenv

Activate virtual environment:

>> python3 -m venv -system-site-packages {path_to_virtualenv}/paircars

>> source {path_to_virtualenv}/paircars/bin/activate

Check the virtual environment path:

>> which python3

It should show “{path_to_virtualenv}/paircars/bin/python3”

Obtaining P-AIRCARS¶

P-AIRCARS source code can be downloaded from https://github.com/devojyoti96/P-AIRCARS.git.

P-AIRCARS is not public now. It will be public very soon. If you want to use P-AIRCARS before it becomes public, please reach us at dkansabanik@ncra.tifr.res.in, paircarsnotification@gmail.com

Installing P-AIRCARS¶

Move to the P-AIRCARS directory.

Run setup.py

>> python3 setup.py install

That’s all. It will automatically install all required packages and libraries. This installation process is in very early stage. If you find any issues during installation, please reach us at dkansabanik@ncra.tifr.res.in, paircarsnotification@gmail.com.

Note

If you installed P-AIRCARS in a virtual environment, please make sure you are inside the virtual environment. P-AIRCARS can not be accessed outside the virtual environment in this case.

>> which python3

It should show “{path_to_virtualenv}/paircars/bin/python3”. If you are not in virtual environment, please activate it.

Tutorial¶

P-AIRCARS is a user-friendly and robust pipeline. In this tutorial we will learn how to run P-AIRCARS on a set of solar observations from MWA in default mode.

Start the P-AIRCARS¶

Open your terminal
Type ‘go-paircars’ and press enter

>> go-paircars
If user want to use P-AIRCARS to download data from MWA ASVO, follow the steps below:
If user does not have MWA ASVO accunt :

Go to https://asvo.mwatelescope.org/

Click on REGISTER button at top-right corner.

Fill up the details and submit.
1. It may take some days to accept the registration form by MWA ASVO administration.
  After the registration is done:
  
  Go to https://asvo.mwatelescope.org/
  
  Click on LOGIN button at top-right corner.
  
  Give your username and password and click on Login.
  
  After login, click on your name on the top right corner and go to MY PROFILE.
  
  In that page user can find the API key
  
  Copy the MWA ASVO API key
Type ‘go-paircars–ASVO_API_key={API_key_of_user}’ and press enter.

>>go-paircars --ASVO_API_key={API_key_of_user}

User only need to supply the MWA ASVO key once and it will be saved in the P-AIRCARS cache directory. If user delete the cache, user needs to supply it again.

P-AIRCARS input window will pop up.

Download the data¶

Click Download Data on P-AIRCARS user interface.

A popup window will open.

Fill up the details and click on Start download. Data jobs will be submitted and waiting for download. A new popup window will show the progress of the data download. Data will be saved in the Data Directory.

Fill up the inputs¶

Manual input¶

Two inputs are mandatory

Data directory : Name of the directory where all measurement setsto be imaged are present. User can type the full directory path in the entry box or use the browse button to select the directory.
Base directory : Name of the base directory where all the calibration and imaging will be done. User can type the full directory path in the entry box or use the browse button to select the directory.

Load from previous input file¶

If a previous P-AIRCARS input file with ‘.paircars’ exists, inputs can be filled automatically from that input file. To do that follow the steps

Press the load button at the right of the Input file at the bottom of the INPUTS block and select the input file.
Select the file and click open and it will automatically load the input file and fill the inputs.

Validating inputs¶

After filling the inputs either manually or from a previous input file, user can validate if all inputs are correct or not.

Click Validate Inputs button at the bottom of the INPUTS.
If inputs are correct a pop up message will be shown saying, “Inputs are correct”.

3.If mandatory inputs are missing it will show “Mandatory inputs are missing”.: Check if all mandatory inputs are given or not.

Save inputs¶

If user wants P-AIRCARS inputs can be saved in a ‘.paircars’ file.

Click Save Inputs button at the bottom.
A file dialog box will pop up. Default directory is your present derectory and default file name is ‘inputs.paircars’. User may change the directory and file name.
Click on save to the inputs.
If the input file saved successfully a pop up message will show the location where the input file saved. Otherwise it will show error.

Run the P-AIRCARS¶

After giving all the mandatory inputs, we are now ready to start the P-AIRCARS.

Click on Run P-AIRCARS button at the bottom.
If P-AIRCARS is already running or interupted at the same base directory, a pop up message will ask ‘P-AIRCARS already running in the base directory. Do you really want to over run?’
If you want to re-run P-AIRCARS in the same base directory overriding the present running P-AIRCARS, press yes , otherwise press no.
If you press yes, P-AIRCARS will start. It may take some time to start. When started a pop up message will show ‘P-AIRCARS has started. Job ID : your_job_id’.
Note down the Job ID. You can use this Job ID to track the progress of the P-AIRCARS.
If you have started P-AIRCARS, it will start after pressing Run P-AIRCARS button and pop message will show the Job ID.

Notification over e-mail¶

User can get the updates about the progress of p-AIRCARS in their e-mail. To enable this feature follow the steps

Select the Notification button.
Put the e-mail id of the user in the e-mail entry box.

Locate the final images¶

If you give some custom path to save final images in the Image Directory, final images will be saved there. Otherwise, final images will be saved in a sub-directory named ‘final_images’ inside the Base Directory.

Details of Inputs¶

In this section the details of the P-AIRCARS inputs are given.

Inputs¶

Data Directory : Name of the directory where all measurement sets to be imaged are present. If user want to run in offline mode, please be sure that all metafits files correcponding to all measurement sets are also present in the Data Directory. This is a mandatory input.
Base Directory : Name of the directory where all calibration and imaging analysis will be done. Either give an existing directory, otherwise P-AIRCARS will create the Base Directory in the given path. This is a mandatory input.
Image Directory : Name of the directory to save final images. If no input is given final images will be saved in a sub-directory named ‘final_images’ inside the Base Directory.
Time range : Time ranges to be imaged. Time range format is hh0:mm0:ss0.ff0~hh1:mm1:ss1.ff1,hh2:mm2:ss2.ff2~hh3:mm3:ss3.ff3,…
Channel range : Channel ranges to be imaged. Channel range format is ch0~ch1,ch2~ch3,….
Caltable Names : Prior caltable names obtained from calibrator observations. Give the full paths separated by commas.
Robustness : Determine the robustness of the calibration solutions and sef-calibration convergence. It takes three values 0, 1, 2 with increasing robustness. Default value is 1.
Quality : Determine the image quality. It takes three values 0, 1, 2 with increasing quality of the images. Default value is 1.
Verbose : If selected, verbose output logs will be saved. Also all the images, measurement sets and calibration tables are also saved. In verbose mode amount of disk space requirements will be more. Turn on only for debuggung purposes. Default it is turned off.
Save Logs : If selected, logs for self-calibration and imaging rounds will be saved. Default it is turned off.
Interactive : By default it is turned off. If selected, user can do interactive cleaning and also can change inputs manually during runtime. Only useful for specific debugging.
Decorrelation Correction : If selected, perform the amplitude decorrelatation correction. By default it is switched on. Details of the correction can be found in Mondal et al. 2020 https://doi.org/10.3847/1538-4357/ab0a01
Reference Antenna : Select the reference antenna from first 30 core antennas. Default is 1.
Auto-calculate Parameters : If selected, P-AIRCARS will calculate calibration and imaging parameters automatically from the data. By default, it is switched on. When it is turned off, user can give their own calibration and imaging parameters in advanced inputs. Only experienced users should turn this off.
Use WSClean : Use WSClean for imaging. By default it is switched on. If it is turned off, CASA will be used for imaging. Imaging using CASA is significantly slow compared to WSClean. Thus users are instructed to use WSClean. If WSClean is not installed, P-AIRACARS will automatically use CASA for imaging even if Use WSClean is turned on.
CASA Auto-masking : When CASA is being used for imaging, CASA auto-masking can be used by enabling this option. By default it is not activated.
Notification : If selected, P-AIRCARS will automatically send the updates about the progress of calibration and imaging to the user specific e-mail id.
CASA Mask : When using CASA for imaging, user may supply a CASA mask file here. By default it is not activated. It will be activated when Use WSClean is turned off.
CASA Mask String : CASA soecific mask string when using CASA for imaging. By default it is not activated. It will be activated when Use WSClean is turned off.
e-mail : User e-mail address to send notifications. If no e-mail id is given, no notification can be sent.
Bandpass : Whether perform bandpass self-calibration or not. By default it is turned on.
Polcal : Whether perform polarisation calibration and imaging. By default it is turned on. When it off, P-AIRCARS will only make Stokes-I images.
Frequency Interval : Frequency interval to make images. By default it is 160 kHz.
Time Interval : Time interval to make images. By default it is 0.5 s.
Frequency Width : Bandwidth to average. By default it is 160 kHz.
Time Width : Temporal averaging. By default it is 0.5 s.
CPU(%) : Percentage of total available cpus to be used. By default it is 50%.
Clear Virtual SCreens : Clear the virtual screens opened during the P-AIRCARS run.
Save models : If selected, save final model images. By default it is turned off.
Save residuals : If selected, save final residual images. By default it is turned off.
XY cutout : Size of the final image cutout. Default size is 3 degree.
Perform Flagging : Perform flagging during the calibration or not. By default it is on.
Use aNKflag : If selected, use aNKflag for flagging. If aNKflag is not installed, do not use.
HPC environment : Switch on if running is high performance computing (HPC) environment. When it is on fill the settings for HPC environment.
Input file : Load the P-AIRCARS inputs from previous saved P-AIRCARS input file. Extension of P-AIRCARS input file is ‘.paircars’. After loading inputs will be filled automatically.
Restart P-AIRCARS : Restart P-AIRCARS from the interupted step. It is off by default.
Fresh start P-AIRCARS : Start P-AIRCARS in fresh. If selected, all previous analysis in the same Base Directory will be deleted. By default it is switched on.

Advanced Inputs¶

Advanced input block will be activated when Auto-calculate Parameters is switched off. Use Advanced Inputs carefully and suggest to use only by experienced users.

Pixelsize : Pixel size of the image in arcsec or arcmin, e.g. ‘20.0arcsec’ or ‘1.0arcmin’.
Number of pixels : Total number of pixels in the image. Product of Pixelsize and Number of pixels determine the size of the field-of-view.
Multiscale scales : Multiscale scales in units of number of pixels.
UV-taper : UV taper string in CASA format
Start Sigma : Starting value of sigma for rms based thresholding in self-calibration.
Sigma Step : Stepsize to reduce the sigma value with self-calibration iterations.
Minimum Sigma : Minimum value of sigma to reduce during self-calibration.
Residual Flux Fraction : Fraction of flux remained in the residual image to stop the self-calibration.
UV range : UV range in CASA format for self-calibration.
Skip Frequency : Frequency step to skip during calibration. Calibration will be performed after Skip Frequency steps.
Skip Time : Time step to skip during calibration. Calibration will be performed after Skip Time steps.
Gain Minimum SNR : Minimum SNR of the gain solutions to be used. Gain solutions less than minimum SNR will be rejected.
DR RMS step : If rms based dynamic range increases less than this value, self-calibration converged.
DR Negative step : If negative based dynamic range increases less than this value, self-calibration converged.
Minimum DR : Minimum dynamic range to save a final image.
Maximum DR : Maximum dynamic range required. If the imaging dynamic range reached this value, self-calibration stopped.
Minimum Selfcal SNR : Minimum antenna based SNR required to allow the self-calibration to start.
Extra time averaging : If large calibration solutions flagged, average over Extra time to gain SNR. The value is in second.
Maximum average time : Maximum allowed value of the Extra time.

High Performance Computing (HPC) settings¶

Robustness and Quality¶

In this section two major input parameters for P-AIRCARS, Quality and Robustness, will be discussed.

Quality (Q)¶

Quality parameters determines the quality of the final images. With the increase of Quality value, antennas will be added in smaller steps, self-calibration will be continued for smaller value of residual fluxes and final CLEANing of the images will be deeper. Quality takes three values; 0, 1, 2. Default value is 1.

Robustness (R)¶

Robustness parameter determines the robustness of the calibration and imaging. Increasing the robustness parameter will use strong convergence criteria for self-calibration, uses higher value of minimum SNR for gain solutions, performs calibration at smaller time and frequency intervals. Robustness takes three values; 0, 1, 2. Default value is 1.

The calibration and imaging parameters are determined based on the combination of Quality and Robustness. The calibration and imaging parameters for each of the combinations with increasing significance are listed below. With the increasing significance, the calibration and imaging quality will be better, but the computational time will also increase. Default value of 1 for Quality and Robustness is an optimization between final image quality and computational time.

Combinations¶

Q = 0, R = 0

residual_frac = 0.03

start_sigma = 9.0

min_sigma = 6

gain_minsnr = 3.0

DR_delta_rms = 25

DR_delta_neg = 20

min_DR = 20

max_DR = 100

skip_time = 480 s

skip_freq = 2560 kHz

Q = 0, R = 1

residual_frac = 0.03

start_sigma = 9.0

min_sigma = 7

gain_minsnr = 3.0

DR_delta_rms = 22

DR_delta_neg = 18

min_DR = 25

max_DR = 500

skip_time = 240 s

skip_freq = 2560 kHz

Q = 0, R = 2

residual_frac = 0.03

start_sigma = 9.0

min_sigma = 8

gain_minsnr = 3.0

DR_delta_rms = 20

DR_delta_neg = 15

min_DR = 30

max_DR = 1000

skip_time = 120 s

skip_freq = 2560 kHz

Q = 1, R = 0

residual_frac = 0.015

start_sigma = 10.0

min_sigma = 7

gain_minsnr = 4

DR_delta_rms = 20

DR_delta_neg = 15

min_DR = 30

max_DR = 1000

skip_time = 120 s

skip_freq = 1280 kHz

Q = 1, R = 1 (Default)

residual_frac = 0.015

start_sigma = 10.0

min_sigma = 8

gain_minsnr = 4

DR_delta_rms = 18

DR_delta_neg = 12

min_DR = 35

max_DR = 5000

skip_time = 60 s

skip_freq = 1280 kHz

Q = 1, R = 2

residual_frac = 0.015

start_sigma = 10.0

min_sigma = 9

gain_minsnr = 4

DR_delta_rms = 15

DR_delta_neg = 10

min_DR = 40

max_DR = 10000

skip_time = 30 s

skip_freq = 1280 kHz

Q = 2, R = 0

residual_frac = 0.01

start_sigma = 11.0

min_sigma = 8

gain_minsnr = 4.5

DR_delta_rms = 18

DR_delta_neg = 12

min_DR = 40

max_DR = 10000

skip_time = 15 s

skip_freq = 640 kHz

Q = 2, R = 1

residual_frac = 0.01

start_sigma = 11.0

min_sigma = 9

gain_minsnr = 4.5

DR_delta_rms = 15

DR_delta_neg = 10

min_DR = 45

max_DR = 50000

skip_time = 12 s

skip_freq = 640 kHz

Q = 2, R = 2

residual_frac = 0.01

start_sigma = 11.0

min_sigma = 10

gain_minsnr = 4.5

DR_delta_rms = 12

DR_delta_neg = 8

min_DR = 50

max_DR = 100000

skip_time = 10 s

skip_freq = 640 kHz

paircars¶

paircars.access_ms¶

paircars.basic_func¶

paircars.decor¶

paircars.dynamic_spectrum¶

paircars.flagger¶

paircars.fullpol_selfcal_LTS¶