Run P-AIRCARS Pipeline¶
Basic run¶
To run P-AIRCARS pipeline, with default settings for full analysis, run the following command from terminal. Work directory needs not to be created before hand, but the path where it will be created should exist.
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory>
Note
User needs to provide full path. Providing short path or only the directory name will cause error.
Advanced run¶
For advanced run, user is requested to first check the parameters of run-mwa-paircars.
run-mwa-paircars -h
Click here to see parameters
P-AIRCARS data directory and docker environment is not setup yet
usage: run-mwa-paircars [-h] --workdir WORKDIR --outdir OUTDIR
[--target_metafits TARGET_METAFITS]
[--cal_datadir CAL_DATADIR]
[--cal_metafits CAL_METAFITS] [--solint SOLINT]
[--cal_uvrange CAL_UVRANGE] [--no_polcal]
[--only_amplitude] [--redo_basic_cal] [--redo_selfcal]
[--use_solarflagger] [--freqrange FREQRANGE]
[--timerange TIMERANGE]
[--image_freqres IMAGE_FREQRES]
[--image_timeres IMAGE_TIMERES] [--pol POL]
[--minuv MINUV] [--weight WEIGHT] [--robust ROBUST]
[--no_multiscale] [--clean_threshold CLEAN_THRESHOLD]
[--no_pbcor] [--cutout_rsun CUTOUT_RSUN]
[--no_solar_mask] [--do_overlay] [--make_msplot]
[--non_solar_data] [--no_ds]
[--do_forcereset_weightflag] [--no_cal_flag]
[--no_import_model] [--no_basic_cal]
[--do_sidereal_cor] [--no_solarcenter_move]
[--no_selfcal] [--no_ap_selfcal] [--no_solar_selfcal]
[--no_applycal] [--no_apply_selfcal] [--no_imaging]
[--verbose] [--cpu_frac CPU_FRAC]
[--mem_frac MEM_FRAC] [--max_worker MAX_WORKER]
[--keep_backup] [--no_calibrated_ms]
[--no_remote_logger] [--log2term]
[--job_password JOB_PASSWORD] [--cluster]
[--port PORT] [--partition PARTITION]
[--account ACCOUNT] [--walltime WALLTIME]
target_datadir
Run P-AIRCARS for calibration and imaging of solar observations.
options:
-h, --help show this help message and exit
###################
Essential parameters
###################:
target_datadir Target measurement set directory
--workdir WORKDIR Working directory (default: None)
--outdir OUTDIR Output products directory (default: None)
--target_metafits TARGET_METAFITS
Target metafits file (default: )
--cal_datadir CAL_DATADIR
Calibrator measurement set directory (default: )
--cal_metafits CAL_METAFITS
Calibrator metafits file (default: )
###################
Advanced calibration parameters
###################:
--solint SOLINT Solution interval for calibration (e.g. 'int', '10s',
'5min', 'inf') (default: 60s)
--cal_uvrange CAL_UVRANGE
UV range to filter data for calibration (e.g.
'>100klambda', '100~10000lambda') (default: )
--no_polcal Disable polarization calibration
--only_amplitude Apply only amplitude part of gain solution from
calibrator or not
--redo_basic_cal Redo basic calibration or not
--redo_selfcal Redo self-calibration or not
--use_solarflagger Use solar flagger
###################
Advanced imaging parameters
###################:
--freqrange FREQRANGE
Frequency range in MHz to select during imaging
(comma-seperate, e.g. '100~110,130~140') (default: )
--timerange TIMERANGE
Time range to select during imaging (comma-seperated,
e.g. '2014/09/06/09:30:00~2014/09/06/09:45:00,2014/09/
06/10:30:00~2014/09/06/10:45:00') (default: )
--image_freqres IMAGE_FREQRES
Output image frequency resolution in MHz (-1 = full)
(default: 1.28)
--image_timeres IMAGE_TIMERES
Output image time resolution in seconds (-1 = full)
(default: 10.0)
--pol POL Stokes parameter(s) to image ('I' or 'IQUV') (default:
IQUV)
--minuv MINUV Minimum baseline length (in wavelengths) to include in
imaging (default: 0)
--weight WEIGHT Imaging weighting scheme (e.g. 'briggs', 'natural',
'uniform') (default: briggs)
--robust ROBUST Robust parameter for Briggs weighting (-2 to +2)
(default: 0.0)
--no_multiscale Disable multiscale CLEAN for extended structures
--clean_threshold CLEAN_THRESHOLD
Clean threshold in sigma for final deconvolution
(default: 1.0)
--no_pbcor Do not apply primary beam correction after imaging
--cutout_rsun CUTOUT_RSUN
Field of view cutout radius in solar radii (default:
10.0)
--no_solar_mask Disable use solar disk mask during deconvolution
--do_overlay Make overlay plot on EUV images
--make_msplot Make diagnostic plots of measurement sets
###################
Advanced pipeline parameters
###################:
--non_solar_data Disable solar data mode
--no_ds Disable making solar dynamic spectra
--do_forcereset_weightflag
Force reset of weights and flags (disabled by default)
--no_cal_flag Disable initial flagging of calibrators
--no_import_model Disable model import
--no_basic_cal Disable basic gain calibration
--do_sidereal_cor Sidereal motion correction for Sun (disabled by
default)
--no_solarcenter_move
Disable moving phaseceneter to solar center
--no_selfcal Disable self-calibration
--no_ap_selfcal Disable amplitude-phase self-calibration
--no_solar_selfcal Disable solar-specific self-calibration parameters
--no_applycal Disable application of basic calibration solutions
--no_apply_selfcal Disable application of self-calibration solutions
--no_imaging Disable final imaging
--verbose Verbose logs
###################
Advanced hardware resource parameters for local system or per node on HPC cluster
###################:
--cpu_frac CPU_FRAC Fraction of CPU usuage per node (default: 0.8)
--mem_frac MEM_FRAC Fraction of memory usuage per node (default: 0.8)
--max_worker MAX_WORKER
Maximum number of workers (default: -1)
--keep_backup Keep backup of intermediate steps
--no_calibrated_ms Keep calibrated measurement sets or not
--no_remote_logger Disable remote logger
--log2term Show logs in terminal
--job_password JOB_PASSWORD
User specified job password (default: None)
--cluster Running in cluster environment
--port PORT Prefect port (default: 4260)
###################
Advanced slurm cluster settings
###################:
--partition PARTITION
Partition name (Required) (default: None)
--account ACCOUNT Account name (If your cluster requires this, you
should provide. Otherwise job can not be started)
(default: None)
--walltime WALLTIME Wall time, each slurm job can execute in maximum this
time (default: None)
Multiple options demonstrated below can be combined to have all of them together.
To view details of measurement set:
show-paircars-ms </full/path/to/measurement_set>
Runs with advanced calibration paramaters¶
If one do not have calibrator observations:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory>
Note
P-AIRCARS will self-calibrate and perform flux-density and polarisation calibration based on some assumptions. In these cases, absolute values of the quantities should be carefully considered.
If user wants to provide specific path to target and calibrator metafits file
run-mwa-paircars </full/path/to/data_directory> --target_metafits </full/path/to/data_metafits> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --cal_metafits </full/path/to/calibrator_metafits>
If user wants to run with multiple target and calibrator observations
run-mwa-paircars </full/path/to/data_directory_1>,</full/path/to/data_directory_2> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory_1>,</full/path/to/calibrator_data_directory_2>
Do calibration with custom calibration parameters. There are two parameters: cal_uvrange and solint which can be changed. Example, run the following command to perform gain solutions at 10second interval and >200lambda data:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --solint "10s" --uvrange ">200lambda"
By default for full-polar data, polarization calibration will be performed. To disable it:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --no_polcal
Runs with advanced imaging paramaters¶
Run pipeline to image specific time and frequency range. Default is to use entire time and frequency range. Example for imaging two time ranges given in UTC and frequency ranges given in MHz:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --timerange "2024/06/10/09:00:00~2024/06/10/09:30:00,2024/06/10/10:15:00~2024/06/10/10:45:00" --freqrange 100~150,200~230
Run imaging with custom time and frequency resolution. Default is to use each 1.28 MHz coarse bands and 10s of integration. Example run for imaging at 0.5 second time resolution and 160 kHz (0.16 MHz) frequency resolution:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --image_timeres 0.5 --image_freqres 0.16
Default is to make only Stokes I images if do_polcal=False and Stokes IQUV, if do_polcal=True. To run only Stokes I imaging, even if do_polcal=True, run:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --pol I
Similarly, all other advanced imaging parameters can be used.
Switching off particular pipeline step(s)¶
By default, all steps will be performed by pipeline. Even pipeline was run upto certain stages and then stopped, all steps from beginning will be performed to avoid any potential issue in failure in previous runs. If user is certain that previous run was successful upto certain stages, those stages can be switched.
Caution
User should not modify any file and directory structure in the work directory. Switching off certain parameters will only allow to run the pipeline forward, if the expected output products from those steps are present with appropriate name in appropriate directory. Otherwise, it will fail.
Take a look at the Advanced pipeline parameters in the help page of run-mwa-paircars. Each parameters are self explanatory. Some examples are given below:
To switch off self-calibration:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --no_selfcal
To stop final imaging:
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --no_imaging
To switch off self-calibration and final imaging
run-mwa-paircars </full/path/to/data_directory> --workdir </full/path/to/work_directory> --outdir </full/path/to/output_product_directory> --cal_datadir </full/path/to/calibrator_data_directory> --no_selfcal --no_imaging