helper.py¶
as_list(x)
¶
check_exp_input_parameters(exp)
¶
Validates the input parameters and experiment configuration.
This function performs the following checks: 1. Verifies that the ‘agebins’ parameter contains all age groups referenced in ‘age_groups_aggregates’. 2. Ensures that ‘agebins’ includes both the values 2 and 10. 3. Confirms that the ‘seasonality’ parameter contains no more than two seasonality patterns.
Parameters: |
|
---|
Raises: |
|
---|
Source code in utility\helper_simulation.py
check_exp_inputs(exp, attribute_check=False)
¶
Helper function to validate experiment inputs.
Parameters: |
|
---|
Raises: |
|
---|
Source code in utility\helper.py
check_exp_inputs_emod(exp)
¶
Checks input parameter compatibility with the framework configurations for EMOD.
This function performs the following checks: 1. Ensures that only a single seed is specified for EMOD burn-in simulations. 2. Issues a warning if the burn-in period is less than 100 years, which may lead to inaccurate results. 3. Issues a warning if the population size is below 1000, which may lead to inaccurate results. 4. Issues a warning if the population size exceeds 10000, which may cause computational or memory issues.
Parameters: |
|
---|
Raises: |
|
---|
Source code in utility\helper_simulation.py
check_exp_inputs_malariasimulation(exp)
¶
Checks input parameter compatibility with the framework configurations for malariasimulation
This function performs the following checks: 1. Raises a ValueError if the ‘forced EIR’ option is selected for entomology mode, as it is not supported. 2. Issues a warning if the burn-in period is less than 80 years, which may lead to inaccurate results. 3. Issues a warning if the population size is below 1000, which may lead to inaccurate results.
Parameters: |
|
---|
Raises: |
|
---|
Source code in utility\helper_simulation.py
check_exp_inputs_openmalaria(exp)
¶
Checks input parameter compatibility with the framework configurations for OpenMalaria.
This function performs the following checks: 1. Issues a warning if OpenMalaria is configured with 30-day survey steps and a 5-day analyzer, as this configuration will cause errors during post-processing. 2. Raises a ValueError if the ‘ccstep’ intervention is specified with 30-day survey steps, which is incompatible. 3. Issues a warning if the burn-in period is less than 80 years, which may lead to inaccurate results. 4. Issues a warning if the population size is below 10000, which may lead to inaccurate results.
Parameters: |
|
---|
Raises: |
|
---|
Source code in utility\helper_simulation.py
create_scenarios_csv(exp, col_list)
¶
Generate a scenarios DataFrame based on the provided experiment object and column list.
This function constructs a DataFrame of all possible combinations of attributes specified
in col_list
and maps additional experiment-specific parameters, including entomology mode,
number of seeds, and calibration target inputs. It also processes specific interventions
such as ccstep
if present.
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper.py
create_simsetup_csv(exp)
¶
Creates a CSV file containing the simulation setup parameters from the given experiment object.
This function collects various parameters from the exp
(Experiment) object and writes them to a CSV file.
The CSV contains the following fields:
- run_mode
, job_directory
, output_directory
, emod_serialized_id
, and other simulation-specific parameters.
- Additionally, various burnin, population size, age group settings, seasonal data, and intervention lists are included.
The resulting CSV is saved in the directories specified by job_directory
and output_directory
.
Parameters: |
|
---|
Source code in utility\helper.py
exec(command)
¶
Executes the specified command in a subprocess.
Parameters: |
|
---|
Returns: |
|
---|
Raises: |
|
---|
Notes
- This function runs the command in a new shell and captures both standard output and standard error.
- The output can be accessed through the Popen object returned by this function.
Source code in utility\helper_slurm.py
exp_obj_to_csv(file_path)
¶
Converts the experiment object (stored as a pickle file) to a CSV file.
This function reads a pickle file containing an experiment object (exp.obj
), and then converts it to a CSV
file format (exp_obj.csv
). The function supports two data types: dictionaries and lists.
- If the data is a dictionary, it writes each key-value pair to the CSV with columns “Key” and “Value”.
- If the data is a list, it writes each item in the list as a row in the CSV (assuming each item is a list or tuple).
Parameters: |
|
---|
Source code in utility\helper.py
get_intervention_params(exp)
¶
Retrieve intervention parameters for a specific experiment.
Parameters: - exp: Experiment object.
Returns: Intervention parameters based on the specified experiment.
Source code in utility\helper_simulation.py
get_param_from_dataframe(df, name, sim_params_list)
¶
Retrieves a parameter value from a DataFrame and returns it in the correct format for storage in an experiment object (exp).
Example
exp.seasonality = get_param(exp_scen_df, ‘seasonality’, listparam=True)
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper.py
get_seasonal_eir(exp=None)
¶
Generates seasonal EIR (Entomological Inoculation Rate) values.
Examples:
season_daily, season_month, seasonal, perennial = get_seasonal_eir() exp = get_seasonal_eir(exp)
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper_simulation.py
get_simulation_time_params(exp)
¶
Calculates simulation time parameters for EMOD, malariasimulation and OpenMalaria based on the provided arguments.
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper_simulation.py
make_dirs(exp, overwrite=False)
¶
Creates necessary directories for a simulation experiment.
Args: exp (Experiment): The experiment object containing project directory information. overwrite (bool, optional): If True, existing directories will be overwritten. Defaults to False.
Returns: exp (Experiment): The experiment object with updated directory paths.
Source code in utility\helper.py
map_model_calib_inputs(exp, df)
¶
Maps model inputs based on output targets.
Parameters: |
|
---|
Returns: pandas.DataFrame: Updated DataFrame.
Source code in utility\helper.py
monthly_to_daily_EIR(monthly_EIR)
¶
Convert monthly EIR values to daily using cubic spline interpolation.
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper_simulation.py
param_variation(df, exp)
¶
Perform parameter variation for malariasimulation simulations.
Parameters: - df: DataFrame containing data from scenarios.csv. - exp: Experiment object.
Returns: DataFrame with added column ‘malariasimulation_pv’ representing malariasimulation parameter variation values.
Source code in utility\helper_simulation.py
parse_args()
¶
Parses command-line arguments for simulation specifications.
Returns: argparse.Namespace: Parsed command-line arguments.
Source code in utility\helper.py
parse_launch_args()
¶
Parses command-line arguments for launching a simulation experiment.
This function defines and parses several command-line arguments required for running a simulation experiment. The arguments specify parameters such as the job directory, suite name, experiment name, model to run, user directory, interventions, run mode, and output type. The parsed arguments are returned as a Namespace object.
Returns: |
|
---|
Command-Line Arguments
-d, –directory (str) : Job directory for burn-in; ignored if no EMOD burn-in is provided. –suite (str) : Name of the suite; defaults to the current date followed by “_test”. –expname (str) : Name of the experiment; defaults to include the current time. –emodstep (str) : Specifies the EMOD step (e.g., ‘burnin’, ‘pickup’, ‘new_pickup’). –serializedid (str) : Serialized ID for EMOD, if applicable. –models (list) : List of models to run (default: [‘EMOD’, ‘malariasimulation’, ‘OpenMalaria’]). –user (str) : Subdirectory for simulations, based on the current user or ‘b1139’ for NUCLUSTER. –csv (str) : Name of the scenario CSV file. –rownum (int) : Row number from the specified scenario CSV (default: 0). -r, –run_mode (str) : Type of run (‘custom’, ‘calibration’, ‘simple’, ‘production’, or ‘csv’). –emod_calib_params (bool) : Whether to include calibration parameters for EMOD. –mpv (bool) : Whether to run malariasimulation with parameter variation (default: True). -t, –test (bool) : Runs the simulation with lower population, shorter burn-in, and fewer seeds. -n, –note (str) : An optional tag to distinguish different runs.
Source code in utility\helper.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 |
|
process_case_management(df)
¶
Processes and splits the case management column.
Handles cases where case_management
contains either a list of two values
or a single value in a list.
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper.py
rep_scen_df(df)
¶
Repeats scenario data in a DataFrame based on the number of seeds.
Parameters: |
|
---|
Returns: |
|
---|
Source code in utility\helper.py
save_exp_scen(exp, scen_df, save_dirs)
¶
Saves the experiment and scenario data to the specified save directories. Args: exp (Experiment): Experiment object containing experiment specifications. scen_df (pandas.DataFrame): The scenario data to be saved. save_dirs (list): List of directories to save the data.
Source code in utility\helper.py
save_scen(scen_df, fname, save_dirs=None)
¶
Saves a scenario DataFrame to a CSV file in the specified directories. Args: scen_df (pandas.DataFrame): The DataFrame containing the scenario data. fname (str): The name of the CSV file to be saved. save_dirs (list of str, optional): List of directories to save the CSV file. Defaults to None.
Source code in utility\helper.py
shell_header_quest(sh_account, time_str='03:59:00', memG=3, job_name='myjob', arrayJob=None, mem_scl=1)
¶
Generates the SLURM shell script header for submitting jobs to a high-performance computing cluster.
Parameters: |
|
---|
Returns: |
|
---|
Raises: |
|
---|
Notes
- The function checks if the ‘log’ directory exists and creates it if it does not.
- The partition is selected based on the job time limit if the account is not a buy-in account.
- The generated header includes job error and output log file paths based on whether the job is an array job or a single job.
Source code in utility\helper_slurm.py
str_to_digit(x)
¶
Converts a string to a float if possible, otherwise returns the original string. Args: x (str): The input string. Returns: float or str: The converted float if conversion is successful, otherwise the original string.
Source code in utility\helper.py
submit_run_plotters(exp)
¶
Submits a plotter job for execution, either locally or via SLURM based on the HPC configuration: - If running locally, it generates and submits the plotter job using a local submission script. - If using SLURM, it delegates the submission to the SLURM-specific plotter submission function.
Parameters: |
|
---|
Source code in utility\helper_simulation.py
submit_run_plotters_slurm(exp)
¶
Submits and manages the creation of shell scripts to run standardized plots based on the models specified in the experiment.
Parameters: |
|
---|
Returns: |
|
---|
Notes
- The function checks which models (EMOD, malariasimulation, OpenMalaria) are
specified in
exp.models
and sets corresponding job IDs. - It prepares a submission script for a set of default plots (
relationship
,timeseries
,agecurves
) and also handles custom plots if specified inexp.plots_to_run
. - Memory requirements for each plot are defined and passed to the submission
script. The default is 20 GB, but this can be adjusted based on specific
plot needs (e.g.,
ccstep
requires 80 GB). - If
exp.plots_to_run
is set to'all'
, all available plots will be processed.
Source code in utility\helper_slurm.py
submit_run_pyscript(exp, pyscript='plotter/plot_relationship.py', shname='run_relationship_plots.sh', custom_args='--models EMOD malariasimulation OpenMalaria', t='05:00:00', memG=20, job_id_EMOD=False, job_id_malariasimulation=False, job_id_OpenMalaria=False, job='pyjob', wdir=None, write_only=False)
¶
Submits a job to run a specified Python script using SLURM.
Parameters: |
|
---|
Returns: |
|
---|
Raises: |
|
---|
Notes
- The function generates a shell script that includes the SLURM header and the command to run the specified Python script.
- It handles job dependencies based on the provided job IDs for EMOD, malariasimulation, and OpenMalaria.
- The script is written to the job directory and submitted to the SLURM workload manager.
- The submitted job ID is printed to the console for tracking purposes.
Source code in utility\helper_slurm.py
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 |
|
write_txt(txtobj, path, fname)
¶
Writes a text object to a file. Args: txtobj (str): The text object to write. path (str): The path to the directory where the file will be saved. fname (str): The filename to use for the file.