simopt.experiment_base

Base classes for problem-solver pairs and I/O/plotting helper functions.

Module Contents

simopt.experiment_base.model_directory

simopt.experiment_base.problem_directory

simopt.experiment_base.solver_directory

simopt.experiment_base.SIMOPT_TOPLEVEL

simopt.experiment_base.EXPERIMENT_DIR

Base class for running one solver on one problem.

Initializes a ProblemSolver object.

You can either: 1. Provide solver and problem names to look them up via directory.py, or 2. Provide solver and problem objects directly.

Parameters:

solver_name (str, optional) – Name of the solver.
problem_name (str, optional) – Name of the problem.
solver_rename (str, optional) – User-defined name for the solver.
problem_rename (str, optional) – User-defined name for the problem.
solver (Solver, optional) – Simulation-optimization solver object.
problem (Problem, optional) – Simulation-optimization problem object.
solver_fixed_factors (dict, optional) – Fixed solver parameters.
problem_fixed_factors (dict, optional) – Fixed problem parameters.
model_fixed_factors (dict, optional) – Fixed model parameters.
file_name_path (Path | str, optional) – Path to save a pickled ProblemSolver object.
create_pickle (bool, optional) – Whether to save the object as a pickle file.

property solver: simopt.base.Solver: Simulation-optimization solver.

property problem: simopt.base.Problem: Simulation-optimization problem.

property n_macroreps: int: Number of macroreplications run.

property file_name_path: pathlib.Path: Path of .pickle file for saving ProblemSolver object.

property all_recommended_xs: list[list[tuple]]: Sequences of recommended solutions from each macroreplication.

property all_intermediate_budgets: list[list]: Sequences of intermediate budgets from each macroreplication.

property timings: list[float]: Runtimes (in seconds) for each macroreplication.

property n_postreps: int: Number of postreps to take at each recommended solution.

property crn_across_budget: bool: Whether CRN is used across solutions recommended at different times.

property crn_across_macroreps: bool: Whether CRN is used across solutions from different macroreplications.

property all_post_replicates: list[list[list]]: All post-replicates from all solutions from all macroreplications.

property all_est_objectives: list[list[float]]: Estimated objective values of all solutions from all macroreplications.

property n_postreps_init_opt: int: Number of postreplications at initial (x0) and optimal (x*) solutions.

property crn_across_init_opt: bool: Whether CRN is used for postreplications at x0 and x* solutions.

property x0: tuple: Initial solution (x0).

property x0_postreps: list: Post-replicates at x0.

property xstar: tuple: Proxy for optimal solution (x*).

property xstar_postreps: list: Post-replicates at x*.

property objective_curves: list[simopt.curve.Curve]: Estimated objective function curves, one per macroreplication.

property progress_curves: list[simopt.curve.Curve]: Progress curves, one for each macroreplication.

property has_run: bool: True if the solver has been run on the problem, otherwise False.

property has_postreplicated: bool: True if the solver has been postreplicated, otherwise False.

property has_postnormalized: bool: True if the solver has been postprocessed, otherwise False.

create_pickle = True

all_stoch_constraints = []

all_est_lhs = []

feasibility_curves = []

model_created(model: simopt.base.Model) → None

Hook called after the problem’s model is instantiated.

Parameters:: model – The initialized model associated with the experiment’s problem.

This is a helper function to customize the experiment’s input model.

before_replicate(model: simopt.base.Model, rng_list: list[mrg32k3a.mrg32k3a.MRG32k3a]) → None

Hook executed immediately before each replication during an experiment.

Parameters:

model – The model about to be simulated.
rng_list – The list of RNGs used for the replication.

This is a helper function to customize behavior before each replication.

check_compatibility() → str

Check whether the experiment’s solver and problem are compatible.

Returns:: Error message in the event problem and solver are incompatible.
Return type:: str

run(n_macroreps: int, n_jobs: int = -1) → None

Runs the solver on the problem for a given number of macroreplications.

Note

RNGs for random problem instances are reserved but currently unused. This method is under development.

Parameters:

n_macroreps (int) – Number of macroreplications to run.
n_jobs (int, optional) – Number of jobs to run in parallel. Defaults to -1. -1: use all available cores 1: run sequentially

Raises:

ValueError – If n_macroreps is not positive.

run_multithread(mrep: int, solver: simopt.base.Solver, problem: simopt.base.Problem) → tuple

Runs one macroreplication of the solver on the problem.

Parameters:

mrep (int) – Index of the macroreplication.
solver (Solver) – The simulation-optimization solver to run.
problem (Problem) – The problem to solve.

Returns:

A tuple containing:

int: Macroreplication index.
list: Recommended solutions.
list: Intermediate budgets.
float: Runtime for the macroreplication.

Return type:

tuple

Raises:

ValueError – If mrep is negative.

post_replicate(n_postreps: int, crn_across_budget: bool = True, crn_across_macroreps: bool = False) → None

Runs postreplications at the solver’s recommended solutions.

Parameters:

n_postreps (int) – Number of postreplications at each recommended solution.
crn_across_budget (bool, optional) – If True, use CRN across solutions from different time budgets. Defaults to True.
crn_across_macroreps (bool, optional) – If True, use CRN across solutions from different macroreplications. Defaults to False.

Raises:

ValueError – If n_postreps is not positive.

post_replicate_multithread(mrep: int) → tuple

Runs postreplications for a given macroreplication’s recommended solutions.

Parameters:

mrep (int) – Index of the macroreplication.

Returns:

A tuple containing:

int: Macroreplication index.
list: Postreplicates for each recommended solution.
float: Runtime for the macroreplication.

Return type:

tuple

Raises:

ValueError – If mrep is negative.

bootstrap_sample(bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False, disable_macrorep_bootstrap: bool = False) → tuple[list[simopt.curve.Curve], list[simopt.curve.Curve]]

Generates bootstrap samples of objective/progress and feasibility curves.

Parameters:

bootstrap_rng (MRG32k3a) – Random number generator used for bootstrapping.
normalize (bool, optional) – If True, normalize progress curves with respect to optimality gaps. Defaults to True.
feasibility_score_method (Literal["inf_norm", "norm"], optional) – Feasibility scoring method. Defaults to “inf_norm”.
feasibility_norm_degree (int, optional) – Degree of the norm when
1. (feasibility_score_method == "norm". Defaults to)
feasibility_two_sided (bool, optional) – Whether to award a non-zero score to feasible solutions based on the best violation.
disable_macrorep_bootstrap (bool, optional) – Whether to disable bootstrap across macroreplications. Defaults to False.

Returns:

Bootstrapped progress curves and feasibility curves for all macroreplications.

Return type:

tuple[list[Curve], list[Curve]]

bootstrap_terminal_objective_and_feasibility(bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = True) → tuple[list[float], list[float]]: Bootstraps terminal objective and feasibility scores.

feasibility_score_history(feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = True) → None: Compute feasibility score history.

record_experiment_results(file_name: str) → None

Saves the ProblemSolver object to a .pickle file.

Parameters:: file_name (str) – Name of the pickle file. It is saved under the EXPERIMENT_DIR path.

log_experiment_results(print_solutions: bool = True) → None

Creates a readable .txt log file from a problem-solver pair’s .pickle file.

Parameters:: print_solutions (bool, optional) – If True, include recommended solutions in the .txt file. Defaults to True.

simopt.experiment_base.trim_solver_results(problem: simopt.base.Problem, recommended_solutions: list[simopt.base.Solution], intermediate_budgets: list[int]) → tuple[list[simopt.base.Solution], list[int]]

Trims solver-recommended solutions beyond the problem’s maximum budget.

Parameters:

problem (Problem) – The problem the solver was run on.
recommended_solutions (list[Solution]) – Solutions recommended by the solver.
intermediate_budgets (list[int]) – Budgets at which solutions were recommended.

Returns:

A tuple containing:

list[Solution]: Trimmed list of recommended solutions.
list[int]: Trimmed list of corresponding intermediate budgets.

Return type:

tuple

simopt.experiment_base.read_experiment_results(file_name_path: pathlib.Path | str) → ProblemSolver

Reads a ProblemSolver object from a .pickle file.

Parameters:

file_name_path (Path | str) – Path to the .pickle file.

Returns:

Loaded problem-solver pair that was previously run or: post-processed.

Return type:

ProblemSolver

Raises:

ValueError – If the file does not exist.

simopt.experiment_base.post_normalize(experiments: list[ProblemSolver], n_postreps_init_opt: int, crn_across_init_opt: bool = True, proxy_init_val: float | None = None, proxy_opt_val: float | None = None, proxy_opt_x: tuple | None = None, create_pair_pickles: bool = False) → None

Constructs objective and normalized progress curves for a set of experiments.

Parameters:

experiments (list[ProblemSolver]) – Problem-solver pairs for different solvers on the same problem.
n_postreps_init_opt (int) – Number of postreplications at initial (x0) and optimal (x*) solutions.
crn_across_init_opt (bool, optional) – If True, use CRN for postreplications at x0 and x*. Defaults to True.
proxy_init_val (float, optional) – Known objective value of the initial solution.
proxy_opt_val (float, optional) – Proxy or bound for the optimal objective value.
proxy_opt_x (tuple, optional) – Proxy for the optimal solution.
create_pair_pickles (bool, optional) – If True, create a pickle file for each problem-solver pair. Defaults to False.

simopt.experiment_base.bootstrap_sample_all(experiments: list[list[ProblemSolver]], bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False) → tuple[list[list[list[simopt.curve.Curve]]], list[list[list[simopt.curve.Curve]]]]

Generates bootstrap samples of progress and feasibility curves.

Parameters:

experiments (list[list[ProblemSolver]]) – Grid of problem-solver pairs, where each inner list corresponds to different problems for a given solver.
bootstrap_rng (MRG32k3a) – Random number generator used for bootstrapping.
normalize (bool, optional) – If True, normalize progress curves by optimality gaps. Defaults to True.
feasibility_score_method (Literal["inf_norm", "norm"], optional) – Feasibility scoring rule.
feasibility_norm_degree (int, optional) – Degree of the norm when feasibility_score_method == "norm".
feasibility_two_sided (bool, optional) – Whether to give feasible solutions a non-zero score based on the best violation.

Returns:

Bootstrapped progress/objective curves and feasibility curves for all solutions from all macroreplications, grouped by solver and problem.

Return type:

tuple[list[list[list[Curve]]], list[list[list[Curve]]]]

class simopt.experiment_base.PlotType(*args, **kwds)

Bases: enum.Enum

Enum class for different types of plots and metrics.

ALL = 'all'

MEAN = 'mean'

QUANTILE = 'quantile'

AREA_MEAN = 'area_mean'

AREA_STD_DEV = 'area_std_dev'

SOLVE_TIME_QUANTILE = 'solve_time_quantile'

SOLVE_TIME_CDF = 'solve_time_cdf'

CDF_SOLVABILITY = 'cdf_solvability'

QUANTILE_SOLVABILITY = 'quantile_solvability'

DIFFERENCE_OF_CDF_SOLVABILITY = 'difference_of_cdf_solvability'

DIFFERENCE_OF_QUANTILE_SOLVABILITY = 'difference_of_quantile_solvability'

AREA = 'area'

BOX = 'box'

VIOLIN = 'violin'

TERMINAL_SCATTER = 'terminal_scatter'

FEASIBILITY_SCATTER = 'feasibility_scatter'

FEASIBILITY_VIOLIN = 'feasibility_violin'

ALL_FEASIBILITY_PROGRESS = 'all_feasibility_progress'

MEAN_FEASIBILITY_PROGRESS = 'mean_feasibility_progress'

QUANTILE_FEASIBILITY_PROGRESS = 'quantile_feasibility_progress'

static from_str(label: str) → PlotType: Converts a string label to a PlotType enum.

simopt.experiment_base.bootstrap_procedure(experiments: list[list[ProblemSolver]], n_bootstraps: int, conf_level: float, plot_type: PlotType, beta: float | None = None, solve_tol: float | None = None, estimator: float | simopt.curve.Curve | None = None, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False) → tuple[float, float] | tuple[simopt.curve.Curve, simopt.curve.Curve]

Performs bootstrapping and computes confidence intervals for progress curves.

Parameters:

experiments (list[list[ProblemSolver]]) – Grid of problem-solver pairs.
n_bootstraps (int) – Number of bootstrap samples to generate.
conf_level (float) – Confidence level for the interval (0 < conf_level < 1).
plot_type (PlotType) – Type of plot/metric for which to compute the interval.
beta (float, optional) – Quantile level (0 < beta < 1), used with some plot types.
solve_tol (float, optional) – Relative optimality gap that defines a “solved” instance.
estimator (float or Curve, optional) – Reference estimator for difference plot types.
normalize (bool, optional) – Whether to normalize progress curves. Defaults to True.
feasibility_score_method (Literal["inf_norm", "norm"], optional) – Feasibility scoring rule used when plot_type corresponds to feasibility metrics.
feasibility_norm_degree (int, optional) – Degree of the norm when feasibility_score_method == "norm".
feasibility_two_sided (bool, optional) – Whether to assign a non-zero score to feasible solutions based on the best violation.

Returns:

Lower and upper bounds of the CI.

Return type:

tuple[float, float] or tuple[Curve, Curve]

simopt.experiment_base.functional_of_curves(bootstrap_curves: list[list[list[simopt.curve.Curve]]], plot_type: PlotType, beta: float | None = 0.5, solve_tol: float | None = 0.1) → float | simopt.curve.Curve

Computes a functional of bootstrapped objective or progress curves.

Parameters:

bootstrap_curves (list[list[list[Curve]]]) – Bootstrapped curves for all solutions across all macroreplications.
plot_type (PlotType) – Type of functional to compute: - PlotType.MEAN - PlotType.QUANTILE - PlotType.AREA_MEAN - PlotType.AREA_STD_DEV - PlotType.SOLVE_TIME_QUANTILE - PlotType.SOLVE_TIME_CDF - PlotType.CDF_SOLVABILITY - PlotType.QUANTILE_SOLVABILITY - PlotType.DIFFERENCE_OF_CDF_SOLVABILITY - PlotType.DIFFERENCE_OF_QUANTILE_SOLVABILITY
beta (float, optional) – Quantile level (0 < beta < 1). Defaults to 0.5.
solve_tol (float, optional) – Optimality gap for defining a solved instance (0 < solve_tol ≤ 1). Defaults to 0.1.

Returns:

The computed functional of the curves.

Return type:

Curve or float

Raises:

ValueError – If input values are invalid or unsupported for the given plot_type.

simopt.experiment_base.compute_bootstrap_conf_int(observations: list[float | int], conf_level: float, bias_correction: bool = True, overall_estimator: float | None = None) → tuple[numpy.ndarray, numpy.ndarray]

Construct a bootstrap confidence interval for an estimator.

Parameters:

observations (list[float | int]) – Estimators from all bootstrap instances.
conf_level (float) – Confidence level for confidence intervals, i.e., 1 - gamma; must be in (0, 1).
bias_correction (bool, optional) – Whether to use bias-corrected bootstrap CIs (via the percentile method). Defaults to True.
overall_estimator (float | None, optional) – The estimator to compute the CI around. Required if`bias_correction` is True.

Returns:

A tuple containing the lower and upper bounds of: the bootstrap confidence interval.

Return type:

tuple[np.ndarray, np.ndarray]

Raises:

ValueError – If conf_level is not in (0, 1), or if overall_estimator is None when bias_correction is True.

simopt.experiment_base.plot_bootstrap_conf_ints(bs_conf_int_lower_bounds: simopt.curve.Curve, bs_conf_int_upper_bounds: simopt.curve.Curve, color_str: str = 'C0') → None

Plot bootstrap confidence intervals.

Parameters:

bs_conf_int_lower_bounds (Curve) – Lower bounds of bootstrap confidence intervals, as curves.
bs_conf_int_upper_bounds (Curve) – Upper bounds of bootstrap confidence intervals, as curves.
color_str (str, optional) – String indicating line color, e.g., “C0”, “C1”, etc. Defaults to “C0”.

simopt.experiment_base.report_max_halfwidth(curve_pairs: list[list[simopt.curve.Curve]], normalize: bool, conf_level: float, difference: bool = False) → None

Print caption for the max halfwidth of bootstrap confidence interval curves.

Parameters:

curve_pairs (list[list[Curve]]) – A list of paired bootstrap CI curves.
normalize (bool) – Whether to normalize progress curves with respect to optimality gaps.
conf_level (float) – Confidence level for confidence intervals (must be in (0, 1)).
difference (bool, optional) – Whether the plot is for difference profiles. Defaults to False.

Raises:

ValueError – If conf_level is not in (0, 1) or if curve_pairs is empty.

simopt.experiment_base.check_common_problem_and_reference(experiments: list[ProblemSolver]) → None

Check if a collection of experiments share the same problem, x0, and x*.

Parameters:: experiments (list[ProblemSolver]) – Problem-solver pairs of different solvers on a common problem.
Raises:: ValueError – If any experiments have different problem instances, starting solutions (x0), or optimal solutions (x*).

simopt.experiment_base.plot_progress_curves(experiments: list[ProblemSolver], plot_type: PlotType, beta: float = 0.5, normalize: bool = True, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') → list[pathlib.Path]

Plots individual or aggregate progress curves for solvers on a single problem.

Parameters:

experiments (list[ProblemSolver]) – Problem-solver pairs for different solvers on the same problem.
plot_type (PlotType) – Type of plot to produce (ALL, MEAN, or QUANTILE).
beta (float, optional) – Quantile level to plot (0 < beta < 1). Defaults to 0.50.
normalize (bool, optional) – If True, normalize curves by optimality gaps. Defaults to True.
all_in_one (bool, optional) – If True, plot all curves in one figure. Defaults to True.
n_bootstraps (int, optional) – Number of bootstrap samples. Defaults to 100.
conf_level (float, optional) – Confidence level for CIs (0 < conf_level < 1). Defaults to 0.95.
plot_conf_ints (bool, optional) – If True, plot bootstrapped confidence intervals. Defaults to True.
print_max_hw (bool, optional) – If True, print caption with max half-width. Defaults to True.
plot_title (str, optional) – Custom title for the plot (used only if all_in_one=True).
legend_loc (str, optional) – Location of legend (e.g., “best”, “lower right”).
ext (str, optional) – File extension for saved plots (e.g., “.png”). Defaults to “.png”.
save_as_pickle (bool, optional) – If True, save plot as a pickle file. Defaults to False.
solver_set_name (str, optional) – Label for solver group in plot titles. Defaults to “SOLVER_SET”.

Returns:

List of file paths where the plots were saved.

Return type:

list[Path]

Raises:

ValueError – If beta, conf_level, or n_bootstraps have invalid values.

simopt.experiment_base.plot_solvability_cdfs(experiments: list[ProblemSolver], solve_tol: float = 0.1, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') → list[pathlib.Path]

Plots solvability CDFs for one or more solvers on a single problem.

Parameters:

experiments (list[ProblemSolver]) – Problem-solver pairs for different solvers on a common problem.
solve_tol (float, optional) – Optimality gap that defines when a problem is considered solved (0 < solve_tol ≤ 1). Defaults to 0.1.
all_in_one (bool, optional) – If True, plot all curves together. Defaults to True.
n_bootstraps (int, optional) – Number of bootstrap samples. Defaults to 100.
conf_level (float, optional) – Confidence level for intervals (0 < conf_level < 1). Defaults to 0.95.
plot_conf_ints (bool, optional) – If True, include bootstrapped confidence intervals. Defaults to True.
print_max_hw (bool, optional) – If True, print the max half-width in the caption. Defaults to True.
plot_title (str, optional) – Custom title to override the generated one (used only if all_in_one is True).
legend_loc (str, optional) – Location of the plot legend (e.g., “best”).
ext (str, optional) – File extension for saved plots. Defaults to “.png”.
save_as_pickle (bool, optional) – If True, save plots as pickle files. Defaults to False.
solver_set_name (str, optional) – Label for solver group in plot titles. Defaults to “SOLVER_SET”.

Returns:

List of file paths for the generated plots.

Return type:

list[Path]

Raises:

ValueError – If any input parameter is out of bounds or invalid.

simopt.experiment_base.plot_area_scatterplots(experiments: list[list[ProblemSolver]], all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str = 'best', ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') → list[pathlib.Path]

Plots scatterplots of mean vs. standard deviation of area under progress curves.

Can generate either one plot per solver or a combined plot for all solvers.

Note

The print_max_hw flag is currently not implemented.

Parameters:

experiments (list[list[ProblemSolver]]) – Problem-solver pairs used for plotting.
all_in_one (bool, optional) – If True, plot all solvers together. Defaults to True.
n_bootstraps (int, optional) – Number of bootstrap samples. Defaults to 100.
conf_level (float, optional) – Confidence level for CIs (0 < conf_level < 1). Defaults to 0.95.
plot_conf_ints (bool, optional) – If True, show bootstrapped confidence intervals. Defaults to True.
print_max_hw (bool, optional) – Placeholder for printing max half-widths. Currently unused.
plot_title (str, optional) – Custom title for the plot (applies only if all_in_one=True).
legend_loc (str, optional) – Location of the legend (e.g., “best”, “lower right”).
ext (str, optional) – File extension for saved plots. Defaults to “.png”.
save_as_pickle (bool, optional) – If True, save plot as a pickle file. Defaults to False.
solver_set_name (str, optional) – Label for solver group in plot titles. Defaults to “SOLVER_SET”.
problem_set_name (str, optional) – Label for problem group in plot titles. Defaults to “PROBLEM_SET”.

Returns:

List of file paths for the plots produced.

Return type:

list[Path]

Raises:

ValueError – If n_bootstraps is not positive or conf_level is outside (0, 1).

simopt.experiment_base.plot_solvability_profiles(experiments: list[list[ProblemSolver]], plot_type: PlotType, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, solve_tol: float = 0.1, beta: float = 0.5, ref_solver: str | None = None, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') → list[pathlib.Path]

Plots solvability or difference profiles for solvers on multiple problems.

Parameters:

experiments (list[list[ProblemSolver]]) – Problem-solver pairs used for plotting.
plot_type (PlotType) – Type of solvability plot to produce: - PlotType.CDF_SOLVABILITY - PlotType.QUANTILE_SOLVABILITY - PlotType.DIFFERENCE_OF_CDF_SOLVABILITY - PlotType.DIFFERENCE_OF_QUANTILE_SOLVABILITY
all_in_one (bool, optional) – If True, plot all curves together. Defaults to True.
n_bootstraps (int, optional) – Number of bootstrap samples. Defaults to 100.
conf_level (float, optional) – Confidence level for intervals (0 < conf_level < 1). Defaults to 0.95.
plot_conf_ints (bool, optional) – If True, show bootstrapped confidence intervals. Defaults to True.
print_max_hw (bool, optional) – If True, print max half-width in caption. Defaults to True.
solve_tol (float, optional) – Optimality gap defining when a problem is considered solved (0 < solve_tol ≤ 1). Defaults to 0.1.
beta (float, optional) – Quantile level to compute (0 < beta < 1). Defaults to 0.5.
ref_solver (str, optional) – Name of the reference solver for difference plots.
plot_title (str, optional) – Custom title for the plot (used only if all_in_one=True).
legend_loc (str, optional) – Location of the legend (e.g., “best”, “upper right”).
ext (str, optional) – File extension for saved plots (e.g., “.png”). Defaults to “.png”.
save_as_pickle (bool, optional) – If True, save plots as pickle files. Defaults to False.
solver_set_name (str, optional) – Name of solver group for plot titles. Defaults to “SOLVER_SET”.
problem_set_name (str, optional) – Name of problem group for plot titles. Defaults to “PROBLEM_SET”.

Returns:

List of file paths for the plots produced.

Return type:

list[Path]

Raises:

ValueError – If any input parameter is out of bounds or invalid.

simopt.experiment_base.plot_terminal_progress(experiments: list[ProblemSolver], plot_type: PlotType = PlotType.VIOLIN, normalize: bool = True, all_in_one: bool = True, plot_title: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') → list[pathlib.Path]

Plots terminal progress as box or violin plots for solvers on a single problem.

Parameters:

experiments (list[ProblemSolver]) – ProblemSolver pairs for different solvers on a common problem.
plot_type (str, optional) – Type of plot to generate: “box” or “violin”. Defaults to “violin”.
normalize (bool, optional) – If True, normalize progress curves by optimality gaps. Defaults to True.
all_in_one (bool, optional) – If True, plot all curves in one figure. Defaults to True.
plot_title (str, optional) – Custom title to override the default. Used only if all_in_one is True.
ext (str, optional) – File extension for saved plots (e.g., “.png”). Defaults to “.png”.
save_as_pickle (bool, optional) – If True, save the plot as a pickle file. Defaults to False.
solver_set_name (str, optional) – Label for solver group in plot titles. Defaults to “SOLVER_SET”.

Returns:

List of file paths for the plots produced.

Return type:

list[str]

Raises:

ValueError – If an unsupported plot type is specified.

simopt.experiment_base.plot_terminal_scatterplots(experiments: list[list[ProblemSolver]], all_in_one: float = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') → list[pathlib.Path]

Plot scatter plots of the mean and standard deviation of terminal progress.

Either creates one plot per solver or a combined plot for all solvers.

Parameters:

experiments (list[list[ProblemSolver]]) – Problem-solver pairs used to produce plots.
all_in_one (bool, optional) – Whether to plot all solvers in one figure. Defaults to True.
plot_title (str | None, optional) – Title to override the autogenerated one (only applies if all_in_one is True).
legend_loc (str | None, optional) – Location of the legend (e.g., “best”). Defaults to None.
ext (str, optional) – File extension to use for output images. Defaults to “.png”.
save_as_pickle (bool, optional) – Whether to also save the plots as .pickle files. Defaults to False.
solver_set_name (str, optional) – Name for the solver group used in plot titles. Defaults to “SOLVER_SET”.
problem_set_name (str, optional) – Name for the problem group used in plot titles. Defaults to “PROBLEM_SET”.

Returns:

A list of file paths to the plots produced.

Return type:

list[Path]

simopt.experiment_base.setup_plot(plot_type: PlotType, solver_name: str = 'SOLVER SET', problem_name: str = 'PROBLEM SET', normalize: bool = True, budget: int | None = None, beta: float | None = None, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, solve_tol: float | None = None, plot_title: str | None = None) → None

Create a new figure, add labels to the plot, and reformat axes.

Parameters:

plot_type (PlotType) – Type of plot to produce. Valid options include: - ALL: All estimated progress curves. - MEAN: Estimated mean progress curve. - QUANTILE: Estimated beta quantile progress curve. - SOLVE_TIME_CDF: CDF of solve time. - CDF_SOLVABILITY: CDF solvability profile. - QUANTILE_SOLVABILITY: Quantile solvability profile. - DIFFERENCE_OF_CDF_SOLVABILITY: Difference of CDF solvability profiles. - DIFFERENCE_OF_QUANTILE_SOLVABILITY: Difference of quantile solvability profiles. - AREA: Area scatterplot. - BOX: Box plot of terminal progress. - VIOLIN: Violin plot of terminal progress. - TERMINAL_SCATTER: Scatterplot of mean and std dev of terminal progress.
solver_name (str, optional) – Name of the solver. Defaults to “SOLVER SET”.
problem_name (str, optional) – Name of the problem. Defaults to “PROBLEM SET”.
normalize (bool, optional) – Whether to normalize with respect to optimality gaps. Defaults to True.
budget (int, optional) – Function evaluation budget.
beta (float, optional) – Quantile to compute (must be in (0, 1)).
feasibility_score_method (Literal["inf_norm", "norm"], optional) – Method to compute the feasibility score. Defaults to “inf_norm”.
feasibility_norm_degree (int, optional) – Degree of the norm to use for the feasibility score. Defaults to 1.
solve_tol (float, optional) – Relative optimality gap for declaring a solve (must be in (0, 1]).
plot_title (str, optional) – Title to override the automatically generated one.

Raises:

ValueError – If any inputs are invalid.

simopt.experiment_base.save_plot(solver_name: str, problem_name: str, plot_type: PlotType, normalize: bool, extra: float | list[float] | None = None, plot_title: str | None = None, ext: str = '.png', save_as_pickle: bool = False) → pathlib.Path

Create and save a plot with appropriate labels and formatting.

Parameters:

solver_name (str) – Name of the solver.
problem_name (str) – Name of the problem.
plot_type (PlotType) – Type of plot to produce. Valid options include: - ALL: All estimated progress curves. - MEAN: Estimated mean progress curve. - QUANTILE: Estimated beta quantile progress curve. - SOLVE_TIME_CDF: CDF of solve time. - CDF_SOLVABILITY: CDF solvability profile. - QUANTILE_SOLVABILITY: Quantile solvability profile. - DIFFERENCE_OF_CDF_SOLVABILITY: Difference of CDF solvability profiles. - DIFFERENCE_OF_QUANTILE_SOLVABILITY: Difference of quantile solvability profiles. - AREA: Area scatterplot. - TERMINAL_SCATTER: Scatterplot of mean and std dev of terminal progress. - FEASIBILITY_SCATTER: Scatterplot of terminal objective vs feasibility. - FEASIBILITY_VIOLIN: Violin plot of terminal feasibility. - ALL_FEASIBILITY_PROGRESS: Feasibility progress curves for all macroreps. - MEAN_FEASIBILITY_PROGRESS: Mean feasibility progress curve. - QUANTILE_FEASIBILITY_PROGRESS: Quantile feasibility progress curve.
normalize (bool) – Whether to normalize with respect to optimality gaps.
extra (float | list[float], optional) – Extra number(s) specifying quantile (e.g., beta) and/or solve tolerance.
plot_title (str | None, optional) – If provided, overrides the default title and filename.
ext (str, optional) – File extension for the saved plot. Defaults to “.png”.
save_as_pickle (bool, optional) – Whether to save the plot as a pickle file. Defaults to False.

Returns:

Path pointing to the location where the plot will be saved.

Return type:

Path

simopt.experiment_base.plot_terminal_feasibility(experiments: list[list[ProblemSolver]], plot_type: Literal['scatter', 'violin'] = 'scatter', score_type: Literal['inf_norm', 'norm'] = 'inf_norm', two_sided: bool = True, plot_zero: bool = True, plot_optimal: bool = True, norm_degree: int = 1, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, bias_correction: bool = True, solver_set_name: str = 'SOLVER_SET', plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False) → list[str]

Plot the feasibility of one solver problem pair. (for now).

Parameters:

experiments (list [list [experiment_base.ProblemSolver]]) – Problem-solver pairs used to produce plots.
plot_type (str, default = 'scatter') –

String indicating which type of plot to produce:
”scatter” : scatter plot with terminal objective on x-axis and terminal feasiblity score on y-axis for all macroreps “violin” : violin plot showing density of terminal feasiblity scores for all macroreps
score_type (str, default = "inf_norm") – “inf_norm” : use infinite norm of lhs of violated constraints to calculate feasiblity score “norm” : use “norm_degree” to specify degree of norm of lhs of violated constriants
two_sided (bool, default = "True") – Two-sided or one-sided feasiblity score
plot_zero (bool, default = True) – Plot a dashed red line a feasiblity score = 0
plot_optimal (default = True) – Plot a dashed red line at beast feasible objective across all postreplications
norm_degree (int, default = 1) – if not using inf_norm, specifies degree of norm taken of lhs of violated constraints for feasibility score
all_in_one (bool, default = True) – plot all solvers on same plot
n_bootstraps (int, default = 100) – number of bootsrap replications for CI construction
conf_level (float, default = 0.95) – confidence level of created CI
plot_conf_ints (bool, default = True) – plot CI’s for each maroreplication
bias_correction (bool, default = True) – use bias correction for CI construction
solver_set_name (str, default = "SOLVER_SET") – Override solver names in plot, only applies if all_in_one = True
plot_title (str, optional) – Optional title to override the one that is autmatically generated
legend_loc (str, default="best") – specificies location of legend
ext (str, default = '.png') – Extension to add to image file path to change file type
save_as_pickle (bool, default = False) – True if plot should be saved to pickle file, False otherwise.
Returns
-------
file_list (list [str]) – List compiling path names for plots produced.
Raises
------
TypeError
ValueError

simopt.experiment_base.plot_feasibility_progress(experiments: list[list[ProblemSolver]], plot_type: PlotType = PlotType.ALL_FEASIBILITY_PROGRESS, score_type: Literal['inf_norm', 'norm'] = 'inf_norm', norm_degree: int = 1, two_sided: bool = True, plot_zero: bool = True, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, beta: float = 0.5, solver_set_name: str = 'SOLVER_SET', plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False) → list[str]

Plot feasibility over solver progress.

Parameters:

experiments (list [list [experiment_base.ProblemSolver]]) – Problem-solver pairs used to produce plots.
plot_type (str, default = 'scatter') –

String indicating which type of plot to produce:
”all” : show all macroreps “mean” : plot mean of all macroreps “quantile” : plot quantile of all macroreps
score_type (str, default = "inf_norm") – “inf_norm” : use infinite norm of lhs of violated constraints to calculate feasiblity score “norm” : use “norm_degree” to specify degree of norm of lhs of violated constriants
two_sided (bool, default = "True") – Two-sided or one-sided feasiblity score. Two-sided only supported for plot_type = “all”
plot_zero (bool, default = True) – Plot a dashed red line a feasiblity score = 0
norm_degree (int, default = 1) – if not using inf_norm, specifies degree of norm taken of lhs of violated constraints for feasibility score
all_in_one (bool, default = True) – plot all solvers on same plot
n_bootstraps (int, default = 100) – number of bootsrap replications for CI construction
conf_level (float, default = 0.95) – confidence level of created CI
plot_conf_ints (bool, default = True) – plot CI’s for each maroreplication
print_max_hw (bool, default = True) – report max halfwidth for CI’s’
beta (float, default = .5) – quantile to computue must be between 0 and 1
solver_set_name (str, default = "SOLVER_SET") – Override solver names in plot, only applies if all_in_one = True
plot_title (str, optional) – Optional title to override the one that is autmatically generated
legend_loc (str, default="best") – specificies location of legend
ext (str, default = '.png') – Extension to add to image file path to change file type
save_as_pickle (bool, default = False) – True if plot should be saved to pickle file, False otherwise.
Returns
-------
file_list (list [str]) – List compiling path names for plots produced.
Raises
------
TypeError
ValueError

class simopt.experiment_base.ProblemsSolvers(solver_factors: list[dict] | None = None, problem_factors: list[dict] | None = None, solver_names: list[str] | None = None, problem_names: list[str] | None = None, solver_renames: list[str] | None = None, problem_renames: list[str] | None = None, fixed_factors_filename: str | None = None, solvers: list[simopt.base.Solver] | None = None, problems: list[simopt.base.Problem] | None = None, experiments: list[list[ProblemSolver]] | None = None, file_name_path: pathlib.Path | None = None, create_pair_pickles: bool = False, experiment_name: str | None = None)

Base class for running one or more solver on one or more problem.

Initialize a ProblemsSolvers object.

There are three ways to initialize a ProblemsSolvers object: 1. Provide the names of solvers and problems (for lookup in directory.py). 2. Provide lists of solver and problem objects to pair directly. 3. Provide a full list of ProblemSolver objects (as nested lists).

Parameters:

solver_factors (list[dict] | None) – List of solver factor dictionaries, one per design point. Requires solver_names to match the number of entries.
problem_factors (list[dict] | None) – List of problem/model factor dictionaries, one per design point. Requires problem_names to match the number of entries.
solver_names (list[str] | None) – List of solver names to look up.
problem_names (list[str] | None) – List of problem names to look up.
solver_renames (list[str] | None) – User-specified labels for solvers.
problem_renames (list[str] | None) – User-specified labels for problems.
fixed_factors_filename (str | None) – Name of a .py file containing fixed factor dictionaries.
solvers (list[Solver] | None) – List of Solver objects to use directly.
problems (list[Problem] | None) – List of Problem objects to use directly.
experiments (list[list[ProblemSolver]] | None) – Explicit problem-solver pairings.
file_name_path (Path | None) – Output path for saving the ProblemsSolvers object.
create_pair_pickles (bool) – Whether to create individual .pickle files for each problem-solver pair.
experiment_name (str | None) – Optional name to prefix output files.

property solver_names: list[str]: List of solver names.

property n_solvers: int: Number of solvers.

property problem_names: list[str]: List of problem names.

property n_problems: int: Number of problems.

property solvers: list[simopt.base.Solver]: List of solvers.

property problems: list[simopt.base.Problem]: List of problems.

property all_solver_fixed_factors: dict[str, dict]: Fixed solver factors for each solver.

property all_problem_fixed_factors: dict[str, dict]: Fixed problem factors for each problem.

property all_model_fixed_factors: dict[str, dict]: Fixed model factors for each problem.

property experiments: list[list[ProblemSolver]]: All problem-solver pairs.

property file_name_path: pathlib.Path: Path to the .pickle file for saving the ProblemsSolvers object.

property create_pair_pickles: bool: Whether to create pickle files for each problem-solver pair.

property experiment_name: str: Name of experiment to be appended to the beginning of output files.

check_compatibility() → str

Check whether all experiments’ solvers and problems are compatible.

Returns:: Error message in the event any problem and solver are incompatible.
Return type:: str

run(n_macroreps: int) → None

Run n_macroreps of each solver on each problem.

Parameters:: n_macroreps (int) – Number of macroreplications to run per problem-solver pair.
Raises:: ValueError – If n_macroreps is not positive.

post_replicate(n_postreps: int, crn_across_budget: bool = True, crn_across_macroreps: bool = False) → None

Runs postreplications for each problem-solver pair on all macroreplications.

Parameters:

n_postreps (int) – Number of postreplications per recommended solution.
crn_across_budget (bool, optional) – If True, use CRN across solutions from different time budgets. Defaults to True.
crn_across_macroreps (bool, optional) – If True, use CRN across solutions from different macroreplications. Defaults to False.

Raises:

ValueError – If n_postreps is not positive.

post_normalize(n_postreps_init_opt: int, crn_across_init_opt: bool = True) → None

Builds objective and progress curves for all experiment collections.

Parameters:

n_postreps_init_opt (int) – Number of postreplications at initial (x0) and optimal (x*) solutions.
crn_across_init_opt (bool, optional) – If True, use CRN for postreplications at x0 and x*. Defaults to True.

Raises:

ValueError – If n_postreps_init_opt is not positive.

check_postreplicate() → bool

Checks whether all experiments have been postreplicated.

Returns:: Whether all experiments have been postreplicated
Return type:: bool

check_postnormalize() → bool

Checks whether all experiments have been postnormalized.

Returns:: Whether all experiments have been postnormalized.
Return type:: bool

record_group_experiment_results() → None: Saves a ProblemsSolvers object to a .pickle file in the outputs directory.

log_group_experiment_results() → None

Creates a .txt summary of solvers and problems in the ProblemSolvers object.

The file is saved in the ‘logs/’ folder next to the experiment’s pickle file.

report_group_statistics(solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') → None

Reports statistics for all solvers across all problems.

Parameters:

solve_tols (list[float], optional) – Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50].
csv_filename (str, optional) – Name of the output CSV file (without ‘.csv’). Defaults to “df_solver_results”.

Raises:

ValueError – If any solve tolerance is not in the range (0, 1].

report_statistics(pair_list: list[ProblemSolver], solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') → None

Calculates statistics from macroreplications and saves results to a CSV file.

Parameters:

pair_list (list[ProblemSolver]) – List of ProblemSolver objects.
solve_tols (list[float], optional) – Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50].
csv_filename (str, optional) – Name of the CSV file to write results to. Defaults to “df_solver_results”.

Raises:

ValueError – If any solve tolerance is not in the range (0, 1].

simopt.experiment_base.read_group_experiment_results(file_path: pathlib.Path | str) → ProblemsSolvers

Reads a ProblemsSolvers object from a .pickle file.

Parameters:

file_path (Path) – Path to the .pickle file.

Returns:

A group of problem-solver experiments that were run: or post-processed.

Return type:

ProblemsSolvers

Raises:

FileNotFoundError – If the file does not exist.

simopt.experiment_base.find_unique_solvers_problems(experiments: list[ProblemSolver]) → tuple[list[simopt.base.Solver], list[simopt.base.Problem]]

Finds unique solvers and problems from a list of ProblemSolver experiments.

Parameters:

experiments (list[ProblemSolver]) – List of problem-solver pairs.

Returns:

A tuple containing:

A list of unique solvers.
A list of unique problems.

Return type:

tuple[list[Solver], list[Problem]]

simopt.experiment_base.find_missing_experiments(experiments: list[ProblemSolver]) → tuple[list[simopt.base.Solver], list[simopt.base.Problem], list[tuple[simopt.base.Solver, simopt.base.Problem]]]

Finds missing problem-solver pairs from a list of experiments.

Parameters:

experiments (list[ProblemSolver]) – List of problem-solver pairs.

Returns:

A tuple containing:

list[Solver]: Unique solvers present in the experiments.
list[Problem]: Unique problems present in the experiments.
list[tuple[Solver, Problem]]: Problem-solver pairs that are missing.

Return type:

tuple

simopt.experiment_base.make_full_metaexperiment(existing_experiments: list[ProblemSolver], unique_solvers: list[simopt.base.Solver], unique_problems: list[simopt.base.Problem], missing_experiments: list[tuple[simopt.base.Solver, simopt.base.Problem]]) → ProblemsSolvers

Creates experiments for missing problem-solver pairs.

Parameters:

existing_experiments (list[ProblemSolver]) – Existing problem-solver experiments.
unique_solvers (list[Solver]) – Solvers present in the existing experiments.
unique_problems (list[Problem]) – Problems present in the existing experiments.
missing_experiments (list[tuple[Solver, Problem]]) – Problem-solver pairs that have not yet been run.

Returns:

A new ProblemsSolvers object containing the completed set.

Return type:

ProblemsSolvers

simopt.experiment_base.create_design_list_from_table(design_table: pandas.DataFrame) → list[dict[str, Any]]

Create a list of solver or problem objects for each design point.

Parameters:: design_table (DataFrame) – DataFrame containing the design table. Each row represents a design point, and each column represents a factor.
Returns:: List of dictionaries, where each list entry corresponds to a design point, and each dictionary contains the name and values for each factor in that design point.
Return type:: list[dict[str, Any]]

simopt.experiment_base.create_design(name: str, factor_headers: list[str], factor_settings: list[tuple[float, float, int]] | pathlib.Path, fixed_factors: dict | None = None, cross_design_factors: dict | None = None, design_type: Literal['nolhs'] = 'nolhs', n_stacks: int = 1) → list[dict[str, Any]]

Creates a design of solver, problem, or model factors.

Please ensure the indexing of the factor_headers argument matches the indexing of the factor_settings argument.

Parameters:

name (str) – Name of the solver, problem, or model.
factor_headers (list[str]) – Names of factors that vary in the design.
factor_settings (list[tuple[float, float, int]] | Path) – A list of tuples, each of the form (min, max, # decimals) or a Path to a .txt file containing those factor settings.
fixed_factors (dict, optional) – Dictionary of fixed factor values that override defaults.
cross_design_factors (dict, optional) – Dictionary of lists of cross-design factor values. Defaults to None.
design_type (Literal["nolhs"], optional) – Type of design. Defaults to “nolhs”.
n_stacks (int, optional) – Number of stacks. Defaults to 1.

Returns:

A list of dictionaries, where each dictionary represents: a design.

Return type:

list[dict[str, Any]]

Raises:

ValueError – If input validation fails.
Exception – If the design type is unsupported.