simopt.experiment_base ====================== .. py:module:: simopt.experiment_base .. autoapi-nested-parse:: Base classes for problem-solver pairs and I/O/plotting helper functions. Module Contents --------------- .. py:data:: model_directory .. py:data:: problem_directory .. py:data:: solver_directory .. py:data:: SIMOPT_TOPLEVEL .. py:data:: EXPERIMENT_DIR .. py:class:: ProblemSolver(solver_name: str | None = None, problem_name: str | None = None, solver_rename: str | None = None, problem_rename: str | None = None, solver: simopt.base.Solver | None = None, problem: simopt.base.Problem | None = None, solver_fixed_factors: dict | None = None, problem_fixed_factors: dict | None = None, model_fixed_factors: dict | None = None, file_name_path: pathlib.Path | str | None = None, create_pickle: bool = True) Base class for running one solver on one problem. Initializes a ProblemSolver object. You can either: 1. Provide solver and problem names to look them up via `directory.py`, or 2. Provide solver and problem objects directly. :param solver_name: Name of the solver. :type solver_name: str, optional :param problem_name: Name of the problem. :type problem_name: str, optional :param solver_rename: User-defined name for the solver. :type solver_rename: str, optional :param problem_rename: User-defined name for the problem. :type problem_rename: str, optional :param solver: Simulation-optimization solver object. :type solver: Solver, optional :param problem: Simulation-optimization problem object. :type problem: Problem, optional :param solver_fixed_factors: Fixed solver parameters. :type solver_fixed_factors: dict, optional :param problem_fixed_factors: Fixed problem parameters. :type problem_fixed_factors: dict, optional :param model_fixed_factors: Fixed model parameters. :type model_fixed_factors: dict, optional :param file_name_path: Path to save a pickled ProblemSolver object. :type file_name_path: Path | str, optional :param create_pickle: Whether to save the object as a pickle file. :type create_pickle: bool, optional .. py:property:: solver :type: simopt.base.Solver Simulation-optimization solver. .. py:property:: problem :type: simopt.base.Problem Simulation-optimization problem. .. py:property:: n_macroreps :type: int Number of macroreplications run. .. py:property:: file_name_path :type: pathlib.Path Path of .pickle file for saving ProblemSolver object. .. py:property:: all_recommended_xs :type: list[list[tuple]] Sequences of recommended solutions from each macroreplication. .. py:property:: all_intermediate_budgets :type: list[list] Sequences of intermediate budgets from each macroreplication. .. py:property:: timings :type: list[float] Runtimes (in seconds) for each macroreplication. .. py:property:: n_postreps :type: int Number of postreps to take at each recommended solution. .. py:property:: crn_across_budget :type: bool Whether CRN is used across solutions recommended at different times. .. py:property:: crn_across_macroreps :type: bool Whether CRN is used across solutions from different macroreplications. .. py:property:: all_post_replicates :type: list[list[list]] All post-replicates from all solutions from all macroreplications. .. py:property:: all_est_objectives :type: list[list[float]] Estimated objective values of all solutions from all macroreplications. .. py:property:: n_postreps_init_opt :type: int Number of postreplications at initial (x0) and optimal (x*) solutions. .. py:property:: crn_across_init_opt :type: bool Whether CRN is used for postreplications at x0 and x* solutions. .. py:property:: x0 :type: tuple Initial solution (x0). .. py:property:: x0_postreps :type: list Post-replicates at x0. .. py:property:: xstar :type: tuple Proxy for optimal solution (x*). .. py:property:: xstar_postreps :type: list Post-replicates at x*. .. py:property:: objective_curves :type: list[simopt.curve.Curve] Estimated objective function curves, one per macroreplication. .. py:property:: progress_curves :type: list[simopt.curve.Curve] Progress curves, one for each macroreplication. .. py:property:: has_run :type: bool True if the solver has been run on the problem, otherwise False. .. py:property:: has_postreplicated :type: bool True if the solver has been postreplicated, otherwise False. .. py:property:: has_postnormalized :type: bool True if the solver has been postprocessed, otherwise False. .. py:attribute:: create_pickle :value: True .. py:attribute:: all_stoch_constraints :value: [] .. py:attribute:: all_est_lhs :value: [] .. py:attribute:: feasibility_curves :value: [] .. py:method:: model_created(model: simopt.base.Model) -> None Hook called after the problem's model is instantiated. :param model: The initialized model associated with the experiment's problem. This is a helper function to customize the experiment's input model. .. py:method:: before_replicate(model: simopt.base.Model, rng_list: list[mrg32k3a.mrg32k3a.MRG32k3a]) -> None Hook executed immediately before each replication during an experiment. :param model: The model about to be simulated. :param rng_list: The list of RNGs used for the replication. This is a helper function to customize behavior before each replication. .. py:method:: check_compatibility() -> str Check whether the experiment's solver and problem are compatible. :returns: Error message in the event problem and solver are incompatible. :rtype: str .. py:method:: run(n_macroreps: int, n_jobs: int = -1) -> None Runs the solver on the problem for a given number of macroreplications. .. note:: RNGs for random problem instances are reserved but currently unused. This method is under development. :param n_macroreps: Number of macroreplications to run. :type n_macroreps: int :param n_jobs: Number of jobs to run in parallel. Defaults to -1. -1: use all available cores 1: run sequentially :type n_jobs: int, optional :raises ValueError: If `n_macroreps` is not positive. .. py:method:: run_multithread(mrep: int, solver: simopt.base.Solver, problem: simopt.base.Problem) -> tuple Runs one macroreplication of the solver on the problem. :param mrep: Index of the macroreplication. :type mrep: int :param solver: The simulation-optimization solver to run. :type solver: Solver :param problem: The problem to solve. :type problem: Problem :returns: A tuple containing: - int: Macroreplication index. - list: Recommended solutions. - list: Intermediate budgets. - float: Runtime for the macroreplication. :rtype: tuple :raises ValueError: If `mrep` is negative. .. py:method:: post_replicate(n_postreps: int, crn_across_budget: bool = True, crn_across_macroreps: bool = False) -> None Runs postreplications at the solver's recommended solutions. :param n_postreps: Number of postreplications at each recommended solution. :type n_postreps: int :param crn_across_budget: If True, use CRN across solutions from different time budgets. Defaults to True. :type crn_across_budget: bool, optional :param crn_across_macroreps: If True, use CRN across solutions from different macroreplications. Defaults to False. :type crn_across_macroreps: bool, optional :raises ValueError: If `n_postreps` is not positive. .. py:method:: post_replicate_multithread(mrep: int) -> tuple Runs postreplications for a given macroreplication's recommended solutions. :param mrep: Index of the macroreplication. :type mrep: int :returns: A tuple containing: - int: Macroreplication index. - list: Postreplicates for each recommended solution. - float: Runtime for the macroreplication. :rtype: tuple :raises ValueError: If `mrep` is negative. .. py:method:: bootstrap_sample(bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False, disable_macrorep_bootstrap: bool = False) -> tuple[list[simopt.curve.Curve], list[simopt.curve.Curve]] Generates bootstrap samples of objective/progress and feasibility curves. :param bootstrap_rng: Random number generator used for bootstrapping. :type bootstrap_rng: MRG32k3a :param normalize: If True, normalize progress curves with respect to optimality gaps. Defaults to True. :type normalize: bool, optional :param feasibility_score_method: Feasibility scoring method. Defaults to "inf_norm". :type feasibility_score_method: Literal["inf_norm", "norm"], optional :param feasibility_norm_degree: Degree of the norm when :type feasibility_norm_degree: int, optional :param ``feasibility_score_method == "norm"``. Defaults to 1.: :param feasibility_two_sided: Whether to award a non-zero score to feasible solutions based on the best violation. :type feasibility_two_sided: bool, optional :param disable_macrorep_bootstrap: Whether to disable bootstrap across macroreplications. Defaults to False. :type disable_macrorep_bootstrap: bool, optional :returns: Bootstrapped progress curves and feasibility curves for all macroreplications. :rtype: tuple[list[Curve], list[Curve]] .. py:method:: bootstrap_terminal_objective_and_feasibility(bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = True) -> tuple[list[float], list[float]] Bootstraps terminal objective and feasibility scores. .. py:method:: feasibility_score_history(feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = True) -> None Compute feasibility score history. .. py:method:: record_experiment_results(file_name: str) -> None Saves the ProblemSolver object to a .pickle file. :param file_name: Name of the pickle file. It is saved under the EXPERIMENT_DIR path. :type file_name: str .. py:method:: log_experiment_results(print_solutions: bool = True) -> None Creates a readable .txt log file from a problem-solver pair's .pickle file. :param print_solutions: If True, include recommended solutions in the .txt file. Defaults to True. :type print_solutions: bool, optional .. py:function:: trim_solver_results(problem: simopt.base.Problem, recommended_solutions: list[simopt.base.Solution], intermediate_budgets: list[int]) -> tuple[list[simopt.base.Solution], list[int]] Trims solver-recommended solutions beyond the problem's maximum budget. :param problem: The problem the solver was run on. :type problem: Problem :param recommended_solutions: Solutions recommended by the solver. :type recommended_solutions: list[Solution] :param intermediate_budgets: Budgets at which solutions were recommended. :type intermediate_budgets: list[int] :returns: A tuple containing: - list[Solution]: Trimmed list of recommended solutions. - list[int]: Trimmed list of corresponding intermediate budgets. :rtype: tuple .. py:function:: read_experiment_results(file_name_path: pathlib.Path | str) -> ProblemSolver Reads a ProblemSolver object from a .pickle file. :param file_name_path: Path to the .pickle file. :type file_name_path: Path | str :returns: Loaded problem-solver pair that was previously run or post-processed. :rtype: ProblemSolver :raises ValueError: If the file does not exist. .. py:function:: post_normalize(experiments: list[ProblemSolver], n_postreps_init_opt: int, crn_across_init_opt: bool = True, proxy_init_val: float | None = None, proxy_opt_val: float | None = None, proxy_opt_x: tuple | None = None, create_pair_pickles: bool = False) -> None Constructs objective and normalized progress curves for a set of experiments. :param experiments: Problem-solver pairs for different solvers on the same problem. :type experiments: list[ProblemSolver] :param n_postreps_init_opt: Number of postreplications at initial (x0) and optimal (x*) solutions. :type n_postreps_init_opt: int :param crn_across_init_opt: If True, use CRN for postreplications at x0 and x*. Defaults to True. :type crn_across_init_opt: bool, optional :param proxy_init_val: Known objective value of the initial solution. :type proxy_init_val: float, optional :param proxy_opt_val: Proxy or bound for the optimal objective value. :type proxy_opt_val: float, optional :param proxy_opt_x: Proxy for the optimal solution. :type proxy_opt_x: tuple, optional :param create_pair_pickles: If True, create a pickle file for each problem-solver pair. Defaults to False. :type create_pair_pickles: bool, optional .. py:function:: bootstrap_sample_all(experiments: list[list[ProblemSolver]], bootstrap_rng: mrg32k3a.mrg32k3a.MRG32k3a, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False) -> tuple[list[list[list[simopt.curve.Curve]]], list[list[list[simopt.curve.Curve]]]] Generates bootstrap samples of progress and feasibility curves. :param experiments: Grid of problem-solver pairs, where each inner list corresponds to different problems for a given solver. :type experiments: list[list[ProblemSolver]] :param bootstrap_rng: Random number generator used for bootstrapping. :type bootstrap_rng: MRG32k3a :param normalize: If True, normalize progress curves by optimality gaps. Defaults to True. :type normalize: bool, optional :param feasibility_score_method: Feasibility scoring rule. :type feasibility_score_method: Literal["inf_norm", "norm"], optional :param feasibility_norm_degree: Degree of the norm when ``feasibility_score_method == "norm"``. :type feasibility_norm_degree: int, optional :param feasibility_two_sided: Whether to give feasible solutions a non-zero score based on the best violation. :type feasibility_two_sided: bool, optional :returns: Bootstrapped progress/objective curves and feasibility curves for all solutions from all macroreplications, grouped by solver and problem. :rtype: tuple[list[list[list[Curve]]], list[list[list[Curve]]]] .. py:class:: PlotType(*args, **kwds) Bases: :py:obj:`enum.Enum` Enum class for different types of plots and metrics. .. py:attribute:: ALL :value: 'all' .. py:attribute:: MEAN :value: 'mean' .. py:attribute:: QUANTILE :value: 'quantile' .. py:attribute:: AREA_MEAN :value: 'area_mean' .. py:attribute:: AREA_STD_DEV :value: 'area_std_dev' .. py:attribute:: SOLVE_TIME_QUANTILE :value: 'solve_time_quantile' .. py:attribute:: SOLVE_TIME_CDF :value: 'solve_time_cdf' .. py:attribute:: CDF_SOLVABILITY :value: 'cdf_solvability' .. py:attribute:: QUANTILE_SOLVABILITY :value: 'quantile_solvability' .. py:attribute:: DIFFERENCE_OF_CDF_SOLVABILITY :value: 'difference_of_cdf_solvability' .. py:attribute:: DIFFERENCE_OF_QUANTILE_SOLVABILITY :value: 'difference_of_quantile_solvability' .. py:attribute:: AREA :value: 'area' .. py:attribute:: BOX :value: 'box' .. py:attribute:: VIOLIN :value: 'violin' .. py:attribute:: TERMINAL_SCATTER :value: 'terminal_scatter' .. py:attribute:: FEASIBILITY_SCATTER :value: 'feasibility_scatter' .. py:attribute:: FEASIBILITY_VIOLIN :value: 'feasibility_violin' .. py:attribute:: ALL_FEASIBILITY_PROGRESS :value: 'all_feasibility_progress' .. py:attribute:: MEAN_FEASIBILITY_PROGRESS :value: 'mean_feasibility_progress' .. py:attribute:: QUANTILE_FEASIBILITY_PROGRESS :value: 'quantile_feasibility_progress' .. py:method:: from_str(label: str) -> PlotType :staticmethod: Converts a string label to a PlotType enum. .. py:function:: bootstrap_procedure(experiments: list[list[ProblemSolver]], n_bootstraps: int, conf_level: float, plot_type: PlotType, beta: float | None = None, solve_tol: float | None = None, estimator: float | simopt.curve.Curve | None = None, normalize: bool = True, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, feasibility_two_sided: bool = False) -> tuple[float, float] | tuple[simopt.curve.Curve, simopt.curve.Curve] Performs bootstrapping and computes confidence intervals for progress curves. :param experiments: Grid of problem-solver pairs. :type experiments: list[list[ProblemSolver]] :param n_bootstraps: Number of bootstrap samples to generate. :type n_bootstraps: int :param conf_level: Confidence level for the interval (0 < conf_level < 1). :type conf_level: float :param plot_type: Type of plot/metric for which to compute the interval. :type plot_type: PlotType :param beta: Quantile level (0 < beta < 1), used with some plot types. :type beta: float, optional :param solve_tol: Relative optimality gap that defines a "solved" instance. :type solve_tol: float, optional :param estimator: Reference estimator for difference plot types. :type estimator: float or Curve, optional :param normalize: Whether to normalize progress curves. Defaults to True. :type normalize: bool, optional :param feasibility_score_method: Feasibility scoring rule used when `plot_type` corresponds to feasibility metrics. :type feasibility_score_method: Literal["inf_norm", "norm"], optional :param feasibility_norm_degree: Degree of the norm when ``feasibility_score_method == "norm"``. :type feasibility_norm_degree: int, optional :param feasibility_two_sided: Whether to assign a non-zero score to feasible solutions based on the best violation. :type feasibility_two_sided: bool, optional :returns: Lower and upper bounds of the CI. :rtype: tuple[float, float] or tuple[Curve, Curve] .. py:function:: functional_of_curves(bootstrap_curves: list[list[list[simopt.curve.Curve]]], plot_type: PlotType, beta: float | None = 0.5, solve_tol: float | None = 0.1) -> float | simopt.curve.Curve Computes a functional of bootstrapped objective or progress curves. :param bootstrap_curves: Bootstrapped curves for all solutions across all macroreplications. :type bootstrap_curves: list[list[list[Curve]]] :param plot_type: Type of functional to compute: - PlotType.MEAN - PlotType.QUANTILE - PlotType.AREA_MEAN - PlotType.AREA_STD_DEV - PlotType.SOLVE_TIME_QUANTILE - PlotType.SOLVE_TIME_CDF - PlotType.CDF_SOLVABILITY - PlotType.QUANTILE_SOLVABILITY - PlotType.DIFFERENCE_OF_CDF_SOLVABILITY - PlotType.DIFFERENCE_OF_QUANTILE_SOLVABILITY :type plot_type: PlotType :param beta: Quantile level (0 < beta < 1). Defaults to 0.5. :type beta: float, optional :param solve_tol: Optimality gap for defining a solved instance (0 < solve_tol ≤ 1). Defaults to 0.1. :type solve_tol: float, optional :returns: The computed functional of the curves. :rtype: Curve or float :raises ValueError: If input values are invalid or unsupported for the given plot_type. .. py:function:: compute_bootstrap_conf_int(observations: list[float | int], conf_level: float, bias_correction: bool = True, overall_estimator: float | None = None) -> tuple[numpy.ndarray, numpy.ndarray] Construct a bootstrap confidence interval for an estimator. :param observations: Estimators from all bootstrap instances. :type observations: list[float | int] :param conf_level: Confidence level for confidence intervals, i.e., 1 - gamma; must be in (0, 1). :type conf_level: float :param bias_correction: Whether to use bias-corrected bootstrap CIs (via the percentile method). Defaults to True. :type bias_correction: bool, optional :param overall_estimator: The estimator to compute the CI around. Required if`bias_correction` is True. :type overall_estimator: float | None, optional :returns: A tuple containing the lower and upper bounds of the bootstrap confidence interval. :rtype: tuple[np.ndarray, np.ndarray] :raises ValueError: If `conf_level` is not in (0, 1), or if `overall_estimator` is None when `bias_correction` is True. .. py:function:: plot_bootstrap_conf_ints(bs_conf_int_lower_bounds: simopt.curve.Curve, bs_conf_int_upper_bounds: simopt.curve.Curve, color_str: str = 'C0') -> None Plot bootstrap confidence intervals. :param bs_conf_int_lower_bounds: Lower bounds of bootstrap confidence intervals, as curves. :type bs_conf_int_lower_bounds: Curve :param bs_conf_int_upper_bounds: Upper bounds of bootstrap confidence intervals, as curves. :type bs_conf_int_upper_bounds: Curve :param color_str: String indicating line color, e.g., "C0", "C1", etc. Defaults to "C0". :type color_str: str, optional .. py:function:: report_max_halfwidth(curve_pairs: list[list[simopt.curve.Curve]], normalize: bool, conf_level: float, difference: bool = False) -> None Print caption for the max halfwidth of bootstrap confidence interval curves. :param curve_pairs: A list of paired bootstrap CI curves. :type curve_pairs: list[list[Curve]] :param normalize: Whether to normalize progress curves with respect to optimality gaps. :type normalize: bool :param conf_level: Confidence level for confidence intervals (must be in (0, 1)). :type conf_level: float :param difference: Whether the plot is for difference profiles. Defaults to False. :type difference: bool, optional :raises ValueError: If `conf_level` is not in (0, 1) or if `curve_pairs` is empty. .. py:function:: check_common_problem_and_reference(experiments: list[ProblemSolver]) -> None Check if a collection of experiments share the same problem, x0, and x*. :param experiments: Problem-solver pairs of different solvers on a common problem. :type experiments: list[ProblemSolver] :raises ValueError: If any experiments have different problem instances, starting solutions (`x0`), or optimal solutions (`x*`). .. py:function:: plot_progress_curves(experiments: list[ProblemSolver], plot_type: PlotType, beta: float = 0.5, normalize: bool = True, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') -> list[pathlib.Path] Plots individual or aggregate progress curves for solvers on a single problem. :param experiments: Problem-solver pairs for different solvers on the same problem. :type experiments: list[ProblemSolver] :param plot_type: Type of plot to produce (ALL, MEAN, or QUANTILE). :type plot_type: PlotType :param beta: Quantile level to plot (0 < beta < 1). Defaults to 0.50. :type beta: float, optional :param normalize: If True, normalize curves by optimality gaps. Defaults to True. :type normalize: bool, optional :param all_in_one: If True, plot all curves in one figure. Defaults to True. :type all_in_one: bool, optional :param n_bootstraps: Number of bootstrap samples. Defaults to 100. :type n_bootstraps: int, optional :param conf_level: Confidence level for CIs (0 < conf_level < 1). Defaults to 0.95. :type conf_level: float, optional :param plot_conf_ints: If True, plot bootstrapped confidence intervals. Defaults to True. :type plot_conf_ints: bool, optional :param print_max_hw: If True, print caption with max half-width. Defaults to True. :type print_max_hw: bool, optional :param plot_title: Custom title for the plot (used only if `all_in_one=True`). :type plot_title: str, optional :param legend_loc: Location of legend (e.g., "best", "lower right"). :type legend_loc: str, optional :param ext: File extension for saved plots (e.g., ".png"). Defaults to ".png". :type ext: str, optional :param save_as_pickle: If True, save plot as a pickle file. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Label for solver group in plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :returns: List of file paths where the plots were saved. :rtype: list[Path] :raises ValueError: If beta, conf_level, or n_bootstraps have invalid values. .. py:function:: plot_solvability_cdfs(experiments: list[ProblemSolver], solve_tol: float = 0.1, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') -> list[pathlib.Path] Plots solvability CDFs for one or more solvers on a single problem. :param experiments: Problem-solver pairs for different solvers on a common problem. :type experiments: list[ProblemSolver] :param solve_tol: Optimality gap that defines when a problem is considered solved (0 < solve_tol ≤ 1). Defaults to 0.1. :type solve_tol: float, optional :param all_in_one: If True, plot all curves together. Defaults to True. :type all_in_one: bool, optional :param n_bootstraps: Number of bootstrap samples. Defaults to 100. :type n_bootstraps: int, optional :param conf_level: Confidence level for intervals (0 < conf_level < 1). Defaults to 0.95. :type conf_level: float, optional :param plot_conf_ints: If True, include bootstrapped confidence intervals. Defaults to True. :type plot_conf_ints: bool, optional :param print_max_hw: If True, print the max half-width in the caption. Defaults to True. :type print_max_hw: bool, optional :param plot_title: Custom title to override the generated one (used only if all_in_one is True). :type plot_title: str, optional :param legend_loc: Location of the plot legend (e.g., "best"). :type legend_loc: str, optional :param ext: File extension for saved plots. Defaults to ".png". :type ext: str, optional :param save_as_pickle: If True, save plots as pickle files. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Label for solver group in plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :returns: List of file paths for the generated plots. :rtype: list[Path] :raises ValueError: If any input parameter is out of bounds or invalid. .. py:function:: plot_area_scatterplots(experiments: list[list[ProblemSolver]], all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, plot_title: str | None = None, legend_loc: str = 'best', ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') -> list[pathlib.Path] Plots scatterplots of mean vs. standard deviation of area under progress curves. Can generate either one plot per solver or a combined plot for all solvers. .. note:: The `print_max_hw` flag is currently not implemented. :param experiments: Problem-solver pairs used for plotting. :type experiments: list[list[ProblemSolver]] :param all_in_one: If True, plot all solvers together. Defaults to True. :type all_in_one: bool, optional :param n_bootstraps: Number of bootstrap samples. Defaults to 100. :type n_bootstraps: int, optional :param conf_level: Confidence level for CIs (0 < conf_level < 1). Defaults to 0.95. :type conf_level: float, optional :param plot_conf_ints: If True, show bootstrapped confidence intervals. Defaults to True. :type plot_conf_ints: bool, optional :param print_max_hw: Placeholder for printing max half-widths. Currently unused. :type print_max_hw: bool, optional :param plot_title: Custom title for the plot (applies only if `all_in_one=True`). :type plot_title: str, optional :param legend_loc: Location of the legend (e.g., "best", "lower right"). :type legend_loc: str, optional :param ext: File extension for saved plots. Defaults to ".png". :type ext: str, optional :param save_as_pickle: If True, save plot as a pickle file. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Label for solver group in plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :param problem_set_name: Label for problem group in plot titles. Defaults to "PROBLEM_SET". :type problem_set_name: str, optional :returns: List of file paths for the plots produced. :rtype: list[Path] :raises ValueError: If `n_bootstraps` is not positive or `conf_level` is outside (0, 1). .. py:function:: plot_solvability_profiles(experiments: list[list[ProblemSolver]], plot_type: PlotType, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, solve_tol: float = 0.1, beta: float = 0.5, ref_solver: str | None = None, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') -> list[pathlib.Path] Plots solvability or difference profiles for solvers on multiple problems. :param experiments: Problem-solver pairs used for plotting. :type experiments: list[list[ProblemSolver]] :param plot_type: Type of solvability plot to produce: - PlotType.CDF_SOLVABILITY - PlotType.QUANTILE_SOLVABILITY - PlotType.DIFFERENCE_OF_CDF_SOLVABILITY - PlotType.DIFFERENCE_OF_QUANTILE_SOLVABILITY :type plot_type: PlotType :param all_in_one: If True, plot all curves together. Defaults to True. :type all_in_one: bool, optional :param n_bootstraps: Number of bootstrap samples. Defaults to 100. :type n_bootstraps: int, optional :param conf_level: Confidence level for intervals (0 < conf_level < 1). Defaults to 0.95. :type conf_level: float, optional :param plot_conf_ints: If True, show bootstrapped confidence intervals. Defaults to True. :type plot_conf_ints: bool, optional :param print_max_hw: If True, print max half-width in caption. Defaults to True. :type print_max_hw: bool, optional :param solve_tol: Optimality gap defining when a problem is considered solved (0 < solve_tol ≤ 1). Defaults to 0.1. :type solve_tol: float, optional :param beta: Quantile level to compute (0 < beta < 1). Defaults to 0.5. :type beta: float, optional :param ref_solver: Name of the reference solver for difference plots. :type ref_solver: str, optional :param plot_title: Custom title for the plot (used only if `all_in_one=True`). :type plot_title: str, optional :param legend_loc: Location of the legend (e.g., "best", "upper right"). :type legend_loc: str, optional :param ext: File extension for saved plots (e.g., ".png"). Defaults to ".png". :type ext: str, optional :param save_as_pickle: If True, save plots as pickle files. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Name of solver group for plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :param problem_set_name: Name of problem group for plot titles. Defaults to "PROBLEM_SET". :type problem_set_name: str, optional :returns: List of file paths for the plots produced. :rtype: list[Path] :raises ValueError: If any input parameter is out of bounds or invalid. .. py:function:: plot_terminal_progress(experiments: list[ProblemSolver], plot_type: PlotType = PlotType.VIOLIN, normalize: bool = True, all_in_one: bool = True, plot_title: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET') -> list[pathlib.Path] Plots terminal progress as box or violin plots for solvers on a single problem. :param experiments: ProblemSolver pairs for different solvers on a common problem. :type experiments: list[ProblemSolver] :param plot_type: Type of plot to generate: "box" or "violin". Defaults to "violin". :type plot_type: str, optional :param normalize: If True, normalize progress curves by optimality gaps. Defaults to True. :type normalize: bool, optional :param all_in_one: If True, plot all curves in one figure. Defaults to True. :type all_in_one: bool, optional :param plot_title: Custom title to override the default. Used only if all_in_one is True. :type plot_title: str, optional :param ext: File extension for saved plots (e.g., ".png"). Defaults to ".png". :type ext: str, optional :param save_as_pickle: If True, save the plot as a pickle file. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Label for solver group in plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :returns: List of file paths for the plots produced. :rtype: list[str] :raises ValueError: If an unsupported plot type is specified. .. py:function:: plot_terminal_scatterplots(experiments: list[list[ProblemSolver]], all_in_one: float = True, plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False, solver_set_name: str = 'SOLVER_SET', problem_set_name: str = 'PROBLEM_SET') -> list[pathlib.Path] Plot scatter plots of the mean and standard deviation of terminal progress. Either creates one plot per solver or a combined plot for all solvers. :param experiments: Problem-solver pairs used to produce plots. :type experiments: list[list[ProblemSolver]] :param all_in_one: Whether to plot all solvers in one figure. Defaults to True. :type all_in_one: bool, optional :param plot_title: Title to override the autogenerated one (only applies if `all_in_one` is True). :type plot_title: str | None, optional :param legend_loc: Location of the legend (e.g., "best"). Defaults to None. :type legend_loc: str | None, optional :param ext: File extension to use for output images. Defaults to ".png". :type ext: str, optional :param save_as_pickle: Whether to also save the plots as `.pickle` files. Defaults to False. :type save_as_pickle: bool, optional :param solver_set_name: Name for the solver group used in plot titles. Defaults to "SOLVER_SET". :type solver_set_name: str, optional :param problem_set_name: Name for the problem group used in plot titles. Defaults to "PROBLEM_SET". :type problem_set_name: str, optional :returns: A list of file paths to the plots produced. :rtype: list[Path] .. py:function:: setup_plot(plot_type: PlotType, solver_name: str = 'SOLVER SET', problem_name: str = 'PROBLEM SET', normalize: bool = True, budget: int | None = None, beta: float | None = None, feasibility_score_method: Literal['inf_norm', 'norm'] = 'inf_norm', feasibility_norm_degree: int = 1, solve_tol: float | None = None, plot_title: str | None = None) -> None Create a new figure, add labels to the plot, and reformat axes. :param plot_type: Type of plot to produce. Valid options include: - ALL: All estimated progress curves. - MEAN: Estimated mean progress curve. - QUANTILE: Estimated beta quantile progress curve. - SOLVE_TIME_CDF: CDF of solve time. - CDF_SOLVABILITY: CDF solvability profile. - QUANTILE_SOLVABILITY: Quantile solvability profile. - DIFFERENCE_OF_CDF_SOLVABILITY: Difference of CDF solvability profiles. - DIFFERENCE_OF_QUANTILE_SOLVABILITY: Difference of quantile solvability profiles. - AREA: Area scatterplot. - BOX: Box plot of terminal progress. - VIOLIN: Violin plot of terminal progress. - TERMINAL_SCATTER: Scatterplot of mean and std dev of terminal progress. :type plot_type: PlotType :param solver_name: Name of the solver. Defaults to "SOLVER SET". :type solver_name: str, optional :param problem_name: Name of the problem. Defaults to "PROBLEM SET". :type problem_name: str, optional :param normalize: Whether to normalize with respect to optimality gaps. Defaults to True. :type normalize: bool, optional :param budget: Function evaluation budget. :type budget: int, optional :param beta: Quantile to compute (must be in (0, 1)). :type beta: float, optional :param feasibility_score_method: Method to compute the feasibility score. Defaults to "inf_norm". :type feasibility_score_method: Literal["inf_norm", "norm"], optional :param feasibility_norm_degree: Degree of the norm to use for the feasibility score. Defaults to 1. :type feasibility_norm_degree: int, optional :param solve_tol: Relative optimality gap for declaring a solve (must be in (0, 1]). :type solve_tol: float, optional :param plot_title: Title to override the automatically generated one. :type plot_title: str, optional :raises ValueError: If any inputs are invalid. .. py:function:: save_plot(solver_name: str, problem_name: str, plot_type: PlotType, normalize: bool, extra: float | list[float] | None = None, plot_title: str | None = None, ext: str = '.png', save_as_pickle: bool = False) -> pathlib.Path Create and save a plot with appropriate labels and formatting. :param solver_name: Name of the solver. :type solver_name: str :param problem_name: Name of the problem. :type problem_name: str :param plot_type: Type of plot to produce. Valid options include: - ALL: All estimated progress curves. - MEAN: Estimated mean progress curve. - QUANTILE: Estimated beta quantile progress curve. - SOLVE_TIME_CDF: CDF of solve time. - CDF_SOLVABILITY: CDF solvability profile. - QUANTILE_SOLVABILITY: Quantile solvability profile. - DIFFERENCE_OF_CDF_SOLVABILITY: Difference of CDF solvability profiles. - DIFFERENCE_OF_QUANTILE_SOLVABILITY: Difference of quantile solvability profiles. - AREA: Area scatterplot. - TERMINAL_SCATTER: Scatterplot of mean and std dev of terminal progress. - FEASIBILITY_SCATTER: Scatterplot of terminal objective vs feasibility. - FEASIBILITY_VIOLIN: Violin plot of terminal feasibility. - ALL_FEASIBILITY_PROGRESS: Feasibility progress curves for all macroreps. - MEAN_FEASIBILITY_PROGRESS: Mean feasibility progress curve. - QUANTILE_FEASIBILITY_PROGRESS: Quantile feasibility progress curve. :type plot_type: PlotType :param normalize: Whether to normalize with respect to optimality gaps. :type normalize: bool :param extra: Extra number(s) specifying quantile (e.g., beta) and/or solve tolerance. :type extra: float | list[float], optional :param plot_title: If provided, overrides the default title and filename. :type plot_title: str | None, optional :param ext: File extension for the saved plot. Defaults to ".png". :type ext: str, optional :param save_as_pickle: Whether to save the plot as a pickle file. Defaults to False. :type save_as_pickle: bool, optional :returns: Path pointing to the location where the plot will be saved. :rtype: Path .. py:function:: plot_terminal_feasibility(experiments: list[list[ProblemSolver]], plot_type: Literal['scatter', 'violin'] = 'scatter', score_type: Literal['inf_norm', 'norm'] = 'inf_norm', two_sided: bool = True, plot_zero: bool = True, plot_optimal: bool = True, norm_degree: int = 1, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, bias_correction: bool = True, solver_set_name: str = 'SOLVER_SET', plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False) -> list[str] Plot the feasibility of one solver problem pair. (for now). :param experiments: Problem-solver pairs used to produce plots. :type experiments: list [list [``experiment_base.ProblemSolver``]] :param plot_type: String indicating which type of plot to produce: "scatter" : scatter plot with terminal objective on x-axis and terminal feasiblity score on y-axis for all macroreps "violin" : violin plot showing density of terminal feasiblity scores for all macroreps :type plot_type: str, default = 'scatter' :param score_type: "inf_norm" : use infinite norm of lhs of violated constraints to calculate feasiblity score "norm" : use "norm_degree" to specify degree of norm of lhs of violated constriants :type score_type: str, default = "inf_norm" :param two_sided: Two-sided or one-sided feasiblity score :type two_sided: bool, default = "True" :param plot_zero: Plot a dashed red line a feasiblity score = 0 :type plot_zero: bool, default = True :param plot_optimal: Plot a dashed red line at beast feasible objective across all postreplications :type plot_optimal: default = True :param norm_degree: if not using inf_norm, specifies degree of norm taken of lhs of violated constraints for feasibility score :type norm_degree: int, default = 1 :param all_in_one: plot all solvers on same plot :type all_in_one: bool, default = True :param n_bootstraps: number of bootsrap replications for CI construction :type n_bootstraps: int, default = 100 :param conf_level: confidence level of created CI :type conf_level: float, default = 0.95 :param plot_conf_ints: plot CI's for each maroreplication :type plot_conf_ints: bool, default = True :param bias_correction: use bias correction for CI construction :type bias_correction: bool, default = True :param solver_set_name: Override solver names in plot, only applies if all_in_one = True :type solver_set_name: str, default = "SOLVER_SET" :param plot_title: Optional title to override the one that is autmatically generated :type plot_title: str, optional :param legend_loc: specificies location of legend :type legend_loc: str, default="best" :param ext: Extension to add to image file path to change file type :type ext: str, default = '.png' :param save_as_pickle: True if plot should be saved to pickle file, False otherwise. :type save_as_pickle: bool, default = False :param Returns: :param -------: :param file_list: List compiling path names for plots produced. :type file_list: list [str] :param Raises: :param ------: :param TypeError: :param ValueError: .. py:function:: plot_feasibility_progress(experiments: list[list[ProblemSolver]], plot_type: PlotType = PlotType.ALL_FEASIBILITY_PROGRESS, score_type: Literal['inf_norm', 'norm'] = 'inf_norm', norm_degree: int = 1, two_sided: bool = True, plot_zero: bool = True, all_in_one: bool = True, n_bootstraps: int = 100, conf_level: float = 0.95, plot_conf_ints: bool = True, print_max_hw: bool = True, beta: float = 0.5, solver_set_name: str = 'SOLVER_SET', plot_title: str | None = None, legend_loc: str | None = None, ext: str = '.png', save_as_pickle: bool = False) -> list[str] Plot feasibility over solver progress. :param experiments: Problem-solver pairs used to produce plots. :type experiments: list [list [``experiment_base.ProblemSolver``]] :param plot_type: String indicating which type of plot to produce: "all" : show all macroreps "mean" : plot mean of all macroreps "quantile" : plot quantile of all macroreps :type plot_type: str, default = 'scatter' :param score_type: "inf_norm" : use infinite norm of lhs of violated constraints to calculate feasiblity score "norm" : use "norm_degree" to specify degree of norm of lhs of violated constriants :type score_type: str, default = "inf_norm" :param two_sided: Two-sided or one-sided feasiblity score. Two-sided only supported for plot_type = "all" :type two_sided: bool, default = "True" :param plot_zero: Plot a dashed red line a feasiblity score = 0 :type plot_zero: bool, default = True :param norm_degree: if not using inf_norm, specifies degree of norm taken of lhs of violated constraints for feasibility score :type norm_degree: int, default = 1 :param all_in_one: plot all solvers on same plot :type all_in_one: bool, default = True :param n_bootstraps: number of bootsrap replications for CI construction :type n_bootstraps: int, default = 100 :param conf_level: confidence level of created CI :type conf_level: float, default = 0.95 :param plot_conf_ints: plot CI's for each maroreplication :type plot_conf_ints: bool, default = True :param print_max_hw: report max halfwidth for CI's' :type print_max_hw: bool, default = True :param beta: quantile to computue must be between 0 and 1 :type beta: float, default = .5 :param solver_set_name: Override solver names in plot, only applies if all_in_one = True :type solver_set_name: str, default = "SOLVER_SET" :param plot_title: Optional title to override the one that is autmatically generated :type plot_title: str, optional :param legend_loc: specificies location of legend :type legend_loc: str, default="best" :param ext: Extension to add to image file path to change file type :type ext: str, default = '.png' :param save_as_pickle: True if plot should be saved to pickle file, False otherwise. :type save_as_pickle: bool, default = False :param Returns: :param -------: :param file_list: List compiling path names for plots produced. :type file_list: list [str] :param Raises: :param ------: :param TypeError: :param ValueError: .. py:class:: ProblemsSolvers(solver_factors: list[dict] | None = None, problem_factors: list[dict] | None = None, solver_names: list[str] | None = None, problem_names: list[str] | None = None, solver_renames: list[str] | None = None, problem_renames: list[str] | None = None, fixed_factors_filename: str | None = None, solvers: list[simopt.base.Solver] | None = None, problems: list[simopt.base.Problem] | None = None, experiments: list[list[ProblemSolver]] | None = None, file_name_path: pathlib.Path | None = None, create_pair_pickles: bool = False, experiment_name: str | None = None) Base class for running one or more solver on one or more problem. Initialize a ProblemsSolvers object. There are three ways to initialize a ProblemsSolvers object: 1. Provide the names of solvers and problems (for lookup in `directory.py`). 2. Provide lists of solver and problem objects to pair directly. 3. Provide a full list of `ProblemSolver` objects (as nested lists). :param solver_factors: List of solver factor dictionaries, one per design point. Requires `solver_names` to match the number of entries. :type solver_factors: list[dict] | None :param problem_factors: List of problem/model factor dictionaries, one per design point. Requires `problem_names` to match the number of entries. :type problem_factors: list[dict] | None :param solver_names: List of solver names to look up. :type solver_names: list[str] | None :param problem_names: List of problem names to look up. :type problem_names: list[str] | None :param solver_renames: User-specified labels for solvers. :type solver_renames: list[str] | None :param problem_renames: User-specified labels for problems. :type problem_renames: list[str] | None :param fixed_factors_filename: Name of a `.py` file containing fixed factor dictionaries. :type fixed_factors_filename: str | None :param solvers: List of `Solver` objects to use directly. :type solvers: list[Solver] | None :param problems: List of `Problem` objects to use directly. :type problems: list[Problem] | None :param experiments: Explicit problem-solver pairings. :type experiments: list[list[ProblemSolver]] | None :param file_name_path: Output path for saving the `ProblemsSolvers` object. :type file_name_path: Path | None :param create_pair_pickles: Whether to create individual `.pickle` files for each problem-solver pair. :type create_pair_pickles: bool :param experiment_name: Optional name to prefix output files. :type experiment_name: str | None .. py:property:: solver_names :type: list[str] List of solver names. .. py:property:: n_solvers :type: int Number of solvers. .. py:property:: problem_names :type: list[str] List of problem names. .. py:property:: n_problems :type: int Number of problems. .. py:property:: solvers :type: list[simopt.base.Solver] List of solvers. .. py:property:: problems :type: list[simopt.base.Problem] List of problems. .. py:property:: all_solver_fixed_factors :type: dict[str, dict] Fixed solver factors for each solver. .. py:property:: all_problem_fixed_factors :type: dict[str, dict] Fixed problem factors for each problem. .. py:property:: all_model_fixed_factors :type: dict[str, dict] Fixed model factors for each problem. .. py:property:: experiments :type: list[list[ProblemSolver]] All problem-solver pairs. .. py:property:: file_name_path :type: pathlib.Path Path to the .pickle file for saving the ProblemsSolvers object. .. py:property:: create_pair_pickles :type: bool Whether to create pickle files for each problem-solver pair. .. py:property:: experiment_name :type: str Name of experiment to be appended to the beginning of output files. .. py:method:: check_compatibility() -> str Check whether all experiments' solvers and problems are compatible. :returns: Error message in the event any problem and solver are incompatible. :rtype: str .. py:method:: run(n_macroreps: int) -> None Run `n_macroreps` of each solver on each problem. :param n_macroreps: Number of macroreplications to run per problem-solver pair. :type n_macroreps: int :raises ValueError: If `n_macroreps` is not positive. .. py:method:: post_replicate(n_postreps: int, crn_across_budget: bool = True, crn_across_macroreps: bool = False) -> None Runs postreplications for each problem-solver pair on all macroreplications. :param n_postreps: Number of postreplications per recommended solution. :type n_postreps: int :param crn_across_budget: If True, use CRN across solutions from different time budgets. Defaults to True. :type crn_across_budget: bool, optional :param crn_across_macroreps: If True, use CRN across solutions from different macroreplications. Defaults to False. :type crn_across_macroreps: bool, optional :raises ValueError: If n_postreps is not positive. .. py:method:: post_normalize(n_postreps_init_opt: int, crn_across_init_opt: bool = True) -> None Builds objective and progress curves for all experiment collections. :param n_postreps_init_opt: Number of postreplications at initial (x0) and optimal (x*) solutions. :type n_postreps_init_opt: int :param crn_across_init_opt: If True, use CRN for postreplications at x0 and x*. Defaults to True. :type crn_across_init_opt: bool, optional :raises ValueError: If `n_postreps_init_opt` is not positive. .. py:method:: check_postreplicate() -> bool Checks whether all experiments have been postreplicated. :returns: Whether all experiments have been postreplicated :rtype: bool .. py:method:: check_postnormalize() -> bool Checks whether all experiments have been postnormalized. :returns: Whether all experiments have been postnormalized. :rtype: bool .. py:method:: record_group_experiment_results() -> None Saves a ProblemsSolvers object to a .pickle file in the outputs directory. .. py:method:: log_group_experiment_results() -> None Creates a .txt summary of solvers and problems in the ProblemSolvers object. The file is saved in the 'logs/' folder next to the experiment's pickle file. .. py:method:: report_group_statistics(solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') -> None Reports statistics for all solvers across all problems. :param solve_tols: Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50]. :type solve_tols: list[float], optional :param csv_filename: Name of the output CSV file (without '.csv'). Defaults to "df_solver_results". :type csv_filename: str, optional :raises ValueError: If any solve tolerance is not in the range (0, 1]. .. py:method:: report_statistics(pair_list: list[ProblemSolver], solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') -> None Calculates statistics from macroreplications and saves results to a CSV file. :param pair_list: List of ProblemSolver objects. :type pair_list: list[ProblemSolver] :param solve_tols: Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50]. :type solve_tols: list[float], optional :param csv_filename: Name of the CSV file to write results to. Defaults to "df_solver_results". :type csv_filename: str, optional :raises ValueError: If any solve tolerance is not in the range (0, 1]. .. py:function:: read_group_experiment_results(file_path: pathlib.Path | str) -> ProblemsSolvers Reads a ProblemsSolvers object from a .pickle file. :param file_path: Path to the .pickle file. :type file_path: Path :returns: A group of problem-solver experiments that were run or post-processed. :rtype: ProblemsSolvers :raises FileNotFoundError: If the file does not exist. .. py:function:: find_unique_solvers_problems(experiments: list[ProblemSolver]) -> tuple[list[simopt.base.Solver], list[simopt.base.Problem]] Finds unique solvers and problems from a list of ProblemSolver experiments. :param experiments: List of problem-solver pairs. :type experiments: list[ProblemSolver] :returns: A tuple containing: - A list of unique solvers. - A list of unique problems. :rtype: tuple[list[Solver], list[Problem]] .. py:function:: find_missing_experiments(experiments: list[ProblemSolver]) -> tuple[list[simopt.base.Solver], list[simopt.base.Problem], list[tuple[simopt.base.Solver, simopt.base.Problem]]] Finds missing problem-solver pairs from a list of experiments. :param experiments: List of problem-solver pairs. :type experiments: list[ProblemSolver] :returns: A tuple containing: - list[Solver]: Unique solvers present in the experiments. - list[Problem]: Unique problems present in the experiments. - list[tuple[Solver, Problem]]: Problem-solver pairs that are missing. :rtype: tuple .. py:function:: make_full_metaexperiment(existing_experiments: list[ProblemSolver], unique_solvers: list[simopt.base.Solver], unique_problems: list[simopt.base.Problem], missing_experiments: list[tuple[simopt.base.Solver, simopt.base.Problem]]) -> ProblemsSolvers Creates experiments for missing problem-solver pairs. :param existing_experiments: Existing problem-solver experiments. :type existing_experiments: list[ProblemSolver] :param unique_solvers: Solvers present in the existing experiments. :type unique_solvers: list[Solver] :param unique_problems: Problems present in the existing experiments. :type unique_problems: list[Problem] :param missing_experiments: Problem-solver pairs that have not yet been run. :type missing_experiments: list[tuple[Solver, Problem]] :returns: A new ProblemsSolvers object containing the completed set. :rtype: ProblemsSolvers .. py:function:: create_design_list_from_table(design_table: pandas.DataFrame) -> list[dict[str, Any]] Create a list of solver or problem objects for each design point. :param design_table: DataFrame containing the design table. Each row represents a design point, and each column represents a factor. :type design_table: DataFrame :returns: List of dictionaries, where each list entry corresponds to a design point, and each dictionary contains the name and values for each factor in that design point. :rtype: list[dict[str, Any]] .. py:function:: create_design(name: str, factor_headers: list[str], factor_settings: list[tuple[float, float, int]] | pathlib.Path, fixed_factors: dict | None = None, cross_design_factors: dict | None = None, design_type: Literal['nolhs'] = 'nolhs', n_stacks: int = 1) -> list[dict[str, Any]] Creates a design of solver, problem, or model factors. Please ensure the indexing of the factor_headers argument matches the indexing of the factor_settings argument. :param name: Name of the solver, problem, or model. :type name: str :param factor_headers: Names of factors that vary in the design. :type factor_headers: list[str] :param factor_settings: A list of tuples, each of the form (min, max, # decimals) or a Path to a .txt file containing those factor settings. :type factor_settings: list[tuple[float, float, int]] | Path :param fixed_factors: Dictionary of fixed factor values that override defaults. :type fixed_factors: dict, optional :param cross_design_factors: Dictionary of lists of cross-design factor values. Defaults to None. :type cross_design_factors: dict, optional :param design_type: Type of design. Defaults to "nolhs". :type design_type: Literal["nolhs"], optional :param n_stacks: Number of stacks. Defaults to 1. :type n_stacks: int, optional :returns: A list of dictionaries, where each dictionary represents a design. :rtype: list[dict[str, Any]] :raises ValueError: If input validation fails. :raises Exception: If the design type is unsupported.