simopt.experiment.multiple ========================== .. py:module:: simopt.experiment.multiple .. autoapi-nested-parse:: Multiple problem-solver pairs. Module Contents --------------- .. py:data:: model_directory .. py:data:: problem_directory .. py:data:: solver_directory .. py:class:: ProblemsSolvers(solver_factors: list[dict] | None = None, problem_factors: list[dict] | None = None, solver_names: list[str] | None = None, problem_names: list[str] | None = None, solver_renames: list[str] | None = None, problem_renames: list[str] | None = None, fixed_factors_filename: str | None = None, solvers: list[simopt.base.Solver] | None = None, problems: list[simopt.base.Problem] | None = None, experiments: list[list[simopt.experiment.single.ProblemSolver]] | None = None, file_name_path: pathlib.Path | None = None, create_pair_pickles: bool = False, experiment_name: str | None = None) Base class for running one or more solver on one or more problem. Initialize a ProblemsSolvers object. There are three ways to initialize a ProblemsSolvers object: 1. Provide the names of solvers and problems (for lookup in `directory.py`). 2. Provide lists of solver and problem objects to pair directly. 3. Provide a full list of `ProblemSolver` objects (as nested lists). :param solver_factors: List of solver factor dictionaries, one per design point. Requires `solver_names` to match the number of entries. :type solver_factors: list[dict] | None :param problem_factors: List of problem/model factor dictionaries, one per design point. Requires `problem_names` to match the number of entries. :type problem_factors: list[dict] | None :param solver_names: List of solver names to look up. :type solver_names: list[str] | None :param problem_names: List of problem names to look up. :type problem_names: list[str] | None :param solver_renames: User-specified labels for solvers. :type solver_renames: list[str] | None :param problem_renames: User-specified labels for problems. :type problem_renames: list[str] | None :param fixed_factors_filename: Name of a `.py` file containing fixed factor dictionaries. :type fixed_factors_filename: str | None :param solvers: List of `Solver` objects to use directly. :type solvers: list[Solver] | None :param problems: List of `Problem` objects to use directly. :type problems: list[Problem] | None :param experiments: Explicit problem-solver pairings. :type experiments: list[list[ProblemSolver]] | None :param file_name_path: Output path for saving the `ProblemsSolvers` object. :type file_name_path: Path | None :param create_pair_pickles: Whether to create individual `.pickle` files for each problem-solver pair. :type create_pair_pickles: bool :param experiment_name: Optional name to prefix output files. :type experiment_name: str | None .. py:property:: solver_names :type: list[str] List of solver names. .. py:property:: n_solvers :type: int Number of solvers. .. py:property:: problem_names :type: list[str] List of problem names. .. py:property:: n_problems :type: int Number of problems. .. py:property:: solvers :type: list[simopt.base.Solver] List of solvers. .. py:property:: problems :type: list[simopt.base.Problem] List of problems. .. py:property:: all_solver_fixed_factors :type: dict[str, dict] Fixed solver factors for each solver. .. py:property:: all_problem_fixed_factors :type: dict[str, dict] Fixed problem factors for each problem. .. py:property:: all_model_fixed_factors :type: dict[str, dict] Fixed model factors for each problem. .. py:property:: experiments :type: list[list[simopt.experiment.single.ProblemSolver]] All problem-solver pairs. .. py:property:: file_name_path :type: pathlib.Path Path to the .pickle file for saving the ProblemsSolvers object. .. py:property:: create_pair_pickles :type: bool Whether to create pickle files for each problem-solver pair. .. py:property:: experiment_name :type: str Name of experiment to be appended to the beginning of output files. .. py:method:: check_compatibility() -> str Check whether all experiments' solvers and problems are compatible. :returns: Error message in the event any problem and solver are incompatible. :rtype: str .. py:method:: run(n_macroreps: int) -> None Run `n_macroreps` of each solver on each problem. :param n_macroreps: Number of macroreplications to run per problem-solver pair. :type n_macroreps: int :raises ValueError: If `n_macroreps` is not positive. .. py:method:: post_replicate(n_postreps: int, crn_across_budget: bool = True, crn_across_macroreps: bool = False) -> None Runs postreplications for each problem-solver pair on all macroreplications. :param n_postreps: Number of postreplications per recommended solution. :type n_postreps: int :param crn_across_budget: If True, use CRN across solutions from different time budgets. Defaults to True. :type crn_across_budget: bool, optional :param crn_across_macroreps: If True, use CRN across solutions from different macroreplications. Defaults to False. :type crn_across_macroreps: bool, optional :raises ValueError: If n_postreps is not positive. .. py:method:: post_normalize(n_postreps_init_opt: int, crn_across_init_opt: bool = True) -> None Builds objective and progress curves for all experiment collections. :param n_postreps_init_opt: Number of postreplications at initial (x0) and optimal (x*) solutions. :type n_postreps_init_opt: int :param crn_across_init_opt: If True, use CRN for postreplications at x0 and x*. Defaults to True. :type crn_across_init_opt: bool, optional :raises ValueError: If `n_postreps_init_opt` is not positive. .. py:method:: check_postreplicate() -> bool Checks whether all experiments have been postreplicated. :returns: Whether all experiments have been postreplicated :rtype: bool .. py:method:: check_postnormalize() -> bool Checks whether all experiments have been postnormalized. :returns: Whether all experiments have been postnormalized. :rtype: bool .. py:method:: record_group_experiment_results() -> None Saves a ProblemsSolvers object to a .pickle file in the outputs directory. .. py:method:: log_group_experiment_results() -> None Creates a .txt summary of solvers and problems in the ProblemSolvers object. The file is saved in the 'logs/' folder next to the experiment's pickle file. .. py:method:: report_group_statistics(solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') -> None Reports statistics for all solvers across all problems. :param solve_tols: Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50]. :type solve_tols: list[float], optional :param csv_filename: Name of the output CSV file (without '.csv'). Defaults to "df_solver_results". :type csv_filename: str, optional :raises ValueError: If any solve tolerance is not in the range (0, 1]. .. py:method:: report_statistics(pair_list: list[simopt.experiment.single.ProblemSolver], solve_tols: list[float] | None = None, csv_filename: str = 'df_solver_results') -> None Calculates statistics from macroreplications and saves results to a CSV file. :param pair_list: List of ProblemSolver objects. :type pair_list: list[ProblemSolver] :param solve_tols: Optimality gaps defining when a problem is considered solved (values in (0, 1]). Defaults to [0.05, 0.10, 0.20, 0.50]. :type solve_tols: list[float], optional :param csv_filename: Name of the CSV file to write results to. Defaults to "df_solver_results". :type csv_filename: str, optional :raises ValueError: If any solve tolerance is not in the range (0, 1].