Welcome to SimOpt’s documentation!

The purpose of the SimOpt testbed is to encourage development and constructive comparison of simulation-optimization (SO) solvers (algorithms). We are particularly interested in the finite-time performance of solvers, rather than the asymptotic results that one often finds in related literature.

For the purposes of this site, we define simulation as a very general technique for estimating statistical measures of complex systems. A system is modeled as if the probability distributions of the underlying random variables were known. Realizations of these random variables are then drawn randomly from these distributions. Each replication gives one observation of the system response, i.e., an evaluation of the objective function. By simulating a system in this fashion for multiple replications and aggregating the responses, one can compute statistics and use them for evaluation and design.

The paper Pasupathy and Henderson (2006) explains the original motivation for the testbed, and the follow-up paper Pasupathy and Henderson (2011) describes an earlier interface for MATLAB implementations of problems and solvers. The paper Dong et al. (2017) conducts an experimental comparison of several solvers in SimOpt and analyzes their relative performance. The recent Winter Simulation Conference paper Eckman et al. (2019) describes in detail the recent changes to the architecture of SimOpt and the control of random number streams.

The models library contains the simulation logic to simulate a variety of systems and SO test problems built around these models. The solvers library provides users with the latest SO solvers to solve different types of SO problems. The two libraries are intended to help researchers evaluate and compare the finite-time performance of existing solvers.

The source code consists of the following modules:

• The base.py module contains class definitions for models, problems, and solvers.

• The experiment_base.py module contains class definitions and functions for running experiments with simulation-optimization solvers.

• The data_farming_base.py module contains class definitions and functions for running data-farming experiments.

• The directory.py module contains a listing of models, problems, and solvers in the library.

Getting Started

Please make sure you have the following dependencies installed: Python 3, numpy, scipy, matplotlib, seaborn, and mrg32k3a. Then clone the repo. To see an example of how to run an experiment on a solver and problem pair, please view or run demo/demo_problem_solver.py.

Contents

• latest
  • simopt package
    • Subpackages
      • simopt.models package
      • simopt.solvers package
    • Submodules
    • simopt.GUI module
      • Cross_Design_Window
      • Experiment_Window
      • Experiment_Window.self.frame
      • Experiment_Window.self.experiment_master_list
      • Experiment_Window.self.widget_list
      • Experiment_Window.self.experiment_object_list
      • Experiment_Window.self.problem_var
      • Experiment_Window.self.solver_var
      • Experiment_Window.self.maco_var
      • Plot_Window
      • Post_Normal_Window
      • Post_Processing_Window
      • main()
      • problem_solver_abbreviated_name_to_unabbreviated()
      • problem_solver_unabbreviated_to_object()
    • simopt.base module
      • Summary
      • Model
      • Problem
      • Solution
      • Solver
    • simopt.data_farming_base module
      • DataFarmingExperiment
      • DataFarmingMetaExperiment
      • DesignPoint
    • simopt.directory module
      • Summary
    • simopt.experiment_base module
      • Summary
      • Curve
      • ProblemSolver
      • ProblemsSolvers
      • bootstrap_procedure()
      • bootstrap_sample_all()
      • cdf_of_curves_crossing_times()
      • check_common_problem_and_reference()
      • compute_bootstrap_CI()
      • difference_of_curves()
      • find_missing_experiments()
      • find_unique_solvers_problems()
      • functional_of_curves()
      • make_full_metaexperiment()
      • max_difference_of_curves()
      • mean_of_curves()
      • plot_area_scatterplots()
      • plot_bootstrap_CIs()
      • plot_progress_curves()
      • plot_solvability_cdfs()
      • plot_solvability_profiles()
      • plot_terminal_progress()
      • plot_terminal_scatterplots()
      • post_normalize()
      • quantile_cross_jump()
      • quantile_of_curves()
      • read_experiment_results()
      • read_group_experiment_results()
      • report_max_halfwidth()
      • save_plot()
      • setup_plot()
      • trim_solver_results()
    • Module contents

Acknowledgments

An earlier website for SimOpt was developed through work supported by the National Science Foundation under grant nos. DMI-0400287, CMMI-0800688 and CMMI-1200315. Recent work on the development of SimOpt has been supported by the National Science Foundation under grant nos. DGE-1650441, CMMI-1537394, CMMI-1254298, CMMI-1536895, IIS-1247696, and TRIPODS+X DMS-1839346, by the Air Force Office of Scientific Research under grant nos. FA9550-12-1-0200, FA9550-15-1-0038, and FA9550-16-1-0046, and by the Army Research Office under grant no. W911NF-17-1-0094. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation (NSF).