biosimulators_test_suite.test_case package#

Submodules#

biosimulators_test_suite.test_case.cli module#

Methods for test cases involving checking command-line interfaces

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-21
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.cli.CliDescribesSupportedEnvironmentVariablesInline(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that the inline help for a command-line interface describes the environment variables that the simulator supports.

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.cli.CliDisplaysHelpInline(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that a command-line interface provides inline help.

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.cli.CliDisplaysVersionInformationInline(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that a command-line interface provides version information inline.

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

biosimulators_test_suite.test_case.combine_archive module#

Methods for test cases involving checking support for the COMBINE/OMEX standards

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-21
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.combine_archive.CombineArchiveHasSedDocumentsInNestedDirectories(*args, **kwargs)[source]#

Bases: WhenACombineArchiveHasAMasterFileSimulatorOnlyExecutesThisFile

Test that SED documents in nested directories can be executed

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

SED_DOCUMENTS_HAVE_SAME_NAMES = False[source]#

SED_DOCUMENT_LOCATIONS_ARE_NESTED = True[source]#

class biosimulators_test_suite.test_case.combine_archive.CombineArchiveHasSedDocumentsWithSameNamesInDifferentInNestedDirectories(*args, **kwargs)[source]#

Bases: WhenACombineArchiveHasNoMasterFileSimulatorExecutesAllSedDocuments

Test that SED documents with the same names in nested directories can be executed and their outputs are saved to distinct paths

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

SED_DOCUMENTS_HAVE_SAME_NAMES = True[source]#

SED_DOCUMENT_LOCATIONS_ARE_NESTED = True[source]#

class biosimulators_test_suite.test_case.combine_archive.CombineArchiveTestCase(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Base class for testing for the execution of master and non-master files in COMBINE/OMEX archives

_expected_report_ids[source]#

ids of expected reports

Type:: list of str

SED_DOCUMENTS_HAVE_SAME_NAMES = False[source]#

SED_DOCUMENT_LOCATIONS_ARE_NESTED = False[source]#

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

get_expected_reports(archive, sed_documents)[source]#

Get the ids of the reports expected to be produced from a COMBINE/OMEX archive

Parameters:

archive (CombineArchive) – COMBINE/OMEX archive
sed_documents (dict of str to SedDocument) – dictionary that maps the location of each SED document to the document

Returns:

ids of the reports expected to be produced from a COMBINE/OMEX archive

Return type:

set of str

class biosimulators_test_suite.test_case.combine_archive.WhenACombineArchiveHasAMasterFileSimulatorOnlyExecutesThisFile(*args, **kwargs)[source]#

Bases: CombineArchiveTestCase

Test that when a COMBINE/OMEX archive defines a (single) master file, the simulator only executes this file.

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

class biosimulators_test_suite.test_case.combine_archive.WhenACombineArchiveHasNoMasterFileSimulatorExecutesAllSedDocuments(*args, **kwargs)[source]#

Bases: CombineArchiveTestCase

Test that when a COMBINE/OMEX archive does not have a defined master file, the simulator executes all SED-ML files.

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

biosimulators_test_suite.test_case.docker_image module#

Methods for test cases involving checking Docker images

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-21
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.docker_image.DeclaresSupportedEnvironmentVariables(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test if a Docker image declares the environment variables that is supports

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.docker_image.DefaultUserIsRoot(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that the default user of a Docker image is root

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None, expected_user=(None, '', '0'))[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
expected_user (tuple, optional) – expected user
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.docker_image.HasBioContainersLabels(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that a Docker image has BioContainers labels with metadata about the image

EXPECTED_LABELS = ['about.documentation', 'about.home', 'about.license', 'about.license_file', 'about.summary', 'about.tags', 'base_image', 'extra.identifiers.biotools', 'maintainer', 'software', 'software.version', 'version'][source]#

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.docker_image.HasOciLabels(id=None, name=None, description=None, output_medium=OutputMedium.console)[source]#

Bases: TestCase

Test that a Docker image has Open Container Initiative (OCI) labels with metadata about the image

EXPECTED_LABELS = ['org.opencontainers.image.authors', 'org.opencontainers.image.description', 'org.opencontainers.image.documentation', 'org.opencontainers.image.licenses', 'org.opencontainers.image.revision', 'org.opencontainers.image.source', 'org.opencontainers.image.title', 'org.opencontainers.image.url', 'org.opencontainers.image.vendor', 'org.opencontainers.image.version', 'org.opencontainers.image.created'][source]#

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

Exception – if the simulator did not pass the test case

class biosimulators_test_suite.test_case.docker_image.SingularityImageExecutesSimulationsSuccessfully(*args, **kwargs)[source]#

Bases: SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports

Test that the Singularity version of a Docker image can sucessfully execute COMBINE archives

EXEC_WITH_SINGULARITY = True[source]#

biosimulators_test_suite.test_case.log module#

Methods for checking support for reports of the execution status of modeling projects

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-29
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.log.LoggingTestCase(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that simulation tool can log its execution

REPORT_ERROR_AS_SKIP = False[source]#

VALIDATE_OUTPUT_LOGS = False[source]#

VALIDATE_SED_DOCUMENT_LOGS = False[source]#

VALIDATE_TASK_LOGS = False[source]#

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

abstract is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.log.SimulatorReportsTheStatusOfTheExecutionOfCombineArchives(*args, **kwargs)[source]#

Bases: LoggingTestCase

Test that simulator logs the execution of COMBINE/OMEX archives

is_concrete()[source]#

Whether the class is concrete

Returns:: whether the class is concrete
Return type:: bool

class biosimulators_test_suite.test_case.log.SimulatorReportsTheStatusOfTheExecutionOfSedDocuments(*args, **kwargs)[source]#

Bases: LoggingTestCase

Test that simulator logs the execution of individual SED document in COMBINE/OMEX archives

VALIDATE_SED_DOCUMENT_LOGS = True[source]#

is_concrete()[source]#

Whether the class is concrete

Returns:: whether the class is concrete
Return type:: bool

class biosimulators_test_suite.test_case.log.SimulatorReportsTheStatusOfTheExecutionOfSedOutputs(*args, **kwargs)[source]#

Bases: LoggingTestCase

Test that simulator logs the execution of individual SED outputs in COMBINE/OMEX archives

VALIDATE_OUTPUT_LOGS = True[source]#

VALIDATE_SED_DOCUMENT_LOGS = True[source]#

is_concrete()[source]#

Whether the class is concrete

Returns:: whether the class is concrete
Return type:: bool

class biosimulators_test_suite.test_case.log.SimulatorReportsTheStatusOfTheExecutionOfSedTasks(*args, **kwargs)[source]#

Bases: LoggingTestCase

Test that simulator logs the execution of individual SED tasks in COMBINE/OMEX archives

VALIDATE_SED_DOCUMENT_LOGS = True[source]#

VALIDATE_TASK_LOGS = True[source]#

is_concrete()[source]#

Whether the class is concrete

Returns:: whether the class is concrete
Return type:: bool

biosimulators_test_suite.test_case.published_project module#

Methods for test cases involving manually curated COMBINE/OMEX archives

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-21
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.published_project.ConfigurableMasterCombineArchiveTestCase(*args, **kwargs)[source]#

Bases: SyntheticCombineArchiveTestCase

Class for generating synthetic archives with a single master SED-ML file or two non-master copies of the same file

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

_model_change_filter[source]#

filter for model changes to keep

Type:: types.FunctionType

_remove_algorithm_parameter_changes[source]#

if True, remove instructions to change the values of the parameters of algorithms

Type:: bool

_use_single_variable_data_generators[source]#

if True, replace data generators that involve multiple variables or parameters (and data sets, curves, and surfaces) with multiple data generators for each variable

Type:: bool

_remove_plots[source]#

if True, remove plots

Type:: bool

_keep_one_task_one_report[source]#

if True, keep only 1 task and only 1 report

Type:: bool

_expected_report_ids[source]#

ids of expected reports

Type:: list of str

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with master and non-master SED documents

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

class biosimulators_test_suite.test_case.published_project.ExpectedResultOfSyntheticArchive(archive, sed_documents, is_success_expected=True, environment=None)[source]#

Bases: object

An expected result of executing a synthetic COMBINE/OMEX archive

archive[source]#

synthetic COMBINE/OMEX archive for testing the simulator

Type:: CombineArchive

sed_documents[source]#

map from locations to SED documents in synthetic archive

Type:: dict of str to SedDocument

is_success_expected[source]#

whether the execution of the archive is expected to succeed

Type:: bool, optional

environment[source]#

environment variables that the archive should be executed with

Type:: dict, optional

class biosimulators_test_suite.test_case.published_project.SimulatorCanExecutePublishedProject(id=None, name=None, filename=None, task_requirements=None, skipped_simulators=None, expected_reports=None, expected_plots=None, runtime_failure_alert_type=AlertType.exception, assert_no_extra_reports=False, assert_no_extra_datasets=False, assert_no_missing_plots=False, assert_no_extra_plots=False, r_tol=0.0001, a_tol=0.0, minimum_number_of_synthetic_uniform_time_steps=10, output_medium=OutputMedium.console)[source]#

Bases: TestCase

A test case for validating a simulator that involves executing a COMBINE/OMEX archive

id[source]#

Type:: str

name[source]#

name

Type:: str

filename[source]#

path to archive

Type:: str

task_requirements[source]#

list of the required model format and simulation algorithm for each task in the COMBINE/OMEX archive

Type:: list of SedTaskRequirements

skipped_simulators[source]#

list of simulation tools to not test the COMBINE archive with

Type:: list of str

expected_reports[source]#

list of reports expected to be produced by algorithm

Type:: list of ExpectedSedReport

expected_plots[source]#

list of plots expected to be produced by algorithm

Type:: list of ExpectedSedPlot

runtime_failure_alert_type[source]#

whether a run-time failure should be raised as an error or warning

Type:: AlertType

assert_no_extra_reports[source]#

if True, raise an exception if the simulator produces unexpected reports

Type:: bool

assert_no_extra_datasets[source]#

if True, raise an exception if the simulator produces unexpected datasets

Type:: bool

assert_no_missing_plots[source]#

if True, raise an exception if the simulator doesn’t produce the expected plots

Type:: bool

assert_no_extra_plots[source]#

if True, raise an exception if the simulator produces unexpected plots

Type:: bool

r_tol[source]#

relative tolerence

Type:: float

a_tol[source]#

absolute tolerence

Type:: float

minimum_number_of_synthetic_uniform_time_steps[source]#

minimum number of steps to use for derived simulation experiments

Type:: int

compatible_with_specifications(specifications)[source]#

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Raises:

SkippedTestCaseException – if the test case is not applicable to the simulator
Exception – if the simulator did not pass the test case

exec_sedml_docs_in_archive(specifications, out_dir, cli=None)[source]#

Parameters:

specifications (dict) – specifications of the simulator to validate
out_dir (str) – path to save simulation results
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

from_dict(data)[source]#

Read test case from dictionary

Parameters:: data (dict) – dictionary with test case data
Returns:: this object
Return type:: SimulatorCanExecutePublishedProject

from_json(base_path, filename)[source]#

Read test case from JSON file

Parameters:

base_path (str) – bath directory for test cases
filename (str) – JSON file relative to base_path

Returns:

this object

Return type:

SimulatorCanExecutePublishedProject

get_description()[source]#

Get a description of the case

Returns:: description of the case
Return type:: str

class biosimulators_test_suite.test_case.published_project.SingleMasterSedDocumentCombineArchiveTestCase(*args, **kwargs)[source]#

Bases: ConfigurableMasterCombineArchiveTestCase

Class for generating synthetic COMBINE/OMEX archives with a single master SED-ML file

_archive_has_master[source]#

whether the synthetic archive should have a master file

Type:: bool

class biosimulators_test_suite.test_case.published_project.SyntheticCombineArchiveTestCase(id=None, name=None, description=None, output_medium=OutputMedium.console, published_projects_test_cases=None)[source]#

Bases: TestCase

Test that involves a computationally-generated COMBINE/OMEX archive

id[source]#

Type:: str

name[source]#

name

Type:: str

description[source]#

description

Type:: str

output_medium[source]#

medium the description should be formatted for

Type:: OutputMedium

published_projects_test_cases[source]#

curated COMBINE/OMEX archives that can be used to generate example archives for testing

Type:: list of SimulatorCanExecutePublishedProject

_published_projects_test_case[source]#

COMBINE/OMEX archive that is used to generate example archives for testing

Type:: SimulatorCanExecutePublishedProject

EXEC_WITH_SINGULARITY = False[source]#

REPORT_ERROR_AS_SKIP = False[source]#

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

eval(specifications, working_dirname, synthetic_archives_dir=None, dry_run=False, cli=None)[source]#

Evaluate a simulator’s performance on a test case

Parameters:

specifications (dict) – specifications of the simulator to validate
working_dirname (str) – directory for temporary files for evaluating test case
synthetic_archives_dir (str, optional) – Directory to save the synthetic COMBINE/OMEX archives generated by the test cases
dry_run (bool) – if True, do not use the simulator to execute COMBINE/OMEX archives.
cli (str, optional) – command-line interface to use to execute the tests involving the simulation of COMBINE/OMEX archives rather than a Docker image

Returns:

whether there were no warnings about the outputs

Return type:

bool

Raises:

Exception – if the simulator did not pass the test case

abstract eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

is_curated_archive_suitable_for_building_synthetic_archive(specifications, archive, sed_docs)[source]#

Find an archive with at least one report

Parameters:

specifications (dict) – specifications of the simulator to validate
archive (CombineArchive) – curated COMBINE/OMEX archive
sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

True, if the curated archive is suitable for generating a synthetic: archive for testing

Return type:

bool

is_curated_sed_algorithm_suitable_for_building_synthetic_archive(specifications, algorithm)[source]#

Determine if a SED algorithm is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
algorithm (Algorithm) – SED algorithm in curated archive

Returns:

whether the algorithm is suitable for testing

Return type:

bool

is_curated_sed_doc_suitable_for_building_synthetic_archive(specifications, sed_doc, sed_doc_location)[source]#

Determine if a SED document is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
sed_doc (SedDocument) – SED document in curated archive
sed_doc_location (str) – location of the SED document within its parent COMBINE/OMEX archive

Returns:

whether the SED document is suitable for testing

Return type:

bool

is_curated_sed_model_suitable_for_building_synthetic_archive(specifications, model)[source]#

Determine if a SED model is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
model (Model) – SED model in curated archive

Returns:

whether the model is suitable for testing

Return type:

bool

is_curated_sed_report_suitable_for_building_synthetic_archive(specifications, report, sed_doc_location)[source]#

Determine if a SED report is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
report (Report) – SED report in curated archive
sed_doc_location (str) – location of the SED document within its parent COMBINE/OMEX archive

Returns:

whether the report is suitable for testing

Return type:

bool

is_curated_sed_simulation_suitable_for_building_synthetic_archive(specifications, simulation)[source]#

Determine if a SED simulation is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
simulation (Simulation) – SED simulation in curated archive

Returns:

whether the simulation is suitable for testing

Return type:

bool

is_curated_sed_task_suitable_for_building_synthetic_archive(specifications, task)[source]#

Determine if a SED task is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
task (Task) – SED task in curated archive

Returns:

whether the task is suitable for testing

Return type:

bool

class biosimulators_test_suite.test_case.published_project.UniformTimeCourseTestCase(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that a simulator supports multiple reports per SED document

TEST_TIME = True[source]#

add_time_data_set(doc, task, report)[source]#

Rename or add a time data set to a SED report

Parameters:

doc (SedDocument) – SED document
task (task) – SED task
report (Report) – SED report

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

is_curated_sed_task_suitable_for_building_synthetic_archive(specifications, task)[source]#

Determine if a SED task is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
task (Task) – SED task in curated archive

Returns:

whether the task is suitable for testing

Return type:

bool

abstract modify_simulation(simulation)[source]#

Modify a simulation

Parameters:: simulation (UniformTimeCourseSimulation) – simulation

biosimulators_test_suite.test_case.published_project.find_cases(specifications, dir_name=None, output_medium=OutputMedium.console)[source]#

Collect test cases

Parameters:

specifications (dict) – specifications of the simulator to validate
dir_name (str, optional) – path to find example COMBINE/OMEX archives
output_medium (OutputMedium, optional) – medium the description should be formatted for

Returns:

test cases

Return type:

list of SimulatorCanExecutePublishedProject

biosimulators_test_suite.test_case.results_report module#

Methods for checking support for reports of simulation results

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-29
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.results_report.SimulatorGeneratesReportsOfSimulationResults(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that when a COMBINE/OMEX archive defines a (single) master file, the simulator only executes this file.

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

biosimulators_test_suite.test_case.sedml module#

Methods for testing that simulators support the features of SED-ML

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2020-12-21
Copyright:: 2020, Center for Reproducible Biomedical Modeling
License:: MIT

class biosimulators_test_suite.test_case.sedml.SimulatorProducesLinear2DPlots(*args, **kwargs)[source]#

Bases: SimulatorProduces2DPlotsTestCase

Test that a simulator produces linear 2D plots

class biosimulators_test_suite.test_case.sedml.SimulatorProducesLinear3DPlots(*args, **kwargs)[source]#

Bases: SimulatorProduces3DPlotsTestCase

Test that a simulator produces linear 3D plots

class biosimulators_test_suite.test_case.sedml.SimulatorProducesLogarithmic2DPlots(*args, **kwargs)[source]#

Bases: SimulatorProduces2DPlotsTestCase

Test that a simulator produces logarithmic 2D plots

class biosimulators_test_suite.test_case.sedml.SimulatorProducesLogarithmic3DPlots(*args, **kwargs)[source]#

Bases: SimulatorProduces3DPlotsTestCase

Test that a simulator produces logarithmic 3D plots

class biosimulators_test_suite.test_case.sedml.SimulatorProducesMultiplePlots(*args, **kwargs)[source]#

Bases: SimulatorProduces2DPlotsTestCase

Test that a simulator produces multiple plots

class biosimulators_test_suite.test_case.sedml.SimulatorProducesReportsWithCuratedNumberOfDimensions(*args, **kwargs)[source]#

Bases: SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports

Test that that the curated number of output dimensions matches the actual number of output dimensions

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

is_curated_sed_algorithm_suitable_for_building_synthetic_archive(specifications, algorithm)[source]#

Determine if a SED algorithm is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
algorithm (Algorithm) – SED algorithm in curated archive

Returns:

whether the algorithm is suitable for testing

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsAlgorithmParameters(*args, **kwargs)[source]#

Bases: SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports

Test that a simulator supports setting the values of parameters of algorithms

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

is_curated_sed_algorithm_suitable_for_building_synthetic_archive(specifications, algorithm)[source]#

Determine if a SED algorithm is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
algorithm (Algorithm) – SED algorithm in curated archive

Returns:

whether the algorithm is suitable for testing

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsModelAttributeChanges(*args, **kwargs)[source]#

Bases: SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports

Test that a simulator supports changes to the attributes of model elements

REPORT_ERROR_AS_SKIP = True[source]#

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that a simulator supports the core elements of SED: models, simulations, tasks, data generators for individual variables, and reports

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsMultipleReportsPerSedDocument(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that a simulator supports multiple reports per SED document

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsMultipleTasksPerSedDocument(*args, **kwargs)[source]#

Bases: SingleMasterSedDocumentCombineArchiveTestCase

Test that a simulator supports multiple tasks per SED document

_expected_reports[source]#

list of pairs of original reports and their expected duplicates

Type:: list of tuple of str

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with a copy of each task and each report

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

eval_outputs(specifications, synthetic_archive, synthetic_sed_docs, outputs_dir)[source]#

Test that the expected outputs were created for the synthetic archive

Parameters:

specifications (dict) – specifications of the simulator to validate
synthetic_archive (CombineArchive) – synthetic COMBINE/OMEX archive for testing the simulator
synthetic_sed_docs (dict of str to SedDocument) – map from the location of each SED document in the synthetic archive to the document
outputs_dir (str) – directory that contains the outputs produced from the execution of the synthetic archive

Returns:

whether there were no warnings about the outputs

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithFunctionalRangeVariables(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over nested functional ranges based on model (specification) variables

FUNCTONAL_RANGE_USES_VARIABLES = True[source]#

HAS_CHANGES = True[source]#

NUM_NESTED_RANGES = 0[source]#

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of FunctionalRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithFunctionalRanges(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over functional ranges

FUNCTONAL_RANGE_USES_VARIABLES = False[source]#

NUM_NESTED_RANGES = 0[source]#

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of FunctionalRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithLinearUniformRanges(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over uniform ranges

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of UniformRange

UNIFORM_RANGE_TYPE = 'linear'[source]#

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithLogarithmicUniformRanges(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over uniform ranges

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of UniformRange

UNIFORM_RANGE_TYPE = 'log'[source]#

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithMultipleSubTasks(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks with multiple subtasks

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 2[source]#

RANGE_TYPE[source]#: alias of VectorRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithNestedFunctionalRanges(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over nested functional ranges based on model (specification) variables

FUNCTONAL_RANGE_USES_VARIABLES = False[source]#

NUM_NESTED_RANGES = 1[source]#

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of FunctionalRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithNestedRepeatedTasks(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports nested repeated tasks

MIXED_SUB_TASK_TYPES = False[source]#

NUM_NESTED_REPEATED_TASKS = 1[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_LENS = (3, 3)[source]#

RANGE_TYPE[source]#: alias of VectorRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithSubTasksOfMixedTypes(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks whose sub-tasks have mixed types. Also tests that sub-types executed in order of the values of their order attributes and that reports of the results of repeated tasks handle sub-tasks to produce results of different shapes.

MIXED_SUB_TASK_TYPES = True[source]#

NUM_NESTED_REPEATED_TASKS = 1[source]#

NUM_SUB_TASKS = 2[source]#

RANGE_LENS = (2, 3)[source]#

RANGE_TYPE[source]#: alias of VectorRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsRepeatedTasksWithVectorRanges(*args, **kwargs)[source]#

Bases: RepeatedTasksTestCase

Test that a simulator supports repeated tasks over vector ranges

NUM_NESTED_REPEATED_TASKS = 0[source]#

NUM_SUB_TASKS = 1[source]#

RANGE_TYPE[source]#: alias of VectorRange

is_concrete()[source]#

Whether the class is abstract

Returns:: whether the class is abstract
Return type:: bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsSubstitutingAlgorithms(*args, **kwargs)[source]#

Bases: SimulatorSupportsModelsSimulationsTasksDataGeneratorsAndReports

Check that a simulator can substitute algorithms that it doesn’t implement with similar algorithms when the algorithm substitution policy is less restrictive than AlgorithmSubstitutionPolicy.SAME_METHOD. Also check that a simulator ignores unsupported algorithm parameters when the algorithm substitution policy is less restrictive than AlgorithmSubstitutionPolicy.NONE.

REPORT_ERROR_AS_SKIP = True[source]#

build_synthetic_archives(specifications, curated_archive, curated_archive_dir, curated_sed_docs)[source]#

Generate a synthetic archive with master and non-master SED documents

Parameters:

specifications (dict) – specifications of the simulator to validate
curated_archive (CombineArchive) – curated COMBINE/OMEX archive
curated_archive_dir (str) – directory with the contents of the curated COMBINE/OMEX archive
curated_sed_docs (dict of str to SedDocument) – map from locations to SED documents in curated archive

Returns:

list of ExpectedResultOfSyntheticArchive

is_curated_sed_algorithm_suitable_for_building_synthetic_archive(specifications, algorithm)[source]#

Determine if a SED algorithm is suitable for testing

Parameters:

specifications (dict) – specifications of the simulator to validate
algorithm (Algorithm) – SED algorithm in curated archive

Returns:

whether the algorithm is suitable for testing

Return type:

bool

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsUniformTimeCoursesWithNonZeroInitialTimes(*args, **kwargs)[source]#

Bases: UniformTimeCourseTestCase

Test that a simulator supports multiple time courses with non-zero initial times

REPORT_ERROR_AS_SKIP = True[source]#

TEST_TIME = True[source]#

modify_simulation(simulation)[source]#

Modify a simulation

Parameters:: simulation (UniformTimeCourseSimulation) – simulation

class biosimulators_test_suite.test_case.sedml.SimulatorSupportsUniformTimeCoursesWithNonZeroOutputStartTimes(*args, **kwargs)[source]#

Bases: UniformTimeCourseTestCase

Test that a simulator supports time courses with non-zero output start times

TEST_TIME = True[source]#

modify_simulation(simulation)[source]#

Modify a simulation

Parameters:: simulation (UniformTimeCourseSimulation) – simulation

biosimulators_test_suite.test_case.utils module#

Utility methods

Author:: Jonathan Karr <karr@mssm.edu>
Date:: 2021-01-02
License:: MIT

biosimulators_test_suite.test_case.utils.are_array_shapes_equivalent(shape_1, shape_2, same_dims=False)[source]#

Determine if two NumPy arrays have equivalent shape

Trim trailing ones
Check that remaining non-zero dimension sizes are equal
If same_dims is True, all check that the length of the shapes are equal

Parameters:

shape_1 (list of int) – shape of first array
shape_2 (list of int) – shape of second array
same_dims (bool, optional) – if True, check that the dimensions are also the same

Returns:

True if the shapes are equivalent

Return type:

bool

biosimulators_test_suite.test_case.utils.reduce_array_shape(shape)[source]#

Reduce the shape of a NumPy array

Trim trailing ones
Check that remaining non-zero dimension sizes are equal

Parameters:: shape (list of int) – shape of array
Returns:: reduced shape
Return type:: :obj:`list of int