FAMoS in PEtab Select

In this notebook, the FAMoS algorithm [1] is demonstrated. This is designed as an example for tool developers to be able to ensure that they utilize PEtab Select correctly, such that complex methods like FAMoS work.

[1] Gabel M, Hohl T, Imle A, Fackler OT, Graw F (2019) FAMoS: A Flexible and dynamic Algorithm for Model Selection to analyse complex systems dynamics. PLOS Computational Biology 15(8): e1007230. https://doi.org/10.1371/journal.pcbi.1007230

The model space contains 65536 models. In normal PEtab Select workflows, a calibration tool would take the place of the example_cli_famos_calibration_tool.py script. This script emulates a calibration tool: it takes PEtab Select models and assigns criterion values to them, based on previous calibration results for the same models.

# Cleanup the state and candidate models output by a previous run of this notebook
import shutil
from pathlib import Path

from example_cli_famos_helpers import (
    parse_summary_to_progress_list,
    petab_select_problem_yaml,  # noqa: F401
)

output_path = Path().resolve() / "output_famos"
output_path_str = str(output_path)
if output_path.exists():
    shutil.rmtree(output_path_str)
output_path.mkdir(exist_ok=False, parents=True)

from petab_select import Method

state = str(output_path / "state.dill")

# Each iteration of model selection is described as a 2-tuple here.
# First value is the model selection method.
# Second value are relative change in parameter indices that correspond
# to the best model from this iteration.
# e.g. `(Method.FORWARD, {3})` states that the best model from a forward move
# is the model that now estimates the parameter at index 3.
expected_progress_list = [
    (Method.LATERAL, set()),
    (Method.LATERAL, {4, 15}),
    (Method.LATERAL, {9, 13}),
    (Method.FORWARD, set()),
    (Method.FORWARD, {3}),
    (Method.FORWARD, {11}),
    (Method.BACKWARD, set()),
    (Method.BACKWARD, {6}),
    (Method.BACKWARD, {10}),
    (Method.BACKWARD, {8}),
    (Method.BACKWARD, {14}),
    (Method.BACKWARD, {1}),
    (Method.BACKWARD, {16}),
    (Method.BACKWARD, {4}),
    (Method.FORWARD, set()),
    (Method.LATERAL, set()),
    (Method.MOST_DISTANT, {2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15}),
    (Method.LATERAL, {16, 7}),
    (Method.LATERAL, {5, 12}),
    (Method.LATERAL, {13, 15}),
    (Method.LATERAL, {1, 6}),
    (Method.FORWARD, set()),
    (Method.FORWARD, {3}),
    (Method.FORWARD, {7}),
    (Method.FORWARD, {2}),
    (Method.FORWARD, {11}),
    (Method.BACKWARD, set()),
    (Method.BACKWARD, {7}),
    (Method.BACKWARD, {16}),
    (Method.BACKWARD, {4}),
    (Method.FORWARD, set()),
    (Method.LATERAL, set()),
    (Method.LATERAL, {9, 15}),
    (Method.FORWARD, set()),
    (Method.BACKWARD, set()),
    (Method.LATERAL, set()),
]

The predecessor model is some model from the model space, and is defined in the PEtab Select problem YAML file.

%%bash -s "$petab_select_problem_yaml" "$output_path_str"
petab_select_problem_yaml=$1
output_path_str=$2

problem=$petab_select_problem_yaml
state=$output_path_str/state.dill

file_uncalibrated_models=$output_path_str/uncalibrated_models.yaml
file_calibrated_models=$output_path_str/calibrated_models.yaml
file_models=$output_path_str/iteration_models.yaml
file_metadata=$output_path_str/metadata.yaml

for i in {1..40}
do
    echo "Executing iteration $i"

    petab_select start_iteration \
    --problem="$problem" \
    --state="$state" \
    --output-uncalibrated-models="$file_uncalibrated_models" \
    --relative-paths

    # Replace this line with a tool that calibrates the models in `$output`
    # and stores the calibrated models in `$calibrated_output`.
    # This script also changes model IDs for easier analysis in this example.
    python example_cli_famos_calibration_tool.py $file_uncalibrated_models $file_calibrated_models

    petab_select end_iteration \
    --state="$state" \
    --calibrated-models="$file_calibrated_models" \
    --output-models="$file_models" \
    --output-metadata="$file_metadata" \
    --relative-paths

    terminate=$(cat $file_metadata | grep terminate | awk '{print $NF}')
    if [ "$terminate" = "true" ]; then
        echo "Model selection has terminated."
        break
    fi
done

Executing iteration 1
Executing iteration 2

/home/docs/checkouts/readthedocs.org/user_builds/petab-select/envs/latest/lib/python3.11/site-packages/petab_select/candidate_space.py:1143: RuntimeWarning: Model `model_subspace_1-1100110111000111` has been previously excluded from the candidate space so is skipped here.
  return_value = self.inner_candidate_space.consider(model)

Executing iteration 3
Executing iteration 4
Executing iteration 5
Executing iteration 6
Executing iteration 7
Executing iteration 8
Executing iteration 9
Executing iteration 10
Executing iteration 11
Executing iteration 12
Executing iteration 13
Executing iteration 14
Executing iteration 15
Executing iteration 16
Executing iteration 17
Executing iteration 18
Executing iteration 19

/home/docs/checkouts/readthedocs.org/user_builds/petab-select/envs/latest/lib/python3.11/site-packages/petab_select/candidate_space.py:1143: RuntimeWarning: Model `model_subspace_1-0001011010010010` has been previously excluded from the candidate space so is skipped here.
  return_value = self.inner_candidate_space.consider(model)

Executing iteration 20
Executing iteration 21
Executing iteration 22
Executing iteration 23
Executing iteration 24
Executing iteration 25
Executing iteration 26
Executing iteration 27
Executing iteration 28
Executing iteration 29
Executing iteration 30
Executing iteration 31
Executing iteration 32
Executing iteration 33
Executing iteration 34
Executing iteration 35
Executing iteration 36
Executing iteration 37
Model selection has terminated.

progress_list = parse_summary_to_progress_list(output_path / "summary.tsv")

assert progress_list == expected_progress_list