FAMoS in PEtab Select

In this notebook, the FAMoS algorithm [1] is demonstrated.

[1] Gabel M, Hohl T, Imle A, Fackler OT, Graw F (2019) FAMoS: A Flexible and dynamic Algorithm for Model Selection to analyse complex systems dynamics. PLOS Computational Biology 15(8): e1007230. https://doi.org/10.1371/journal.pcbi.1007230

The model space contains 65536 models. In normal PEtab Select workflows, a calibration tool would take the place of the example_cli_famos_calibration_tool.py script. This script emulates a calibration tool: it takes PEtab Select models and assigns criterion values to them, based on previous calibration results for the same models.

# Cleanup the state and candidate models output by a previous run of this notebook
import shutil
from pathlib import Path

from example_cli_famos_helpers import (
    expected_criterion_values,
    parse_summary_to_progress_list,
    petab_select_problem_yaml,
)

output_path = Path().resolve() / "output_famos"
output_path_str = str(output_path)
shutil.rmtree(output_path_str)
output_path.mkdir(exist_ok=False, parents=True)

import numpy as np
import pandas as pd
import petab

import petab_select
from petab_select import ESTIMATE, FamosCandidateSpace, Method, Model
from petab_select.constants import Criterion
from petab_select.model import default_compare

state = str(output_path / 'state.dill')

# Each iteration of model selection is described as a 2-tuple here.
# First value is the model selection method.
# Second value are relative change in parameter indices that correspond
# to the best model from this iteration.
# e.g. `(Method.FORWARD, {3})` states that the best model from a forward move
# is the model that now estimates the parameter at index 3.
expected_progress_list = [
    (Method.LATERAL, set()),
    (Method.LATERAL, {4, 15}),
    (Method.LATERAL, {9, 13}),
    (Method.FORWARD, set()),
    (Method.FORWARD, {3}),
    (Method.FORWARD, {11}),
    (Method.BACKWARD, set()),
    (Method.BACKWARD, {6}),
    (Method.BACKWARD, {10}),
    (Method.BACKWARD, {8}),
    (Method.BACKWARD, {14}),
    (Method.BACKWARD, {1}),
    (Method.BACKWARD, {16}),
    (Method.BACKWARD, {4}),
    (Method.FORWARD, set()),
    (Method.LATERAL, set()),
    (Method.MOST_DISTANT, {2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15}),
    (Method.LATERAL, {16, 7}),
    (Method.LATERAL, {5, 12}),
    (Method.LATERAL, {13, 15}),
    (Method.LATERAL, {1, 6}),
    (Method.FORWARD, set()),
    (Method.FORWARD, {3}),
    (Method.FORWARD, {7}),
    (Method.FORWARD, {2}),
    (Method.FORWARD, {11}),
    (Method.BACKWARD, set()),
    (Method.BACKWARD, {7}),
    (Method.BACKWARD, {16}),
    (Method.BACKWARD, {4}),
    (Method.FORWARD, set()),
    (Method.LATERAL, set()),
    (Method.LATERAL, {9, 15}),
    (Method.FORWARD, set()),
    (Method.BACKWARD, set()),
    (Method.LATERAL, set()),
]

The predecessor model is some model from the model space, and is defined in the PEtab Select problem YAML file.

%%bash -s "$petab_select_problem_yaml" "$output_path_str"

petab_select_problem_yaml=$1
output_path_str=$2

problem=$petab_select_problem_yaml
state=$output_path_str/state.dill

newly_calibrated_models=""
calibrated_models=""
empty_output=$(echo -e "null\n...")

for i in {1..40}
do

output=$output_path_str/models_$i.yaml
calibrated_output=$output_path_str/calibrated_models_$i.yaml

petab_select candidates \
--problem $problem \
--state $state \
--output $output \
$newly_calibrated_models \
$calibrated_models \
--relative-paths

# Replace this line with a tool that calibrates the models.
# The script also changes model IDs for easier analysis in this example.
python example_cli_famos_calibration_tool.py $output $calibrated_output

if [ "$(cat $calibrated_output)" == "$empty_output" ]
then
        # End the model selection if no models were provided by PEtab Select.
        break
else
        newly_calibrated_models="--newly-calibrated-models "$calibrated_output
        calibrated_models+="--calibrated-models "$calibrated_output" "
fi

done

progress_list = parse_summary_to_progress_list('output_famos/summary.tsv')

assert progress_list == expected_progress_list