Example usage with the CLI

This notebook demonstrates usage of petab_select to perform model selection with commands.

Note that the criterion values in this notebook are for demonstrative purposes only, and are not real. An additional point is that models store the iteration where they were calibrated, but the iteration counter is stored in the candidate space. Hence, when the candidate space (or method) changes in this notebook, the iteration counter is reset.

# Cleanup the state and candidate models output by a previous run of this notebook
import shutil
from pathlib import Path

output_path = Path().resolve() / "output_cli"
output_path_str = str(output_path)
if output_path.exists():
    shutil.rmtree(output_path_str)
output_path.mkdir(exist_ok=False, parents=True)

First iteration

Iterations of model selection start and end with start_iteration and end_iteration.

In each call to petab_select start_iteration, the following options are required:

• --problem: The PEtab Select problem YAML file, which normally defines the method too

• --state: A file that is used to store the state of the problem (e.g., such that models are not revisited)

• --output: A file to store the models proposed by PEtab Select

Other options can be viewed with petab_select start_iteration --help.

In this initial call, a PEtab Select problem is used to identify possible models for selection. Instead of using the method defined in the PEtab Select problem, we use the brute force method, which normally outputs all possible models. Here, the number of models in the output is explicitly limited to 3. Subsequent calls with the same command will output different models.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select start_iteration \
--problem model_selection/petab_select_problem.yaml \
--state $output_path_str/state.dill \
--method brute_force \
--limit 3 \
--output-uncalibrated-models $output_path_str/uncalibrated_models_1.yaml \
--relative-paths

The output format is a list of the PEtab Select model YAML format.

with open(output_path / "uncalibrated_models_1.yaml") as f:
    print(f.read())

- model_subspace_id: M1_0
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_0-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_0-000
  model_label: null
  parameters:
    k1: 0
    k2: 0
    k3: 0
  predecessor_model_hash: virtual_initial_model-
- model_subspace_id: M1_1
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_1-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_1-000
  model_label: null
  parameters:
    k1: 0.2
    k2: 0.1
    k3: estimate
  predecessor_model_hash: virtual_initial_model-
- model_subspace_id: M1_2
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_2-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_2-000
  model_label: null
  parameters:
    k1: 0.2
    k2: estimate
    k3: 0
  predecessor_model_hash: virtual_initial_model-

At this point, the calibration tool should calibrate the models, and save the calibration results to disk in the PEtab Select model YAML format. For this example, we have stored the results in model_selection/calibrated_models_1.yaml.

Next, we finalize the iteration by calling petab_select end_iteration, which requires:

• --state: see start_iteration

• --output-models: A file used to store the models from this iteration of calibration. Note that, if the user has supplied calibration results from previous model selection jobs, then this end_iteration output might differ from the output of the calibration tool. This end_iteration output should be considered the real set of models calibrated in this iteration.

• --output-metadata: A file where metadata from the iteration is stored. This includes the signal of whether model selection has terminated.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select end_iteration \
--state=$output_path_str/state.dill \
--calibrated-models=model_selection/calibrated_models_1.yaml \
--output-models=$output_path_str/models_1.yaml \
--output-metadata=$output_path_str/metadata.yaml \
--relative-paths

Second iteration

Between iterations, the models from the first iteration have been calibrated, and the model with the best criterion value is M1_2. In this iteration, we will switch to the forward method and manually specify M1_2 as the predecessor model. In subsequent iterations, the predecessor model will default to the best model of the previous iteration.

with open("model_selection/calibrated_models_1.yaml") as f:
    print(f.read())

- model_subspace_id: M1_0
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria:
    AIC: 180
  model_hash: M1_0-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: {}
  iteration: 1
  model_id: M1_0-000
  parameters:
    k1: 0
    k2: 0
    k3: 0
  predecessor_model_hash: virtual_initial_model-
- model_subspace_id: M1_1
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria:
    AIC: 100
  model_hash: M1_1-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: {}
  iteration: 1
  model_id: M1_1-000
  parameters:
    k1: 0.2
    k2: 0.1
    k3: estimate
  predecessor_model_hash: virtual_initial_model-

- model_subspace_id: M1_2
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria:
    AIC: 50
  model_hash: M1_2-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: {}
  iteration: 1
  model_id: M1_2-000
  parameters:
    k1: 0.2
    k2: estimate
    k3: 0
  predecessor_model_hash: virtual_initial_model-

To perform the method change, we: delete the current state, select and store the model M1_2 to disk via petab_select get_best, customize the problem to use the predecessor model via candidate_space_arguments, and supply the new method to petab_select start_iteration.

%%bash -s "$output_path_str"
output_path_str=$1

# save the best model of the previous iteration (`M1_2`)
petab_select get_best \
--problem model_selection/petab_select_problem.yaml \
--models model_selection/calibrated_models_1.yaml \
--output $output_path_str/predecessor_model.yaml
# create a copy of the original PEtab select problem and update its paths
cp model_selection/petab_select_problem.yaml $output_path_str/custom_problem.yaml
sed -i 's|- model_space.tsv|- ../model_selection/model_space.tsv|' $output_path_str/custom_problem.yaml
# add the predecessor model to the problem copy
echo """candidate_space_arguments:
  predecessor_model: predecessor_model.yaml
""" >> $output_path_str/custom_problem.yaml
# remove the state from the previous iteration
rm $output_path_str/state.dill

# request models with the customized problem with the predecessor model `M1_2`, using the forward method
petab_select start_iteration \
--problem $output_path_str/custom_problem.yaml \
--state $output_path_str/state.dill \
--output-uncalibrated-models $output_path_str/uncalibrated_models_2.yaml \
--method forward \
--relative-paths

M1_2 has one estimated parameter, k2(*). As expected, the new candidates identified with the forward method have two estimated parameters, and one of them is k2.

(*) There may be additional estimated parameters specified in the PEtab problem, which are not a part of the model selection problem.

with open(output_path / "uncalibrated_models_2.yaml") as f:
    print(f.read())

- model_subspace_id: M1_4
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_4-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_4-000
  model_label: null
  parameters:
    k1: 0.2
    k2: estimate
    k3: estimate
  predecessor_model_hash: M1_2-000
- model_subspace_id: M1_6
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_6-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_6-000
  model_label: null
  parameters:
    k1: estimate
    k2: estimate
    k3: 0
  predecessor_model_hash: M1_2-000

The calibration tool does not need to calibrate every uncalibrated model. For example, the calibration tool might return all calibration results, as soon as an improvement over the previous iteration is identified. Here, we only return the results for the M1_4 model.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select end_iteration \
--state=$output_path_str/state.dill \
--calibrated-models=model_selection/calibrated_M1_4.yaml \
--output-models=$output_path_str/models_2.yaml \
--output-metadata=$output_path_str/metadata.yaml \
--relative-paths

Third iteration

The models from the previous iteration (i.e. M1_4) were stored in the state. Here, we perform an iteration of the forward method, which is automatically initialized with the M1_4 model.

with open("model_selection/calibrated_M1_4.yaml") as f:
    print(f.read())

model_subspace_id: M1_4
model_subspace_indices:
- 0
- 0
- 0
criteria:
  AIC: 15
model_hash: M1_4-000
model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
estimated_parameters:
  k2: 0.15
  k3: 0.0
iteration: 1
model_id: M1_4-000
parameters:
  k1: 0
  k2: estimate
  k3: estimate
predecessor_model_hash: M1_2-000

%%bash -s "$output_path_str"
output_path_str=$1

petab_select start_iteration \
--problem $output_path_str/custom_problem.yaml \
--state $output_path_str/state.dill \
--output-uncalibrated-models $output_path_str/uncalibrated_models_3.yaml \
--method forward \
--relative-paths

As we are performing a forward search from M1_4, which has two parameters, then all models in this iteration will have 3+ parameters. This model space contains only one model with 3 or more estimated parameters. We finalize the iteration with its calibration results.

with open(output_path / "uncalibrated_models_3.yaml") as f:
    print(f.read())

- model_subspace_id: M1_7
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_7-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 2
  model_id: M1_7-000
  model_label: null
  parameters:
    k1: estimate
    k2: estimate
    k3: estimate
  predecessor_model_hash: M1_4-000

%%bash -s "$output_path_str"
output_path_str=$1

petab_select end_iteration \
--state=$output_path_str/state.dill \
--calibrated-models=model_selection/calibrated_M1_7.yaml \
--output-models=$output_path_str/models_3.yaml \
--output-metadata=$output_path_str/metadata.yaml \
--relative-paths

Fourth iteration

As there are no models in the model space with 4+ parameters, subsequent forward searches will return no candidate models. Tools can detect when to terminate by inspecting the metadata produced by end_iteration, as demonstrated at the end of this iteration.

with open("model_selection/calibrated_M1_7.yaml") as f:
    print(f.read())

model_subspace_id: M1_7
model_subspace_indices:
- 0
- 0
- 0
criteria:
  AIC: 20
model_hash: M1_7-000
model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
estimated_parameters:
  k1: 0.25
  k2: 0.1
  k3: 0.0
iteration: 2
model_id: M1_7-000
parameters:
  k1: estimate
  k2: estimate
  k3: estimate
predecessor_model_hash: M1_4-000

%%bash -s "$output_path_str"
output_path_str=$1

petab_select start_iteration \
--problem model_selection/petab_select_problem.yaml \
--state $output_path_str/state.dill \
--output-uncalibrated-models $output_path_str/uncalibrated_models_4.yaml \
--method forward \
--relative-paths

with open(output_path / "uncalibrated_models_4.yaml") as f:
    print(f.read())

[]

%%bash -s "$output_path_str"
output_path_str=$1

petab_select end_iteration \
--state=$output_path_str/state.dill \
--output-models=$output_path_str/models_4.yaml \
--output-metadata=$output_path_str/metadata.yaml \
--relative-paths

with open("output_cli/metadata.yaml") as f:
    print(f.read())

terminate: true

Fifth iteration

Although no additional models are found with a forward search initialized at the best model so far (M1_7), there are additional models in the model space that were not calibrated, which can be identified by using the brute force method with exclusions for the calibrated models.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select start_iteration \
--problem model_selection/petab_select_problem.yaml \
--state $output_path_str/state_5.dill \
--output-uncalibrated-models $output_path_str/uncalibrated_models_5.yaml \
--method brute_force \
--excluded-models $output_path_str/models_1.yaml \
--excluded-models $output_path_str/models_2.yaml \
--excluded-models $output_path_str/models_3.yaml \
--relative-paths

with open(output_path / "uncalibrated_models_5.yaml") as f:
    print(f.read())

- model_subspace_id: M1_3
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_3-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_3-000
  model_label: null
  parameters:
    k1: estimate
    k2: 0.1
    k3: 0
  predecessor_model_hash: virtual_initial_model-
- model_subspace_id: M1_5
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_5-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_5-000
  model_label: null
  parameters:
    k1: estimate
    k2: 0.1
    k3: estimate
  predecessor_model_hash: virtual_initial_model-
- model_subspace_id: M1_6
  model_subspace_indices:
  - 0
  - 0
  - 0
  criteria: {}
  model_hash: M1_6-000
  model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
  estimated_parameters: null
  iteration: 1
  model_id: M1_6-000
  model_label: null
  parameters:
    k1: estimate
    k2: estimate
    k3: 0
  predecessor_model_hash: virtual_initial_model-

Post-processing

After the selection algorithm has terminated, the best model can be stored separately by supplying a list of calibrated models.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select get_best \
--problem model_selection/petab_select_problem.yaml \
--models $output_path_str/models_1.yaml \
--models $output_path_str/models_2.yaml \
--models $output_path_str/models_3.yaml \
--output $output_path_str/best_model.yaml \
--state $output_path_str/state.dill \
--relative-paths

with open(output_path / "best_model.yaml") as f:
    print(f.read())

model_subspace_id: M1_4
model_subspace_indices:
- 0
- 0
- 0
criteria:
  AIC: 15.0
model_hash: M1_4-000
model_subspace_petab_yaml: ../model_selection/petab_problem.yaml
estimated_parameters:
  k2: 0.15
  k3: 0.0
iteration: 1
model_id: M1_4-000
model_label: null
parameters:
  k1: 0
  k2: estimate
  k3: estimate
predecessor_model_hash: M1_2-000

This model can be converted to a PEtab problem with either model_to_petab or models_to_petab.

%%bash -s "$output_path_str"
output_path_str=$1

petab_select model_to_petab \
--model $output_path_str/best_model.yaml \
--output $output_path_str/best_model_petab

/home/docs/checkouts/readthedocs.org/user_builds/petab-select/checkouts/latest/doc/examples/output_cli/best_model_petab/problem.yaml