Example usage with the CLI

This notebook demonstrates usage of petab_select to perform model selection with commands.

Note that the criterion values in this notebook are for demonstrative purposes only, and are not real (the models were not calibrated).

# Cleanup the state and candidate models output by a previous run of this notebook
import os

try:
    os.remove('output/state.dill')
except FileNotFoundError:
    pass

try:
    os.remove('output/models.yaml')
except FileNotFoundError:
    pass

First iteration

In each call to petab_select search, the following options are required: - -y: The PEtab Select problem YAML file; - -s: A file that is used to stored the state of the problem (e.g., such that models are not repeated); - -o: A file to store the output of the model space search; and - -m: The method used to search or identify models in the model space.

Other options can be viewed with petab_select search --help.

In this initial call, a PEtab Select problem is used to identify possible models for selection. The brute force method is used, which normally outputs all possible models. Here, the number of models in the output is explicitly limited to 3. Subsequent calls with the same command will output different models.

%%bash

petab_select candidates \
--problem model_selection/petab_select_problem.yaml \
--state output/state.dill \
--output output/models_1.yaml \
--method brute_force \
--relative-paths \
-l 3

The output format is a list of the PEtab Select model YAML format.

with open('output/models_1.yaml') as f:
    print(f.read())

- criteria: {}
  estimated_parameters: {}
  model_hash: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
  model_id: 65f94e223024ef684fe3e1a3aa2a54cc3ffd08895fbe4539512522d49d703ceda483aff4aa207b160dc358c458b76b25d88fbd94cacfc78bd0c70f4a46a42191
  model_subspace_id: M1_0
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: 0
    k2: 0
    k3: 0
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: null
- criteria: {}
  estimated_parameters: {}
  model_hash: 112c344171a01874a0b400640c2e0f72f2924b91712966cb868bf53b6d8ce2d09bb8e56f52b5aaca506a64754629147047646ea0c0cf568d76e74df2c5e2487a
  model_id: 112c344171a01874a0b400640c2e0f72f2924b91712966cb868bf53b6d8ce2d09bb8e56f52b5aaca506a64754629147047646ea0c0cf568d76e74df2c5e2487a
  model_subspace_id: M1_1
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: 0.2
    k2: 0.1
    k3: estimate
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: null
- criteria: {}
  estimated_parameters: {}
  model_hash: df2e1cd0744275a154036b1e1b09eaa67a76f4c08615b3e36849e3eaddcb05d1ccaedb62d148abcc41579314b2e8bec2871a8f925e3d53b90c0a4c6e9ea098ab
  model_id: df2e1cd0744275a154036b1e1b09eaa67a76f4c08615b3e36849e3eaddcb05d1ccaedb62d148abcc41579314b2e8bec2871a8f925e3d53b90c0a4c6e9ea098ab
  model_subspace_id: M1_2
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: 0.2
    k2: estimate
    k3: 0
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: null

Second iteration

Between iterations, the models from the first iteration have been calibrated, and the model with the best criterion value is M1_2. Here, PEtab Select will identify M1_2 as the best model from a YAML file of calibrated models. In the following iterations, a specific model will be provided.

with open('model_selection/calibrated_first_iteration.yaml') as f:
    print(f.read())

- criteria:
    AIC: 180
  model_id: M1_0
  parameters:
    k1: 0.0
    k2: 0.0
    k3: 0.0
  estimated_parameters: null
  petab_yaml: ../model_selection/petab_problem.yaml
- criteria:
    AIC: 100
  model_id: M1_1
  parameters:
    k1: 0.2
    k2: 0.1
    k3: estimate
  estimated_parameters:
    k3: 0.0
  petab_yaml: ../model_selection/petab_problem.yaml
- criteria:
    AIC: 50
  model_id: M1_2
  parameters:
    k1: 0.2
    k2: estimate
    k3: 0.0
  estimated_parameters:
    k2: 0.05
  petab_yaml: ../model_selection/petab_problem.yaml

This is used as the predecessor model for the next search, this time using the forward method. The same state file can be used in each call. If the output file already exists, it will be overwritten.

%%bash

petab_select candidates \
--problem model_selection/petab_select_problem.yaml \
--state output/state.dill \
--output output/models_2.yaml \
--method forward \
--newly-calibrated-models model_selection/calibrated_first_iteration.yaml \
--relative-paths

M1_2 has one estimated parameter, k2 [1]. As expected, the new candidates identified with the forward method have two estimated parameters, and one of them is k2.

[1] There may be additional estimated parameters specified in the PEtab problem.

with open('output/models_2.yaml') as f:
    print(f.read())

- criteria: {}
  estimated_parameters: {}
  model_hash: 38c95dd428b3e31da6969a50db4a1ccbcefe6d8824617d27ec2360e57d55647a25f3fdd45e5f0270786698606cbe496cd94be9495986dade4d1f1d166a4bf911
  model_id: 38c95dd428b3e31da6969a50db4a1ccbcefe6d8824617d27ec2360e57d55647a25f3fdd45e5f0270786698606cbe496cd94be9495986dade4d1f1d166a4bf911
  model_subspace_id: M1_4
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: 0.2
    k2: estimate
    k3: estimate
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: c95cc8f9f2b945c996856212e7d49784f72cc429dbdbb9bc04adad610f1a8103a58298c714d0518f16a5a002db5a9c27847f5da6a863637178a15a15f2fc294d
- criteria: {}
  estimated_parameters: {}
  model_hash: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
  model_id: db8700c079c8347123adc89b7f5112256c4aaebd2af0f6e32e7582f398b2c1e5e85e588cdcc56bab054c001b96a9b42b02174266927f879d7f78e8ac5d2c33e6
  model_subspace_id: M1_6
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: estimate
    k2: estimate
    k3: 0
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: c95cc8f9f2b945c996856212e7d49784f72cc429dbdbb9bc04adad610f1a8103a58298c714d0518f16a5a002db5a9c27847f5da6a863637178a15a15f2fc294d

Third iteration

Models M1_4 and M1_6 were calibrated, with M1_4 having the superior criterion value, so is used as the predecessor model in the next forward search.

with open('model_selection/calibrated_M1_4.yaml') as f:
    print(f.read())

criteria:
  AIC: 30
model_id: M1_4_0
parameters:
  k1: 0.2
  k2: estimate
  k3: estimate
estimated_parameters:
  k2: 0.15
  k3: 0.0
petab_yaml: ../model_selection/petab_problem.yaml

%%bash

petab_select candidates \
--problem model_selection/petab_select_problem.yaml \
--state output/state.dill \
--output output/models_3.yaml \
--method forward \
--newly-calibrated-models model_selection/calibrated_M1_4.yaml \
--relative-paths

The model space contains only one model with 3 or more estimated parameters, which is a valid candidate model in this case.

with open('output/models_3.yaml') as f:
    print(f.read())

- criteria: {}
  estimated_parameters: {}
  model_hash: 7c105406ec11716473939a0bbb5281066c1014b54e2480ba126030f5c18a597a27a2ca9247aa60d8262f488165079d1c9e040f9d712ec4e19c2d2122a586f3e5
  model_id: 7c105406ec11716473939a0bbb5281066c1014b54e2480ba126030f5c18a597a27a2ca9247aa60d8262f488165079d1c9e040f9d712ec4e19c2d2122a586f3e5
  model_subspace_id: M1_7
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: estimate
    k2: estimate
    k3: estimate
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: 0e0f00ee37d03db0f6ad7bb9cb92eb4ae3aef85fa1e1d9d38dfc5954fbcf62e8768803bedd42fb3638b681d4840e352e5bfc29a1800247ebeaddfbdb4baca46b

Fourth iteration

As there are no models in the model space with additional estimated parameters, subsequent forward searches will return no candidate models.

with open('model_selection/calibrated_M1_7.yaml') as f:
    print(f.read())

criteria:
  AIC: 20
model_id: M1_7_0
parameters:
  k1: estimate
  k2: estimate
  k3: estimate
estimated_parameters:
  k1: 0.25
  k2: 0.1
  k3: 0.0
petab_yaml: ../model_selection/petab_problem.yaml

%%bash

petab_select candidates \
--problem model_selection/petab_select_problem.yaml \
--state output/state.dill \
--output output/models_4.yaml \
--method forward \
--newly-calibrated-models model_selection/calibrated_M1_7.yaml \
--relative-paths

with open('output/models_4.yaml') as f:
    print(f.read())

null
...

Fifth iteration

Although no additional models are found with a forward search initialized at the best model so far (M1_7), there are additional models in the model space that are yet to be calibrated, which can be identified with the brute force method.

%%bash

petab_select candidates \
--problem model_selection/petab_select_problem.yaml \
--state output/state.dill \
--output output/models.yaml \
--method brute_force \
--excluded-models output/models_1.yaml \
--excluded-models output/models_2.yaml \
--excluded-models output/models_3.yaml \
--excluded-models output/models_4.yaml \
--relative-paths

with open('output/models.yaml') as f:
    print(f.read())

- criteria: {}
  estimated_parameters: {}
  model_hash: b7584bfd6f35206dfe32fa0143e53cea808faf965e0c0547bf6ee1cdce7a75cd3ff0aa2bcb1faa27625166454f83e3fcac52cdf43b28e8186fff9a01ac3f8006
  model_id: b7584bfd6f35206dfe32fa0143e53cea808faf965e0c0547bf6ee1cdce7a75cd3ff0aa2bcb1faa27625166454f83e3fcac52cdf43b28e8186fff9a01ac3f8006
  model_subspace_id: M1_3
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: estimate
    k2: 0.1
    k3: 0
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: null
- criteria: {}
  estimated_parameters: {}
  model_hash: de4a2f17d8b0228a31d7451631cf3662d0ecf4dc7738ab6ca3d1de65e817844c9c1df806ec9daf81644b9c10f00185dc8c8de880d9db23a98acadb817f5d481c
  model_id: de4a2f17d8b0228a31d7451631cf3662d0ecf4dc7738ab6ca3d1de65e817844c9c1df806ec9daf81644b9c10f00185dc8c8de880d9db23a98acadb817f5d481c
  model_subspace_id: M1_5
  model_subspace_indices:
  - 0
  - 0
  - 0
  parameters:
    k1: estimate
    k2: 0.1
    k3: estimate
  petab_yaml: ../model_selection/petab_problem.yaml
  predecessor_model_hash: null

Post-processing

After the selection algorithm has terminated, the best model can be stored separately by supplying a list of calibrated models.

%%bash

petab_select best \
--problem model_selection/petab_select_problem.yaml \
--models model_selection/calibrated_first_iteration.yaml \
--models model_selection/calibrated_M1_4.yaml \
--models model_selection/calibrated_M1_7.yaml \
--output output/best_model.yaml \
--state output/state.dill \
--relative-paths

with open('output/best_model.yaml') as f:
    print(f.read())

criteria:
  AIC: 20
estimated_parameters:
  k1: 0.25
  k2: 0.1
  k3: 0.0
model_hash: null
model_id: M1_7_0
model_subspace_id: null
model_subspace_indices: null
parameters:
  k1: estimate
  k2: estimate
  k3: estimate
petab_yaml: ../model_selection/petab_problem.yaml
predecessor_model_hash: null

This model can be converted to a PEtab problem with either model_to_petab or models_to_petab.

%%bash

petab_select model_to_petab \
--model output/best_model.yaml \
--output output/best_model_petab

output/best_model_petab/problem.yaml