{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6c1bc86d",
   "metadata": {},
   "source": [
    "# Example usage with the CLI\n",
    "This notebook demonstrates usage of `petab_select` to perform model selection with commands.\n",
    "\n",
    "Note that the criterion values in this notebook are for demonstrative purposes only, and are not real. An additional point is that models store the iteration where they were calibrated, but the iteration counter is stored in the candidate space. Hence, when the candidate space (or method) changes in this notebook, the iteration counter is reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18dbcbbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup the state and candidate models output by a previous run of this notebook\n",
    "import shutil\n",
    "from pathlib import Path\n",
    "\n",
    "output_path = Path().resolve() / \"output_cli\"\n",
    "output_path_str = str(output_path)\n",
    "if output_path.exists():\n",
    "    shutil.rmtree(output_path_str)\n",
    "output_path.mkdir(exist_ok=False, parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e42ea66",
   "metadata": {},
   "source": [
    "## First iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "427cbd52",
   "metadata": {},
   "source": [
    "Iterations of model selection start and end with `start_iteration` and `end_iteration`.\n",
    "\n",
    "In each call to `petab_select start_iteration`, the following options are required:\n",
    "\n",
    "- `--problem`: The PEtab Select problem YAML file, which normally defines the method too\n",
    "- `--state`: A file that is used to store the state of the problem (e.g., such that models are not revisited)\n",
    "- `--output`: A file to store the models proposed by PEtab Select\n",
    "\n",
    "Other options can be viewed with `petab_select start_iteration --help`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3c4c95",
   "metadata": {},
   "source": [
    "In this initial call, a PEtab Select problem is used to identify possible models for selection. Instead of using the method defined in the PEtab Select problem, we use the brute force method, which normally outputs all possible models. Here, the number of models in the output is explicitly limited to `3`. Subsequent calls with the same command will output different models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eab391ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select start_iteration \\\n",
    "--problem model_selection/petab_select_problem.yaml \\\n",
    "--state $output_path_str/state.dill \\\n",
    "--method brute_force \\\n",
    "--limit 3 \\\n",
    "--output-uncalibrated-models $output_path_str/uncalibrated_models_1.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c34cca4",
   "metadata": {},
   "source": [
    "The output format is a list of the PEtab Select model YAML format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f6ac569",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"uncalibrated_models_1.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee59532-f6bd-4d9e-8a94-a9b0302efab1",
   "metadata": {},
   "source": [
    "At this point, the calibration tool should calibrate the models, and save the calibration results to disk in the PEtab Select model YAML format. For this example, we have stored the results in `model_selection/calibrated_models_1.yaml`.\n",
    "\n",
    "Next, we finalize the iteration by calling `petab_select end_iteration`, which requires:\n",
    "\n",
    "- `--state`: see `start_iteration`\n",
    "- `--output-models`: A file used to store the models from this iteration of calibration. Note that, if the user has supplied calibration results from previous model selection jobs, then this `end_iteration` output might differ from the output of the calibration tool. This `end_iteration` output should be considered the real set of models calibrated in this iteration.\n",
    "- `--output-metadata`: A file where metadata from the iteration is stored. This includes the signal of whether model selection has terminated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73665662-60ea-425c-843e-24a98c64c6a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select end_iteration \\\n",
    "--state=$output_path_str/state.dill \\\n",
    "--calibrated-models=model_selection/calibrated_models_1.yaml \\\n",
    "--output-models=$output_path_str/models_1.yaml \\\n",
    "--output-metadata=$output_path_str/metadata.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db725d4",
   "metadata": {},
   "source": [
    "## Second iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8008447",
   "metadata": {},
   "source": [
    "Between iterations, the models from the first iteration have been calibrated, and the model with the best criterion value is `M1_2`. In this iteration, we will switch to the forward method and manually specify `M1_2` as the predecessor model. In subsequent iterations, the predecessor model will default to the best model of the previous iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "703da45d",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"model_selection/calibrated_models_1.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "839a1b35",
   "metadata": {},
   "source": [
    "To perform the method change, we: delete the current state, select and store the model `M1_2` to disk via `petab_select get_best`, customize the problem to use the predecessor model via `candidate_space_arguments`, and supply the new method to `petab_select start_iteration`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22dfcc1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "# save the best model of the previous iteration (`M1_2`)\n",
    "petab_select get_best \\\n",
    "--problem model_selection/petab_select_problem.yaml \\\n",
    "--models model_selection/calibrated_models_1.yaml \\\n",
    "--output $output_path_str/predecessor_model.yaml\n",
    "# create a copy of the original PEtab select problem and update its paths\n",
    "cp model_selection/petab_select_problem.yaml $output_path_str/custom_problem.yaml\n",
    "sed -i 's|- model_space.tsv|- ../model_selection/model_space.tsv|' $output_path_str/custom_problem.yaml\n",
    "# add the predecessor model to the problem copy\n",
    "echo \"\"\"candidate_space_arguments:\n",
    "  predecessor_model: predecessor_model.yaml\n",
    "\"\"\" >> $output_path_str/custom_problem.yaml\n",
    "# remove the state from the previous iteration\n",
    "rm $output_path_str/state.dill\n",
    "\n",
    "# request models with the customized problem with the predecessor model `M1_2`, using the forward method\n",
    "petab_select start_iteration \\\n",
    "--problem $output_path_str/custom_problem.yaml \\\n",
    "--state $output_path_str/state.dill \\\n",
    "--output-uncalibrated-models $output_path_str/uncalibrated_models_2.yaml \\\n",
    "--method forward \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1f8a5f",
   "metadata": {},
   "source": [
    "`M1_2` has one estimated parameter, `k2`(*). As expected, the new candidates identified with the forward method have two estimated parameters, and one of them is `k2`.\n",
    "\n",
    "(*) There may be additional estimated parameters specified in the PEtab problem, which are not a part of the model selection problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd2f8850",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"uncalibrated_models_2.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d8a3623-80f9-4c4f-8bf6-0465d6914207",
   "metadata": {},
   "source": [
    "The calibration tool does not need to calibrate every uncalibrated model. For example, the calibration tool might return all calibration results, as soon as an improvement over the previous iteration is identified. Here, we only return the results for the `M1_4` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29cb0d84-4399-4e6b-895c-e92f9cc82d68",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select end_iteration \\\n",
    "--state=$output_path_str/state.dill \\\n",
    "--calibrated-models=model_selection/calibrated_M1_4.yaml \\\n",
    "--output-models=$output_path_str/models_2.yaml \\\n",
    "--output-metadata=$output_path_str/metadata.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "541a3e5d",
   "metadata": {},
   "source": [
    "## Third iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afba880c",
   "metadata": {},
   "source": [
    "The models from the previous iteration (i.e. `M1_4`) were stored in the state. Here, we perform an iteration of the forward method, which is automatically initialized with the `M1_4` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54c5b027",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"model_selection/calibrated_M1_4.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "818e59e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select start_iteration \\\n",
    "--problem $output_path_str/custom_problem.yaml \\\n",
    "--state $output_path_str/state.dill \\\n",
    "--output-uncalibrated-models $output_path_str/uncalibrated_models_3.yaml \\\n",
    "--method forward \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889dedc1",
   "metadata": {},
   "source": [
    "As we are performing a forward search from `M1_4`, which has two parameters, then all models in this iteration will have 3+ parameters. This model space contains only one model with 3 or more estimated parameters. We finalize the iteration with its calibration results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f393030",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"uncalibrated_models_3.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4084bd1-5bd7-4e12-8146-67137da4909a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select end_iteration \\\n",
    "--state=$output_path_str/state.dill \\\n",
    "--calibrated-models=model_selection/calibrated_M1_7.yaml \\\n",
    "--output-models=$output_path_str/models_3.yaml \\\n",
    "--output-metadata=$output_path_str/metadata.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de959e53",
   "metadata": {},
   "source": [
    "## Fourth iteration\n",
    "As there are no models in the model space with 4+ parameters, subsequent forward searches will return no candidate models. Tools can detect when to terminate by inspecting the metadata produced by `end_iteration`, as demonstrated at the end of this iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef2fe2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"model_selection/calibrated_M1_7.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35ed7ceb-6783-4956-9951-dbc55bfa9239",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select start_iteration \\\n",
    "--problem model_selection/petab_select_problem.yaml \\\n",
    "--state $output_path_str/state.dill \\\n",
    "--output-uncalibrated-models $output_path_str/uncalibrated_models_4.yaml \\\n",
    "--method forward \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fe1e848-e112-4ad2-ae09-57cdb7506ff8",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"uncalibrated_models_4.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02df7ed9-422d-4f28-9b01-8670be873933",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select end_iteration \\\n",
    "--state=$output_path_str/state.dill \\\n",
    "--output-models=$output_path_str/models_4.yaml \\\n",
    "--output-metadata=$output_path_str/metadata.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57e483fd-5ffa-48a4-8c2a-359f6ebd1422",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"output_cli/metadata.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0b1123",
   "metadata": {},
   "source": [
    "## Fifth iteration\n",
    "Although no additional models are found with a forward search initialized at the best model so far (`M1_7`), there are additional models in the model space that were not calibrated, which can be identified by using the brute force method with exclusions for the calibrated models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5b5087d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select start_iteration \\\n",
    "--problem model_selection/petab_select_problem.yaml \\\n",
    "--state $output_path_str/state_5.dill \\\n",
    "--output-uncalibrated-models $output_path_str/uncalibrated_models_5.yaml \\\n",
    "--method brute_force \\\n",
    "--excluded-models $output_path_str/models_1.yaml \\\n",
    "--excluded-models $output_path_str/models_2.yaml \\\n",
    "--excluded-models $output_path_str/models_3.yaml \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30721bfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"uncalibrated_models_5.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "655f8efc",
   "metadata": {},
   "source": [
    "## Post-processing\n",
    "After the selection algorithm has terminated, the best model can be stored separately by supplying a list of calibrated models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73d54111",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select get_best \\\n",
    "--problem model_selection/petab_select_problem.yaml \\\n",
    "--models $output_path_str/models_1.yaml \\\n",
    "--models $output_path_str/models_2.yaml \\\n",
    "--models $output_path_str/models_3.yaml \\\n",
    "--output $output_path_str/best_model.yaml \\\n",
    "--state $output_path_str/state.dill \\\n",
    "--relative-paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c36564f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path / \"best_model.yaml\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e610bb84",
   "metadata": {},
   "source": [
    "This model can be converted to a PEtab problem with either `model_to_petab` or `models_to_petab`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5d03cd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$output_path_str\"\n",
    "output_path_str=$1\n",
    "\n",
    "petab_select model_to_petab \\\n",
    "--model $output_path_str/best_model.yaml \\\n",
    "--output $output_path_str/best_model_petab"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}