{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "100c8a75",
   "metadata": {},
   "source": [
    "# Khéops 101"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "98d4907b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jez/prj/bell/dev/kheops\n"
     ]
    }
   ],
   "source": [
    "cd ${KHEOPS_EXAMPLES_DIR:-/dev/null}\n",
    "echo $PWD\n",
    "export KHEOPS_NAMESPACE=ex1_enc \n",
    "export KHEOPS_CONFIG=examples/kheops.yml\n",
    "rm -rf \"examples/$KHEOPS_NAMESPACE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327de4e6",
   "metadata": {},
   "source": [
    "## Command line"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28501b9d",
   "metadata": {},
   "source": [
    "Let's check first that kheops works correclty, and start to read the manual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6ede46a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: kheops [OPTIONS] COMMAND [ARGS]...\n",
      "\n",
      "  Khéops, hierarchical key/value store\n",
      "\n",
      "Options:\n",
      "  -v, --verbose                   [default: 0]\n",
      "  -c PATH                         Last name of person to greet.  [env var:\n",
      "                                  KHEOPS_CONFIG; default: kheops.yml]\n",
      "  --install-completion [bash|zsh|fish|powershell|pwsh]\n",
      "                                  Install completion for the specified shell.\n",
      "  --show-completion [bash|zsh|fish|powershell|pwsh]\n",
      "                                  Show completion for the specified shell, to\n",
      "                                  copy it or customize the installation.\n",
      "  --help                          Show this message and exit.\n",
      "\n",
      "Commands:\n",
      "  config\n",
      "  lookup  Lookup database\n"
     ]
    }
   ],
   "source": [
    "kheops --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d08340e",
   "metadata": {},
   "source": [
    "So we have a working `kheops` command, and we will focus on the `lookup` command. On it's simplest form, a lookup consists in querying a `key` for a given `scope`. The output of the `key` will change depending the `scope` value. A `key` is in simple word."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7727c7",
   "metadata": {},
   "source": [
    "## Defining a hierarchy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9615ef3",
   "metadata": {},
   "source": [
    "To illustrate how Khéops works, let's start with a simple example, we will try to lookup the `profile` key of the following two (fictive) servers:\n",
    "\n",
    "* web.domain.org: which act as a webserver role\n",
    "* mysql.domain.org: which act as mysql role\n",
    "\n",
    "But first we need to create our hierarchy. It's as simple as creating directories and put some json or yaml data into different files. Let's create our hierarchy. We will first create the default profile:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "746300bc",
   "metadata": {},
   "source": [
    "From our use case, we will build a lookup tree. We want to be able to organise data depending the 3 criterias:\n",
    "\n",
    "* node: name of the node\n",
    "* role: assigned role to the node\n",
    "* environment: it can either be dev or prod\n",
    "\n",
    "Let's create our lookup hierarchy:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9802b48",
   "metadata": {},
   "source": [
    "default:\n",
    "  lookups:\n",
    "    - path: default # Simplest form, just a path\n",
    "    - path: \"roles/{role}\" # If list, it's auto expanded like in bash\n",
    "    - path: \"env_{env}\" # If list, it's auto expanded like in bash\n",
    "    - path: \"nodes/{node}\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6afc7669",
   "metadata": {},
   "source": [
    "So for a given key, Khéops will iterate all over those paths to find the requested `key` , and then it will merge all results. Some paths are variabilized, and those variable comes from the scope. The scope come along the `key`, it's can be any metadata. For complex metadata you may want to store those in a file and load your scope with the `-f <yaml_scope_file>` option:\n",
    "\n",
    "```\n",
    "kheops lookup -e <var1=val1> -e <var2=val2> <key>\n",
    "```\n",
    "\n",
    "A scope is completely optional while keys are required."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d509fa19",
   "metadata": {},
   "source": [
    "## Basic hierarchy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd5b6b9",
   "metadata": {},
   "source": [
    "Let's create a firest hierachy, we will define a first basic hierarchy. In `kheops.yml`, we can find:\n",
    "\n",
    "```\n",
    "ex1_enc:\n",
    "  \n",
    "  config:\n",
    "    file_path_prefix: \"ex1_enc/\"\n",
    "    file_path_suffix: \"/ansible\"\n",
    "\n",
    "  lookups:\n",
    "\n",
    "    - path: default           # Simplest form, just a path\n",
    "    - path: \"roles/{role}\"    # If list, it's auto expanded like in bash\n",
    "    - path: \"env_{env}\"       # If list, it's auto expanded like in bash\n",
    "    - path: \"nodes/{node}\"\n",
    "\n",
    "```\n",
    "\n",
    "Now we have our hierachy, let's create our files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e510a46d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mexamples/ex1_enc\u001b[0m\n",
      "└── default.yml\n",
      "\n",
      "0 directories, 1 file\n",
      "---\n",
      "profile:\n",
      "  env: \"NO_ENV\"\n",
      "  product: \"NO_PRODUCT\"\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# We create a fresh hierachy\n",
    "mkdir -p examples/$KHEOPS_NAMESPACE\n",
    "\n",
    "# We create a profile key, which is a dict\n",
    "cat > examples/$KHEOPS_NAMESPACE/default.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  env: \"NO_ENV\"\n",
    "  product: \"NO_PRODUCT\"\n",
    "  \n",
    "EOF\n",
    "\n",
    "# Let's inspect our hierarchy\n",
    "tree examples/$KHEOPS_NAMESPACE\n",
    "cat examples/$KHEOPS_NAMESPACE/default.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "931d682a",
   "metadata": {},
   "source": [
    "From this point, we defined our profile with two attribute, `team` and `product`. As it's the default case, we set them both unconfigured. \n",
    "\n",
    "You are now already able to query your hierarchy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "06d85b1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: NO_PRODUCT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a627822",
   "metadata": {},
   "source": [
    "Good, no surprise. But, we mentionned we wanted to get the profile of two instances, this how would do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c5fad63b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: NO_PRODUCT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e node=web.infra.net profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0649351",
   "metadata": {},
   "source": [
    "Same result, let's check how we can change this behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c43757",
   "metadata": {},
   "source": [
    "## Roles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5418032e",
   "metadata": {},
   "source": [
    "However, same result as before, which is expected as we did not finished to configure our hierarchy. Among our instances, we identified 2 roles: web and mysql. Let's create those two roles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9f5aabaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mexamples/ex1_enc\u001b[0m\n",
      "├── default.yml\n",
      "└── \u001b[01;34mroles\u001b[0m\n",
      "    ├── mysql.yml\n",
      "    └── web.yml\n",
      "\n",
      "1 directory, 3 files\n"
     ]
    }
   ],
   "source": [
    "mkdir -p examples/$KHEOPS_NAMESPACE/roles\n",
    "\n",
    "# We create a new web role\n",
    "cat > examples/$KHEOPS_NAMESPACE/roles/web.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  product: \"httpd_server\"\n",
    "\n",
    "  web_top_domain: \"\"\n",
    "  web_app: \"NO_APP\"\n",
    "  web_port: 80\n",
    "  web_user_list:\n",
    "    - sysadmins\n",
    "  \n",
    "EOF\n",
    "\n",
    "# We create a new mysql role\n",
    "cat > examples/$KHEOPS_NAMESPACE/roles/mysql.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  product: \"mysql_server\"\n",
    "\n",
    "  mysql_database: \"NO_DATABASE\"\n",
    "  mysql_users:\n",
    "    - \"sysadmin@10.0.42%\"\n",
    "  mysql_port: 3306\n",
    "  mysql_cluster: False\n",
    "  \n",
    "EOF\n",
    "\n",
    "# Let's inspect our hierarchy\n",
    "tree examples/$KHEOPS_NAMESPACE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4762fa0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> examples/ex1_enc/default.yml <==\n",
      "---\n",
      "profile:\n",
      "  env: \"NO_ENV\"\n",
      "  product: \"NO_PRODUCT\"\n",
      "  \n",
      "\n",
      "==> examples/ex1_enc/roles/mysql.yml <==\n",
      "---\n",
      "profile:\n",
      "  product: \"mysql_server\"\n",
      "\n",
      "  mysql_database: \"NO_DATABASE\"\n",
      "  mysql_users:\n",
      "    - \"sysadmin@10.0.42%\"\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: False\n",
      "  \n",
      "\n",
      "==> examples/ex1_enc/roles/web.yml <==\n",
      "---\n",
      "profile:\n",
      "  product: \"httpd_server\"\n",
      "\n",
      "  web_top_domain: \"\"\n",
      "  web_app: \"NO_APP\"\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "    - sysadmins\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "tail -n 999 examples/$KHEOPS_NAMESPACE/{*.yml,*/*.yml}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0294ec50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: httpd_server\n",
      "  web_top_domain: ''\n",
      "  web_app: NO_APP\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - sysadmins\n",
      "\n",
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: mysql_server\n",
      "  mysql_database: NO_DATABASE\n",
      "  mysql_users:\n",
      "  - sysadmin@10.0.42%\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: false\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e node=web.infra.net -e role=web profile\n",
    "kheops lookup -e node=mysql.infra.net -e role=mysql profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed199d61",
   "metadata": {},
   "source": [
    "## Per node override"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "496c936a",
   "metadata": {},
   "source": [
    "It's getting better, we can see that the profile key has been merged with the key values, across the different locations. \n",
    "\n",
    "However, we will have those placeholders, and we want to have personalized value, depending if it's aweb server, it need an unique domain and some unique parameters. So let's create a `nodes` directory and place some data inside.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c1acb199",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mexamples/ex1_enc\u001b[0m\n",
      "├── default.yml\n",
      "├── \u001b[01;34mnodes\u001b[0m\n",
      "│   ├── mysql.infra.net.yml\n",
      "│   └── web.infra.net.yml\n",
      "└── \u001b[01;34mroles\u001b[0m\n",
      "    ├── mysql.yml\n",
      "    └── web.yml\n",
      "\n",
      "2 directories, 5 files\n"
     ]
    }
   ],
   "source": [
    "mkdir -p examples/$KHEOPS_NAMESPACE/nodes\n",
    "\n",
    "# We create a new web role\n",
    "cat > examples/$KHEOPS_NAMESPACE/nodes/web.infra.net.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  web_app: 'myapp'\n",
    "  web_user_list:\n",
    "    - domain_org\n",
    "    - domain_org_external\n",
    "  \n",
    "EOF\n",
    "\n",
    "# We create a new mysql role\n",
    "cat > examples/$KHEOPS_NAMESPACE/nodes/mysql.infra.net.yml <<EOF\n",
    "---\n",
    "profile:  \n",
    "  mysql_database: \"app_domain_org\"\n",
    "  mysql_users:\n",
    "    - \"app_domain_org@10.0.51%\"\n",
    "  \n",
    "EOF\n",
    "\n",
    "# Let's inspect our hierarchy\n",
    "tree examples/$KHEOPS_NAMESPACE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0492ba27",
   "metadata": {},
   "source": [
    "And we try again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b0d34416",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: httpd_server\n",
      "  web_top_domain: ''\n",
      "  web_app: myapp\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - domain_org\n",
      "  - domain_org_external\n",
      "\n",
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: mysql_server\n",
      "  mysql_database: app_domain_org\n",
      "  mysql_users:\n",
      "  - app_domain_org@10.0.51%\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: false\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e node=web.infra.net -e role=web profile\n",
    "kheops lookup -e node=mysql.infra.net -e role=mysql profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0457c5f9",
   "metadata": {},
   "source": [
    "## Environment override"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4dc7e54",
   "metadata": {},
   "source": [
    "Let's say you want to support environment, it's the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6c92b0cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mexamples/ex1_enc\u001b[0m\n",
      "├── default.yml\n",
      "├── env_dev.yml\n",
      "├── env_prod.yml\n",
      "├── \u001b[01;34mnodes\u001b[0m\n",
      "│   ├── mysql.infra.net.yml\n",
      "│   └── web.infra.net.yml\n",
      "└── \u001b[01;34mroles\u001b[0m\n",
      "    ├── mysql.yml\n",
      "    └── web.yml\n",
      "\n",
      "2 directories, 7 files\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# We create a new dev environment\n",
    "cat > examples/$KHEOPS_NAMESPACE/env_dev.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  env: dev\n",
    "  \n",
    "  # We change the top domain for dev environment, and reduce the cache\n",
    "  web_top_domain: dev.infra.net\n",
    "  web_cache: 1m\n",
    "  \n",
    "  # We want a debug users\n",
    "  web_user_list:\n",
    "    - debug_user\n",
    "  mysql_users:\n",
    "    - debug@10.0.%\n",
    "\n",
    "  debug: true\n",
    "\n",
    "EOF\n",
    "\n",
    "# We create a new mysql role\n",
    "cat > examples/$KHEOPS_NAMESPACE/env_prod.yml <<EOF\n",
    "---\n",
    "profile:\n",
    "  env: prod\n",
    "  \n",
    "  # On production environment, we always want to use public faced domain and 12 hour cache.\n",
    "  web_top_domain: infra.com\n",
    "  web_cache: 12h\n",
    "  \n",
    "EOF\n",
    "\n",
    "# Let's inspect our hierarchy\n",
    "tree examples/$KHEOPS_NAMESPACE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5413f84",
   "metadata": {},
   "source": [
    "So it's become quite easy to compare the difference between environment, with a simple variable switch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9c3200b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: prod\n",
      "  product: httpd_server\n",
      "  web_top_domain: infra.com\n",
      "  web_app: myapp\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - domain_org\n",
      "  - domain_org_external\n",
      "  web_cache: 12h\n",
      "\n",
      "profile:\n",
      "  env: dev\n",
      "  product: httpd_server\n",
      "  web_top_domain: dev.infra.net\n",
      "  web_app: myapp\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - domain_org\n",
      "  - domain_org_external\n",
      "  web_cache: 1m\n",
      "  mysql_users:\n",
      "  - debug@10.0.%\n",
      "  debug: true\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e node=web.infra.net -e role=web -e env=prod profile\n",
    "kheops lookup -e node=web.infra.net -e role=web -e env=dev profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb8302b",
   "metadata": {},
   "source": [
    "Same for mysql:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7c5e86f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: prod\n",
      "  product: mysql_server\n",
      "  mysql_database: app_domain_org\n",
      "  mysql_users:\n",
      "  - app_domain_org@10.0.51%\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: false\n",
      "  web_top_domain: infra.com\n",
      "  web_cache: 12h\n",
      "\n",
      "profile:\n",
      "  env: dev\n",
      "  product: mysql_server\n",
      "  mysql_database: app_domain_org\n",
      "  mysql_users:\n",
      "  - app_domain_org@10.0.51%\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: false\n",
      "  web_top_domain: dev.infra.net\n",
      "  web_cache: 1m\n",
      "  web_user_list:\n",
      "  - debug_user\n",
      "  debug: true\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e node=mysql.infra.net -e role=mysql -e env=prod profile\n",
    "kheops lookup -e node=mysql.infra.net -e role=mysql -e env=dev profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4bf6d8d",
   "metadata": {},
   "source": [
    "You have to keep in mind you can query the key with a different scope, and get different views:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "103cb37d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: NO_PRODUCT\n",
      "\n",
      "==> Per environment view\n",
      "profile:\n",
      "  env: prod\n",
      "  product: NO_PRODUCT\n",
      "  web_top_domain: infra.com\n",
      "  web_cache: 12h\n",
      "\n",
      "profile:\n",
      "  env: dev\n",
      "  product: NO_PRODUCT\n",
      "  web_top_domain: dev.infra.net\n",
      "  web_cache: 1m\n",
      "  web_user_list:\n",
      "  - debug_user\n",
      "  mysql_users:\n",
      "  - debug@10.0.%\n",
      "  debug: true\n",
      "\n",
      "==> Per role and  environment view\n",
      "profile:\n",
      "  env: prod\n",
      "  product: mysql_server\n",
      "  mysql_database: NO_DATABASE\n",
      "  mysql_users:\n",
      "  - sysadmin@10.0.42%\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: false\n",
      "  web_top_domain: infra.com\n",
      "  web_cache: 12h\n",
      "\n",
      "profile:\n",
      "  env: prod\n",
      "  product: httpd_server\n",
      "  web_top_domain: infra.com\n",
      "  web_app: NO_APP\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - sysadmins\n",
      "  web_cache: 12h\n",
      "\n",
      "==> Per node view\n",
      "profile:\n",
      "  env: dev\n",
      "  product: httpd_server\n",
      "  web_top_domain: dev.infra.net\n",
      "  web_app: myapp\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - domain_org\n",
      "  - domain_org_external\n",
      "  web_cache: 1m\n",
      "  mysql_users:\n",
      "  - debug@10.0.%\n",
      "  debug: true\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup profile\n",
    "\n",
    "echo \"==> Per environment view\"\n",
    "kheops lookup -e env=prod profile\n",
    "kheops lookup -e env=dev profile\n",
    "\n",
    "echo \"==> Per role and  environment view\"\n",
    "kheops lookup -e role=mysql -e env=prod profile\n",
    "kheops lookup -e role=web -e env=prod profile\n",
    "\n",
    "echo \"==> Per node view\"\n",
    "kheops lookup -e node=web.infra.net -e role=web -e env=dev profile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de496c59",
   "metadata": {},
   "source": [
    "Even if somwaht clunky, this method can help to troubleshoot wrong data by dichotomy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d591f865",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f3331037",
   "metadata": {},
   "source": [
    "## Tooling and applications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fbf9e36",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a2ee419",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5905ab3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38f1a2ae",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ec8ebb5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7d501ef9",
   "metadata": {},
   "source": [
    "## Troubleshooting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce4b7029",
   "metadata": {},
   "source": [
    "Sometimes, it can may be hard to navigate across file and hierachy, but GNU Utils are here to help. There is a selection of small tips:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "52f5033b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ : Find where a key has been defined\n",
      "+ : ==========================\n",
      "+ grep --colour=auto -r '^profile:' examples/ex1_enc\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_prod.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/web.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K  \n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/web.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/default.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K\u001b[01;31m\u001b[Kprofile:\u001b[m\u001b[K\n",
      "+ : Find where a key has been defined and 5 first lines\n",
      "+ : ==========================\n",
      "+ grep --colour=auto -r -A 5 web_user_list: examples/ex1_enc\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/web.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K  \u001b[01;31m\u001b[Kweb_user_list:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/web.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - sysadmins\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/web.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  \n",
      "\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/web.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K  \u001b[01;31m\u001b[Kweb_user_list:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/web.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - domain_org\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/web.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - domain_org_external\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/web.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  \n",
      "\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K  \u001b[01;31m\u001b[Kweb_user_list:\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - debug_user\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  mysql_users:\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - debug@10.0.%\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/env_dev.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  debug: true\n",
      "+ : Search from anything related to database\n",
      "+ : ==========================\n",
      "+ grep --colour=auto -R -C 3 database examples/ex1_enc\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[Kprofile:\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  product: \"mysql_server\"\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K  mysql_\u001b[01;31m\u001b[Kdatabase\u001b[m\u001b[K: \"NO_DATABASE\"\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  mysql_users:\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - \"sysadmin@10.0.42%\"\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/roles/mysql.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  mysql_port: 3306\n",
      "\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K---\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[Kprofile:  \n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[K  mysql_\u001b[01;31m\u001b[Kdatabase\u001b[m\u001b[K: \"app_domain_org\"\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  mysql_users:\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K    - \"app_domain_org@10.0.51%\"\n",
      "\u001b[35m\u001b[Kexamples/ex1_enc/nodes/mysql.infra.net.yml\u001b[m\u001b[K\u001b[36m\u001b[K-\u001b[m\u001b[K  \n",
      "+ set +x\n"
     ]
    }
   ],
   "source": [
    "set -x\n",
    "\n",
    ": Find where a key has been defined\n",
    ": ==========================\n",
    "grep -r '^profile:' examples/$KHEOPS_NAMESPACE\n",
    "\n",
    "\n",
    ": Find where a key has been defined and 5 first lines\n",
    ": ==========================\n",
    "grep -r -A 5 'web_user_list:' examples/$KHEOPS_NAMESPACE\n",
    "\n",
    "\n",
    ": Search from anything related to database\n",
    ": ==========================\n",
    "grep -R -C 3  'database' examples/$KHEOPS_NAMESPACE\n",
    "\n",
    "set +x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ce952f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fc4a20ef",
   "metadata": {},
   "source": [
    "The tail/head command is quite usefull to look at multiple files at the same time, it add a nice header for each file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "49bc3fc3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> examples/ex1_enc/roles/mysql.yml <==\n",
      "---\n",
      "profile:\n",
      "  product: \"mysql_server\"\n",
      "\n",
      "  mysql_database: \"NO_DATABASE\"\n",
      "  mysql_users:\n",
      "    - \"sysadmin@10.0.42%\"\n",
      "  mysql_port: 3306\n",
      "  mysql_cluster: False\n",
      "  \n",
      "\n",
      "==> examples/ex1_enc/roles/web.yml <==\n",
      "---\n",
      "profile:\n",
      "  product: \"httpd_server\"\n",
      "\n",
      "  web_top_domain: \"\"\n",
      "  web_app: \"NO_APP\"\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "    - sysadmins\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "head -n 999 examples/ex1_enc/roles/*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c6dc222",
   "metadata": {},
   "source": [
    "You can also have a view of all files with this command:\n",
    "```\n",
    "find . -type f| xargs head -n 999 | less\n",
    "```\n",
    "\n",
    "From there, you will be able to have a nice overview of your data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69421fd2",
   "metadata": {},
   "source": [
    "You can even diff your change with this command. There is this simple trick to compare the data difference between 2 lookups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e4a1e8d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- /dev/fd/63\t2022-02-14 13:45:59.223619144 -0500\n",
      "+++ /dev/fd/62\t2022-02-14 13:45:59.223619144 -0500\n",
      "@@ -1,11 +1,14 @@\n",
      " profile:\n",
      "-  env: prod\n",
      "+  env: dev\n",
      "   product: httpd_server\n",
      "-  web_top_domain: infra.com\n",
      "+  web_top_domain: dev.infra.net\n",
      "   web_app: myapp\n",
      "   web_port: 80\n",
      "   web_user_list:\n",
      "   - domain_org\n",
      "   - domain_org_external\n",
      "-  web_cache: 12h\n",
      "+  web_cache: 1m\n",
      "+  mysql_users:\n",
      "+  - debug@10.0.%\n",
      "+  debug: true\n",
      " \n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "diff -u \\\n",
    "<(kheops lookup -e node=web.infra.net -e role=web -e env=prod profile) \\\n",
    "<(kheops lookup -e node=web.infra.net -e role=web -e env=dev profile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b1730b",
   "metadata": {},
   "source": [
    "You can also ask Kheops to explain you how he built the result, you can use the `-x` flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3ac0cc53",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    INFO: Explain lookups:\n",
      "+------------------------+------------------------------+\n",
      "| Config                 | Runtime                      |\n",
      "+------------------------+------------------------------+\n",
      "|                        |                              |\n",
      "| Config:{               | Runtime:{                    |\n",
      "|   \"path\": \"default\",   |   \"scope\": {                 |\n",
      "|   \"backend\": \"file\",   |     \"role\": \"web\"            |\n",
      "|   \"continue\": true     |   },                         |\n",
      "| }                      |   \"key\": \"profile\",          |\n",
      "|                        |   \"conf\": {                  |\n",
      "|                        |     \"index\": 0               |\n",
      "|                        |   },                         |\n",
      "|                        |   \"raw_path\": \"default\"      |\n",
      "|                        | }                            |\n",
      "|                        |                              |\n",
      "| Config:{               | Runtime:{                    |\n",
      "|   \"path\": \"roles/web\", |   \"scope\": {                 |\n",
      "|   \"backend\": \"file\",   |     \"role\": \"web\"            |\n",
      "|   \"continue\": true     |   },                         |\n",
      "| }                      |   \"key\": \"profile\",          |\n",
      "|                        |   \"conf\": {                  |\n",
      "|                        |     \"index\": 1               |\n",
      "|                        |   },                         |\n",
      "|                        |   \"raw_path\": \"roles/{role}\" |\n",
      "|                        | }                            |\n",
      "+------------------------+------------------------------+\n",
      "    INFO: Explain candidates:\n",
      "+----------------------------------------------------------------------------------+-------------------------------+------------------------------+\n",
      "| Status                                                                           | Runtime                       | Key Value                    |\n",
      "+----------------------------------------------------------------------------------+-------------------------------+------------------------------+\n",
      "|                                                                                  |                               |                              |\n",
      "| Status:{                                                                         | Runtime:{                     | Key:{                        |\n",
      "|   \"path\": \"/home/jez/volumes/data/prj/bell/dev/kheops/examples/ex1_enc/defau ... |   \"scope\": {                  |   \"env\": \"NO_ENV\",           |\n",
      "|   \"status\": \"found\",                                                             |     \"role\": \"web\"             |   \"product\": \"NO_PRODUCT\"    |\n",
      "|   \"rel_path\": \"examples/ex1_enc/default.yml\"                                     |   },                          | }                            |\n",
      "| }                                                                                |   \"key\": \"profile\",           |                              |\n",
      "|                                                                                  |   \"conf\": {                   |                              |\n",
      "|                                                                                  |     \"index\": 0                |                              |\n",
      "|                                                                                  |   },                          |                              |\n",
      "|                                                                                  |   \"raw_path\": \"default\",      |                              |\n",
      "|                                                                                  |   \"backend_index\": 0          |                              |\n",
      "|                                                                                  | }                             |                              |\n",
      "|                                                                                  |                               |                              |\n",
      "| Status:{                                                                         | Runtime:{                     | Key:{                        |\n",
      "|   \"path\": \"/home/jez/volumes/data/prj/bell/dev/kheops/examples/ex1_enc/roles ... |   \"scope\": {                  |   \"product\": \"httpd_server\", |\n",
      "|   \"status\": \"found\",                                                             |     \"role\": \"web\"             |   \"web_top_domain\": \"\",      |\n",
      "|   \"rel_path\": \"examples/ex1_enc/roles/web.yml\"                                   |   },                          |   \"web_app\": \"NO_APP\",       |\n",
      "| }                                                                                |   \"key\": \"profile\",           |   \"web_port\": 80,            |\n",
      "|                                                                                  |   \"conf\": {                   |   \"web_user_list\": [         |\n",
      "|                                                                                  |     \"index\": 1                |     \"sysadmins\"              |\n",
      "|                                                                                  |   },                          |   ]                          |\n",
      "|                                                                                  |   \"raw_path\": \"roles/{role}\", | }                            |\n",
      "|                                                                                  |   \"backend_index\": 1          |                              |\n",
      "|                                                                                  | }                             |                              |\n",
      "+----------------------------------------------------------------------------------+-------------------------------+------------------------------+\n",
      "profile:\n",
      "  env: NO_ENV\n",
      "  product: httpd_server\n",
      "  web_top_domain: ''\n",
      "  web_app: NO_APP\n",
      "  web_port: 80\n",
      "  web_user_list:\n",
      "  - sysadmins\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kheops lookup -e role=web profile -X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07eeed03",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}