kheops/docs/learn/learn101.md
2022-06-19 23:20:15 -04:00

820 lines
24 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Khéops 101
```bash
cd ${KHEOPS_EXAMPLES_DIR:-/dev/null}
echo $PWD
export KHEOPS_NAMESPACE=ex1_enc
export KHEOPS_CONFIG=examples/kheops.yml
rm -rf "examples/$KHEOPS_NAMESPACE"
```
/home/jez/prj/bell/dev/kheops
## Command line
Let's check first that kheops works correclty, and start to read the manual.
```bash
kheops --help
```
Usage: kheops [OPTIONS] COMMAND [ARGS]...
Khéops, hierarchical key/value store
Options:
-v, --verbose [default: 0]
-c PATH Last name of person to greet. [env var:
KHEOPS_CONFIG; default: kheops.yml]
--install-completion [bash|zsh|fish|powershell|pwsh]
Install completion for the specified shell.
--show-completion [bash|zsh|fish|powershell|pwsh]
Show completion for the specified shell, to
copy it or customize the installation.
--help Show this message and exit.
Commands:
config
lookup Lookup database
So we have a working `kheops` command, and we will focus on the `lookup` command. On it's simplest form, a lookup consists in querying a `key` for a given `scope`. The output of the `key` will change depending the `scope` value. A `key` is in simple word.
## Defining a hierarchy
To illustrate how Khéops works, let's start with a simple example, we will try to lookup the `profile` key of the following two (fictive) servers:
* web.domain.org: which act as a webserver role
* mysql.domain.org: which act as mysql role
But first we need to create our hierarchy. It's as simple as creating directories and put some json or yaml data into different files. Let's create our hierarchy. We will first create the default profile:
From our use case, we will build a lookup tree. We want to be able to organise data depending the 3 criterias:
* node: name of the node
* role: assigned role to the node
* environment: it can either be dev or prod
Let's create our lookup hierarchy:
default:
lookups:
- path: default # Simplest form, just a path
- path: "roles/{role}" # If list, it's auto expanded like in bash
- path: "env_{env}" # If list, it's auto expanded like in bash
- path: "nodes/{node}"
So for a given key, Khéops will iterate all over those paths to find the requested `key` , and then it will merge all results. Some paths are variabilized, and those variable comes from the scope. The scope come along the `key`, it's can be any metadata. For complex metadata you may want to store those in a file and load your scope with the `-f <yaml_scope_file>` option:
```
kheops lookup -e <var1=val1> -e <var2=val2> <key>
```
A scope is completely optional while keys are required.
## Basic hierarchy
Let's create a firest hierachy, we will define a first basic hierarchy. In `kheops.yml`, we can find:
```
ex1_enc:
config:
file_path_prefix: "ex1_enc/"
file_path_suffix: "/ansible"
lookups:
- path: default # Simplest form, just a path
- path: "roles/{role}" # If list, it's auto expanded like in bash
- path: "env_{env}" # If list, it's auto expanded like in bash
- path: "nodes/{node}"
```
Now we have our hierachy, let's create our files:
```bash
# We create a fresh hierachy
mkdir -p examples/$KHEOPS_NAMESPACE
# We create a profile key, which is a dict
cat > examples/$KHEOPS_NAMESPACE/default.yml <<EOF
---
profile:
env: "NO_ENV"
product: "NO_PRODUCT"
EOF
# Let's inspect our hierarchy
tree examples/$KHEOPS_NAMESPACE
cat examples/$KHEOPS_NAMESPACE/default.yml
```
examples/ex1_enc
└── default.yml
0 directories, 1 file
---
profile:
env: "NO_ENV"
product: "NO_PRODUCT"
From this point, we defined our profile with two attribute, `team` and `product`. As it's the default case, we set them both unconfigured.
You are now already able to query your hierarchy:
```bash
kheops lookup profile
```
profile:
env: NO_ENV
product: NO_PRODUCT
Good, no surprise. But, we mentionned we wanted to get the profile of two instances, this how would do that:
```bash
kheops lookup -e node=web.infra.net profile
```
profile:
env: NO_ENV
product: NO_PRODUCT
Same result, let's check how we can change this behavior.
## Roles
However, same result as before, which is expected as we did not finished to configure our hierarchy. Among our instances, we identified 2 roles: web and mysql. Let's create those two roles:
```bash
mkdir -p examples/$KHEOPS_NAMESPACE/roles
# We create a new web role
cat > examples/$KHEOPS_NAMESPACE/roles/web.yml <<EOF
---
profile:
product: "httpd_server"
web_top_domain: ""
web_app: "NO_APP"
web_port: 80
web_user_list:
- sysadmins
EOF
# We create a new mysql role
cat > examples/$KHEOPS_NAMESPACE/roles/mysql.yml <<EOF
---
profile:
product: "mysql_server"
mysql_database: "NO_DATABASE"
mysql_users:
- "sysadmin@10.0.42%"
mysql_port: 3306
mysql_cluster: False
EOF
# Let's inspect our hierarchy
tree examples/$KHEOPS_NAMESPACE
```
examples/ex1_enc
├── default.yml
└── roles
├── mysql.yml
└── web.yml
1 directory, 3 files
```bash
tail -n 999 examples/$KHEOPS_NAMESPACE/{*.yml,*/*.yml}
```
==> examples/ex1_enc/default.yml <==
---
profile:
env: "NO_ENV"
product: "NO_PRODUCT"
==> examples/ex1_enc/roles/mysql.yml <==
---
profile:
product: "mysql_server"
mysql_database: "NO_DATABASE"
mysql_users:
- "sysadmin@10.0.42%"
mysql_port: 3306
mysql_cluster: False
==> examples/ex1_enc/roles/web.yml <==
---
profile:
product: "httpd_server"
web_top_domain: ""
web_app: "NO_APP"
web_port: 80
web_user_list:
- sysadmins
```bash
kheops lookup -e node=web.infra.net -e role=web profile
kheops lookup -e node=mysql.infra.net -e role=mysql profile
```
profile:
env: NO_ENV
product: httpd_server
web_top_domain: ''
web_app: NO_APP
web_port: 80
web_user_list:
- sysadmins
profile:
env: NO_ENV
product: mysql_server
mysql_database: NO_DATABASE
mysql_users:
- sysadmin@10.0.42%
mysql_port: 3306
mysql_cluster: false
## Per node override
It's getting better, we can see that the profile key has been merged with the key values, across the different locations.
However, we will have those placeholders, and we want to have personalized value, depending if it's aweb server, it need an unique domain and some unique parameters. So let's create a `nodes` directory and place some data inside.
```bash
mkdir -p examples/$KHEOPS_NAMESPACE/nodes
# We create a new web role
cat > examples/$KHEOPS_NAMESPACE/nodes/web.infra.net.yml <<EOF
---
profile:
web_app: 'myapp'
web_user_list:
- domain_org
- domain_org_external
EOF
# We create a new mysql role
cat > examples/$KHEOPS_NAMESPACE/nodes/mysql.infra.net.yml <<EOF
---
profile:
mysql_database: "app_domain_org"
mysql_users:
- "app_domain_org@10.0.51%"
EOF
# Let's inspect our hierarchy
tree examples/$KHEOPS_NAMESPACE
```
examples/ex1_enc
├── default.yml
├── nodes
│   ├── mysql.infra.net.yml
│   └── web.infra.net.yml
└── roles
├── mysql.yml
└── web.yml
2 directories, 5 files
And we try again:
```bash
kheops lookup -e node=web.infra.net -e role=web profile
kheops lookup -e node=mysql.infra.net -e role=mysql profile
```
profile:
env: NO_ENV
product: httpd_server
web_top_domain: ''
web_app: myapp
web_port: 80
web_user_list:
- domain_org
- domain_org_external
profile:
env: NO_ENV
product: mysql_server
mysql_database: app_domain_org
mysql_users:
- app_domain_org@10.0.51%
mysql_port: 3306
mysql_cluster: false
## Environment override
Let's say you want to support environment, it's the same:
```bash
# We create a new dev environment
cat > examples/$KHEOPS_NAMESPACE/env_dev.yml <<EOF
---
profile:
env: dev
# We change the top domain for dev environment, and reduce the cache
web_top_domain: dev.infra.net
web_cache: 1m
# We want a debug users
web_user_list:
- debug_user
mysql_users:
- debug@10.0.%
debug: true
EOF
# We create a new mysql role
cat > examples/$KHEOPS_NAMESPACE/env_prod.yml <<EOF
---
profile:
env: prod
# On production environment, we always want to use public faced domain and 12 hour cache.
web_top_domain: infra.com
web_cache: 12h
EOF
# Let's inspect our hierarchy
tree examples/$KHEOPS_NAMESPACE
```
examples/ex1_enc
├── default.yml
├── env_dev.yml
├── env_prod.yml
├── nodes
│   ├── mysql.infra.net.yml
│   └── web.infra.net.yml
└── roles
├── mysql.yml
└── web.yml
2 directories, 7 files
So it's become quite easy to compare the difference between environment, with a simple variable switch:
```bash
kheops lookup -e node=web.infra.net -e role=web -e env=prod profile
kheops lookup -e node=web.infra.net -e role=web -e env=dev profile
```
profile:
env: prod
product: httpd_server
web_top_domain: infra.com
web_app: myapp
web_port: 80
web_user_list:
- domain_org
- domain_org_external
web_cache: 12h
profile:
env: dev
product: httpd_server
web_top_domain: dev.infra.net
web_app: myapp
web_port: 80
web_user_list:
- domain_org
- domain_org_external
web_cache: 1m
mysql_users:
- debug@10.0.%
debug: true
Same for mysql:
```bash
kheops lookup -e node=mysql.infra.net -e role=mysql -e env=prod profile
kheops lookup -e node=mysql.infra.net -e role=mysql -e env=dev profile
```
profile:
env: prod
product: mysql_server
mysql_database: app_domain_org
mysql_users:
- app_domain_org@10.0.51%
mysql_port: 3306
mysql_cluster: false
web_top_domain: infra.com
web_cache: 12h
profile:
env: dev
product: mysql_server
mysql_database: app_domain_org
mysql_users:
- app_domain_org@10.0.51%
mysql_port: 3306
mysql_cluster: false
web_top_domain: dev.infra.net
web_cache: 1m
web_user_list:
- debug_user
debug: true
You have to keep in mind you can query the key with a different scope, and get different views:
```bash
kheops lookup profile
echo "==> Per environment view"
kheops lookup -e env=prod profile
kheops lookup -e env=dev profile
echo "==> Per role and environment view"
kheops lookup -e role=mysql -e env=prod profile
kheops lookup -e role=web -e env=prod profile
echo "==> Per node view"
kheops lookup -e node=web.infra.net -e role=web -e env=dev profile
```
profile:
env: NO_ENV
product: NO_PRODUCT
==> Per environment view
profile:
env: prod
product: NO_PRODUCT
web_top_domain: infra.com
web_cache: 12h
profile:
env: dev
product: NO_PRODUCT
web_top_domain: dev.infra.net
web_cache: 1m
web_user_list:
- debug_user
mysql_users:
- debug@10.0.%
debug: true
==> Per role and environment view
profile:
env: prod
product: mysql_server
mysql_database: NO_DATABASE
mysql_users:
- sysadmin@10.0.42%
mysql_port: 3306
mysql_cluster: false
web_top_domain: infra.com
web_cache: 12h
profile:
env: prod
product: httpd_server
web_top_domain: infra.com
web_app: NO_APP
web_port: 80
web_user_list:
- sysadmins
web_cache: 12h
==> Per node view
profile:
env: dev
product: httpd_server
web_top_domain: dev.infra.net
web_app: myapp
web_port: 80
web_user_list:
- domain_org
- domain_org_external
web_cache: 1m
mysql_users:
- debug@10.0.%
debug: true
Even if somwaht clunky, this method can help to troubleshoot wrong data by dichotomy.
```bash
```
## Tooling and applications
```bash
```
```bash
```
```bash
```
```bash
```
```bash
```
## Troubleshooting
Sometimes, it can may be hard to navigate across file and hierachy, but GNU Utils are here to help. There is a selection of small tips:
```bash
set -x
: Find where a key has been defined
: ==========================
grep -r '^profile:' examples/$KHEOPS_NAMESPACE
: Find where a key has been defined and 5 first lines
: ==========================
grep -r -A 5 'web_user_list:' examples/$KHEOPS_NAMESPACE
: Search from anything related to database
: ==========================
grep -R -C 3 'database' examples/$KHEOPS_NAMESPACE
set +x
```
+ : Find where a key has been defined
+ : ==========================
+ grep --colour=auto -r '^profile:' examples/ex1_enc
examples/ex1_enc/env_prod.yml:profile:
examples/ex1_enc/roles/mysql.yml:profile:
examples/ex1_enc/roles/web.yml:profile:
examples/ex1_enc/nodes/mysql.infra.net.yml:profile:
examples/ex1_enc/nodes/web.infra.net.yml:profile:
examples/ex1_enc/default.yml:profile:
examples/ex1_enc/env_dev.yml:profile:
+ : Find where a key has been defined and 5 first lines
+ : ==========================
+ grep --colour=auto -r -A 5 web_user_list: examples/ex1_enc
examples/ex1_enc/roles/web.yml: web_user_list:
examples/ex1_enc/roles/web.yml- - sysadmins
examples/ex1_enc/roles/web.yml-
--
examples/ex1_enc/nodes/web.infra.net.yml: web_user_list:
examples/ex1_enc/nodes/web.infra.net.yml- - domain_org
examples/ex1_enc/nodes/web.infra.net.yml- - domain_org_external
examples/ex1_enc/nodes/web.infra.net.yml-
--
examples/ex1_enc/env_dev.yml: web_user_list:
examples/ex1_enc/env_dev.yml- - debug_user
examples/ex1_enc/env_dev.yml- mysql_users:
examples/ex1_enc/env_dev.yml- - debug@10.0.%
examples/ex1_enc/env_dev.yml-
examples/ex1_enc/env_dev.yml- debug: true
+ : Search from anything related to database
+ : ==========================
+ grep --colour=auto -R -C 3 database examples/ex1_enc
examples/ex1_enc/roles/mysql.yml-profile:
examples/ex1_enc/roles/mysql.yml- product: "mysql_server"
examples/ex1_enc/roles/mysql.yml-
examples/ex1_enc/roles/mysql.yml: mysql_database: "NO_DATABASE"
examples/ex1_enc/roles/mysql.yml- mysql_users:
examples/ex1_enc/roles/mysql.yml- - "sysadmin@10.0.42%"
examples/ex1_enc/roles/mysql.yml- mysql_port: 3306
--
examples/ex1_enc/nodes/mysql.infra.net.yml----
examples/ex1_enc/nodes/mysql.infra.net.yml-profile:
examples/ex1_enc/nodes/mysql.infra.net.yml: mysql_database: "app_domain_org"
examples/ex1_enc/nodes/mysql.infra.net.yml- mysql_users:
examples/ex1_enc/nodes/mysql.infra.net.yml- - "app_domain_org@10.0.51%"
examples/ex1_enc/nodes/mysql.infra.net.yml-
+ set +x
```bash
```
The tail/head command is quite usefull to look at multiple files at the same time, it add a nice header for each file:
```bash
head -n 999 examples/ex1_enc/roles/*
```
==> examples/ex1_enc/roles/mysql.yml <==
---
profile:
product: "mysql_server"
mysql_database: "NO_DATABASE"
mysql_users:
- "sysadmin@10.0.42%"
mysql_port: 3306
mysql_cluster: False
==> examples/ex1_enc/roles/web.yml <==
---
profile:
product: "httpd_server"
web_top_domain: ""
web_app: "NO_APP"
web_port: 80
web_user_list:
- sysadmins
You can also have a view of all files with this command:
```
find . -type f| xargs head -n 999 | less
```
From there, you will be able to have a nice overview of your data.
You can even diff your change with this command. There is this simple trick to compare the data difference between 2 lookups:
```bash
diff -u \
<(kheops lookup -e node=web.infra.net -e role=web -e env=prod profile) \
<(kheops lookup -e node=web.infra.net -e role=web -e env=dev profile)
```
--- /dev/fd/63 2022-02-14 13:45:59.223619144 -0500
+++ /dev/fd/62 2022-02-14 13:45:59.223619144 -0500
@@ -1,11 +1,14 @@
profile:
- env: prod
+ env: dev
product: httpd_server
- web_top_domain: infra.com
+ web_top_domain: dev.infra.net
web_app: myapp
web_port: 80
web_user_list:
- domain_org
- domain_org_external
- web_cache: 12h
+ web_cache: 1m
+ mysql_users:
+ - debug@10.0.%
+ debug: true
You can also ask Kheops to explain you how he built the result, you can use the `-x` flag:
```bash
kheops lookup -e role=web profile -X
```
INFO: Explain lookups:
+------------------------+------------------------------+
| Config | Runtime |
+------------------------+------------------------------+
| | |
| Config:{ | Runtime:{ |
| "path": "default", | "scope": { |
| "backend": "file", | "role": "web" |
| "continue": true | }, |
| } | "key": "profile", |
| | "conf": { |
| | "index": 0 |
| | }, |
| | "raw_path": "default" |
| | } |
| | |
| Config:{ | Runtime:{ |
| "path": "roles/web", | "scope": { |
| "backend": "file", | "role": "web" |
| "continue": true | }, |
| } | "key": "profile", |
| | "conf": { |
| | "index": 1 |
| | }, |
| | "raw_path": "roles/{role}" |
| | } |
+------------------------+------------------------------+
INFO: Explain candidates:
+----------------------------------------------------------------------------------+-------------------------------+------------------------------+
| Status | Runtime | Key Value |
+----------------------------------------------------------------------------------+-------------------------------+------------------------------+
| | | |
| Status:{ | Runtime:{ | Key:{ |
| "path": "/home/jez/volumes/data/prj/bell/dev/kheops/examples/ex1_enc/defau ... | "scope": { | "env": "NO_ENV", |
| "status": "found", | "role": "web" | "product": "NO_PRODUCT" |
| "rel_path": "examples/ex1_enc/default.yml" | }, | } |
| } | "key": "profile", | |
| | "conf": { | |
| | "index": 0 | |
| | }, | |
| | "raw_path": "default", | |
| | "backend_index": 0 | |
| | } | |
| | | |
| Status:{ | Runtime:{ | Key:{ |
| "path": "/home/jez/volumes/data/prj/bell/dev/kheops/examples/ex1_enc/roles ... | "scope": { | "product": "httpd_server", |
| "status": "found", | "role": "web" | "web_top_domain": "", |
| "rel_path": "examples/ex1_enc/roles/web.yml" | }, | "web_app": "NO_APP", |
| } | "key": "profile", | "web_port": 80, |
| | "conf": { | "web_user_list": [ |
| | "index": 1 | "sysadmins" |
| | }, | ] |
| | "raw_path": "roles/{role}", | } |
| | "backend_index": 1 | |
| | } | |
+----------------------------------------------------------------------------------+-------------------------------+------------------------------+
profile:
env: NO_ENV
product: httpd_server
web_top_domain: ''
web_app: NO_APP
web_port: 80
web_user_list:
- sysadmins
```bash
```