Troubleshoot Ansible with an Ara over your shoulder

class: center, middle, inverse

# Troubleshoot Ansible with an ARA over your shoulder

## Haïkel Guémar - @hguemar

---
## whoami

.right-column[
<img src="img/me.jpeg" width="80%" />
]

.left-column[

* CentOS Cloud SIG contributor
* Senior Software Engineer @ Red Hat
* Self-proclaimed Chief Cloud Fairy

]

---
class: center, middle, inverse

# What's an Ara?

---
class: middle

## It's a bird!

Well, Macaw or Ara in French are long-tailed often colourful (and very noisy) new world parrots.

---
class: middle

## And it's also ...

ARA Records Ansible which is a very handy Ansible plugin!

<span style="font-size:x-small; font-style:oblique">(Which makes you feel like a space hunter finding a mysterious alien powerful item)</span>

---
class: middle

## Ansible 101 - ad-hoc

```sh
# ping webservers nodes
$ ansible -i inventory -m ping webservers
# Update kernel on all nodes
$ ansible -i inventory -m package -a "name=kernel state=latest" all
```

---
class: middle

## Ansible 101 - playbook

```yaml
- hosts: all
  serial: "25%"
  tasks:
    - name: Read kernel configuration
      shell: cat /boot/config-$(uname -r)
      register: kernel_config
      changed_when: false

- when: '"CONFIG_KAISER=y" not in kernel_config.stdout'
      block:
        - name: Install latest kernel
          package:
            name: kernel
            state: latest
          register: kernel_update

- name: Reboot the server
          reboot:
          when: kernel_update is changed
```

---
class: middle

## Ansible 101 concepts

- Readable
- Self-documenting
- Orchestration
- Facts
- Roles for composability
- Handlers
- Plugins/Modules

---
class: middle

## Running a playbook

```sh
$ ansible-playbook -i hosts playbook.yml

PLAY [all] *******************************************

TASK [Gathering Facts] *******************************
ok: [centos]

TASK [Read kernel configuration] *********************
ok: [centos]

TASK [Install latest kernel] *************************
changed: [centos]

TASK [Reboot the server] *****************************
changed: [centos]

PLAY RECAP *******************************************
centos: ok=4    changed=2    unreachable=0    failed=0
```

---
class: middle

## Actual playbook output

---
class: middle

## RDO story

* hundreds of servers
* Thousands jobs
* Millions lines of logs

A true nightmare to debug

---
class: middle

## How to fix that situation?

Ansible callback plugins?

https://rndmh3ro.github.io/

---
class: middle

## JSON ain't for humans

---
class: middle

## So ARA is

* an open source project from the OpenStack community
* an Ansible callback plugin to save logs into a queryable database
* a command-line interface to query that database
* a web inteface to display Ansible output in a clear way
* a REST API since ARA 1.0 if you like to.

---
class: middle, inverse, center

# I want to use ARA!

---
class: middle

## ARA 101

```sh
# From PyPI
$ pip install ara

# Configure Ansible to use it
$ export ANSIBLE_CALLBACK_PLUGINS=$(python -m ara.setup.callback_plugins)

# No need to change your playbook!
$ ansible-playbook playbook.yml

# Launch web server to display output
$ ara-manage runserver

# Open your browser!
$ xdg-open http://127.0.0.1:9191
```

---
class: middle

## That's it?

---
class: middle

## ARA Manifesto

https://ara.readthedocs.io/en/stable/manifesto.html

1) Simplicity is fundamental

2) Do one thing and do it well

3) Empower users to get their work done

4) Don’t require users to change their workflows

5) De-centralized, offline and standalone by default

---
class: middle, inverse, center

# How does it work?

---
class: middle

## ARA provides a callback plugin to Ansible

```python
# https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback
def v2_on_any(self, *args, **kwargs):
def v2_runner_on_failed(self, result, ignore_errors=False):
def v2_runner_on_ok(self, result):
def v2_runner_on_skipped(self, result):
def v2_runner_on_unreachable(self, result):
def v2_runner_on_no_hosts(self, task):
def v2_playbook_on_start(self, playbook):
def v2_playbook_on_task_start(self, task, is_conditional):
def v2_playbook_on_setup(self):
def v2_playbook_on_play_start(self, play):
def v2_playbook_on_stats(self, stats):
def v2_playbook_on_include(self, included_file):
[...]
```

---
class: middle

## ARA provides a callback plugin to Ansible

```python
# https://github.com/openstack/ara/blob/master/ara/plugins/callbacks/log_ara.py
[...]
class CallbackModule(CallbackBase):
    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = 'notification'
    CALLBACK_NAME = 'ara'

# [...]
    def v2_playbook_on_start(self, playbook):
        self.playbook = models.Playbook(
            ansible_version=ansible_version,
            path=os.path.abspath(playbook._file_name)
        )
        db.session.add(self.playbook)
    
    def v2_runner_on_ok(self, result, **kwargs):
        self.log_task(result, 'ok', **kwargs)
    [...]
```

---
class: middle

## ARA also provides an Ansible module

```python
- name: Test playbook
  hosts: localhost
  tasks:
    - name: Get git version of playbooks
      command: git rev-parse HEAD
      register: git_version

- name: Record git version
      ara_record:
        key: "git_version"
        value: "{{ git_version.stdout }}"
```
<img src="img/Screenshot_from_2017-04-08_23-29-32.png" width="80%" />

---
class: middle, center, inverse

## ARA and friends

---
class: middle

## CI and CD with Zuul

* Developped initially for RDO
* Adopted by OpenStack

Zuul uses it in various places:
* as callback plugin
* a role to save reports in database
* display jobs reports

---
class: middle

## Why no big fat database?

* Context of a CI: one database, one report
* Performances and latency
* Scaling up to millions of playbooks execution per month
* what happens in case of power outage
* SQLite is good enough

---
class: middle, center, inverse

# ARA 1.0 (TBA)

---
class: middle

## ARA is growing

* ara-server
* ara-clients (offline/online)
* ara-plugins (aka ARA core features)
* ara-web
* plus: ansible-role-ara! (Now you can deploy ARA with Ansible too)

---
class: middle

## Must reads from ARA development blog

* [Introducing new projects for the upcoming 1.0 release](https://ara.recordsansible.org/blog/2019/01/16/introducing-new-projects-for-the-upcoming-1.0-release)
* [Status update: ARA 1.0](https://ara.recordsansible.org/blog/2017/11/22/status-update-ara-1.0/)
* [What's coming in ARA 1.0 ?](https://ara.recordsansible.org/blog/2017/08/16/whats-coming-in-ara-1.0/)

---
class: middle

## Thanks

Thanks to David Moreau Simmard for creating ARA and helping me to prepare this presentation.

---
class: middle, center, inverse

# Q/A