Cours Génie Logiciel

class: center, middle, inverse

# Cours Génie Logiciel

---
class: middle

## Disclaimer

Slides are in English, audio will be available in either French or English

---
class: center, middle, inverse

# Industrializing an Open Source Project
## Study case: OpenStack

Haïkel Guémar - haikel.guemar@gmail.com

---
class: center, middle, inverse

# Introduction

---
class: middle

## WhoAmI

1. FOSS developer for longer than I'd like to admit
2. RDO Release Wrangler @ Red Hat since 2014
3. Former student of University of Lyon 1 ;-)

---
class: middle

## Why I'm here?

This is a Software Engineering class, you learnt how to write good code (includes good test!)

But ...

---
class: middle

## Why I'm here?

Software Engineering is not a science but an __engineering__ discipline.
Our job is to solve problems with software (and sometimes without it), not write code.

---
class: middle

## Software Engineering definition

"the systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software"
_IEEE Systems and software engineering_

---
class: middle

## Software Engineering encompasses:

* Writing
* Building
* Testing
* Deploying

And automating all those steps because we're lazy people.

---
class: middle

## Release Engineering in few words

[Release engineering is about building and delivering software.](https://www.oreilly.com/webops-perf/free/release-engineering.csp)
We use our development knowledge, configuration management (source version control if you like), testing, system administration and customer engagement to build, assemble, deliver code into finished products.

---
class: middle

## Software Engineering is a team discipline

So it's important to understand how to collaborate with your team
![](img/cubicle-fight.gif)

---
class: middle

## General Plan

1. Explain what's OpenStack, RDO, RHOSP
2. Our issues in delivering such Behemoth
3. How we fixed it and reorganized ourselves to achieve this purpose

---
class: middle, center, inverse

# OpenStack
---
class: middle

---
class: middle

## Cloud Computing 101

_Involve a lot of hands gestures_

---
class: middle

## OpenWhat?

OpenStack is a collection of FOSS projects to build/manage a Cloud Computing Platform.
It is run by the OpenStack Foundation (Non-Profit)

---
class: middle

## Buckle up, let's dive into details

---
class: middle

## Openstack architecture

---
class: middle

## RDO

RDO is a distribution of OpenStack packaged and tested on Enterprise Linux (CentOS/RHEL and friends)
It is a community project run by Red Hat

---
class: middle

## RDO in numbers

* 4 currently supported releases: Ocata, Pike, Queens, Rocky + master (Steins)
* 1637 binary packages
* 321 sources packages (OpenStack) + 337 more (dependencies)
* 36 clients, 92 puppet modules
* 5 installers: Puppet, TripleO, Packstack, Kolla, Openstack Ansible

---
class: middle

## More statistics (Pike cycle)

* 1966 commits (10 commits merged per day avg)
* 87 contributors (13 not from Red Hat)

In comparison, upstream:

* 1800+ contributors
* 250 commits per day

---
class: middle

## RDO largest deployment

Yes we power the LHC (well, LHC data analysis cloud)
<img src="img/lhc17.jpg" width="90%" />

---
class: middle

## Red Hat OpenStack Platform

Red Hat OpenStack Platform is Red Hat commercial offering based on OpenStack.
It builds upon RDO

---
class: middle, inverse

## Shameless self-promotion

_Mostly because I'm proud that we released it_

---
class: middle

## Why Red Hat OpenStack Platform ?

* Longer shelf life than upstream OpenStack
* Integration with other Red Hat Products (RHEL, OpenShift, Ceph)
* Certification Program
* Reliable upgrades! (Yes, we're only vendor to do that)

---
class: middle, inverse

## End of Shameless self-promotion

---
class: middle

## The mission

OpenStack is made of:
* 696 projects
* 1458 contributors
* 145 organizations

An heavy weight in the Open Source world and a challenge to package

---
class: middle

## Packaging OpenStack has been a challenge

* Used to be very tied to Ubuntu
* Fast-paced development
* And relies heavily on system components (virtualization, storage etc.)
* Components are interdependent
* Our Red Hat OpenStack org grew very fast from few people to hundredfolds in few years.

---
class: middle

## Our solution

* Package (almost) every single commit
* Shape our infra to follow as closely as possible upstream's
* Packaging changes are reviewed through gerrit
* Builds are automated into our Koji build systems (little to no human action to trigger builds)
* Every build goes through CI
* CI is driven by OpenStack Zuul (+Ansible for automation

---
class: middle

## DLRN: Continuous Packaging platform

* DLRN continuous checkout OpenStack repositories
* Rebuild package if there's a new commit
* Trigger a review when there's a build failure (so developers can fix it)

---
class: middle

## DLRN: Continuous Packaging platform

```bash
+------------+    +---------+    +------------+    +-------------------+
|            |    |         |    |            |    |                   |
| New commit +----> current +----> consistent +----> current+passed+ci |
|            |    |         |    |            |    |                   |
+------------+    +---------+    +-----^------+    +---------^---------+
                                       |                     |
                                       |                     |
                                 +-----+------+    +---------+---------+
                                 |   RDO CI   |    |    Upstream CI    |
                                 +------------+    +-------------------+
```

* master: every commit or almost is built
* consistent: set of packages without build failures
* current-passed-ci: snapshot that passed CI
* stable releases: packages from upstream tagged tarballs + sanity checks

* RDO CI targets consistent snapshots
* Upstream CI projects targets current-passed-ci (e.g. packstack, tripleO, kolla, puppet)

---
class: middle

## Red Hat OpenStack stakeholders

* OpenStack Infrastructure: engineering team responsible of producing both RDO and RHOSP
* OpenStack QE: QE teams responsible for testing and validating RHOSP
* Business Partners: includes various hardware vendors, companies building their products upon RHOSP etc.
* RDO Community: the people building RDO

---
class: middle, center, inverse

# Production Chain

---
class: middle

## Prerequisites

1. Red Hat OpenStack Platform must deliver vanilla experience
2. RHOSP production pipeline starts from upstream development branch to customer experience

---
class: middle

## Context

* A vibrant open source community
* Hundreds of engineers located in various area and most of them being remotees
* Tight schedule: 2 major releases per year
* Skills split accross teams, and various organizations (Engineering, QE etc.)
* When I joined Red Hat, RDO release process was still handcrafted
* Inevitable delays between upstream and downstream releases
* PM shoving features/backports requests last minute

---
class: middle

## Classic issue: Engineering vs QE

---
class: middle

## Our answer

* Engineering/QE were reorganized into Delivery Focus Groups
* Rethink the production pipeline from the ground
* Call for rescue our super duper agile coaches to build our own model rather than dumbly apply Scrum or whatever methodology
* Embed PM in DFG so that they can provide early feedback and not be frustrated!
* Philosophy: all work must be done upstream, then in RDO (Community),

---
class: middle

## Delivery Focused Group (DFG)

* End-to-end responsibility for delivering a story
* Cross-functional groups (engineers, QE, Product managers, customer experience etc.)
* Focus is on delivering value to customer (not code, not tests etc.)
* Self-organized group that defines its own mission statement, day-to-day work

---
class: middle

## Delivery Focused Group (DFG) - roles

Making a parallel with Scrum but do not assume 1:1 mapping

* Team catalyst: Scrum master
* User Advocate: Product Owner proxy
* Product Manager: Product Owner
* Stewards: Adult supervision (this is a joke!)

---
class: middle

## Delivery Focused Group (DFG) - specifics

Since team are self-organized, some have specifics organization:

* Squads (and sometimes squad leaders)
* Team lead
* Rotating roles

---
class: middle

## DFG was the result of a long process

* Feature teams (Uncertainty for associates)
* Manager role shifted from traditional command/control model (C2) to support role (business, people, production¸ system)
* Management Team: responsible to keep the show running
* Leadership team: responsible to define shared vision

---
Class: middle

## The book

One of the architects of this reorganization Alexis Monville wrote a
book ["Changing your team from the inside"](http://alexis.monville.com/en/) and it is inspired by our experience.

![](img/persp-book.png)

[The devconf.cz talk (video + slides)](http://alexis.monville.com/en/blog/2018/01/28/changing-your-team-from-the-inside-devconf-cz/)

---
class: middle, center, inverse

# The production chain

---
class: middle

## Before

OpenStack Release Engineering was split accross:
* RDO Engineering team
* Productization team
* Internal Red Hat Release Engineering
* RDO CI team
* Various Product QE teams

Of course, we knew and worked with each other!

---
class: middle

## After

* One Production Chain DFG
* No organigram change
* Everyone had to adjust to different work approach

---
class: middle

## First, build common vision

* We gathered everyone in a room and made them define dream land OpenStack Release pipeline
* Define our mission statement
* Management acknowledged that and provided means to achieve it
* Nobody truly believe it was possible
* For some people, it was their first time to meet their coworkers face to face

---
class: middle

## Then, the first painful steps

* Start laying the foundation for the pipeline
* TBH, RDO release engineering was still a Work-In-Process
* Everyone had to adjust (e.g: my calendar was suddenly full of calls)

---
class: middle

## One year later

* Another workshop to measure progress and define upcoming actions
* Dreamland vision was mostly there
* We had better idea on how to organize ourselves
* Each branch had 2 leads and we organized people across squads
* Despite being trapped in a snowstorm, nobody tried to kill their coworkers.

---
class: middle

## Production Chain

* A set of tools and processes
* Upstream code is built into RDO packages
* RDO packages goes through CI, then are imported into RHOSP (devel)
* RHOSP builds are gated and then published on CDN

---
class: middle

## How it looks like

---
class: middle

## Tooling

* [Software Factory](https://softwarefactory-project.io/sf/welcome.html) (Gerrit, Zuul) 
* DLRN
* rdopkg
* graffiti, sinkhole
* Red Hat/CentOS build/delivery infrastructure

---
class: middle, center, inverse

# Prodchain Responsibility

---
class: middle

## General

* Release Engineering is reponsible of the whole product delivery
* Not writing features, nor fixing bugs unless it is related to the distro
* Release paperwork (e.g crypto export compliance, errata, updates)

---
class: middle

## CI

* Initial Triage of CI failures
* Appropriately filing bugs and escalating CI failures
* Maintenance of CI systems
* Ensuring that the correct pipeline of tests is in place
* Ensuring that the right tests are running at each level

---
class: middle

## Packaging

* Responsible for RDO and OSP builds
* Imports from RDO to OSP
* Setting quality standards

---
class: middle, center, inverse

# DFG responsibility

---
class: middle

## General

* Handling CI escalations
* Implementing/defining CI job
* CI coverage for new features

---
class: middle

## Packaging

* Initial package reviews
* Packaging updates
* Downstream patches
* Requesting new packages (dependencies)
* Hotfixes, CVE

---
class: middle, center, inverse

# Conclusion

---
class: middle, center, inverse

# Q/A