Disclaimer

Slides are in English, audio will be available in either French or English

WhoAmI

1. FOSS developer for longer than I'd like to admit
2. RDO Release Wrangler @ Red Hat since 2014
3. Former student of University of Lyon 1 ;-)

Why I'm here?

This is a Software Engineering class, you learnt how to write good code (includes good test!)

But ...

Why I'm here?

Software Engineering is not a science but an engineering discipline. Our job is to solve problems with software (and sometimes without it), not write code.

Software Engineering definition

"the systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software" IEEE Systems and software engineering

Software Engineering encompasses:

• Writing
• Building
• Testing
• Deploying

And automating all those steps because we're lazy people.

Release Engineering in few words

Release engineering is about building and delivering software. We use our development knowledge, configuration management (source version control if you like), testing, system administration and customer engagement to build, assemble, deliver code into finished products.

Software Engineering is a team discipline

So it's important to understand how to collaborate with your team

General Plan

1. Explain what's OpenStack, RDO, RHOSP
2. Our issues in delivering such Behemoth
3. How we fixed it and reorganized ourselves to achieve this purpose

Cloud Computing 101

Involve a lot of hands gestures

OpenWhat?

OpenStack is a collection of FOSS projects to build/manage a Cloud Computing Platform. It is run by the OpenStack Foundation (Non-Profit)

Buckle up, let's dive into details

Openstack architecture

RDO

RDO is a distribution of OpenStack packaged and tested on Enterprise Linux (CentOS/RHEL and friends) It is a community project run by Red Hat

RDO in numbers

• 4 currently supported releases: Ocata, Pike, Queens, Rocky + master (Steins)
• 1637 binary packages
• 321 sources packages (OpenStack) + 337 more (dependencies)
• 36 clients, 92 puppet modules
• 5 installers: Puppet, TripleO, Packstack, Kolla, Openstack Ansible

More statistics (Pike cycle)

• 1966 commits (10 commits merged per day avg)
• 87 contributors (13 not from Red Hat)

In comparison, upstream:

• 1800+ contributors
• 250 commits per day

RDO largest deployment

Yes we power the LHC (well, LHC data analysis cloud)

Red Hat OpenStack Platform

Red Hat OpenStack Platform is Red Hat commercial offering based on OpenStack. It builds upon RDO

Why Red Hat OpenStack Platform ?

• Longer shelf life than upstream OpenStack
• Integration with other Red Hat Products (RHEL, OpenShift, Ceph)
• Certification Program
• Reliable upgrades! (Yes, we're only vendor to do that)

The mission

OpenStack is made of:

• 696 projects
• 1458 contributors
• 145 organizations

An heavy weight in the Open Source world and a challenge to package

Packaging OpenStack has been a challenge

• Used to be very tied to Ubuntu
• Fast-paced development
• And relies heavily on system components (virtualization, storage etc.)
• Components are interdependent
• Our Red Hat OpenStack org grew very fast from few people to hundredfolds in few years.

Our solution

• Package (almost) every single commit
• Shape our infra to follow as closely as possible upstream's
• Packaging changes are reviewed through gerrit
• Builds are automated into our Koji build systems (little to no human action to trigger builds)
• Every build goes through CI
• CI is driven by OpenStack Zuul (+Ansible for automation

DLRN: Continuous Packaging platform

• DLRN continuous checkout OpenStack repositories
• Rebuild package if there's a new commit
• Trigger a review when there's a build failure (so developers can fix it)

DLRN: Continuous Packaging platform

+------------+    +---------+    +------------+    +-------------------+
|            |    |         |    |            |    |                   |
| New commit +----> current +----> consistent +----> current+passed+ci |
|            |    |         |    |            |    |                   |
+------------+    +---------+    +-----^------+    +---------^---------+
                                       |                     |
                                       |                     |
                                 +-----+------+    +---------+---------+
                                 |   RDO CI   |    |    Upstream CI    |
                                 +------------+    +-------------------+

• master: every commit or almost is built
• consistent: set of packages without build failures
• current-passed-ci: snapshot that passed CI

• stable releases: packages from upstream tagged tarballs + sanity checks

• RDO CI targets consistent snapshots

• Upstream CI projects targets current-passed-ci (e.g. packstack, tripleO, kolla, puppet)

Red Hat OpenStack stakeholders

• OpenStack Infrastructure: engineering team responsible of producing both RDO and RHOSP
• OpenStack QE: QE teams responsible for testing and validating RHOSP
• Business Partners: includes various hardware vendors, companies building their products upon RHOSP etc.
• RDO Community: the people building RDO

Prerequisites

1. Red Hat OpenStack Platform must deliver vanilla experience
2. RHOSP production pipeline starts from upstream development branch to customer experience

Context

• A vibrant open source community
• Hundreds of engineers located in various area and most of them being remotees
• Tight schedule: 2 major releases per year
• Skills split accross teams, and various organizations (Engineering, QE etc.)
• When I joined Red Hat, RDO release process was still handcrafted
• Inevitable delays between upstream and downstream releases
• PM shoving features/backports requests last minute

Classic issue: Engineering vs QE

Our answer

• Engineering/QE were reorganized into Delivery Focus Groups
• Rethink the production pipeline from the ground
• Call for rescue our super duper agile coaches to build our own model rather than dumbly apply Scrum or whatever methodology
• Embed PM in DFG so that they can provide early feedback and not be frustrated!
• Philosophy: all work must be done upstream, then in RDO (Community),

Delivery Focused Group (DFG)

• End-to-end responsibility for delivering a story
• Cross-functional groups (engineers, QE, Product managers, customer experience etc.)
• Focus is on delivering value to customer (not code, not tests etc.)
• Self-organized group that defines its own mission statement, day-to-day work

Delivery Focused Group (DFG) - roles

Making a parallel with Scrum but do not assume 1:1 mapping

• Team catalyst: Scrum master
• User Advocate: Product Owner proxy
• Product Manager: Product Owner
• Stewards: Adult supervision (this is a joke!)

Delivery Focused Group (DFG) - specifics

Since team are self-organized, some have specifics organization:

• Squads (and sometimes squad leaders)
• Team lead
• Rotating roles

DFG was the result of a long process

• Feature teams (Uncertainty for associates)
• Manager role shifted from traditional command/control model (C2) to support role (business, people, production¸ system)
• Management Team: responsible to keep the show running
• Leadership team: responsible to define shared vision

Before

OpenStack Release Engineering was split accross:

• RDO Engineering team
• Productization team
• Internal Red Hat Release Engineering
• RDO CI team
• Various Product QE teams

Of course, we knew and worked with each other!

After

• One Production Chain DFG
• No organigram change
• Everyone had to adjust to different work approach

First, build common vision

• We gathered everyone in a room and made them define dream land OpenStack Release pipeline
• Define our mission statement
• Management acknowledged that and provided means to achieve it
• Nobody truly believe it was possible
• For some people, it was their first time to meet their coworkers face to face

Then, the first painful steps

• Start laying the foundation for the pipeline
• TBH, RDO release engineering was still a Work-In-Process
• Everyone had to adjust (e.g: my calendar was suddenly full of calls)

One year later

• Another workshop to measure progress and define upcoming actions
• Dreamland vision was mostly there
• We had better idea on how to organize ourselves
• Each branch had 2 leads and we organized people across squads
• Despite being trapped in a snowstorm, nobody tried to kill their coworkers.

Production Chain

• A set of tools and processes
• Upstream code is built into RDO packages
• RDO packages goes through CI, then are imported into RHOSP (devel)
• RHOSP builds are gated and then published on CDN

How it looks like

Tooling

• Software Factory (Gerrit, Zuul)
• DLRN
• rdopkg
• graffiti, sinkhole
• Red Hat/CentOS build/delivery infrastructure

General

• Release Engineering is reponsible of the whole product delivery
• Not writing features, nor fixing bugs unless it is related to the distro
• Release paperwork (e.g crypto export compliance, errata, updates)

CI

• Initial Triage of CI failures
• Appropriately filing bugs and escalating CI failures
• Maintenance of CI systems
• Ensuring that the correct pipeline of tests is in place
• Ensuring that the right tests are running at each level

Packaging

• Responsible for RDO and OSP builds
• Imports from RDO to OSP
• Setting quality standards

General

• Handling CI escalations
• Implementing/defining CI job
• CI coverage for new features

Packaging

• Initial package reviews
• Packaging updates
• Downstream patches
• Requesting new packages (dependencies)
• Hotfixes, CVE