Skip to main content
CentOS Stream Kernel Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage
  1. CentOS Stream Kernel Documentation
  2. /
  3. Docs
  4. /
  5. Backport Automation

Backport Automation

OVERVIEW & PURPOSE

This document attempts to detail Red Hat’s current Linux kernel patch backport automation infrastructure.

OBJECTIVES

  • Detail all the software components that comprise the backport automation infrastructure.

  • Detail all the ways humans can interact with the backport automation infrastructure.

  • Detail the ways to verify the backport automation infrastructure is operating correctly.

SOFTWARE COMPONENTS

The pieces of the puzzle:

  • Red Hat Jira – Every automated backport is tied directly to a single issue in Red Hat’s Jira instance, in the RHEL project.

  • Public-inbox copy of upstream linux-cve-announce mailing list.

  • cve_assigner.py – Automated population of Jira data based on linux-cve-announce.

  • backporter.py – Webhook listening for Jira webhook events.

  • check_for_backports.py – Periodic cron job searching for z-stream Jira issues with sufficiently progressed y-stream MRs to backport from.

  • libbackport.py – The core backporting library shared by both of the above backport routines.

  • GitLab backport bot branches – Where the automated backports are pushed to create merge requests.

  • GitLab Merge Requests (MRs) – If you don’t know what these are and where to find them, you probably shouldn’t be reading this document yet.

INTERFACES (Ways in which humans interact with the automation)

There are two different ways in which automated backports are initiated.

First Way: Populating the Commit Hashes field

The first way requires the Jira issue’s Commit Hashes field being populated with valid upstream kernel commit hashes.

After updating this field, a webhook event is emitted from Jira, which is received by the kernel webhooks listener and then delivered to the backport engine (backporter.py).

An automated backport will be attempted if and when the following conditions are met:

  • The issue Status is New or Planning.

  • The issue has a valid value in the Jira “Fix Versions” field.

  • Backporting is allowed for the given branch (it is typically disallowed in pre-release branches and inactive retired streams).

  • The issue does not already have a merge request linked to it.

  • The issue does not already have a prior backport attempt label on it (kwf-backport-success or kwf-backport-failure).

  • At least one line in the “Commit Hashes” field looks like a commit hash.

The backporter.py tool will attempt backporting using the values in the “Commit Hashes” Jira field that match the format of git commit hashes – cherry-picking them one by one, into a branch based on the current HEAD of the target git branch that matches the Fix Version.

Second Way: Periodic Cron Job

The second way is via a cron job (check_for_backports.py) which runs a few times a day. It searches for z-stream Jira issues (determined by the Fix Version field) with discoverable y-stream issues (also determined by their Fix Version field) that have merge requests attached which are sufficiently progressed (with varying rules based on issue type and severity).

When check_for_backports.py finds a sufficiently progressed y-stream merge request, it will attempt to cherry-pick the commits from the y-stream merge request into a branch based on the HEAD of the target z-stream branch.

Actions Taken in Both Pathways

In both automated backport pathways, similar actions are taken:

  • On Failure: A kwf-backport-failure label is added to the Jira issue.

  • On Success:

    • An MR is submitted.

    • A kwf-backport-success label is added to the Jira issue.

  • In both success and failure cases, a comment is added to the relevant Jira issue, providing updates on actions taken and required next steps.

When is the “Commit Hashes” field set?

The "Commit Hashes" field in a given Jira issue is typically populated by one of three methods:

  1. Automated Population via cve_assigner.py:

    • Product Security uses this script immediately after filing CVE bugs.

    • The script reassigns issues to appropriate kernel sub-components or maintainers and populates the "Commit Hashes" field with upstream kernel commit hashes that address the CVE.

  2. Manual Population:

    • Developers manually research and identify commits, then populate the field.

    • This action also triggers an automated backport attempt, even if unintended.

  3. Post-backport-population by check_for_backports.py:

    • If the cron job finds a z-stream backport it is able to complete successfully, it will also populate the Commit Hashes field, but with the y-stream commit hashes it has backported.

Regardless of the Commit Hashes population method used, the backport automation system treats the Jira issue the same. This feature supports both CVE and non-CVE bugs, enabling developers to leverage the tooling to reduce the overhead of handling MRs across multiple streams.

Result of Automation: Merge Requests

Successful backport attempts result in draft MRs authored by the "CKI Backport Bot". These MRs are pushed to a bot branch namespaces on gitlab. All bot branches are accessible to Red Hat GitLab users with Developer-level permissions or higher.

If changes are needed, developers can pull, update, and force-push changes to these branches, provided they:

  • Add their Signed-off-by: line to updated commits.

  • Include their Signed-off-by: in the MR description.

When a developer believes an automated backport is correct, they must:

  • Update the "Dev Target Milestone" field in Jira.

  • Move the MR from "Draft" to "Ready" status for further review.

  • Provide approval for the MR if they are the assigned developer.

Future Enhancements

  • A kernel workflow cron job will automatically close stale Draft MRs linked to Jira issues resolved as "Won’t Do".

  • Currently, stale Draft MRs are closed manually (as of 2024-11-05).

  • Warning developers that their stale draft MR’s are being closed?

MONITORING (How do we know the backport automation infrastructure is working?)

This area is still under development!

There are some Jira dashboards that have been developed to track success and failure, but they need some further refinement to properly reflect all possible states. Also, Jira is messy for sharing read/write access on things like this, making collaboration a bit of a pain so far.

gdoc_arrow_left_alt RHEL Kernel Workflow General Build Instructions gdoc_arrow_right_alt