Kernel Variant Jira Process

How we test kernel and its growing list of variants is an ongoing issue. Holding up merge requests by awaiting test verification of not only kernel but an occasionally missed variant counterpart tracker that may even be untriaged is a pain point for developers. Awaiting verification of a late-discovered variant counterpart tracker can lead to delays, more context switching, and more time spent rebasing. However from a QE and customer-centric view, we cannot be releasing kernel variants untested in scenarios where a kernel patch may affect said variant, so it is imperative we maintain a procedure to track kernel variant testing.

This issue has made itself largely visible in how we handled kernel-rt. There are many scenarios where a kernel and RT bug are filed for a single issue and must be manually kept in sync. Some improvements have been made to kernel webhooks and automations to discover and gate kernel MRs on kernel-rt counterpart trackers. However, this discovery procedure is not perfect as it depends on trackers having correct fields set (summary, fix version, and so on). As we ramp up additional variants, this problem grows larger and manual efforts to manage it cannot scale.

Managing kernel variants like RT in bugs completely separate from the kernel itself has led to pain points for all members of kernel subsystems. Therefore kernel variant testing will now be handled via linked RHEL Jira tickets of type "Task."

A labeling system is utilized to create linked tasks to the parent kernel issue to indicate any kernel variant testing that needs to be performed.

The labels used in this process are:

Any future variant added for kernel will follow the same naming convention.

All testing of the RHEL kernel itself will still be tracked on the parent bug/story as usual, whereas kernel variant testing is now tracked tasks linked with an "is blocked by" relationship.

When a parent kernel issue has a linked variant task, KWF will block a kernel MR from merging until the parent ticket and all its linked variant tasks have completed test evaluation. That means:

When a kernel variant task is linked to a parent kernel issue, they will inherit the following fields from the kernel issue:

All of these field synchronizations are one-way from the kernel to its linked dependents. That is to say, any change to the fields above in the kernel ticket will propagate the changes to its linked variant dependents. However, a change to any of these fields in the linked dependents will not sync back to the kernel parent issue.

Tasks have a limited set of fields:

To ensure that kernel MRs are not gated for a prolonged period of time due to the need to synchronize the testing of kernel and any respective variants requiring testing for that MR, then it is requested that only one functional QE owns the kernel main issue and its respective variant tasks.

As discussed in the prior section, Jira automations will ensure that the QA Contact for the parent kernel issue is synchronized to any linked variant tasks as the Assignee.

The assigned QE is the expert in this area of functional expertise and must make the decisions as to what level of testing to perform for the kernel and its variants.

Once a kernel MR merges, the parent kernel ticket continues following the standard RHEL development procedure for final verification and release.

For kernel-rt prior to RHEL 9.3 where it is built as a separate component and shipped in its own advisories, the parent kernel ticket will additionally be attached to the respective kernel-rt advisory. Developers and QE may therefore see two "Errata Link" entries on these pre-9.3 kernel bugs.

For any linked kernel variant tasks, testing has been completed pre-merge and is not strictly required again for final verification. Thanks to the simplified states of the Jira "Task" type, these may remain Closed. However, QE may still wish to conduct variant testing post-merge as well and may do so.

In the event a kernel bug or story is closed with any resolution that is not "Done" (such as "Not a Bug," "Won’t Fix," etc), any existing linked variant tasks will be automatically closed as well by Jira automations, matching the resolution of the parent kernel ticket. Conversely, when a kernel ticket is reopened, any linked variant tasks will also be reopened.

The variant workflow procedure applies for kernel CVEs addressed in RHEL as well.

For RHEL kernel maintainers, note that this procedure means there will no longer be separate kernel-rt CVE bugs to reference in either changelog or "Resolves:" line. Instead, the kernel CVE Jira ID shall be used for both. When a kernel CVE is for a pre-9.3 RHEL stream, there will be two Fixed-In Builds linked to the ticket: one for the kernel build, and one for the kernel-rt build. Internal automations will handle adding this kernel CVE to both kernel and kernel-rt advisories, and customer-facing CVE pages will continue to properly reflect the CVE fix status for both kernel and kernel-rt.