CentOS Stream Kernel Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Bugzilla Lifecycle

For every modification against the RHEL kernel, an approved Bugzilla (or BZ) is required for that change to make it into the release. This document provides an overview of a modification to the RHEL kernel from the perspective of the BZ and how it evolves through the journey.

The status journey on a BZ can be any one of the following values. Note that there are additional status settings, but they are not generally used.

It is not explicitly required that this process is followed in 100% order. Meaning, that a bugfix may be prepared ahead of the BZ being ready, but the overall process cannot be completed until each of the detailed steps below are completed and satisfied.

For guidance around the overall Kernel Workflow process, please visit the Quick Start Guide.

All Status
All Status
All Status
All Status
All Status
All Status
All Status

Step 1. BZ Opened

All Status
All Status
All Status
All Status
All Status
All Status
All Status

As with every process, it must begin somewhere. For Bugzilla and the Kernel Workflow process, this status ends up being NEW. This indicates that the BZ is just that, new. It has not been reviewed, triaged, or considered for fixing or addressing yet.

BZ’s that are in this state will be periodically reviewed by their current default assignee, which is determined by the component and sub-component that was chosen. A BZ may be transferred between multiple components and sub-components until the appropriate destination has been determined.

Every team has their own review process, but common to them all is that once a BZ has been triaged, it will then be moved to ASSIGNED to an individual to review and evaluate the BZ. As part of the triage process, a BZ can also just move directly to the CLOSED status if it is not appropriate to keep open. Reasons could be that the BZ has already been addressed (and therefore would be CLOSEDDUPLICATE) or maybe it’s something else (e.g. user error).

Step 2. BZ Triaged and Scheduled

All Status
All Status
All Status
All Status
All Status
All Status
All Status

Once a BZ has been triaged and ASSIGNED to an individual, they will review the details of it and identify what needs to be done. Assuming a change is required, the BZ will then be scheduled for a particular release, which requires setting a number of internal-only visible fields such as Development Target Milestone, Internal Target Milestone, and Internal Target Release.

In the most basic scenario (before ITM 26, and not for a Z-Stream), a BZ is approved once the release+ flag is granted. For more information, visit the Get Release Approval section in the Quick Start Guide.

Step 3. Change Prepared

All Status
All Status
All Status
All Status
All Status
All Status
All Status

Once a change has been prepared for the kernel, it needs to be made available for review. To indicate that the update is ready to be looked at, the state of the BZ should be changed to POST. Almost along the lines of it being POST-ed to a bulletin board for people to see. This step will be automatically completed by the webhooks once the BZ has been detected in the description.

For the RHEL Kernel Workflow process, the update is provided via the GitLab Merge Request interface.

Step 4. MR / GitLab Process

All Status
All Status
All Status
All Status
All Status
All Status
All Status

All changes to the RHEL kernel are done via the GitLab Merge Request interface. For more detailed information about this process, please visit the Quick Start Guide. As soon as the web hooks in the GitLab infrastructure notices a referenced BZ in the description, a link will be added to the BZ pointing at the relevant Merge Request(s).

The loose process, however, looks like this:

  1. Tag all commits in the Merge Request as specified in the Commit Rules document.

  2. Open MR.

  3. Obtain at least 2 reviews and reviews from the affected code owners.

  4. Verify the Continuous Kernel Integration (CKI) test has run and completed successfully.

  5. Once all but the Bugzilla check labels receives the ::OK status, the readyForQA label will be applied.

    1. This would include checks such as Acks, Dependencies, Signoff, etc.

  6. Work with QE to properly set Verified field in BZ to Tested.

  7. Upon the time where the MR receives the ::OK status on all labels, the readyForQA label is removed and readyForMerge label is applied.

  8. Kernel Maintainer pulls in the Merge Request once the readyForMerge label has been obtained.

Step 5. QE Verifies the fix

All Status
All Status
All Status
All Status
All Status
All Status
All Status

Once the Merge Request has received sufficient reviews and the CKI test has passed, the readyForQA label will be applied to it. It is at this point that the BZ status will be moved from POST to MODIFIED. The assigned QE engineer will verify that the change does what was intended. This is typically done by using the reproduction steps that were noted in the BZ’s description field when it was originally opened. From there, the QE Engineer will set the Verified Field to Tested.

Step 6. Maintainers stage for the next build.

All Status
All Status
All Status
All Status
All Status
All Status
All Status

After the BZ’s Verified field is updated to Tested (or equivalent), the Merge Request should now have the readyForMerge label applied to it. This is the signal to the Kernel Maintainer that the merge request is ready to be pulled into a delivery candidate kernel build. When the update does get pulled in, the affected BZ statuses will be updated to MODIFIED.

Step 7. New kernel is available

All Status
All Status
All Status
All Status
All Status
All Status
All Status

The maintainers assemble a new kernel periodically to deliver the various updates that they were provided. The schedule on this can vary, but when times are busy enough it’s not uncommon to see a new kernel at least once per week. As part of this process, the kernel undergoes its own set of CKI tests just like every other Merge Request. Once the new kernel has gotten past the gating phase, it is tagged in the infrastructure to be used in the next upcoming composes. It is at this point that the BZ will transition from MODIFIED to ON_QA by adding it to an existing errata advisory.

Step 8. QE Performs Final Verification

All Status
All Status
All Status
All Status
All Status
All Status
All Status

Just like in previous steps, when the BZ reaches the ON_QA status, it is a signal to QE that they can perform final verification of the BZ based on the description steps that were provided.A If everything checks out, the QE engineer will mark the BZ as VERIFIED. The BZ will remain in this state until it is released with an update for RHEL.

Step 9. BZ Update is delivered with a release and finally the BZ is Closed

All Status
All Status
All Status
All Status
All Status
All Status
All Status

During the release process, all BZ’s attached to the relevant errata advisories for a particular RHEL release will be closed. The BZ status is then updated to CLOSED and resolution CURRENT_RELEASE. It is at this point that the involvement of the BZ has concluded its responsibilities in relation to RHEL, the RHEL Kernel, and the issues that it addresses.