A Kernel Developer's Guide to kABI

The enabling of CONFIG_MODVERSIONS in a Linux kernel’s configuration enforces compatibility of the Kernel Module Interface (KMI), more commonly referred to as “_module versioning_”. ^{[2] [3]} This causes all kernel global symbols exported with the EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL() macros to be augmented with 8-hex-digit checksums computed from the names of all relevant C tokens that define the interface. ^[4]

The checksumming augmentation – genksyms symbol versioning information – is calculated by expanding the symbol to its full definition where all structures, unions, enums, and typedefs are expanded down to their basic parts, recursively. This final string is then used as input to a CRC algorithm yielding an integer that will change when any constituent part(s) of the symbol’s included definitions change. ^[5]

Note that it’s not just the globally exported symbol’s names themselves that are versioned. Encompassed is everything associated with the symbol’s full prototype: “_all relevant C tokens that essentially define the interface_”.

Whenever there is a change to an exported symbol’s return type, name, any of its argument’s type, or to the type of any field of any structure contained in, or pointed to, by any of its arguments, including all data structures within data structures (recursively), then its checksum changes. When a module attempts loading, each symbol’s CRC value contained in the kernel’s symbol table is compared with the corresponding symbol’s CRC value in the module’s object file, and if they are not equal the kernel refuses to load the module. Thus, kernel interface changes cause the ’insmod’ of previously built modules to fail.

Kernel data structures are of particular interest with respect to KMI as they contain multiple members, some of which may be other data structures themselves, all of which become recursively included as part of the symbol’s versioning checksum, making the total scope of what must be preserved in order for an existing module to continue to load and run, potentially quite large.

As a concrete example, consider the exported symbol ’pci_bus_read_config_word()’ within the kernel’s PCI core that accesses PCI device configuration space. As an exported symbol, it’s available for external modules to call. ^[6]

From its function prototype, the ’int’ return type, its name, and all its arguments, must remain preserved, which is somewhat self-evident. Much more subtle, however, is its ’bus’ argument which is a pointer to a ’struct pci_bus’. ^[7] If, as the result of an upstream patch series incorporation, the type of any member of ’struct pci_bus’, including members of structures within ’struct pci_bus’, were to change – including adding to, removing from, or changing the name or ordering of any of its members – then the ’pci_bus_read_config_word()’ symbol’s versioning checksum would correspondingly change.

Module versioning does not, nor does the Linux kernel, provide a stable kernel interface (a.k.a., a binary kernel interface). ^[8] The KMI is solely a module to kernel consistency checking mechanism for validating external kernel modules with respect to the kernels they are attempting to link (attach) to, and run against.

RHEL’s kABI commitment does not cover all exported symbols but rather is limited to, and thus only concerned with, the subset of global symbols that partners have requested be preserved, and that Red Hat’s kernel development engineers have reviewed, and explicitly agreed to protect. Such symbols become white-listed [see: kABI Lists], which is one of the components used in implementing RHEL’s kABI compliance.

Agreeing to partners’ external symbol requests has significant consequences. More acutely, as a direct result of partners and customers developing and compiling their external modules against a release’s inception (i.e., X.0), kABI’s impacts, and restrictions, begin with its origin, and persist throughout its entire lifecycle.

While RHEL’s kABI is not limited to kernel data structures – like KMI, it focuses on kernel external symbols and their entire prototype’s makeup – they are of particular interest with respect to kABI also. Data structures have all the same impacts to kABI as they do to KMI: ramifications can be very subtle, often outright hidden, yet potentially encompass a very large scope.

Continuing with the example, ’pci_bus_read_config_word()’ is a kernel exported symbol that is white-listed, thus considered protected with respect to kABI.

0x1aa4c8bf pci_bus_read_config_word vmlinux EXPORT_SYMBOL` `^[11]

The kernel’s ’pci_dev’ data structure, via its indirect correlation from the ’bus’ argument (’struct pci_dev’ contains members of type ’struct pci_bus’) of ’pci_bus_read_config_word()’, also becomes effectively kABI protected. ^[12] So: “What does that imply with respect to kABI?”

Consider upstream commit 31c2b8153c ‘‘PCI: Add support for DMA alias quirks’’. As a core aspect of that commit, a new ’dma_alias_devfn’ member was added to ’struct pci_dev’.

Prior to commit 31c2b8153c, ’struct pci_dev’ consumed N bytes of memory, and the ’driver’ member was at some known offset from the structure’s beginning - assume 0x148. After the commit, ’struct pci_dev’ now consumes N + 1 bytes of memory, and the pre-existing ’driver’ member no longer resides at the same 0x148 byte offset location in memory that backs the structure (all ’struct pci_dev’ members at, or past, the ’driver’ member will have been shifted to new, larger, offset locations).

Now consider a partner’s external module which was compiled against RHEL’s 7.0 kernel, and relies upon (i.e., utilizes) the kernel’s ’pci_bus_read_config_word()’ external symbol. If upstream commit 31c2b8153c were back-ported into RHEL in a straightforward manner, say during 7.1’s kernel development phase, changes to the kernel’s version of ’struct pci_dev’ would affect the module in two ways.

First, any module’s references to the kernel’s structure’s members up to, but not including, the ’driver’ member would be fine, as none of their offsets would have changed. However, unbeknownst to the external module, a new ’dma_alias_devfn’ member now resides within the kernel where the ’driver’ member used to in the memory backing ’struct pci_dev’. So, in effect, all member references by modules of the ’driver’ member onward will dereference memory that no longer corresponds to what the module expects. Second, since the structure is effectively larger now, if the module accesses memory immediately after an automatic allocation of the structure (i.e., a routine’s local variable of type ’struct pci_dev’ within the module’s source file) in any manner, those memory dereferences also will not correspond to what the module is expecting. Both conditions – member dereferences at, or past, the newly added member, and the structure’s overall size – would cause external modules to dereference memory now backing other content resulting in silent data corruption at best, and much more likely, outright failure (i.e., a kernel panic), as a result of the kernel’s kABI commitment regarding ’pci_bus_read_config_word()’ having been broken.

The broken kABI commitment is not limited to the ’pci_bus_read_config_word()’ symbol. The change to ’struct pci_dev’ would correspondingly cause all partners external modules that utilized any external symbols with correlations to ’struct pci_dev’ (in any capacity, directly, or indirectly) to no longer successfully link and run against such an updated kernel.

So, not only are kABI protected external symbols effectively locked down with respect to any changes – API permanence – all data structures correlated with such symbols also become entangled and are effectively locked down with respect to any changes, including member additions, deletions, and re-naming or re-arrangement - kernel data structure invariance. In general: kABI protected external symbols, and all kernel data structures that are correlated with such symbols, are effectively locked down with respect to any changes, starting with the symbol becoming protected (which may occur with the release’s origin), and persists throughout the release’s entire lifecycle. ^[13]

Due to the restrictive constraints that kABI imposes onto RHEL, Red Hat does not automatically guarantee 100% compliance of symbol protection requests received. Some requested symbols are purposely rejected, to allow the symbols to change in future minor releases in order to minimize the encumbrances of integrating new, or maturing, technologies from upstream. In such cases, partners may have to recompile their modules for every minor update kernel. ^[14]

While not all types of kABI induced ramifications and repercussions could ever be covered, reviewing and understanding some key examples from past experiences should help developers gather enough knowledge with which to tackle future kABI breakages they encounter on their own. There are cases of frequently occurring issues that are naturally coincident with, and known to be vulnerable to, kABI and kernel development. The most common can be roughly categorized into general areas susceptible to kABI and include: exported symbols, data structures, header files, static inline functions, enumerations, and the /proc (procfs) and /sys (sysfs) filesystems.

RHEL kernel developers must be aware of the conditions that may introduce kABI failures when developing, and back-porting, patches. This is mostly a trial and error process at first but, with time, and experience, there are general situations developers become familiar with, having learned that such types of changes are particularly susceptible to introducing kABI failures.

Within the classes outlined next are references to kABI case studies presented in Appendix B “kABI Case Studies: Methods of Working Around kABI’s Impacts“ - providing specific examples of methods developed for solving specific kABI failure types. It is recommended to read the class’ content first to gain an overall understanding, and context, and then reference the noted, class specific, case studies for further details.

When building a RHEL kernel, either from a Source RPM (SRPM) or via ’rh-brew’, a kABI checking script, ’check-kabi’, is automatically run as part of the build process. ^[19] The script attempts to check for any kABI breakages introduced by changes to the kernel. The checking is constrained to RHEL’s white-listed symbols only; it does not check the kernel’s non white-listed exported symbols

kABI protection applies to all supported architectures. Hence building via ’rh-brew’ is the recommended method, as it will run ’check-kabi’ on all architectures. If ’check-kabi’ is executed manually, only one specific architecture is checked. When building via the ’rpmbuild’ process, the SPEC file sets the ’with_kabichk’ flag – redhat/rpm/SPECS/kernel.spec::%define with_kabichk – enacting the exact same ’check-kabi’ based checking.

To manually check if there is going to be a kABI issue, prepare by first enacting the patch series’ changes, and make the kernel:

With the kernel built, manually run ’check-kabi’ to check for kABI failures:

If there is no output, then the build was successful and there were no kABI related failures. Otherwise, ’check-kabi’ outputs a list of broken symbols.

Removal or change to white-listed symbols is considered a serious offense as partners and customers are actively using these symbols. Changing a white-listed symbol’s checksum would cause their modules to fail loading in future minor updates, negating Red Hat’s commitment. Red Hat has removed white-listed symbols in the past, but only after contacting partners and securing knowledge they were not using those symbols.

Developers will, on occasion, include a patch that incidentally modifies a white-listed symbol. This will lead to the checksum changing and is considered “_breaking kABI_”. In such cases, developers are expected to modify their patches to accommodate these symbols and patches will be rejected unless they are revised to work around such breakages.

Exported symbols that Red Hat has committed to not break throughout the subsequent lifecycle of a RHEL major release make up the “white-list”. ^[21]

Each minor RHEL release has a set of symbols constituting a white-list. ^[22] Once a symbol has been added to the kABI white-list for a particular release it will not be removed, nor is it allowed to change during the major release’s remaining lifecycle. New symbols may be added to the kABI white-list in minor releases.

Exported symbols that Red Hat’s partners have requested for review and addition to the while-list but have not yet been explicitly accepted nor rejected (e.g., they are in the internal review phase where Red Hat’s developers discuss whether or not to add them to the white-list).

There is a database server that maintains grey-listed entries for each specific RH partner. This server also offers a monitoring service that notifies partners if any of the symbols on their respective grey-list changes.

Partners can request additions to RHEL’s kABI white-list by opening a RHEL Jira under the “kernel / kabi-whitelists” component. Symbol requests are sent to the Red Hat Kernel development team’s Mail List (RHKML) for internal review. Symbols explicitly accepted for protection are added to the kABI white-list at the end of the next minor release’s development phase. Symbols that were not explicitly accepted are automatically added to the grey-list.

Upstream reference: N/A

RHEL7 reference: commit c9be03bbc1c ^[23] PCI: Add kABI shadow structs for the kernel’s PCI subsystem c9be03bbc1c PCI: Create ’pci_dev_rh’ KABI shadowing structure 9599688156b PCI: Convert ’pci_dev’ allocations and frees 5abdc3d2dac PCI: Convert specific ’pci_dev’ related constructs 829d829c1b7 PCI: Create ’pci_bus_rh’ KABI shadowing structure 8af377d9592 PCI: Convert ’pci_bus’ allocations and frees 610b1c95049 PCI: Create ’pci_driver_rh’ KABI shadowing structure]

This case study covers the pre-requisite setup that enables logically adding new members to existing kernel data structures that are becoming, or expected to become, encumbered with kABI-protected symbols. While specific implementations have evolved over the years – hash based associations to shadow structures, directly associated opaque extension structures, and pre-reserved additions with directly associated opaque extension structures – its central tenant – adding a new data structure that is directly associated to an existing data structure unto which all future member additions will be added so as to not effect any symbol versioning aspects of the original data structure – has not.

During RHEL7’s initial kernel development phase – 7.0 – commit c9be03bbc1c “PCI: Create ’pci_dev_rh’ KABI shadowing structure” was pro-actively put into place adding a pointer to an opaque, RHEL kABI-specific, extension structure onto the end of ’struct pci_dev’.[23] This pre-requisite step allowed RHEL7 to maintain kABI with respect to ’pci_dev’ yet also accommodates logical upstream updates to such – the addition of new members only - via its ’pci_dev_rh’ extension structure.

The RHEL specific extension member addition to existing kernel structures – in this instance, a pointer to an opaque extension structure – must be put into place prior to, or coincident with, the correlated symbol becoming protected. This assures that any external modules will have always been compiled against a kernel that has the extension’s member included, prior to the symbol having become protected, and thus relevant symbol version checksums will not have changed.

Once a symbol becomes protected, any encumbered data structures are effectively locked down from any further changes for the remaining life cycle of the release. Otherwise, if data structures were altered going forward, an external module built against a prior kernel, would cease to link successfully due to Red Hat not having maintained its kABI commitment.

The implication is that a symbol may become protected, and its encumbered data structures thus locked down, with the release’s origin (i.e., RHEL X.0). In fact, this is fairly common occurrence for the kernel’s foundational data structures as they are the most likely to become encumbered with symbols becoming protected. Therefore, pro-actively implementing extension members for key kernel data structures during the release’s initial phase is a prudent action that subsystem maintainers should strongly consider.

The subsequent step of allocating the extension structure itself can be postponed until such time as needing to effectively add a new member to the existing structure occurs [see: Case Study: Adding Members to Centrally Allocated Kernel Data Structures and Case Study: Adding Members to Non-Centrally Allocated Kernel Data Structures].

Upstream reference: commit 338c3149a22

RHEL7 back-port ref: commit 8e7c115586d ^[24]

The case study involves adding new members to existing kernel data structures. Its focus is the specific case where all allocations of the existing structure needing augmented are performed by a single, centralized, in-kernel-core routine. There are multiple aspects involved to making this technique work without breaking kABI. This technique requires a pre-requisite setup step [see: Case Study: Pro-actively Extending Core Kernel Data Structures] that itself must be fully comprehended before proceeding.

During RHEL 7.3’s development phase, upstream commit 338c3149a22 “PCI: Add support for multiple DMA aliases” was needed. The new feature was dependent upon a new ’dma_alias_mask’ member being added to ’struct pci_dev’.

The ’pci_dev’ data structure from the kernel’s PCI subsystem is foundational to its subsystem – thus is altered by upstream changes on a consistent basis.[7] Additionally, being such a core data structure, there are numerous kABI-protected symbols within PCI’s subsystem that reference it, so any changes to it would normally break kABI for many symbols. Fortunately, ’struct pci_dev’ meets very strict requirements that allows new members to be logically added to it: it is strictly allocated from the kernel’s heap via a common function - ’pci_alloc_dev()’. ^[25]

If a data structure is strictly allocated from the kernel’s heap (i.e., no static definitions of the structure are allowed, nor are automatic allocations on the stack) then there is a technique that can be used for adding new members to it without breaking kABI. ^[26]

Intuitive, or experienced, developers will eventually ask: “What prevents an external driver module from allocating a ’struct pci_dev’ within itself, either statically (within the source file’s scope) or automatically (as an automatic variable within a routine; thus resides on the routine’s stack)?” ^[27] Nothing, however if one researches, via ’cscope(1)’ or some similar tool, all ’struct pci_dev’ allocations within the kernel and it’s built-in modules, they will see that all allocations of ’pci_dev’ utilize ’pci_alloc_dev()’. This is where kABI enters a “grey area”: nothing prevents an external driver module from doing such, but, it would be considered ill formed; improper programming by a module is not considered a kABI breakage.

With the aforementioned setup step – the extension mechanism from commit c9be03bbc1c – in place, the subsystem’s common allocation related functions next need to be augmented to include allocating the additional extension structure along side its parent. While it makes sense to do this coincident with the addition of the extension structure in this particular case – where allocations are handled by a common in-kernel-core function within the subsystem – it is not absolutely necessary to perform this specific step until a specific need arises as is covered in a different case study [see: Case Study: Adding Members to Non-Centrally Allocated Kernel Data Structures]. This was accomplished via RHEL7 commit 9599688156b “PCI: Convert ’pci_dev’ allocations and frees”.[23]

With an opaque, RHEL specific extension structure now in place, and fully backed (allocated), future upstream changes to the structure which would normally break kABI can now be brought into RHEL by placing new members into the corresponding extension structure such as was done in this case study’s RHEL7 commit - 8e7c115586d “PCI: Add support for multiple DMA aliases”.

Any existing external module references to members of ’struct pci_dev’ will remain valid. Yes, there will effectively be new members to ’struct pci_dev’, placed into the extension structure; however, external modules compiled against any earlier RHEL kernel will not be aware of such additions and so will never attempt to access the member or its backing memory.

With a solution in place that is understood not to logically break existing external modules, the change would still end up altering all symbols with associations to ’struct pci_dev’. The final piece of the overall solution is to use RHEL’s ’RH_KABI_EXTEND()’ macro to circumvent GENKSYMS based checking from detecting the change.[18]

Upstream reference: commit 2d984ad132a

RHEL7 back-port reference: commit bef916144ab ^[28]

This case study is similar to Case Study: Adding Members to Centrally Allocated Kernel Data Structures in that it is concerned with adding new members to an existing kernel data structure, in this case ’struct device’. ^[29]What differs, however, is that the existing kernel data structure does not have a centralized allocation routine with which to collocate the extension’s allocation, introducing an additional consideration that has to be taken into account, making things quite a bit more complex, and worth specifically looking at.

The same, pre-requisite, associated extension structure setup is necessary and took place during RHEL7’s initial kernel development phase, via commit 9f84482488d “device: Create ’device_rh’ KABI shadowing structure”, which put a pointer to a RHEL specific kABI extension structure, ’struct device_rh’, onto the end of ’struct device’. ^[30] driver-core: Add KABI shadow structs to kernel’s driver subsys 3ba6b481780 device: Create ’device_driver_rh’ KABI shadowing structure 9f84482488d device: Create ’device_rh’ KABI shadowing structure]

The ’device’ structure, from the kernel’s device driver subsystem, underlies every device that runs on Linux. The sequence that device drivers typically use with respect to getting a device initialized – device registration – is to first obtain an allocated ’device’ structure, typically via inheritance from a device specific subsystem, and then, at some later time, call ’device_add()’ which adds the device entry to the kernel with reference counting. As such, ’struct device’ does not have a centralized function for its allocation; allocations are distributed throughout the kernel.

Distributed allocations of ’device’ structures, typically from within each of the kernel’s specific device related subsystem’s themselves, introduces an additional consideration that must be taken into account: “Where is a good place to add the allocation of the corresponding kABI extension structure?” When the need to add a member to RHEL’s ’device’ structure finally arose – during the kernel development phase of RHEL 7.2 – commit 928a66966c2 “device: allocate/free Red Hat only struct device_rh” took the approach of not collocating RHEL’s extension structure allocations with their respective ’device’ structure allocations, but instead, to augment the ’device_add()’ function to perform the extension’s allocation.

As device registration first obtains an allocated ’device’ structure and then, some separate time later, calls ’device_add()’, there is a window of time in which ’struct device’ exists, and can be accessed, prior to device registration’s completion via ’device_add()’, and there are driver, and kernel, paths that access members of ’struct device’ within this window. This is not an issue for upstream, as all accesses obviously occur after the allocation. A consequence of the approach taken in this case for RHEL, however, is accesses within this window that, now by extension, reference ’struct device_rh’. Such accesses cause the kernel to panic de-referencing a NULL pointer with RHEL as a consequence of ’struct device_rh’ not yet having been allocated. As such, some code paths need the extension structure allocated prior to ’device_add()’ being called, which subsequently requires extra checking within RHEL’s customized ’device_add()’ routine to see if ’struct device_rh’ has already been allocated or not. This was covered with commit ef52e4a544c “device: add device_rh_alloc()”.

The default extension related allocation remains within ’device_add()’; however, extension allocations necessary prior to such now have to be enacted throughout the kernel in a somewhat ad hoc, case-by-case, manner; usually upon each occurrence of a de-referencing panic (along with the additional work of root cause analysis and re-working of the path’s specific, allocation related, need). ^{[31] [32]}

From here on out the case study is similar to Case Study: Adding Members to Centrally Allocated Kernel Data Structures with the additional caveat that additional latent issues yet uncovered may yet occur and need addressed as outlined.

Upstream reference: commit 9d26d3a8f1b

RHEL7 back-port reference: commit a040fb72b14 ^[33]

This case study covers the easiest approach for adding members to existing kernel data structures without breaking kABI. As the study’s title suggests, it involves exploiting existing holes within kernel data structures. Kernel developers typically take data structure member alignment and padding aspects into consideration during development, attempting to both align, and pack, structures as efficiently, and tightly, as possible. ^[34] Even so, holes within structures sometimes exist.

If an existing hole can be found within a protected kernel data structure that, including all alignment concerns, is equal to or greater than the size of a new member needing to be added, the hole can be exploited for adding the member while still maintaining kABI.

An upstream patch, targeted for back-porting into RHEL, added a new ’bridge_d3’ member to ’struct pci_dev’.[7]

The ’pci_dev’ structure is the base data structure of the kernel’s PCI subsystem, and as such, there are numerous kABI-protected symbols with associations to it. A straight back-port would break kABI due to its side-effect of changing the version checksums of all symbols ’pci_dev’ is correlated with.

The new member is a single bit bit-field declaration, and with careful inspection of RHEL’s ’struct pci_dev’ (RHEL 7’s structure varies from upstream’s), there looked to be a “hole” between the end of the previously existing bit-fields and the ’d3_delay’ member. Running the ’pahole(1)’ utility against ’struct pci_dev’ confirmed that a 14-bit hole existed:

RHEL7 Patch 09/23 “PCI: Put PCIe ports into D3 during suspend” used just such an exploitation. During the back-port, the new member was inserted within the existing hole. As the hole already existed, both the size of the overall structure, and the offsets of all existing members within the structure, were not altered as long as the insertion was performed at the end of the bit fields. Incidentally, any existing external modules would not know about the newly introduced member, thus should never attempt to access that part of the structure’s memory.

While the solution is understood not to logically effect existing external modules, the change would still end up altering all symbols with associations to ’struct pci_dev’. The final piece of the overall solution is to use RHEL’s ’RH_KABI_FILL_HOLE()’ macro to specifically circumvent GENKSYMS based checking from detecting the change.[18]

Upstream reference: commit 589fcc23074

RHEL7 back-port reference: commit 0c9beb71177 ^[35] Re-base PCI subsystem with v3.18 content 0c9beb71177 [RHEL7.3 PATCH 15/16] PCI: Move pci_configure_slot() to drivers/pci/probe.c]

There is an interesting situation concerning forward declarations of kernel data structures that yields “false kABI failures” from the checking tools. The scenario typically involves header files in which prior to the patch series’ changes, a source file only knows about a specific data structure via an incomplete forward declaration (i.e., struct pci_dev;). Then, as a direct result of the patch including an additional header file, the source file now knows the structure’s full declaration (i.e., struct pci_dev { …​ };).

Basically what has occurred with respect to the source file is that what was an opaque reference to a structure subsequently becomes non-opaque due to the new header file inclusion that adds – directly, or indirectly – the structure’s full declaration.

Including header files may change the parsing of structures, which can result in a false-positive kABI failure.

These situations can be particularly hard to track down and isolate as the root of the kABI failure.

During RHEL7.3’s kernel development phase, commit 0c9beb71177 "` PCI: Move pci_configure_slot() to drivers/pci/probe.c`" was pro-actively back-ported, moving a function from one source file to another.[35] As a necessary result of the move, an additional header file was included within the destination’s source file.

What was particularly interesting about this circumstance was that RHEL’s build related kABI checking did not flag a kABI failure whereas the ’ksc(1)’ kABI checking tool, which was used by a partner, indicated that the following symbol’s kABI checksums changed and affects (breaks) their modules, leading them to request for the symbols to become protected:

These specific symbols were not kABI white-listed (thus why this situation was not caught during the back-port’s kernel build related checking); however, the pci_read/write_config functions are really just wrappers around pci_bus_read/write_config_ which are kABI white-listed. ^[36]

This specific instance’s circumstances are unique. As the symbols were not white-listed, RHEL kernel developers were very reluctant to add them as the partner had requested as this could be a slippery slope that leading to RHEL being requested to preserve all the kernel’s exported symbols.

Regardless of the specific circumstances, once isolated, these types of false-failures are generally fairly easy to rectify as they can be worked around by the use of the GENKSYMS pre-processing directive.

For this specific case, it exposed a bug in the ’ksc(1)’ tool, so RHEL’s kernel development team acquiesced, enacting the easy fix to restore the original checksums, but in a manner explicitly not adding the symbols to the white-list, via RHEL7 commit 0c348a6cd87. ^[37]

This same technique can often be used to work around false kABI failures with similar – indirect forward declarations of data structures becoming full forward declarations – scenarios.

Upstream reference: commit 95e300c052f

RHEL7 back-port reference: commit a57906f6a8f ^[38] ipmi: Make the DMI probe into a generic platform probe]

Changes to enumeration ordering can affect external modules. RHEL’s current kABI checking capabilities may not catch simple enumeration related changes that indeed affect modules, leading to silent kABI failures.

Note how RHEL7.6 commit a57906f6a8f “IPMI: Make the DMI probe into a generic platform probe” was able to maintain the values of previously existing enumerations to preserve kABI.

Fortunately, updates of enumerations are fairly easy to adhere to kABI’s requirements. The most common tactic is to convert un-assigned enumeration lists into explicitly assigned lists, and then if adding new entries is needed, do so in a manner that does not affect any previous entries (i.e., if a new, upstream, enumeration was added within an existing list, keep the list ordered as it was and add the new entry to the end of the list instead).

With respect to substitutions (exchanges) using the pre-reserved entries:

With respect to substitutions (exchanges) using the “extension” structure: