CentOS 7.4 & Realtek Wireless issues

Updated: June 2, 2018

Several weeks ago, I upgraded my CentOS instance, forming a part in the eight-boot setup on the Lenovo G50 laptop, to the latest release, version 7.4. Remember? Well, everything was peachy, including the networking. Then, a couple of reboots later (and more importantly, full system power offs), CentOS 7.4 would no longer recognize the Wireless interface! The Realtek RTL8723BE module was not being loaded anymore!

I decided to explore and troubleshoot this in detail, because I hate seemingly random issues, especially since it worked once, it should continue working. Intermittent problems are the worst. Let's see if we can understand the problem and fix it. After me.

Problem in more detail

If you recall my original test on the G50 machine, CentOS did not have the right network drivers, and I had to manually load the Realtek rpm (from EPEL), and then use the repo for additional pimping. But other than this one time effort to get underway, everything was fine. Even CentOS 7.4 worked without a hitch until suddenly it did not.

What does the log say

Looking at the system logs, we have:

rtl8723_common: loading out-of-tree module taints kernel.
WARNING: module 'rtl8723_common' built without retpoline-enabled compiler, may affect Spectre v2 mitigation
rtl8723_common: module verification failed: signature and/or required key missing - tainting kernel
rtl8723be: disagrees about version of symbol rtl8723_phy_set_bb_reg
rtl8723be: Unknown symbol rtl8723_phy_set_bb_reg (err -22)

Very interesting. First, we have a mention of the Spectre mitigations, but it's only a warning. The real problem is with an unknown symbol, which prevents the module from being loaded into the kernel. Reading online, I did find a partial reference (annoyingly, the full explanation is for RHEL customers only) that this is actually related to the retpoline patches, as part of the bigger Spectre thingie, which prevents compilation as well as loading of kernel modules. If this is indeed correct, this would be the first instance (that I've encountered) of the Spectre mitigation interfering with the system.

So I decided to try several things, to verify whether the issue is related to the patches, and if somehow I could work around this, by manually compiling a new version of the RTL8723BE driver, and loading it manually. But before all that, I rebooted into an older kernel, and indeed everything was fine. The Wireless driver was loading and working without any problems. This is definitely related to the kernel 693 revision. I do not know why it was triggered with a system power off rather than just a reboot - probably because of the associated firmware, much like say turning VT on or off in the UEFI/BIOS - but let's focus on the solution.

Solution attempt 1

I uninstalled the Intel microcode package, rebooted, no luck. This did not change anything. I believe the firmware patch is only effective when used in conjunction with a BIOS update, and I did not use that on the G50 system. Moreover, the actual compilation capability is related to kernel patches, and not the firmware itself. Still.

Solution attempt 2

Speaking of compilations, I decided to try to compile the RTL family of drivers. The drivers for Realtek devices are available under a single bundle on GitHub, and I've already used this source several times in the past, dealing with different Linux-Realtek issues over the years. When trying to compile from the master tree, the following error vomit showed up on the command line:

make[1]: Entering directory `/usr/src/kernels/3.10.0-693.21.1.el7.x86_64'
CC [M]  /home/roger/Downloads/rtlwifi_new-master/pci.o
In file included from /home/roger/Downloads/rtlwifi_new-master/pci.c:30:0:
/home/roger/Downloads/rtlwifi_new-master/wifi.h:45:27: error: ‘IEEE80211_NUM_BANDS’ undeclared here (not in a function)

I thought that this might be specifically limited to one of the other drivers that I did not need, so I stepped into the rtl8723be sub-directory, and tried to compile only this particular module. Alas, the Makefile inside is virtually empty, and it does not have any target. I had to manually create my own Makefile. BTW, if you are planning on similar endeavors, then here's the content, should you wish to replicate - please note that all indentations should be tab stops (if you copy & paste from here).

SHELL := /bin/sh
CC = gcc
KVER  ?= $(shell uname -r)
KSRC := /lib/modules/$(KVER)/build
PWD := $(shell pwd)
CLR_MODULE_FILES := *.mod.c *.mod *.o .*.cmd *.ko *~ .tmp_versions* modules.order Module.symvers
SYMBOL_FILE := Module.symvers

ifneq ("","$(wildcard /lib/modules/$(KVER)/kernel/drivers/net/wireless/realtek)")
MODDESTDIR := /lib/modules/$(KVER)/kernel/drivers/net/wireless/realtek/rtlwifi
else
MODDESTDIR := /lib/modules/$(KVER)/kernel/drivers/net/wireless/rtlwifi
endif

rtl8723be-objs :=        \
        dm.o        \
        fw.o        \
        hw.o        \
        led.o        \
        phy.o        \
        pwrseq.o    \
        pwrseqcmd.o    \
        rf.o        \
        sw.o        \
        table.o        \
        trx.o        \

obj-m += rtl8723be.o

all:
    $(MAKE) -C $(KSRC) M=$(PWD) modules
install: all

    @mkdir -p $(MODDESTDIR)/rtl8723be
    @install -p -D -m 644 rtlwifi.ko $(MODDESTDIR)
    @install -p -D -m 644 ./rtl8723be/rtl8723be.ko $(MODDESTDIR)/rtl8723be

    @depmod -a $(KVER)

ccflags-y += -D__CHECK_ENDIAN__

There were a few glitches with the Makefile - spaces instead of tabs, and all that, but eventually I had the Makefile all nice and dandy, and I could proceed with the compilation. For a moment, it looked as if this was going to work, but then it did not.

make[1]: Entering directory `/usr/src/kernels/3.10.0-693.21.1.el7.x86_64'
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/dm.o
In file included from /home/roger/Downloads/rtlwifi_new-master/rtl8723be/dm.c:26:0:
/home/roger/Downloads/rtlwifi_new-master/rtl8723be/../wifi.h:45:27: error: ‘IEEE80211_NUM_BANDS’ undeclared here (not in a function)
#define NUM_NL80211_BANDS IEEE80211_NUM_BANDS

However, on an older kernel (327), it works fine:

make all
make -C /lib/modules/3.10.0-327.28.2.el7.x86_64/build M=/home/roger/Downloads/rtlwifi_new-master/rtl8723be modules
make[1]: Entering directory `/usr/src/kernels/3.10.0-327.28.2.el7.x86_64'
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/dm.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/fw.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/hw.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/led.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/phy.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/pwrseq.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/pwrseqcmd.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/rf.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/sw.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/table.o
CC [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/trx.o
/home/roger/Downloads/rtlwifi_new-master/rtl8723be/trx.c: In function ‘rtl8723be_rx_query_desc’:
/home/roger/Downloads/rtlwifi_new-master/rtl8723be/trx.c:392:3: warning: passing argument 1 of ‘ieee80211_is_robust_mgmt_frame’ from incompatible pointer type [enabled by default]
if ((!_ieee80211_is_robust_mgmt_frame(hdr)) &&
^
In file included from include/net/mac80211.h:21:0,
from /home/roger/Downloads/rtlwifi_new-master/rtl8723be/../wifi.h:36,
from /home/roger/Downloads/rtlwifi_new-master/rtl8723be/trx.c:26:
include/linux/ieee80211.h:2410:20: note: expected ‘struct sk_buff *’ but argument is of type ‘struct ieee80211_hdr *’
static inline bool ieee80211_is_robust_mgmt_frame(struct sk_buff *skb)
^
LD [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.o
Building modules, stage 2.
MODPOST 1 modules
WARNING: "efuse_power_switch" [/home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko] undefined!
WARNING: "rtl_get_hwinfo" [/home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko] undefined!
WARNING: "rtl_tx_report_handler" [/home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko] undefined!
WARNING: "rtl_c2hcmd_enqueue" [/home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko] undefined!
WARNING: "rtl_get_tx_report" [/home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko] undefined!
CC      /home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.mod.o
LD [M]  /home/roger/Downloads/rtlwifi_new-master/rtl8723be/rtl8723be.ko
make[1]: Leaving directory `/usr/src/kernels/3.10.0-327.28.2.el7.x86_64'

On a side note, I tried finding new/updated Realtek drivers from three different third-party repos (ELREPO, EPEL and RPM Fusion), but none of that seemed to help. And so it seems, there's nothing we could do here, with this approach.

Solution attempt 3

In the CentOS 7.4 article, I also wrote I intended to test new 4.x kernels. Well, constrained by this silly and pointless issue, which renders CentOS into an inexcusable piece of junk on the Lenovo laptop - but mind, it's a server distro, not intended for home use in the first place, there's nothing to lose by going all the way.

Short answer: this worked majestically.

However, you will hate me now, but the full details and all that, that's going to be a separate article. It's not a trivial thing, so bear with me.

Conclusion

Well, if you have a system with a Realtek Wireless network card, and CentOS does not have the drivers for your machine out of the box, then you should avoid upgrading to the post-Spectre kernel 3.10.x, as it contains patches that may break your existing modules as well as prevent you from compiling. In other words, this renders CentOS useless - perfect from the security perspective, but then security is overrated, even more so when it conflicts with functionality. There's no point to a secure system that cannot be used. But then, you're left without an upgrade path really.

I deliberately chose not to disclose the last part of my solution journey in this article as it veers completely off the CentOS path. Even with all the modifications and pimping that I've included, it was/is still CentOS proper. Adding a non-standard kernel completely breaks the model. That's not bad, especially considering this is a server distro, so any home usage is a miracle of a sort, but that's a different story altogether. We will do that, I promise, and we will see how the new (mainline) kernel allows more home-use flexibility that was not factored into the server nature of the RHEL/CentOS line. For now, while I did fail in giving you the details of the solution, there's something to look forward in the coming weeks. Stay tuned.

Cheers.