hansdegoede | Entries tagged with fedora

I have been working on getting the camera on the ThinkPad X1 Carbon Gen 12 to work under Fedora.

This requires 3 things:

Some ov08x40 sensor patches, these are available as downstream cherry-picks in Fedora kernels >= 6.12.13
A small pipewire fix to avoid WirePlumber listing a bunch of bogus extra "ipu6" Video Sources, these fixes are available in Fedora's pipewire packages >= 1.2.7-4
I2C and GPIO drivers for the new Lattice USB IO-expander, these drivers are not available in the upstream / mainline kernel yet

I have also rebased the out of tree IPU6 ISP and proprietary userspace stack in rpmfusion and I have integrated the USBIO drivers into the intel-ipu6-kmod package. So for now getting the cameras to work on the X1 Carbon Gen 12 requires installing the out of tree drivers through rpmfusion. Follow these instructions to enable rpmfusion, you need both the free and nonfree repos.

Then make sure you have a new enough kernel installed and install the rpmfusion akmod for the USBIO drivers:

sudo dnf update 'kernel*'
sudo dnf install akmod-intel-ipu6

The latest version of the out of tree IPU6 ISP driver can co-exist with the mainline / upstream IPU6 CSI receiver kernel driver. So both the libcamera software ISP FOSS stack and Intel's proprietary stack can co-exist now. If you do not want to use the proprietary stack you can disable it by running 'sudo ipu6-driver-select foss'.

After installing the kmod package reboot and then in Firefox go to Mozilla's webrtc test page and click on the "Camera" button, you should now get a camera permisson dialog with 2 cameras: "Built in Front Camera" and "Intel MIPI Camera (V4L2)" the "Built in Front Camera" is the FOSS stack and the "Intel MIPI Camera (V4L2)" is the proprietary stack. Note the FOSS stack will show a strongly zoomed in (cropped) image, this is caused by the GUM test-page, in e.g. google-meet this will not be the case.

I have also been making progress with some of the other open IPU6 issues:

Camera's failing on Dell XPS laptops due to iVSC errors (rhbz#2316918, rhbz#2324683) after a long debugging session this is finally fixed, the fix for this will be available in Fedora kernels >= 6.13.4 which should show up in updates-testing today
Camera's no working on Microsoft Surface book with ov7251 sensor, the fix for this has landed upstream

The initial IPU6 camera support landed in Fedora 41 only works on a limited set of laptops. The reason for this is that with MIPI cameras every different sensor and glue-chip like IO-expanders needs to be supported separately.

I have been working on making the camera work on more laptop models. After receiving and sending many emails and blog post comments about this I have started filing Fedora bugzilla issues on a per sensor and/or laptop-model basis to be able to properly keep track of all the work.

Currently the following issues are being either actively being worked on, or are being tracked to be fixed in the future.

Issues which have fixes pending (review) upstream:

IPU6 camera on TERRA PAD 1262 V2 not working, fix has been accepted upstream.
IPU6 camera on Dell XPS 9x40 models with ov02c10 sensor not working, sensor driver has been submitted upstream.

Open issues with various states of progress:

See all the individual bugs for more details. I plan to post semi-regular status updates on this on my blog.

This above list of issues can also be found on my Fedora 42 change proposal tracking this and I intent to keep an updated complete list of all x86 MIPI camera issues (including closed ones) there.

Unfortunately an incomplete backport of IPU6 DMA handling changes has landed in kernel 6.11.11.

This not only causes IPU6 cameras to not work, this causes the kernel to (often?) crash on boot on systems where the IPU6 is in use and thus enabled by the BIOS.

Kernels 6.12.2 - 6.12.4 are also affected by this. A fix for this is pending for the upcoming 6.12.5 release.

6.11.11 is the last stable release in the 6.11.y series, so there will be no new stable 6.11.y release with a fix.

As a workaround users affected by this can stay with 6.11.10 or 6.12.1 until 6.12.5 is available in your distributions updates(-testing) repository.

I'm happy to announce that the last tweaks have landed and that the fully FOSS libcamera software ISP based IPU6 camera support in Fedora 41 now has no known bugs left. See the Changes page for testing instructions.

Supported hardware

Unlike USB UVC cameras where all cameras work with a single kernel driver, MIPI cameras like the Intel IPU6 cameras require multiple drivers. The IPU6 input-system CSI receiver driver is common to all laptops with an IPU6 camera, but different laptops use different camera sensors and each sensor needs its own driver and then there are glue ICs like the LJCA USB IO-expander and the iVSC (Intel Visual Sensing Controller) and there also is the ipu-bridge code which translates Windows oriented ACPI tables with sensor info into the fwnodes which the Linux drivers expect.

This means that even though IPU6 support has landed in Fedora 41 not all laptops with an IPU6 camera will work. Currently the IPU6 integrated in the following CPU models works if the sensor + glue hw/sw is also supported:

Tiger Lake
Alder Lake
Raptor Lake

Jasper Lake and Meteor Lake also have an IPU6 but there is some more integration work necessary to get things to work there. Getting Meteor Lake IPU6 cameras to work is high on my TODO list.

The mainline kernel IPU6 CSI receiver + libcamera software ISP has been successfully tested on the following models:

Various Lenovo ThinkPad models with ov2740 (INT3474) sensor (1)
Various Dell models with ov01a10 (OVTI01A0) sensor
Dell XPS 13 PLus with ov13b10 (OVTIDB10/OVTI13B1)
Some HP laptops with hi556 sensor (INT3537)

To see which sensor your laptop has run: "ls /sys/bus/i2c/devices" this will show e.g. "i2c-INT3474:00" if you have an ov2740, with INT3474 being the ACPI Hardware ID (HID) for the sensor. See here for a list of currently known HID to sensor mappings. Note not all of these have upstream drivers yet. In that cases chances are that there might be a sensor driver for your sensor here.

We could really use help with people submitting drivers from there upstream. So if you have a laptop with a sensor which is not in the mainline but is available there, you know a bit of C-programming and you are willing to help, then please drop me an email so that we can work together to get the driver upstream.

1) on some ThinkPads the ov2740 sensor fails to start streaming most of the time. I plan to look into this next week and hopefully I can come up with a fix.

MIPI camera Integration work done for Fedora 41

After landing the kernel IPU6 CSI receiver and libcamera software ISP support upstream early in the Fedora 41 cycle, there still was a lot of work to do with regards to integrating this into the rest of the stack so that the cameras can actually be used outside of the qcam test app.

The whole stack looks like this "kernel → libcamera → pipewire | pipewire-camera-consuming-app". Where the 2 currently supported pipewire-camera consuming apps are Firefox and GNOME Snapshot.

Once this was all up and running testing found quite a few bugs which have all been fixed now:

Firefox showing 13 different cameras in its camera selection pulldown for a single IPU6 camera (fix).
Installing pipewire-plugin-libcamera leads to UVC cameras being powered on all the time causing significant battery drain (bug, bug, discussion, fix).
Pipewire does not always recognizes cameras on login (bug, bug, bug, fix).
Pipewire fails to show cameras with relative controls (fix).
spa_libcamera_buffer_recycle sometimes fails, causing stream to freeze on first frame (bug, fix)
Firefox chooses bad default resolution of 640x480. I worked with Jan Grulich to get this fixed and this is fixed as of firefox-130.0.1-3.fc41. Thank you Jan!
Snapshot prefers 4:3 mode, e.g. 1280x1080 on 16:9 camera sensors capable of 1920x1080 (pending fix)
Added intel-vsc-firmware, pipewire-plugin-libcamera, libcamera-ipa to the Fedora 41 Workstation default package-set (pull, pull, pull)

Recently there have been a number of reports (bug 2183743, bug 2276698, bug 2283839, bug 2312355) about the plymouth boot splash not showing properly on PCs using AMD GPUs.

The problem without plymouth and AMD GPUs is that the amdgpu driver is a really really big driver, which easily takes up to 10 seconds to load on older PCs. The delay caused by this may cause plymouth to timeout while waiting for the GPU to be initialized, causing it to fallback to the 3 dot text-mode boot splash.

There are 2 workaround for this depending on the PCs configuration:

1. With older AMD GPUs the radeon driver is actually used to drive the GPU but even though it is unused the amdgpu driver still loads slowing things down.

To check if this is the case for your PC start a terminal in a graphical login session and run: "lsmod | grep -E '^radeon|^amdgpu'" this will output something like this:

amdgpu 17829888 0
radeon 2371584 37

The second number after each is the usage count. As you can see in this example the amdgpu driver is not used. In this case you can disable the loading of the amdgpu driver by adding "modprobe.blacklist=amdgpu" to your kernel commandline:

sudo grubby --update-kernel=ALL --args="modprobe.blacklist=amdgpu"

2. If the amdgpu driver is actually used on your PC then plymouth not showing can be worked around by telling plymouth to use the simpledrm drm/kms device created from the EFI framebuffer early on boot, rather then waiting for the real GPU driver to load. Note this depends on your PC booting in EFI mode. To do this run:

sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm"

After using 1 of these workarounds plymouth should show normally again on boot (and booting should be a bit faster).

The firmware which drm/kms drivers need is becoming bigger and bigger and there is a push to move to generating a generic initramfs on distro's builders and signing the initramfs with the distro's keys for security reasons. When targetting desktops/laptops (as opposed to VMs) this means including firmware for all possible GPUs which leads to a very big initramfs.

This has made me think about dropping the GPU drivers from the initramfs and instead make plymouth work well/better with simpledrm (on top of efifb). A while ago I discussed making this change for Fedora with the Red Hat graphics team spoiler: For now nothing is going to change.

Let me repeat that: For now there are no plans to implement this idea so if you believe you would be impacted by such a change: Nothing is going to change.

Still this is something worthwhile to explore further.

Advantages:

1. Smaller initramfs size:

* E.g. a host specific initramfs with amdgpu goes down from 40MB to 20MB
* No longer need to worry about Nvidia GSP firmware size in initrd
* This should also significantly shrink the initrd used in liveimages

2. Faster boot times:

* Loading + unpacking the initrd can take a surprising amount of time. E.g. on my old AMD64 embedded PC (with BobCat cores) the reduction of 40MB -> 20MB in initrd size shaves approx. 3 seconds of initrd load time + 0.6s seconds from the time it takes to unpack the initrd
* Probing drm connectors can be slow and plymouth blocks the initrd -> rootfs transition while it is busy probing

3. Earlier showing of splash. By using simpledrm for the splash the splash can be shown earlier, avoiding the impression the machine is hanging during boot. An extreme example of this is my old AMD64 embedded PC, where the time to show the first frame of the splash goes down from 47 to 9 seconds.

4. One less thing to worry about when trying to create a uniform desktop pre-generated and signed initramfs (these would still need support for nvme + ahci and commonly used rootfs + lvm + luks).

Disadvantages:

Doing this will lead to user visible changes in the boot process:

1. Secondary monitors not lit up by the efifb will stay black during full-disk encryption password entry, since the GPU drivers will now only load after switching to the encrypted root. This includes any monitors connected to the non boot GPU in dual GPU setups.

Generally speaking this is not really an issue, the secondary monitors will light up pretty quickly after the switch to the real rootfs. However when booting a docked laptop, with the lid closed and the only visible monitor(s) are connected to the non boot GPU, then the full-disk encryption password dialog will simply not be visible at all.

This is the main deal-breaker for not implementing this change.

Note because of the strict version lock between kernel driver and userspace with nvidia binary drivers, the nvidia binary drivers are usually already not part of the initramfs, so this problem already exists and moving the GPU drivers out of the initramfs does not really make this worse.

2. With simpledrm plymouth does not get the physical size of the monitor, so plymouth will need to switch to using heuristics on the resolution instead of DPI info to decide whether or not to use hidpi (e.g. 2x size) rendering and even when switching to the real GPU driver plymouth needs to stay with its initial heuristics based decision to avoid the scaling changing when switching to the real driver which would lead to a big visual glitch / change halfway through the boot.

This may result in a different scaling factor for some setups, but I do not expect this really to be an issue.

3. On some (older) systems the efifb will not come up in native mode, but rather in 800x600 or 1024x768.

This will lead to a pretty significant discontinuity in the boot experience when switching from say 800x600 to 1920x1080 while plymouth was already showing the spinner at 800x600.

One possible workaround here is to add: 'video=efifb:auto' to the kernel commandline which will make the efistub switch to the highest available resolution before starting the kernel. But it seems that the native modes are simply not there on systems which come up at 800x600 / 1024x768 so this does not really help.

This does not actually break anything but it does look a bit ugly. So we will just need to document this as an unfortunate side-effect of the change and then we (and our users) will have to live with this (on affected hardware).

4. On systems where a full modeset is done the monitor going briefly black from the modeset will move from being just before plymouth starts to the switch from simpledrm drm to the real driver. So that is slightly worse. IMHO the answer here is to try and get fast modesets working on more systems.

5. On systems where the efifb comes up in the panel's native mode and a fast modeset can be done, the spinner will freeze for a (noticeable) fraction of a second as the switch to the real driver happens.

Preview:

To get an impression what this will look / feel like on your own systems, you can implement this right now on Fedora 40 with some manual configuration changes:

1. Create /etc/dracut.conf.d/omit-gpu-drivers.conf with:

omit_drivers+=" amdgpu radeon nouveau i915 "

And then run "sudo dracut -f" to regenerate your current initrd.

2. Add to kernel commandline: "plymouth.use-simpledrm"

3. Edit /etc/selinux/config, set SELINUX=permissive this is necessary because ATM plymouth has issues with accessing drm devices after the chroot from the initrd to the rootfs.

Note this all assumes EFI booting with efifb used to show the plymouth boot splash. For classic BIOS booting it is probably best to stick with having the GPU drivers inside the initramfs.

Many recent Intel laptops have replaced the standard UVC USB camera module with a raw MIPI camera-sensor connected to the IPU6 found in recent Intel laptop chips.

Both the hw interface of the ISP part of the IPU6 as well as the image processing algorithms used are considered a trade secret and so far the only Linux support for the IPU6 relies on an out of tree kernel driver with a proprietary userspace stack on top, which is currently available in rpmfusion.

Both Linaro and Red Hat have identified the missing ISP support for various ARM and X86 chips as a problem. Linaro has started a project to add a SoftwareISP component to libcamera to allow these cameras to work without needing proprietary software and Red Hat has joined Linaro in working on this.

FOSDEM talk

Bryan O'Donoghue (Linaro) and I are giving a talk about this at FOSDEM.

Fedora COPR repository

This work is at a point now where it is ready for wider testing. A Fedora COPR repository with a patched kernel and libcamera is now available for users to test, see the COPR page for install and test instructions.

This has been tested on the following devices:

Lenovo ThinkPad X1 yoga gen 8 (should work on any ThinkPad with ov2740 sensor)
Dell Latitude 9420 (ov01a1s sensor)
HP Spectre x360 13.5 (2023 model, hi556 sensor)

Description of the stack

Kernel driver for the camera sensor, for the ov2740 used on current Lenovo designs (excluding MTL) I have landed all necessary kernel changes for this upstream.
Kernel support for the CSI receiver part of the IPU6 Intel is working on upstreaming this and has recently posted v3 of their patch series for this upstream and this is under active review.
A FOSS Software ISP stack inside libcamera to replace the missing IPU6 ISP (processing-system/psys) support. Work on this is under way. I've recently send out v2 of the patch-series for this.
Firefox pipewire camera support and support for the camera portal to get permission to access the camera. My colleague Jan Grulich has been working on this, see Jan's blogpost. Jan's work has landed in the just released Firefox 122.

There is an issue with the rpmfusion packaged IPU6 camera stack for Fedora is not working on many Dell laptop models after upgrading the kernel to a 6.5.y kernel.

This is caused by a new mainline ov0a10 sensor driver which takes precedence over the akmod ov0a10 driver but lacks VSC integration.

This can be worked around by running the following command:

sudo rm /lib/modules/$(uname -r)/kernel/drivers/media/i2c/ov01a10.ko.xz; sudo depmod -a

After the rm + depmod run:

sudo rmmod ov01a10; sudo modprobe ov01a10

Or reboot. After this your camera will hopefully work again.

I have submitted a pull-request to disable the mainline kernel's non working ov01a10 driver, so after the next Fedora kernel update this workaround should no longer be necessary.

I have just become aware that Fedora users using the packaged IPU6 camera stack are having an issue where the output from the camera is black. We have been updating the stack and the new version of ipu6-camera-bins has hit the stable updates repo, while the matching new version of ipu6-camera-hal is currently in the updates-testing repo.

This causes the versions of ipu6-camera-bins vs ipu6-camera-hal to get out of sync (unless you have updates-testing enabled) which leads to the camera output being all black

You can fix this issue by running the following command:

sudo dnf update --enablerepo=rpmfusion-nonfree-updates-testing 'ipu6-camera-*'

Sorry about the inconvenience, we'll make sure to push both packages at the same time for the next set of updates.

I have tagged all the new ipu6-camera-hal builds to be moved to the stable update repositories, so on the next rpmfusion updates push this should be resolved.

Installation

I am happy to announce that Intel's IPU6 camera stack has been packaged in rpmfusion and now can be installed under Fedora 37 and newer with a single `dnf install` command.

Note since this uses an out of tree kernel module build as unsigned akmod you need to disable secureboot for this to work; or alternatively sign the kmod with your own local key (instructions here).

First enable both the rpmfusion-free and rpmfusion-nonfree repositories, for instructions see https://rpmfusion.org/Configuration

The IPU6 support requires kernel >= 6.3.1 which is in updates-testing for now and v4l2loopback also needs to be updated to the latest version (in case you already have it installed):

sudo dnf update \
--enablerepo=updates-testing \
--enablerepo=rpmfusion-free-updates-testing \
--enablerepo=rpmfusion-nonfree-updates-testing \
'kernel*' '*v4l2loopback'

And now things are ready to install the IPU6 driver stack:

sudo dnf install \
--enablerepo=updates-testing \
--enablerepo=rpmfusion-free-updates-testing \
--enablerepo=rpmfusion-nonfree-updates-testing \
akmod-intel-ipu6

After this command reboot and you should be able to test your camera with https://mozilla.github.io/webrtc-landing/gum_test.html under firefox now.

This relies on Intel's partly closed-source hw-enablement for the IPU6, as such this known to not work on laptop models which are not covered by Intel's hw-enablement work. If your laptop has an option to come with Linux pre-installed and that SKU uses the IPU6 cameras then this should work. Currently known to work models are:

Dell Latitude 9420 (ov01a1s sensor)
Dell Precision 5470 (ov01a10 sensor)
Dell XPS 13 Plus 9320 (ov01a10 sensor)
Lenovo ThinkPad X1 Carbon Gen 10 (ov2740 sensor)
Lenovo ThinkPad X1 Nano Gen 2 (ov2740 sensor)
Lenovo ThinkPad X1 Yoga Gen 7 (ov2740 sensor)

If the IPU6 driver works for you on an unlisted model please drop mean email at <[email protected]> so that the above list can be updated.

Description of the stack

The IPU6 camera stack consists of the following layers:

akmod-intel-ipu6 the IPU6 kernel drivers. These are currently out of tree. Work is ongoing on getting various IO-expander, sensor drivers and the CSI2 receiver patches upstream. This is a slow process though and currently there is no clear path to getting the actual ISP part of the IPU supported upstream.
ipu6-camera-bins this is a set of closed-source userspace libraries which the rest of the userspace stack builds on top of. There is a separate set of libraries for each IPU6 variant. Currently there are 2 sets, "ipu6" for Tiger Lake and "ipu6ep" for Alder Lake.
ipu6-camera-hal this is a library on top of the set of libraries in ipu6-camera-bins. This needs to be built separately for the "ipu6" and "ipu6ep" library sets from ipu6-camera-bins.
gstreamer1-plugins-icamerasrc a gstreamer plugin built on top of ipu6-camera-hal. This allows using the camera through gstreamer.
akmod-v4l2loopback + v4l2-relayd. Most apps don't use gstreamer for camera access and even those that do don't know they need to use the icamerasrc element. v4l2-relayd will monitor a v4l2loopback /dev/video0 node and automatically start a gstreamer pipeline to send camera images into the loopback when e.g. firefox opens the /dev/video0 node to capture video.

Packaging challenges and technical details

akmod-intel-ipu6: There were 2 challenges to overcome before the IPU6 kernel drivers could be packaged:
1. The sensor drivers required patches to the main kernel package, specifically to the INT3472 driver which deals with providing GPIO, clk, regulator and LED resources to the sensor drivers. Patches have been written for both the main kernel, including some LED subsystem core additions, as well as patches to the IPU6 sensor drivers to bring them inline with mainline kernel conventions for GPIOs, clks and LEDs. All the necessary patches for this are upstream now, allowing the latest ipu6-drivers code to work with an unmodified mainline kernel.
2. Until now the IPU6 drivers seem to have been used with a script which manually loads the modules in a specific order. Using automatic driver loading by udev exposed various probe-ordering issues. Requiring numerous patches (all upstreamed to Intel's github repo) to fix.
ipu6-camera-bins: Since there were 2 sets of libraries for different IPU6 versions, these are now installed in separate /usr/lib64/ipu6[ep] directories and the headers and pkgconfig files are also installed in 2 different variants.
ipu6-camera-hal: This needs to be built twice against the 2 different sets of ipu6-camera-bins libraries. Letting the user pick the right libcamhal.so build to install is not very user friendly, to avoid the user needing to manually chose:
1. Both builds are installed in separate /usr/lib64/ipu6[ep] directories.
2. The libcamhal.so under these directories is patched to include the directory it is installed in as RPATH, so that dynamic-linking against that libcamhal.so will automatically link against the right set of ipu6-camera-bins libraries.
3. To make all this all work transparently the actual /usr/lib64/libcamhal.so is a symlink to /run/libcamhal.so and /run/libcamhal.so is set by udev rules to point to /usr/lib64/ipu6[ep]/libcamhal.so depending on the actual hw. The /run/libcamhal.so indirection is there so that things will also work with an immutable /usr .
4. ipu6-camera-hal's udev rules also set a /run/v4l2-relayd config file symlink to configure the gstreamer pipeline use by v4l2-relayd to match the ipu6 vs ipu6ep capabilities.
akmod-v4l2loopback + v4l2-relayd: Getting this to work with Firefox was somewhat tricky, there were 2 issues which had to be solved:
1. Firefox does not accept the NV12 image format generated by ipu6ep pipelines. To work around this a conversion to YUV420 has been added to the v4l2-relayd pipeline feeding into v4l2loopback. This workaround can be dropped once Firefox 114, which will have NV12 support, is released.
2. Gstreamer sends v4l2-buffers with a wrong bytesused field value into v4l2loopback causing Firefox to reject the video frames. A patch has been written and merged upstream to make v4l2loopback fix up the bytesused value, fixing this.

Many thanks to my colleague Kate Hsuan for doing most of the packaging work for this.

And also a big thank you to the rpmfusion team for maintaining the rpmfusion repo and infrastructure which allows packaging software which does not meet Fedora's strict guidelines outside of the Fedora infra.

As mentioned in my previous blog post, I have written a new patch series for 6.2 to try to avoid having multiple entries in /sys/class/backlight for a single panel again.

This new series might cause regressions on a different set of even older laptop models then the one affected by the 6.1 backlight work. So I'm again looking for people willing to run a few quick tests.

To see if your laptop is possibly affected by the 6.2 change please run:

ls /sys/class/backlight

if the output of this command contains both:

A GPU native backlight device, with intel/nv/nvidia/amd/radeon/psb/oaktrail in the name; and
A vendor backlight device, with either a model-series: eeepc, ideapad, thinkpad, etc; or a vendor-name: acer, asus, dell, toshiba, sony, etc. in the name

then 6.2 will cause a behavior change on your device, it will hide the vendor backlight device in preference of the native backlight device.

If your laptop shows only a native or only a vendor backlight device (possibly in combination with another type of backlight device such as acpi_video#), then your laptop will not be affected by the planned changes for 6.2.

Note that I expect only very old models to be affected, e.g. the Sony Vaio PCG-FRV3
from 2003 is known to be affected.

If your laptop has both native + vendor backlight devices, then please do 2 things:

Please run the following commands:
1. ls /sys/class/backlight > ls-backlight.txt
2. sudo dmesg > dmesg.txt
3. sudo dmidecode > dmidecode.txt
4. sudo acpidump -o acpidump.txt
Please test if the native backlight interface works, the example below assumes the native backlight is called "intel_backlight":
1. cd /sys/class/backlight/intel_backlight
2. cat max_brightness
3. <the "cat max_brightness" will show the maximum brightness value supported>
4. echo $max_brightness_value > brightness
5. echo $half-of-max_brightness_value > brightness

Where if for example cat max_brightness returns 255 then $max_brightness_value
is 255 and $half-of-max_brightness_value is 127. And then check if the brightness of the backlight actually changes when you do this ?

After generating the 4 .txt files and running the native backlight tests, please send me an email about this at [email protected] with the results of the native backlight tests (panel brightness changes when echo-ing values to brightness or not?) and with the 4 generated .txt files attached.

If the native backlight interface works then things should keep working fine with 6.2 and typically you will get more fine-grained brightness control as an added bonus. Please send me an email with the test results even if the native backlight interface works.

I have received quite a few test reports in response to my previous blog post. Many thanks to everyone who has run the tests and send me their results!

These tests show that as a result of the current 6.1 changes quite a few laptop models will end up with an empty "/sys/class/backlight", breaking users ability to control their laptop panel's brightness.

I have submitted a patch-set for 6.1 upstream to fix this.

More detailed summary/analysis of the received test reports:

Known Windows laptop models affected by this:
- Acer Aspire 1640
- HP Compaq nc6120
- IBM ThinkPad X40
- System76 Starling Star1
Known MacBook models affected by this:
- Apple MacBook 2.1
- Apple MacBook 4.1
- Apple MacBook Pro 7.1
30 unaffected models
The Dell Inspiron N4010 has acpi_video support and acpi_osi_is_win8() returns false, so acpi_video_get_backlight_type() returns acpi_video, but acpi_video fails to register a backlight device due to a _BCM eval error. This model already has a DMI quirk for this, so it is unaffected.
The following laptop models use vendor backlight control method, while also having a native backlight entry under /sys/class/backlight:
- Asus EeePC 901 (native backlight confirmed to work better then vendor)
- Dell Latitude D610 (native backlight confirmed to also work)
- Sony Vaio PCG-FRV3 (native backlight control does not work, BIOS from 2003!)

The fixes for 6.1 restore the behavior where userspace can see multiple entries under "/sys/class/backlight" for a single panel and the kernel leaves figuring out which one actually works up to userspace. This is undesirable and having more then 1 backlight device for a single panel also blocks the new backlight userspace API work which I have planned.

This first round of testing has shown that native works well even on systems so old that they don't have acpi_video backlight control support.

So I have prepared a patch series to try again with 6.2 by making native be preferred over vendor, which should avoid the problems seen with the 6.1 changes before the fixes.

I have landed a large(ish) refactor of the ACPI/x86 backlight detection code in the kernel for 6.1. I have been very careful to try and not break things but there is a special group of laptops where the ability to control the backlight brightness may disappear because of this.

The most likely laptops to be hit by this are laptops which are either pretty old and or which are weird in some other way (e.g. flashed with coreboot, did not ship with Windows as factory os, ...). Note Chromebooks are affected by this too, but that special category has already been fixed.

You can check if your laptop is affected by this by running "ls /sys/class/backlight" if this shows only 1 entry and that entry is named "intel_backlight", "nouveau_bl", "amdgpu_bl0" or "radeon_bl0" then your laptop might be affected.

Note this is quite normal on modern(ish) laptops, a second check is to boot with "acpi_backlight=video" added to the kernel commandline and then run "ls /sys/class/backlight" again, if you now additionally also have an "acpi_video0" entry then your laptop should work fine with 6.1, if you don't have an "acpi_video0" entry please first do "cat /proc/cmdline" and check that "acpi_backlight=video" is present there.

If you have e.g. only the "intel_backlight" entry and adding "acpi_backlight=video" does not cause an "acpi_video0" entry to appear then 6.1 will likely break backlight control!

If you have a laptop which is likely affected by this then please run the following commands:

ls /sys/class/backlight > ls-backlight.txt
sudo dmesg > dmesg.txt
sudo dmidecode > dmidecode.txt
sudo acpidump -o acpidump.txt

And send me an email about this at [email protected] with the 4 generated .txt files attached. If possible please also give an actual 6.1 kernel a try and see if that indeed breaks things. E.g. for Fedora you can find 6.1 kernel builds here and see here for some install instructions for these Fedora kernel builds.

Now that FESCo has decided that Fedora will keep supporting BIOS booting, the people working on Fedora's bootloader stack will need help from the Fedora community to keep Fedora booting on systems which require Legacy BIOS to boot.

To help with this the Fedora BIOS boot SIG (special interest group) has been formed. The main goal of this SIG are to help the Fedora bootloader people by:

Doing regular testing of nightly Fedora N + 1 composes on hardware
which only supports BIOS booting
Triaging and/or fixing BIOS boot related bugs.

A [email protected] mailinglist and bugzilla account has been created, which will be used to discuss testing result and as assignee / Cc for bootloader bugzillas which are related to BIOS booting.

If you are interested in helping with Fedora BIOS boot support please:

Subscribe to the email-list
Add yourself to the Members section of the SIG's wiki page

Starting with kernel 5.17 the kernel supports the builtin privacy screens built into the LCD panel of some new laptop models.

This means that the drm drivers will now return -EPROBE_DEFER from their probe() method on models with a builtin privacy screen when the privacy screen provider driver has not been loaded yet.

To avoid any regressions distors should modify their initrd generation tools to include privacy screen provider drivers in the initrd (at least on systems with a privacy screen), before 5.17 kernels start showing up in their repos.

If this change is not made, then users using a graphical bootsplash (plymouth) will get an extra boot-delay of up to 8 seconds (DeviceTimeout in plymouthd.defaults) before plymouth will show and when using disk-encryption where the LUKS password is requested from the initrd, the system will fallback to text-mode after these 8 seconds.

I've written a patch with the necessary changes for dracut, which might be useful as an example for how to deal with this in other initrd generators, see: https://github.com/dracutdevs/dracut/pull/1666

I've also filed bugs for tracking this for Fedora, openSUSE, Arch, Debian and Ubuntu.

For the hw-enablement for Bay- and Cherry-Trail devices which I do as a side project, sometimes it is useful to play with the Android which comes pre-installed on some of these devices.

Sometimes the Android-X86 boot-loader (kerneflinger) is locked and the standard "Developer-Options" -> "Enable OEM Unlock" -> "Run 'fastboot oem unlock'" sequence does not work (e.g. I got the unlock yes/no dialog, and could move between yes and no, but I could not actually confirm the choice).

Luckily there is an alternative, kernelflinger checks a "OEMLock" EFI variable to see if the device is locked or not. Like with some of my previous adventures changing hidden BIOS settings, this EFI variable is hidden from the OS as soon as the OS calls ExitBootServices, but we can use the same modified grub to change this EFI variable. After booting from an USB stick with the relevant grub binary installed as "EFI/BOOT/BOOTX64.EFI" or "BOOTIA32.EFI", entering the
following command on the grub cmdline will unlock the bootloader:

setup_var_cv OEMLock 0 1 1

Disabling dm-verity support is pretty easy on these devices because they can just boot a regular Linux distro from an USB drive. Note booting a regular Linux distro may cause the Android "system" partition to get auto-mounted after which dm-verity checks will fail! Once we have a regular Linux distro running step 1 is to find out which partition is the android_boot partition to do this as root run:

blkid /dev/mmcblk?p#

Replacing the ? for the mmcblk number for the internal eMMC and then for # is 1 to n, until one of the partitions is reported as having 'PARTLABEL="android_boot"', usually "mmcblk?p3" is the one you want, so you could try that first.

Now make an image of the partition by running e.g.:

dd if=/dev/mmcblk1p3" of=android_boot.img

And then copy the "android_boot.img" file to another computer. On this computer extract the file and then the initrd like this:

abootimg -x android_boot.img
mkdir initrd
cd initrd
zcat ../initrd.img | cpio -i

Now edit the fstab file and remove "verify" from the line for the system partition. after this update android_boot.img like this:

find . | cpio -o -H newc -R 0.0 | gzip -9 > ../initrd.img
cd ..
abootimg -u android_boot.img -r initrd.img

The easiest way to test the new image is using fastboot, boot the tablet into Android and connect it to the PC, then run:

adb reboot bootloader
fastboot boot android_boot.img

And then from an "adb shell" do "cat /fstab" verify that the "verify" option is gone now. After this you can (optionally) dd the new android_boot.img back to the android_boot partition to make the change permanent.

Note if Android is not booting you can force the bootloader to enter fastboot mode on the next boot by downloading this file and then under regular Linux running the following command as root:

cat LoaderEntryOneShot > /sys/firmware/efi/efivars/LoaderEntryOneShot-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f

This all started with a Mele PCG09 before testing Linux on this I took a quick look under Windows and the device-manager there showed an exclamation mark next to a Realtek 8723BS bluetooth device, so BT did not work. Under Linux I quickly found out why, the device actually uses a Broadcom Wifi/BT chipset attached over SDIO/an UART for the Wifi resp. BT parts. The UART connected BT part was described in the ACPI tables with a HID (Hardware-ID) of "OBDA8723", not good.

Now I could have easily fixed this with an extra initrd with DSDT-overrride but that did not feel right. There was an option in the BIOS which actually controls what HID gets advertised for the Wifi/BT named "WIFI" which was set to "RTL8723" which obviously is wrong, but that option was grayed out. So instead of going for the DSDT-override I really want to be able to change that BIOS option and set it to the right value. Some duckduckgo-ing found this blogpost on changing locked BIOS settings.

The flashrom packaged in Fedora dumped the BIOS in one go and after build UEFITool and ifrextract from source from their git repos I could extract the interface description for the BIOS Setup menus without issues (as described in the blogpost). Here is the interesting part of the IFR for changing the Wifi/BT model:

0xC521 One Of: WIFI, VarStoreInfo (VarOffset/VarName): 0x110, VarStore: 0x1, QuestionId: 0x1AB, Size: 1, Min: 0x0, Max 0x2, Step: 0x0 {05 91 53 03 54 03 AB 01 01 00 10 01 10 10 00 02 00}
0xC532 One Of Option: RTL8723, Value (8 bit): 0x1 (default) {09 07 55 03 10 00 01}
0xC539 One Of Option: AP6330, Value (8 bit): 0x2 {09 07 56 03 00 00 02}
0xC540 One Of Option: Disabled, Value (8 bit): 0x0 {09 07 01 04 00 00 00}
0xC547 End One Of {29 02}

So to fix the broken BT I need to change the byte at offset 0x110 in the "Setup" EFI variable which contains the BIOS settings from 0x01 to 0x02. Easy, one problem though, the "dd on /sys/firmware/efi/efivars/Setup-..." method described in the blogpost does not work on most devices. Most devices protect the BIOS settings from being modified this way by having 2 Setup-${GUID} EFI variables (with different GUIDs), hiding the real one leaving a fake one which is only a couple of bytes large.

But the BIOS Setup-menu itself is just another EFI executable, so how can this access the real Setup variable ? The trick is that the hiding happens when the OS calls exitbootservices to tell EFI it is ready to take over control of the machine. This means that under Linux the real Setup EFI variable has been hidden early on during Linux boot, but when grub is running it is still available! And there is a patch adding a new setup_var command to grub, which allows changing BIOS settings from within grub.

The original setup_var command picks the first Setup EFI variable it finds, but as mentioned already in most cases there are 2, so later an improved setup_var_3 command was added which instead skips Setup EFI variables which are too small (as the fake ones are only a few bytes). After building an EFI version of grub with the setup_var* commands added it is just a matter of booting into a grub commandline and then running "setup_var_3 0x110 2" and from then on the BIOS shows the WIFI type as being AP6330 and the ACPI tables will now report "BCM2E67" as HID for the BT and just like that the bluetooth issue has been fixed.

( Read more... )

As you may know I've been doing a lot of hw-enablement work on Bay- and Cherry-Trail tablets as a side-project for the last couple of years.

Some of these tablets have one interesting feature intended to "flash" Android on them. When turned on with both the volume-up and the volume-down buttons pressed at the same time they enter something called DNX mode, which it will then also print to the LCD panel, this is really just a variant of the android fastboot protocol built into the BIOS. Quite a few models support this, although on Bay Trail it sometimes seems to be supported (it gets shown on the screen) but it does not work since many models which only shipped with Windows lack the external device/gadget-mode phy which the Bay Trail SoC needs to be able to work in device/gadget mode (on Cherry Trail the gadget phy has been integrated into the SoC).

So on to the topic of this blog-post, I recently used DNX mode to unbrick a tablet which was dead due to the BIOS settings get corrupted in a way where it would not boot and it was also impossible to enter the BIOS setup. After some duckduckgo-ing I found a thread about how in DNX mode you can upload a replacement for the efilinux.efi bootloader normally used for "fastboot boot" and how you can use this to upload a binary to flash the BIOS. I did not have a BIOS image of this tablet, so that approach did not work for me. But it did point me in the direction of a different, safer (no BIOS flashing involved) solution to unbrick the tablet.

If you run the following 2 commands on a PC with a Bay- or Cherry-Trail connected in DNX mode:

fastboot flash osloader some-efi-binary.efi
fastboot boot some-android-boot.img

Then the tablet will execute the some-efi-binary.efi. At first I tried getting an EFI shell this way, but this failed because the EFI binary gets passed some arguments about where in RAM it can find the some-android-boot.img. Then I tried booting a grubx64.efi file and that result in a grub commandline. But I had not way to interact with it and replacing the USB connection to the PC with a OTG / USB-host cable with a keyboard attached to it did not result in working input.

So my next step was to build a new grubx64.efi with "-c grub.cfg" added to the commandline for the final grub2-mkimage step, embedding a grub.cfg with a single line in there: "fwsetup". This will cause the tablet to reboot into its BIOS setup menu. Note on some tablets you still will not have keyboard input if you just let the tablet sit there while it is rebooting. But during the reboot there is enough time to swap the USB cable for an OTG adapter with a keyboard attached before the reboot completes and then you will have working keyboard input. At this point you can select "load setup defaults" and then "save and exit" and voila the tablet works again.

For your convenience I've uploaded a grubia32.efi and a grubx64.efi with the necessary "fwsetup" grub.cfg here. This is build from this branch at this commit (this was just a random branch which I had checked out while working on this).

Note the image used for the "fastboot boot some-android-boot.img" command does not matter much, but it must be a valid android boot.img format file otherwise fastboot will refuse to try to boot it.

A while ago I worked on improving Logitech G15 LCD-screen support under Linux. I recently got an email from someone who wanted to add support for the LCD panel in the Logitech Z-10 speakers to lcdproc, asking me to describe the process I went through to improve G15 support in lcdproc and how I made it work without requiring the unmaintained g15daemon code.

So I wrote a long email describing the steps I went through and we both thought this could be interesting for others too, so here it is:

1. For some reason I decided that I did not have enough projects going on at the same time already and I started looking into improving support for the G15 family of keyboards.

2. I started studying the g15daemon code and did a build of it to check that it actually works. I believe making sure that you have a known-to-work codebase as a starting point, even if it is somewhat crufty and ugly, is important. This way you know you have code which speaks the right protocol and you can try to slowly morph it into what you want it to become (making small changes testing every step). Even if you decide to mostly re-implement from scratch, then you will likely use the old code as a protocol documentation and it is important to know it actually works.

3. There were number of things which I did not like about g15daemon:

3.1 It uses libusb, taking control of the entire USB-interface used for the gaming functionality away from the kernel. 

3.2 As part of this it was not just dealing with the LCD, it also was acting as a dispatches for G-key key-presses. IMHO the key-press handling clearly belonged in the kernel. These keys are just extra keys, all macro functionality is handled inside software on the host/PC side. So as a kernel dev I found that these really should be handled as normal keys and emit normal evdev event with KEY_FOO codes from a /dev/input/event# kernel node.

3.3 It is a daemon, rather then a library; and most other code which would deal with the LCD such as lcdproc was a daemon itself too, so now we would have lcdproc's LCDd talking to g15daemon to get to the LCD which felt rather roundabout.
So I set about tackling all of these

4. Kernel changes: I wrote a new drivers/hid/hid-lg-g15.c kernel driver:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/hid/hid-lg-g15.c
Which sends KEY_MACRO1 .. KEY_MACRO18, KEY_MACRO_PRESET1 .. KEY_MACRO_PRESET3, KEY_MACRO_RECORD_START, KEY_KBD_LCD_MENU1 .. KEY_KBD_LCD_MENU4 keypresses for all the special keys. Note this requires that the kernel HID driver is left attached to the USB interface, which required changes on the g15dameon/lcdproc side.

This kernel driver also offers a /sys/class/leds/g15::kbd_backlight/ interface which allows controller the backlight intensity through e.g. GNOME kbd-backlight slider in the power pane of the control-center. It also offers a bunch of other LED interfaces under /sys/class/leds/ for controlling things like the LEDs under the M1 - M3 preset selection buttons. The idea here being that the kernel abstract the USB protocol for these gaming-kbd away and that a single userspace daemon for doing gaming kbd macro functionality can be written which will rely only on the kernel interfaces and will thus work with any kbd which has kernel support.

5. lcdproc changes

5.1 lcdproc already did have a g15 driver, which talked to g15daemon. So at first I started testing with this (at this point all my kernel work would not work, since g15daemon would take full control of the USB interface unbinding my kernel driver). I did a couple of bug-fixes / cleanups in this setup to get the code to a starting point where I could run it and it would not show any visible rendering bugs in any of the standard lcdproc screens

5.2 I wrote a little lcdproc helper library, lib_hidraw, which can be used by lcdproc drivers to open a /dev/hidraw device for them. The main feature is, you give the lib_hidraw_open() helper a list of USB-ids you are interested in; and it will then find the right /dev/hidraw# device node (which may be different every boot) and open it for you.

5.3 The actual sending of the bitmap to the LCD screen is quite simple, but it does need to be a very specific format. The format rendering is done by libg15render. So now it was time to replace the code talking to g15daemon with code to directly talk to the LCD through a /dev/hidraw# interface. I kept the libg15render dependency since that was fine. After a bit of refactoring, the actual change over to directly sending the data to the LCD was not that big.

5.4 The change to stop using the g15daemon meant that the g15 driver also lost support for detecting presses on the 4 buttons directly under the LCD which are meant for controlling the menu on the LCD. But now that the code is using /dev/hidraw# the kernel driver I wrote would actually work and report KEY_KBD_LCD_MENU1 .. KEY_KBD_LCD_MENU4 keypresses. So I did a bunch of cleanup / refactoring of lcdproc's linux_input driver and made it take over the reporting of those button presses.

5.5 I wanted everything to just work out of the box, so I wrote some udev rules which automatically generate a lcdproc.conf file configuring the g15 + linux_input drivers (including the key-mapping for the linux_input driver) when a G15 keyboard gets plugged in and the lcdproc.conf file does not exist yet.

All this together means that users under Fedora, where I also packaged all this can now do "dnf install lcdproc", pluging their G15 keyboard and everything will just work.

p.s.

After the email exchange I got curious and found a pair of these speakers 2nd hand for a nice price. The author of the initial email was happy with me doing the work on this. So I added support for the Z-10 speakers to lcdproc (easy) and wrote a set of kernel-patches to support the display and 1-4 keys on the speaker as LCD menu keys.

I've also prepared an update to the Fedora lcdproc packages so that they will now support the Z-10 speakers OOTB, if you have these and are running Fedora, then (once the update has reached the repos) a "sudo dnf install lcdproc" followed by unplugging + replugging the Z-10 speakers should make the display come alive and show the standard lcdproc CPU/mem usage screens.

For a long time Logitech produced wireless keyboards using 27 MHz as communications band. Although these have not been produced for a while now these are still pretty common and a lot of them are still perfectly serviceable.

But when using them under Linux, there is one downside, since the communication is one way by default the wireless link is unencrypted by default, which is kinda bad from a security pov. These keyboards do support using an encrypted link, but this requires a one-time setup where the user manually enters a key on the keyboard.

I've written a small Linux utility to do this under Linux, which should help give these keyboards an extra lease on life and stop them unnecessarily becoming e-waste. Sometimes these keyboards appear to be broken, while the only problem is that the key in the keyboard and receiver are not in sync, the README also contains instructions on howto reset the keyboard, without needing the utility, restoring (unencrypted) functionality.

The 'lg-27MHz-keyboard-encryption-setup' utility is available on Fedora in the 'logitech-27mhz-keyboard-encryption-setup package.