Fedora 33: Fedora version upgrade breaks a production web server, and Fedora’s reputation for smooth version upgrades

My brother hosts his personal website and blog malak.ca on a baremetal DSL server. My brother uses Fedora on his laptop and server.

A perfectly good pre-fork mod_php MPM-ITK PHP handler was in place, and serving web pages.

Upon reboot after the major version upgrade, the web server was showing error 503 for PHP requests on the blog. The config files were a mismatched mess, so we ended up having to do a baremetal format. My brother keeps his data on a separate drive so the baremetal evac only involved a mysql dump file and a few config files, but still.

Note this post about Fedora 33: “Several relatively controversial changes are currently under discussion on the project’s mailing lists…”

https://lwn.net/Articles/824620/

“The default doesn’t matter, there’s absolutely no reason to take away the sysadmin’s choice here. There are at least 40 servers I personally am responsible for where I see no reason to move from mod_php to php-fpm, for example.” John M. Harris Jr.

I was a CentOS web server admin for many years, and used Fedora on my personal laptop until last year. My brother ran CentOS in the past, but towards the end of the Long Term Support (LTS) cycle, CentOS had absurdly outdated but security-patched versions of libraries. My brother started using Fedora on his web server, and we have been able to do several major version upgrades without incident. The reliability of this upgrade process is what made Fedora suitable for a web server.

Ubuntu has trouble with major version upgrades. On some Ubuntu version upgrades, the installer freezes, requiring that a rescue kernel be entered, apt-get update –fix-missing, dpkg –repair, and other exotic interventions take place before the upgrade process can be resumed and completed.

By comparison, Fedora version upgrading has a better track record, and is usually smooth. A technical error or unforeseen incompatibility would be understandable. A deliberate policy choice to break production web servers to enforce a policy opinion: not cool.

Fortunately, Remi RPM has come to the rescue:

https://rpms.remirepo.net/

https://rpmfind.net/linux/rpm2html/search.php?query=mod_php

Read my brother’s post on this:

Dell XPS 13 (9300): considerations for Linux

The Dell XPS 13 (9300) is a high-end laptop designed for Windows, which can be reformatted to run Linux.

You can pay as much as you want, or you can be patient

My patience was rewarded with a chance to buy a full spec system at a good price on a factory refurb with full warranty. So the lesson is: you can pay as much as you want. In addition to being a factory refurb, this model is now technically obsolete: a refreshed model Dell XPS 13 (9310) with a slightly newer CPU was announced in October 2020. See my blog post about the eBay Global Shipping Program run by Pitney Bowes:

The best computer is the one you have with you

Before this purchase, my main personal laptop was a 15.6” i7-4712HQ laptop 16GB RAM 1TB SSD, which is usually connected to a monitor, keyboard, and mouse on a desk at home.

However, when leaving the house, I have usually reached for my 11.6” netbook – a single-core Celeron (essentially a 64 bit Atom) with 4GB RAM and a 64GB MMC.

The golf cart of computers

This system is the golf cart of computers, but with enough performance tuning, can provide acceptable performance under Linux.

So I knew that my next personal computer purchase had to be small.

More power

For this new purchase, I needed the horsepower of a server: lots of CPU cores, lots of RAM, lots of SSD drive space.

The Dell XPS 13 (9300) has a 13.4” screen, but fits into the Amazon Basics 11.6” Laptop Sleeve:

Specifications

This machine has some serious horsepower. I waited till i could get 32GB of low-powered DDR RAM in this small a laptop, only available as of the 10th generation Intel Ice Lake processor.

Intel i7-1065G7 CPU (4 dual-thread cores)

32GB RAM

1TB SSD

13.4” 4K touchscreen

USB-C/Thunderbolt 3 docking stations, dongles, and the BIOS compatibility issue

There are discussions on Reddit and other places that allude to compatibility problems between the Dell XPS 13 (9300) and certain USB-C/Thunderbolt 3 docking stations. My first USB-C dongle purchase, which worked with a Dell XPS 15 (9550), was not compatible with the Dell XPS 13 (9300). Several posts on Reddit said this was a known issue and spoke of rumors of a Dell XPS 13 BIOS update in November 2020. However, I was able to find a part that is compatible:

Dell support for Linux hardware drivers

Dell offers support for Ubuntu 20.04LTS hardware drivers for this model. There are apt repositories offered by Dell. However, i found that Ubuntu 20.04LTS detected all of the hardware, aside from the fingerprint reader, which required an apt repo:

Dell deserves credit for building a system which is so Linux-friendly.

Dell is still Windows-centric for utilities like Dell Update and BIOS firmware updates

Dell has done a great job in supporting desktop Linux on the Dell XPS 13 (9300). However, some things are still Windows-centric: the Dell Update utility, and the BIOS and firmware updates provided by Dell, assume you are running Windows. For this reason alone, you should consider formatting your system as dual-boot between Windows and Linux baremetal partitions.

Formatting the Dell XPS 13 for dual boot between Windows and Linux

This Dell XPS 13 came with a “digital license” for Windows 10 Pro. If you buy a new laptop, it is almost certainly associated with a license for Windows 10 Home or Windows 10 Pro that will activate as soon as you reformat.

Run the factory-installed Windows to test the hardware

The Dell XPS 13 (9300) shipped with a factory-installed copy of Windows 10 Pro.

If you buy the Dell XPS 13 (9300) for use with Linux, I suggest that you run it on the factory-installed Windows first. Test the hardware: connect to the Internet, play a video. Then run the Dell Update utility to see if there are any updates, especially firmware or BIOS updates, you will need.

RAID vs AHCI setting in BIOS for Linux

If you intend to reformat this computer with Linux, you need to go into the BIOS and change the SATA drive controller from RAID mode to AHCI mode.

Change the SATA controller from RAID to AHCI in BIOS

Go into the BIOS of the laptop by pressing F2 during bootup, and change the SATA controller mode from RAID to AHCI.

Stay in UEFI mode

It may be tempting to switch to Legacy BIOS mode, but stay in UEFI mode: it will help Linux to create a multiple-boot menu later on during the build.

Creating a bootable USB-C memory stick installer (I used a USB-C micro-SD card reader)

I found that even with a USB-C male to USB-A female adapter, i was unable to boot from a USB-A memory stick. I was almost able to boot from a micro-SD card (after enabling the option in BIOS). However, I received a device driver error from the Windows 10 installer. I succeeded with a micro-SD card in a USB-C adapter:

(I assume there is such a thing as a USB-C memory stick, but they are not yet common.)

Rufus ISO utility

Rufus is an ISO burning utility:

[rufus screenshot]

https://rufus.ie/

Download a fresh ISO of Windows 10 from Microsoft:

https://www.microsoft.com/en-ca/software-download/windows10ISO

Use Rufus, and specify partion type GPT and target system UEFI-CSM. Burn the micro-SD card.

Run the Windows installer. Destroy all existing partitions. Then, install Windows in a 200GB partition, but leave the rest of the drive space as “unallocated space.”

Download a fresh ISO of Ubuntu 20.04LTS Linux:

https://releases.ubuntu.com/20.04/

Use Rufus, and specify partion type GPT and target system UEFI-CSM. Burn the micro-SD card.

Run the Ubuntu Linux installer. Let the installer use the remaining “unallocated space” on the hard drive.

Linux will install with a multiple-boot menu

It will also install a multiple-boot menu that appears when you start the computer. By default, Ubuntu Linux will boot first, but you can choose Windows as the boot within 10 seconds.

Consider using KVM to host Windows and MacOS under Linux

KVM allows you to host Windows, MacOS, and Linux as guest virtual machines under Linux. See my blog post on the subject:

eBay Global Shipping Program (GSP), Pitney Bowes, and 23 days between Kentucky and Canada

US-based merchants on eBay are sometimes cheaper than other sources. Many of these merchants use the eBay Global Shipping Program (GSP) run by Pitney Bowes. The GSP is cheap, slow, frustrating, but effective. I ordered a laptop from a reseller based in North Carolina. The reseller did their part promptly, shipping to the GSP warehouse in Erlanger KY within 2 days. However, after that, the package spent 11 days in limbo, without even a status update. Then it spent 12 days in a bonded warehouse in Canada before being released and shipped to me.

The brotherhood of wealthy middle powers

There is a strange brotherhood of buyers from wealthy middle powers like Canada, the UK, and Australia trading horror stories about the delays in the Global Shipping Program. There are stories about lost packages but for the most part the packages finally arrive. I am grateful to the posters for sharing tracking numbers that reassured that it is possible to not see an update for 2 weeks, then get a parcel delivered with 4 status updates all uploaded to the system at once:

https://www.ecommercebytes.com/C/letters/blog.pl?/pl/2020/7/1595105662.html

https://www.canadapost.ca/trackweb/en#/details/4011220526949650

(local archive: https://blog.gordonbuchan.com/files/eBayGlobalShippingDisappointsCollectiblesBuyer.pdf)

(Power tip: a friend pointed out that some tracking numbers, such as those provided by Canada Post, offer options that can change shipping choices before delivery. So if you do plan to share a tracking number on the Internet, better to wait until your package arrives.)

Re-estimating shipping times based on tracking numbers

Learn to use the eBay, Pitney Bowes, USPS, and your local post office tracking sites. Learn how to pull apart your status number and get statuses on local legs of the trip. Suggestion, use:

https://parcelsapp.com/en

to re-calculate your expected shipping times.

Stressful, but…

I found the Global Shipping Program to be stressful and irritating, but I saved a significant amount of money on a laptop purchase.

Apple moving to ARM CPUs for MacBooks : implications for Windows, Linux, and desktop computing

Apple has announced that it will be moving its line of MacBook laptops from Intel-based CPUs to ARM-based CPUs. While this does not have any immediate impact on the Windows or Linux desktop computing markets, Apple’s move validates the platform and may give Microsoft an incentive to again experiment with Windows on ARM (RIP Windows RT 2012-2015).

This has implications for Linux desktop computing, which relies upon the availability of Windows-on-Intel “Wintel” hardware. The transition to ARM CPUs on Mac and Windows hardware may be accompanied by bootloader lockdown and the inability to install unsigned binaries or other operating systems.

Since a Mac person may have found this blog post by searching for “Mac” and “ARM” I should address their concerns first

For the 80% of consumers using 20% of software features, the Mac experience will not change.

No doubt, Adobe’s Creative Suite applications like Photoshop, and Document Cloud applications like Acrobat, will be ported and available on the first day the MacBook on ARM goes on sale. I would expect that the open source movement will embrace the platform and have applications like VLC and Tunnelblick ready as well. Some specialized applications will not be available right away.

There will be disappointing performance using Rosetta 2 or other Intel-to-ARM CPU emulators.

Advanced Mac consumers will likely keep at least 1 Intel-based Mac for power and compatibility for the first year or 2 of the transition, to run the Mac applications that are not yet available for the native ARM.

Advanced Mac consumers will find that Parallels is less effective in virtualizing Windows, as it will be emulating the Intel CPU at the same time as it hosts the guest operating system. Boot Camp, which allowed multiple-boot to Windows in the past, will not be available, even when Windows on ARM is again available.

Advanced Mac consumers will likely keep at least 1 Intel-based Windows computer for compatibility as they lose the ability to run Windows in emulation or multiple-boot.

Software piracy is likely to be more difficult, due to a lack of Mac-on-ARM software available to pirate, and may be blocked altogether by App store controls and binary signing.

Market conditions are different from 2015

Market conditions and customer expectations are different from 2015. The inability to install 3rd-party software directly without using an app store intermediary was a deal-breaker in 2015. Now, 12 years after the introduction of the iPhone, customers are comfortable using app stores to access software. It may not be a coincidence that Windows 10 now has a relatively usable App store with free-as-in-beer and sometime even free-as-in-Libre apps like VLC.

The App store can be used to stop piracy

This will have profound effects on the Mac desktop ecosystem going forward, and may validate a similar path to be taken by Microsoft going forward. If Apple is able to transition half of its unit shipments to ARM (I predict that Mac will retain Intel models at the high end for performance and binary compatibility for a few years), then Microsoft will follow.

Open source communities like Linux and BSD rely upon hardware from the Wintel hardware ecosystem.

Desktop Linux needs Wintel hardware. There are hobbyist solutions based on ARM, like the Raspberry Pi and the Pinebook, and these platforms get better with every release. The software is there: both Red Hat (Fedora) and Canonical (Ubuntu) now support AARCH64 ARM CPUs. However the performance is not there, the hardware tuning is not there. We need to encourage Pi and Pine, because they may be all we have in 10 years.

Linux was almost blocked from Wintel hardware in 2008 with the UEFI bootloader

In 2008, there was concern about viruses that would infect the master boot record (MBR). A solution called unified extensible firmware interface (UEFI) essentially locked the computer so that it could only be formatted by an installer that had a certificate issued to a software publisher for a fee. Whether intentional or not, this had the potential to prevent open operating systems like Linux from installing on this new generation of Wintel hardware. A technical and political solution was found, and the UEFI threat to open source software was neutralized.

Android hardware is a vision of this possible future

Android telephones are essentially Linux-on-ARM computers. Most of them have bootloaders that are “locked” and will only allow software with a specific digital signature to be installed on the phone’s hardware. There are some phones that are easier to “unlock” than others.

When Windows RT came out in 2012, it did not allow “unsigned binaries.” the only software you could install was via the RT app store. When the Mac introduced an app store, it initially set the default to off, allowing the installation of unsigned binaries. Later versions set this value to on by default, requiring consumers to find the option and disable it before “unsigned” software was permitted on the computer.

Consumers have been trained by the app stores on iOS and Android

Now that consumers have been trained to accept app stores as the intermediary between their computer and the software they wish to install, it is not hard to imagine a future where the app store is all that is left. To protect consumers from security threats, Apple (and later Windows) may use app-store-on-ARM to eliminate piracy, while carefully cultivating the sense of openness by allowing open source apps into the app stores, as is the case with iOS and Android.

What happens if the Windows-on-ARM hardware has a locked bootloader?

Losing the ability to install desktop Linux on Mac hardware is not a big deal, in numerical terms. From a Linux hacker’s point of view, the issue will be Wintel and its eventual replacement Windows-on-ARM (“WinARM.”)

The server hardware market will accommodate Linux-on-ARM. But what about the desktop and tablet hardware markets?

Let’s imagine the following hypothetical timeline:

2020. Macbook-on-ARM released to market, does not suck. Software available via app store only, no sideload of unsigned binaries, locked bootloader, like an iPhone or most Android phones.

2021. Windows-on-ARM released to market, does not suck. Software available via app store only, no sideload of unsigned binaries, locked bootloader, like an iPhone or most Android phones.

Linux hackers will be able to use Windows-on-Intel junk for 5 years

For the first 5 years, this will not be a problem in practical terms. Linux hackers will be able to find and reformat used Windows-on-Intel hardware. But after that?

Back to the Raspberry Pi and Pinebook

This brings us back to the Raspberry Pi, and the Pinebook. We have to hope that these projects succeed. They may be the only hardware platform we will have left, on which to install free-as-in-Libre software.

Desktop Linux needs Microsoft Office and Adobe Photoshop

I know of a professional firm that uses a Linux server to store documents for a network of Mac users. The server has a screen, keyboard, and mouse at a desk in the photocopier room.

Linux is a weird kind of Mac

The office manager was asked to sit at console, enter a password, and check connectivity using the web browser. When the office manager saw a desktop with Chrome and TeamViewer, she seemed to relax: “oh, it’s a weird kind of Mac.”

Desktop computer market share 2013 vs 2020

Let’s take a look at the desktop computer market 2013 vs 2020:

2013

Windows: 90%

MacOS: 8%

Linux: 1%

2020

Windows: 78%

MacOS: 17%

Linux: 2%

(source: https://www.statista.com/statistics/218089/global-market-share-of-windows-7/)

12% of the desktop computer market has moved away from Windows

Desktop computing has not fundamentally changed for the past 7 years. But due to problems with system stability and security challenges, 12% of the desktop computer market has moved away from Windows. 9% of the market switched to MacOS. 1% of the market switched to Linux.

Windows on Intel, UNIX on Intel

At the time of this writing, Windows, MacOS, and Linux desktop computers mostly use Intel-compatible CPUs. MacOS and Linux are both essentially UNIX-on-Intel computers, with roughly the same performance and security advantages relative to Windows running on similar hardware. As our friend the office manager put it, “Linux is a weird kind of Mac.”

Applications are brand names

Our friend the office manager recognized a few brand names: Chrome and TeamViewer. Their presence validated the platform. They allowed her to consider the Linux desktop as a viable alternative to her Mac.

Some standard applications are already present on the Linux desktop

On the Linux desktop on which I am writing this post, I have Chrome, Filezilla, VNC, VLC, TeamViewer, Zoom, and Teams. These same applications can be installed on Windows and MacOS computers.

The Linux desktop does not have Microsoft Office or Adobe Photoshop

At the time of this writing, Microsoft Office and Adobe Photoshop are not available for the Linux desktop.

Work-alike replacements are not valid in the eyes of consumers

This post is being written in LibreOffice Writer, which is open source software that tries to reproduce the Microsoft Office suite. Like Pages for the Mac, it does a competent job of reading and writing word processing files. But it is not Microsoft Office 365. Same for Photoshop: there are many image editing programs, but they are not the brand name, and really not the same thing. Word processing might be one thing, but aside from the simplest spreadsheets and presentations, Excel and PowerPoint are not really replaceable.

Macs have Office and Photoshop

Macs are relevant to consumers because they can run Microsoft Office 365 and Adobe Photoshop. They may not be able to run every accounting or engineering program available for Windows, but having Office and Photoshop covers most needs. That is why when Windows lost 12% of of the entire market, MacOS was able to capitalize on the opportunity to capture the consumers making a change, but Linux was not.

If Linux had Microsoft Office 365 and Adobe Photoshop

If Linux had Microsoft Office 365 and Adobe Photoshop, we would see the emergence of a second strong UNIX on Intel desktop computer platform. Linux, the “weird kind of Mac,” would get more interest, more relevance, and more market share.