Dear reader, please note that this article was written anonymously, as I am a person deep inside the Enterprise Linux ecosystem. I know "anonymous" is a big word, and if anyone wants to identify the author, they can.
I'm against changes to RHEL code redistribution, and I believe that Red Hat is as much a "freeloader" on open source as any other company is on the RHEL codebase. However, I would like to share reasons why I will never ever install Rocky Linux on my servers.
In this article, I would like to share some of the stories, misdeeds, and bad practices that explain why so many people do not like Rocky Linux. I also want to show that things are a bit more complicated than most Linux users think.
Story 0: The original founder of CentOS
The central figure in the case against Rocky Linux is Gregory Kurtzer. He is, for many people, the hero of the RHEL rebuild world. Many believe he is the father of CentOS, and Rocky is/will remain his beloved and, more importantly, free child.
The term "The original founder of CentOS" comes from Kurtzer himself and has been spread like wildfire by PR companies (more on illegal, in most countries, and shady PR practices later). The only problem with this statement is that, in the opinion of many, including myself, it is not true.
My doubts about Rocky Linux came exactly from this point, as one of my much older colleagues said that this was "not entirely true" (his exact expression was a “little” stronger).
I would like to present some basic but important facts:
- Rebuilds started on the RHEL rebuilds mailing list.
- The name CentOS was announced by Rocky McGaugh.
- The Kurtzer was not interested in building a Red Hat Enterprise Linux (then Red Hat Linux) clone.
- The idea of creating a true distribution from RHEL was not his and had been discussed much earlier.
- Well, you can read a bit more on HN. And it's written by quite a famous person in the Linux world :)
So why does Gregory Kurtzer claim to be the "original founder of CentOS?" Mainly because he started the Chaos Foundation, which took over the efforts to rebuild under a common name.
[1] https://www.mail-archive.com/[email protected]/msg07038.html
[3] https://web.archive.org/web/20040824032738/http://www.caosity.org/pipermail/caos/2003-July/000701.html - Gregory Kurtzer was not interested in building a RHEL clone
[4] https://www.mail-archive.com/[email protected]/msg00022.html - Idea of creating Distribution on it's own
[5] https://news.ycombinator.com/item?id=33907452
Story 1: The Departure
One of the main misconceptions about Gregory is that he left CentOS when, in fact, CentOS left his foundation. The people leading CentOS development didn't want to be part of Gregory's scheme. To this day, some of the "old guard" of CentOS are not "on the best of terms" with him.
Fifteen years later, the CIQ CEO is much smarter - but unfortunately for the open-source community, in a very bad way. So, when articles like "What happened to CentOS will not happen to Rocky Linux” make the rounds, I think not only about bad changes to the lifecycle, but also about the fact that CentOS was able to leave the Gregory Foundation, but Rocky Linux cannot.
Generally speaking, the person in question himself has a few versions of it; my favorite was presented by himself in the Changelog podcast:
I was associated with it until Red Hat sued me.
And then he said:
It wasn’t a lawsuit, it was a threaten of a lawsuit, which was enough in my book to do it.
[1] https://lists.centos.org/pipermail/centos-devel/2005-March/077502.html
[2] https://www.reddit.com/r/CentOS/comments/s77p49/comment/ht9v86o/
[3] https://changelog.com/podcast/427#transcript-87
Story 2: Foundation
You may know that Rocky Linux is part of RESF—Rocky Enterprise Software Foundation. But if you are familiar with "foundations," especially US-based "foundations,” you know that many of them are broken by design. The Rocky Linux Foundation loosely promised to be a non-profit. But it isn't. It's for-profit and it has a single owner (Gregory Kurtzer).
But there is more to the smoke and mirrors surrounding RESF. As you may know, they have governance and bylaws. So look at them and count how many people in that governance are not employed by or connected to the CIQ. By the way, the foundation broke its own statutes by not having a diverse governance :). I'm not going to read the bylaws again to see if they still do that, but if you find it, feel free to comment. Also, remember to read older bylaws and older about pages since it’s changed.
The whole story is strong in media and PR. Take, for example, the ITWorld Canada article from 2022
In literally the same paragraph, you have the following sentences:
The key for an open source initiative to grow and flourish,
Kurtzer said, is to is to register it as a non-profit organisation,
which is what the cAos Foundation. He did the same with Rocky Linux.
And then:
The Rocky Enterprise Software Foundation (RESF) is a Public Benefit Corporation (PBC).
Do you see the "little" problem here? I do not like to be lied to. And I do not like people being lied to, especially when it is such a primitive lie, literally in the same paragraph.
[1] https://www.zdnet.com/article/rocky-linux-foundation-launches/ Look at the manipulative language used in this article
[2] https://www.resf.org/about
[3] https://www.reddit.com/r/linuxadmin/comments/15p1gbt/comment/jvyat0h
Story 3: Trademark ownership
As I said, have you ever wondered why CentOS could leave the Greg Foundation? Well, one of the main reasons was that there was no clear ownership of trademarks/domains.
Unfortunately, for the open-source community, that's not the case with RESF and Rocky Linux. As I said before, RESF has a single owner, and the Rocky Linux project does not have its trademarks. Rocky Linux, as an individual project, if it tries to leave RESF, can't use its name, logo, and other trademarks without permission from the RESF owner. So, good luck.
So with CIQ/RESF milking Rocky Linux, there is no way the project can become independent. It's not a community project, it's a corporate project.
From this point on, I will use arguments against CIQ/RESF/RL interchangeably.
[1] https://www.zdnet.com/article/goodbye-centos-hello-rocky-linux/
[2] https://www.reddit.com/r/AlmaLinux/comments/15dr141/giving_rebuilders_a_bad_name_ciq_and_ansible/
[3] https://www.reddit.com/r/linuxadmin/comments/15p1gbt/comment/jw90zhc
Story 4: Buying advertising against community-owned open-source projects
These are the early days, and RL is losing to AlmaLinux. AlmaLinux is the first diversely governed foundation (and it is non-profit for real). So, what does CIQ do? They buy ads against several Linux distros, including AlmaLinux and CentOS.
Is this legal? Yes, it is!
Is this something that a "self-imposed non-profit" (this term is a joke) would benefit from? YES!
Is this what the open-source community would do? NO!
Buying ads against community-owned non-profit open-source projects while pretending to make one is against the general spirit of open source. Of course, Rocky Linux did not buy these ads; CIQ did, and they did it out of the goodness of their hearts.
[1] https://twitter.com/ChrisLAS/status/1460668344156114944
[2] https://twitter.com/ChrisLAS/status/1420123460982894599
[3] https://www.reddit.com/r/linux/comments/qv6mg2/comment/hkw046b/
Story 5: PR without disclosure
In many cases, the CIQ/RESF/RL use PR firms to spread the word about themselves. Sometimes, they even sponsor Linux influencers, and they tend to leave out some facts :) (like CentOS leaving the former Kurtzer Foundation or RESF being a for-profit company owned by one person). Anyway, some PR tends not to mention the obvious conflict of interest and that the article is sponsored.
And since it's very difficult, if not impossible, to verify the source of the PR, it's very hard to say whether the article is sponsored or not if it's not labeled. In my country and many EU countries, it's legal. Even in the US, it's not a legal problem; it shows poor ethics and morals of the:
- Writer
- PR company
- Publisher
- The company that sponsored it
As I said, it's extremely difficult to catch someone red-handed. However, in the case of RL and CIQ, it has become quite well-known in the Linux community.
[1] https://twitter.com/GordonMessmer/status/1675997483573612546
Story 6: Spreading FUD
One of the biggest damages done to CentOS Stream has been the FUD spread by CIQ and others. The main point is that CentOS Stream is unstable and will never be production-ready. As someone who regularly compares packages and distributions, I can say this:
-
CentOS Stream is stable and, in many ways, even better than RHEL, and that means it is even better than clones.
-
The number of engineers working on CentOS Stream is greater than that of any other RHEL clone.
-
The amount of testing done on CentOS Stream is orders of magnitude greater than any other RHEL clone.
-
The CentOS Stream gives users real power to fix bugs and influence the future of the distribution.
There has also been a lot of FUD spread about AlmaLinux. That AlmaLinux has taken "shortcuts,” especially when it comes to the subscription manager. If you're familiar with Subscription Manager, you know that it's designed to connect to Red Hat infrastructure, and Enterprise Linux clones don't install it by default. So, not including it in AlmaLinux is not a bad shortcut but a good decision. Later, it was added because it might also work with Foreman/Katello/Candlepin. The whole "AlmaLinux takes shortcuts" era was FUD in its purest form.
Unfortunately, I can't find the source of this ATM, but I've seen it, and I'm sure one of the readers will be able to add it in the comments section.
Other popular FUD spread about AlmaLinux, although I did not find CIQ employees saying it publicly:
- AlmaLinux uses CloudLinux infrastructure (they used it during the bootstrap phase, and it was very well-known)
- AlmaLinux is controlled by Russians (this one goes strong after Putler attacks Ukraine).
- CloudLinux controls AlmaLinux (not that CloudLinux CEO literally stepped down from the foundation; also, AlmaLinux has a much healthier board than RESF)
Story 7: Being fu****g hypocrites
So, you remember all that Red Hat drama about paywalled GPL sources? When Red Hat a "cancer" on open source? And users were "freeloaders"? One of the people who was very vocal about this was Gregory Kurtzer :). No matter what all the shouting was about:
- "Rocky going strong."
- "Protect the community."
- "Protect open source."
Or nice PR from RESF:
“I believe that open source should always be freely available and completely stable.
It should never be hidden behind a paywall, nor should it be controlled by a single company,”
states Gregory Kurtzer, founder of the Rocky Linux project and chair of the board of the
Rocky Enterprise Software Foundation,
And other big words that are not worth mentioning because when it comes to real action, they are not there. What's worse, they are doing the same or even more restrictive things than Red Hat.
Now, CIQ is giving back to the community by putting a paywall around the sources of CIQ Rocky Linux 8.8:
Restrictions. The license granted in this Section 3 is conditioned upon
Customer’s and its Authorized Users’ compliance with this Agreement. Customer shall
not and shall ensure its Authorized Users do not: (i) permit any third party to use or
access the Software (except for the Authorized Users as permitted herein); (ii) install the
Software on more than the number of Licensed Hosts permitted under the applicable
Order; (iii) share access to the Software (including log in information or notifications)
with anyone who is not intended to be an Authorized User; (iv) provide, license,
sublicense, sell, resell, rent, lease, share, lend, or otherwise transfer or make available
the Software to any third parties, except as expressly permitted by Ctrl IQ in writing; (v)
except with respect to any access to Software that is licensed under an open source
license, modify, copy or create derivative works based on any content accessed through
the Software; (vi) except with respect to any Software that is licensed under an open
source license, disassemble, reverse engineer, decompile or otherwise seek access to
the source code of the Software; (vii) access the Software in order to build a competitive
product or services; (viii) remove, delete, alter, or obscure any copyright, trademark,
patent, or other notice of intellectual property or documentation, including any copy
thereof; (ix) transmit unlawful, infringing or harmful data or code to or from; or (x)
otherwise use the Software except as expressly permitted hereunder
There is more. They made rebuilds of other RH products Red Hat Ansible Tower (or whatever they call it, ATM). They made it closed source. Then, after the community started noticing that it should be open source, after two weeks they decided to release code.
So that's it when it comes to open-source heroes and giving back to the community.
[1] https://www.reddit.com/r/AlmaLinux/comments/1aurwrr/comment/kri9xpc/
[2] https://www.reddit.com/r/AlmaLinux/comments/15dr141/comment/ju3t9mv/
Story 8: Borrowed content - erratas
So the CIQ/RESF/RL use Red Hat content, that's obvious. However, you need to remember that components have different licenses. So, if you are using the source code, the most important and, in many cases, only license is the one that is in the source code. If you are using Red Hat Portal or Support, the new restrictive EULA applies. If you use Red Hat errata, the errata text and content are protected. That's why some vendors have rather empty errata information, including one generated for CentOS. The most important parts of erratas are:
- List of packages
- CVEs numbers (for security erratas)
The text is not that useful; you should probably read the CVE anyway, but some projects cannot help themselves and cannot be used without any consideration.
Have a look at https://errata.rockylinux.org/RLSA-2023:6818 where you can read about it:
Rocky Enterprise Software Foundation Satellite is a systems management tool for Linux-based systems.
You can also find other examples, such as:
Rocky Enterprise Software Foundation Identity Management (IdM)
or
Rocky Enterprise Software Foundation Entitlement Platform.
Is it legal? I do not know. Ask a lawyer. Is it ethical? No.
Story 9: Killing Open Source - CIQ versus Sylabs
In late August 2023, news broke of a lawsuit against CIQ. Note that if I'm right, it's still ongoing. But for the moment, the court "DISMISSES all of Sylabs' claims WITH LEAVE TO AMEND."
I'm not a lawyer, but I am an engineer. An open source engineer who's heard "we won't open it because it might hurt our business" or "it makes us vulnerable" time and time again. And I never liked that, but as I got older and saw people like Gregory Kurtzer, I started to understand.
What Gregory did is the worst-case scenario for any company trying to build open-source projects.
- A company employs Kurtzer as CTO.
- This company makes software vital to its survival and expansion, just like any software company.
- Gregory was CTO when the company decided to open source its critical assets. (March 2020).
- Gregory leaves the company with extensive knowledge of customers, system internals, users, and contacts. As a C-level executive, he literally knows it all.
- Kurtzer started a new company, took part in the development team, and used it against the former company.
- Kurtzer profits from the old company code that was open-sourced during his time as CTO.
By pulling this kind of stunt, one is killing open source—killing trust in open source companies and developers. You work with people you trust, and you never expect them to be your enemy the next day. It leaves a bad taste in your mouth that's hard to describe.
Finally, I do not care about the future of Sylab, CIQ, or who is legally right. I care about the future of open source, and this kind of practice is a mortal wound to open source and free software.
[1] https://www.theregister.com/2023/08/24/lawsuit_claims_ciq_was_founded/
[2] https://storage.courtlistener.com/recap/gov.uscourts.cand.408847/gov.uscourts.cand.408847.59.0.pdf
[3] https://www.courtlistener.com/docket/66863695/sylabs-inc-v-rose/
Other stories
There are other stories, like why EPEL stopped doing count_me analyses for some time and why nobody takes them seriously anymore. Or about some key people officially working in public institutions and de facto doing work for a for-profit foundation. But I think the most important ones are covered. Some of these stories are only known to a small circle of people who go to the conference and know each other. Personally, I have left out some of them because it would literally say who I am or where I work.
Summary
When Red Hat "bought" the CentOS project, there were many red flags. We all know how it ended. Red Hat was then, and for many still is, the champion of open source. RESF/CIQ/RL was never there and never will be with its current smelly track record. And the sheer volume of red flags regarding them is by far more than it ever was with Red Hat.
If you read this and install Rocky Linux, you are not only a fool but a dangerous fool. You are the fool who likes to be lied to and is incapable of understanding the bigger picture.
I also omitted the technical part of the CIQ/RESF/RL. Perhaps I will return to this to this topic.
In the next article, I will cover the joke called OpenELA.