Requests are undeniably unmissable for anyone willing to start in the Python ecosystem. Yet, the community is broadly unaware of how sad the situation has become.
The Incident
Due to dramatic circumstances, it was partially locked and put in "maintenance mode" only, meaning it would no longer evolve (features speaking).
Some of you may have heard that more than six years ago, Requests was working toward a major version, namely the V3. There was even a completed fundraiser with a substantial amount, around $30k.
At the time, there was a plan to bring Async and HTTP/2 support (most notably). You'll guess that none of the exposed promises ever landed.
Impacts
This incident led to several impacts.
- Maintainers felt too much pressure to make steps toward a major; some of them said: "We just can't deliver the promises, and it's going hard to explain why".
- All the back pressure landed at urllib3.
- Abruptly started to refuse any changes, saying: "Requests is feature complete, and it's popularity makes it nearly impossible to fix bugs that may be considered feature to this point."
- Newer projects started ditching HTTP client Requests by learning to use a new one.
- Some large-scale projects bound to Requests did not migrate because of the potentially high cost of doing it.
Let's be clear, this article is not about saying "Requests maintainers are bad" but rather saying we don't agree, and everyone is entitled to their choices.
For many years now, Requests have been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.
The Journey Into Waking Up
Initially, like many other reasonable people, I said to myself "Let's wait, Requests will eventually unblock the situation!" Yet, the years are still passing, and nothing.
Let's realize something, Internet Explorer 11 just celebrated its tenth anniversary last year, and you’ll never guess that it has support for HTTP/2 integrated! Yet, Requests is unable to do so.
I said to myself: "Why are we waiting for this indefinitely? They don't owe us anything! There's a reasonable chance we are waiting... trapped in a circular reference… Let's act."
Then, I tried to get a firm grip on the urllib3 base code, contributing this and there until I was ready to kick things up with a proof of concept that would have put urllib3 far ahead. Without any breaking changes. I was delusional. This was a bit of a shock, but six months passed between my initial kick-off and my formal give-up, and here's why in a nutshell:
- Every bit of idea brought to the table was dismissed, and never seriously analyzed.
This felt as good as taking the coldest wind on your face while exiting your warm home. It's the hard truth of the OSS; it's not a democracy, and it's not ruled by pure logic, but by humans.
Not far later on, urllib3 called out a fundraiser to help with HTTP/2 support (only) and claimed around $40k to be able to deliver. They probably meant well, but[...]
This was the last straw for me, knowing the past mistakes, and actually the best incentive to break the cycle. Everything points to another year before having the slightest sign of a PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n into our production pipes. They clearly lack the time to move forward (or incentives? or both?). No worries, as developers, we should share this burden for the greater good.
From my perspective (my opinion only), the OSS ecosystem should not fundraise money for a decade-old protocol, and most of the time should deliver prior to asking. Entities raising money with no solid proof of concept; that's rare!
This felt like a bit of a hostage situation, but it could only be me seeing it this way.
The Solution
So many years and various expertise have been brought to Requests and urllib3… Do we throw all of it into the garbage disposal? Engineers should be reasonable scientists and not revisionists.
We don't have to reinvent the wheel all over again; HTTP client Requests are established and really pleasant in their usage. We believe that Requests has the most inclusive and developer-friendly interfaces.
So, how to propose an attractive continuity for Requests? How hard could this be to (i) bring a true upgrade experience and (ii) keep the compatibility as high as possible?
Sometimes, the OSS ecosystem acts like it is trying to land a rover on the Moon, let's be more realistic and bring our vision down to firm ground.
There's a chance offered to us to rehabilitate Requests (and urllib3) and give a fresh breath of air to us, the developers. Without having to learn something as basic as handling an HTTP client.
So, the work started, without looking back.
Just around my intent to give up on urllib3 evolution, we proposed a fork of both urllib3 and Requests, namely urllib3-future and Niquests.
They are (truly) drop-in replacements, at the time of writing, those forks are six months old! And the whole work around this is roughly a year old. We listened to what the community needed by... triaging all the issues on major repositories (Requests, urllib3, and dependants of them) so that the upgrade would be truly useful.
Are you seeing it coming? Great! Let's look at the feature table comparison against requests, httpx, and aiohttp!
|
Feature |
niquests |
requests |
httpx |
aiohttp |
|---|---|---|---|---|
|
|
✅ |
✅ |
✅ |
✅ |
|
|
✅ |
❌ |
✅ |
❌ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
✅ |
✅ |
❌ |
|
|
✅ |
❌ |
✅ |
✅ |
|
|
✅ |
✅ |
❌ |
N/A |
|
|
✅ |
N/A |
✅ |
✅ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
Limited |
❌ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
❌ |
✅ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
❌ |
Limited |
Limited |
|
|
✅ |
❌ |
❌ |
❌ |
|
|
✅ |
✅ |
✅ |
✅ |
|
|
✅ |
❌ |
Limited |
Limited |
|
|
✅ |
✅ |
✅ |
❌ |
|
|
✅ |
✅ |
✅ |
✅ |
|
|
✅ |
✅ |
✅ |
✅ |
|
|
✅ |
N/A |
N/A |
N/A |
|
|
✅ |
❌ |
❌ |
✅ |
|
|
✅ |
❌ |
❌ |
✅ |
And you would think... performance..? are bad?
Look at the performance comparison against them!
Scenario: Fetch a thousand requests using 10 tasks or threads, each with a hundred requests using a single pool of 10 connection.
High-Level APIs
|
Client |
Average Delay to Complete |
Notes |
|---|---|---|
|
requests |
987 ms |
ThreadPoolExecutor. HTTP/1.1 |
|
httpx |
735 ms |
Asyncio. HTTP/2 |
|
niquests |
470 ms |
Asyncio. HTTP/2 |
Simplified APIs
|
Client |
Average Delay to Complete |
Notes |
|---|---|---|
|
requests core |
643 ms |
ThreadPoolExecutor. HTTP/1.1 |
|
httpx core |
550 ms |
Asyncio. HTTP/2 |
|
aiohttp |
220 ms |
Asyncio. HTTP/1.1 |
|
niquests core |
210 ms |
Asyncio. HTTP/2 |
Did you give up on HTTP/2 due to performance concerns? Think again! Multiplexing and response laziness open up a wide range of possibilities!
And everything you are witnessing is in fact "Requests"-extended, and you have nothing to learn again to get started. We kept 98% of the test cases from both Requests and urllib3, then extended them to cover newer features.
We are looking forward to seeing you at the Niquests repository! Let's say it's a promotional offer, get everything and more for $̶4̶0̶k̶ $0!
-
#1 You depend on Requests? Discover Niquests instead.
-
Drop-in replacement, simple patch needed, tiny effort required.
-
-
#2 You depend on urllib3? Discover urllib3.future instead.
- Drop-in replacement, no patch required, atom-like sized effort required.
By visiting the GitHub repository and documentation, you will be able to learn more about this, and hopefully make a decisive switch!