github.com
Back to storiesRanked by HN
Comments
57 total · 57 loaded
As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
Nice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use.
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.
uv is the backbone of any modern Python library. I’m excited to see improvements.
Good article. But the Makefile part should really be replaced by tox as a best practice. Tox even has a uv runner now that will handle setting up all the environments.
I like just or mise tasks as often my python projects contain many other language commands (SQL, docker, pnpm, etc ...).
There’s a bit of a bootstrap problem there in that Tox (and its dependencies) have to be installed first, unless your makefile is calling “uv run tox” or similar. Uv’s standalone nature makes it ideal for bootstrapping projects in a way that tox isn’t.
Can uv even be used with C extensions now?
they just implement the modern PEPs
Nonsense. uv is one of many existing build tools. All the items can be done by other programs.
Funny to list OpenAI as an example at the end of the blog post ...
A 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.
Making things faster is much easier than making things smaller. I'm sure we can win back a 4% slowdown elsewhere since we have so many more levers to pull from.
Percentages aren't always the right gauge. uv isn't something I run frequently or for long periods of time. It's like a couple seconds every month or whatever. I would rather spend an extra second waiting every year to have hundreds more megabytes all the time.
When actively developing Python projects, running uv a dozen times per hour isn't unusual. Those seconds add up fast. Whereas freed disk space remains unused.
>especially when it comes at an increase in complexity.
Or (nasty) bugs hard to debug
Whereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage.
Disk isn't free, especially now.
Exactly my experience, even worse because for me it's 2x256GB drives. Eventually I bit the bullet and upgraded one of them to 1TB, but not the one my root partition is mounted on (too lazy for that), so now I find myself moving various large directories across to my "extra" drive and symlinking them. Funnily enough I don't remember this ever being a problem back when 50GB was considered a lot.
Because when 50gb was a lot, by the time you filled it, 200 GB was cheap and a non event.
Yea, I remember when the family computer had a 40MB hard drive and it was never ever filled to capacity. Now I’m wondering how many new disks to add to my ZFS pool, and when. And how to budget for it.
I don't disagree. But if you're in that position, surely you'd be better off emptying the cache and disabling it?
The things which use my disk are things like package caches in several languages, build artifacts in both release and debug, multiplied by worktrees, things like iOS device support and release files + simulators, docker containers and related.
I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there.
The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc.
Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
> deduplication at the file level: every file is now stored under its BLAKE3 hash
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)
Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits).
I wasn't very clear.
> I use it for my own "deduplication / integrity / berzerker" utility
Do you have it in a public repo you could share?
No sadly I don't have any public repo: it's mostly really a collection of shell scripts and then the database one (for the berzerker/renamer) is in Clojure and I run it from my always-on REPL.
But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.
Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.
It's annoying that most file formats don't checksum their own content.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.
Though fair enough, it could offer it as an opt-in thing.
> It's annoying that most file formats don't checksum their own content.
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
And backups. So much backups.
What is the point of using packaging software that belongs to ClosedAI?
Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.
> By using uv, you are literally supporting the people who want to make you unemployed for stock options.
By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.
It's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.
And it is machine like that many software engineers still cheer on being replaced once it is happening.
Software engineers are the socially dumbest creatures on the planet and deserve every bit of scorn that was heaped on them at high school. The industry used them for multi level marketing since about 2010 and now has instructed them to direct the MLM against themselves.
And they comply!
That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.
The AI cartel is the new Microsoft.
It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000.
That is another fallacy. The Internet is a common carrier and was supposed to lead to freedom in 2000. It has nothing to do with outsourcing your thinking to a bunch of wannabe trillionaires.
> That is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control.
"Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language.
And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree that big tech sucks, and many places in the world need less surveillance and less corporate control, but I don't think the entire LLM-space (besides very obviously the biggest companies) are responsible for that kind of usage.
Is 'tired' here in scare quotes or are you referencing something? And if it wasn't faster from the get-go, how did it address the fatigue?
The Internet relies on physical infrastructure managed by sovereign nations and multinational corporations, it would never be free from those constraints. Until we reach post-scarcity utopia and harmony between all of humanity—ironically, something that could be enabled by AI—the Internet will never be "free".
What is the word? "Facetious"?
Well, it's fast. Perhaps "they" want me and everyone else unemployed, but it's not immediate and not particularly certain. However, I can use uv here and now to greatly simplify my workflows, so I use it.
If you're going to be unemployed anyway, and they don't gain a single penny by you using it, then what does this protest gain?
I think `pdm` is much nicer. I think if pdm had become a think earlier people would have switched from poetry to pdm rather than this uv thing which is a lot of different things at once.
> Astral people are already in the comments making pro AI coding statements.
I can't see any other comment on this post mentioning AI, did you just make this up?
People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.
Maybe I've misunderstood you, but that's not a special advantage of uv. pip has been able to do this for many years.
pip install git+https://github.com/some-org/repoI have never gotten the speed argument either. Yes it's noticeably faster - but in all my projects it's 1s vs 0.1s, not enough to make a meaningful difference. Even in their benchmarks, the worst number they ever give for pip is 7s [1] (and those numbers are 2+ years old, with python's improvements pip should be doing a little better now). I just can't say I've ever cared about installing dependencies, something you do once, taking 7s longer. Nor do I care how long CI takes, it running in the background is the point.
The one use I can give it is for running scripts with inline dependencies. I have found it a noticeable improvement over pipx there. But that's more due to needing to parse dependencies every single launch, if it's something I use regularly I end up just installing them normally instead, and it's faster than either.
I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but uvx was not great. On the other hand, it did seem to install the packages faster than pip.
>I didn't find it fast at all.
Compared to what? Fast is relative. Compared to pip it's miles ahead.
YMMV depending on specific needs but generally even on Windows on an average machine with an imperfect setup I've found it exceptionally fast, often well inside the sub-second range, to the point that I do sometimes worry if it really ran.
Sounds like it's worth another look at your settings to make sure they are right.
I do avoid uv run for this reason but it’s useful for managing python projects. The speed claims actually have a lot to do with efficient caching, and I run several projects on the same Python (patch) version with similar packages on the same system
uv is the backbone of any modern Python library.