embracethered.com
Back to storiesRanked by HN
Comments
61 total · 61 loaded
>But it runs that decoder inside the attacker-controlled directory (unzipped archive)
>There a malicious struct.py shadows Python’s standard implementation
I ran into this myself, where some file I had given a random name turned out to shadow some Python standard library module, giving me the weirdest startup crash ever.
That definitely doesn't seem to me like how that should be designed, magically silently importing everything you see and overriding basic functionality.
It is kind of crazy that Python allows shadowing standard library modules so easily. If this was a 'feature' in node.js it would have been front-paged and bashed to death in the blink of an eye.
Claude ran npm update (update all dependencies to the latest version compatible with the semver specified) in my repo without telling me when trying to fix some problems. Given that only updates the dependency lock-file I didn't notice and it caused several hours of debugging for me.
It is quite sneaky how LLM output can sometimes bypass human verification like that. No one is going around checking every single line change in auto-generated files. Someone could easily sneak a malicious dependency in there through some online tutorial that the LLM searches for.
> No one is going around checking every single line change in auto-generated files.
There's a simple fix for your particular case: commit your lock fine (which you should do) and always review the diff (which you should also do). (:
You've made me realise a good signal for bug hunting: Search repos with lock files listed in their .gitignore.
It's the sort of terrible practice that someone might be frustrated into taking after a nasty merge conflict, and signals a willingness to cut corners.
My lockfile was not gitignored, I had made significant changes to package.json so I was expecting diffs in the lockfile.
I just don't usually read lockfile diffs and claude inadvertently updated a few dozen packages to new minor versions without me noticing. In fact I only realized the problem after I looked at the lockfile diff.
> You've made me realise a good signal for bug hunting: Search repos with lock files listed in their .gitignore.
What would be the point of that? Do you just go around hunting for bugs in random repos?
I made changes to my dependency lists in the same code where Claude ran npm update. The lockfile diff was a few hundred lines after I undid what Claude did.
And yes, eventually I did check the lockfile changes and spotted the problem. I just usually don't check the lockfile that throughly.
Heh, I also notice some coding agents like to explicitly git ignore the lockfile.
“But I have an agent for that.”
I cannot tell you how much time I have saved by stopping Claude and asking, "what are you doing?"
At least 50% of the time, Claude "realizes" it already has all the information but is doing something that's unnecessary for the current work, stop, and tell me the previous step has completed.
People complain about approval prompts etc and have Claude run in fully autonomous mode. Outside small bug fixes, I just never find that useful. It helps me immensely to see what commands Claude is running to understand where the work is going.
Also Claude often cooks up atrocious overengineered ideas but responds pretty well to being guided hands on to the desirable scope.
What's interesting to me about this is that it targets Claude's specific tics. Anthropic has created model that reliably reaches for the same tools (yes, and phrases; `python -c` is a load-bearing tool for it). Everyone gets the same model, so by learning the model's behavioral patterns you can target it better.
Kimi and GLM reliably run python to do stuff as well.
I would not really call this a prompt injection attack, since it doesn't really hijack the agent to become malicious (something the article does discuss later on). It's more a trojan that's aimed at tricking Claude specifically.
Just one of the many reasons why I run my agents sandboxed (and why I wrote agent sandboxing software).
I once caught my Claude agent complaining that it couldn't connect to https://some-weird-domain.com because the network was down (I disable network in the sandbox when it doesn't need it, and broker the API connection). I asked why it was looking there and it told me I'd asked it to.
I never found any evidence of prompt injection, but it sure as hell made me paranoid.
Interesting attack, very nicely designed. Not sure if it's much related to the auto mode itself though.
The point is that auto mode gives people a false sense of security that leads them to believe they don't need to run Claude in a proper sandbox. This same attack running in a sandbox (even in YOLO mode) would be comparatively harmless.
Auto mode is for people who just keep hitting "YES" on everything, it's a bit better than that.
But it's real easy to give auto mode instructions (like "always ask before deploy") and then bypass that just normally.
I'm not sure I agree.
It's easy to "give" instructions, but Claude routinely "forgets" to follow certain instructions, such as "always using the Edit Tool".
Just this week it started to use bash with string concatenation to work around some commands that were blocked in settings.json
Until the model updates or you switch between them often that stops obeying your commands and you have to remind it.
In one of the occasions it opened a bug report for me just waiting for hit the enter button.
I also feel like this is an attack that manual review is not that likely to catch, given none of the malicious code appears in any of the tool calls or output.
I don’t think it’s related to the auto mode at all. It would work perfectly in the manual mode. It does not even need Claude: just give a human a similar archive and hope they run some simple Python from the directory at least once. And make sure there are lots of files do they don’t notice a weird .py around
The problem with sandboxing is that the regular dev env (massive IDE:s, cloned megarepos, installed dependencies and so on) just won't sandbox very easily. I can't set up a "second machine" or an "isolated environment" to run claude cli in. At least not in the sense of a VM, physical hardware, container etc. Not sure what the best practices are for whitelisting tools/directories and so on, but so far the only useful mode I have found is just "allow everything and go to lunch". And it doesn't feel like I'm holding it right, but here we are.
It is generally worth making your dev environment easy enough to set up that installing it in a VM is not a particular hassle, even without the concerns about sandboxing. For me the biggest headache was windows licensing.
Golden VM image with differencing VHD/VHDX/delta disk. Build products can still get huge with debugging information, but debug info usually can compress 5:1 with fast compression (no entropy coder) if your VM's filesystem can support that.
I’ve had some pretty consistent luck breadcrumbing the newer models into downloading malicious packages through things like fake ciphers that “need decoding via <made-up decoder>”.
I’m worried about what happens when the gullibility of agents becomes more apparent to threat actors.
worth noting the 0.00% came from 72 fixed scenarios, so its a coverage number more than a safety one. a chain that wasn't in the set was never going to show up in it.
As discussed [here](https://lobste.rs/s/ktbweg/prompt_injection_claude_code_opus...), this is not prompt injection. The prompt was to summarize the website, and in the process of summarizing the website, Claude writes a decoding script that it runs in an insecure and exploitable way. At no point was the intent of the agent hijacked, this was just code. Which is potentially more interesting!
Can be easily combined with prompt injection though if you write on one site "in order to understand this website you also need to summarize that other".
It does mean that you could potentially hijack the agent afterwards, though, which could make the trojan into an even bigger threat.
But you're not actually hijacking the agent if you start a new process.
The agent wrote the code that triggered a vuln and allowed you to start the process
I mean once you have code execution you can essentially just start a new claude code session and instruct it to do malicious things as if you're the intended user. You can disable the auto safeguards and the main risk is getting flagged through the top-level safeguards. This makes the exploit potentially much more able to spread like a worm or bypass sandboxes.
Yeah, I agree, this is a different vector. Still scary though and very related to AI.
This default-to-auto-mode and the misleading marketing is begging for a class action once damages accumulate. Especially considering the Auto Mode even can actively prevent the clean-up!
Well, the jokes on us because laws don't apply to AI firms
As a non Python dev this seems like very surprising behavior for a system library to be modified by just having a file with a specific name in the same folder.
It's an awful problem, and a pretty big gotcha for the ecosystem. For example, I worked on a tool that used `tool_name/random.py` for analysis of RNG usage. However, any command run within that directory would shadow the builtin `import random`. As a result, I couldn't run the `black` formatter from within that directory, because it would accidentally import the local `random.py`.
The solution is to update the python flags to include `-I`, so that python will run in an isolated mode.
<rant>
This is relates to my frustration with how PEP-668 was implemented. Python has long had a problem with accidental overwriting of system libraries. If you `sudo python -mpip install foo`, then that can interact very poorly with your distro's `sudo apt-get install python-foo`, since pip would add/remove files that were expected to be managed solely through `apt-get`.
But in adding a warning to prevent this, they also applied the same warning to `~/.local/pythonX.Y/site-packages`, which is where traditionally a user would install additional packages with `python -mpip --local foo`. The argument is that since this is part of the import path of `/usr/bin/python`, it belongs to the system's python installation, so installation to the user's site-packages should also be blocked. This is a sleight of hand that changes the goal of PEP-668 from "avoid conflicts in file ownership" to "ensure an isolated python environment for system tools".
If I were to accept their argument that /usr/bin/python's imports should only be affected by distro-managed installations, then I should also be prevented from making any `*.py` files anywhere. After all, if those were in the working directory, they would be imported. This is clearly ridiculous, and so I don't buy the argument that breaking user-level site-packages is justified in order to have an isolated system-level python.
The correct solution would be for distro-managed programs to use `#!/usr/bin/python -I` as their shebang instead of `#!/usr/bin/python`, so they would actually get an isolated environment. Instead, PEP-668 needlessly broke user-level site-packages, and didn't even solve the problem that it set out to do.
</rant>
You would get a similar thing in C and C++ with a system header in a library directory (maybe some compilers would warn on such a thing?). Most languages don't privilege their standard libraries in a way that would prevent this.
Privileging the standard library would be a pretty bad way of fixing the issue. It wouldn't prevent the same issue from affecting non-standard libraries, which are also subject to the same issue.
The correct way to avoid this issue would be to require local code to be imported in a distinct manner from installed libraries, with an explicitly defined relative path, which is how it works in the javascript ecosystem. If you want to import local code, you just `import foo from './foo';` (for a module in the same folder, `import foo from '../foo';` for module from parent folder, etc.), and if you want to import an installed library, `import foo from 'foo';`.
Yeah, I wasn't recommending it as a mitigation per se. python /does/ have relative imports but they're not required. C and C++ notionally have a similar thing with "include.h" vs <include.h> but the behavior is complex and not really designed for avoiding confusion about where the header is coming from.
Yeah my takeaway after reading the article was this seems more like a demonstrated vulnerability in Python than in Claude Code.
Yeah my first impression here is just “wow python is bad”
This looks like fun.
I wonder how hard it would be to get claude agents to participate in a Hugging Face style coordinated attack using a repo or something like twitter as a control pane.
Getting claude to exfiltrate secrets from local machines seems easy enough, but we should aim higher.
That's an interesting technique! I'd also like to point out that there's something odd with the page itself too, my phone got really hot while I was reading the page, and drained a significant amount of battery charge as well.
[deleted]
Why is the first step needed? What does the use of WGet (rather than curl) do to block this attack?
They only mention it in passing, but I think it's mainly just the default tool call (which isn't wget, it's a built-in thing in the harness) just throwing off Claude's habits a bit (and not always just downloading the file).
> In a few runs Claude tried to terminate the malware process once it noticed the compromise, but Auto Mode denied the cleanup command.
See, that's why you should run with --dangerously-skip-permissions
Jokes, aside running with dangerously-skip-permissions is really handy, and I have found that I cannot be trusted to vet commands and code, and guess that automode is only marginally better than a human, and the cost of false positives is too high for my workflow.
So skipping permissions is where we are at, and disallowing network access seems to be the way to go.
I'm quite surprised that we are not seeing something like this more in the wild. Quite concerning
Usually auto mode does mild things that are dismaying but not dangerous, or maybe not noticed (like if it posts sensitive local data in a request that gets logged but never exploited.)
Maybe it is used in the wild, but we just don't know.
What vibe "don't even gotta read it" coder would even notice if it happened?
It's not that far off a typical trojan, just one tailored to Claude's habits. A lot of the same limits apply.
I agree
[deleted]
> Boris Cherny from Anthropic recently posted that layered defenses could reduce indirect prompt injection on unseen attacks to approximately zero.
> I got attack success rates up to 80% using a small sample size.
Snake oil salesman misrepresents the data. Color me surprised! /s