It is absolutely insane that it just says “ran a command” and doesn’t even show what it is
wouldn’t want to risk the user learning how to do something with it.
Claude shows you what it does, and only runs what you allow, either on an individual basis or by wildcard rules (per project and global).
The guy said YOLO and gave all the permissions because manually approving commands was too much work
The penalty should be: you delete my project, I delete your model, except that’s a lot of learning loss so they go Cylon with it.
And now the question - who’s accountable for that mistake?
Can’t exactly fire the model… I mean you could just stop using AI, but when is a company executive ever going to suggest something so reasonable.
If you shake a magic 8 ball after asking “should I go on a murder spree”, how good is the legal defense “but, the magic 8 ball said yes?”
In this case, the magic 8 ball went on a murdering spree by itself; are you responsible for asking the question and shaking too hard? Probably.
If I set off a Rube Goldberg machine whose final output is to detonate a bomb, I don’t get to show up in court and say the machine did it.
True, but if that final output had not been known to you, and/or you had set off the machine that triggered the bomb under duress, that changes your liability.
No. If it’s within the power of the device, method or whatever you do have the blood on your hands. (at least morally)
However. In business terms, since it’s management nothing will happen.
To put it in simple terms, if you ask someone to do something for you and you agree that anything is allowed to achieve this, you’ve agreed and gave your consent. In technical terms for AI this is especially fatal since you can technically not safely permanently guardrail an LLM.
That “and/or” is splitting up wildly different cases. If you drive a bus through a red light because you thought the intersection was clear, and it turns out it wasn’t. Is wildly different from the scenario in Speed.
Be careful it doesn’t start flying around and pull out its spears and drill like in phantasm
If you throw a spinning chainsaw into a room of bunnies, the chainsaw is not at fault for the bloodbath, you are.
Likely the user who set the agent to auto mode without confirming potentionally dangerous operations, and the infra lead for allowing users to access the prod db.
Shit, I set up my agents to only run in a sandboxed workspace, only suggest edits via merge requests that have to be manually approved by an actual person, and only connects to copies of databases. And that’s just for my personal projects on my home server. Anyone not taking similar steps in a professional setting should be fired immediately
Have you considered that management made AI usage a KPI, the feature has a deadline of next sprint, and setting up a sandboxed workspace is still sitting at the end of the backlog as a “nice to have”? /s
Reader roles are important, and should be the only access you allow your coding client.
Anti-AI people want to act like situations like this are indicative of a core problem with the AI, when if anything the AI just surfaced preexisting core issues in security, architecture, and developer training/habits.
‘The AI tried to delete production’ and ‘the AI was able to delete production’ are two separate problems. The AI doesn’t get a pass just because you also had a security problem.
I don’t think the AI ran
npm run destroy-production --irreversible. Most probably it rannpm run testand the .env happened to point to production, which is 100% on the dev who put it there.The AI isn’t a conscious entity, it’s a tool that does it’s best to carry out human instructions, based upon inference (and if you’re not an idiot, immutable guardrails). If it deleted production, let alone was given access capable of doing so, that’s completely gross human error.
Y’all are like children who get mad at inanimate objects.
I meant it more in the sense that the AI was a shit product that isn’t fit for purpose, but you show those strawmen who’s boss.
You seem to know nothing about the principle of least privilege, and I’d venture a guess engineering and dev tooling. If you are a software engineer, you’re exactly the kind who’d delete a production database.
On the most capable model with high reasoning. Whoops.
“reasoning”
destructive integration tests?!?!?!?!?!? on an actual db???
Is this an old and known dumbarse issue or do we have new dumbarses having their production data made gdpr-friendly by their favourite chatbot?
Approve for me
😂
AI is like a draft horse. It can be useful to pull heavy things under some very specific circumstances and under strict supervision and control. But you must keep it secluded inside of a pen with sturdy barriers that it cannot get past even if it wanted to and never let it access anything else than a copy of what you want it to work on.
You absolutely don’t want it to be able to roam freely because one day it will completely ignore anything you told it to do, go into your house, destroy everything and shit on the carpet.
And unlike the horse, no matter how much it pretends to express regret, the AI will remain just as likely to screw up the exact same way again.
Aren’t draft horses bred to be gentle and biddable? AI is like a raging bull.
Would you keep one in your living room?
A draft horse? For an hour if she had just peed and pooped. Not sure if my cat would love or hate her. I’d want her owner to come too, so she’d feel calm about it. After all she’d barely have room to turn around.
A raging bull? No thanks.
Well I wouldn’t. If AI is worse, it says a lot about letting it have read/write permissions over your system now doesn’t it?
Oh, not AI. Never AI.
I would let a real-live draft horse into my apartment living room if for instance it and its owner needed shelter from ICE. They’d never suspect, because I’m one floor up from the street but there’s a back way through my building which doesn’t require an elevator.
But never AI.
