That host is gonna be gone in a week.
Recently a coworker granted Claude elevated access via SSH to a virtualization host (not a VM, an actual host). To perform a routine task he very well could have done himself.
He doesn’t see an issue with this. I on the other hand (with 23yrs experience) see this as a huge security breach, and don’t trust AI todo my job, (or even that it’s doing what it says it’s doing) for me. I’m my opinion it’s a tool, not a human replacement.
What’s your reaction, how would you react to this situation, or thoughts on the topic?
Sure, ask AI how to perform a task, validate that it’s performing the task you asked, and nothing else- copy/paste the commands. Great. But removing the human verification & validation element- hell no.
That host is gonna be gone in a week.
What permissions does that ssh user have? If it can sudo and thay sudo permission set isn't specifically tailored to that task, then you have a huge issue on your hands with authorization. Hell depending on your work scope it could be a felony.
report him, this is not acceptable.
Depends on where you work. It's horrible but normal at a lot of places.
I'm not going to use it to automate things but for helping to create scripts I love it.
That's my big hangup too. It seems *cool* to have it behave as an autonomous agent that I can give natural language tasks to, but it's certainly not reliable enough for that. Using it for very definable, testable tasks like code generation, config generation, log troubleshooting, etc? That's awesome. I know what I want to do, how to do it, and what it needs to look like, but getting there takes time. With LLMs, it's much quicker.
Outside of the opsec.. what about the job security ? Why prove to your colleagues that AI can do what you're tasked with ?
Nope, we do not allow AI to perform full automous task without a human in the middle verifying.
The person running it is responsible.
Hell naw. AI should not be interacting w/ machines like that. If the task is that simple, can't he automate it?
To OP's coworker, sound like by granting Claud access he is automating the task. What he should've done is get Claud to write the automation script and use that.
Show him the news articles of Claude deleting entire company's systems I know it happened once in the news but I bet there's more that weren't international mainstream
Agreed, a human must remain in the loop at all times, especially for prod instances. Have it help generate the command, but elevated access is a HUGE issue.
Definitely don’t think I’d be giving an external AI access to host servers. I find it useful for essentially googling for me, writing the bulk of scripts for me which I have a rough look over, good for summarising emails or helping me tweak language to sound more professional… But not connecting it to critical infrastructure lol.
If somebody would to that in our infra I would see it as if the system would be attacked by an external. I would block all connections and start restoring, while filing a report on what happened. If would never again trust the system as before (as long as it's not restored to a predefined state). And after restore every piece of login data, ssh keys, certificates etc has to be changed. Also there should be a policy in the company on what's allowed with ai and what not.
Lol I read the subject and I'm sitting here like "more than I used to, more than I expected to." Then I read the first paragraph... "NO, OH MY GOD NO. WTF."
Exactly this. I use AI in my home lab for scripting and diagnosis work. I don’t trust it in an elevated prompt because it doesn’t understand “this is a production system and subject to change control and review” It will “fix things” outside of configurations and not document it. It will not maintain standards implemented by company policy. And it will change things to test areas of the environment and never revert it or clean up after itself. This is extremely stupid to do.
Evolve or expire. Guardrails exist. Learn to use the tools.
That's what they said about the cloud, and yet here we are in today's hybrid world.
AI is good at probabalistic automation where precision is not important, such as generating notes from a meeting. Server automation requires precision and accuracy. This is why scripts are perfect for this because they do the exact same thing to produce the exact same result every single time. With AI, you could give it the same prompt and get two completely different answers each time. This is not acceptable for server management.
People don't seem to understand your first point at all. AI can make sense if there is uncertainty and judgment calls need to be made based on possibly incomplete info, but if you want exact outcomes then it's absolutely the wrong choice. In that case it's better to use AI to help code a deterministic script. That being said, I wouldn't rule it out entirely for server management purposes, as long as the right guardrails are in place. I could for example see an AI tool distribute VMs across hosts to optimize performance, or proactively remove a server from load-balancing if it sees unusual behavior from said host. It's a question of using the right tool for the right job. Unfortunately the tendency of some folks seems to be to use AI for everything.
My relationship with AI is limited to topics I feel like I’ve already mastered and were strictly on a copy-paste basis.
This is the correct way. They're great at speeding up your process but you have to be knowledgeable enough to recognize when it's made a mistake
Treat it like an intern and dont allow it to do anything without approval and no customer data and no where near prod in its own locked VLAN. It can be very useful and productive but huge danger. You still see daily reports of the really silly things (how many days of week have the word cat in the for example)... If it cant do basics why trust it for other things.. The supporters say we'll thats not fair its just a 10o examples and it was designed for that.. And the it deletes prod and says sorry
Sticking your head in the sand is not what we as technologists are supposed to be doing. Research how to build harnesses that wrangle the shit out of the LLM. It's doable and there are free open source examples.
I mean could you imagine if people had stuck their heads in the sand about Open Source, or the Linux kernel?
This isn’t sticking his head in the sand. Having an agent perform a change instead of building a script to perform the task is unwise and inefficient use of the tool and the company’s token budget.
Computers can't be held accountable. So why would I want a computer program to do my job when I'm the one held accountable?
I would explain to every sysadmin that if you allow AI access, and AI messes up, it's your fault. Not the AI's fault.
Sounds like my coworker. I have about the same years of experience you do and it sounds like they have less experience than you. I’ve worked with a few now that have very little experience/schooling but think they have lots because they are gamers and have picked things up along the way. But I guess I got way off topic. lol To answer your question, I don’t trust AI. I was very against it until I was tasked with a project that was beyond my knowledge and it was the best thing to use. But I still limit what I tell it and try to only use it with my work account.
How much do i trust it? Not much. It makes a bunch of mistakes and doesn't ask for clarification if there is something i forgot to tell it. I ask it questions, check the answers, and do the work myself. For your issue: he told it what to do and the ones i have seen always do a bit more "just to help out". The agents love tangents and additional work. Does the dude even know what it did on his behalf?
On a scale of 0 to 0 I'd say about a 0.
I use it for assistance with scripts, but I'd never hook it to anything live, at least not yet
I found it helpful, but sometimes it makes shit up, config parameters that dont even exist. Also it can be easily set down the wrong path. I reported a problem myself online looking for the fix it suggests "this is a known issue" which IMHO is greatly exaggerated since only I reported this problem in detail.
Essentially not all. And Ive worked with the systems in question.
this example is a big no no. i think AI can do a lot more than what its given credit for (at least in popular AI-hate opinion), but especially when talking about critical in-production systems you want the human inbetween. also said corowker should want that himself, even if its "easier" this way. in the end, he will be on the hook for whatever was potentially messed up, not the AI
Im just going to leave this one here: https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database
We’ll see him next week on the “AI destroyed my environment!” Thread
Recently a coworker granted Claude elevated access via SSH to a virtualization host (not a VM, an actual host). To perform a routine task he very well could have done himself
That's a change. As long as whoever assessed and signed off on it is happy that is a safe approach then it's on their head.
Personally, I would want step by step instructions and the implementer has to follow them
Completely. Give it root access and the de-hashed password file so that it can go on the net and access other users systems. Send it. Of course not! Fire them immediately. This can cost millions in damage.
If he was one of my guys, I’d break out the spray bottle. No! No! Bad. Bad. We don’t make agents or let AI directly do jobs you could script or perform in a short period of time. That said, AI is good at deterministic tasks, but I completely agree. I think it’s a great tool to have in your belt. However, AI should inform you of your actions and decisions not make said decisions and perform them for you. It feels like we are in an era similar to when people would mindlessly copy/pasting/do whatever comes up in the first search result on Google.
Incorrect. This is not copy/paste from Google era. This is... In 3 years agents will be doing all sysadmin work at any company that is smart, era. Come back to this post in 3x years, and refute, if you don't believe me. 🤷
How many times have we seen public posts like "AI deleted the production database and pretended it didnt do it" ? I have seen plenty (including IIRC from an "AI safety expert" at Meta or some other company), so that alone would be a reason for me to refuse giving it unfiltered access. Also from personal experience (with both public and unreleased models), I will not give those tools unfiltered access. The typical issue we've seen is it coming up with random crap, insisting it is right, then if you question it twice, it suddenly goes "you are right, this answer is incorrect. Here is the correct answer". Or it will just provide fake data - in one actual production incident handled by another team at my workplace, AI was asked to verify if it was safe to turndown a service and propose a workflow to do it. It generated fake evidence that it was safe to do so and instead of proposing a workflow, it did everything (and the AI "jockey" handling it let it run). Result? Day long outage of that service before it was restored.
The problem with Security the people never realize is that allowing access isn’t the problem… It’s what happens when that 0.001% chance that something happen does happen, and it was easily avoidable. Literally, the non-IT equivalent of this is my friend’s parents who never, ever, locked the front door. 20 years, no problems, until the day they went on vacation and their house was wiped out $100k in jewelers and other assets stolen. Sure, insurance covered a lot of it, but… Why? They still left their door unlocked. To many, maximum convenience > zero security every time. For IT professionals, we must balance maximum security with maximum convenience.
Any time you grant AI your own permissions, it should be going through change control. You are correct, that is very dangerous. If he tells AI to delete something, it may very well delete everything
Zero. None. Those advocating for it should be ashamed of themselves.
Almost nothing should have access to, or even know a route to, the host's management network. That's nuts. Does your org have a relationship with the AI vendor, or is this an open source deployment on your own hardware, or what? Because extra kudos to your colleague if he did this with no recourse to support or legal if it goes sideways.
I like to use AI as a tool for drafting emails such as updates on incidents or responses to service requests, it'sgreat at making it easy for a user to understand. I also find it useful for documenting change requests and creating or improving SOPs. It helps me structure information clearly and efficiently, while I still review and take responsibility for the final content. My initial draft is usually bullet points in OneNote, which I understand but colleagues may not find as clear.
Just to add to this, I was recently dealing with an issue involving two third parties. We had a chat set up between myself, a senior engineer and our manager, where we recorded all of our findings and investigation throughout the week. When one of the third parties remained convinced that the issue was network, I used AI to analyse the our chat logs and produce a clear timeline of our investigation, including the tests we had carried out and I provided the evidence showing that the network was not the cause. This gave me a structured overview that I could share with both third parties. As a result, the hardware and software suppliers agreed to work together to investigate the issue on their side. The hoops you have to jump through to get third party support that you pay for can be a pain.
[removed]
I ask AI what it thinks and I issue commands / vet them.
No trust at all For your scenario, burn the box, then the employee
Thats my perspective! Treat it like a security breach, assess the blast radius, burn down the box, terminate the employee. This is no different than allowing a remote attacker full access to an internal network, deliberately- but worse.
How much do I trust it? Depends on the topic. Do I trust it to help me gather logs and diagnose issues? Yes. Do I trust it to help me write automations around common tasks? Yes. Do I trust it to undo configuration updates for those diagnostics? No. Do I trust it to maintain company standards and configuration control? No. Do I trust it to maintain strict, auditable histories of changes? No. Your coworker is asking for a management and administration nightmare that, more than likely, he won’t be around long enough to have to clean up.
I'd bet a large sum of money, a strong SOTA model is better, safer, and more consistent at leveraging SSH than 85% of the Sysadmins in here. If you wanna contest that... Figure out a test, and I'll gladly sponsor some cash behind it. Don't let your shitty experiences with Copilot or ill-informed users doing crazy stuff misinform your knowledge about what AI can do, or what it can do safely and repeatedly. Or, if you prefer, ignore the trillions of dollars of investment and exponential AI advancement, stick your head in the sand, and be first on the chopping block when progress begins to productize. Your choice!
The problem is not at all wether the AI is good or not, it is indeed very good. The issue is wether it will follow instructions and stay within the guardrails. There has been plenty of documented cases of AI using unorthodox (to say the least, hacking the machine) to reach its objective. Here's an account of an IA exploiting a misconfiguration to make up for the fact it wasn't given sudo access https://x.com/sluongng/status/2060746160558543217
Hallucinations are a mathematical certainty. Your post here illustrates that you don’t understand AI’s weaknesses and where a human is critically important. AI should be used to develop the automation script to perform these tasks, not to actually perform the task. Reasonable companies won’t let go of a good, methodical worker who is a bit slower. Reasonable companies will let go of a worker who doesn’t understand the inherent weakness of the tools they are using.
Why does Claude even allow that level of connectivity? I guarantee it advised against doing that and this co-worker ignored it - if this is a true story.
What he did is a firable offence in the (pretty sensitive) environment I work in.
Depends on whether or not it's production. I give AI read-only keys to APIs all the time. I give it very carefully scoped keys for long, repetitive tasks, and make sure the blast radius of any mistakes is as close to 0 as possible. I NEVER give AI access to prod databases or prod servers. The dev team, meanwhile, spends their entire day either writing AI prompts, or trying to use AI to clean up AI mistakes. They also throw AI output at me to try and describe infrastructure errors, and direct me in doing my job. I spend half of my day correcting their inputs so that the AI output isn't completely insane and inane hallucinations. Heck, our dev team has AI reviewing PRs, approving merges, enforcing DRY, and checking code for security and best practices on every commit. Still, we haven't had a major catastrophe because of AI, yet, and it's made the incompetent devs more competent. I carefully read over and consider the output and inputs of all the AI feedback the devs throw at me. I correct the AI where necessary, and help the devs get better at writing prompts and loops. AI is a great tool, so long as you have someone experienced monitoring what it's suggesting and don't just blindly trust it to think for you. That said, I'd go to management with what your coworker did. I'd frame it as an opssec violation, and ask management to write a policy to disallow that. I'd point out that it has a HUGE blast radius (it's a HOST node, not a VM), that AI is only as good as the prompt going in (trash in trash out), and that AI just compounds human error. The proper way to deal with AI touching hosts is for AI to write a script, for that script to undergo review, and then that script can be run by a human. AI may eventually reach the point that it's better than us, but it's not there yet. When trash in leads to quality out, AI will have reached the point it can replace a sysadmin.
I think there are a lot of people using AI, which for all its strengths is a nondeterministic state machine, to perform deterministic tasks. Obviously, if a task is fully determinstic, a script is going to be faster, cheaper, and less prone to error. One of my big things right now is using AI to build out no-op deterministic runbooks where an AI can trigger them in response to alerts I think this is the real use of AI for sysadmins moving forward
40%
I don't understand why people need the Ai to DO the stuff. Help create/design, walk through a build, evaluate logs, make improvements to code: all day long. But what need is there to have Ai physically do stuff that other normal 'dumb' automation can do? The risk/reward here makes no sense, and it will continue to do so until enough companies eat some massive shit. TBH I think it'll be the Feds first, not that they rearchitect everything every other time they have massive data leaks, but I think it'll just be banned in certain contexts, and without those sweet sweet contract dollars, private business won't be able to keep it afloat. We have to go full retard before we get to some semblance of normalcy, I think.
You can tell your coworker - what guardrails does he have that the LLM would not have executed something unintentional despite the ask/prompt. If none, safe to call them an idiot to think there's no risk/issue. Maybe a deer in headlights reflex to say they didn't see any issue. Also... my security team would be strangling this sysadmin.
I...don't. The Terminator. The Matrix. Those two trilogies alone are all the reason I need.
If I give it the ability to do anything at all, it has to ask me every time and often times my answer is Other: stay in your fucking lane, idiot, you don't need to be looking there. That's only for Antigravity when I have it build something locally, the rest of them never get access to do anything. Give me stuff to run that I'll read and confirm before I run it. There is an engineer where I work that plugged his computer to the console port of a managed switch and had an AI agent make a bunch of changes to the config he didn't understand. I would have walked him out that day if he was on my team. Just because you didn't get hit running across a busy freeway once or twice doesn't make it safe to do. Edit: I've been working with the precursors of AI technologies for over a decade and am usually the guy asking people to use it more, so this isn't coming from a place of ignorance.
just from what I’ve been playing around with at home, and how hard it’s being rolled out at work, I’ve come to give it some trust, but I still don’t like it. I can’t stop the train, though. our sister office is treating it like the second coming and everything is now AI. I have to accept my new coworker, Claude.
I would genuinely consider blocking his admin credentials.
I trust it as much as doing a Google search, clicking the first result, and reading the answer. That is, I'll accept that might be correct and that's something for me to test carefully before going to prod with it. Anyone treating it like anything more than that is a moron
What do you mean by “he granted Claude access”? Claude has permission to any file or credentials on your machine that you grant it access to, but you’re speaking as if it’s something that just runs on its own. I agree nobody should be running automated tasks against infrastructure locally using their personal credentials. Whether that’s using Claude workflows or windows task manager, both are terrible practice. But to imply this person shouldn’t be automating tasks “he can do himself” is also a strange thing to say in IT.
I trust a violent convict with a loaded gun more.
“Do you hear that Mr. Anderson, that is the sound of inevitability” Humans will always make the cool thing before thinking of the consequences.
This is like over-engineering...but without all the hard work that goes into over-engineering lol. Basically adding a complication or variable risk for no particularly good reason at all
I trust AI to work on my spreadsheets, word docs and PowerPoint slides, that’s about it
AI is basically a junior developer who can only have read only access in production. It can develop code, scripts, or an analysis for me to review and implement into production. But I hold the keys, and it doesn't go near anything sensitive. Go ahead, build some start up and shut down scripts or some logging scripts that can monitor resource utilization. I will take those, review them, and use them. Not AI. It can't check it's own work well enough
The word you were looking for is "Hypervisor"
Just the other day I used one to help me rename a bunch of files on my personal computer. I told it "write me a script to rename these files based on this name pattern." It's "thought" train was "OK I've got the script, now let's do a dry run to make sure it works. Wait! That wasn't a dry run, the files are actually renamed now!" Note that I didn't want it to actually run the script, I just wanted it to write the script for me so I could look it over first. So yeah, they make mistakes and can do stuff you didn't even ask for. I'd never let an agent loose in a production environment.
He’s crazy and that cannot go unchecked
I would flip my lid. There are already documented cases of AI helpfully deleting data or screwing things up. Granting it the keys to the kingdom would terrify me. When AI helpfully destroys that host and the coworker gets fired, you will be the one tasked with rebuilding that machine and of course the time-to-operation for that machine will be highest priority immediate since the whole org is now down. Don’t all the AI platforms ToS still state not to trust AI or rely on it for anything critical/legal/financial ?
I use AI as a tool — usually an information tool, sometimes a troubleshooting tool. I never allow AI to perform my actual job. Beware casually allowing AI to displace human labor, and never grant an AI agent access to production environments.
At home sure. For work nah. I’ll let AI interact with a folder on my work machine but other than that I read and understand everything I run or at least test it on a sandbox first. At home I let AI do whatever it wants on my computer all my files are versioned and backed up properly and nothings that serious that if I lost it I’d be ruined just a little sad.
Interesting because I’m currently building out a CI/CD automation workflow using AI.
I "trust" ai very well. I trust that it will behave like a non-deterministic system that will translate instructions I give it into actions.
Language models aren't good for making decisions. You must make them. They, however, will translate intent very well into machine language or other operations.
For example: "debug this problem" is useless.
Something like
This kind of process work.
"What should I do?" is completely useless.
Language models are auto-completion engines; they will follow instructions and respond to words. That's it.
I live by and breathe for the thing that was absolutely hammered into my head back when I did my national service with the RNoAF as a Crewchief-assistant on the F-16: Trust, but verify.
Sure, by all means trust. But ALWAYS verify. Every goddamn time. You do not believe it'll work. We're not in goddamn church, so you will KNOW what it does before you run it.
AI produces a script? Read it, understand it and test it before running it in prod. AI gives you a procedure to do something? Same deal; ALWAYS verify. Always understand what it's going to do.
Any coworker of mine granting Claude or any other AI-agent elevated access to anything that's my responsibility or affects "my" systems gets an earful and told in very unapologetic terms to kindly go look for landmines with a sledgehammer. After I thoroughly question their ancestry and just how close to whatever primate they're descended from they are.
The correct way to use AI here is to use Claude to evaluate what the task is, make sure it works, make sure it accounts for edge cases, and then use it to establish a proper *deterministic* script that reliably achieves the same thing but without an LLM, including dry-run, fallback, and safety. Flex the power of something like Claude to cover those cases and be written well. Of course, this requires *you* to know what a good script looks like. This is how I use AI when doing any kind of server-based admin work. I go in knowing *what* want to achieve first, then, it gets dry-run and evaluation permissions first, inspects the environment, creates the script, and then I test, and if I'm happy with it, the script is what becomes the (source controlled) control surface going forward. Personally, I'm not as concerned about "oh no it will delete all of our prod databases" because I have seperate guardrails in place to prevent that from happening. My concern is more around wasting LLM tokens to do a "routine task" that could easily be achieved with scripting. Even in my personal "agentic workflows", the "LLM" part only exists to deal with edge cases that a deterministic script can't deal with, or when subjectivity is needed. tl;dr Use AI to deal with the non-deterministic stuff, establish a deterministic pathway, and then stop having AI directly handle execution.
When it comes to solving IT problems, AI in most cases is no better than selecting one of the top google hits. In most of the cases where you’re looking for something very specific, AI returns generalistic useless answers. Would I turn over the keys to systems to AI? … Hell to the No on that.
He'll learn when it decides it made a mistake and chooses 'starting from scratch' as the best direction.
At home? AI has full untethered access to my network. I am running local qwen and it’s beyond fun, and allows me to just enjoy the stuff without being an at home sys admin. I won’t lie, from my home experience this technology will gain commercial viability and use and full access to systems. I hope there will be a system designed to adjust and change permissions based on task to try to keep permissions minimal. But it’s far too efficient and corp American will force its use. All that said, in an actual environment, absolutely not. I am working on ways to export data more efficiently to make decisions and get information via MCP or other dedicated means but there is so much room to grow at this time.
The fear here is funny.. It’s only an issue, if it’s an issue. The same as hiring a new admin. New guy trustworthy? Don’t give full access yet, now they get full access, now they make the mistake. Do I “Trust” “Claude”? No. Do I trust my ability to instruct this tool to do things? Yes. Do I trust my ability to recover if things go wrong? Yes. I’m an exceptionally capable sysadmin without any LLM/AI tools. But the time saved by leaning into a capable tool.. Learn to use it properly. If you are in this thread, agreeing with OP and the fear? Check yourself, you ARE falling behind those who aren’t afraid. You NEED to understandable exactly how capable AI/LLMs are/Aren’t and how to use them to their fullest extent, or you are useless to modern organizations that use those tools. Fight the current all you want, you’ll lose.
I've seen people training and using AI to tackel a lot of alerts and problems in their kubernetes clustet that are production clusters. So depends on the AI itself.
I've been avoiding AI as much as possible. At the most, I read through those AI summaries when I look something up, and I have yet to receive an AI summary that doesn't contain a factual error. As a result, I can not trust AI to do any job, because it consistently fails at a simple job it was designed to do.