Managing a Team That Uses AI: What I Learned

Managing a team that uses AI is now part of most IT leaders’ jobs, whether they’ve planned for it or not. I use Claude, Anthropic’s AI assistant, almost every day at work. It has helped me troubleshoot VPN faults, write deployment scripts, review vendor contracts and draft change communications. It has made me noticeably faster, and I’d find it hard to go back to working without it.

It has also been wrong. Not usually in an obvious way, and that’s what makes it interesting. It has been wrong in ways that sounded completely reasonable and that, if I’d accepted the answer without checking, could have caused real problems.

That’s the part I think matters for IT leaders. Your team is almost certainly using AI assistants already, approved or not. The real question isn’t whether they should. It’s how you manage a team when some of the work is drafted by a tool that is usually right, occasionally confidently wrong, and never accountable for the outcome. This is what I’ve learned from correcting an AI assistant on the job, and what I’d put in place for a team that uses one.

The mistakes weren’t random

Looking back over the times the answer was wrong, I noticed a pattern. The errors were rarely nonsense. They were plausible, specific and delivered with exactly the same confidence as the correct answers.

A version number that was off

While I was building an Azure Automation runbook, the assistant correctly warned that a PowerShell cmdlet had changed to return tokens as a secure string, but it attributed the change to the wrong major version of the module. The code handled both cases, so nothing broke. Anyone relying on the explanation to plan an upgrade would have been misled, though, and it was only corrected when the explanation was checked against Microsoft’s migration guide. The code was fine. The explanation was wrong.

A convincing explanation of vendor behaviour

After I fixed an application control problem, the assistant and I settled on an explanation of how the product’s policy hierarchy worked. The fix was right, but the explanation was the opposite of what the vendor’s own documentation describes. I only discovered that when I went to write the problem up and checked the documentation properly. If I’d published that explanation, someone could have accepted it and carried the wrong mental model into their next problem.

A feature that was no longer unsupported

An AI-generated draft I reviewed stated confidently that a Microsoft 365 feature didn’t work in a particular Citrix configuration. That had been true once, but it isn’t any more: Microsoft now documents a supported configuration. The draft took an old answer, presented it as current, and wrote it so convincingly that nothing in the prose suggested it was out of date. An answer can be completely correct historically and still be wrong for the environment you’re working in today.

Invented detail

This one is more dangerous, because it looks like evidence. An AI-generated draft gave a precise-sounding success rate for a setting and claimed experience across “hundreds of deployments”. Neither had any source. They sounded like something an experienced engineer might say, but they were invented. Another draft even ended with the assistant offering to “provide sample scripts, just let me know!”, which would have been published as the article’s final line if I hadn’t noticed.

A shared misdiagnosis

The example I learned most from was a misdiagnosis the AI and I shared. During a long troubleshooting session, we agreed that a particular component was causing a hang, and I removed it. Eventually I found the component wasn’t causing the problem at all. It was hiding the real one, so I put it back.

That was a useful reminder that AI doesn’t remove confirmation bias. It can make it easier to reinforce a wrong diagnosis, because it gives your theory another articulate voice.

The pattern

AI errors are most dangerous around things that change: version numbers, licensing, vendor behaviour, product support, defaults, new features, and anything that used to be true. They also appear when the assistant fills a gap in its knowledge with an explanation that sounds like it belongs there, instead of saying “I don’t know”. So the lesson isn’t “be smarter than the AI”. It’s to build the check into the process.

Accountability doesn’t transfer to the AI

The first rule I’d set for any IT team is simple: “the AI said so” is never a reason. If someone deploys a script, changes a firewall rule or tells a user something, they own that decision, however the work was produced. An AI assistant can’t attend the post-incident review, it won’t be on call when the change goes wrong at 2am, and it doesn’t have to explain the decision to the business. Accountability stays with the person who pressed Enter.

This matters because AI output feels authoritative. It’s well written, well structured and confident, which can make it easier to trust than a colleague’s rough notes, even when those notes come from someone with ten years’ experience of the technology. I’ve caught myself about to accept an explanation simply because it was articulate.

As a manager, the practical test is to ask: “How do you know that’s right?” If the answer is “Claude told me”, the work isn’t finished. If it’s “Claude suggested it, I checked it against the vendor documentation and tested it on a pilot device”, that’s a very different conversation. The AI did the heavy lifting, and the engineer still did the engineering.

Review AI work like a capable new starter’s

The mental model I keep coming back to is to treat an AI assistant like a very capable, very fast new starter. It’s knowledgeable, hard-working and can produce an impressive amount of work quickly. It also doesn’t know your environment or your organisation’s history, can rely on outdated information, and is sometimes far too confident when it should say “I’m not sure”.

You wouldn’t let a new starter push a production script without review, however talented they were, and the same applies here. The good news is that you don’t need a new governance framework, because your existing processes already cover most of it:

Peer review. AI-assisted scripts, configuration and documentation get reviewed like any other work. The ease with which AI produces polished work makes review more important, not less.
Change management. The change process doesn’t care who wrote the PowerShell. Risk assessment, testing, rollback plans and approvals still apply.
Pilot first. Several of my AI-assisted scripts worked perfectly on one machine and failed on the second. That second machine is usually where the real learning starts.
Primary sources. For anything version-specific, licensing-related or about how a vendor’s product behaves, check the vendor’s documentation. That one habit would have caught most of the errors above.

AI should accelerate thinking, not replace it

This is the distinction I most want my team to understand. There’s a big difference between “use AI to solve this problem for me” and “use AI to help me solve this problem”. The first encourages dependency, and replaces the thinking. The second accelerates it, and can make an engineer considerably better.

When I’m troubleshooting, I use AI to generate possible causes, suggest tests, explain unfamiliar behaviour and challenge my current theory. But I still decide which test to run, I still look at the result, and I still decide whether the evidence supports the diagnosis.

Where AI shines, and where to slow down

I don’t want this to read as a warning against AI, because the benefits are real. A team that learns to use it well can produce more, communicate better and spend less time on repetitive work. The skill is knowing where to trust it more and where to slow down.

AI is excellent at:

Structuring troubleshooting. It helps you work through a problem methodically, suggests what to test next and keeps track of what you’ve ruled out.
First drafts. Scripts, documentation, change notices and emails go from a blank page to a useful starting point in minutes.
Explaining unfamiliar territory. Getting up to speed on a new technology is dramatically faster when you can ask questions conversationally.
Reviewing work. Asking AI to find problems with something often surfaces issues you’d missed. Some of the most useful corrections I’ve had came from asking an AI assistant to review a draft that an AI had produced.
Communication. It turns technical detail into language a business audience understands, which is especially useful for engineers who know the technology but struggle to explain the business impact.

Slow down for:

Anything version-specific, such as module versions, firmware, build numbers, API versions and required updates.
Licensing, which changes regularly and is expensive to get wrong.
Vendor behaviour, especially policy evaluation, precedence, defaults and undocumented behaviour.
Anything that has changed since the answer was written, including new features, deprecations, support statements and product limitations.
Security decisions, where a plausible but wrong recommendation creates real exposure.
Statistics and claims of experience. If an AI quotes a precise-sounding percentage or a number of deployments, ask where it came from. If there’s no source, don’t publish it.

Confidence isn’t verification

It’s useful to ask an AI assistant how confident it is, what it’s assuming and what documentation supports its answer, because that can expose uncertainty. But confidence is not verification. An AI can tell you it’s very sure and still be wrong.

Verification has to happen outside the answer: checking the vendor’s documentation, testing the command, reproducing the behaviour in a lab, or asking another engineer. The higher the cost of being wrong, the stronger the verification should be.

What your team shouldn’t paste into an AI tool

The fastest way to troubleshoot with an AI assistant is to paste in the error, the log or the configuration. That’s also where the biggest data risk sits, and IT teams handle more sensitive material than most of the business. I’d set clear rules in three tiers:

Never paste secrets, into any tool. That means passwords, API keys, client secrets, storage account keys, private keys, tokens and recovery codes. Scrub them from scripts, logs and configuration files before sending anything to an AI service.
Remove personal information unless your organisation has explicitly approved that use. That includes customer and staff records, HR matters, contact details and support tickets containing personal data. If you don’t need the personal information to solve the technical problem, take it out.
Treat sensitive business information with care. Internal network diagrams, detailed vulnerability findings, security incident details, proprietary code and commercially sensitive documents all fall here. A useful AI tool isn’t the same as an approved destination for that information.

The tool matters too

There’s a real difference between an AI service your organisation has deliberately approved and staff individually choosing whatever free tool they like. The organisation should understand how the service handles data, what controls and contractual terms apply, and what employees may enter. If you haven’t made that decision deliberately, your team will make it individually.

The Australian Signals Directorate (ASD) makes similar points. Its introduction to artificial intelligence recommends applying the same basic security principles to AI systems as to any online service, and lists questions organisations should ask, including how an AI system will affect their privacy and data protection obligations. ASD’s Engaging with artificial intelligence paper gives guidance on using AI systems securely, including third-party systems your organisation doesn’t control. For organisations building or running AI systems of their own, ASD and its international partners have also published guidance on AI data security. Both are listed on ASD’s AI publications page.

The practical lesson for an IT manager is simple: know which AI tools your team is using, and decide which ones are approved.

Protecting how junior staff learn

This is the risk I think about most as a manager. I caught most of the errors above because years of experience told me something didn’t look right. A version number felt wrong, an explanation contradicted something I’d seen before, or a configuration didn’t behave the way the assistant said it would. That instinct came from doing the work the slow way: reading documentation, breaking things, fixing them, and seeing the same problem several times.

A junior engineer who reaches for an AI assistant first may never build that instinct. They’ll get working answers faster, but they may not understand why the answers work, and more importantly, they may not know when an answer is wrong. That’s a problem for them, and eventually for the team. A few things help:

Ask them to explain it. If someone brings you an AI-assisted fix, ask them to walk you through why it works, not just what the command does. Why that command? What did they rule out? What happens if it fails? If they can’t explain it, the job isn’t finished.
Use AI to learn, not just to skip. Encourage them to ask “why does this work?” rather than only “what command fixes this?” Used that way, AI is an excellent tutor that can explain a concept five different ways without getting frustrated.
Make primary-source checks a habit. If the assistant recommends a Microsoft setting, read Microsoft’s documentation, and if it explains a vendor’s behaviour, check the vendor’s documentation. That habit builds exactly the judgement engineers need.
Keep pairing on difficult problems. Watching a senior engineer form a hypothesis, decide what evidence matters and change direction when the evidence disagrees is still one of the best ways to develop judgement. AI should be another tool in that process, not a replacement for it.

Make it a team practice, not a private habit

In many teams, AI use is private. Each person has their own conversations, their own prompts and their own lessons about where the tool goes wrong, and none of it gets shared. That’s a missed opportunity, and a few simple practices turn individual use into team knowledge:

Share the corrections. When someone catches an AI assistant being wrong, especially about a product your team uses, tell the team. “It keeps getting this licensing detail wrong” or “the documentation it quotes is three years old” is useful operational knowledge.
Share what works. Useful prompts, good ways to structure a troubleshooting session, and techniques for getting AI to challenge its own answer mean nobody has to reinvent the same workflow.
Be open about AI use. Nobody should feel they need to hide that AI helped them, because hidden use is hard to review. The goal isn’t to catch people using AI. It’s to know how the work was produced, so it gets the right level of scrutiny.
Keep your own documentation authoritative. If AI helped solve a problem, don’t leave the answer in a chat history. Verify it, then put the verified answer into your runbooks or knowledge base. Your documentation should contain what you know to be true, not a transcript of what an AI thought might be true.

The real lesson from correcting Claude

The most valuable thing AI has done for me isn’t writing scripts faster. It’s made me more deliberate about verification, because I know the answer in front of me might be wrong. That changes how I use it. I still use it enthusiastically, to write the first version, challenge my thinking, explain something unfamiliar or work through a difficult problem. But I don’t outsource the final judgement.

That’s the mindset I’d want in an IT team. Use the tool, get faster, ask better questions, and stay sceptical enough to check the answer when it matters. The assistant can do a lot of the work, but it can’t take responsibility for it, and it shouldn’t be the reason anyone stops thinking. Managed well, AI makes a good team faster. Managed badly, it makes a team confidently wrong at scale. The difference isn’t the tool. It’s the habits you build around it.

Team AI guideline checklist

If I were writing a one-page guideline for an IT team using AI assistants, it would say:

You own the output. “The AI said so” is never a justification.
Use an approved tool, and don’t use personal AI accounts for company work unless the organisation has explicitly approved it.
Never paste secrets. Scrub passwords, tokens, keys and certificates first.
Protect personal and sensitive information. If the organisation hasn’t approved putting it into the service, don’t.
Verify anything that changes, checking version numbers, licensing, vendor behaviour and support statements against primary sources.
Review and test as normal. AI-assisted work still goes through peer review, change management and testing.
Understand before you deploy. If you can’t explain why it works, don’t deploy it.
Use AI to build judgement, not replace it. Ask why, not just what.
Share what you learn, including where the assistant got things wrong.
Record the verified answer in team documentation, not just in an AI conversation.

Sources

Daniel Lutton

From my early days on the helpdesk through roles as a service desk manager, systems administrator, and network engineer, I’ve spent more than 25 years in the IT world. As I transition into cyber security, my goal is to make tech a little less confusing by sharing what I’ve learned and helping others wherever I can.

Managing a Team That Uses AI: What Correcting Claude Taught Me

ByDaniel Lutton