How to Tell If an Agency Actually Uses AI Well — or Is Just Charging You for the Buzzword

It doesn't mean owning the latest tools — almost everyone does now. Google's DORA programme (the long-running research on software delivery performance) put it bluntly in 2025: AI 'amplifies what's already there.' Drop AI into a team with strong testing, version control and review, and it gets faster. Drop it into a team without those, and it ships problems faster. The tools are a multiplier, not a fix.

So 'uses AI well' is a statement about practice, not purchasing. The mature version is AI for first drafts, test scaffolding, documentation and refactoring suggestions — always inside a pipeline where a human reviews, tests and is accountable for what merges. The opposite is 'vibe coding' (accepting AI output with minimal scrutiny) shipped straight to production. When you evaluate an agency, you're really evaluating the discipline wrapped around the AI, not the AI.

Make them get specific. Which AI tools do you use, and for which parts of the work? Where in your workflow does a human review AI-generated code, and who signs off before it merges? How do you stop AI from introducing security flaws or duplicated code? Can you show test coverage, or cycle-time data before and after adopting AI? Have you ever rejected AI output — when, and why?

The point isn't any single 'correct' answer; it's whether they answer in artifacts or in adjectives. A genuine team reaches for examples, names tools, and describes a review pipeline without hesitation. A buzzword shop returns to marketing language — 'cutting-edge', 'AI-driven', 'next-generation' — and gets vague when you ask who reviews the output. Specificity under follow-up questions is the single best signal you have.

Look for the surrounding engineering practice. DORA's 2025 model ties real AI value to concrete capabilities: a clear, communicated AI policy; healthy access to internal data; strong automated testing; mature version control; small, reviewable batches of change; and a user-centred focus. An agency that can speak to those is using AI inside a system, not as a sticker.

The clearest single flag is human-in-the-loop review. GitHub framed it well in 2025 — 'developers will always own the merge button' — and ThoughtWorks went further, placing 'complacency with AI-generated code' firmly on its do-not-do list and prescribing test-driven development, static analysis and human review. So ask to see the review step. A team that treats AI output as a first draft to be tested and reviewed — never as finished work — is one using it well.

'AI washing' — overstating AI capability for marketing — is common enough that regulators now act on it. In March 2024 the US SEC charged two firms a combined $400,000 for false AI claims; in September 2024 the FTC ran 'Operation AI Comply', a sweep of actions against deceptive AI marketing, with the blunt line that 'there is no AI exemption from the laws on the books.' If public companies get fined for it, a sales deck certainly isn't above it.

The practical red flags: AI as an adjective with no detail; an inability to name tools or describe where humans review output; AI pitched purely as a discount or a headcount cut with no quality story attached; and 'we let AI write it' with no mention of testing or accountability. None of these are illegal in an agency pitch — but each is a sign you're buying the buzzword, not the capability.

No — and assuming so is how buyers get burned. On cost, AI's measured gains are real on small tasks but evaporate on complex work, so 'we use AI' is not a reason to expect a discount. On safety, the evidence is pointed: Veracode's 2025 review found about 45% of AI-generated code samples failed security tests, and an earlier NYU study found roughly 40% of one tool's generated programs contained vulnerabilities. Stack Overflow's 2025 survey shows developers themselves now trust AI accuracy less, not more (around 33%).

The takeaway is not 'avoid AI' — it's that AI well-used is a quality safeguard, not a price cut. An agency that pitches AI as 'faster and cheaper' with no mention of review and testing is describing the exact path to insecure, throwaway code. One that pitches AI as 'faster, with the same review rigour' understands what it's doing.

If your real need is a prototype, a demo, or an internal tool a few trusted people will use, you may not need an agency — AI tools or a capable freelancer can get you there, and you can review the output yourself. The AI-maturity question only earns its weight once something real depends on the build.

Hold off on an agency, too, when you intend to own the product in-house long term — then you should be hiring and building the practice internally — or when the project is genuinely throwaway and not worth the diligence. Use this checklist when you're commissioning software that has to ship, scale, handle real users or data, and be maintained for years. That's when how well a team actually uses AI — inside real engineering discipline — starts to decide your outcome.