How to select an AI vendor in Australia: a due-diligence checklist

Most AI vendor pitches in Australia sound the same. Confident deck, a few logos, a demo that works flawlessly on rehearsed data, and a promise to “partner on your AI journey”. The pitch is not where you find out whether the firm can actually ship. You find that out in due diligence — by asking questions the reseller can’t answer and the practitioner answers without flinching.

The summary you can act on: AI vendor selection in Australia comes down to one distinction — can this firm actually build and run the thing, or are they reselling someone else’s platform with a services wrapper? Seven areas of evidence will tell you, and none of them are on the slide deck. This checklist gives you those seven, plus how to structure the evaluation so the answers are comparable rather than a beauty contest of confidence.

This piece is the diligence companion to your procurement process. If you’re still drafting the request, start with our guide on how to run an AI consulting RFP, then use the checklist below to interrogate the shortlist.

Why AI vendor selection is different from normal software procurement

Traditional software procurement rewards the vendor with the most complete feature matrix. AI delivery doesn’t work that way. The value isn’t in the model — anyone can call the same APIs you can. The value is in the engineering discipline around the model: retrieval quality, evaluation, guardrails, data handling, and the unglamorous work of making a probabilistic system reliable enough to trust in production.

That means the usual procurement signals mislead you. A polished platform demo tells you the platform works, not that the firm can adapt it to your data, your risk appetite, and your regulatory obligations. AI vendor selection Australia should weight demonstrated delivery over demonstrated product. The seven areas below are ordered roughly by how quickly they expose a reseller dressed as a builder.

1. Delivery track record — shipped, not just started

Ask for engagements that reached production and stayed there. “We ran a pilot” is not a track record; most pilots quietly die, which is a topic worth understanding in its own right — see why most enterprise AI pilots fail.

Questions that separate builders from resellers:

• What did you ship in the last 12 months that is still running? Named systems, real users, current status.
• What broke, and how did you find out? A firm that has operated AI in production has war stories about hallucinations caught by evaluation, retrieval that degraded, or a model update that changed behaviour overnight. No war stories means no production experience.
• Show me an architecture you regret. Seniors will happily critique their own past decisions. Salespeople change the subject.

If every reference is a proof-of-concept or a workshop, you are buying slideware.

2. Who actually does the work

This is the single most common gap between what’s sold and what’s delivered. The senior who runs the pitch is frequently not the person who writes your code. You get a pyramid: one experienced lead spread across six accounts, and the day-to-day done by juniors learning on your budget.

Pin this down before you sign:

• Name the individuals who will be on your engagement, their seniority, and their allocation percentage.
• What proportion of billable hours are delivered by people with three-plus years of production AI experience?
• Contractual continuity — can key people be swapped out without your consent?

Pricing and staffing are linked, and the honest version of that trade-off is worth reading before you negotiate: our note on the honest cost of AI consulting in Australia explains why a cheaper day rate usually means a more junior team, not a better deal.

3. Data handling and residency

For Australian organisations this is often the deciding factor, not a footnote. You need specifics in writing, not reassurance.

• Where does data physically reside at rest and in transit — Australian regions, or offshore? If offshore, which jurisdictions?
• Is your data used to train or improve any model, the vendor’s or a third party’s? Get a contractual “no” if that’s your requirement.
• Sub-processors — which third parties touch your data, and are their terms flowed down?
• Retention and deletion — how long is prompt and output data kept, and can you compel deletion?

If you’re in a regulated sector, general-only references to obligations under the Privacy Act and the Australian Privacy Principles, and sector rules such as APRA’s prudential standards, should be second nature to the vendor. A firm that hasn’t thought about AI and the Australian Privacy Act is not ready for a regulated workload.

4. Security posture

Treat an AI vendor like any other supplier with access to your systems — then add the AI-specific attack surface.

• Baseline security: independent certification or attestation, access controls, secrets management, incident response commitments. Ask for evidence, not adjectives.
• AI-specific threats: how do they defend against prompt injection, data exfiltration through model outputs, and over-broad tool or agent permissions? A firm building agentic systems that can’t discuss least-privilege tool scoping is a risk.
• Supply chain: which model providers and open-source components are in scope, and how are they patched and monitored?

The tell here is specificity. Builders describe controls they’ve actually implemented. Resellers describe the platform vendor’s marketing.

5. Evaluation and quality discipline

This is the area that most cleanly separates engineering-led firms from everyone else, because it’s invisible in a demo and expensive to fake.

Ask: how do you know the system is good enough to ship, and how do you know it stays good?

A serious answer includes:

• A held-out evaluation set built from your real data and edge cases, not vibes.
• Metrics tied to business outcomes — accuracy, groundedness, refusal rates — with target thresholds agreed before launch.
• Regression testing so a prompt change or model update doesn’t silently degrade quality.
• Human review loops for the cases that matter.

If the answer is “we test it and it looks good”, they are shipping hope. Evaluation rigour is also what makes ROI measurable rather than anecdotal; the CFO framework for measuring AI ROI depends on exactly this kind of instrumentation.

6. Exit and intellectual property terms

Optimism about the relationship is not a strategy. Read the exit terms as if the engagement will end badly, because some do.

• Who owns the deliverables — code, prompts, fine-tuned weights, evaluation sets, and documentation? “The vendor licenses it to you” is very different from “you own it”.
• Lock-in — is the solution portable across model providers, or welded to one platform you’ll pay to leave?
• Transition assistance — is there a contractual obligation to hand over cleanly to you or another supplier, and at what cost?
• Source access — do you receive the actual source and configuration, or a black box you can’t maintain without them?

Resellers structure deals so you can’t operate without them. Builders are comfortable handing over, because their value is repeatable, not hostage-taking.

7. References that mean something

Vendor-supplied references are curated to say nice things. Make them useful anyway by asking the referee questions the vendor didn’t prep them for:

• What went wrong, and how did the vendor handle it?
• Who was actually on your team, and did the people stay?
• Is the system still in production, and would you buy from them again at the same price?
• What would you do differently?

A reference who can only offer generic praise either had a shallow engagement or was chosen for their diplomacy. Push for a referee with a comparable workload to yours.

How to structure the evaluation

Don’t run this as a single scored spreadsheet where a strong sales performance drowns out weak delivery evidence. Instead:

1. Gate on the non-negotiables first. Data residency, security baseline, and IP ownership are pass or fail. A firm that fails a gate is out regardless of how good the pitch was.
2. Score the depth areas separately — track record, staffing seniority, and evaluation discipline — and weight them heavily. These predict delivery success.
3. Run a small paid trial on your real data before the large commitment. A two-to-four week scoped exercise reveals more than any reference call, and it exposes the staffing-continuity question immediately.
4. Compare like with like. Insist every shortlisted firm answers the same seven areas in writing, so you’re weighing evidence rather than charisma.

Grounding all of this in a clear strategy — what you’re actually trying to achieve and why — keeps the evaluation honest; our view on AI strategy covers how to frame the problem before you shop for a solution.

Where to start

Good AI vendor selection in Australia is unglamorous. It’s asking the same seven questions of every firm and paying attention to who answers with specifics and who reaches for the deck. The builders welcome the scrutiny; the resellers resent it. That reaction is itself a data point.

If you’d like a second opinion on a shortlist, a set of diligence questions tailored to your workload, or an independent view before you sign, get in touch. We’re happy to be the awkward questions in the room.