How to Choose a Generative AI Development Company: 12 Criteria

Buying generative AI development is harder than buying most software work, because the thing you are evaluating is easy to fake for an hour and hard to run for a year. A convincing demo takes a weekend. A system that survives real volume, real data rules and a real auditor takes a different kind of team, and the difference does not show up in a pitch deck.

So the useful question is not which generative AI development company is best. It is what you can actually test before you sign.

What follows is twelve criteria you can apply to every vendor on your shortlist, including us. Each one comes with a test: a document to ask for, a question whose answer is checkable, or a clause to read. A vendor who cannot pass a criterion has told you something useful. A vendor who passes all twelve on a slide but cannot produce evidence for any of them has told you something more useful still.

This post is about evaluating the builder. If you want to know what we build, that sits on our AI capability page; if you want the engineering detail of how agents get built, that is on AI agent development.

1. Evidence of production systems, not demos

Almost every vendor can show you something impressive. Far fewer can show you something that has been running for a year with someone’s name against it.

The test. Ask for the two systems that have been live longest. For each, ask who operates it today, how many people use it, and what broke after launch. Production answers name a system, an owner and an incident. Demo answers name a technology.

Ask one more question that separates the honest from the polished: was this a proof of concept or a system with real users? Both are legitimate work, and they are frequently described in identical language. Two of ours make the distinction concrete. For RadarFirst we built an AI assistant that turns conversational incident reports into structured compliance forms on AWS Bedrock, SageMaker and LangChain, using retrieval-augmented generation and NLP, over 3+ months. For Harvard Business Publishing we built TutorBot, a Microsoft Copilot assistant over an AWS ingestion pipeline with OpenSearch, across 10+ months. The second one was a proof of concept, and we say so.

2. A written definition of “working” before the build starts

The single most reliable predictor of whether a generative AI project reaches production is whether anyone wrote down, in advance, what success means. An evaluation set is that definition made executable: real inputs from your workload, known-good outputs, including the awkward cases, graded automatically on every change.

The test. Ask who writes the evaluation set, at what point in the engagement, and what happens contractually if the system misses the bar. A vendor who proposes to define “working” after the build has reserved the right to declare victory. A vendor who asks you for fifty messy real examples in week one is doing the job.

This is the mechanism behind most of the pilots that never make it to production, and it is the same discipline that turns an approved use case into an owned production system.

3. Data handling, residency, and what the model provider keeps

Your data leaves your perimeter the moment a prompt is sent. Where it goes next is a configuration question with a documented answer, and a competent vendor will point you at that documentation rather than reassure you.

Amazon Bedrock, for example, documents retention as an explicit account-level or project-level setting: in its zero retention mode, no request or response data is written to durable storage by AWS or shared with the model provider. Some models require retention so that human review can occur, in which case prompts and completions are held within the AWS boundary for a bounded period. Residency has a documented answer too: where cross-region inference is enabled, retained inputs and outputs are stored in the region where the request is processed, which is not necessarily the region you thought you were running in.

The test. Ask the vendor to show you, in the provider’s own documentation, the setting that governs retention for the models in your design, and the region behaviour that applies. If the answer is a verbal assurance, they have not checked. On our side, our compliance position is SOC 2 Type II and ISO 27001, and we are an AWS Consulting Partner and a Databricks Partner.

4. Model portability instead of provider lock-in

Model choice ages badly. Whatever is best for your use case today is a decision you will revisit, so the question is not which provider a vendor prefers but how expensive it would be to change your mind.

The test. Ask how many places in the codebase name the model provider. One is a healthy answer, because it means provider access sits behind an interface. Ask whether your prompts and evaluation sets live in your repository in a provider-neutral form, and whether the evaluation harness can re-score a different model without a rewrite. A vendor whose entire practice is built on a single provider’s stack is not disqualified, but you should know that is what you are buying.

5. Who owns prompts, fine-tunes, embeddings and evaluation sets

Most contracts say something sensible about source code and nothing at all about the artifacts that make a generative AI system valuable. Those artifacts are where months of work accumulate.

The test. Read the intellectual property clause and check that it names, individually: prompt templates, fine-tuned model weights or adapters, embeddings and the vector index, the ingestion and retrieval pipeline, the evaluation set, and the grading harness that scores it. Then ask a second question that matters as much as ownership: which cloud account does each artifact live in today? Owning something that only exists inside a vendor’s tenancy is a weaker position than it sounds.

6. Security posture against the OWASP LLM risks

Generative AI has its own failure modes, and there is a published, vendor-neutral list of them. The OWASP Top 10 for LLM Applications 2025 names LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM08 Vector and Embedding Weaknesses and LLM10 Unbounded Consumption, among others.

The test. Hand a vendor the list and ask them to walk it item by item against the system they are proposing, saying for each whether it applies, and if so which architectural control addresses it. You are listening for architecture: least-privilege tool scoping, treating retrieved content as untrusted input, validating output before a downstream system consumes it, approval gates on irreversible actions. A vendor who answers every item with “we use a good system prompt” has not read the list. The controls behind these answers are described in our AI agent development work.

7. A stated hallucination and human-in-the-loop policy

Every generative system is wrong sometimes. What distinguishes a serious builder is that they have decided in advance which wrong answers are acceptable and which must never reach a person or a system of record unreviewed.

The test. Ask which outputs can reach a user without human review, which require approval, and what the reviewer actually sees when they approve. Grounding and attribution belong in that answer: on the Harvard Business Publishing engagement, every answer the assistant produced carried source attribution, which is what makes a claim checkable rather than plausible. A policy of “we will tune it until it stops hallucinating” is not a policy. The contractual version of this conversation is covered in our post on explainability before you scale.

8. Observability and a cost ceiling in production

Generative AI systems degrade quietly. Quality drifts as your data, your users and the models change, and spend scales with usage rather than sitting still after launch. OWASP treats the spend side as a security risk in its own right, listed as Unbounded Consumption.

The test. Ask what is logged for each request, who is alerted when quality regresses against the evaluation set, and what happens when spend crosses a threshold: an alert, a throttle, or nothing at all. Ask to see the dashboard from an existing engagement. For RadarFirst, observability layers for continuous accuracy tracking were part of the architecture rather than an afterthought, which is the shape you want. Notice what this criterion is not: it is not a quote. Anyone who prices your system before seeing your integration surface is quoting a different project.

9. MLOps maturity: how changes actually reach production

A prompt edit is a production change. So is a model version bump, a new embedding model, or a re-indexed knowledge base. Teams that treat these as configuration tweaks ship regressions they cannot explain.

The test. Ask how a prompt change reaches production: who reviews it, what runs automatically before it lands, how it is rolled back, and how often the full evaluation suite runs. Ask what triggers a re-index or a retrain, and who decides. The answer tells you whether you are buying an engineering practice or a research project. Where the surrounding data platform is the real constraint, that is a data and AI center of excellence conversation before it is a model conversation, and the expensive mistakes usually live here rather than in model selection.

10. Domain evidence in your regulated context

If you operate under supervision, a vendor’s fluency in your regulatory vocabulary is a proxy for how many review cycles they have survived.

Two reference points make this checkable. The NIST AI Risk Management Framework, released on 26 January 2023 for voluntary use, organises risk work around four functions, GOVERN, MAP, MEASURE and MANAGE, and has a Generative AI Profile companion, NIST-AI-600-1, released on 26 July 2024. In Europe, the European Commission’s own summary of the AI Act records that the Act entered into force on 1 August 2024 and became applicable on 2 August 2026, with rules for general-purpose AI models effective from August 2025 and obligations for high-risk systems starting on 2 December 2027.

The test. Ask which framework the vendor’s governance artifacts map to, then ask for the artifact names, not the framework name. A vendor who has done this work can list what they produce and who signs it. Our post on governance evidence you can pull on demand sets out what that record looks like in practice.

11. Team composition: who actually does the work

The people in the pitch are rarely the people in the standup. In generative AI this matters more than usual, because the work needs data engineers and QA at least as much as it needs model specialists.

The test. Ask for the named profiles of the engineers who would be assigned, ask who built the demo you were shown, and ask what happens when someone leaves mid-engagement. Then ask for the vetting standard in specifics. Ours: we hire fewer than 2% of applicants, and every engineer clears four gates before meeting a client, a technical assessment, a live coding review, English proficiency and culture fit. We run 1,000+ engineers across Latin America in US time zones, carry a 76 Employee NPS, and a team is generally productive in 2–4 weeks. If you are staffing rather than outsourcing an outcome, hiring AI engineers is a different commercial shape with different risks, and the staff augmentation, dedicated team and fixed-price comparison is worth reading before you choose one.

12. The exit path

The best moment to negotiate leaving is before you arrive. This criterion is last because it tests every other answer: a vendor who has genuinely handed systems over will describe the process without flinching.

The test. Ask what a handover looks like in concrete terms: how much notice, who holds the cloud accounts and the API keys throughout, what runbooks and architecture documentation exist, whether the evaluation set and its history transfer with the system, and how long the vendor stays available afterwards. Ask whether anything in the system requires their tooling or licences to keep running. A vendor whose answer is that nobody has ever left should be asked why nobody has ever needed to.

How to actually run this list

Twelve criteria are too many to weight equally, so do not try. Pick the three that map to your real exposure. In a regulated environment, that is usually data handling, security posture and domain evidence. For a team that has already burned a budget on a pilot, it is the definition of working, the MLOps practice and the exit path.

Then apply one rule throughout: ask for the artifact, not the answer. A contract clause, an evaluation set, a dashboard, a documentation link, a named engineer. Anything a vendor can produce in a week, they already have. Anything they promise to produce after signature, they are building on your budget.

The same discipline applies to every engineering vendor decision you make, and the general version of it is in our guide to evaluating software development outsourcing companies. Applied to generative AI it simply has sharper edges, because the technology makes it unusually cheap to look capable.

If you want to run these twelve questions past a team that expects them, talk to us. We would rather answer them before a contract than after one.