Skip to content

What's new

  • Refact.ai: deployment, models and repository updated
  • Aider, Arize (AX, Phoenix) and 19 more: repository now published
  • Omnara added
All changes
AI SDLC Landscape

Evaluate an AI SDLC tool

29 questions to put to a vendor, in six groups. Each says why a buyer asks it. Open a tool's questions to see which ones its vendor has answered in public, with the source, and which it has not.

Download as MarkdownDownload as CSVPick a tool on the map

Data

Where code and prompts go, who processes them and what is kept.

  1. Which model providers and hosting providers process our code and prompts, model by model?

    One vendor documents that prompts and code context go to several outside model providers, and that its own models may add inference providers. Cursor, privacy and data governance (opens in a new tab)

  2. Does zero data retention apply to every model and every preview feature, or only some?

    One product's models run on several hosts, and retention terms and exceptions differ between them. GitHub Docs, model hosting (opens in a new tab)

  3. Is our code used to train any model, yours or a provider's, and where is that written down?

    Buyers name uncertainty about what models learn as a reason not to buy. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

  4. What happens to our terms when a provider changes its own?

    A contract that spans several providers can change under the buyer without the vendor changing anything. GitHub Docs, model hosting (opens in a new tab)

  5. Can the tool run in our cloud account, on premises or in a single tenant?

    Where code goes decides what a security team will sign. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

Continuity

What happens if the vendor, its owner or its model supplier changes.

  1. Who owns the vendor today, and has it been acquired or announced a sale?

    A change of owner changes terms, roadmap and support. The market moves page lists the recorded ones.

  2. Does the product depend on one model provider, and what happens if that supplier cuts access?

    Some tools route to several providers and some to one. Many vendors do not publish which. Cursor, privacy and data governance (opens in a new tab)

  3. What can we export on exit: repository context, prompts, policy configuration, audit logs and usage history?

    We found no public standard for exit and portability terms of AI coding tools, so the answer has to come from the vendor.

  4. Is any part open source, so we can keep running it if the vendor stops?

    Published source is a hedge against the vendor failing or being bought.

  5. Which companies of our size use it in production, and will they talk to us?

    Teams revert to manual process when they do not trust a tool, so references from peers carry weight. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

Pricing

What it costs at your scale, beyond the seat.

  1. Is pricing per seat, per usage, a subscription or a mix?

    Seat price alone understates cost: total cost can include token overages, premium model tiers, agentic compute and governance infrastructure. DX, AI coding assistant pricing (opens in a new tab)

  2. What triggers overage charges, and is there a cap we can set?

    The same analysis says overages can multiply the headline price for heavy agentic use. DX, AI coding assistant pricing (opens in a new tab)

  3. Which models sit in premium tiers, and who decides which one a developer uses?

    Premium model tiering is a named cost driver. DX, AI coding assistant pricing (opens in a new tab)

  4. What does it cost us per accepted change at today's usage, before we expand?

    The analysis recommends a measured baseline before rollout. We found no public benchmark for companies of 50 to 2,000 engineers that separates productivity gain from added review and governance cost. DX, AI coding assistant pricing (opens in a new tab)

Integration

How it fits the tools you already run.

  1. Which source control, CI/CD and issue tracker does it connect to, and how?

    Evaluation advice is to prefer assistants that integrate with the repository and the other development tools. The New Stack, AI coding assistants: 12 dos and don'ts (opens in a new tab)

  2. Which other tools on this map does it pair with, and which does it replace?

    The records name which tools integrate, ship inside one another or compete.

  3. Can our policies decide who may use which feature, per team and repository?

    Policy permission is on the list of things to test in a trial. The New Stack, AI coding assistants: 12 dos and don'ts (opens in a new tab)

  4. What integration commitments do you publish, and where?

    We found no comparable published integration commitments across IDEs, source control, CI/CD, identity and scanners.

Security

Controls, assurance and what you can show an auditor.

  1. Which compliance attestations does the vendor hold, and for which products?

    Buyers cite the inability to prove output reliability to auditors as a reason not to buy. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

  2. Do you keep audit logs of prompts, outputs and accepted changes, and for how long?

    Governance built for traditional software often does not map to AI workflows. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

  3. Do you answer a standard security questionnaire for AI-generated code?

    One security vendor publishes a template covering scanning, fix accuracy, policy alignment and scale. It is a vendor's own, so treat it as a starting point. Snyk, GenAI coding security RFP template (opens in a new tab)

  4. Can we start on low-stakes work and widen the scope as we build trust?

    The published rollout pattern starts with boilerplate, tests and documentation, keeps people accountable for generated code, then widens. Stack Overflow Blog, closing the developer AI trust gap (opens in a new tab)

Acceptance

What the tool gives the person who must answer for the changes it produces. These come from the maintainer's thesis, and few vendors publish them.

  1. Can a named person accept or refuse each AI-generated change, and can an administrator set who holds that authority?

    Acceptance that cannot be refused is not acceptance, so the authority and the right to refuse come first. The thesis, accountable acceptance (opens in a new tab)

  2. What does the reviewer see before accepting: the change's intent, its footprint, what changed in the system meanwhile, the evidence and its gaps?

    A person can only decide from a brief that is short enough to read and complete enough to trust. The thesis, accountable acceptance (opens in a new tab)

  3. Can we set a boundary below which changes are accepted by policy, so only consequential changes reach a person?

    The number of decisions each person faces has to stay within what they can judge. The thesis, accountable acceptance (opens in a new tab)

  4. Does the tool record who accepted which change, on what evidence, and can we export that record?

    A decision has to be reconstructable before anyone can answer for it. The thesis, accountable acceptance (opens in a new tab)

  5. After acceptance, can a person halt, roll back or disable an AI-generated change, and is doing so safe by design?

    Authority without a way to stop and reverse is not control. The thesis, accountable acceptance (opens in a new tab)

  6. Does the tool show a person how the changes they accepted behaved in production?

    People keep their judgment accurate only when they see how past decisions turned out. The thesis, accountable acceptance (opens in a new tab)