Developersummit
  • HOME
  • SPEAKERS
  • SESSIONS
  • SCHEDULE
  • BUY TICKETS
  • FAQ
  • CONTACT
  • ONDEMAND
saltmarch

GIDS news media, articles, insights and virtual events educate and illuminate its audiences so they can be fully prepared to deal with the new realities at work and in their professions.

Saltmarch On-Demand
Media

Our Experts

Videos On Demand

Insights

Connect

About Us

Privacy Policy

Code of Conduct

Terms & Conditions

Contact Us

Sponsor GIDS 2027

Subscribe to Developersummit

Get the latest event updates, and insights from today's leading voices.

© 2026-2027 Saltmarch. All rights reserved.

When AI Tests, Systems, and Specifications Disagree
RegisterTwitterLinkedInFacebook

< session />

When AI Tests, Systems, and Specifications Disagree

Fri, April 30AI-Assisted Software EngineeringReliability, Observability & SecurityPlatform Engineering & DevOps

When the same AI influences both code and tests, a green build is no longer sufficient evidence that software is correct. The code and its tests can reinforce the same misunderstanding, while coverage dashboards measure activity rather than demonstrate that business requirements have been satisfied.

In this session, you will examine a production verification architecture built around a probabilistic proposer, a deterministic verifier, graded evidence, and a computed release verdict. At its centre is a Git-versioned rulebook containing executable business rules written in plain language, approved by the person responsible for the requirement, and reviewed as decision tables. The rulebook computes expected results, allows product owners to examine behaviour before implementation, and connects every verification run to its underlying requirement. An LLM-driven agent explores the system, identifies uncovered scenarios, and drafts candidate rules and tests, while a separate deterministic verification layer, with no model in the verdict path, compares those claims with the approved rules and observed system behaviour.

Through a live insurance-rating example, you will see what happens when an AI-generated test expects one premium, a locally running Guidewire PolicyCenter instance returns another, and an actuary-approved executable rule computes a third. The resulting evidence graph and release verdict of READY, CONDITIONAL, or NOT READY help engineering, QA, and business teams investigate the disagreement and determine which artifact requires correction. The session also examines the more difficult case in which the approved rule itself has become stale. Although insurance provides the primary case study, the verification architecture applies wherever AI-generated software must implement evolving business rules.

What You Will Learn

  • Why AI generation and verification require independent systems

  • How executable rules, graded evidence, and deterministic verdicts support requirement-level assurance

  • Why operation coverage, self-grading agents, and snapshot assertions can create false confidence

  • How to investigate disagreements between generated tests, system behaviour, and business specifications

Who Should Attend

Staff and Principal Engineers, Software and Solution Architects, AI and Platform Engineers, Test and Quality Engineering Leaders, SREs and MLOps practitioners, and technical leaders responsible for AI-assisted software delivery.

< speaker_info />

About the speaker

Peter Thomas

Peter Thomas

CTO, Karate Labs

Peter Thomas is CTO, Co-Founder of Karate Labs and the original creator of the Karate open-source framework adopted by 650+ companies including 76 of the Fortune 500, with over two million monthly downloads and 8,800+ GitHub stars. Karate is featured in the Gartner Market Guide for API & MCP Testing Tools (March 2026) and is embedded as the foundation of Guidewire’s own API testing framework, making Karate Labs Guidewire’s first and sole testing technology partner.

His current work is Karate Agent, an AI-native verification platform built on the architecture this talk teaches: an LLM-driven agent explores systems and drafts tests and candidate business rules, while a deterministic verification layer (with no model in the decision path) grades every claim against product-owner-approved rules and computes a ship verdict with requirement-level traceability. The platform is self-hosted and BYO-LLM by design, and runs in production across banking, insurance, and financial services.

Peter speaks internationally on test architecture, API design, and the engineering of trustworthy agentic systems.

Related Talks

Using AI to Upgrade Applications from One Language to Another: Strategies, Dos, and Don'ts

Wed, April 28

Using AI to Upgrade Applications from One Language to Another: Strategies, Dos, and Don'ts

Venkat Subramaniam
Twelve Essential Engineering Practices for AI-Assisted Development

Wed, April 28

Twelve Essential Engineering Practices for AI-Assisted Development

Venkat Subramaniam
Specification-Driven Development by Example

Wed, April 28

Specification-Driven Development by Example

Venkat Subramaniam

On-Demand Talks

All On-Demand »