Developersummit
  • HOME
  • SPEAKERS
  • SESSIONS
  • SCHEDULE
  • BUY TICKETS
  • FAQ
  • CONTACT
  • ONDEMAND
saltmarch

GIDS news media, articles, insights and virtual events educate and illuminate its audiences so they can be fully prepared to deal with the new realities at work and in their professions.

Saltmarch On-Demand
Media

Our Experts

Videos On Demand

Insights

Connect

About Us

Privacy Policy

Code of Conduct

Terms & Conditions

Contact Us

Sponsor GIDS 2027

Subscribe to Developersummit

Get the latest event updates, and insights from today's leading voices.

© 2026-2027 Saltmarch. All rights reserved.

Operating Agentic Systems in Production: Evaluation, Observability, and Runtime Control
RegisterTwitterLinkedInFacebook

< session />

Operating Agentic Systems in Production: Evaluation, Observability, and Runtime Control

Fri, April 30Reliability, Observability & SecurityAI-Native Software

Once agentic systems reach production, the model quickly stops being the hardest engineering problem. The greater challenge is understanding why the system made particular execution decisions, observing how its behaviour changes over time, and establishing the operational controls needed to keep adaptive agent systems reliable, inspectable, and safe.

In this session, you will explore what it takes to operate agentic software in production by looking beyond whether the final answer was correct to the runtime decisions that produced it. You will examine why a system selected a particular tool, plan, memory, model, handoff, or recovery strategy, and how engineering teams can evaluate those decisions and detect changes in behaviour. The session covers execution traces, evaluation signals, policy evolution, safety boundaries, observability, and debugging practices, with the model treated as one component within a larger agent harness. You will leave with a practical framework for deciding what to observe, what to evaluate, where runtime controls belong, and how to keep adaptive behaviour inspectable, governable, and understandable as systems evolve.

What You Will Learn

  • How to observe and evaluate runtime decisions made by intelligent software systems

  • How execution traces, evaluation signals, observability, and runtime controls support production operations

  • How to keep adaptive system behaviour inspectable, governable, and understandable over time

Who Should Attend

AI Engineers, Platform Engineers, SREs and Reliability Engineers, Software Architects, Staff and Principal Engineers, Technical Leads, and teams responsible for operating intelligent software in production.

< speaker_info />

About the speaker

Nicole Koenigstein

Nicole Koenigstein

AI Systems Researcher & Author

Nicole Koenigstein is an AI Researcher and Practitioner in Agentic Systems, working across research, consulting, teaching, and direct system implementation to build reliable, production-ready AI systems. Her work focuses on multi-agent architectures, evaluation, safety, and long-term system behavior.

She served as an external evaluator for a European Commission AI Grand Challenge and has advised IOSCO on generative AI in regulated environments. She also serves on advisory boards for leading AI and quantitative finance conferences. Nicole regularly delivers invited talks and technical workshops across academia, industry, and international events. She is the author of Math for Machine Learning and Transformers in Action with Manning Publications. Her books Transformers: The Definitive Guide: Applications Beyond NLP and AI Agents: The Definitive Guide have been published by O’Reilly Media, and her forthcoming book Harness Engineering for AI Agents will also be published by O’Reilly Media.

Related Talks

When Healthy Agents Fail: 12 Patterns from Production

Thu, April 29

When Healthy Agents Fail: 12 Patterns from Production

Tuhin Sharma, Soham Dutta
Policy Must Commit: Securing Agent Actions Beyond Guardrails

Thu, April 29

Policy Must Commit: Securing Agent Actions Beyond Guardrails

Rohit Bhardwaj
Progressive Collapse: How To Avoid Cascading Failure

Wed, April 28

Progressive Collapse: How To Avoid Cascading Failure

Sam Newman

On-Demand Talks

All On-Demand »