Building Resilient AI-powered Software, And Planning For When Things Go Wrong
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, covering disaster recovery and resilience for AI-powered software. The material discusses recovery targets, infrastructure constraints, changing AI models, observability and strategies for managing vendor uncertainty; it does not document a specific outage or provide evidence of a measured rise in incidents.

The Pragmatic Engineer has published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, outlining how software teams can prepare for major outages and the reliability challenges of AI-powered services. The report focuses on disaster recovery planning and operational practices for systems using AI models, rather than announcing a particular outage, product or new industry-wide reliability finding.

The excerpts cover planning for failures that affect an entire cloud region, data center or other infrastructure site, including events such as fires, floods, power problems and failures in cloud infrastructure. Newman presents disaster recovery as a set of procedures for restoring service after those broad disruptions. The report explains two common planning measures: recovery point objective (RPO), the maximum acceptable period of data loss, and recovery time objective (RTO), the acceptable duration of service unavailability after an incident.

The article also outlines a range of disaster recovery patterns, from backup and restore to pilot-light and warm failover arrangements and multisite active-active systems. These options differ in cost, complexity and recovery capability. The source does not prescribe one approach for every system; it says teams need to select targets and designs suited to their own requirements.

For AI-powered software, the report identifies model-provider resilience, changes to models, observability and uncertainty as design concerns. It also points to CPU and GPU availability limits, recommending that teams understand vendor constraints and prioritize critical services. Suggested ways to manage uncertainty include using more than one model or vendor, making costs transparent and regularly reviewing vendors. These are recommendations in the excerpt, not reported results from a comparative study.

At a glance
reportWhen: Published in 2026; the exact publicatio…
The developmentThe Pragmatic Engineer published excerpts from Sam Newman’s new book on disaster recovery and building resilient AI-powered systems.

Recovery Targets Shape Outage Costs

The guidance matters because recovery expectations drive architecture and spending. A system that must return quickly and lose little data may require more infrastructure, operational work and testing than one that can tolerate a longer interruption or a larger recovery gap. Defining RTO and RPO gives engineering and business teams concrete terms for making those trade-offs before an incident.

AI services add dependencies that can change beyond a team’s direct control. If an application relies on an external model provider, availability and model behavior may be affected by that provider’s systems or updates. Monitoring, fallback planning and vendor review can help teams identify and manage those dependencies, though the excerpts do not claim that any one tactic prevents outages.

Amazon

disaster recovery software for distributed systems

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

From Site Failures to AI Dependencies

The source is a report in The Pragmatic Engineer that includes permitted excerpts from Newman’s book, published by O’Reilly Media in 2026. The editor says Newman spent three years writing the book, rather than the two years he initially expected, and revised material to account for AI’s relevance to system design. The book has 16 chapters; the report says its excerpts cover only a portion of the material.

Newman’s disaster recovery discussion uses “site” as a general term for the location of computing infrastructure, whether that is a public-cloud region or a private data center. The article’s opening also links increased code deployment and production bugs to AI agents writing more code, but provides no supporting statistics or incident data for that observation. It should be read as the report’s framing, not as a quantified finding.

“The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.”

— Sam Newman, in the book excerpt

Amazon

AI system monitoring tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Evidence and Implementation Details

The source is an excerpt and editorial introduction, not a survey of outages or a technical evaluation of AI providers. It does not establish that AI coding agents have caused a broad increase in production incidents; the report offers no figures, time period or comparison baseline for that claim. It also does not provide specific vendor availability data, test results for the recovery patterns, or a recommended RTO or RPO for a particular type of service.

How to apply the advice depends on each system’s data, user needs, vendor contracts and available infrastructure. The source flags possible CPU and GPU constraints but does not quantify their scale or say which providers or regions are affected. Details of the book’s AI resilience recommendations beyond the excerpted topics are also not included.

Amazon

cloud infrastructure backup solutions

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Teams Must Set Their Own Targets

The immediate next step for engineering teams applying the material is to define acceptable data loss and downtime, then choose and test a recovery design against those targets. That involves checking whether backup, failover capacity and traffic redirection work under realistic conditions, and confirming that critical services receive priority when infrastructure is limited.

For AI-dependent applications, teams can map model and vendor dependencies, monitor service behavior, make operating costs visible and review alternatives as providers or models change. The report does not announce a follow-up study or give a timetable for further findings. Newman’s book is available from its publisher, according to the article.

Amazon

multi-vendor AI model management tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Key Questions

What is the news?

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, about disaster recovery and resilience for AI-powered software.

What do RPO and RTO mean?

RPO is the maximum acceptable period of data loss, based on the last backup or synchronization. RTO is the acceptable time a service can remain unavailable after an incident.

Does the report show that AI agents are causing more outages?

No. The report raises that possibility in its introduction but supplies no outage data, time window or comparison baseline to demonstrate an increase.

What approaches does the excerpt discuss for AI system resilience?

It identifies model-provider resilience, model changes and observability as concerns, and suggests considering multiple models or vendors, transparent costs and regular vendor reviews. These are recommendations, not guarantees against failure.

Source: rss

EVERGREEN BESTSE

Evergreen bestsellers Picks

As an affiliate, we earn on qualifying purchases.

You May Also Like

Porting the ThinkPad X61 to Coreboot

A detailed report on how a developer successfully ported the ThinkPad X61 firmware to Coreboot, aided by AI-assisted reverse engineering techniques.

RHEO On The Web: Find Your Flow

Discover RHEO’s web version—an instant, private fluid simulation that offers calming, breathing, and creative experiences without downloads or sign-up.

Minerva. The opposite path.

Italy’s Minerva-3B, trained from scratch on 2.5 trillion tokens, scored only 4.9% on Italian academic tests, raising questions about scale and investment.

Apple Silicon’s Quiet Memory Advantage

Apple Silicon’s unified memory architecture offers a significant capacity advantage for large AI models, despite lower bandwidth and speed compared to NVIDIA GPUs.