TL;DR
Microsoft has released pg_durable as an open-source PostgreSQL extension to enable fault-tolerant, durable execution of SQL workflows directly inside the database. This development aims to simplify complex data and AI pipelines by reducing reliance on external orchestrators and infrastructure.
Microsoft has officially open-sourced pg_durable, a PostgreSQL extension that enables fault-tolerant, durable execution of workflows directly within the database, eliminating the need for external orchestrators or complex app-layer logic. This move aims to streamline data and AI pipeline development by embedding durable execution capabilities into PostgreSQL, making it accessible via Microsoft’s Azure HorizonDB cloud service.
pg_durable allows defining workflows as SQL graphs that execute and checkpoint at each step. If a crash, restart, or failure occurs, execution resumes from the last checkpoint, ensuring reliability without manual recovery. The extension is designed for use cases such as vector embedding pipelines, ingestion workflows, scheduled maintenance, fan-out aggregation, and external API calls, replacing traditional architectures involving cron jobs, queues, and external orchestrators.
Microsoft’s Azure HorizonDB, a new PostgreSQL cloud service, now includes pg_durable as part of its offering, highlighting Microsoft’s commitment to bringing compute close to data. The extension is available as a Debian package supporting PostgreSQL 17 and 18, with instructions for installation and configuration provided in the open-source repository. The extension runs as a PostgreSQL library, requiring no additional infrastructure, and integrates seamlessly with existing SQL workflows.
Why It Matters
This development matters because it introduces a new paradigm for building resilient, scalable data and AI pipelines entirely within PostgreSQL. By embedding durable execution in the database, organizations can reduce complexity, improve reliability, and gain operational visibility without managing separate orchestration layers or external services. This aligns with industry trends toward in-database compute and simplifies workflows for data engineers, DBAs, and SREs.
PostgreSQL extension pg_durable
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
Traditional data workflows often rely on external orchestrators like Airflow or Temporal, combined with queues and status tables, leading to complex architectures prone to failures and manual recovery. PostgreSQL extensions like pg_cron have provided scheduling, but lack true durability and fault tolerance. Microsoft’s release of pg_durable builds on the growing need for in-database workflow management, following broader industry moves toward embedded compute solutions. Previously, fault-tolerant workflows required custom app logic or external services, which added complexity and operational overhead.
“pg_durable brings fault-tolerant, in-database workflow execution directly into PostgreSQL, reducing infrastructure and simplifying data pipelines.”
— Microsoft spokesperson
“Integrating pg_durable into HorizonDB demonstrates our commitment to providing robust, high-performance PostgreSQL solutions optimized for modern data workloads.”
— Microsoft Azure HorizonDB team
Production-Grade AGENTIC AI Systems: Enterprise Orchestration & Multi-Agent Systems | Advanced Engineering Guide to Architect Zero-Trust, Fault-Tolerant Swarms and Scale Securely in Production
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It is not yet clear how widely adopted pg_durable will become outside of Microsoft’s Azure HorizonDB ecosystem, or how it will perform at scale in diverse production environments. Details about long-term support, community contributions, and compatibility with other PostgreSQL extensions are still emerging. Additionally, the extent to which organizations will replace existing orchestration solutions with pg_durable remains to be seen.
durable SQL workflow tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What’s Next
Next steps include broader community testing, feedback, and potential integration into other PostgreSQL distributions. Microsoft plans to continue developing pg_durable with new features and optimizations, and expects organizations to experiment with embedding durable workflows in their data pipelines. Further documentation, tutorials, and case studies are likely to follow, facilitating wider adoption and best practices.
PostgreSQL 17 compatible extensions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is pg_durable?
pg_durable is a PostgreSQL extension that enables fault-tolerant, durable execution of SQL workflows, allowing steps to checkpoint and resume after failures without external orchestration.
Who can benefit from pg_durable?
Data engineers, DBAs, SREs, and teams building AI or data pipelines that require reliable, in-database workflow execution will find it useful.
Is pg_durable available outside Microsoft Azure HorizonDB?
Yes, it is open-source and can be installed on PostgreSQL 17 and 18 instances, but its integration is currently promoted within HorizonDB’s cloud environment.
What are the limitations of pg_durable?
It is designed for SQL-shaped workflows; complex logic requiring arbitrary code, extensive in-memory control flow, or spanning many heterogeneous systems may still require external orchestrators.
How does pg_durable improve over traditional methods?
It consolidates workflow definition, checkpointing, and recovery within PostgreSQL, reducing reliance on external services, simplifying architecture, and increasing reliability.
Source: Hacker News
