Databricks, the company founded around the open-source Apache Spark engine, has unveiled a new architecture called LTAP (lake transactional/analytical processing) that it claims unifies online transactional processing (OLTP) and online analytical processing (OLAP) with 'no data duplication.' The marketing message — 'one data, zero compromises, zero copies' — has drawn scepticism from engineers who have examined the technical details.
LTAP works with Reyden, a new compute engine, and Lakebase, Databricks' serverless PostgreSQL database built on technology from Neon, which Databricks acquired last year. The architecture stores transactional data in a pageserver format for PostgreSQL, then propagates it to object storage in Parquet format for analytical queries. A Databricks engineer acknowledged on a private messaging community that technically there are two copies: one in the pageserver and one in object storage, with the pageserver acting as a cache or materialization layer.
The distinction matters because Databricks is positioning LTAP as a solution for workloads created by the booming deployment of AI agents in software development and business applications. Unifying OLTP and OLAP is a long-standing database challenge — OLTP performs small, row-oriented reads and frequent writes, while OLAP performs large, column-oriented reads and batch writes. Getting both to coexist in a single system is difficult at the physical storage level.
For UK businesses, the technology could reduce the complexity of managing separate databases for transactions and analytics, potentially lowering costs and speeding up AI-driven applications. However, the 'zero copies' claim may mislead organisations about storage costs and data consistency. The UK Information Commissioner's Office (ICO) and the EU AI Act both require transparency in data processing, and any ambiguity about data duplication could raise compliance questions for firms handling personal data.
Databricks is not alone in pursuing this goal. SingleStore, for example, began working on an in-memory row store and on-disk column store with tiered storage in 2014, and launched a cloud database service in 2020 that automatically manages data across memory, local cache, and storage. 'The market is heating up as AI agents demand real-time transactions and analytics on the same data,' said a data engineering analyst who spoke on condition of anonymity. 'But buyers should look past the marketing and examine the actual architecture.'
For UK consumers, the implications are indirect but significant: if LTAP delivers on its promise of faster, cheaper data processing, it could enable more responsive AI-powered services in banking, retail, and healthcare. But if the architecture introduces hidden costs or data consistency issues, businesses may pass those on to customers. The debate over 'zero copies' is likely to continue as Databricks refines its message and competitors challenge the claim.