Delta Lake to be hosted by the Linux Foundation
All organizations want to get more value from their data, but can be hampered by the lack of reliable information within data lakes.
The Delta Lake project addresses data reliability challenges by making transactions ACID (Atomicity, Consistency, Isolation, Durability) compliant enabling concurrent reads and writes. It also helps to ensure that the data lake is free of corrupt and not-conformant data.
From today Delta Lake will become a Linux Foundation project and use an open governance model. First announced by Databricks earlier this year, Delta Lake has already been adopted by thousands of organisations and has a thriving ecosystem of supporters.
"Bringing Delta Lake under the neutral home of the Linux Foundation will help the open source community dependent on the project develop the technology addressing how big data is stored and processed, both on-prem and in the cloud," says Michael Dolan, VP of strategic programs at the Linux Foundation. "The Linux Foundation helps open source communities leverage an open governance model to enable broad industry contribution and consensus building, which will improve the state of the art for data storage and reliability."
Delta Lake will have an open governance model that encourages participation and technical contribution and will provide a framework for long-term stewardship by an ecosystem invested in Delta Lake's success. Although initially designed to work with Apache Spark, Delta Lake the community is adding support for other open source data systems.
Databricks' CEO and co-founder Ali Ghodsi -- one of the original co-creators of the open source Apache Spark project -- is excited to be going through this journey again with Delta Lake. "Our team has continued to create and contribute to open source projects because we know it is the fastest, most comprehensive way to innovate. To address organisations’ data challenges, we want to ensure this project is open source in the truest form. Through the strength of the Linux Foundation community and contributions, we're confident that Delta Lake will quickly become the standard for data storage in data lakes."
You can find out more on the Delta Lake site.
Image credit: thampapon1/depositphotos.com