![]() In order to fully understand Delta Lake and the business benefits it brings, let’s take a step back and look at its beginnings. The evolution from data warehouse to Delta Lake So, how can UK businesses get away from this complexity? Where do they begin? How do they also avoid the ‘lock in’ which comes from using proprietary vendors? These are questions which have been at the heart of the development of our open source Delta Lake. If this all sounds complex on paper then it most definitely is in practice! Many businesses will be forgiven for finding it almost impossible to scale and maintain a useful data layer. Moreover, the major issue with many data architectures is the sheer number of data technologies needed to cover structured, unstructured, semi-structured and binary data formats coupled with the need for further different technologies for SQL, data engineering, streaming/batch, data science and machine learning (ML). See also: Lindsay Carter on ITV’s ambitious data transformation Why? Because their current data architectures require a whole host of different skill sets to run and manage, not to mention the ability to govern data across so many technologies and still be compliant with legislation like GDPR. Getting a handle on data drives operational efficiencies, and can even create real-world impact such as slashing CO2 emissions or speeding up drug discovery.īut, whilst the intent to innovate with data is certainly there, the reality for many businesses is that they still cannot unlock the true value of their data, writes Toby Balfre, VP Field Engineering, Databricks. Many UK businesses now realise the power of data and AI. The Stack asked Toby Balfre, VP Field Engineering, Databricks, to tell us more about the technology and the rationale behind providing the crown jewels to the open source community. It's the foundation of Databricks’ flagship platform the Data Lakehouse - the company defines a "Lakehouse" as "a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization" - and widely used, with some six million downloads per month. ![]() ![]() Delta Lake is a open-source storage framework that lets users build a "Lakehouse architecture" with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python. Last month Databricks made Delta Lake entirely open source. ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |