Databricks Launches Data Intelligence Platform
- By Paul Mah
- November 22, 2023
Databricks last week announced the launch of its Data Intelligence Platform, which is built on top of the lakehouse and will use generative AI to enhance data management.
The new offering incorporates the generative AI capabilities inherited from its USD1.3 billion acquisition of MosaicML earlier this year to make it easy for enterprises to create AI applications that understand their data.
According to Databricks, the Data Intelligence Platform will help enterprises deeply understand the semantics of their data, tailoring itself to each organization’s jargon and acronyms.
By observing how data is used in existing workloads, it will offer a tailored natural language interface for all users, from nonexperts to data engineers. Other benefits include automated management, semantic cataloging, as well as optimization for AI models, enhanced governance, and privacy.
Built for AI
Crucially, the platforms will enhance enterprise AI applications by connecting to the relevant business data and leveraging the semantics they learn to deliver accurate results. AI application developers no longer have to rely on what Databricks terms “brittle” prompt engineering.
Other benefits include end-to-end RAG (Retrieval Augmented Generation) to build high-quality conversational agents on custom data and the ability to train customer models either from scratch on an organization’s data or by continued pretraining of existing models such as MPT and Llama 2 to enhance AI applications further.
Enterprises will also be able to roll out end-to-end MLOps based on the popular MLflow open-source project, with all data produced automatically actionable, tracked, and monitorable in the lakehouse.
According to Databricks, data intelligence platforms will greatly simplify the development of enterprise AI applications. It will integrate with Mosaic AI to make it easier for enterprises to create AI applications that understand their data.
“Historically, data platforms have been hard for end-users to access and for data teams to manage and govern. Data intelligence platforms are set to transform this landscape by directly tackling both these challenges – making data much easier to query, manage, and govern,” explained Databricks in a blog signed off by top Databricks engineers and cofounder and CTO Matei Zaharia.
“As AI reshapes the software world, we believe that the leaders in every industry will be those who leverage data and AI deeply to power their organizations.”
Paul Mah is the editor of DSAITrends. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose. You can reach him at [email protected].
Image credit: iStockphoto/ArtemisDiana