New cloud platform enables cost-effective data engineering
Businesses encounter a variety of challenges in building systems on and around Spark to meet the needs of data engineering.
Often engineers need to perform mission-critical data cleansing, transformations, and manipulations, to make business activities real-time dashboards or fraud detection possible. Mastering data engineering is therefore an essential step to automating systems and making data-driven decisions.
Spark-based data science platform Databricks is launching a new edition of its cloud platform optimized specifically for data engineering workloads.
Databricks for Data Engineering enables more cost-effective data engineering using Spark while empowering data engineers to easily combine SQL, structured streaming, Extract, Transform, Load (ETL), and machine learning workloads to rapidly and securely deploy data pipelines into production.
The new release will complement the company’s existing cloud platform by providing all enterprises with a unified data analytics platform that enables seamless collaboration to accelerate data-driven decisions across the organization.
"The expansion of our product portfolio to meet the needs of data engineering workloads is a major step in our journey to make big data simple for very complex data problems," says Ali Ghodsi, CEO and co-founder at Databricks. "Databricks for Data Engineering will offer organizations a unified environment for data science and data engineering users alike, while optimizing Apache Spark performance -- all with the reliability of an enterprise data and analytics platform at an efficient price."
Key features include performance optimization for faster processing speeds, plus optimizing for the AWS S3 access layer. Cluster management capabilities such as auto-scaling and AWS Spot instances reduces operational costs by avoiding time-consuming tasks needed to build, configure, and maintain complex Spark infrastructure.
It's designed to integrate with tools and services, such as Redshift and Kinesis, and machine learning frameworks such as TensorFlow. An integrated data sources catalog makes every data source immediately available to all Databricks users without duplicating data ingest work.
Enterprise levels of security are built-in with turnkey security standards including SOC 2 Type 1 certification and HIPAA compliance, end-to-end data encryption, detailed logs easily accessible in AWS S3 for debugging, and IT admin capabilities such as Single Sign-On with SAML 2.0 support and role-based access controls for clusters, jobs, and notebooks.
You can find out more about Databricks and start a free trial on the company's website.
Photo Credit: agsandrew/Shutterstock