Tag: Azure

This article aims to explain the differences between TOGAF and the Cloud Architecture Frameworks / Well-Architected Framework (WAF). Are you interested in understanding how these frameworks can be merged? This article provides some insight into these topics without delving too deeply into the specif...

Starting a Data Engineering project can be daunting, especially for newcomers. One of the initial challenges is understanding the flow of activities required to begin processing data effectively. This post aims to provide an example of a fundamental ETL process using Excel files, orchestrated with D...

In this article it’s gonna be explained two different methods that can be used for upserting data from the Databricks lakehouse platform to Azure SQL DB. Code snippets can be found on my github: ssaenzf/databricksLearning: Repo with code from articles related with the Databricks lakehouse plat...

In this story, we will learn how to configure Azure Databricks Unity Catalog with Terraform, and we will talk about how to design External Storage Accounts for Multiple Applications. In particular, we will learn: Creating Databricks External Storage Account for Multiple Applications ...

Choose the right data platform between Azure Synapse vs Databricks in terms of pricing, benefits, and use cases for your business. Azure Synapse Vs Databricks: A Comprehensive Guide For Businesses The role of data has had a fascinating evolution over the decades. What began as a byproduct ...

We can now provision Azure and Databricks infrastructure via Terraform as IaC. In this tutorial, I’m going to show you how. Requirements Terraform installed Azure CLI installed Azure account (free plan is enough) Terraform Code To get the latest version of this code please...

This series comprises 4 articles showcasing the comprehensive data engineering practice on the Azure platform. It encompasses the utilization of Azure Data Lake, DataBricks, Azure Data Factory, Python, Power BI, and Spark technology. In this initial part (part 1), we will acquaint ourselves with the...

Spot VMs make use of unused compute capacity in Azure, allowing you to save up to 90% on compute costs for workloads, with minimal impact on workload completion times. Azure Databricks automatically handles eviction/termination of Spot VMs by replacing them with new on-demand VMs at regular pri...

If you use external tables for your Delta lake, and want to migrate to Unity, this is a simple guide for accomplishing this. The differences between how to do this with a streaming table and a non streaming table are very minor, with the major difference being you must stop your stream, make your ne...

1. Intro: What is Azure Databricks, and What is it Used For? Azure Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale. The Azure Databricks Lakehouse Platform integrates with cloud stora...

In this story, we will learn how to configure Azure Databricks Unity Catalog with Terraform. In particular, we will learn: Creating Databricks Access Connector for the External Storage Account Creating Databricks Storage Credential Creating the External Azure Storage Account Creating ...

Introduction This blog post explains how to configure and build end to end CI/CD pipeline solutions on Databricks using Azure devops and best practices to deploy libraries in workspace using azure service principal in CI/CD pipeline for security aspects. A Typical Azure Databricks pipeline inc...

Introduction In today’s era of data-driven decision-making, a well-architected data pipeline is pivotal for any business. It not only empowers businesses to process large volumes of data but also delivers actionable insights in a timely manner. In this article, we’ll guide you through...

Azure Databricks is a collaborative analytics platform that is fully integrated with Microsoft Azure. It’s an Apache Spark-based analytics platform optimized for Azure, designed in collaboration between Microsoft and Databricks. Key Features 1. Managed Apache Spark: Azure Databricks p...

In the modern world of data-driven decision-making, developers and data scientists play a crucial role in harnessing the potential of data. Databricks is a unified analytics platform designed to help developers, data scientists, and analysts collaborate seamlessly on big data projects. Leveraging th...

TLTR: Clone this git project, set params and run 0_script.sh to deploy 1 ALDSgen2 hub and N Databricks spokes A data lake is a centralized repository of data that allows enterprises to create business value from data. Azure Databricks is a popular tool to analyze data and build dat...

As a small team of data engineers and data scientists, we often work on projects that involve designing and implementing data processing solutions for various customers. Recently, we had an interesting challenge: one of our customers had developed an optimisation algorithm t...

Streamlit is an amazing open-source app framework commonly used by data scientists to quickly make responsive web apps. Streamlit’s intuitive framework is very simple to learn, and can create beautiful dashboards and even more complex web applications within minutes. However, many data...

Streaming data from Kafka, Azure Event Hubs, etc. is a common practice in modern data processing and analytics. These platforms are designed to handle large volumes of real-time data and provide a reliable way to ingest and process streaming data. Event Hub is commonly used in scenarios where data n...

Kubernetes (K8s) has emerged as a popular container orchestration platform, and as organizations adopt K8s for their workloads, they must also consider robust secret management solutions. Azure Key Vault, a cloud-based secret management service provided by Microsoft Azure, is a powerful choice for s...

Today, we will discuss a common error when deploying an Azure Azure Storage Account with a Private Endpoint with Terraform. After we try to add a Storage Account Container, we receive the following error: Error: containers.Client#GetProperties: Failure responding to request: S...

This tutorial provides a comprehensive and easy-to-follow guide on how to use Terraform to create Azure resources. Whether you are a beginner or an experienced user, this tutorial will walk you through the process, from setting up your Azure account to deploying your first resource. Prerequires ...

Intro With the exciting new announcement for Terraform Tests I couldn’t help but take them for a test drive. For our use case we will build on a prior post Terraform Azure: Reusable SQL Database Configurations. In the last post we created a module to deploy SQL Servers. Today...

In this story, we will learn how to configure Azure Databricks Unity Catalog with Terraform. In particular, we will learn: Deploying SQL Warehouse Creating the Databricks Access Connector for the Unity Catalog Creating the Azure Storage Account for the Unity Catalog Creating...

Recently, I embarked on a particularly challenging automation task: automating the SCIM (System for Cross-domain Identity Management) provisioning within the Azure Databricks environment. This journey, spurred by the necessity to efficiently manage user access and identities, highlighted the importa...

After my last post about An Azure DevOps Pipeline to manage Azure resources using Terraform, I want to go a step further and build the Azure DevOps Project, Repository and Pipelines etc. using Terraform. That means we can manage the Azure DevOps resources using Infrastructure as Code and can cr...

Welcome aboard, tech enthusiasts and digital trailblazers! Today, we embark on a journey into the vast yet captivating world of Azure Kubernetes, where efficiency, scalability, and seamless performance align perfectly. In this article, we will not only explore the best practices that ensure a smooth...

In today’s ever-evolving technological landscape, complexity can often overshadow the immense potential of groundbreaking technology. One such technological marvel is Linux Kubernetes clusters on Azure, renowned for their ability to empower businesses with unmatched scalability, flexibility, a...

Title: Unleashing the Full Potential of Your Cloud Infrastructure: Insider Tips for Kubernetes on Azure Introduction: In this age of rapid technological advancements, businesses are increasingly embracing cloud computing to elevate their performance and efficiency to new heights. Among the myr...

Microsoft Azure App Service is a fully managed platform for building, deploying and scaling web apps. It is a PaaS (Platform as a service) that enables us to quickly deploy our app without worrying about infrastructure and performance. We can develop our favourite language, be it .NET, .NET Core,...

Science thrives on pushing boundaries, venturing into the unknown to unlock new possibilities. Yet, some questions lie hidden beyond the reach of classical computers, their complexities shrouded in a quantum-mechanical mist. This is where Azure Quantum Elements, a revolutionary cloud platform f...