Tag: pipelines

Not all data issues are created equal. While some issues may be merely annoying other issues spark hire-on-fire-moments causing your company to bleed $100,000 each day. At Synq, we work with dozens of companies that are seeing an explosion in business-critical data use cases. A few years ago data...

Today, this is what I am selling: awesome_pipeline.fit(X, y) awesome_pipeline may look just like another variable, but here is what it does to poor X and y under the hood: Automatically isolates numerical and categorical features of X. Imputes missing va...

Building reliant, scalable, efficient and production-ready data pipelines is vital to many modern businesses today to manage and visualize data effectively and make data-driven business decisions. What’s more, teams are often required to integrate the data pipeline with a full-fledged GUI a...

Consider the journey of clean water from its source to your tap! In much the same way, data within organizations follows a complex path, starting from various sources and databases and requiring processing before it becomes valuable for decision-making. This intricate journey is facilitated by...

In this article, let's quickly discuss (hack) how to monitor UC+DLT pipelines owned by the Service Principals. As per Databricks Documentation, it can be done using a Table Valued Function, event_log. Start a Shared Compute Cluster or DB SQL Warehouse and execute the followi...

Introduction GitOps is a process for automating cloud deployment using Git, infrastructure-as-code, and CI/CD tools. Your code repository will be the single source of truth for your infrastructure. At any point in time, your infrastructure-as-code configuration will mirror the existing ...