1. Part 1 talked about what Delta Tables are and why they’re ideal for long-term archiving

2. A hands-on demo: converting a large CSV to a Delta Table and comparing the size ← you’re here

3. Querying Delta Tables with DuckDB, no Spark required

I wanted a dataset big enough to feel real. Something closer to what actually accumulates in a production database over a few years. We’ll be using the NYC Parking Violations dataset from Kaggle. It contains four fiscal years of parking tickets issued across New York City.

• 42,339,438 rows

• 8.4 GB as a CSV

That felt like enough to work with.

The Migration

I used delta-rs, the Python package, to write the data to a local Delta Table. No Spark, no cluster. Just Python reading the CSV in chunks and writing it out as Parquet files within the Delta Table format.

The full code is in the companion repo here. The README.md file has instructions to follow along.

import pandas as pd
import pyarrow as pa
from deltalake import write_deltalake

CHUNKSIZE = 1_000_000
first = True

for chunk in pd.read_csv("data/nyc_parking_violations.csv", chunksize=CHUNKSIZE, dtype=str):
    chunk.columns = [c.replace(" ", "_").replace("/", "_") for c in chunk.columns]
    table = pa.Table.from_pandas(chunk, preserve_index=False)
    write_deltalake("data/delta_table", table, mode="overwrite" if first else "append", schema_mode="merge")
    first = False

The script reads a million rows at a time, writes to Delta, and repeats until done.

Share

What do the numbers say

The delta table was 4.9x smaller on the first write, and 6.5x after running a couple of extra steps.

The initial write produces one Parquet file per chunk. 43 Snappy-compressed files in this case. Running OPTIMIZE compacts those into fewer, larger files and rewrites them using ZSTD compression, which squeezes them down further. VACUUM then cleans up the old Snappy versions. Together they saved another 443 MB.

from deltalake import DeltaTable
dt = DeltaTable("data/delta_table")
dt.optimize.compact()
dt.vacuum(retention_hours=0, enforce_retention_duration=False, dry_run=False)

Dictionary Encoding

Where Parquet shines is when you have columns with repeated values. It applies a concept called Dictionary Encoding automatically, when the ratio of unique values to total rows is low enough to make it worthwhile. A colour column with 5,745 unique values across 42 million rows is a perfect candidate. A column of unique transaction IDs is not.

In this case, we will take the column Vehicle_Color as an example.

Across 42 million rows, there are only 5,745 unique colour values (5,745 unique colours? — clearly a free-text field?!). Things like WHITE, BLACK, GREY, BLK, GY. In a CSV, every row stores the full string. In Parquet, each unique value is stored once in a dictionary, and each row just stores a small integer pointing to it.

5x smaller on a single column.

A Note on the Delta Log

You may ask if Parquet does all the heavy lifting why store data in a delta table format. The Delta Table format is what maintains ACID compliance. It keeps track of all the changes to the table from the time when it was created. When you look at the Delta Table folder after running everything, you’ll see this:

data/delta_table/
├── _delta_log/
│   ├── 00000000000000000000.json   ← WRITE (overwrite, chunk 1, 1M rows)
│   ├── 00000000000000000001.json   ← WRITE (append, chunk 2, 1M rows)
│   ├── ...                         ← WRITE (append, chunks 3-42)
│   ├── 00000000000000000043.json   ← OPTIMIZE (43 files to 20 files)
│   ├── 00000000000000000044.json   ← VACUUM START (43 files to delete)
│   └── 00000000000000000045.json   ← VACUUM END (completed)
├── part-00000-c6e7a34b-...zstd.parquet
├── part-00000-bc09554a-...zstd.parquet
└── ...

Opening the early log entries, you’ll find snappy.parquet in the add records. That’s the initial write using Snappy compression by default. In the OPTIMIZE entry, you’ll find those replaced with .zstd.parquet files. Snappy is the default in most tools because it’s fast to read and write. ZSTD trades a bit of compute for better compression ratios, making it a better fit for files you’re storing long term and reading less frequently. The Parquet docs cover the full list of supported codecs if you want to go deeper.

Each JSON entry records exactly what happened: which files were added, which were removed, how many rows, min/max values per column. Here’s what a single append entry looks like:

{ "commitInfo": { "operation": "WRITE", "operationParameters": { "mode": "Append" },
    "operationMetrics": { "num_added_files": 1, "num_added_rows": 1000000 } } }
{ "add": { "path": "part-00000-27e284b6-...snappy.parquet", "size": 36893533,
    "stats": { "numRecords": 1000000,
      "minValues": { "Issue_Date": "01/01/2013", ... },
      "maxValues": { "Issue_Date": "12/25/2013", ... },
      "nullCount": { "Vehicle_Make": 5589, ... } } } }

And here’s what the OPTIMIZE entry looks like:

{ "commitInfo": { "operation": "OPTIMIZE",
    "operationMetrics": {
      "numFilesAdded": 20,
      "numFilesRemoved": 43,
      "filesAdded":   { "totalFiles": 20, "totalSize": 1370418125 },
      "filesRemoved": { "totalFiles": 43, "totalSize": 1834829450 } } } }
{ "remove": { "path": "part-00000-357b4b21-...snappy.parquet", "size": 51763870 } }
{ "add":    { "path": "part-00000-c6e7a34b-...zstd.parquet",   "size": 77495774,
    "stats": { "numRecords": 2000000, ... } } }

43 small Snappy files compacted into 20 larger ZSTD files. Total size dropped from 1.83 GB to 1.37 GB.

And once VACUUM runs:

{ "commitInfo": { "operation": "VACUUM START",
    "operationMetrics": { "numFilesToDelete": 43, "sizeOfDataToDelete": 1834829450 } } }
{ "commitInfo": { "operation": "VACUUM END",
    "operationParameters": { "status": "COMPLETED" },
    "operationMetrics": { "numDeletedFiles": 43 } } }

43 files, 1.75 GB reclaimed.

It’s a complete audit trail. Every operation, in order, with the numbers to back it up. In a future post I will write in detail about how you can infer more information from these json files using a function built within delta_rs.

Bringing It Back to the Cost

The numbers here are for a CSV, not a SQL Server table. I don’t have a SQL Server instance that can hold 43 million rows for free, but I’m planning to do that comparison in a future post. But the point from Part 1 still stands. SQL Server stores data row by row, adds index overhead, and keeps an instance running whether you’re querying or not.

What’s Next

Part 3: querying this Delta Table with DuckDB. No Spark, no cluster. Just SQL on your laptop against 42 million rows. I hope this series has been helpful. Please subscribe if you want to be notified for the future posts. I will add an additional post to this series to show how the above example works the same way in a cost effective object storage like ADLS. See you soon.

This is the first article in a series on Delta Lake storage. Over the next few posts, we’ll cover:

1. What Delta Tables are and why they’re ideal for long-term archiving ← you’re here

2. A hands-on demo: converting a large CSV to a Delta Table in ADLS (object storage) and comparing the size

3. Querying Delta Tables with DuckDB, no Spark required

Let’s start with the foundation.

Not All Data Needs the Same Home

In most data teams, there’s a natural split in the data you manage. There’s the stuff that gets hit constantly: current records, recent transactions, the tables your dashboards and reports depend on. And then there’s the historical data: audit logs, aged-out transactions, archived records that might get queried once a quarter, during a compliance review, or when someone needs to investigate something from two years ago.

SQL Server and Postgres are great at what they do. Fast lookups, structured storage, supporting the day-to-day reporting and transformation workloads your team runs. But when you’re also storing hundreds of millions of rows that nobody touches regularly, it’s worth asking: does this data need to live here?

For cold data, there’s often a better fit. Blob storage, structured as a Delta Table, gives you the same query ability at a fraction of the cost, without the overhead of a database instance sitting idle most of the time.

What Is a Parquet File?

Before we get to Delta Tables, we need to talk about Parquet, because Delta Tables are built on top of it.

Parquet is a columnar storage format. That distinction matters more than it sounds.

In a row-based format like a CSV, data is stored row by row. If an analyst wants to calculate the average of a single column across 100 million rows, the system has to scan every row to get there.

Parquet flips this. Data is stored column by column. If you only need three columns out of fifty, Parquet reads only those three. For the kinds of workloads data teams run (aggregations, filters, summaries) this is a meaningful difference.

On top of that, Parquet applies compression per column. Because each column tends to hold similar values (think of a status column that only ever contains “active”, “inactive”, or “pending”), compression algorithms like Snappy or ZSTD can squeeze that data down significantly. A CSV that’s 2GB on disk might land at 200-300MB as a Parquet file. We’ll show this with a real demo in the next article.

What Is a Delta Table?

Delta Lake is an open source storage framework that brings reliability to data lakes. A Delta Table is a layer built on top of Parquet files that adds reliability and manageability.

Here’s how it looks on disk:

/my-delta-table/
├── _delta_log/
│   ├── 00000000000000000000.json
│   ├── 00000000000000000001.json
│   └── ...
├── part-00000-abc123.snappy.parquet
├── part-00001-def456.snappy.parquet
└── ...

The Parquet files are the actual data. The _delta_log/ folder is what makes it a Delta Table. It’s a transaction log that records every operation ever performed on the table: inserts, updates, deletes, schema changes. But it also stores statistics about the data itself — things like the minimum and maximum values in each column, null counts, and row counts per file. This metadata is what allows Delta to skip over entire files when you run a query with a filter, without having to open and scan them first.

This gives you things that raw Parquet files don’t. Writes are atomic, so a failed pipeline run doesn’t leave your table in a broken state. Schema enforcement means Delta rejects data that doesn’t match the table’s schema, so you don’t silently corrupt your archive. Time travel lets you query the table as it existed at any point in the past. VERSION AS OF 5 or TIMESTAMP AS OF ‘2024-01-01’ are real queries you can run. And partition pruning means that if you partition your table by date or region, queries that filter on that column skip entire partitions, making reads much faster even on large datasets.

Putting It Together

Let’s say your team has five years of order records. They’re not going anywhere, but they’re also not being queried every day. Here’s what the two approaches look like side by side:

Keeping it in SQL Server / Postgres:

You’re paying for a managed database instance even when nobody’s querying that data. Storage costs are higher than blob storage. Backups and maintenance windows cover this data too. Over time, the database grows heavier, which can affect the performance of your active workloads.

Moving it to a Delta Table on Object Storage:

Data sits in blob storage, one of the cheapest storage tiers available. You pay for storage only and compute comes in only when you query. No instance to manage, no maintenance windows. It scales to petabytes without many infrastructure changes, and tools like Spark, Databricks, Azure Synapse, Microsoft Fabric and (as we’ll cover later) DuckDB can query it directly.

A Pattern That Works in Practice

A common approach: your database handles current and recent data, and a scheduled pipeline (Airflow, Azure Data Factory, or even GitHub Actions, we’ve covered those in earlier posts) moves aged-out records to a Delta Table in object storage on a regular cadence. Your database stays lean, your archive stays query-able, and your storage costs stay predictable.

Different data, different home. It’s a simple idea, but it makes a real difference at scale.

What’s Next

In the next article, we’ll make this concrete. I’ll take a large CSV file, write it to ADLS as a Delta Table using Python and the delta-rs library, and show you the size difference side by side. No Spark cluster required.

Share

The goal: every time dbt runs, log the results including model names, execution times, test outcomes, and timestamps into a dedicated table for monitoring and analysis.

You can find the updated repository here: DBTActionRunner

Subscribe now

Why Log Run Results?

When dbt runs in production (or CI/CD), you lose visibility into what happened unless you dig through logs. By persisting run metadata to a database table, you can:

• Track performance trends over time

• Monitor test failures and model execution status

• Audit pipeline runs with unique invocation IDs

• Query execution history using SQL instead of parsing log files

The Implementation

dbt-core provides on-run-end hooks which are SQL or macros that execute after a dbt run completes. These hooks have access to the results context variable, which contains metadata about every model and test that ran.

Step 1: Add the Hook to dbt_project.yml

First, configure dbt to call a custom macro after each run:

on-run-end:
  - "{{ log_run_results() }}"

This tells dbt to execute the log_run_results() macro when the run finishes.

Step 2: Create the Logging Macro

Create a new file at macros/log_run_results.sql:

{% macro log_run_results() %}
    {% if execute %}

        -- Create the logging table if it doesn't exist
        {% set create_table_sql %}
            IF NOT EXISTS (SELECT * FROM sys.tables WHERE name = 'dbt_run_results' AND schema_id = SCHEMA_ID('{{ target.schema }}'))
            BEGIN
                CREATE TABLE {{ target.schema }}.dbt_run_results (
                    run_id UNIQUEIDENTIFIER DEFAULT NEWID(),
                    invocation_id NVARCHAR(255),
                    run_started_at DATETIME2,
                    model_name NVARCHAR(255),
                    status NVARCHAR(50),
                    execution_time FLOAT,
                    rows_affected INT,
                    message NVARCHAR(MAX),
                    created_at DATETIME2 DEFAULT GETDATE()
                )
            END
        {% endset %}

        {% do run_query(create_table_sql) %}

        -- Insert run results
        {% for result in results %}
            {% set insert_sql %}
                INSERT INTO {{ target.schema }}.dbt_run_results 
                (invocation_id, run_started_at, model_name, status, execution_time, rows_affected, message)
                VALUES (
                    '{{ invocation_id }}',
                    '{{ run_started_at }}',
                    '{{ result.node.unique_id }}',
                    '{{ result.status }}',
                    {{ result.execution_time }},
                    {% if result.adapter_response.get('rows_affected') %}{{ result.adapter_response.rows_affected }}{% else %}NULL{% endif %},
                    '{{ result.message | replace("'", "''") }}'
                )
            {% endset %}

            {% do run_query(insert_sql) %}
        {% endfor %}

        {{ log("Run results logged to " ~ target.schema ~ ".dbt_run_results", info=True) }}

    {% endif %}
{% endmacro %}

What this does:

1. Creates the table (dbt_run_results) if it doesn’t exist

2. Loops through each result from the dbt run

3. Inserts metadata for every model and test executed

4. Captures key metrics: execution time, status, invocation ID, timestamps

What Gets Logged

After running dbt build or dbt run, the table contains entries like this:

Key fields:

• invocation_id: Groups all models/tests from a single dbt run

• model_name: Full identifier (e.g., model.Football_Analytics.stg_goals_per_season)

• status: success, error, pass, fail, skipped

• execution_time: Seconds to execute

• rows_affected: Actual row count for models (NULL for tests)

• created_at: When the log entry was written

Querying the Results

Now you can analyze your dbt runs using SQL:

Find the slowest models

SELECT 
    model_name,
    AVG(execution_time) as avg_execution_time,
    MAX(execution_time) as max_execution_time,
    COUNT(*) as run_count
FROM dbt_run_results
WHERE model_name LIKE 'model.%'
GROUP BY model_name
ORDER BY avg_execution_time DESC

Check recent test failures

SELECT 
    run_started_at,
    model_name,
    status,
    message
FROM dbt_run_results
WHERE model_name LIKE 'test.%'
  AND status = 'fail'
ORDER BY run_started_at DESC

Track runs over time

SELECT 
    CAST(run_started_at AS DATE) as run_date,
    COUNT(DISTINCT invocation_id) as total_runs,
    SUM(CASE WHEN status IN ('success', 'pass') THEN 1 ELSE 0 END) as successful_items,
    SUM(CASE WHEN status IN ('error', 'fail') THEN 1 ELSE 0 END) as failed_items
FROM dbt_run_results
GROUP BY CAST(run_started_at AS DATE)
ORDER BY run_date DESC

Integration with CI/CD

Since this runs as an on-run-end hook, it works automatically in your GitHub Actions pipeline. Every time the workflow executes dbt build, the results get logged and no additional configurations are needed.

From the workflow file:

- name: Run dbt Build and Test
  run: |
    echo "Running dbt debug to confirm connection..."
    dbt debug --target ci

    echo "Running dbt build (runs, tests, seeds) on CI target..."
    dbt build --target ci

After this step completes, the dbt_run_results table is automatically populated.

How Row Counts Are Captured

The macro queries each model after it’s built to get the actual row count:

{% set count_query %}
    SELECT COUNT(*) as row_count FROM {{ relation }}
{% endset %}

This adds minimal overhead (one COUNT query per model) but provides valuable insight into data volume trends over time. Tests show NULL for rows_affected since they’re validation queries, not data transformations.

Summary

By leveraging dbt’s on-run-end hooks, you can build a lightweight monitoring system that:

• Requires no external tools

• Stores data in your existing database

• Integrates seamlessly with CI/CD

• Provides queryable execution history

This approach gives you full visibility into your dbt pipeline without needing dbt Cloud or third-party observability platforms.

Check out the full implementation in the DBTActionRunner repo, and feel free to adapt it for your own setup.

You can find the full repository here: DBTActionRunner

The Architecture

The setup involves three main components:

1. dbt Project: The standard models, seeds, and tests.

2. Profiles.yml: Configured to use environment variables for database connectivity.

3. GitHub Workflow: The .yml file that orchestrates the environment setup and execution.

Subscribe now

1. Handling the Profile

The biggest hurdle in running dbt in a CI/CD environment is the profiles.yml. You don’t want to check credentials into version control. Instead, I’ve configured the profile to pull from environment variables:

football_analytics:
  target: ci

  outputs:
    ci:
      type: sqlserver
      driver: 'ODBC Driver 18 for SQL Server' # Adjust the version if necessary
      
      # Use environment variables (which will be populated by GitHub Secrets)
      server: "{{ env_var('DBT_SERVER') }}" 
      port: 1433
      database: "{{ env_var('DBT_DATABASE') }}" 
      schema: "{{ env_var('DBT_SCHEMA') }}" 
      user: "{{ env_var('DBT_USER') }}" 
      password: "{{ env_var('DBT_PASSWORD') }}" 
      

2. Managing Secrets

To make the above work, I’ve stored the actual database credentials in GitHub Secrets (Settings > Secrets and variables > Actions). This ensures that the runner has access to the database during execution without exposing sensitive data in the logs.

3. The Workflow Implementation

The workflow file (.github/workflows/main.yml) defines the lifecycle of the runner. Here’s the breakdown of the steps I implemented:

• Trigger: Set to on: push to the main branch.

• Environment: Uses ubuntu-latest.

• Python Setup: Installs Python 3.12 to handle the dbt core and adapter installation.

The core steps in the YAML:

name: dbt CI/CD Pipeline - Football Analytics
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  workflow_dispatch: null
jobs:
  dbt_ci_job:
    runs-on: ubuntu-latest
    env:
      DBT_SERVER: ${{ secrets.DBT_SERVER }}
      DBT_DATABASE: ${{ secrets.DBT_DATABASE }}
      DBT_SCHEMA: ${{ secrets.DBT_SCHEMA }}
      DBT_USER: ${{ secrets.DBT_USER }}
      DBT_PASSWORD: ${{ secrets.DBT_PASSWORD }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: 3.12
      - name: Install dbt and Dependencies
        run: |
          # 1. Install prerequisites for the MS SQL Driver (unixODBC)
          sudo apt-get update
          sudo apt-get install -y unixodbc unixodbc-dev
          
          # 2. Add Microsoft repositories to fetch the official driver
          curl https://packages.microsoft.com/keys/microsoft.asc | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc
          # Assuming Ubuntu 22.04 runner, adjust if using a different image
          curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list

          # 3. Update repos and install the ODBC Driver 18 (msodbcsql18)
          sudo apt-get update
          sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18

          # 4. Install dbt adapter and packages
          pip install dbt-sqlserver
          dbt deps
      - name: Run dbt Build and Test
        run: >
          echo "Running dbt debug to confirm connection..."
          dbt debug --target ci
          echo "Running dbt build (runs, tests, seeds) on CI target..."
          dbt build --target ci
      - name: Clean up
        run: >
          echo "Dbt run complete."

Summary

By offloading the execution to a GitHub Action Runner, the project is no longer dependent on a local machine’s uptime or configuration. It provides a clean, reproducible environment for every change pushed to the repo.

If you’re looking to implement this, feel free to fork the DBTActionRunner repo and swap out the adapter and credentials for your own.

Share

A powerful but often overlooked way to do this is with GitHub Actions runners. These allow you to execute Python (or any language) scripts inside a CI/CD pipeline, triggered by events such as code pushes, schedules, or manual dispatches. In other words, you can automate data tasks without standing up servers or relying on heavier ETL platforms.

Why GitHub Actions Matter for Data Engineers

Traditionally, data engineers rely on managed ETL tools like Airflow, Azure Data Factory, or AWS Glue. These are excellent in many cases, but they do not always fit every scenario. Sometimes you need to integrate with platforms that don’t have a ready-made connector. Sometimes you want lightweight automation without setting up infrastructure. Other times, you need fine-grained control over how data is pulled, transformed, and pushed.

GitHub Actions runners fill that gap. If you can write it in Python, you can run it in a runner. You can set up jobs to run on a schedule, after a code push, or whenever you manually trigger them. Secrets and credentials can be stored securely in GitHub Secrets, and the entire process lives alongside your code under version control. This combination of automation, security, and flexibility makes runners especially powerful for data engineers.

An Example Using My Repository

To make this more concrete, let’s look at my GitHub repository . It contains an example integration script that uses pandas to capture data from the GitHub API, writes the results to a CSV file, and saves the file as a build artifact. The GitHub Actions workflow takes care of setting up Python, installing dependencies, running the script, and saving the output.

Here is the workflow file:

name: Run Python Script

on: 
  workflow_dispatch:   # run manually from GitHub
  push:                # run when code is pushed to main
    branches: [ main ]

jobs:
  run-script:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run my script
        run: python test.py

      - name: Upload CSV artifact
        uses: actions/upload-artifact@v4
        with:
          name: github-events
          path: github_events_*.csv
          retention-days: 1

This workflow ensures that whenever it runs, the script executes in a fresh environment and the resulting CSV is stored for one day as an artifact that can be downloaded from the Actions page. It demonstrates the core idea: runners can automate the boring parts of data movement while letting you focus on the logic inside your script.

Subscribe now

Why This Matters for Your Career

Being able to set up lightweight integrations with GitHub Actions shows that you understand more than just databases and SQL. You are applying software engineering practices like version control and CI/CD to data engineering work. You can handle cross-platform data flows without being tied to a single vendor. And you can deliver secure, flexible automation at a moment’s notice.

For hiring managers, this signals that you are adaptable and resourceful. For your team, it means faster prototyping and fewer infrastructure headaches. For you, it is an extra tool in your toolkit that will make you stand out as a data engineer who is not only comfortable with data, but also with the software practices that surround it.

Managed ETL tools are not going away, and they will always play a big role. But data engineers who know how to use runners and workflows have a secret weapon. You can integrate anything with an API. You can spin up quick jobs without waiting for infrastructure. You can deliver value faster and more flexibly.

The reality is, agility in a tech environment isn’t about rituals or tools. It’s about how we think, and how willing we are to change when it matters.

For me, being agile means staying open to feedback, making room for course correction, and moving forward without getting stuck in our own plans. It’s not about skipping structure. It’s about not being owned by it. You still plan, just in smaller chunks. You still commit, but with eyes open to what might shift. Agile teams listen more. They adapt quickly. They know that learning is part of the process, not something that happens after a launch.

And honestly, none of it works without the right team culture. You can follow the ceremonies perfectly and still be rigid. Or you can have no formal framework and still respond well to change. It comes down to whether people feel trusted to take ownership and safe to challenge ideas. The tools don’t create that. The people do.

Agility is less about speed and more about staying aligned. With your users, your purpose, and your team. It’s about working in a way that lets you learn something, and then actually do something with it.

So if you're in tech, ask yourself. Are you using agile tools, or are you actually working in an agile way?

If this resonated, I write about tech, teams, and working with intent. Subscribe to get future posts straight to your inbox.

I’ve found that work goes better when I’m intentional. It’s not always easy, and I don’t always get it right, but I try to be mindful of how I show up. How I speak, how I listen, how I respond. But every time I put in the effort to be intentional, it has turned out positive.

It can be as simple as taking a moment before sending a message, or making sure I really hear what someone’s saying in a meeting instead of just planning what I’ll say next. Sometimes it’s checking my tone. Other times, it’s holding back a reaction I’d rather not lead with.

I try to give feedback with some care. To ask questions instead of assuming. To notice when something feels off and not ignore it.

It’s not about being perfect or polished. It’s just about trying to be present. To match what I do with what I mean, and to do my part in making things a bit easier for the people I work with.

This isn’t something I’m writing to boast. I’m just trying to figure out how to keep relationships at work feeling human in a world that can feel increasingly flat and repetitive. If you’ve been feeling the same way, I hope this helps you feel a little less alone.

Subscribe now

I started my career wrangling data with SSIS (SQL Server Integration Services). Later, I shifted to Python as the industry evolved. But wow, has the landscape changed. These days, being a data engineer doesn’t just mean moving data around. You’re expected to be part DevOps, part software engineer, part cloud whisperer. And sometimes, all of that.

For a while, I was drowning in the learning curve. Every new tool came with a learning path longer than my weekend to-do list. I began to think, maybe I’d get more value from diving deep into one tool or concept at a time, and learning it by doing. In my experience, the real “aha!” moments come when you’re knee-deep in terminal errors and finally get something working.

That moment hit when I got curious about Apache Airflow. I’d heard all the buzz: DAGs this, orchestration that. I wanted to get hands-on, but running it in the cloud? Not so fast. Most cloud platforms don’t hand out free credits like they used to, and I wasn’t exactly keen to hand over my credit card just to spin up a test environment.

But then it hit me. Airflow is open source. Linux is open source. And I’m on Windows, with WSL (Windows Subsystem for Linux). Lightbulb moment. Why not just run Airflow locally for free, using Linux on Windows? No billing alerts, no surprise charges. Just me, my laptop, and a fresh Ubuntu instance.

So that’s what I did. And it worked.

I’ve included a step-by-step guide for setting up Airflow on WSL, for anyone else who wants to skip the cloud bills and get straight to learning. It’s not perfect, but it’s real, it’s practical, and most importantly, it’s yours.

Instructions to Install Apache Airflow Using WSL with Ubuntu

1. Enable WSL and Install Ubuntu

Open PowerShell as Administrator and run:

wsl --install

Restart your computer if prompted.

2. Install Ubuntu from Microsoft Store

Open the Microsoft Store, search for Ubuntu 22.04 LTS, and install it.

Or you can download it from https://releases.ubuntu.com/jammy/

3. Set Up Ubuntu

Launch Ubuntu from the Start menu and complete the initial setup by creating a username and password.

4. Update Package Lists

In the Ubuntu terminal, run:

Bash is used in cmd (Command Prompt) in Windows to provide a Linux-like environment for running Linux-based commands and tools, allowing users to work with Linux-compatible software on Windows

bash
sudo apt update

5. Install Required Dependencies

Install Python, pip, and venv:
sudo apt install python3 python3-pip python3-venv

6. Create and Activate a Virtual Environment

Create a virtual environment:

bash
python3 -m venv airflow_env

Activate the virtual environment:

bash
source airflow_env/bin/activate

7. Install Apache Airflow

Install Airflow:

bash
pip install apache-airflow

8. Initialize the Airflow Database

Initialize the database:

bash
airflow db init

9. Start Airflow Components

Open Two Terminal Windows:

In the first terminal, activate the virtual environment and start the scheduler:

bash
source airflow_env/bin/activate
airflow scheduler

In the second terminal, activate the virtual environment again and start the web server:

bash
source airflow_env/bin/activate
airflow webserver --port 8080

10. Access Airflow UI

Open a web browser and navigate to http://localhost:8080 to access the Airflow UI.

If you have been in tech for a while the username and password should be obvious to you ;). If it isn’t please reach out!

Start Airflow-ing, the fun (and free) way!

You might be wondering why these topics? Its mainly because I can’t pick one….. So I thought why not write about the the things I love and have some level of experience in. On their own, they’re powerful. Lets see where this goes together.

What can you expect?

Every week (or two or three 😋), I’ll be sharing:

• Reflections from the tech world that I can keep on top of

• Insights on using data to make smarter, braver decisions

• Graphs I wish existed in sports

• Stories and lessons from coaching

• Frameworks, questions, and mental models for thoughtful leadership

This newsletter is for the curious, the analytical, and the quietly ambitious. For people who build, lead, coach, or just want to think more deeply about how we work and what drives us.

Why am I doing this?

I often find myself wondering why certain tools succeed, why some sports teams consistently win, and why some IT teams outperform others. I usually have opinions on these things—but I’ve rarely shared them. Part of that’s fear, part of it’s the habit of keeping ideas to myself. Let’s see if I can overcome this fear and be willing to share my ideas.

I’ve also spent countless late nights buried in Stack Overflow threads (yeah, before ChatGPT was around), searching for answers to tricky technical problems. Over time, I’ve built up a quiet library of knowledge—tucked away in notes and saved files—that might actually help someone else. So this newsletter is my way of starting to share what I’ve learned, what I think (not that everything is useful 😁) , and what I’m still figuring out.

So… if any of this sounds interesting, feel free to hit subscribe, follow along, or click whatever button feels right. I’d love to have you on the journey.

Subscribe now