Automating Azure Databricks with Databricks SDK (Python) – Step by Step Guide

Automating Azure Databricks with Databricks SDK (Python) – Step by Step Guide

When working with Azure Databricks, creating clusters, jobs, repos, and secrets manually from the UI does not scale. For real-world data engineering projects, automation is essential.

In this post, I explain how to automate Azure Databricks resource creation using the Databricks SDK for Python, step by step, in a clear and practical way.

Why use Databricks SDK?

  • Infrastructure as Code (IaC) for Databricks
  • Repeatable and consistent environments
  • Easy CI/CD integration
  • Less manual UI work, fewer errors

Prerequisites

  • An existing Azure Databricks workspace
  • Permission to create clusters, jobs, repos, and secrets
  • Python 3.9 or above
  • Databricks Personal Access Token (PAT)

Step 1: Create Python Virtual Environment

python -m venv .venv
source .venv/bin/activate   # macOS / Linux

Step 2: Install Databricks SDK

pip install databricks-sdk

Step 3: Configure Authentication

Set the Databricks workspace URL and PAT token as environment variables.

export DATABRICKS_HOST="https://adb-<workspace-id>.<region>.azuredatabricks.net"
export DATABRICKS_TOKEN="<your_personal_access_token>"

This allows the SDK to authenticate securely without hardcoding credentials.

Step 4: Test Connection to Databricks

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
me = w.current_user.me()
print("Connected as:", me.user_name)

If this runs successfully, your SDK connection is working.

Step 5: Create a Databricks Cluster using SDK

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

cluster = w.clusters.create(
    cluster_name="demo-sdk-cluster",
    spark_version="13.3.x-scala2.12",
    node_type_id="Standard_DS3_v2",
    num_workers=1,
    autotermination_minutes=20
)

print("Cluster ID:", cluster.cluster_id)

This creates a cluster programmatically instead of using the Databricks UI.

Step 6: Create a Databricks Job

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import Task, NotebookTask, NewCluster

w = WorkspaceClient()

job = w.jobs.create(
    name="demo-sdk-job",
    tasks=[
        Task(
            task_key="run_notebook",
            notebook_task=NotebookTask(
                notebook_path="/Shared/demo_notebook"
            ),
            new_cluster=NewCluster(
                spark_version="13.3.x-scala2.12",
                node_type_id="Standard_DS3_v2",
                num_workers=1
            )
        )
    ]
)

print("Job ID:", job.job_id)

This is the recommended way to run production workloads using job clusters.

Step 7: Create Databricks Repo (Git Integration)

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

repo = w.repos.create(
    url="https://github.com/your-org/your-repo.git",
    provider="gitHub",
    path="/Repos/demo-sdk-repo"
)

print("Repo created:", repo.id)

This enables Git-based development directly inside Databricks.

Step 8: Create Secret Scope and Secrets

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

w.secrets.create_scope(scope="demo-scope")
w.secrets.put_secret(
    scope="demo-scope",
    key="storage_key",
    string_value="my-secret-value"
)

Secrets should always be stored securely instead of being hardcoded.

Best Practices

  • Use job clusters instead of all-purpose clusters
  • Store configuration in YAML/JSON files
  • Make scripts idempotent (check before create)
  • Integrate SDK scripts into CI/CD pipelines

Conclusion

The Databricks SDK allows data engineers to fully automate workspace setup, making environments reproducible, secure, and production-ready.

If you are serious about Databricks in enterprise projects, SDK-based automation is not optional — it is essential.

Happy Automating 🚀

Popular posts from this blog

Exploring the Largest UK Employers: A Power BI Visualization

Image
Understanding employment distribution among top companies can provide valuable insights into industry dominance and workforce trends. In this blog, I analyze the largest employers in the UK using a Power BI table visualization, sourced from CompaniesMarketCap . Source:  CompaniesMarketCap Key Insights from the Data: Compass Group leads the ranking with 550,000 employees, dominating the food service industry. Tesco, the retail giant, follows with 330,000 employees. HSBC, a major player in banking, employs over 215,000 people. The total workforce among the top companies surpasses 1.98 million employees. Visualizing in Power BI: Using a table visualization, we can clearly compare the number of employees across different companies. Power BI’s sorting, aggregation, and filtering features enhance data readability and analysis. However, incorporating bar charts, conditional formatting, and KPIs could make the insights even more compelling. What’s Next? Would you add more interactive eleme...
Read more

Master Databricks Asset Bundles Through Hands-On Practice

15 min read | 100% Practical Guide Forget theory. Forget abstract examples. This is a hands-on, build-as-you-learn guide to mastering YAML through the lens of Databricks Asset Bundles (DABs) . By the end of this post, you'll go from never writing YAML to confidently deploying production-grade data pipelines as code. 🎯 What You'll Build: A complete Databricks workspace configuration including jobs, clusters, notebooks, and permissions—all defined in YAML and deployable with a single command. Level 0: YAML Basics BEGINNER The Golden Rules Rule #1: YAML uses spaces for indentation , never tabs. Standard is 2 spaces per level. Rule #2: YAML is case-sensitive . Name ≠ name Rule #3: Indentation = Structure . It defines parent-child relationships. ...
Read more

PySpark Important Last-Minute Notes

Image
If you are preparing for Data Engineering interviews , Spark projects , or need a quick PySpark revision , this post consolidates the most important PySpark concepts in one place. Best for: Data Engineers, Big Data Developers, Azure/Databricks/Microsoft Fabric users, and anyone doing last-minute interview preparation. What is PySpark? PySpark is the Python API for Apache Spark , an open-source distributed computing framework used for large-scale data processing. Why PySpark? Distributed, in-memory processing Faster than traditional batch systems Scales across clusters Supports SQL, Streaming, and Machine Learning Common use cases ETL / ELT pipelines Big data analytics Machine learning at scale Real-time and batch processing Spark Cluster Architecture A Spark cluster typically consists of: Master Node: Manages resources and schedules tasks Worker Nodes: Execute tasks in parallel This architecture enables efficient distributed p...
Read more