Databricks Quick Reference & Cheat Sheet

• Cluster mode: Choose based on workload type (development, production, ML)
• Worker nodes: Autoscale between configured bounds for cost efficiency
• Photon: Enable for large-scale query acceleration
• Job clusters: Best for one-time or scheduled workloads — no upfront cost

-- Create a table in Unity Catalog
CREATE TABLE catalog.schema.table_name (
    id BIGINT,
    name STRING,
    value DOUBLE
);

-- Query via Serverless SQL Warehouse
SELECT * FROM catalog.schema.table_name LIMIT 10;

-- Change warehouse / query context
ALTER WAREHOUSE warehouse_name SET AUTOSTOP = 30 MINUTE;

Key takeaway: Databricks SQL warehouses (Standard or Serverless) handle BI/SQL workloads with separate compute from Notebook clusters.

-- Read by delta version
SELECT * FROM table_name VERSION AS OF 42;

-- Read by timestamp
SELECT * FROM table_name TIMESTAMP AS '2025-08-01 00:00:00';

ZORDER BY co-locates related information in the same set of files, drastically speeding up filtered queries.

# Read from various sources
df = spark.read.format("csv").option("header", "true").load("/path/to/file.csv")
df = spark.read.json("/path/to/file.json")
df = spark.read.parquet("/path/to/data/")

# SQL-Style Querying
spark.table("catalog.schema.table_name").createOrReplaceTempView("t")
results = spark.sql("SELECT col1, COUNT(*) FROM t GROUP BY col1")

from pyspark.sql.functions import col, avg, count, sum, max, min, when

df.agg(
    avg("price").alias("avg_price"),
    count("*").alias("total_records"),
    sum("quantity").alias("total_qty"),
    max("date").alias("latest_day")
)

-- Conditional aggregation with CASE WHEN equivalent
df.withColumn("tier", when(df.amount > 1000, "high").otherwise("low"))

-- Grant table read access
GRANT SELECT ON TABLE catalog.schema.table TO role_name;

-- Grant schema usage
GRANT USE SCHEMA ON SCHEMA catalog.schema TO role_name;

-- Revoke
REVOKE ALL ON TABLE catalog.schema.table FROM role_name;

Unity Catalog provides centralized governance across workspaces: one place for access control, data discovery, and audit logs.

%sql -- switch cell language to SQL
%python -- switch to Python  
%sh -- shell commands
%md -- Markdown rendering
%fs -- Databricks File System operations

@dlt.table
def clean_data():
    return (
        spark.readStream.format("delta").load("/path/to/source")
            .filter(col("status") == "active")
    )

DLT defines declarative data pipelines with built-in schema enforcement, incremental processing, and automatic refresh.

# Databricks CLI — create a job
databricks jobs create \
  --name "ETL Pipeline" \
  --new-cluster '{"spark_version":"14.3.x-scala2.12","NodeType":"r6id.xlarge", ...}'

# Run a job now
databricks jobs run-now --job-id 12345

# Listing files
dbutils.fs.ls("/path/to/dir")

# Copying
dbutils.fs.cp("file:/local/path", "/dbfs/mnt/gcs/bucket/folder")

# Creating directories
dbutils.fs.mkdirs("/dbfs/tmp/new_folder")

# Downloading from external storage (mount example)
dbutils.fs.mount(
  source="gs://bucket-name/path",
  mount_point="/mnt/gs/bucket-name",
  extra_configs={"fs.gs.auth.service.account.json.keyfile": "…"}
)

• Cluster mode: Use Standard for dev, High Concurrency for BI/shared SQL warehouses. Use Photon-enabled runtimes where possible.
• Delta Lake is the storage layer; leverage time travel and OPTIMIZE/ZORDER for performance.
• Unity Catalog centralizes governance (metastore → catalog → schema → table).
• Notebook cells support multi-language switching via %sql, %python, etc. — use them inline as needed.
• DLT & Workflows turn notebooks into production-ready, scheduled pipelines with clear lineage and SLAs.