MiniLake emulates the Databricks API locally with real Spark, SQL and Delta Lake execution

An open-source Databricks emulator

MiniLake, an open-source project that surfaced on the Hacker News front page on 29 October 2026, is a local emulator for the Databricks API. According to the project's site, it serves the Databricks REST surface on a single local port, so the databricks-sdk Python library, the Databricks CLI, the Terraform provider, Asset Bundles, PySpark and LLM agents can all be pointed at localhost with no cloud account, no API key and no compute bill. It is MIT-licensed, requires Python 3.11 or newer, and installs either through pip or as a Docker image.

Real execution instead of canned responses

The project positions itself against mock servers, on the argument that fake endpoints let broken code pass unnoticed. MiniLake instead runs SQL on DuckDB, writes genuine Delta Lake files (Parquet data plus a _delta_log transaction log) and executes spark-submit jobs in sibling containers. Statement execution supports the INLINE and EXTERNAL_LINKS dispositions with JSON_ARRAY, ARROW_STREAM and CSV result formats, and result manifests carry actual column types.

Unity Catalog objects — catalogs, schemas, tables and volumes — get full create, read, update and delete support, with each catalog stored as its own DuckDB database file so catalog.schema.table addressing works natively rather than through string rewriting. Writes to external Delta tables go through a real Spark job, reads use DuckDB's delta_scan, and PySpark clients resolve tables through the official Unity Catalog connector for Spark, the same protocol used when Spark talks to a real workspace. Job scheduling honours depends_on and run_if and runs independent branches in parallel. Types such as STRING, LONG, DECIMAL(10,2), ARRAY, MAP and STRUCT round-trip between the engines, and when a declared column type disagrees with the underlying files, the files win.

Drop-in for existing clients

Because the emulator speaks the Databricks REST API on the wire, the only change most clients need is the host address, and any token is accepted for local development. The project claims more than 30 API groups are covered, from Unity Catalog to SCIM. Errors come back in Databricks' own shape, with error_code and message fields, so the SDK raises its usual typed exceptions, and unimplemented endpoints answer with an explicit 501 rather than a fabricated success.

A browser-based SQL workspace is included, with a Monaco editor, catalog-aware autocomplete, charts, export to CSV or JSON, saved queries, query history and an embedded JupyterLab. Terraform support is exercised continuously: the project's test suite runs the genuine Databricks provider through plan, apply and destroy on every run and checks that a follow-up plan reports no changes, while databricks bundle deploy and bundle run work end to end.

Built for CI, agents and offline use

Several features target automated environments. A single POST request to a reset endpoint gives each test a clean workspace without restarting the container, health and readiness endpoints form a small control plane, and a MINILAKE_PERSIST setting snapshots state on shutdown and restores it at startup. The image is designed to run fully offline: the DuckDB Delta extension and the Delta and Unity Catalog jars are baked in at build time, and the project says this can be verified by running the container with networking disabled.

There is also a Model Context Protocol server on the same port exposing 67 tools, 8 resources and 4 prompts for LLM agents, including composite tools that collapse multi-step call sequences and a resource describing the SQL dialect so agents do not write Spark-only syntax. The project reports a suite of more than 378 tests driven by the real Databricks SDK rather than mocks.

Why it matters

Developing against Databricks normally requires a cloud workspace: an account, credentials and clusters that bill by the hour. That cost shapes behaviour — teams skip integration tests, avoid experimenting with Terraform and Asset Bundles, and hesitate to let agents loose on real infrastructure. A wire-compatible local emulator with real execution semantics changes that calculus: infrastructure code and applications can be exercised against the same API surface for free, in CI and offline, and behaviour that mocks hide becomes visible. The obvious caveat is fidelity — DuckDB and containerised Spark approximate a production workspace rather than replicate it, and gaps surface as 501 responses. But for the widening set of tooling that speaks the Databricks API, including MCP-driven agents, a free local target makes each iteration cost nothing.