Databricks Architecture: Where Your Code Runs and Who Holds the Data

Two questions decide most of what matters about Databricks, and neither is about features. Where does the code actually run, and who holds the data it reads? The answers determine your isolation boundaries, your network design, your compliance story, and the shape of your bill. Everything else is detail on top.

This article describes the architecture as documented on September 15, 2026 — the Azure reference cited was itself updated on September 11, 2026. Cloud product architecture changes, so treat the structure as durable and the specifics as dated, and check the current documentation for anything you are about to depend on. Examples are invented for illustration, and no pricing figures appear here because they vary by cloud, region, and contract.

Account, workspace, metastore

Three containers organize everything, and confusing them is the source of most early mistakes.

The account is the top-level construct. Identity and access — users, groups, service principals, provisioning — live here, as does the creation of workspaces across regions, the management of Unity Catalog metastores, and billing, compliance, and policy. Anything you want to be true for the whole organization is configured at this level.

A workspace is the collaboration environment where people actually work: ingestion, interactive exploration, scheduled jobs, model training. Workspaces are where notebooks, dashboards, and jobs live, which makes them the natural unit for separating teams or environments.

A Unity Catalog metastore is the governance system for data assets. It is attached to workspaces, and the documented behavior is the part with architectural consequences: a single metastore can be linked to multiple workspaces in the same region, which gives each of them the same view of the data and lets access controls be managed across all of them at once.

That sentence resolves a question teams otherwise answer badly. Separating teams into workspaces does not have to mean separating their data, and it should not. Workspaces are for separating work — who has which notebooks, which jobs run where, who can disturb whom. The metastore is for governing data. A design that gives every team its own workspace and its own metastore has quietly created several islands with independent permission models, which is the thing governance was supposed to prevent.

Two planes, and who owns each

The platform runs across a control plane and a compute plane, and the split is the basis of its security model.

The control plane holds the backend services Databricks manages in the Databricks account, including the web application. It is not in your cloud account.

The compute plane is where data is processed, and it comes in two forms. With serverless compute, the resources run in a serverless compute plane inside the Databricks account, created in the same region as the workspace’s classic compute plane. With classic compute, the resources run in your own cloud subscription or account, with new compute created inside each workspace’s virtual network.

	Classic compute	Serverless compute
Where it runs	Your cloud subscription, in the workspace’s virtual network	Databricks’ account, in the same region as the workspace’s classic plane
Isolation basis	Natural isolation, because it is your own account and network	A network boundary per workspace, with layers separating customers and further network controls between clusters of the same customer
Network control	Yours: subnets, routing, private endpoints, egress rules	Databricks’ design, configured through its own controls
Startup	Provisioning virtual machines, which takes time	Managed by Databricks, oriented to fast start
Who is billed for the machines	You, by your cloud provider, in addition to the platform charge	Included in the platform charge

The practical decision usually comes down to two things. If a security requirement is expressed in terms of your own network — traffic that must traverse specific endpoints, egress restricted by your rules, machines inside a perimeter you audit — classic compute is the model that satisfies it directly, because the machines are yours. If the priority is fast start and not operating a fleet, serverless removes work that classic makes you do. Notice that this is a genuine trade rather than a maturity ladder: classic gives control at the price of running things, and serverless gives operational simplicity at the price of configuring isolation through someone else’s controls.

One detail worth carrying into a design review: for the same customer, the documentation describes additional network controls between clusters in the serverless plane, not only between customers. Whether a given workload’s isolation requirement is met by that is a question to answer against the current security documentation rather than an assumption either way.

What lives in workspace storage

Every workspace has storage, and it holds something different from the data everyone talks about. It contains two categories, both separate from your own data objects such as tables and volumes.

• Workspace file system data: what users create through the interface — notebooks, SQL queries and dashboards, alerts, Git-attached folders, libraries, and small configuration files.
• Workspace system data: what the platform generates internally — query results and cached results, job run results, notebook revisions, query plans kept for observability, and cluster logs.

In a classic workspace that storage account sits in your own cloud subscription, which means it is visible to your infrastructure team and subject to your policies. The documentation attaches an unusually direct warning to it: do not delete or modify the workspace storage in your cloud account, because the workspace depends on both its control plane databases and its workspace storage, and if the workspace storage is deleted the workspace cannot be recovered.

That is a sentence worth acting on before an incident rather than after. A storage account in your subscription will eventually be examined by someone tidying up costs, applying a lifecycle policy, or enforcing a naming standard. It should be labeled, excluded from generic cleanup rules, and covered by whatever change control protects production. The same caution applies to lifecycle rules: a policy that looks reasonable applied to a bucket of old files is destructive applied to this one.

The legacy item to know about is DBFS, and the deprecation is narrower than the name suggests. What Databricks documents as deprecated and not recommended is DBFS root and DBFS mounts, with Unity Catalog volumes, external locations, or workspace files recommended instead. The dbfs:/ scheme itself is not retired — the documentation describes it as an optional scheme when interacting with Unity Catalog volumes, so a path such as dbfs:/Volumes/… is current code, not legacy code.

That distinction matters for anyone auditing a codebase, because grepping for the prefix finds both and flags the wrong things. The question to ask of each path is what it resolves to — the workspace’s own root storage or a mount, which are the deprecated cases, or a Unity Catalog volume, which is not.

Unity Catalog: the governance layer

Unity Catalog is the unified governance layer for data and AI, applying access control across workspaces, tracking lineage, and recording activity for audit. Its structure is a three-level namespace, catalog.schema.object, and the objects it secures include tables, views, volumes, functions, and models.

Three parts of that description have consequences beyond naming.

• Models and functions are governed like tables. The same permission model covers a trained model and the table it was trained on, which closes a gap that otherwise appears when data is well controlled and the artifacts derived from it are not.
• Managed and external are different arrangements, and the dividing line is the storage lifecycle. Tables and volumes can be managed, where Databricks handles both governance and the underlying file storage, or external, where it handles governance only and the files stay where they are. The documented consequence is disposal: dropping a managed table has Databricks delete the data files after a recovery period, while for an external table you must delete the files from your own bucket yourself. Reading from another engine is a separate question, not a reason to choose external. Unity Catalog exposes managed tables to external engines — Trino, DuckDB, Apache Spark, Daft, and Iceberg REST catalog–integrated engines among them — through open APIs and temporary, scoped credentials that inherit the requesting principal’s privileges, so governance still applies. What interoperability actually depends on is the table format, the API, whether your client supports it, and whether the principal has the privileges — not on which of the two arrangements you picked.
• Some objects sit outside the three levels. Storage credentials, external locations, connections, and shares live directly under the metastore. These are the objects that decide what the platform may reach, which makes them the ones to review first when granting access to a new storage account.

Lineage and audit come with it: the catalog tracks how assets flow from source data through to models, services, and dashboards, and keeps a record of data access and system activity in an audit log system table. That changes what is possible operationally. Questions that are normally answered by asking colleagues — which dashboards break if this column changes, who read this table last quarter — become queries. Whether they get asked depends on whether anyone populates the ownership and descriptions around the automatic metadata, which is the gap catalog coverage measures and the reason a named owner per dataset is worth insisting on.

The architectural point from earlier returns here, and it is a product constraint rather than a recommendation. Databricks states that you can have only one metastore per region, and that all workspaces in that region share it. So there is no design choice to make about metastore count within a region, and a multi-region deployment necessarily has more than one, which makes “how do definitions stay consistent across regions” a question every multi-region deployment has to answer rather than one some can avoid.

What that constraint pushes downward is the separation people usually want from separate metastores. Databricks’ guidance is to organize into catalogs and schemas that reflect organizational divisions and projects, with catalogs typically corresponding to an environment scope, team, business unit, or a combination — and to use the single-region metastore across development lifecycle scopes and business units rather than seeking separate ones. Where a team’s workspace and its data have the same isolation requirement, binding a catalog to specific workspaces adds a second boundary. The design work, in other words, is in the catalog layout and the workspace bindings, not in how many metastores to run.

On Azure specifically

The structure above is the same across clouds; what differs is which of your cloud’s primitives it lands on. On Azure, classic compute resources are created in your Azure subscription inside each workspace’s virtual network, and the workspace storage account is likewise in your subscription. One naming quirk is worth knowing when reading the portal: classic workspaces appear there as Hybrid workspaces.

Three integration points follow from that placement, and each belongs to a different team in most organizations.

• Network. Because classic compute sits in your virtual network, its connectivity is designed by whoever designs your networks: subnets, routing, private connectivity to storage, and egress control. This is the integration that most often determines the project timeline, because it requires a team that does not otherwise work on data.
• Storage. Analytical data generally lives in your own storage accounts, reached through external locations and storage credentials registered in the metastore. Access is therefore governed twice — by the catalog and by the storage account’s own controls — and the two need to agree.
• Identity. Users, groups, and service principals are managed at the account level and generally federated from the organization’s directory, so that joining and leaving are handled once rather than per platform.

Because these details change and are specific to a subscription’s configuration, the current Azure documentation and your own cloud team are the authorities here rather than any general description.

Where the cost comes from

Billing has two components, and missing the second is the most common budgeting error. The platform charge is measured in Databricks Units: the pricing page defines a DBU as a normalized unit of processing power used for measurement and pricing, with consumption driven by processing metrics that may include the compute resources used and the amount of data processed. Separately, if you run classic compute in your own cloud account, your cloud provider bills you for the instances used there, and storage and networking costs vary with the services you choose and the provider. Rates differ by workload type and cloud and are not reproduced here.

The structure matters more than the rates, because it tells you where growth comes from. Six paths account for most surprises, and each has a signal you can watch for.

Where it grows	Why	Signal
Idle interactive compute	A cluster left running produces charges with no work done	Long-lived clusters with low utilization; auto-termination disabled
Oversized clusters	Sized for the worst job and used for every job	Jobs whose runtime barely changes when the cluster is halved
Interactive compute running scheduled work	Production jobs on an all-purpose cluster, typically priced above job compute	Scheduled jobs attached to a cluster people also use interactively
Cluster proliferation	Every team creates its own rather than sharing or using policies	Cluster count growing faster than headcount
Inefficient jobs	Scanning far more than needed, or spilling for lack of memory	Bytes read against table size; spill in the query profile
Storage growth in the background	Snapshots, delete files, and small files accumulating without maintenance	Storage rising while the data volume does not; query time drifting upward

The first four are configuration and habit rather than technical problems, and they respond to guardrails more than to reports: auto-termination on by default, cluster policies that bound size and type, jobs running on job compute, and attribution that tells each team what it spends. The last two are engineering work — the same maintenance discussed as compaction and retention elsewhere, which is a scheduled job somebody has to own.

The row that most often goes unnoticed is proliferation, because each individual cluster looks reasonable. It is worth measuring directly, since the pattern only becomes visible in aggregate, and the remedy — shared compute with isolation where it is genuinely needed — is a policy decision rather than a technical one.

A short checklist before committing

Question	If the answer is unclear
Serverless or classic compute, and which requirement decided it?	The choice is made by whoever set up the first workspace
Which regions will you operate in, and how will definitions stay consistent across their separate metastores?	Each region develops its own permission model and its own version of the same table
What does the catalog layout separate — environment, team, business unit — and which catalogs are bound to specific workspaces?	Isolation is assumed to exist somewhere it was never configured
Who owns the workspace storage account, and is it excluded from cleanup automation?	A tidy-up destroys a workspace that cannot be recovered
Which tables are managed and which external, and who then owns their storage lifecycle?	Files are governed by the catalog and cleaned up by nobody
Is any code still reading through legacy DBFS paths?	A deprecated path becomes an urgent migration on someone else’s schedule
Who designs the network, and has that team been engaged?	The timeline slips on an integration the data team cannot do itself
Can each team see its own consumption, and are cluster policies in place?	Cost is optimized centrally by people who cannot judge the workloads

Questions to explore further

• If your security team wrote the isolation requirement in their own words, would classic or serverless satisfy it, and have they read the current documentation for the one you chose?
• What fraction of your compute spend last month came from clusters that were running but not executing anything?
• For your most important table, could another engine read it today — which API would it use, and does that engine’s client support it?
• List your catalogs and say what each one separates. Was that layout a decision, or the order things were created in?

References

All sources were checked on September 15, 2026. The Azure architecture reference carried a documentation date of September 11, 2026. Cloud platform architecture, feature availability, and pricing change frequently; confirm against current documentation before relying on any of it.

• Microsoft Learn, High-level architecture: Azure Databricks
• Databricks Documentation, Databricks architecture overview
• Databricks Documentation, Unity Catalog
• Databricks Documentation, Unity Catalog best practices
• Databricks Documentation, What is DBFS?
• Databricks Documentation, Managed tables in Unity Catalog
• Databricks, Pricing