Databricks Architecture

Two questions decide most of what matters about Databricks, and neither is about features. Where does the code actually run, and who holds the data it reads? The answers determine your isolation boundaries, your network design, your compliance story, and the shape of your bill.

Use the structure to understand the platform, and treat deployment specifics as dated: cloud product architecture changes, so check the current documentation for anything you are about to depend on. Examples are invented for illustration, and no pricing figures appear here because they vary by cloud, region, and contract.

Account, workspace, metastore

Three containers organize everything, and confusing them is the source of most early mistakes.

The account is the top-level construct. Identity and access — users, groups, service principals, provisioning — live here, as does the creation of workspaces across regions, the management of Unity Catalog metastores, and billing, compliance, and policy. Anything you want to be true for the whole organization is configured at this level.

A workspace is the collaboration environment where people actually work: ingestion, interactive exploration, scheduled jobs, model training. Workspaces are where notebooks, dashboards, and jobs live, which makes them the natural unit for separating teams or environments.

A Unity Catalog metastore is the governance system for data assets. It is attached to workspaces, and the documented behavior is the part with architectural consequences: a single metastore can be linked to multiple workspaces in the same region, which gives each of them the same view of the data and lets access controls be managed across all of them at once.

That sentence resolves a question teams otherwise answer badly. Separating teams into workspaces does not have to mean separating their data, and it should not. Workspaces are for separating work — who has which notebooks, which jobs run where, who can disturb whom. The metastore is for governing data. A design that gives every team its own workspace and its own metastore has quietly created several islands with independent permission models, which is the thing governance was supposed to prevent.

Two planes, and who owns each

The platform runs across a control plane and a compute plane, and the split is the basis of its security model.

The control plane holds the backend services Databricks manages in the Databricks account, including the web application. It is not in your cloud account.

The compute plane is where data is processed, and it comes in two forms. With serverless compute, the resources run in a serverless compute plane inside the Databricks account, created in the same region as the workspace’s classic compute plane. With classic compute, the resources run in your own cloud subscription or account, with new compute created inside each workspace’s virtual network.

	Classic compute	Serverless compute
Where it runs	Your cloud subscription, in the workspace’s virtual network	Databricks’ account, in the same region as the workspace’s classic plane
Isolation basis	Your own account, and a network that is yours to design where the deployment lets you	A network boundary per workspace, with layers separating customers and further network controls between clusters of the same customer
Network control	In your subscription, but how much of it you design depends on the deployment: on Azure, a Databricks-managed virtual network by default, or your own through VNet injection, where subnets, routing, private endpoints, and egress rules are yours	Databricks’ design, configured through its own controls such as network connectivity configurations, private connectivity, and egress policies
Startup	Provisioning virtual machines, which takes time	Managed by Databricks, oriented to fast start
Who is billed for the machines	You, by your cloud provider, in addition to the platform charge	Included in the platform charge

Classic and serverless describe compute. A separate choice applies to the workspace itself, and it decides where the workspace’s own storage sits. A classic workspace keeps its workspace storage and its classic compute plane in your cloud account, and it can also use serverless compute. A serverless workspace runs on serverless compute and uses Databricks-managed default storage for its internal workspace storage and for the default catalog created with it; further catalogs can be created in default storage or in your own cloud storage. So using serverless compute in a classic workspace does not relocate the classic workspace storage, while a serverless workspace keeps that storage in default storage from the start; creating one does not migrate an existing workspace’s data. Workspace storage is not the only place data can end up, though. In both kinds of workspace, some features use default storage for control plane metadata, derived data, models, and other artifacts — automated session restoration for serverless notebooks, data classification, and clean rooms among them — so a residency review has to check each feature in use rather than the workspace type alone.

The practical decision usually comes down to two things. If a security requirement is expressed in terms of your own network — traffic that must traverse specific endpoints, egress restricted by your rules, machines inside a perimeter you audit — classic compute is the natural starting point, because the machines run in your account. It satisfies the requirement only when the network is one you control and configure. On Azure, the default deployment places classic compute in a virtual network that Databricks manages; routing through your own endpoints and applying your own egress rules requires VNet injection into a network you provide, and then the routes and rules still have to be verified. It is also worth asking whether the requirement is about placement inside your network or about the controls themselves: serverless compute has its own private connectivity and egress controls, which may meet a requirement stated as an outcome. If the priority is fast start and not operating a fleet, serverless removes work that classic makes you do. Notice that this is a genuine trade rather than a maturity ladder: classic gives control at the price of running things, and serverless gives operational simplicity at the price of configuring isolation through someone else’s controls.

One detail worth carrying into a design review: for the same customer, the documentation describes additional network controls between clusters in the serverless plane, not only between customers. Whether a given workload’s isolation requirement is met by that is a question to answer against the current security documentation rather than an assumption either way.

What lives in workspace storage

Every workspace has storage, and it holds something different from the data everyone talks about. It contains two categories, both logically separate from your own data objects such as tables and volumes — a distinction of kind, which does not always mean a different bucket or account.

• Workspace file system data: what users create through the interface — notebooks, SQL queries and dashboards, alerts, Git-attached folders, libraries, and small configuration files.
• Workspace system data: what the platform generates internally — query results and cached results, job run results, notebook revisions, query plans kept for observability, and cluster logs.

In a classic workspace that storage account sits in your own cloud subscription, which means it is visible to your infrastructure team and subject to your policies. In a serverless workspace it is Databricks-managed default storage instead, so the warning that follows concerns classic workspaces. The documentation attaches an unusually direct warning to it: do not delete or modify the workspace storage in your cloud account, because the workspace depends on both its control plane databases and its workspace storage, and if the workspace storage is deleted the workspace cannot be recovered.

That is a sentence worth acting on before an incident rather than after. A storage account in your subscription will eventually be examined by someone tidying up costs, applying a lifecycle policy, or enforcing a naming standard. It should be labeled, excluded from generic cleanup rules, and covered by whatever change control protects production. The same caution applies to lifecycle rules: a policy that looks reasonable applied to a bucket of old files is destructive applied to this one.

The legacy item to know about is DBFS, and the deprecation is narrower than the name suggests. What Databricks documents as deprecated and not recommended is DBFS root and DBFS mounts, with Unity Catalog volumes, external locations, or workspace files recommended instead. The dbfs:/ scheme itself is not retired — the documentation describes it as an optional scheme when interacting with Unity Catalog volumes, so a path such as dbfs:/Volumes/… is current code, not legacy code.

That distinction matters for anyone auditing a codebase, because grepping for the prefix finds both and flags the wrong things. The question to ask of each path is what it resolves to — the workspace’s own root storage or a mount, which are the deprecated cases, or a Unity Catalog volume, which is not.

Unity Catalog: the governance layer

Unity Catalog is the unified governance layer for data and AI, applying access control across workspaces, tracking lineage, and recording activity for audit. Its structure is a three-level namespace, catalog.schema.object, and the objects it secures include tables, views, volumes, functions, and models.

Three parts of that description have consequences beyond naming.

• Models and functions are governed like tables. The same permission model covers a trained model and the table it was trained on, which closes a gap that otherwise appears when data is well controlled and the artifacts derived from it are not.
• Managed and external are different arrangements, and the dividing line is the storage lifecycle. Tables and volumes can be managed, where Databricks handles both governance and the underlying file storage, or external, where it handles governance only and the files stay where they are. The documented consequence is disposal: dropping a managed table has Databricks delete the data files after a recovery period, while for an external table you must delete the files from your own bucket yourself. Reading from another engine does not by itself require external tables. Unity Catalog can expose managed tables to external engines — Trino, DuckDB, Apache Spark, Daft, and Iceberg REST catalog–integrated engines among them — through open APIs and temporary, scoped credentials that inherit the requesting principal’s privileges. Whether a particular access works depends on more than managed or external. External access has to be enabled on the metastore and the principal granted EXTERNAL USE SCHEMA; the client has to support credential vending and the table’s format and protocol features; support differs between reads and writes; and the storage matters too — tables in Databricks-managed default storage do not support credential vending at all. This is the path for engines that read files directly; engines that query through Databricks over ODBC or JDBC are a different path with different requirements. Some tables are not served through this path at all — tables with row filters or column masks, views, materialized views, and streaming tables among them — which means those tables are not available to an external engine this way, not that their data is returned without the policies. So managed does not rule out external access, and choosing external does not guarantee it. Where an external reader or writer needs something the managed path does not offer — a client or operation it does not support, or a format other than Delta or Iceberg — that is a legitimate reason to choose external.
• Some objects sit outside the three levels. Storage credentials, external locations, connections, and shares live directly under the metastore. These are the objects that decide what the platform may reach, which makes them the ones to review first when granting access to a new storage account.

Lineage and audit come with it: the catalog tracks how assets flow from source data through to models, services, and dashboards, and keeps a record of data access and system activity in an audit log system table. That changes what is possible operationally. Questions normally answered by asking colleagues — which dashboards depend on this column, who read this table last quarter — become queries, within limits. Lineage returns the dependencies that were recorded, which makes it a list of candidates for impact review rather than a verdict on what a change will break; whether a dependent actually breaks depends on how it uses the column. What is recorded has limits of its own: RDDs and global temporary views are not captured, column lineage is captured only when source and target are referenced by table name rather than by path, and UDFs can hide the mapping between columns. How far back a question can reach depends on where it is asked: Catalog Explorer shows lineage captured since September 1, 2024 with no expiry, while the lineage system tables system.access.table_lineage and system.access.column_lineage keep a rolling one-year window, so a query that joins them to other data sees only the last year. Both lineage and audit records are visible only to users with the right permissions, and audit coverage depends on which events are logged and how long they are kept. These are gaps in what was observed, and filling in ownership and descriptions does not recover them. Missing owners and descriptions are a different gap — the one catalog coverage measures, as the share of assets carrying the metadata an organization requires — and the reason a named owner per dataset is worth insisting on.

The architectural point from earlier returns here, and it is a product constraint rather than a recommendation. A Databricks account has one Unity Catalog metastore per region, shared by the workspaces attached to it. So within one account, metastore count is not a way to isolate teams in the same region, and a multi-region deployment necessarily has more than one, which makes “how do definitions stay consistent across regions” a question every multi-region deployment has to answer rather than one some can avoid.

What that constraint pushes downward is the separation people usually want from separate metastores. Databricks’ guidance is to organize into catalogs and schemas that reflect organizational divisions and projects, with catalogs typically corresponding to an environment scope, team, business unit, or a combination — and to use the single-region metastore across development lifecycle scopes and business units rather than seeking separate ones. Where a team’s workspace and its data have the same isolation requirement, binding a catalog to specific workspaces adds a second boundary. The design work, in other words, is in the catalog layout and the workspace bindings, not in how many metastores to run.

On Azure specifically

The structure above is the same across clouds; what differs is which of your cloud’s primitives it lands on. For a classic workspace on Azure, classic compute resources are created in your Azure subscription inside the workspace’s virtual network, and the workspace storage account is likewise in your subscription. One naming quirk is worth knowing when reading the portal: classic workspaces appear there as Hybrid workspaces.

That placement requires the platform team to coordinate network, storage, and identity controls with their owners.

• Network. With VNet injection, classic compute sits in a virtual network you provide, so its connectivity is designed by whoever designs your networks: subnets, routing, private connectivity to storage, and egress control. With the default Databricks-managed virtual network, much less of that is yours to change. This is the integration that most often determines the project timeline, because it requires a team that does not otherwise work on data.
• Storage. Analytical data generally lives in your own storage accounts, reached through external locations and storage credentials registered in the metastore. When access goes through Unity Catalog, two controls act together: the catalog decides who may use the data and issues or uses the credential, and the storage account’s own controls decide what that credential can reach. Access that does not go through the catalog — a separate identity with its own storage role, an account key, a legacy mount — gets none of the catalog’s permissions, audit, or lineage, so those paths have to be closed, or governed and audited separately. Aligning the two sets of rules is not enough while a third path exists.
• Identity. Users, groups, and service principals are managed at the account level and generally federated from the organization’s directory, so that joining and leaving are handled once rather than per platform.

Because these details change and are specific to a subscription’s configuration, the current Azure documentation and your own cloud team are the authorities here rather than any general description.

Where the cost comes from

Billing has two components, and missing the second is an easy budgeting error. The platform charge is measured in Databricks Units: the pricing page defines a DBU as a normalized unit of processing power used for measurement and pricing, with consumption driven by processing metrics that may include the compute resources used and the amount of data processed. Separately, if you run classic compute in your own cloud account, your cloud provider bills you for the instances used there, and storage and networking costs vary with the services you choose and the provider. Rates differ by workload type and cloud and are not reproduced here.

The structure matters more than the rates, because it tells you where growth comes from. Six paths account for many surprises, and each has a signal you can watch for.

Where it grows	Why	Signal
Idle interactive compute	A cluster left running produces charges with no work done	Long-lived clusters with low utilization; auto-termination disabled
Oversized clusters	Sized for the worst job and used for every job	Jobs whose runtime barely changes when the cluster is halved
Interactive compute running scheduled work	Production jobs on an all-purpose cluster, typically priced above job compute	Scheduled jobs attached to a cluster people also use interactively
Cluster proliferation	Clusters created per person or team where sharing or policies would do	Idle time and compute hours rising faster than the work done. Count alone is only a prompt, since short job clusters and justified isolation also raise it
Inefficient jobs	Scanning far more than needed, or spilling for lack of memory	Bytes read against table size; spill in the query profile
Storage growth in the background	Snapshots, delete files, and small files accumulating without maintenance	Storage rising while the data volume does not; file counts and delete information growing on tables whose queries read them

The first four usually respond to guardrails more than to reports: auto-termination on by default, cluster policies that bound size and type, jobs running on job compute, and attribution that tells each team what it spends. The last two need diagnosis rather than one remedy. Excess scanning and spill show up in the query profile and have to be traced to a cause — a missing filter or pruning, a large join or sort, skew, too little memory for the workload, concurrent load — and fixed in the query, the partitioning, or the resources; file layout is one possible cause, not the default one. Background storage growth is table maintenance: compaction where small files are the overhead, and snapshot retention. For Unity Catalog managed tables, predictive optimization can run OPTIMIZE, VACUUM, and ANALYZE automatically, on serverless compute billed as serverless jobs and recorded in a system table; it does not run on external tables. For those, auto compaction and optimized writes can be configured to control file sizes as data is written — auto compaction runs after a write succeeds — but they are not a full replacement for OPTIMIZE, and they do nothing for VACUUM or ANALYZE. So what remains for each table is a decision per operation: which of them it needs, whether automatic management covers it, and otherwise how it is run and scheduled. Either way someone owns the outcome — whether automatic maintenance is enabled, whether retention meets time-travel and erasure needs, what it costs, and whether it works — even when nobody schedules the job.

Proliferation is easy to miss because each individual cluster looks reasonable, and the pattern only becomes visible in aggregate. But a rising count is a prompt to investigate, not proof of waste: more independent jobs, more concurrency, genuine isolation needs, and short-lived job clusters all raise it, and many short clusters that terminate automatically can use less compute than a few that sit idle. Compare run time, idle time, cost, work done, and the reason each cluster exists. Whether to share is partly an organizational rule and partly a technical question — shared compute has to accommodate the workloads’ libraries and runtime versions, their load, and any isolation they genuinely need.

Questions that reveal the wrong commitment

Question	If the answer is unclear
Classic or serverless workspace, which compute, and which requirement decided each?	The choice is made by whoever set up the first workspace
Which regions will you operate in, and how will definitions stay consistent across their separate metastores?	Each region develops its own permission model and its own version of the same table
What does the catalog layout separate — environment, team, business unit — and which catalogs are bound to specific workspaces?	Isolation is assumed to exist somewhere it was never configured
Who owns the workspace storage account, and is it excluded from cleanup automation?	A tidy-up destroys a workspace that cannot be recovered
Which tables are managed and which external, and who then owns their storage lifecycle?	Files are governed by the catalog and cleaned up by nobody
Is any code still reading through legacy DBFS paths?	A deprecated path becomes an urgent migration on someone else’s schedule
Who designs the network, and has that team been engaged?	The timeline slips on an integration the data team cannot do itself
Can each team see its own consumption, and are cluster policies in place?	Cost is optimized centrally by people who cannot judge the workloads

References

Feature availability and pricing also change; confirm them against current documentation for the deployment in use.

• Microsoft Learn, High-level architecture: Azure Databricks
• Databricks Documentation, Databricks architecture overview
• Databricks Documentation, Unity Catalog
• Databricks Documentation, Unity Catalog best practices
• Databricks Documentation, What is DBFS?
• Databricks Documentation, Managed tables in Unity Catalog
• Databricks, Pricing
• Databricks Documentation, Create a workspace
• Databricks Documentation, Default storage in Databricks
• Microsoft Learn, Deploy Azure Databricks in your Azure virtual network (VNet injection)
• Databricks Documentation, Manage network policies for serverless egress control
• Databricks Documentation, Unity Catalog credential vending for external system access
• Databricks Documentation, Enable external data access to Unity Catalog
• Databricks Documentation, Lineage in Unity Catalog
• Databricks Documentation, Audit log system table reference
• Databricks Documentation, Predictive optimization for Unity Catalog managed tables
• Databricks Documentation, Query profile
• Databricks Documentation, Control data file size