Compute pool types
Dedicated
Dedicated compute pools provide isolated resources reserved for your workloads. Your queries don’t compete with other users for processing power, which results in more predictable performance. Use dedicated compute pools when:- Running production workloads where performance consistency matters
- Processing large or complex queries that need guaranteed resources
- Operating time-sensitive pipelines where latency must stay predictable
Shared
Shared compute pools use pooled resources across multiple users. This is more cost-effective but means your query performance may vary depending on current platform load. Use shared compute pools when:- Running exploratory queries or ad-hoc analysis
- Developing and testing queries before promoting to production
- Working with smaller datasets where performance variability is acceptable
Snowflake warehouse
On Snowflake-based data planes, each compute pool maps to a Snowflake virtual warehouse. When you register warehouses through the Snowflake Native App, each warehouse becomes a compute pool on your data plane. You can register multiple warehouses to separate workloads—for example, a smaller warehouse for exploratory queries and a larger one for production pipelines. Each Snowflake compute pool has a collaboration policy that controls which companies can use it, and one pool can be designated as the default for the data plane.AWS EMR
On AWS-based data planes, compute pools map to Amazon EMR Spark clusters that the data plane operator provisions, reuses across jobs targeting the same pool, and terminates when idle. Each pool has a configured size that determines the cluster’s worker memory budget and vCPU count. The operator schedules NQL jobs—includingmaterialize-view, nql-forecast, datasets_sample, and datasets_execute_dml—onto the appropriate cluster based on the compute pool selected for the workload.
Sizes mirror Snowflake’s warehouse vocabulary (x_small through 6x_large). vCPU counts won’t match Snowflake because EMR delivers a given memory budget with fewer, larger cores.
Fixed-size tiers (x_small through large)
The fixed-size tiers run at a fixed node count and set their Spark container sizes explicitly. Containers are uniform — one shape per node class, sized at half a node — so a node holds two containers and the driver simply occupies one of them. No tier spends a whole node on the driver, and every tier on the ladder delivers distinct executor capacity.
x_small runs on a general-purpose node because it is the only way to get a worker below 32 GiB: EMR offers no instance smaller than .xlarge, and a memory-optimized .xlarge is already 32 GiB.
Managed-scaling tiers (x_large and above)
Sizes x_large and above use EMR Managed Scaling: the cluster boots small and expands toward the maximum based on YARN load, then contracts back down when idle. These tiers target the same worker memory budget as the equivalent Snowflake warehouse at full scale, so a workload that fits in a Snowflake warehouse of size N gets equivalent RAM on the EMR cluster of size N.
Instance-storage (_storage) variants
Every size has a sibling _storage variant — x_small_storage, small_storage, medium_storage, large_storage, x_large_storage, 2x_large_storage, 3x_large_storage, 4x_large_storage, 5x_large_storage, 6x_large_storage — with the same worker memory budget and node counts as its base size, but with core and task nodes running on Graviton instances with local NVMe SSD instead of EBS-backed families. x_small_storage uses the general-purpose NVMe family (m7gd/m8gd); every other variant uses the memory-optimized NVMe family (r7gd/r8gd). The master node does no shuffle and needs no scratch disk.
Use a _storage variant when a job spills a large volume of shuffle or scratch data to disk and fails on an EBS-backed pool with No space left on device. Typical culprits are wide joins, large GROUP BY / DISTINCT aggregations, and materialized-view refreshes over very wide datasets. For jobs that are CPU- or memory-bound rather than shuffle-bound, stick with the base size — the local-NVMe families cost more per node.
You switch between the base and _storage tiers the same way you change any other size: update the pool’s provider.size via PATCH /data-planes/{id}/compute-pools/{poolId}, or create a new pool sized for the workload and pin it on the affected job or dataset.
Idle and job execution timeouts
EMR-backed pools expose two optional tunables that control cluster lifetime and per-job runtime. Both are set on theaws_emr provider when you create or update a pool, and both are validated server-side — invalid values return HTTP 400.
When either field is omitted (or set to
null) on create or update, the API fills in the default above before validating and persisting the pool. Explicit values — including the -1 disabled-idle sentinel — are always preserved. This matters when you update the pool’s size (for example, switching to a _storage variant): the provider block is replaced wholesale rather than merged, so a size-only update that omits the timeout fields still gets the standard 4h/15m defaults instead of silently dropping them. Set idle_timeout_seconds to -1 when you want a long-lived cluster — for example, a pool dedicated to back-to-back batch jobs where boot latency dominates run time.
The shared Narrative compute pool is configured with a 1-hour job execution timeout and is intended for small jobs. Pools you provision for yourself get the 4-hour default on create, and can set their own (longer or shorter) limit.
Which compute pools are available
The compute pool options available to you depend on your data plane’s underlying provider:
You select your compute pool through the context selector in the platform’s top navigation.
Default compute pool resolution
When a job is created without an explicit compute pool, the platform resolves one through a four-level fallback chain. Resolution happens at job-creation time, so a job’scompute_pool_id is fixed before it lands in Pending. The first level present wins.
On AWS data planes, a job must resolve to a compute pool. If none of the four levels supplies one, the platform rejects the request with
400 Bad Request and an error message that starts cannot enqueue job: no active compute pool available for data plane. Health check jobs are the only exception. On Snowflake data planes, a job that resolves to no pool is still accepted until the data plane adopts compute pools.
Set the level-3 default for a given data plane with:
compute_pools) write permission and return 204 No Content on success. To set either default from the data plane page in the platform instead, see Set a default compute pool.
Reading the effective default: default_source
Every compute pool in an API response carries a default_source that tells you whether that pool is the one the authenticated company’s jobs run on when they don’t name a pool themselves, and which level of the resolution chain made it the default. At most one active pool per data plane has a non-null value.
The field appears on the list, get-by-id, create, and update responses, and on the pools embedded in a data plane response. Two rules keep the reported value aligned with what job creation actually picks:
- A default that points at an archived pool falls through to the next level, so an archived pool is never reported as the default.
- On the shared Narrative data plane, only the level-3 company default can produce
"company"— the level-4 data plane default is a platform-wide constant that cannot hold a per-company answer, so"data_plane"never appears there.
When to use each type
Archiving compute pools
Deleting a compute pool is a soft delete: the pool’s status changes toarchived, the record stays in place, and it is filtered out of list responses but still resolvable by id. Archival behavior diverges by provider when the pool is still named as a default — by the data plane, a dataset, or a company — because each integration has different operational realities. On Snowflake data planes, the platform reconciles those references itself and the archive succeeds. On AWS data planes, the archive is rejected until each reference is cleared or changed.
Snowflake users frequently mutate their warehouse list outside the platform — through the Snowflake UI, permissions, or infrastructure as code. When a backing warehouse disappears, the platform has to track reality, so the archive must succeed and dependent references are updated automatically. AWS compute pools are wholly managed inside the platform, so the API rejects an archive that would leave a dangling default and forces an explicit decision.
Runtime impact on in-flight jobs
Neither archive flow blocks in-flight jobs at archive time. Instead, the data plane operator fails any job whose compute pool resolves to a non-active status on its next polling iteration — including a job that was mid-cancellation, because the missing pool is the actionable signal and a successful cancellation report would mask the root cause.
See the Archive a compute pool guide for what happens on each reference path, the error in-flight jobs report, and the order in which to clear references.
On AWS data planes, the first compute pool you create is not automatically promoted to the data plane default. This is the permanent behavior: jobs targeting an AWS data plane must either pin an explicit
computePoolId, resolve to a dataset-level default, or have a data plane default set explicitly via PUT /data-planes/{id}/default-compute-pool/{poolId}. The intent is that AWS workloads make a deliberate choice about where they run rather than implicitly routing through a “first pool wins” default.On Snowflake data planes, the first compute pool you create is currently auto-promoted to the data plane default for backward compatibility while the Snowflake-side migration off implicit defaults is in flight. This behavior is temporary and will be removed once every Snowflake workload pins its compute pool explicitly.Every newly registered company is provisioned a private, x-small AWS EMR compute pool on the Narrative data plane and that pool is set as the company-level default for the same data plane. Jobs that don’t pin a pool explicitly resolve to this default through the pool resolution chain (job → dataset → company → data plane). You can rename, archive, or replace the default at any time.
external_id is optional for aws_emr providers
When creating a compute pool with the aws_emr provider via POST /data-planes/{id}/compute-pools, you can omit external_id from the request body — the server fills in the trivial {"type": "aws_emr"} payload automatically. Supplying it explicitly still works, and the type is validated against the provider.
For the snowflake_warehouse provider, external_id is still required because it carries the warehouse name and alias that the platform uses to dispatch queries to the correct Snowflake warehouse.
How compute pools relate to the SDK
When executing queries through the TypeScript SDK, theexecution_cluster parameter maps to the compute pool concept:
execution_cluster.type accepts 'dedicated' or 'shared', corresponding directly to the Dedicated and Shared compute pool types. If omitted, the data plane’s default compute pool is used.
For Snowflake-based data planes, omitting execution_cluster uses the data plane’s default compute pool (the warehouse you’ve designated as default).
Related content
Execution Context
How data plane, compute pool, database, and schema work together
Data Planes
Where your data lives and is processed
Executing NQL Queries
Run queries programmatically with the TypeScript SDK
Migrate to Compute Pools
Transition from a single Snowflake warehouse to compute pools
Archive a Compute Pool
Safely retire a compute pool and clear its references