> ## Documentation Index
> Fetch the complete documentation index at: https://docs.narrative.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets
> Understand how datasets store, organize, and protect your data in Narrative
A dataset is a structured collection of data registered in Narrative. Think of it as a table in a database—it has a defined schema, holds rows of data, and can be queried, shared, and collaborated on. Datasets are one of the core primitives in the Narrative platform.
## What datasets are for
Datasets serve as the primary way to bring data into Narrative and make it available for collaboration:
**Storage and organization.** Datasets provide a structured container for your data. Each dataset has a defined schema that specifies what fields exist, their data types, and how they should be validated.
**Querying.** Once data is in a dataset, you can query it using [NQL](/concepts/nql/design-philosophy). Datasets are the foundation of all query operations in Narrative.
**Collaboration.** Through [access rules](/concepts/primitives/access-rules), you can grant other organizations permission to query your datasets—enabling data sharing, monetization, or joint analysis.
## How datasets are structured
### Schema
Every dataset has a schema that acts as its structural blueprint. The schema defines:
* **Field names** — The columns that exist in the dataset
* **Field types** — The data type for each field (string, number, timestamp, etc.)
* **Descriptions** — Documentation explaining what each field contains
* **Validations** — Rules that ensure data integrity when records are added
```typescript theme={null}
// Example schema definition
{
properties: {
customer_id: { type: 'string' },
event_type: { type: 'string' },
event_timestamp: { type: 'timestamptz' },
event_value: { type: 'double' },
}
}
```
Schemas are designed to be stable. While you can add new fields to a schema, changing or removing existing fields requires careful consideration to avoid breaking queries or integrations that depend on them.
### Records and snapshots
Data in a dataset is organized into records (rows) and snapshots:
* **Records** are individual data entries that conform to the dataset's schema
* **Snapshots** represent a point-in-time collection of files that were ingested together
When you upload data, the ingestion process validates each record against the schema and adds it to the dataset as part of a new snapshot.
## Adding data to datasets
Datasets support multiple ways to add data:
**Append mode.** New data is added alongside existing data. Use this for event-style data where each upload contains new records.
**Overwrite mode.** New data replaces existing data. Use this when you want to refresh the entire dataset with an updated version.
For procedural details on uploading data, see [Uploading Data](/guides/sdk/uploading-data).
## Retention policies
Datasets can have [retention policies](/concepts/primitives/dataset-retention-policies) that automatically manage data lifecycle. A retention policy defines how long data is kept before automatic deletion, helping you manage storage costs and comply with data governance requirements.
Common retention configurations include:
* **Time-based retention** — Automatically remove data older than a specified period (e.g., 90 days, 1 year)
* **Retain everything** — Keep all data indefinitely until manually deleted
For details on how retention policies work, including differences between data planes and how to configure them, see [Dataset Retention Policies](/concepts/primitives/dataset-retention-policies).
## Ownership and access
### Single-company ownership
Every dataset is owned by exactly one company. The owner has full control over:
* The dataset's schema and configuration
* Who can access the data and under what terms
* Whether to archive or delete the dataset
This ownership model ensures clear accountability and prevents ambiguity about who controls sensitive data.
### Access through access rules
By default, a dataset is private to its owner. To share data with other organizations, you create [access rules](/concepts/primitives/access-rules) that define:
* Which organizations can query the dataset
* Which fields and records they can access
* What pricing applies (if any)
This separation between ownership and access provides flexibility—you retain full control while selectively enabling collaboration.
## Where datasets live
Datasets are scoped to a specific [data plane](/concepts/primitives/data-planes). The data plane determines:
* Where the data physically resides (Narrative-hosted or your own infrastructure)
* Which query engine processes queries against the dataset
* What data residency and compliance requirements are met
When you create a dataset, you specify which data plane it belongs to. The [control plane](/concepts/primitives/control-plane) maintains metadata about the dataset—its schema, access rules, and statistics—while the actual data remains in the data plane.
## View datasets
A view dataset is a dataset backed by an NQL query rather than uploaded data. When you execute a query with the `create_as_view` option, the result is stored as a view dataset. Unlike a [materialized view](/concepts/nql/materialized-views), a view dataset does not refresh on a schedule — the stored NQL is inlined and re-evaluated at query time whenever the view dataset is referenced.
View datasets are useful when you want a reusable, queryable subset of your data without duplicating it into a separate physical dataset. The underlying NQL query can reference other datasets (including other view datasets), and the platform resolves those dependencies automatically.
### Restrictions
View datasets have specific restrictions compared to regular datasets:
* **No access rules.** You cannot create [access rules](/concepts/primitives/access-rules) on a view dataset. If you need to share the data with other organizations, create a regular dataset or materialized view instead.
* **No connections.** You cannot create [connections](/concepts/data-activation/overview) to deliver a view dataset to an external platform.
* **No forecasting.** Query cost forecasting is not available for queries against view datasets.
For the full feature comparison with materialized views — including unsupported NQL features like `MERGE ON`, `PARTITIONED_BY`, and chunking strategies — see [view dataset limitations](/concepts/nql/materialized-views#view-dataset-limitations).
### Creating a view dataset
Use the `create_as_view` option when executing an NQL query through the SDK:
```typescript theme={null}
const result = await api.executeNql({
nql: 'SELECT user_id, email, event_type FROM company_data."my_dataset" WHERE event_type = \'purchase\'',
data_plane_id: null,
create_as_view: true,
});
```
For details on executing queries with this option, see [Executing NQL Queries](/guides/sdk/executing-queries#creating-a-view-dataset).
## Datasets, materialized views, and view datasets
Narrative supports three types of data containers:
| Type | Source | Data storage | Updates |
| --------------------- | ------------------------ | ------------------------------- | ------------------------------------- |
| **Dataset** | External data you upload | Physical table | Manual uploads or automated ingestion |
| **Materialized view** | NQL query results | Physical table (cached results) | Automatic refresh on schedule |
| **View dataset** | NQL query definition | No physical storage | Re-evaluated at query time |
[Materialized views](/concepts/nql/materialized-views) are created from NQL queries and automatically refresh their contents. Regular datasets require you to explicitly add data through uploads or ingestion. View datasets store only the NQL query definition and re-evaluate it at query time — see [View datasets](#view-datasets) above.
## Related content
Configure automatic data lifecycle management
Control who can query your datasets and at what price
Understand where your datasets physically reside
Column-level metrics computed over your dataset contents
Create and manage datasets with the SDK
Resolve HTTP 409 errors when a dataset name is already in use
Resolve HTTP 400 errors when a dataset schema doesn't match a connector