# API Keys
Source: https://docs.narrative.io/account-settings/api-keys

Create, manage, and secure API keys for authenticating requests to the Narrative API

API keys authenticate requests to the Narrative API. You can use them when calling APIs directly or when configuring other systems to make requests on your behalf.

## Viewing your API keys

Navigate to **Settings > API Keys** to view your existing API keys. You'll see a list of your keys along with their expiration dates and assigned permissions.

## API key permissions

Each API key permission is a combination of an **access level** (Read or Write) and a **resource** (a functional area of the platform like Datasets, Connections, or Jobs). A key can hold multiple permissions, and they are additive—each one grants additional capabilities.

<Note>
  For a deeper explanation of how permissions work, see [API Key Permissions](/concepts/security/api-key-permissions). For a complete list of all resources and access levels, see the [Permissions Reference](/reference/security/permissions).
</Note>

<Tip>
  **Use minimum permissions**

  When creating an API key, grant only the permissions necessary for its intended function. This security best practice helps protect your data and account from unauthorized access.
</Tip>

### Choosing permissions for common use cases

Use the table below to determine which permissions to assign based on what your key needs to do.

| Use case                | Recommended permissions                                                                                      |
| ----------------------- | ------------------------------------------------------------------------------------------------------------ |
| **Read-only analytics** | Read on Datasets, Queries, and Subscriptions                                                                 |
| **Data ingestion**      | Write on Datasets and DataStreams                                                                            |
| **Workflow automation** | Read and Write on Workflows and Jobs                                                                         |
| **Connector delivery**  | Read and Write on Connections and Destinations                                                               |
| **SDK development**     | Read and Write on the resources your application uses (see [SDK Authentication](/guides/sdk/authentication)) |

## API key expiration

API keys can have an expiration date, after which they're no longer valid.

<Tip>
  **Set expiration dates and rotate keys regularly**

  We highly recommend setting an expiration date for all API keys and rotating them on a regular schedule.
</Tip>

## Creating a new API key

<Steps>
  <Step title="Open the API Keys page">
    Navigate to **Settings > API Keys**.
  </Step>

  <Step title="Create the key">
    Click **New API Key**.
  </Step>

  <Step title="Configure the key">
    Enter a name for the key, select the permissions you want to grant, and set an expiration date. See [common use cases](#choosing-permissions-for-common-use-cases) for guidance on which permissions to select.
  </Step>

  <Step title="Save your key">
    Copy and securely store the API key. You won't be able to view the full key again after this step.
  </Step>
</Steps>

## Revoking an API key

To revoke an API key:

1. Navigate to **Settings > API Keys**.
2. Click the **...** menu next to the key you want to revoke.
3. Select **Delete**.

The key is immediately revoked and can no longer be used to make requests to the Narrative API.

## Related content

<CardGroup>
  <Card title="API Key Permissions" icon="lock-keyhole" href="/concepts/security/api-key-permissions">
    Understand how scoped permissions protect your resources
  </Card>

  <Card title="Permissions Reference" icon="table" href="/reference/security/permissions">
    All available permission resources and access levels
  </Card>

  <Card title="SDK Authentication" icon="code" href="/guides/sdk/authentication">
    Configure API key authentication for the TypeScript SDK
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    How Narrative protects your data
  </Card>
</CardGroup>

## Responding to a security incident

<Warning>
  If you suspect your API keys have been compromised:

  1. **Revoke all affected API keys immediately** using the steps above.
  2. **Create new API keys** with appropriate permissions.
  3. **Report the incident** to Narrative security so we can help you investigate and take appropriate action.
</Warning>


# SSO Configuration
Source: https://docs.narrative.io/account-settings/sso

Configure SAML 2.0 single sign-on for your Narrative I/O organization

This guide walks enterprise administrators through configuring SSO for their organization. Once configured, your team members can authenticate to Narrative using your corporate identity provider.

<Note>
  To understand how SSO works and why organizations use it, see [Single Sign-On (SSO)](/concepts/security/sso).
</Note>

## Prerequisites

Before you begin, ensure:

* Your organization has an enterprise contract that includes SSO
* Narrative has provisioned your organization and provided your **Organization Slug** (e.g., `yourcompany`)
* You have received an email invitation from Narrative and completed initial registration
* Your Narrative account has the **Stytch Admin** role (contact Narrative support if unsure)
* You have administrative access to your organization's identity provider

## First-time login

Before SSO is configured, you'll use a magic link to access the admin portal.

<Steps>
  <Step title="Access Enterprise Login">
    Navigate to [app.narrative.io/platform/login](https://app.narrative.io/platform/login) and click **Enterprise Login** at the bottom of the login page.
  </Step>

  <Step title="Enter your organization slug">
    Enter your organization's slug (provided by Narrative) to identify which organization you're authenticating against.
  </Step>

  <Step title="Authenticate via magic link">
    Enter your email address and click **Send Magic Link**. Check your email for the login link and click it to complete authentication.
  </Step>
</Steps>

Once SSO is configured, you'll authenticate directly through your organization's identity provider instead.

## Configure your SSO connection

### Gather IdP information

From your identity provider (Okta, Azure AD, etc.), collect:

| Information           | Description                                   |
| --------------------- | --------------------------------------------- |
| **IdP SSO URL**       | The endpoint where SAML requests are sent     |
| **IdP Entity ID**     | Unique identifier for your IdP                |
| **X.509 Certificate** | Public certificate for signature verification |

### Create the connection in Narrative

<Steps>
  <Step title="Open SSO settings">
    In Narrative, go to **Settings > Enterprise Login Settings**.
  </Step>

  <Step title="Create new connection">
    Click **New SSO Connection**.
  </Step>

  <Step title="Enter IdP information">
    Enter the IdP SSO URL, Entity ID, and X.509 certificate you collected.
  </Step>
</Steps>

## Configure your identity provider

In your identity provider, create a new SAML application with the following settings from Narrative:

| Field                            | Description                                             | Where to find                                |
| -------------------------------- | ------------------------------------------------------- | -------------------------------------------- |
| **ACS URL / Single Sign-On URL** | The endpoint where your IdP sends SAML assertions       | Narrative UI (SSO Settings > New Connection) |
| **Entity ID / Audience URI**     | Unique identifier for Narrative as the service provider | Narrative UI (SSO Settings > New Connection) |
| **Name ID Format**               | Must be set to `EmailAddress`                           | —                                            |

### Okta configuration

In Okta, when creating the SAML application:

1. Check **Use this for Recipient URL and Destination URL** when entering the Single sign-on URL
2. Set **Application username** to `Email`
3. Set **Update application username on** to `Create and update`

Configure these attribute statements with **Name format** set to `Basic`:

| Attribute name | Value            |
| -------------- | ---------------- |
| `first_name`   | `user.firstName` |
| `last_name`    | `user.lastName`  |
| `id`           | `user.id`        |

### Azure AD (Microsoft Entra ID) configuration

Configure these attribute statements:

| Attribute name | Value            |
| -------------- | ---------------- |
| `first_name`   | `user.givenname` |
| `last_name`    | `user.surname`   |
| `id`           | `user.objectid`  |

### OneLogin configuration

Configure these attribute statements:

| Attribute name | Value         |
| -------------- | ------------- |
| `first_name`   | `{firstname}` |
| `last_name`    | `{lastname}`  |
| `id`           | `{id}`        |

<Warning>
  **Attribute names must match exactly**

  The attribute names sent to Narrative (`first_name`, `last_name`, `id`) must match exactly as shown. Attribute value syntax varies by identity provider—consult your IdP's documentation if you encounter issues.
</Warning>

## Complete the SSO connection

After creating the SAML application in your IdP:

<Steps>
  <Step title="Locate metadata">
    Find the **Metadata URL** (or download the metadata XML) from your identity provider.

    * **Okta**: Find the Metadata URL in the **Sign On** tab. Click **Actions** under SAML Signing Certificates and select **View IdP metadata**.
    * **Azure AD**: Download the **Federation Metadata XML** from the SAML configuration page, or use the **App Federation Metadata Url**.
  </Step>

  <Step title="Add metadata to Narrative">
    Return to Narrative's SSO Settings tab, paste the Metadata URL, and click **Create**.
  </Step>

  <Step title="Verify connection">
    Your SSO connection should appear in the list with status **Active**.
  </Step>
</Steps>

## Assign users in your IdP

Users must be explicitly assigned to the Narrative SAML application in your IdP before they can authenticate.

**Okta**: Go to **Applications > Your Narrative App > Assignments**. Click **Assign > Assign to People** (or **Assign to Groups** for bulk assignment).

**Azure AD**: Go to **Enterprise Applications > Your Narrative App > Users and groups**. Click **Add user/group** and select users or groups.

<Tip>
  Users not assigned to the application will see an error: "User is not assigned to this application."
</Tip>

## Enforce SSO-only authentication

By default, Narrative allows multiple authentication methods (email magic links and SSO). To require SSO for all users:

<Steps>
  <Step title="Open organization settings">
    Go to **Settings > Enterprise Login Settings > Organization Settings**.
  </Step>

  <Step title="Edit authentication settings">
    Under **Authentication settings**, click **Edit**.
  </Step>

  <Step title="Disable other methods">
    Uncheck **Allow all primary auth methods** to reveal individual method checkboxes. Check only **Single Sign-On** and leave **Email Magic Links** unchecked.
  </Step>

  <Step title="Save changes">
    Click **Save**.
  </Step>
</Steps>

<Warning>
  **SSO-only mode**

  All users in your organization will be required to authenticate via SSO. Email magic links will no longer work for standard users.
</Warning>

## Just-in-time (JIT) provisioning

JIT provisioning is enabled by default. When enabled, any user assigned to your IdP's SAML application can automatically create a Narrative account on their first login—no invitation required.

To disable JIT provisioning:

1. Go to **Organization Settings > User onboarding**
2. Click **Edit**
3. Under **JIT Provisioning**, uncheck SSO connections
4. Click **Save**

When JIT is disabled, users must be explicitly invited via the Member Management tab before they can access Narrative.

## Managing members

### Inviting members

To invite a new user (when JIT is disabled, or to pre-provision users):

1. Go to **Member Management**
2. Click **Invite**
3. Enter the user's name and email address
4. Optionally assign a role (leave blank for standard user access)
5. Click **Invite**

The user will receive an email invitation. When they click the link and authenticate via SSO, their account will be activated.

<Tip>
  The invited email address must match the email configured in your identity provider.
</Tip>

### Revoking access

To remove a user's access:

1. In your identity provider, remove the user from the Narrative SAML application (or remove them from an assigned group)
2. The user will immediately lose access and see an error on their next login attempt

You can also archive the member in Narrative's **Member Management** tab to fully remove their account from the platform.

## API token security

<Warning>
  **API tokens are not affected by SSO**

  Long-lived API tokens (provisioned under **Settings > API Keys**) are not tied to an individual user and are not automatically revoked when a user's SSO access is revoked.

  If a user leaves your company or should no longer have access to a long-lived API token, you must manually revoke that API token. See [API Keys](/account-settings/api-keys) for instructions.
</Warning>

## Troubleshooting

| Issue                                                  | Solution                                                                                                        |
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
| "User is not assigned to this application"             | Add the user to the Narrative SAML application in your identity provider                                        |
| "This project is not authorized to call this endpoint" | Contact Narrative support—SSO products may not be enabled for your organization                                 |
| User prompted to register after clicking invite link   | This is expected for first-time users. They'll complete a brief registration flow before accessing the platform |

## Support

For assistance with SSO configuration, contact Narrative support at [support@narrative.io](mailto:support@narrative.io) or reach out to your account representative.

If you use an identity provider not covered in this guide, our team can provide configuration assistance specific to your IdP.

## Related content

<CardGroup>
  <Card title="Single Sign-On (SSO)" icon="shield-keyhole" href="/concepts/security/sso">
    Understand how SSO works and why organizations use it
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys for programmatic access
  </Card>
</CardGroup>


# Create a new Access Rule
Source: https://docs.narrative.io/api-reference/access-rules/create-a-new-access-rule

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /v2/access-rules
Creates a new access rule based on the provided NQL query.



# Delete an access rule
Source: https://docs.narrative.io/api-reference/access-rules/delete-an-access-rule

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /v2/access-rules/{access_rule_id}
Access rule deletion cannot be undone.



# Get all accessible Access Rules (V2)
Source: https://docs.narrative.io/api-reference/access-rules/get-all-accessible-access-rules-v2

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/access-rules
Get all accessible Access Rules.

'owned_only' and 'owned_only' are flags, they don't require values.

'tag', 'dataset_id', and 'company_id' are multi-params.

Query like 'tag=tag1&tag=tag2&company_id=1&company_id=2' is interpreted as "access rules with tags that comprise 'tag1' or 'tag2' AND belonging to companies with id '1' or '2'.



# Get an Access Rule by id (V2)
Source: https://docs.narrative.io/api-reference/access-rules/get-an-access-rule-by-id-v2

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/access-rules/{access_rule_id}
Get an access rule.



# Get an Access Rule by name (V2)
Source: https://docs.narrative.io/api-reference/access-rules/get-an-access-rule-by-name-v2

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/access-rules/{company_slug}/{access_rule_name}
Get an access rule by name and company slug.



# Update an Access Rule by id (V2)
Source: https://docs.narrative.io/api-reference/access-rules/update-an-access-rule-by-id-v2

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /v2/access-rules/{access_rule_id}
Update an access rule.



# Create a System Access Token
Source: https://docs.narrative.io/api-reference/access-tokens/create-a-system-access-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /access-tokens/tokens
Create a System Access Token



# Delete a System Access Token
Source: https://docs.narrative.io/api-reference/access-tokens/delete-a-system-access-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /access-tokens/tokens/{access_token_id}
Delete a System Access Token



# Get a System Access Token by id
Source: https://docs.narrative.io/api-reference/access-tokens/get-a-system-access-token-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /access-tokens/tokens/{access_token_id}
Get a System Access Token by id



# List System Access Tokens
Source: https://docs.narrative.io/api-reference/access-tokens/list-system-access-tokens

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /access-tokens/tokens
List System Access Tokens



# Update a System Access Token
Source: https://docs.narrative.io/api-reference/access-tokens/update-a-system-access-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /access-tokens/tokens/{access_token_id}
Update a System Access Token



# Create an agent conversation
Source: https://docs.narrative.io/api-reference/agent-conversations/create-an-agent-conversation

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /agents/conversations
Creates a new agent conversation with a pinned `system_prompt` and a `defaults` block
that every subsequent run inherits. The conversation starts at `version: 0` — your
first run's `expected_version` will be `0`.

## What this endpoint actually does

1. Validates the `defaults` block. Failures here (alias regex, duplicate aliases,
   wire-name length, missing required fields) return 400 immediately — the
   conversation row is **not** created on validation failure.
2. Inserts a row in `agent_conversations` with `version = 0`, `company_id` from your
   bearer token, and the validated `defaults` serialized as JSONB.
3. Returns the persisted conversation echo.

No workflows are started here. The conversation is a long-lived container; the
actual model execution happens later, in `POST /agents/conversations/{id}/runs`.

## What you cannot change later

- **`system_prompt`** — pinned for the conversation's lifetime. If you need a
  different system prompt, create a new conversation.
- **`company_id`** / **`user_id`** — derived from the bearer token at creation time;
  not exposed in the request body. Both gate every subsequent read endpoint on this
  conversation: peers in the same company **cannot** see each other's conversations,
  runs, or messages. Cross-user access surfaces as 404 — the API does not
  distinguish "doesn't exist" from "owned by another user".

Everything else in `defaults` (model, max_iterations, tools, etc.) is just a default
— any individual run can override any of those fields via `config_override` on its
`POST .../runs` body. See [Agent Conversations
Reference](https://docs.narrative.io/reference/architecture/agent-conversations#run)
for the override semantics.

## Permission

`agent_conversations` resource with `write` verb. The same permission gates
`POST .../runs`; `read` is required for the GET endpoints.

## Example: minimal "hello world" conversation (no tools)

```json
{
  "name": "hello world",
  "system_prompt": "You are a helpful assistant. Answer concisely.",
  "defaults": {
    "model": "anthropic.claude-opus-4.6",
    "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
    "execution_cluster": "shared",
    "max_iterations": 3,
    "max_tokens": 1024,
    "temperature": 0.0,
    "output_format_schema": {
      "type": "object",
      "additionalProperties": false,
      "required": ["text"],
      "properties": { "text": { "type": "string" } }
    }
  }
}
```

## Example: conversation that can search the Narrative docs

The `mcp_servers` entry registers two tools the model can call server-side. The
platform handles the JSON-RPC calls to `https://docs.narrative.io/mcp` transparently.

```json
{
  "name": "docs search",
  "system_prompt": "Use the available tools to look things up before answering.",
  "defaults": {
    "model": "anthropic.claude-opus-4.6",
    "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
    "execution_cluster": "shared",
    "max_iterations": 8,
    "max_tokens": 2048,
    "temperature": 0.0,
    "output_format_schema": {
      "type": "object",
      "additionalProperties": false,
      "required": ["text"],
      "properties": { "text": { "type": "string" } }
    },
    "mcp_servers": [{
      "alias": "docs",
      "url": "https://docs.narrative.io/mcp",
      "description": "Narrative docs MCP server"
    }]
  }
}
```

## Example: conversation with a caller-declared tool (human in the loop)

Caller-declared `tools[]` entries cause the model's call to **pause** the run with
`status: requires_action`. The caller answers by posting a follow-up run with
`payload.kind: tool_outputs`. These tools have no alias — the model sees the bare
`name`, and the platform routes any dash-free tool call to caller-declared resolution.

```json
{
  "name": "booking flow",
  "system_prompt": "You are a booking assistant.",
  "defaults": {
    "model": "anthropic.claude-opus-4.6",
    "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
    "execution_cluster": "shared",
    "max_iterations": 5,
    "output_format_schema": {
      "type": "object",
      "additionalProperties": false,
      "required": ["text"],
      "properties": { "text": { "type": "string" } }
    },
    "tools": [{
      "name": "confirm_booking",
      "description": "Ask the user to confirm a proposed booking slot.",
      "input_schema": {
        "type": "object",
        "additionalProperties": false,
        "required": ["proposed_slot"],
        "properties": { "proposed_slot": { "type": "string" } }
      }
    }]
  }
}
```

## Common 400 errors

| `type` URL slug | Cause |
|---|---|
| `/errors/invalid-tool-alias` | MCP alias doesn't match `^[a-zA-Z][a-zA-Z0-9]{0,7}$` |
| `/errors/duplicate-tool-alias` | Same alias appears twice across `mcp_servers` |
| `/errors/invalid-caller-tool-name` | Caller-declared `tools[].name` is empty, contains a dash, or exceeds 64 chars |
| `/errors/duplicate-caller-tool-name` | Same `tools[].name` appears twice |
| `/errors/tool-name-too-long` | Some MCP wire name `{alias}-{tool_name}` > 64 chars |

See [Agent Conversations error
catalog](https://docs.narrative.io/reference/architecture/agent-conversations/errors/conversation-not-found)
for the full list.



# Get a run's current state
Source: https://docs.narrative.io/api-reference/agent-conversations/get-a-runs-current-state

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /agents/runs/{id}
Returns the run's current state — status, iterations used so far, token usage,
submitted inference job IDs, the model's final answer (if completed), pending tool
calls (if requires_action), or the error (if failed).

## Polling

Runs are asynchronous. After `POST /agents/conversations/{id}/runs` you poll this
endpoint until you see a terminal status (`completed`, `requires_action`, or
`failed`). A reasonable polling cadence:

- Start with 2-second polls — most simple runs complete in a single iteration
  (5–15 seconds wall-clock).
- Back off to 4–8 seconds for longer-running runs (multi-iteration tool loops).
- Hard-cap at maybe 5 minutes — if a run hasn't terminated by then either something
  is wrong (check `submitted_inference_job_ids` in the data plane's `jobs` table) or
  the workflow timed out and will land `failed` shortly.

## Which fields are populated when

| `status` | Populated fields |
|---|---|
| `pending` | `id`, `conversation_id`, `client_op_id`, `effective_config`, `started_at` |
| `running` | above + `submitted_inference_job_ids` (one per iteration completed) |
| `completed` | above + `iterations_used`, `usage`, exactly one of `final_text` / `final_structured_output`, `completed_at` |
| `requires_action` | above + `iterations_used`, `usage`, `pending_tool_calls`, `completed_at` |
| `failed` | above + `error`, possibly partial `iterations_used`/`usage`, `completed_at` |

`effective_config` is **always** populated — it's the fully-merged config used for
this run (conversation defaults + this run's `config_override`). Helpful for
confirming the model saw what you expected.

## Run vs. inference iteration

Each run runs N iterations, where N ≤ `effective_config.max_iterations`. Each
iteration is one inference job submitted to the data plane (one Bedrock or Cortex
API call). `submitted_inference_job_ids` carries the platform's `jobs.id` for each
iteration — use those to drill into the raw model request/response in the data
plane's logs / job table.

`iterations_used` and `submitted_inference_job_ids.length` match for any terminal
state.

## Error structure

On `status: "failed"` the `error` field carries:

- `type` — opaque incident code (e.g. `AgentLoopMaxIterationsExceeded`,
  `AgentLoopInvalidEffectiveConfig`, `UnknownTool`). Stable across versions, used
  for log correlation.
- `message` — human-readable detail.
- `title` and `docs_url` — caller-facing presentation looked up from the platform's
  [error catalog](https://docs.narrative.io/reference/architecture/agent-conversations/errors/unexpected).
  Absent if the API doesn't recognize the `type` (usually because the workflow
  workers have been upgraded ahead of the API); in that case rely on `type` and
  `message`.

Note: this endpoint always returns 200 for a known run — even one with `status:
"failed"`. The HTTP-level error path (RFC 7807) only applies when the call itself
fails (token invalid, run not found, etc.).

## Scope: per-user, not per-company

Runs are scoped to the **`(company_id, user_id)`** pair that created them. Returns
404 for "run does not exist", "run belongs to a different company", and "run was
created by a different user in the same company" alike — the API does not
distinguish them to avoid leaking existence.

## Permission

`agent_conversations` resource with `read` verb.

## Example: completed run

```json
{
  "id": "82eb9cb7-619e-46bb-ac1c-8a32b0112c1c",
  "conversation_id": "bc2505b7-068d-44ed-8055-a6f6ffe54ab1",
  "company_id": 1,
  "user_id": 407,
  "client_op_id": "696b022c-e86d-4b3e-b6d7-ceb7eb1a495e",
  "status": "completed",
  "tool_choice": null,
  "effective_config": { "...": "..." },
  "iterations_used": 1,
  "usage": { "prompt_tokens": 169, "completion_tokens": 8, "total_tokens": 177 },
  "submitted_inference_job_ids": ["e94c6eab-019f-4e96-bb47-049fa60e7988"],
  "pending_tool_calls": [],
  "final_text": "4",
  "final_structured_output": null,
  "error": null,
  "started_at": "2026-05-18T13:21:30.355180Z",
  "completed_at": "2026-05-18T13:21:51.983952Z"
}
```

## Example: completed structured-output run

```json
{
  "id": "8dda70ab-9f81-4613-a3c9-68b3b584e82d",
  "status": "completed",
  "iterations_used": 1,
  "final_text": null,
  "final_structured_output": {
    "event": {
      "type": "login",
      "user_id": "u-123",
      "session_duration_seconds": 1800
    }
  },
  ...
}
```

## Example: requires_action run

```json
{
  "id": "19082f49-0280-4e32-a8b4-7df8dea9dbed",
  "status": "requires_action",
  "iterations_used": 1,
  "pending_tool_calls": [{
    "tool_use_id": "tooluse_DWXPKZ50JDGib5GmShyUgJ",
    "name": "confirm_booking",
    "arguments": { "proposed_slot": "tomorrow afternoon" }
  }],
  "final_text": null,
  "final_structured_output": null,
  ...
}
```

Resume by posting a new run with `payload.kind: "tool_outputs"` carrying the same
`tool_use_id`.

## Example: failed run

```json
{
  "id": "...",
  "status": "failed",
  "iterations_used": null,
  "usage": null,
  "final_text": null,
  "final_structured_output": null,
  "error": {
    "type": "AgentLoopMaxIterationsExceeded",
    "message": "max iterations exceeded after 5 iterations",
    "title": "Max Iterations Exceeded",
    "docs_url": "https://docs.narrative.io/reference/architecture/agent-conversations/errors/max-iterations-exceeded"
  },
  ...
}
```



# Get an agent conversation
Source: https://docs.narrative.io/api-reference/agent-conversations/get-an-agent-conversation

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /agents/conversations/{id}
Fetches the conversation's metadata plus its current `version`. Use the `version`
immediately before posting a run as `expected_version` — that's how the platform's
compare-and-swap guard knows whether the conversation has advanced since you last
looked.

## When to call this

- Before starting a new run: read `version`, post it as `expected_version`. If you
  cache the value across long delays you'll hit
  [Version Conflict](https://docs.narrative.io/reference/architecture/agent-conversations/errors/version-conflict)
  more often.
- When resuming a `requires_action` run: the conversation's version has advanced
  because the assistant turn was persisted on the run row's finalize. Re-read it
  before posting `tool_outputs`.
- To inspect a conversation's `defaults` if you've forgotten what was pinned.

## What you cannot do here

There is no `PATCH` / `PUT` for conversations. The `system_prompt` and `defaults` are
immutable at the conversation level. To change behavior, override per-run via
`config_override`, or create a new conversation.

## Scope: per-user, not per-company

Conversations are scoped to the **`(company_id, user_id)`** pair that created them.
Peers in the same company cannot see each other's conversations — the endpoint
returns 404 for "does not exist", "belongs to a different company", and "belongs to
a different user in the same company" alike. This is deliberate: distinguishing them
would leak existence.

## Permission

`agent_conversations` resource with `read` verb.



# List conversation messages (delta read by version cursor)
Source: https://docs.narrative.io/api-reference/agent-conversations/list-conversation-messages-delta-read-by-version-cursor

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /agents/conversations/{id}/messages
Returns the messages in this conversation with `sequence_no > since`, plus the
conversation's `current_version` at the moment the response was generated. The
`messages` list is ordered by `sequence_no` ascending and is gap-free — if the
response contains messages with `sequence_no` 3, 4, 5, you have all three.

## Polling pattern

```text
since = 0
loop forever:
  r = GET /agents/conversations/{id}/messages?since={since}
  process r.messages       # may be empty
  since = r.current_version # use the head, not r.messages.last.sequence_no
  wait briefly
```

`current_version` is the conversation's head at the moment the response was generated
— using it as the next request's `since` is the only way to ensure gap-free reads
even when new messages land between the time the response was generated and the time
your client processed it.

## When messages get added

Messages are persisted in a single transaction by the workflow's finalize step (when
a run reaches a terminal state). You won't see partial reads — a run that produces 4
messages either lands all 4 or none.

A run's `id` appears in `messages[].run_id`, so you can correlate messages to the
runs that produced them.

## What's in `content_blocks`

Three discriminated variants:

- `text` — plain text. The dominant variant.
- `tool_use` — the model requesting a tool call. Carries `tool_use_id`, the aliased
  `name`, and the model's `arguments`.
- `tool_result` — the result of a tool call, nested as `content_blocks` inside the
  `tool_result` block. Carries the original `tool_use_id` plus an `is_error` flag.

The same `tool_use_id` appears on the assistant's `tool_use` block and on the
subsequent `tool` (or `user`, for resume payloads) turn's `tool_result` block —
that's how a UI matches them up.

## Permission

`agent_conversations` resource with `read` verb.

## Example: paging from scratch

```bash
curl "$API/agents/conversations/{id}/messages?since=0" \
  -H "Authorization: Bearer $TOKEN"
# → { "current_version": 4, "messages": [ <4 entries> ] }
```

Then later:

```bash
curl "$API/agents/conversations/{id}/messages?since=4" \
  -H "Authorization: Bearer $TOKEN"
# → { "current_version": 4, "messages": [] }   # nothing new
```



# List the calling user's agent conversations
Source: https://docs.narrative.io/api-reference/agent-conversations/list-the-calling-users-agent-conversations

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /agents/conversations
Returns a paginated list of conversations created by the bearer-token's user, newest
first (by `created_at`).

## Scope: per-user, not per-company

Results are filtered by **both** `company_id` and `user_id` from the bearer token —
peers in the same company are **not** visible. The endpoint never returns 404; the
natural empty state is `records: []` with `total_records: 0`.

## Pagination

Uses the standard `page` / `per_page` query parameters. Default `per_page` is 50.
The response envelope carries `prev_page`, `current_page`, `next_page`,
`total_records`, `total_pages`, and the `records` array — same shape every other
paginated Narrative endpoint uses.

## Permission

`agent_conversations` resource with `read` verb.

## Example

```bash
curl "$API/agents/conversations?page=1&per_page=20" \
  -H "Authorization: Bearer $TOKEN"
```



# Start a new run on the conversation
Source: https://docs.narrative.io/api-reference/agent-conversations/start-a-new-run-on-the-conversation

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /agents/conversations/{id}/runs
Inserts a row in `agent_runs`, kicks off the workflow that drives the agent loop, and
returns immediately. The response has `status: "pending"` — poll
`GET /agents/runs/{id}` to track progress.

Runs are **asynchronous**. This endpoint does not wait for the model to reply.
Typical end-to-end latency is 3–10 seconds for a single-iteration run, longer for
multi-iteration tool-use loops (60–120 seconds is common for 6–8 iterations against
an MCP server).

## Required fields

- **`client_op_id`** — UUID you generate. Used as the idempotency key, scoped to
  `(conversation_id, client_op_id)`. Re-sending the same value returns the original
  run row unchanged. Always generate a fresh UUID for a fresh logical request;
  reusing one is only safe if the second call is a retry of the first.
- **`expected_version`** — the conversation's current version, read from
  `GET /agents/conversations/{id}` just before this call. If it doesn't match at
  run-creation time, you get a 409
  [Version Conflict](https://docs.narrative.io/reference/architecture/agent-conversations/errors/version-conflict).
  Recovery: refetch the conversation, re-read `version`, retry the run with the
  fresh value.
- **`payload`** — `kind: "user_message"` to start a fresh user turn, or
  `kind: "tool_outputs"` to resume a previously-paused (`requires_action`) run.

## Optional fields

- **`tool_choice`** — bias the model toward (or away from) using tools on iteration 1
  **of this run**. Three values: `auto`, `any`, `specific_tool` (see schema). Applies
  only to iteration 1 of this run; iteration 2+ are back on `auto`. To re-force a
  specific tool on the next run, set `tool_choice` again on its request body.
- **`config_override`** — sparse override of the conversation's `defaults`. Any field
  you set replaces the conversation-level value for this single run. The two list
  fields (`mcp_servers`, `tools`) are **wholesale-replaced**, not merged. The
  `system_prompt` cannot be overridden.

## Run lifecycle (state diagram)

```text
pending  →  running  →  completed         ← model produced a final answer
                    ↘   requires_action   ← model called a caller-declared tool
                    ↘   failed            ← non-recoverable error
```

You stop polling on any of the three terminal states. For `requires_action`, the run
can be **resumed** by posting a new run on the same conversation with
`payload.kind: "tool_outputs"`. For `completed` and `failed`, the run is done.

## Examples

### Example 1 — fresh user message, no tools

```json
{
  "client_op_id": "696b022c-e86d-4b3e-b6d7-ceb7eb1a495e",
  "expected_version": 0,
  "payload": { "kind": "user_message", "text": "What is 2 + 2?" }
}
```

Returns:

```json
{
  "id": "82eb9cb7-619e-46bb-ac1c-8a32b0112c1c",
  "status": "pending",
  "started_at": "2026-05-18T13:21:30.355180Z",
  "submitted_inference_job_ids": [],
  "pending_tool_calls": [],
  "effective_config": { ... },
  ...
}
```

Then poll `GET /agents/runs/{id}` until `status: completed`. Final
`final_text: "4"`.

### Example 2 — force a specific tool, override max_iterations

```json
{
  "client_op_id": "c8aef867-eca6-46f7-8a99-e22dd0ee6062",
  "expected_version": 0,
  "payload": { "kind": "user_message", "text": "Find me a meeting time for tomorrow afternoon." },
  "tool_choice": { "kind": "specific_tool", "name": "confirm_booking" },
  "config_override": { "max_iterations": 10 }
}
```

The model is forced to call `confirm_booking` on iteration 1. Since `confirm_booking`
is a caller-declared tool (dash-free name, listed in `tools[]`), the run terminates
with `status: requires_action` after one iteration; `pending_tool_calls` contains the
model's proposed arguments.

To pin an MCP-resolved tool instead, include the `mcp_alias` field:

```json
"tool_choice": { "kind": "specific_tool", "mcp_alias": "docs", "name": "search_kb" }
```

### Example 3 — resume a paused run

First, refetch the conversation to get the new version (it has advanced because the
paused run's assistant turn was persisted):

```bash
curl "$API/agents/conversations/{id}" -H "Authorization: Bearer $TOKEN"
# → { "version": 2, ... }
```

Then post the resume:

```json
{
  "client_op_id": "7fb6795c-60a2-4cf9-a0db-0cc645279704",
  "expected_version": 2,
  "payload": {
    "kind": "tool_outputs",
    "outputs": [{
      "tool_use_id": "tooluse_DWXPKZ50JDGib5GmShyUgJ",
      "content": "User confirmed the proposed slot.",
      "is_error": false
    }]
  }
}
```

The model sees the tool result and continues; this run usually terminates
`completed` after one iteration with a contextual final answer.

## Idempotency

Re-sending the same body with the same `client_op_id` returns the original
`agent_runs` row verbatim. Idempotency is on `(conversation_id, client_op_id)`:

- **Same `client_op_id`, same conversation** → same run row returned, no new
  workflow started.
- **Same `client_op_id`, different conversation** → new run, new workflow.

Use a fresh UUID per logical request, never per retry of an in-flight request.

## Common error responses

| Status | `type` slug | Cause | Recovery |
|---|---|---|---|
| 400 | `/invalid-tool-alias` | `config_override.mcp_servers[]` has a bad alias | Fix the alias |
| 400 | `/tool-name-too-long` | `config_override` introduced a wire name > 64 chars | Shorten alias or tool name |
| 400 | `/unknown-tool-use-id` | `tool_outputs` references an id not on the latest assistant turn | Refetch the prior run's `pending_tool_calls` |
| 400 | `/not-a-client-tool-call` | `tool_outputs` references a server-resolved call | Filter to client-side ids only |
| 400 | `/incomplete-tool-outputs` | Missing or extra `outputs[]` entries | Match the pending set exactly |
| 400 | `/no-assistant-turn` | `tool_outputs` payload on a conversation with no assistant turn yet | Use `user_message` payload instead |
| 404 | `/conversation-not-found` | Conversation doesn't exist, is in another company, or was created by a different user | Verify the id; conversations are scoped to the bearer-token's `(company_id, user_id)` |
| 409 | `/version-conflict` | `expected_version` mismatch | Refetch the conversation, retry |
| 500 | `/workflow-start-failed` | Platform couldn't reach the workflow engine | Retry with the same `client_op_id` |

## Permission

`agent_conversations` resource with `write` verb.



# Delivery Commands
Source: https://docs.narrative.io/api-reference/app-api/delivery-commands

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v1/app/commands/delivery
Lists delivery commands this app has access to.



# Delivery Files
Source: https://docs.narrative.io/api-reference/app-api/delivery-files

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v1/app/commands/delivery/{delivery_id}/files
Lists delivery files for a particular delivery command.



# Download file
Source: https://docs.narrative.io/api-reference/app-api/download-file

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /v1/app/commands/delivery/{delivery_id}/download/{filename}
Generates a download url for a particular delivery file



# Activate an app invite
Source: https://docs.narrative.io/api-reference/app-invites/activate-an-app-invite

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /app-invites/{app_invite_id}/activate
Activates an app invite, changing its status from `pending` to `active`.

Requires an app client credentials token.



# Archive an app invite
Source: https://docs.narrative.io/api-reference/app-invites/archive-an-app-invite

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /app-invites/{app_invite_id}
Archives an app invite by its ID.

This is a soft delete that sets the invite status to `archived`.



# Create an app invite
Source: https://docs.narrative.io/api-reference/app-invites/create-an-app-invite

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /app-invites
Creates a new app invite for a company.

Requires an app client credentials token. The invite is created in `pending` status
and includes a generated invite code.



# Get an app invite
Source: https://docs.narrative.io/api-reference/app-invites/get-an-app-invite

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /app-invites/{app_invite_id}
Returns a specific app invite by its ID.



# Get app invite by code
Source: https://docs.narrative.io/api-reference/app-invites/get-app-invite-by-code

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /app-invites/codes/{code}
Returns a public view of an app invite by its invite code.

This endpoint does not require authentication and returns a stripped-down response
that omits fields like timestamps and tags from the response.



# List app invites
Source: https://docs.narrative.io/api-reference/app-invites/list-app-invites

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /app-invites
Returns all app invites for the authenticated company and app.

Results are paginated using `page` (1-based, default `1`) and `per_page`
(default `100`) query parameters. Optionally filter results to a specific
installation using `installation_id`.



# Get Data instance
Source: https://docs.narrative.io/api-reference/apps/get-data-instance

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /apps/{app_slug}/profile-store/${profile_id}



# Get Schema
Source: https://docs.narrative.io/api-reference/apps/get-schema

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /apps/{app_slug}/manifest/profile-store-schema



# Get Schema
Source: https://docs.narrative.io/api-reference/apps/get-schema-1

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /apps/{app_slug}/manifest/profile-store-schema/{schema_version_id}



# Lists apps
Source: https://docs.narrative.io/api-reference/apps/lists-apps

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /apps



# Lists profiles
Source: https://docs.narrative.io/api-reference/apps/lists-profiles

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /apps/{app_slug}/profiles



# Post Data instance
Source: https://docs.narrative.io/api-reference/apps/post-data-instance

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /apps/{app_slug}/profile-store/${profile_id}
Submitting (create or update) data for a profile.



# Publish app event
Source: https://docs.narrative.io/api-reference/apps/publish-app-event

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /apps/events
Create a new webhook subscription for the authenticated company.



# Put Schema
Source: https://docs.narrative.io/api-reference/apps/put-schema

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /apps/{app_slug}/manifest/profile-store-schema
Submitting (create or update) JSON Schema (2019-09 version) for an app.



# Update a slug
Source: https://docs.narrative.io/api-reference/apps/update-a-slug

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /admin/apps/{app_id}/slug
Slug can be changed (from NULL) just once.



# Create an attribute
Source: https://docs.narrative.io/api-reference/attributes/create-an-attribute

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /attributes
Create a new attribute.



# Delete an attribute
Source: https://docs.narrative.io/api-reference/attributes/delete-an-attribute

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /attributes/{attribute_name_or_id}
Permanently removes an attribute from the platform.

Attribute deletion will fail if the given attribute is referenced by another attribute.

Attribute deletions cannot be undone.



# Get an attribute
Source: https://docs.narrative.io/api-reference/attributes/get-an-attribute

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /attributes/{attribute_name_or_id}



# Get attribute references.
Source: https://docs.narrative.io/api-reference/attributes/get-attribute-references

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /attributes/{attribute_name_or_id}/references



# Get attributes
Source: https://docs.narrative.io/api-reference/attributes/get-attributes

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /attributes
Return all attributes.



# Request or refresh an OAuth token
Source: https://docs.narrative.io/api-reference/auth/request-or-refresh-an-oauth-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /oauth/token
# /oauth/token
#
Requests or refreshes the OAuth Token.



# Authenticate a user based on a Stytch session token
Source: https://docs.narrative.io/api-reference/authentication/authenticate-a-user-based-on-a-stytch-session-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/login
Authenticate the user based on the Stytch session token, and return the user information, and API Access Token.
The Stych session token can be obtained from a Magic Links or OAuth token thanks to the registration status API.



# Authenticate a user based on a Stytch session token (B2B version)
Source: https://docs.narrative.io/api-reference/authentication/authenticate-a-user-based-on-a-stytch-session-token-b2b-version

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/b2b/login
Authenticate the user based on the Stytch session token, and return the user information, and API Access Token.



# Exchange a user token for a token scoped to a different company
Source: https://docs.narrative.io/api-reference/authentication/exchange-a-user-token-for-a-token-scoped-to-a-different-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/exchange-token
Exchange the current user access token for a new token scoped to the specified company.
The user can only exchange their token to be scoped to another company to which they belong.



# Register a user based on a Stytch session token
Source: https://docs.narrative.io/api-reference/authentication/register-a-user-based-on-a-stytch-session-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/register
Register the user based on the Stytch session token and name/company name, and return the the created user id.
The Stych session token can be obtained from a Magic Links or OAuth token thanks to the registration status API.



# Register a user based on a Stytch session token (B2B)
Source: https://docs.narrative.io/api-reference/authentication/register-a-user-based-on-a-stytch-session-token-b2b

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/b2b/register
Register the user based on the Stytch session token and name/company name, and return the the created user id.
The Stych session token can be obtained from a Magic Links or OAuth token thanks to the registration status API.



# Report whether the user associatd to a Stytch OAuth or Magic Links token has been fully registered on Narrative side
Source: https://docs.narrative.io/api-reference/authentication/report-whether-the-user-associatd-to-a-stytch-oauth-or-magic-links-token-has-been-fully-registered-on-narrative-side

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/registration-status
Report whether the user associatd to a Stytch OAuth or Magic Links token has been fully registered on Narrative side.
Provides a Stytch session token that can be used to access the login and register APIs.



# Report whether the user associatd to a Stytch OAuth or Magic Links token has been fully registered on Narrative side (B2B)
Source: https://docs.narrative.io/api-reference/authentication/report-whether-the-user-associatd-to-a-stytch-oauth-or-magic-links-token-has-been-fully-registered-on-narrative-side-b2b

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /authentication/b2b/registration-status
Report whether the user associatd to a Stytch OAuth or Magic Links token has been fully registered on Narrative side.
Provides a Stytch session token that can be used to access the login and register APIs.



# Get dataset storage billing
Source: https://docs.narrative.io/api-reference/billings/get-dataset-storage-billing

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/billings/dataset-storage-billing
Get dataset storage billings paginated.



# Get dataset storage billing for company
Source: https://docs.narrative.io/api-reference/billings/get-dataset-storage-billing-for-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /billings/dataset-storage-billing
Get dataset storage billings for a company id paginated.
Company id is extracted from token.



# Get a presigned URL to upload company logo
Source: https://docs.narrative.io/api-reference/companies/get-a-presigned-url-to-upload-company-logo

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/companies/logo-upload-url
Sending a file to Narrative is a two step process.
This endpoint allows you to request an `upload_url` which can be used in a `PUT` request to upload the file.
For example, once you have received an `upload_url` you could use it as follows:

```bash
curl -v --upload-file logo.png {upload_url}
```

The generated `upload_url` is only valid for 30 minutes after issuance.

Response also contains `permanent_url`, `expiry`, and `content_type`.
`permanent_url` should be used as long-term reference.
`content_type` should be used as Content-Type header since it is part of URL signature.



# Lists Company Domains
Source: https://docs.narrative.io/api-reference/companies/lists-company-domains

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /companies/{company_id}/domains
# /companies/{company_id}/domains

Returns all domains registered by a specific company



# Get all data-streams provider companies
Source: https://docs.narrative.io/api-reference/company-marketing-information/get-all-data-streams-provider-companies

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /company-info/data-stream-markets
Returns all the companies providing data-stream market offers.



# Get marketing details on a company
Source: https://docs.narrative.io/api-reference/company-marketing-information/get-marketing-details-on-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /company-info/{company_id}
Returns the marketing detailed information on a company.



# Returns the company and user associated with the token
Source: https://docs.narrative.io/api-reference/company-marketing-information/returns-the-company-and-user-associated-with-the-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /company-info/whoami
Returns the company and user associated with the token.



# Clear the company's default compute pool for a data plane
Source: https://docs.narrative.io/api-reference/company/clear-the-companys-default-compute-pool-for-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /company/{company_id}/data-planes/{data_plane_id}/default-compute-pool
Clears the company-level default compute pool for the given data plane. After this call, jobs whose pool is
resolved through the company-level default (level 3 of the resolution chain) on `data_plane_id` will fall through
to the data-plane default (level 4).

Idempotent: clearing when no default is set is a no-op and still returns `204 No Content`.



# Delete (archive) a company
Source: https://docs.narrative.io/api-reference/company/delete-archive-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /company/{company_id}



# Get a company
Source: https://docs.narrative.io/api-reference/company/get-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /companies/{company_id}



# Get a company
Source: https://docs.narrative.io/api-reference/company/get-a-company-1

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /company/{company_id}



# Set the company's default compute pool for a data plane
Source: https://docs.narrative.io/api-reference/company/set-the-companys-default-compute-pool-for-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /company/{company_id}/data-planes/{data_plane_id}/default-compute-pool/{compute_pool_id}
Sets the company-level default compute pool for the given data plane. This is level 3 of the compute-pool
resolution chain — when a job doesn't pin a pool explicitly (level 1) and the dataset has no default (level 2),
this default is used before falling through to the data-plane default (level 4).

The pool must be active, owned by the company or shared with it via `using` collaborators, and live on
`data_plane_id`. Validation mirrors the rule applied to dataset-level defaults.

Replaces any existing default for `data_plane_id`. Other per-data-plane entries on the company are preserved.



# Update a company
Source: https://docs.narrative.io/api-reference/company/update-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /company/{company_id}
Update the company.
Please note that "slug" could be changed only once (from null to some value).
Lack of slug (in the request) considered as change to null and passes validation if current value is null too.
Status can be changed from pending to active but not in the opposite direction.



# Update a user
Source: https://docs.narrative.io/api-reference/company/update-a-user

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /users/{user_id}



# Archive a compute pool
Source: https://docs.narrative.io/api-reference/compute-pools/archive-a-compute-pool

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /data-planes/{data_plane_id}/compute-pools/{compute_pool_id}
Archive the specified compute pool owned by the authenticated company.



# Create a compute pool
Source: https://docs.narrative.io/api-reference/compute-pools/create-a-compute-pool

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-planes/{data_plane_id}/compute-pools
Create a new compute pool owned by the authenticated company.



# Get a compute pool by ID
Source: https://docs.narrative.io/api-reference/compute-pools/get-a-compute-pool-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-planes/{data_plane_id}/compute-pools/{compute_pool_id}
Retrieve a specific compute pool accessible to the authenticated company.



# List compute pools
Source: https://docs.narrative.io/api-reference/compute-pools/list-compute-pools

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-planes/{data_plane_id}/compute-pools
Retrieve all compute pools accessible to the authenticated company.



# Update a compute pool
Source: https://docs.narrative.io/api-reference/compute-pools/update-a-compute-pool

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-planes/{data_plane_id}/compute-pools/{compute_pool_id}
Update an existing compute pool owned by the authenticated company.



# Archive a connection
Source: https://docs.narrative.io/api-reference/connections/archive-a-connection

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /connections/{connection_id}



# Creates a connection
Source: https://docs.narrative.io/api-reference/connections/creates-a-connection

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /connections



# Returns a connection
Source: https://docs.narrative.io/api-reference/connections/returns-a-connection

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /connections/{connection_id}



# Searches for connections from a given dataset id
Source: https://docs.narrative.io/api-reference/connections/searches-for-connections-from-a-given-dataset-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /connections



# Create contract
Source: https://docs.narrative.io/api-reference/contracts/create-contract

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/contracts
Create a new contract with rates



# Get active contract
Source: https://docs.narrative.io/api-reference/contracts/get-active-contract

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/contracts/active/{company_id}
Return active contract with details for a company id.



# Get active contract for company
Source: https://docs.narrative.io/api-reference/contracts/get-active-contract-for-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /contracts/active
Return active contract with details for a company id.
Company id is extracted from token.



# Get contract
Source: https://docs.narrative.io/api-reference/contracts/get-contract

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/contracts/{contract_id}/{revision}
Return single contract with details by id and revision.



# Get contracts
Source: https://docs.narrative.io/api-reference/contracts/get-contracts

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/contracts
Get contracts with details paginated.



# Get contracts for company
Source: https://docs.narrative.io/api-reference/contracts/get-contracts-for-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /contracts
Return all contract with details for a company id.
Company id is extracted from token.



# Get rates for active contract
Source: https://docs.narrative.io/api-reference/contracts/get-rates-for-active-contract

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/contracts/rate/{company_id}
Return all rates for a active contract for company_id



# Update Contract
Source: https://docs.narrative.io/api-reference/contracts/update-contract

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /admin/contracts/{company_id}
Update active contract for a company



# Add a collaborator to a data plane
Source: https://docs.narrative.io/api-reference/data-planes/add-a-collaborator-to-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-planes/{data_plane_id}/collaborators/{collaborator_company_id}
Grants the specified company access to the given data plane.



# Archive a data plane
Source: https://docs.narrative.io/api-reference/data-planes/archive-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /data-planes/{data_plane_id}
Archive the specified data plane owned by the authenticated company.



# Clear the default compute pool for a data plane
Source: https://docs.narrative.io/api-reference/data-planes/clear-the-default-compute-pool-for-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /data-planes/{data_plane_id}/default-compute-pool
Removes the default compute pool setting from this data plane.



# Create a data plane
Source: https://docs.narrative.io/api-reference/data-planes/create-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-planes
Create a new data plane owned by the authenticated company.



# Data plane heartbeat
Source: https://docs.narrative.io/api-reference/data-planes/data-plane-heartbeat

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-planes/{data_plane_id}/heartbeat
Send a heartbeat to update the last active timestamp of the data plane.



# Get a data plane by ID
Source: https://docs.narrative.io/api-reference/data-planes/get-a-data-plane-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-planes/{data_plane_id}
Retrieve a specific data plane owned or shared with the authenticated company.



# List data planes
Source: https://docs.narrative.io/api-reference/data-planes/list-data-planes

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-planes
Retrieve all data planes accessible to the authenticated company.



# Remove a collaborator from a data plane
Source: https://docs.narrative.io/api-reference/data-planes/remove-a-collaborator-from-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /data-planes/{data_plane_id}/collaborators/{collaborator_company_id}
Revokes the specified company's access to the given data plane.



# Replace the collaborators on a data plane
Source: https://docs.narrative.io/api-reference/data-planes/replace-the-collaborators-on-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-planes/{data_plane_id}/collaborators
Overwrites the entire `collaborators` object on the data plane, setting both `participants` (who may
use the data plane) and `manage_compute_pools` (which companies may create / edit / archive compute
pools on it). Mirrors the whole-object PUT pattern used by attributes.

The granular per-company `PUT/DELETE /data-planes/{data_plane_id}/collaborators/{collaborator_company_id}`
endpoints continue to work for incremental mutation of `participants`; this endpoint is the only way
to set `manage_compute_pools`.




# Set the default compute pool for a data plane
Source: https://docs.narrative.io/api-reference/data-planes/set-the-default-compute-pool-for-a-data-plane

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-planes/{data_plane_id}/default-compute-pool/{compute_pool_id}
Sets the specified compute pool as the default for this data plane.
The compute pool must exist, be active, belong to the same company, and be associated with this data plane.



# Trigger a health check
Source: https://docs.narrative.io/api-reference/data-planes/trigger-a-health-check

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-planes/{data_plane_id}/health-check
Enqueue a health check job for the specified data plane. Optionally target a specific compute pool. Returns the created job.



# Create a data-shop.
Source: https://docs.narrative.io/api-reference/data-shops/create-a-data-shop

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shop
Create a new data-shop.



# Fetch many data-shops
Source: https://docs.narrative.io/api-reference/data-shops/fetch-many-data-shops

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shop/ids
Fetch many data-shops, given their `ids`.

Allows only upto 200 data-shops to be fetched in a call. If no `ids` are requested, empty array result is returned.



# Get a data-shop by its host name.
Source: https://docs.narrative.io/api-reference/data-shops/get-a-data-shop-by-its-host-name

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shop/hostname/{host_name}
Gets the data-shop given its host name.



# Get a data-shop by its id.
Source: https://docs.narrative.io/api-reference/data-shops/get-a-data-shop-by-its-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shop/{data_shop_id}
Gets the data-shop given its uuid.



# Get all data-shops for a company.
Source: https://docs.narrative.io/api-reference/data-shops/get-all-data-shops-for-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shop/company/{company_id}
Gets all data-shops belonging to a company or empty if non exist.



# Update a data-shop.
Source: https://docs.narrative.io/api-reference/data-shops/update-a-data-shop

https://docs-cdn.narrative.io/api-reference/main/openapi.json patch /data-shop/{data_shop_id}
Update a new data-shop.

**Note**: Not passing a `fe_manifest` will remove the `fe_manifest` from the data-shop `manifest`.

Only a user belonging to the company can update a data-shop.



# Create a data-stream.
Source: https://docs.narrative.io/api-reference/data-streams/create-a-data-stream

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-stream
Create a new data-stream.

Any datasets involved in the data-stream should exist and be active. Any fields referred



# Fetch many data-streams
Source: https://docs.narrative.io/api-reference/data-streams/fetch-many-data-streams

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-stream/ids
Fetch many data-streams, given their `ids`.

Allows only upto 50 data-streams to be fetched in a call. If no `ids` are requested, empty array result is returned.

**Note**: If `n` `ids` are requested this might return any `m` `(0 <= m <= n)` number of data-streams depending on
how many actually do exist. It is the clients duty to find this discrepancy.



# Find data-streams available on the data-streams-market.
Source: https://docs.narrative.io/api-reference/data-streams/find-data-streams-available-on-the-data-streams-market

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/sales-channel/data-streams-market/search
Gets all data-streams available on the data-streams-market matching the given search expression.

Search expressions follow the postgres 'websearch_to_tsquery' specification:

1. logical operators : `tshirt and apparel or accessories -leather`
2. phrases           : `"find this"  -"but not this"`
3. multiple words    : `goods services`  (this is the same as `goods and services`)

Reference: https://www.postgresql.org/docs/11/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES search for 'websearch_to_tsquery'



# Find data-streams for the given company and a sales-channel and the given search expression.
Source: https://docs.narrative.io/api-reference/data-streams/find-data-streams-for-the-given-company-and-a-sales-channel-and-the-given-search-expression

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/sales-channel/{sales_channel}/search
Gets all data-streams belonging to a company, filtered for the sales-channel and the given search expression.

Search expressions follow the postgres 'websearch_to_tsquery' specification:

1. logical operators : `tshirt and apparel or accessories -leather`
2. phrases           : `"find this"  -"but not this"`
3. multiple words    : `goods services`  (this is the same as `goods and services`)

Reference: https://www.postgresql.org/docs/11/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES search for 'websearch_to_tsquery'



# Find data-streams for the given company and its data-shop hostname and the given search expression.
Source: https://docs.narrative.io/api-reference/data-streams/find-data-streams-for-the-given-company-and-its-data-shop-hostname-and-the-given-search-expression

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/host/{host_name}/search
Gets all data-streams belonging to a company, filtered for the hostname and the given search expression.

Search expressions follow the postgres 'websearch_to_tsquery' specification:

1. logical operators : `tshirt and apparel or accessories -leather`
2. phrases           : `"find this"  -"but not this"`
3. multiple words    : `goods services`  (this is the same as `goods and services`)

Reference: https://www.postgresql.org/docs/11/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES search for 'websearch_to_tsquery'



# Get a data-stream by its id.
Source: https://docs.narrative.io/api-reference/data-streams/get-a-data-stream-by-its-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/{data_stream_id}
Gets the data-stream given its uuid.



# Get all data-streams for a company.
Source: https://docs.narrative.io/api-reference/data-streams/get-all-data-streams-for-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}
Gets all data-streams belonging to a company or empty if non exist.



# Get all data-streams for a company given a sales channel
Source: https://docs.narrative.io/api-reference/data-streams/get-all-data-streams-for-a-company-given-a-sales-channel

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/sales-channel/{sales_channel}
Gets all data-streams belonging to a company, given a sales channel.

Sales channels:
  1. data-streams-market
  2. data-shop



# Get all data-streams for a company given a search expression.
Source: https://docs.narrative.io/api-reference/data-streams/get-all-data-streams-for-a-company-given-a-search-expression

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/search
Gets all data-streams belonging to a company and the given search expression.

Search expressions follow the postgres 'websearch_to_tsquery' specification:

1. logical operators : `tshirt and apparel or accessories -leather`
2. phrases           : `"find this"  -"but not this"`
3. multiple words    : `goods services`  (this is the same as `goods and services`)

Reference: https://www.postgresql.org/docs/11/textsearch-controls.html#TEXTSEARCH-PARSING-QUERIES search for 'websearch_to_tsquery'



# Get all data-streams for the given company and its data-shop hostname.
Source: https://docs.narrative.io/api-reference/data-streams/get-all-data-streams-for-the-given-company-and-its-data-shop-hostname

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/host/{host_name}
Gets all data-streams belonging to a company, filtered for the hostname.



# Get schema
Source: https://docs.narrative.io/api-reference/data-streams/get-schema

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/slug/{slug}/schema
Retrieve schema for deliveries from this data stream.



# Gets the data-stream for a given company and slug.
Source: https://docs.narrative.io/api-reference/data-streams/gets-the-data-stream-for-a-given-company-and-slug

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-stream/company/{company_id}/slug/{slug}
Gets the data-stream for a given company and slug



# Update a data-stream.
Source: https://docs.narrative.io/api-reference/data-streams/update-a-data-stream

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-stream/{data_stream_id}
Update a new data-stream.

The main idea: get a data-stream document/resource, after creation or accessing by id,
then update the document and PUT it as it is.

Of course certain things aren't updateable and are ignored,
for instance identifiers already established (like DS id) and
metadata like creation/update dates and who created/updated the entity.

Things that can be updated

- name
- slug
- content
- tags
- category
- offers
- data_rules
- auth_rules

**Note**: nested structures like content etc. are changed as complete structures,
hence care needs to be taken to include unchanged data at as the same time as the data being changed so as not to
over-write the rest of it.

Any validations applicable on creation time also apply on updates.

Only a user belonging to the company can update a data-stream.



# Preview snapshot expirations
Source: https://docs.narrative.io/api-reference/dataset-configuration/preview-snapshot-expirations

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/retention-policy/preview
Preview snapshot expirations given a retention policy.



# Upsert retention policies for a dataset
Source: https://docs.narrative.io/api-reference/dataset-configuration/upsert-retention-policies-for-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/retention-policy
Accepts either a legacy payload (snapshot-only) or the unified v1 policy object. The response is always returned in the unified v1 shape with a `compat.mode` indicating whether the server interpreted the request as legacy or v1.
**Row TTL clock default:** If `row_ttl.enabled=true` and `row_ttl.clock` is omitted, the server will default to the configured Rosetta Stone attribute (nio_last_modified) **if that mapping exists**; otherwise  it returns HTTP 400.




# Files added
Source: https://docs.narrative.io/api-reference/dataset-files/files-added

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/snapshots/{snapshot_id}/files-added
Lists files added by a snapshot



# Find files
Source: https://docs.narrative.io/api-reference/dataset-files/find-files

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/find-files
Lists snapshots and associated files



# Find snapshots
Source: https://docs.narrative.io/api-reference/dataset-files/find-snapshots

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/find-snapshots
Lists snapshots and associated files



# Issue file download url
Source: https://docs.narrative.io/api-reference/dataset-files/issue-file-download-url

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/snapshots/{snapshot_id}/files-added/{file_id}/download
Retrieve the download URL for a file



# Suggest mappings
Source: https://docs.narrative.io/api-reference/dataset-mappings/suggest-mappings

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/suggest-mappings
Schedule the execution of a mappings suggestion job for a given dataset, using the Rosetta Stone API (Snowflake data plane only).



# Archive the materialized view refresh configuration
Source: https://docs.narrative.io/api-reference/dataset-refresh/archive-the-materialized-view-refresh-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /datasets/{dataset_id}/refresh-schedule



# Create a refresh schedule for an existing materialized view
Source: https://docs.narrative.io/api-reference/dataset-refresh/create-a-refresh-schedule-for-an-existing-materialized-view

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/refresh-schedule
Creates new request for request derivation



# Run a materialized view refresh now
Source: https://docs.narrative.io/api-reference/dataset-refresh/run-a-materialized-view-refresh-now

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/refresh-materialized-view



# Update the materialized view refresh configuration
Source: https://docs.narrative.io/api-reference/dataset-refresh/update-the-materialized-view-refresh-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/refresh-schedule
Creates new request for request derivation



# Delete a sample
Source: https://docs.narrative.io/api-reference/dataset-samples/delete-a-sample

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /datasets/{dataset_id}/sample
Deletes a sample of rows from a dataset.



# Request Dataset Sample
Source: https://docs.narrative.io/api-reference/dataset-samples/request-dataset-sample

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/request-sample
Request a dataset sample job with pre-compiled SQL (for non-Narrative Data Planes only)



# Sample a dataset
Source: https://docs.narrative.io/api-reference/dataset-samples/sample-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/sample
Get a sample of rows from a dataset as JSON.



# Sample a dataset (Admin)
Source: https://docs.narrative.io/api-reference/dataset-samples/sample-a-dataset-admin

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/admin/sample
Get a sample of rows from a dataset as JSON.



# Sampled Mappings
Source: https://docs.narrative.io/api-reference/dataset-samples/sampled-mappings

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/sample/mappings
Returns sampled dataset after attribute mapping for a given `mapping_id`

This endpoint should be used when mapping exist.

A response is a list of JSON objects, where every object will have an `input` field, that represents the original key/value pairs from the dataset, and the `output` filed that will contain mapped fields where the key will be `attribute name`, and value will be JSON object of key/value pairs.



# Disable Statistics Configuration
Source: https://docs.narrative.io/api-reference/dataset-statistics/disable-statistics-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /datasets/{dataset_id}/statistics-configuration
Disable the statistics configuration. Statistics will no longer be computed.



# Get Statistics Configuration
Source: https://docs.narrative.io/api-reference/dataset-statistics/get-statistics-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/statistics-configuration
Retrieve the current statistics configuration for a dataset.



# Put Statistics Configuration
Source: https://docs.narrative.io/api-reference/dataset-statistics/put-statistics-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/statistics-configuration
Create or update the statistics configuration for a dataset. Replaces any existing configuration.



# Activate a dataset
Source: https://docs.narrative.io/api-reference/datasets/activate-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/activate
Activates a dataset.

This locks in the schema and allows Narrative to begin ingesting data.



# Activate a dataset
Source: https://docs.narrative.io/api-reference/datasets/activate-a-dataset-1

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/activate
Activates a dataset.

This locks in the schema and allows Narrative to begin ingesting data.



# Create a dataset
Source: https://docs.narrative.io/api-reference/datasets/create-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets
Create or register a new dataset.

Newly created datasets are in the `pending` status and require activation.

To register an empty dataset in Snowflake Dataplane:

If the request has `data_plane_id` defined but `external_id` is not provided, it will create an empty dataset with the specified schema in `pending` state.

To trigger creation of such dataset in Snowflake Dataplane, it should call `PUT datasets/<dataset_id>/activate`.



# Create or Update detailed statistics
Source: https://docs.narrative.io/api-reference/datasets/create-or-update-detailed-statistics

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/columns-stats
Create or Update detailed statistics. Detailed statistics are collected for datasets belonging to the Narrative Data Plane, but
uploading these stats is necessary for non-Narrative data planes, since Narrative does not have access to the data to compute these statistics.



# Delete a dataset
Source: https://docs.narrative.io/api-reference/datasets/delete-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /datasets/{dataset_id}
Permanently removes a dataset from the platform.

Dataset deletion cannot be undone.



# Derive the configuration
Source: https://docs.narrative.io/api-reference/datasets/derive-the-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/derive-stats-config
Creates new request for request derivation



# Disable configuration
Source: https://docs.narrative.io/api-reference/datasets/disable-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /datasets/{dataset_id}/stats-config
Disable configuration, metrics won't be calculated



# Enable configuration
Source: https://docs.narrative.io/api-reference/datasets/enable-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/enable-stats-config
Enable configuration if it exists



# Get a dataset
Source: https://docs.narrative.io/api-reference/datasets/get-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}



# Get admin info for a dataset
Source: https://docs.narrative.io/api-reference/datasets/get-admin-info-for-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/admin
Get a dataset including information about the backing [Iceberg table](http://iceberg.apache.org/).



# Get dataset statistics
Source: https://docs.narrative.io/api-reference/datasets/get-dataset-statistics

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/stats
Return summary statistics for all ingestion snapshots for a given dataset.
For datasets with "incremental" updates, each snapshot defines a new incremental upload of data.
For datasets with "overwrite" updates, each snapshot overwrites the previous data.
`all_records` query parameter is here because of backward compatibility.
By default API is paginated, so if you don't provide both `per_page` and `offset` parameters, you will get only the latest summary.



# Get datasets
Source: https://docs.narrative.io/api-reference/datasets/get-datasets

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets
Return all datasets belonging to the authenticated company.



# Get detailed statistics
Source: https://docs.narrative.io/api-reference/datasets/get-detailed-statistics

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/columns-stats
Get statistics for each column.



# Get Metrics Configuration
Source: https://docs.narrative.io/api-reference/datasets/get-metrics-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/stats-config
Put new version of configuration.



# Ingest a file
Source: https://docs.narrative.io/api-reference/datasets/ingest-a-file

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/upload
After uploading a file to Narrative via the `upload` API, you can then have the file can then be ingested into a dataset.

Related guides:
  - [How do I create a dataset?](https://kb.narrative.io/how-do-i-create-a-dataset): see the section titled **Upload Files for Ingestion**.



# List connector interfaces compatible with a dataset
Source: https://docs.narrative.io/api-reference/datasets/list-connector-interfaces-compatible-with-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/interfaces
Evaluates the dataset's schema against every connector interface exposed by apps installed for
the caller's company, and reports which interfaces accept the schema and which reject it.

Each installed app may expose one or more interfaces, each with its own JSON-Schema-based
policy. An interface with no policy accepts any dataset schema.

The optional `tags` query parameter restricts evaluation to interfaces whose metadata tags
intersect the supplied set (case-insensitive, OR semantics). When omitted, every interface is
evaluated.



# Metrics
Source: https://docs.narrative.io/api-reference/datasets/metrics

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/metrics
Lists metrics for a dataset



# Metrics
Source: https://docs.narrative.io/api-reference/datasets/metrics-1

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /datasets/{dataset_id}/snapshots/{snapshot_id}/column-stats
Metrics for a snapshot



# Put Metrics Configuration
Source: https://docs.narrative.io/api-reference/datasets/put-metrics-configuration

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}/stats-config
Put new version of configuration.



# Submit recalculation
Source: https://docs.narrative.io/api-reference/datasets/submit-recalculation

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/columns-stats/recalculate
Run recalculation for the dataset.



# Truncate a dataset
Source: https://docs.narrative.io/api-reference/datasets/truncate-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/truncate
Truncates a dataset.

Removes all rows from the dataset but leaves the dataset intact.



# Update a dataset
Source: https://docs.narrative.io/api-reference/datasets/update-a-dataset

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets
Update existing dataset.



# Update a dataset
Source: https://docs.narrative.io/api-reference/datasets/update-a-dataset-1

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /datasets/{dataset_id}
Update a pending or active dataset.

All fields can be updated on a pending dataset.
Schema property metadata (`sensitive`, `required`, `unsellable`) and enumeration (append-only), `description`, `display_name`
can be updated on an active dataset. 

Archived datasets cannot be updated.



# Update a dataset's partition spec
Source: https://docs.narrative.io/api-reference/datasets/update-a-datasets-partition-spec

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /datasets/{dataset_id}/admin/partition-spec
Update the partition specification of the [Iceberg table](http://iceberg.apache.org/) backing the dataset.

See:
  - [What is partitioning?](http://iceberg.apache.org/partitioning/)
  - [Iceberg Specification: Partitioning](http://iceberg.apache.org/spec/#partitioning)



# Creates a destination
Source: https://docs.narrative.io/api-reference/destinations/creates-a-destination

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /destinations



# Get a destination by id
Source: https://docs.narrative.io/api-reference/destinations/get-a-destination-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /destinations/{destination_id}



# Removes the destination from a subscription
Source: https://docs.narrative.io/api-reference/destinations/removes-the-destination-from-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /destinations/{destination_id}



# Search destinations for a profile
Source: https://docs.narrative.io/api-reference/destinations/search-destinations-for-a-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /destinations/profile_id={profile_id}



# Search destinations for a subscription
Source: https://docs.narrative.io/api-reference/destinations/search-destinations-for-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /destinations/subscription_id={subscription_id}



# Updates the values of the quick settings
Source: https://docs.narrative.io/api-reference/destinations/updates-the-values-of-the-quick-settings

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /destinations/{destination_id}



# Archive a connection
Source: https://docs.narrative.io/api-reference/encryption-materials/archive-a-connection

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /encryption-materials/{encryption_material_id}



# Create an encryption material
Source: https://docs.narrative.io/api-reference/encryption-materials/create-an-encryption-material

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /encryption-materials
Create an encryption material



# List encryption materials
Source: https://docs.narrative.io/api-reference/encryption-materials/list-encryption-materials

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /encryption-materials
List encryption materials



# Create an installation
Source: https://docs.narrative.io/api-reference/installations/create-an-installation

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations
Creates a new installation for an app.

The installation associates an app with the authenticated company using a specific tier from the app's manifest.
The tier determines which permissions are granted to the installation.



# Create an installation-scoped access token
Source: https://docs.narrative.io/api-reference/installations/create-an-installation-scoped-access-token

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations/{installation_id}/token
Returns a Narrative API access token scoped to the given installation, carrying the
permissions granted to the application at installation.

With a user-bound bearer token, the access token is created on behalf of the
authenticated user for an installation belonging to the user's company. The token is
used with an application (e.g., the S3 connector) to allow the application to call
Narrative's API on the user's behalf.

With an app client-credentials bearer token (obtained via `POST /oauth/token` with
`grant_type=client_credentials`), the access token is created by the application itself
for one of its own installations, allowing the application to act on behalf of a
company that has installed it. Use `GET /installations` with the app token to discover
installation ids.



# Delete an installation
Source: https://docs.narrative.io/api-reference/installations/delete-an-installation

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /installations/{installation_id}
Deletes an installation by its ID.

The installation must belong to the authenticated company. An installation cannot be deleted
if it has active profiles associated with it. Remove all profiles before uninstalling the app.



# Get an installation
Source: https://docs.narrative.io/api-reference/installations/get-an-installation

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /installations/{installation_id}
Returns a single installation by its ID.

With a company-scoped token (e.g. a user token), the installation must belong to the
authenticated company.

With an app client-credentials token, the installation must belong to the authenticated app.



# Get current installation info
Source: https://docs.narrative.io/api-reference/installations/get-current-installation-info

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /installations/whoami
Returns information about the installation associated with the current authentication token.

This endpoint is useful for apps to identify which installation they are operating under.
The token must be associated with an installation (e.g., an app token).



# List installations
Source: https://docs.narrative.io/api-reference/installations/list-installations

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /installations
Returns a paginated list of installations.

With a company-scoped token (e.g. a user token), lists the installations of the
authenticated company, optionally filtered by app category using the `app_category`
query parameter.

With an app client-credentials token, lists the installations of the authenticated
app across all companies, optionally narrowed to a single company using the
`company_id` query parameter. Use this to look up the installation id for a company
before creating an installation-scoped access token.

Results are paginated using `page` (1-based, default `1`) and `per_page`
(default `100`) query parameters.



# Cancel a job
Source: https://docs.narrative.io/api-reference/jobs/cancel-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/cancel
Finalizes cancellation of a job, moving it from `pending_cancellation` to `cancelled`. It is a no-op for a job
already in a terminal state (`completed`, `cancelled`, `failed`); any other non-terminal state (e.g. `pending` or
`running`) is rejected. Use `request-cancellation` first to move a job into `pending_cancellation`.

Requires write access to jobs.



# Complete a job
Source: https://docs.narrative.io/api-reference/jobs/complete-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/complete
Marks a job as `completed` along with its result. The result shape is selected by the `type` field in the body
and must match the job's type. It is a no-op for a job already in a terminal state (`completed`, `cancelled`,
`failed`); a `pending_cancellation` job can still be completed if it finishes before the cancellation is processed.

Requires write access to jobs.



# Fail a job
Source: https://docs.narrative.io/api-reference/jobs/fail-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/fail
Marks a job as `failed` with a message and optional diagnostic value. It is a no-op for a job already in a
terminal state (`completed`, `cancelled`, `failed`); a `pending_cancellation` job can still be failed if it fails
before the cancellation is processed.

Requires write access to jobs.



# Get a job
Source: https://docs.narrative.io/api-reference/jobs/get-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /jobs/{job_id}
This endpoint retrieves all details associated with a specific job ID from the jobs table.



# Get jobs
Source: https://docs.narrative.io/api-reference/jobs/get-jobs

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /jobs
Returns a list of jobs associated with the specified company, including job ID, job type, timestamp information, and results.

It returns a maximum of 500 jobs at a time to manage response size.



# Request cancellation of a job
Source: https://docs.narrative.io/api-reference/jobs/request-cancellation-of-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/request-cancellation
Requests cancellation of a `pending` or `running` job, moving it to `pending_cancellation` so the data plane can
stop the work. It is a no-op for a job already in a terminal state (`completed`, `cancelled`, `failed`) or already
in `pending_cancellation`.

Cancellation is only supported for specific job types on the Narrative data plane, and on Snowflake data planes.
Use `cancel` to finalize a job once it has reached `pending_cancellation`.

Requires write access to jobs.



# Reschedule a job
Source: https://docs.narrative.io/api-reference/jobs/reschedule-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/reschedule
Puts a job back on the queue: resets it to `pending` and clears its executor so the operator re-dispatches it
onto a fresh cluster. This is typically used when a job's cluster terminated with its step still in flight.

By default this is a no-op for jobs already in a terminal state (`completed`, `cancelled`, `failed`). Supply a
body with `force: true` to reschedule a terminal job anyway (e.g. to re-run one that already finished). The body
is optional — an empty body keeps the default behavior.

The response's `outcome` reports whether the job was actually `rescheduled` or `skipped_terminal` (left untouched
because it was terminal and `force` was not set).

Requires write access to jobs.



# Run a job
Source: https://docs.narrative.io/api-reference/jobs/run-a-job

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /jobs/{job_id}/run
Starts a job by assigning it an executor and moving it to `running`. Applies to jobs in the `pending` or `failed`
state; it is a no-op for a job that is already `running` or in a non-retryable terminal state (`completed`,
`cancelled`). Idempotent.

Requires write access to jobs.



# Accept a mapping
Source: https://docs.narrative.io/api-reference/mappings/accept-a-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /mappings/{mapping_id}/accept
Accept a `system` proposed mapping, i.e. move it from the `status` of `pending` to `active`.

The mapping has to be `system` proposed and not `archived`.

Both owners (users belonging to the company the mapping belongs to) and admins can promote a mapping.



# Create a private mapping
Source: https://docs.narrative.io/api-reference/mappings/create-a-private-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /mappings/companies/{company_id}
Create a private mapping.

Supports mapping types: `cached_mapping`, `object_mapping`, and `value_mapping`.



# Create a system mapping
Source: https://docs.narrative.io/api-reference/mappings/create-a-system-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /mappings/system
Create a `system` mapping.

Only admins can call this endpoint.



# Delete a mapping
Source: https://docs.narrative.io/api-reference/mappings/delete-a-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /mappings/{mapping_id}
Mapping deletion cannot be undone.

If the mapping is still being used then this request fails.

Both owners (users belonging to the company the mapping belongs to) and admins can delete a mapping.



# Get mapping
Source: https://docs.narrative.io/api-reference/mappings/get-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /mappings/{mapping_id}
Return a specific mapping based on its id.



# Get mapping preview
Source: https://docs.narrative.io/api-reference/mappings/get-mapping-preview

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /mappings/{mapping_id}/sample
Returns sampled dataset after attribute mapping for a given `mapping_id`

This endpoint should be used when mapping exist.

A response is a list of JSON objects, where every object will have an `input` field, that represents the original key/value pairs from the dataset, and the `output` filed that will contain mapped fields where the key will be `attribute name`, and value will be JSON object of key/value pairs.



# Get mappings
Source: https://docs.narrative.io/api-reference/mappings/get-mappings

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /mappings
Return the mappings owned by the caller's company. Optionally filter by attribute and/or dataset.



# Get mappings for a company
Source: https://docs.narrative.io/api-reference/mappings/get-mappings-for-a-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /mappings/companies/{company_id}
Return the mappings owned by a specific company based on its id. Optionally filter by attribute
and/or dataset.



# Get system mappings
Source: https://docs.narrative.io/api-reference/mappings/get-system-mappings

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /mappings/system
Return all `system` mappings.

Only admins can call this endpoint.



# Test Mapping
Source: https://docs.narrative.io/api-reference/mappings/test-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /mappings/test
Returns sampled dataset after attribute mapping for a given mapping.

This endpoint should be used when mapping does not exist.

A response is a list of JSON objects, where every object will have an `input` field, that represents the original key/value pairs from the dataset, and the `output` filed that will contain mapped fields where the key will be `attribute name`, and value will be JSON object of key/value pairs.

A `cached_mapping` is resolved as a join at query time and has no per-row preview, so its response is always an empty list of records.



# Update a mapping
Source: https://docs.narrative.io/api-reference/mappings/update-a-mapping

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /mappings/{mapping_id}
Update the definition of a mapping between an attribute and a dataset.

Note: the attribute and dataset with which a mapping is associated cannot be updated.



# Run model inference
Source: https://docs.narrative.io/api-reference/model-inference/run-model-inference

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /model-inference/run
Create a job to perform an LLM call. Supported on both AWS and Snowflake data planes.



# Archive a model
Source: https://docs.narrative.io/api-reference/models/archive-a-model

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /models/{model_id}



# Get a model
Source: https://docs.narrative.io/api-reference/models/get-a-model

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /models/{model_id}
Returns a single model by its ID.



# Get models
Source: https://docs.narrative.io/api-reference/models/get-models

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /models
Returns a paginated list of models accessible to the authenticated company.

Models can be filtered by repository type and sorted by creation date.



# Update a model
Source: https://docs.narrative.io/api-reference/models/update-a-model

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /models/{model_id}
Updates an existing model.

Only the fields provided in the request body will be updated.



# Get the result of a NQL forecast
Source: https://docs.narrative.io/api-reference/nql/get-the-result-of-a-nql-forecast

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /nql/{job_id}
This endpoint is used to poll forecast result after it's ran.



# Run a NQL Query
Source: https://docs.narrative.io/api-reference/nql/run-a-nql-query

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /nql/run
The run endpoint in the NQL API allows users to execute their NQL queries.
This endpoint processes the query and performs the specified operations
within the specified data plane. Depending on the NQL provided, the run
endpoint can create materialized views or generate forecasts.



# Validate an NQL query
Source: https://docs.narrative.io/api-reference/nql/validate-an-nql-query

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /nql/validate
The validate endpoint in the NQL API allows users to verify the syntax and structure
of their NQL queries before executing them. This ensures that the queries are correctly
formatted and will not cause errors when passing through the /run endpoint.
The primary goal of using the validate endpoint is to receive a 200 OK status code,
which indicates that the query is valid.



# Creates a new setup intent
Source: https://docs.narrative.io/api-reference/payment-methods/creates-a-new-setup-intent

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /payment-methods/credit-card/setup-intent
# /payment-methods/credit-card/setup-intent

Creates a new setup intent



# Deletes a payment method
Source: https://docs.narrative.io/api-reference/payment-methods/deletes-a-payment-method

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /payment-methods/{payment_method_id}
# /payment-methods/{payment_method_id}

Deletes a payment method



# Returns payment methods for company
Source: https://docs.narrative.io/api-reference/payment-methods/returns-payment-methods-for-company

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /payment-methods
# /payment-methods

Returns payment methods for company



# Sets a valid SetupIntent as the primary payment method
Source: https://docs.narrative.io/api-reference/payment-methods/sets-a-valid-setupintent-as-the-primary-payment-method

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /payment-methods/credit-card/setup-intent/{setup_intent_id}
# /payment-methods/credit-card/setup-intent/{setup_intent_id}

Sets a valid SetupIntent as the primary payment method



# Create a permission
Source: https://docs.narrative.io/api-reference/permissions/create-a-permission

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/permissions
Create a new permission



# Attach permission to product
Source: https://docs.narrative.io/api-reference/products/attach-permission-to-product

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/products/add-permission
Attaches given permission to a given product



# Create a product
Source: https://docs.narrative.io/api-reference/products/create-a-product

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/products
Create a new product.



# Get product
Source: https://docs.narrative.io/api-reference/products/get-product

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/products/{sku}
Return single product



# Get products
Source: https://docs.narrative.io/api-reference/products/get-products

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /admin/products
Get products paginated



# Update product
Source: https://docs.narrative.io/api-reference/products/update-product

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /admin/products/{sku}
Update product by sku.



# Archives a profile
Source: https://docs.narrative.io/api-reference/profiles/archives-a-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations/{installation_id}/profiles/{app_profile_id}/archive
Archiving prevents using the profile again in the future.



# Creates an app profile
Source: https://docs.narrative.io/api-reference/profiles/creates-an-app-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations/{installation_id}/profiles
The profile will be assigned a unique identifier upon creation. New profiles are "disabled" and should be activated after creation.



# Disables a profile
Source: https://docs.narrative.io/api-reference/profiles/disables-a-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations/{installation_id}/profiles/{app_profile_id}/disable
Only profiles that are enabled will be considered when delivering subscriptions.



# Enables a profile
Source: https://docs.narrative.io/api-reference/profiles/enables-a-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /installations/{installation_id}/profiles/{app_profile_id}/enable
Only profiles that are enabled will be considered when delivering subscriptions.



# Returns app profiles
Source: https://docs.narrative.io/api-reference/profiles/returns-app-profiles

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /installations/{installation_id}/profiles



# Returns profile with {profile_id}
Source: https://docs.narrative.io/api-reference/profiles/returns-profile-with-

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /installations/{installation_id}/profiles/{app_profile_id}



# Updates an existing app profile
Source: https://docs.narrative.io/api-reference/profiles/updates-an-existing-app-profile

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /installations/{installation_id}/profiles/{app_profile_id}



# Create a query
Source: https://docs.narrative.io/api-reference/queries/create-a-query

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /v2/queries
Any number of properties can be updated on the same request.
If a property is not sent on the payload, the value won't be updated.



# Deletes a query
Source: https://docs.narrative.io/api-reference/queries/deletes-a-query

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /v2/queries/{query_id}
Deletes a query



# Get a query by id
Source: https://docs.narrative.io/api-reference/queries/get-a-query-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/queries/{query_id}
Get a query.



# Get accessible queries
Source: https://docs.narrative.io/api-reference/queries/get-accessible-queries

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/queries
Get all accessible queries.



# Updates a query
Source: https://docs.narrative.io/api-reference/queries/updates-a-query

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /v2/queries/{query_id}
Updates a query



# Create bucket
Source: https://docs.narrative.io/api-reference/resources/create-bucket

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /resources/buckets
Create an AWS S3 bucket inside Narrative's AWS account assigned to the authenticated company.

The created bucket is uniquely accessible by the authenticated company and can be used to facilitate the automation of tasks like ingestion of data into a dataset.

Pay attention, that bucket-owner-full-control ACL is strictly required for "bucket_policy" type of access.



# Get buckets
Source: https://docs.narrative.io/api-reference/resources/get-buckets

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /resources/buckets
Return the AWS S3 buckets assigned to the authenticated company.



# Update access type
Source: https://docs.narrative.io/api-reference/resources/update-access-type

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /resources/update-access
Update access type to the bucket.
Pay attention, that bucket-owner-full-control ACL is strictly required for "bucket_policy" type of access.
Also, bucket should be whitelisted for mutation. All buckets created by creation endpoint are mutable by default.
Buckets created before 2023 could be immutable in case of customized policies.
You can call 'Get buckets' endpoint to check `is_access_mutable` property in response.



# Infer the dataset schema of a file passed by multipart file.
Source: https://docs.narrative.io/api-reference/schema-inference/infer-the-dataset-schema-of-a-file-passed-by-multipart-file

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /v2/inference



# Create a private schema preset.
Source: https://docs.narrative.io/api-reference/schema-presets/create-a-private-schema-preset

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /schema-presets
Create a private schema preset owned by the authenticated account. Only members of the same organization are able to see this schema preset.



# Create a public schema preset.
Source: https://docs.narrative.io/api-reference/schema-presets/create-a-public-schema-preset

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /admin/schema-presets
Create a **public** schema preset available platform wide. This endpoint is only available to administrators.



# Delete a schema preset by its id.
Source: https://docs.narrative.io/api-reference/schema-presets/delete-a-schema-preset-by-its-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /schema-presets/{schema_preset_id}
Archive (soft delete) a specific schema preset given its id. The schema preset will not be visible from that point forward.



# Detail information on a schema preset by its id.
Source: https://docs.narrative.io/api-reference/schema-presets/detail-information-on-a-schema-preset-by-its-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /schema-presets/{schema_preset_id}
Returns detailed information on a specific schema preset given its id.



# Returns all available schema presets.
Source: https://docs.narrative.io/api-reference/schema-presets/returns-all-available-schema-presets

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /schema-presets
List the schema presets that are publicly available and the ones owned by the authenticated account.



# Returns all publicly available schema presets.
Source: https://docs.narrative.io/api-reference/schema-presets/returns-all-publicly-available-schema-presets

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /schema-presets/public
List the schema presets that are publicly available.



# Activate a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/activate-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/{subscription_id}/activate
Activation will result in charges to your company's payment method.



# Cancel a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/cancel-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/{subscription_id}/cancel
A cancelled subscription cannot be reactivated.



# Checkout a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/checkout-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/{subscription_id}/checkout
Checkout creates and activates a subscription in a single API call.

Activation will result in charges to your company's payment method.



# Create a cost forecast
Source: https://docs.narrative.io/api-reference/subscriptions/create-a-cost-forecast

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/cost-forecasts
Create an estimate of the "cost" of servicing a set of subscription constraints.

Cost has two components:

- The number of rows which must be read in order to find data which matches the subscription constraints.
- An estimate of the "size" of those rows in bytes.

Note: the number of rows reported in the cost is **not** an estimate of the number of rows which will be purchased as part of a subscription using the given constraints.



# Create a cost forecast for a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/create-a-cost-forecast-for-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/{subscription_id}/cost-forecasts
Create a cost forecast for a subscription, taking into account data already processed for the subscription.



# Create a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/create-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions
Subscriptions created using this endpoint are left in the `pending` state and require activation.



# Delete a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/delete-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /data-shops/subscriptions/{subscription_id}
Subscription deletion cannot be undone.

If a subscription is active it will be cancelled.



# Get a cost forecast
Source: https://docs.narrative.io/api-reference/subscriptions/get-a-cost-forecast

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/cost-forecasts/{cost_forecast_id}
Get a cost forecast by id.



# Get a delivered file
Source: https://docs.narrative.io/api-reference/subscriptions/get-a-delivered-file

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/{subscription_id}/deliveries/download
Retrieve a AWS S3 presigned URL to download a file delivered as part of a subscription.



# Get a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/get-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/{subscription_id}
Get a subscription by id.



# Get cost forecasts
Source: https://docs.narrative.io/api-reference/subscriptions/get-cost-forecasts

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/cost-forecasts
Return all cost forecasts belonging to the authenticated company.



# Get deliveries
Source: https://docs.narrative.io/api-reference/subscriptions/get-deliveries

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/{subscription_id}/deliveries
Retrieve metadata about files containing data purchased as part of the subscription.



# Get matches
Source: https://docs.narrative.io/api-reference/subscriptions/get-matches

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /data-shops/subscriptions/matches
Retrieve datasets matching a set of subscription constraints.



# Get schema
Source: https://docs.narrative.io/api-reference/subscriptions/get-schema

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/{subscription_id}/schema
Retrieve schema for deliveries from this subscription.



# Get subscription cost forecasts
Source: https://docs.narrative.io/api-reference/subscriptions/get-subscription-cost-forecasts

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/{subscription_id}/cost-forecasts
Retrieve cost forecasts for a subscription.



# Get subscription matches
Source: https://docs.narrative.io/api-reference/subscriptions/get-subscription-matches

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions/{subscription_id}/matches
Retrieve datasets matching a subscription's constraints.



# Get subscriptions
Source: https://docs.narrative.io/api-reference/subscriptions/get-subscriptions

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /data-shops/subscriptions
Return all subscriptions belonging to the authenticated company.



# Update a subscription
Source: https://docs.narrative.io/api-reference/subscriptions/update-a-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /data-shops/subscriptions/{subscription_id}
Update a subscription

NOTE: If the cadence of the subscription is modified, the frequency filter will
      automatically be adjusted to match the new cadence.



# Upload a file
Source: https://docs.narrative.io/api-reference/uploads/upload-a-file

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /uploads/{upload_path}
Sending a file to Narrative is a two step process. This endpoint allows you to request an `upload_url` which can be used in a `PUT` request to upload the file.

For example, once you have received an `upload_url` you could use it as follows:

```bash
curl -v --upload-file my-file.csv {upload_url}
```

The generated `upload_url` is only valid for 30 minutes after issuance.



# Create a usage event
Source: https://docs.narrative.io/api-reference/usage/create-a-usage-event

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /usage
Create a usage event to charge a customer for their use of a product.



# Archives a view
Source: https://docs.narrative.io/api-reference/views/archives-a-view

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /v2/views/{view_id}
Archives a view



# Create a view
Source: https://docs.narrative.io/api-reference/views/create-a-view

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /v2/views
Any number of properties can be updated on the same request.
If a property is not sent on the payload, the value won't be updated.



# Get a view by id
Source: https://docs.narrative.io/api-reference/views/get-a-view-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/views/{view_id}
Get a view.



# Get accessible views
Source: https://docs.narrative.io/api-reference/views/get-accessible-views

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /v2/views
Get all accessible views.



# Updates a view
Source: https://docs.narrative.io/api-reference/views/updates-a-view

https://docs-cdn.narrative.io/api-reference/main/openapi.json put /v2/views/{view_id}
Updates a view



# Archive a webhook subscription
Source: https://docs.narrative.io/api-reference/webhooks/archive-a-webhook-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /webhooks/{webhook_subscription_id}
Archive the specified webhook subscription.



# Create a webhook subscription
Source: https://docs.narrative.io/api-reference/webhooks/create-a-webhook-subscription

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /webhooks
Create a new webhook subscription for the authenticated company.



# Get a webhook subscription by ID
Source: https://docs.narrative.io/api-reference/webhooks/get-a-webhook-subscription-by-id

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /webhooks/{webhook_subscription_id}
Retrieve a specific webhook subscription.



# List webhook subscriptions
Source: https://docs.narrative.io/api-reference/webhooks/list-webhook-subscriptions

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /webhooks
Get all webhook subscriptions for the authenticated company.



# Get a current user
Source: https://docs.narrative.io/api-reference/whoami/get-a-current-user

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /whoami



# Archive a workflow
Source: https://docs.narrative.io/api-reference/workflows/archive-a-workflow

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /workflows/{workflow_id}
Archive a workflow. This terminates any active schedule and running execution,
then marks the workflow as archived in the database.



# Cancel a specific workflow run
Source: https://docs.narrative.io/api-reference/workflows/cancel-a-specific-workflow-run

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /workflows/{workflow_id}/runs/{run_id}/cancel
Request cancellation of a specific workflow execution identified by its run ID.
This is the recommended method for cancelling workflow executions, as it guarantees
that only the intended execution is cancelled. Unlike cancelling by workflow ID alone,
this avoids a race condition where:
  1. A request to cancel the last execution is sent.
  2. The last execution completes before the server processes the cancellation.
  3. A new workflow execution starts.
  4. The server processes the cancellation request and cancels the new execution instead of the intended one.



# Cancel a workflow execution
Source: https://docs.narrative.io/api-reference/workflows/cancel-a-workflow-execution

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /workflows/{workflow_id}/cancel
Request cancellation of the currently running execution of a workflow.



# Create a workflow
Source: https://docs.narrative.io/api-reference/workflows/create-a-workflow

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /workflows
Create a new workflow from a YAML specification. The specification is parsed and validated
before the workflow is persisted.

Optionally, the workflow can be scheduled and/or triggered immediately upon creation
using the `trigger_immediately` and `schedule_immediately` request fields.



# Get a workflow
Source: https://docs.narrative.io/api-reference/workflows/get-a-workflow

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /workflows/{workflow_id}
Retrieve a workflow by its ID. Only workflows owned by the authenticated company are accessible.



# List workflow runs
Source: https://docs.narrative.io/api-reference/workflows/list-workflow-runs

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /workflows/{workflow_id}/runs
List execution runs for a workflow. Returns a paginated list of runs with their status,
start time, and close time.

Results are ordered by start time descending (most recent first).



# List workflows
Source: https://docs.narrative.io/api-reference/workflows/list-workflows

https://docs-cdn.narrative.io/api-reference/main/openapi.json get /workflows
List all active workflows for the authenticated company.



# Remove a workflow schedule
Source: https://docs.narrative.io/api-reference/workflows/remove-a-workflow-schedule

https://docs-cdn.narrative.io/api-reference/main/openapi.json delete /workflows/{workflow_id}/schedule
Delete the schedule for a workflow. This stops future scheduled executions from being
created but does not cancel any currently running workflow execution.

Returns an error if the workflow does not have an active schedule.



# Schedule a workflow
Source: https://docs.narrative.io/api-reference/workflows/schedule-a-workflow

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /workflows/{workflow_id}/schedule
Create a schedule for a workflow based on the cron expression defined in its specification.
The workflow specification must contain a `schedule` section.

Optionally, the workflow can also be triggered immediately by setting `trigger_immediately` to true.



# Trigger a workflow
Source: https://docs.narrative.io/api-reference/workflows/trigger-a-workflow

https://docs-cdn.narrative.io/api-reference/main/openapi.json post /workflows/{workflow_id}/trigger
Trigger an immediate execution of a workflow. Returns the run ID of the new execution.



# App Invites
Source: https://docs.narrative.io/concepts/apps/app-invites

How invite links enable third parties to connect accounts without Narrative authentication

App invites let applications generate shareable invitation links for third-party account onboarding. An invitee follows the link, connects their account (for example, via OAuth with Pinterest), and a profile is created in the inviting company's Narrative account — no Narrative authentication required.

## Why app invites exist

Onboarding external partners typically requires coordinating account creation, credentials, and permissions. App invites remove this friction by letting the inviter send a single link. The invitee never needs to sign up for Narrative or understand the platform — they simply authorize access to their external account, and the integration is ready.

This is especially valuable when:

* **Scaling partner networks** — onboard dozens or hundreds of partners without manual account setup
* **Reducing time-to-value** — partners can connect in minutes instead of days
* **Minimizing support burden** — self-service onboarding eliminates back-and-forth coordination

## How app invites work

```mermaid theme={null}
sequenceDiagram
    participant Inviter as Inviter
    participant App as Connector App
    participant API as Narrative API
    participant Invitee as Invitee
    participant OAuth as External Platform

    Inviter->>App: Request invite for partner
    App->>API: Create app invite
    API-->>App: Return invite URL with app code
    App-->>Inviter: Share invite URL
    Inviter->>Invitee: Send invite link
    Invitee->>App: Open invite URL
    App->>API: Look up invite by code
    API-->>App: Return invite details
    App->>OAuth: Redirect to OAuth flow
    OAuth-->>App: Authorization granted
    App->>API: Activate invite and create profile
```

1. The **inviter** requests an invite through the connector app
2. The app calls the Narrative API to **create an app invite**, receiving a unique URL containing an [app code](/reference/glossary#app-code)
3. The inviter **shares the URL** with the intended partner
4. The **invitee** opens the link, which directs them to the **app's landing page**
5. The app uses the invite code to **look up the invite** via the Narrative API's public endpoint
6. The app redirects the invitee to the external platform's **OAuth authorization flow**
7. After the invitee **authorizes access**, the app calls the Narrative API to **activate the invite** and create a profile in the inviter's company account

## Invite lifecycle

Each app invite progresses through a series of states:

| State      | Description                                                            | Transition                                                       |
| ---------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `pending`  | Invite has been created but the invitee has not yet completed the flow | Moves to `active` when the invitee completes OAuth authorization |
| `active`   | The invitee has connected their account and a profile has been created | Can be moved to `archived` when no longer needed                 |
| `archived` | The invite is deactivated and no longer usable                         | Terminal state                                                   |

## Reconnect invites

Some connectors support a second type of app invite: **reconnect invites**. Instead of creating a new profile, a reconnect invite allows an invitee to refresh the OAuth token on an existing profile.

This is useful when a partner's token has expired and you need them to re-authorize without disrupting the existing profile configuration, ad account associations, or active connections.

The flow is the same as a standard invite — the invitee receives a link, completes OAuth, and the token is updated in place. The [Pinterest Connector](/reference/connectors/pinterest) currently supports reconnect invites.

## Security considerations

App invites are designed with several security properties:

* **Scoped access** — each invite is tied to a specific app and company. The resulting profile has only the permissions needed for that integration
* **No Narrative authentication for invitees** — invitees interact only with the external platform's OAuth flow, reducing the attack surface
* **Public code lookup** — invite details can be retrieved using the [app code](/reference/glossary#app-code) without authentication, allowing invitees to verify the invite before proceeding. The app code reveals only non-sensitive metadata about the invite

***

## Related content

<CardGroup>
  <Card title="Data Activation" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    How connectors deliver data to external platforms
  </Card>

  <Card title="Connector Reference" icon="plug" href="/reference/connectors/index">
    Available connectors and their configuration
  </Card>

  <Card title="Permissions Reference" icon="lock-keyhole" href="/reference/security/permissions">
    Required API key permissions for app invite operations
  </Card>
</CardGroup>


# Data Flow
Source: https://docs.narrative.io/concepts/architecture/data-flow

How data moves through the Narrative platform during queries and collaboration

Narrative's architecture is designed around a core principle: data stays in place. Rather than moving data to queries, queries move to data. This approach enables collaboration across organizations while maintaining data residency and security requirements.

## The principle: Queries go to data

Traditional data sharing requires copying data between systems. Narrative inverts this model:

| Traditional approach                      | Narrative approach              |
| ----------------------------------------- | ------------------------------- |
| Copy data to a shared location            | Leave data in place             |
| Queries run against copies                | Queries run against source data |
| Data freshness depends on sync frequency  | Data is always current          |
| Multiple copies increase security surface | Single source reduces exposure  |

When you query data through Narrative, the [control plane](/concepts/primitives/control-plane) sends query instructions to your [data plane](/concepts/primitives/data-planes). The data plane executes the query locally and returns only the results.

## What flows where

Understanding what crosses boundaries helps clarify the architecture:

### From user to control plane

* NQL query text
* Authentication credentials
* Request metadata

### From control plane to data plane

* Compiled SQL (transpiled for the target database)
* Job metadata and parameters
* Execution instructions

### From data plane to control plane

* Status updates and completion signals
* Error messages and diagnostics
* Data samples (when requested for viewing results)

### What never leaves the data plane

* Raw source data
* Intermediate query results
* Materialized view contents (stored locally)

## Query data flow

All queries—including interactive queries run from the Query Editor—follow the same pattern as materialized views. Results are stored as a dataset in the data plane, not returned directly to the user.

```
User                Control Plane              Data Plane
  │                      │                         │
  │──── NQL Query ──────>│                         │
  │                      │                         │
  │                      │──── Compiled SQL ──────>│
  │                      │                         │
  │                      │                    ┌────┴────┐
  │                      │                    │ Execute │
  │                      │                    │ & Store │
  │                      │                    │(Dataset)│
  │                      │                    └────┬────┘
  │                      │                         │
  │                      │<──── Status ───────────│
  │                      │                         │
```

Interactive queries are implemented as [materialized views](/concepts/nql/materialized-views) with two automatic constraints:

* A **24-hour retention policy** that expires the results automatically
* A **row limit** that caps the result size

This architecture ensures that even ad-hoc queries never move full result sets through the control plane—data stays in your data plane.

## Materialized view data flow

[Materialized views](/concepts/nql/materialized-views) follow a different pattern. Results stay in the data plane:

```
Control Plane              Data Plane
     │                         │
     │──── Refresh Job ───────>│
     │                         │
     │                    ┌────┴────┐
     │                    │ Execute │
     │                    │  Query  │
     │                    └────┬────┘
     │                         │
     │                    ┌────┴────┐
     │                    │  Store  │
     │                    │ Results │
     │                    └────┬────┘
     │                         │
     │<──── Status ────────────│
     │                         │
```

The control plane receives only a completion status—never the actual data. Results are stored as a dataset within your data plane.

## How you view query results

Since query results stay in the data plane, how do you actually see them? Through **data sampling**.

```
User                Control Plane              Data Plane
  │                      │                         │
  │── View Results ─────>│                         │
  │                      │                         │
  │                      │── Sample Request ──────>│
  │                      │                         │
  │                      │                    ┌────┴────┐
  │                      │                    │  Read   │
  │                      │                    │ 1000    │
  │                      │                    │  Rows   │
  │                      │                    └────┬────┘
  │                      │                         │
  │                      │<── Sample Data ────────│
  │                      │                         │
  │<── Display Sample ───│                         │
  │                      │                         │
```

When you click to view query results in the UI or request results via the API:

1. **Sample job** — The control plane requests a sample from the dataset in your data plane
2. **Row retrieval** — Up to 1,000 rows are read from the stored results
3. **Sample storage** — The sample is stored in the control plane for quick access
4. **Display** — The sample appears in the UI or API response

This sample is a preview of your results, not the complete dataset. For full access to query results, use [data exports](/guides/nql/creating-materialized-views) or access the underlying dataset directly.

<Tip>
  Samples are one of the few cases where actual data leaves your data plane. For governance implications, see [Sample Data](/concepts/architecture/sample-data). To clear samples programmatically, see [Managing Datasets](/guides/sdk/managing-datasets#sampling-data).
</Tip>

## Cross-data-plane queries

When a query references data in multiple data planes, the control plane coordinates execution:

1. **Query decomposition** — The control plane breaks the query into subqueries, one per data plane
2. **Parallel execution** — Subqueries execute independently in each data plane
3. **Result coordination** — Results are combined as needed, with the control plane coordinating data movement between planes

The specific data movement depends on the query. For joins across data planes, some data may need to move between planes for the join to complete. The control plane optimizes this movement to minimize data transfer.

## Data collaboration flow

When you share data with a partner organization, they query your data through the same architecture:

1. Partner submits NQL query referencing your dataset
2. Control plane verifies their permissions against your [access rules](/concepts/primitives/access-rules)
3. Compiled query is sent to your data plane
4. Your data plane executes and returns results
5. Partner receives results through the control plane

Your raw data never leaves your data plane—only the query results authorized by your access rules.

## Data egress via connectors

While the architecture above keeps data within data planes, you often need to deliver data to external destinations for activation. [Connectors](/reference/connectors/index) enable delivery to platforms like advertising DSPs, cloud storage, and data warehouses.

### How connectors work

Connectors are pre-built integrations that handle authentication, data formatting, and delivery to external platforms. The data flow for connector-based delivery:

```
Data Plane                  Connector                External Destination
    │                          │                            │
    │                          │                            │
    │──── Dataset ────────────>│                            │
    │                          │                            │
    │                     ┌────┴────┐                       │
    │                     │ Format  │                       │
    │                     │  Data   │                       │
    │                     └────┬────┘                       │
    │                          │                            │
    │                          │──── Formatted Data ───────>│
    │                          │                            │
    │                          │<──── Confirmation ─────────│
    │                          │                            │
```

Unlike queries (where only results leave the data plane), connector delivery sends the full dataset to the external destination. This is intentional—connectors exist specifically to move data outside the platform for activation.

### Setting up delivery

Connector-based delivery involves two configuration levels:

| Level                 | What it configures                        | Example                               |
| --------------------- | ----------------------------------------- | ------------------------------------- |
| **Profile**           | Account-level authentication and settings | AWS credentials, ad account ID        |
| **Delivery settings** | Job-specific parameters                   | Destination folder, delivery schedule |

Once a connector is configured, delivery options appear automatically when creating materialized views or working with datasets.

<Note>
  Each connector requires specific identifier types to match users on the destination platform. See [Connectors Reference](/reference/connectors/index) for requirements by platform.
</Note>

***

## Related content

<CardGroup>
  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    The complete query lifecycle from submission to results
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and queries execute
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    How queries are compiled and coordinated
  </Card>

  <Card title="Connectors" icon="plug" href="/reference/connectors/index">
    Available destinations for data delivery
  </Card>
</CardGroup>


# Query Processing
Source: https://docs.narrative.io/concepts/architecture/query-processing

How NQL queries are compiled, dispatched, and executed across the platform

When you run an NQL query, it passes through several stages before returning results. The [control plane](/concepts/primitives/control-plane) compiles and coordinates the query, while [data plane](/concepts/primitives/data-planes) operators execute it against your actual data. This separation ensures your data stays in your infrastructure while enabling cross-organization collaboration.

## The query lifecycle

A query moves through six stages from submission to results:

```
┌─────────────────────────────────────────────────────────────────┐
│                        Control Plane                             │
│                                                                  │
│  ┌──────────┐   ┌──────────────┐   ┌────────────────────────┐   │
│  │  Parse   │ → │  Transpile   │ → │      Job Queue         │   │
│  │   NQL    │   │  to SQL      │   │   (awaiting pickup)    │   │
│  └──────────┘   └──────────────┘   └───────────┬────────────┘   │
│                                                 │                │
└─────────────────────────────────────────────────┼────────────────┘
                                                  │ poll
                                                  ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Data Plane (Your Infrastructure)              │
│                                                                  │
│  ┌──────────┐   ┌──────────────┐   ┌────────────────────────┐   │
│  │ Operator │ → │   Execute    │ → │       Results          │   │
│  │  (polls) │   │  Native SQL  │   │                        │   │
│  └──────────┘   └──────────────┘   └────────────────────────┘   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

## Stage 1: Query submission

You submit an NQL query to the control plane through:

* The **Query Editor** in the Narrative I/O interface
* The **NQL API** for programmatic access
* **Materialized view definitions** that run on a schedule

The control plane receives your query and begins processing.

## Stage 2: Compilation and transpilation

The control plane transforms your NQL into executable SQL through several steps:

**Parsing.** The NQL syntax is parsed and validated. Malformed queries fail here with syntax errors.

**Resolution.** Dataset references are resolved to their actual locations. [Rosetta Stone](/concepts/rosetta-stone/overview) mappings translate standardized attribute names to physical column names.

**Permission verification.** Your [access rules](/concepts/primitives/access-rules) are checked for every dataset and field in the query. If you lack permission to access any referenced data, the query fails before execution.

**Optimization.** The query planner analyzes the query structure to determine an efficient execution strategy. For large dataset scans, this may include [chunking](/concepts/nql/chunking)—splitting the query into time-based segments for improved stability and cost efficiency.

**Transpilation.** The NQL is converted to native SQL for your target data plane's database engine. A Snowflake data plane receives Snowflake SQL; a Spark data plane receives Spark SQL. See [NQL Design Philosophy](/concepts/nql/design-philosophy) for details on transpilation.

At the end of this stage, the control plane has a compiled query ready for execution.

## Stage 3: Job enqueueing

The control plane creates a job containing the compiled SQL and adds it to the job queue. The job queue is a coordination mechanism between the control plane and data planes—it holds work waiting for execution.

Query execution is one type of job. The job queue also handles other operations like dataset management and system tasks. See [Job Types](/reference/architecture/job-types) for a complete list.

## Stage 4: Operator polling

Each data plane runs an operator component that bridges the control plane and your data infrastructure. The operator:

* **Polls** the control plane's job queue for work targeting its data plane
* **Authenticates** to ensure only authorized operators can pick up jobs
* **Claims** jobs for execution

This is a **pull-based architecture**. The control plane never connects directly to your database—it only makes jobs available. Your operator reaches out to claim them. This design means:

* Your data plane can be behind a firewall
* The control plane never needs credentials to your database
* You control when and how jobs are executed

## Stage 5: Query execution

Once the operator picks up a job, it executes the compiled SQL against your data plane's native query engine—Snowflake, Spark, or another supported system.

The query runs entirely within your infrastructure. Your data never leaves the data plane; only query instructions enter and results exit.

## Stage 6: Result handling

After execution completes, results are stored in the data plane—never returned directly through the control plane:

**Interactive queries.** Results are stored as a new dataset in your data plane. Interactive queries are implemented as materialized views with a **24-hour retention policy** and an **automatic row limit**. The control plane receives only a completion status—not the data itself. You view results through [data sampling](/concepts/architecture/data-flow#how-you-view-query-results), which retrieves a preview of up to 1,000 rows.

**Materialized views.** Results are written to a dataset in your data plane. The control plane is notified of completion but doesn't see the actual data. Materialized views can have custom retention policies and refresh schedules.

**Data exports.** Results are written to a designated destination within your data plane.

Throughout execution, status updates flow back to the control plane so you can monitor progress. See [Data Flow](/concepts/architecture/data-flow) for details on how data moves through the platform.

## Why this architecture?

The separation between control plane and data plane is deliberate:

**Security.** Your data stays in your infrastructure. The control plane coordinates work without accessing raw data. This enables collaboration while maintaining data residency requirements.

**Flexibility.** Each organization can use their preferred database system. The control plane transpiles NQL to whatever dialect your data plane requires.

**Scalability.** Operators scale with your data plane's capacity. You can run multiple operators for high-throughput workloads or share capacity across fewer operators.

**Reliability.** The pull-based job queue provides resilience. If an operator is temporarily unavailable, jobs wait in the queue until capacity returns.

***

## Related content

<CardGroup>
  <Card title="NQL Design Philosophy" icon="compass-drafting" href="/concepts/nql/design-philosophy">
    Why NQL is an interpreted language with transpilation
  </Card>

  <Card title="Chunking" icon="puzzle-piece" href="/concepts/nql/chunking">
    How large queries are split into time-based segments
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    The orchestration layer that compiles queries and coordinates jobs
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and queries execute
  </Card>
</CardGroup>


# Sample Data
Source: https://docs.narrative.io/concepts/architecture/sample-data

Understand how data samples cross from data planes to the control plane, and the governance implications

Sample data is one of the rare cases where actual data leaves your [data plane](/concepts/primitives/data-planes) and is stored in the [control plane](/concepts/primitives/control-plane). For organizations with data residency requirements or strict governance policies, this distinction matters.

## What are data samples?

A data sample is a preview of up to 1,000 rows from a dataset. Samples enable you to view query results, validate data quality, and inspect datasets without downloading full result sets.

When you click to view query results in the UI or retrieve results via the API, the platform creates a sample from your dataset and stores it in the control plane for display.

## Why samples cross the boundary

Narrative's architecture is built around a core principle: [data stays in place](/concepts/architecture/data-flow). Queries are sent to your data plane, executed locally, and results remain within your infrastructure.

Samples are the exception. To display query results in the UI or return them via the API, actual row-level data must be transmitted from your data plane to the control plane and stored there.

<Warning>
  Unlike full query results (which stay in your data plane), sample data is transmitted to and persisted in Narrative's control plane infrastructure. If your datasets contain sensitive information, that information will be present in any samples you create.
</Warning>

This is one of only two scenarios where data egresses from your data plane:

| Scenario               | Description                       | Destination                         |
| ---------------------- | --------------------------------- | ----------------------------------- |
| **Data samples**       | Preview rows for viewing results  | Control plane database              |
| **Connector delivery** | Intentional export for activation | External platforms (S3, DSPs, etc.) |

## What data is included in samples

When a sample is created:

* **Row limit** — Up to 1,000 rows are retrieved
* **All columns** — Every column in the dataset is included
* **Rosetta Stone attributes** — If the dataset has [Rosetta Stone mappings](/concepts/rosetta-stone/how-it-works), normalized attribute values are included alongside the raw columns
* **No transformation** — Raw data is not masked, hashed, or modified (Rosetta Stone attributes reflect their normalized transformations)
* **Point-in-time snapshot** — The sample reflects the dataset at the moment of retrieval

## Sample lifecycle

Samples are created on-demand, not automatically.

### Creation

A sample is created when:

* You click to view query results in the Query Editor
* You call `requestDatasetSample()` via the SDK
* You request results through the API

Each sample creation triggers a [sample job](/reference/architecture/job-types#sample-jobs) that retrieves rows from your data plane and transmits them to the control plane.

### Storage

Once created, samples persist in the control plane database until explicitly deleted. They do not expire automatically.

### Deletion

Samples can be removed:

* Programmatically via `deleteDatasetSample()` in the SDK
* Through the UI when clearing query results
* When the underlying dataset is deleted

## Governance considerations

### Data residency

Sample data resides in Narrative's control plane infrastructure, which may be in a different geographic region than your data plane. If you operate a customer-hosted data plane specifically for data residency requirements, be aware that samples bypass this isolation.

### Sensitive data exposure

If your dataset contains PII or other sensitive information, that data will be present in samples stored in the control plane. Consider:

* Whether viewing results is necessary for your workflow
* Clearing samples promptly after inspection
* Using direct data plane access for sensitive datasets

### Retention practices

Since samples persist until deleted, establish practices for clearing them when no longer needed. This reduces the footprint of sensitive data in the control plane.

<Tip>
  For datasets containing sensitive information, clear samples after inspection using the SDK's `deleteDatasetSample()` method. See [Managing Datasets](/guides/sdk/managing-datasets#sampling-data) for implementation details.
</Tip>

### Access control

Sample access follows the same permission model as the underlying dataset. Users who can query a dataset can view its samples.

## Alternatives to sampling

If you need to avoid storing data in the control plane:

* **Direct data plane access** — Query and process data within your data plane infrastructure
* **Cloud storage export** — Use [connectors](/reference/connectors/index) to deliver data to your own S3 bucket or data warehouse
* **Materialized views** — Access the underlying dataset directly rather than viewing samples

These approaches keep data within infrastructure you control, at the cost of not being able to preview results in the Narrative UI.

***

## Related content

<CardGroup>
  <Card title="Data Flow" icon="arrows-split-up-and-left" href="/concepts/architecture/data-flow">
    How data moves through the platform
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    What the control plane manages and stores
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    How Narrative protects your data
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets#sampling-data">
    SDK methods for sample management
  </Card>
</CardGroup>


# Security Model
Source: https://docs.narrative.io/concepts/architecture/security-model

Understand how Narrative protects your data through architecture, access controls, and operational practices

Narrative's security model is built on the principle that your data remains under your control. The platform architecture, access controls, and operational practices work together to protect sensitive information while enabling collaboration.

## Separation of control and data planes

Narrative's most fundamental security design is the separation between the [control plane](/concepts/primitives/control-plane) and [data planes](/concepts/primitives/data-planes).

**The control plane** manages metadata, permissions, and orchestration. It knows *about* your data—schema, access rules, query patterns—but never sees the data itself.

**Data planes** store and process actual data. They can be hosted by Narrative or within your own infrastructure, giving you control over where sensitive information resides.

This separation provides several security benefits:

* **Data residency control** — Keep data within specific geographic or infrastructure boundaries
* **Reduced attack surface** — The control plane has no access to raw data, limiting exposure from any single point of compromise
* **Audit clarity** — Data access and metadata operations are logged separately, making compliance auditing straightforward

## Access control model

Narrative uses a multi-layered access control system:

### Organization-level controls

Organizations are the top-level security boundary. Each organization has:

* **Member management** — Control who belongs to your organization
* **Role assignments** — Assign admin or standard roles to members
* **Authentication policies** — Enforce SSO, MFA requirements, or other authentication standards through your identity provider

### Dataset-level controls

When you register or share data, you define who can access it:

* **Visibility rules** — Control which organizations can discover your datasets
* **[Access rules](/concepts/primitives/access-rules)** — Define who can query your data, what they can access, and pricing terms
* **Field-level permissions** — Restrict access to sensitive columns within a dataset

### Query-time enforcement

Access controls are enforced at query execution, not just at the UI level:

* Every query is validated against current permissions before execution
* Changes to access rules take effect immediately
* Queries that exceed granted permissions fail with clear error messages

## Data isolation

Data from different organizations never mingles unintentionally:

**Storage isolation** — Datasets are stored with clear ownership boundaries. Customer-hosted data planes provide physical isolation when required.

**Query isolation** — When you query across organizations (in data collaboration scenarios), the data plane enforces that you only see data you're explicitly permitted to access.

**Result isolation** — Query results are delivered only to the requesting user. Full result sets are not stored beyond the query session, though [data samples](/concepts/architecture/sample-data) (up to 1,000 rows for viewing) are stored in the control plane until explicitly cleared.

## Encryption

Narrative encrypts data at multiple levels:

| Layer              | Protection                                                              |
| ------------------ | ----------------------------------------------------------------------- |
| **In transit**     | All API and data transfer uses TLS 1.2+ encryption                      |
| **At rest**        | Data planes encrypt stored data using AES-256                           |
| **Key management** | Encryption keys are managed separately from data, with regular rotation |

Customer-hosted data planes can integrate with your organization's key management infrastructure for additional control.

## Authentication

Narrative supports multiple authentication methods:

* **Email magic links** — Passwordless authentication for quick access
* **Single Sign-On (SSO)** — SAML 2.0 integration with enterprise identity providers
* **API keys** — Long-lived tokens for programmatic access with scoped permissions

Enterprise organizations can enforce SSO-only authentication and inherit MFA requirements from their identity provider.

<Note>
  For SSO configuration details, see [SSO Configuration](/account-settings/sso). To learn how API key permissions work, see [API Key Permissions](/concepts/security/api-key-permissions) and the [Permissions Reference](/reference/security/permissions).
</Note>

## Audit and compliance

Narrative maintains comprehensive audit logs:

* **Authentication events** — Login attempts, SSO assertions, API key usage
* **Data access** — Queries executed, datasets accessed, results delivered
* **Administrative actions** — Permission changes, member management, configuration updates

These logs support compliance requirements including SOC 2 Type II. Contact your account representative for compliance documentation.

## Operational security

Beyond architectural controls, Narrative maintains operational security practices:

* **Vulnerability management** — Regular security assessments and penetration testing
* **Incident response** — Documented procedures for security event handling
* **Employee access** — Strict access controls and background checks for personnel with system access
* **Change management** — All production changes go through review and approval processes

## Shared responsibility

Security is a shared responsibility between Narrative and customers:

**Narrative is responsible for:**

* Platform infrastructure security
* Control plane availability and integrity
* Narrative-hosted data plane security
* Security updates and vulnerability remediation

**Customers are responsible for:**

* Managing organization membership and roles
* Configuring appropriate access controls for datasets
* Securing API keys and credentials
* Customer-hosted data plane security (when applicable)

## Related content

<CardGroup>
  <Card title="Access Rules" icon="key" href="/concepts/primitives/access-rules">
    Control who can access your data and at what price
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understand where your data lives and how data planes provide isolation
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    Learn how Narrative orchestrates operations without accessing your data
  </Card>

  <Card title="Single Sign-On (SSO)" icon="shield-keyhole" href="/concepts/security/sso">
    Configure enterprise authentication for your organization
  </Card>

  <Card title="API Key Permissions" icon="lock-keyhole" href="/concepts/security/api-key-permissions">
    How scoped permissions control what API keys can do
  </Card>
</CardGroup>


# CCPA
Source: https://docs.narrative.io/concepts/compliance/ccpa

Understanding the California Consumer Privacy Act and California Privacy Rights Act

The California Consumer Privacy Act (CCPA), as amended by the California Privacy Rights Act (CPRA), is the most comprehensive state-level privacy law in the United States. Effective since January 2020 and significantly strengthened in 2023, CCPA gives California residents substantial rights over their personal information and imposes obligations on businesses that collect or process that data.

## Who CCPA applies to

CCPA applies to for-profit businesses that collect California consumers' personal information and meet any of the following thresholds:

| Threshold             | Criteria                                                                                                                 |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Annual revenue**    | Gross annual revenue exceeding \$25 million                                                                              |
| **Data volume**       | Buys, sells, or shares the personal information of 100,000 or more California consumers, households, or devices annually |
| **Revenue from data** | Derives 50% or more of annual revenue from selling or sharing California consumers' personal information                 |

<Info>
  CCPA applies based on where the consumer resides, not where your business is located. A company in New York with California customers must comply if it meets the thresholds.
</Info>

### Who is a "consumer"?

Under CCPA, a "consumer" is any California resident—defined as a natural person who is in California for other than a temporary or transitory purpose, or who is domiciled in California but currently outside the state temporarily.

***

## Key definitions

CCPA uses specific terminology that differs from GDPR and other privacy frameworks:

| Term                     | Definition                                                                                                                                                                                                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Personal information** | Information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked with a particular consumer or household. Broader than many definitions, including IP addresses, browsing history, and inferences. |
| **Sale**                 | Selling, renting, releasing, disclosing, disseminating, making available, or transferring personal information for monetary or other valuable consideration.                                                                                                        |
| **Share**                | Sharing personal information for cross-context behavioral advertising purposes, whether or not for monetary consideration. Added by CPRA.                                                                                                                           |
| **Service provider**     | An entity that processes personal information on behalf of a business pursuant to a written contract that prohibits retaining, using, or disclosing the information for purposes other than performing the services.                                                |
| **Contractor**           | Similar to a service provider but with additional restrictions. Added by CPRA to close perceived loopholes.                                                                                                                                                         |
| **Business**             | A for-profit entity that collects consumers' personal information and meets CCPA thresholds.                                                                                                                                                                        |

<Tip>
  The "sale" definition is broader than cash transactions. Sharing data with an advertising partner in exchange for services can constitute a sale under CCPA.
</Tip>

***

## Consumer rights

CCPA grants California consumers specific rights over their personal information:

### Right to know

Consumers can request that a business disclose:

* The categories of personal information collected
* The sources from which information was collected
* The business or commercial purpose for collection or sale
* The categories of third parties with whom information is shared
* The specific pieces of personal information collected about them

### Right to delete

Consumers can request deletion of their personal information, with limited exceptions for legal obligations, completing transactions, security purposes, and certain internal uses.

### Right to correct

Added by CPRA, consumers can request correction of inaccurate personal information that a business maintains about them.

### Right to opt-out of sale/sharing

Consumers can direct businesses not to sell or share their personal information. Businesses must provide a clear "Do Not Sell or Share My Personal Information" link on their website.

### Right to limit use of sensitive personal information

Added by CPRA, consumers can limit a business's use of sensitive personal information (such as Social Security numbers, financial accounts, precise geolocation, or racial/ethnic origin) to purposes necessary for providing the requested goods or services.

### Right to non-discrimination

Businesses cannot discriminate against consumers who exercise their CCPA rights through:

* Denying goods or services
* Charging different prices
* Providing different quality of goods or services
* Suggesting that exercising rights will result in any of the above

***

## Business obligations

Businesses subject to CCPA must fulfill several requirements:

### Privacy notices

Provide a privacy policy that discloses:

* Categories of personal information collected in the past 12 months
* Purposes for each category
* Categories of sources and third parties
* Consumer rights and how to exercise them
* Whether information is sold or shared, with opt-out instructions

### Responding to consumer requests

* Provide at least two methods for submitting requests (for online businesses, this must include a web form)
* Verify the identity of consumers making requests
* Respond within 45 days (extendable by an additional 45 days with notice)
* Provide information free of charge for up to two requests per year

### Service provider contracts

Maintain written contracts with service providers and contractors that:

* Specify the business purposes for processing
* Prohibit selling or sharing the information
* Require the same level of privacy protection as required by CCPA
* Grant the business rights to monitor compliance

### Training

Ensure that personnel handling consumer inquiries about privacy practices are informed of CCPA requirements.

### Record keeping

Businesses processing personal information of 10 million or more consumers must maintain records of requests and responses for 24 months.

***

## Sensitive personal information

CPRA created a special category of "sensitive personal information" with additional protections:

* Social Security, driver's license, state ID, or passport numbers
* Account log-in credentials (username with password or security questions)
* Financial account, debit card, or credit card numbers with access codes
* Precise geolocation
* Racial or ethnic origin, religious or philosophical beliefs, union membership
* Contents of mail, email, or text messages (unless the business is the intended recipient)
* Genetic data
* Biometric information for identification purposes
* Health information
* Sex life or sexual orientation information

Consumers can limit the use of sensitive personal information to what is necessary to perform the services or provide the goods reasonably expected.

***

## Enforcement and penalties

CCPA is enforced by the California Privacy Protection Agency (CPPA) and the California Attorney General:

| Violation type               | Penalty                                                                                                             |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Intentional violations**   | Up to \$7,500 per violation                                                                                         |
| **Unintentional violations** | Up to \$2,500 per violation                                                                                         |
| **Data breaches**            | Private right of action with statutory damages of $100-$750 per consumer per incident, or actual damages if greater |

<Warning>
  Penalties are assessed per violation, meaning each affected consumer or each instance of non-compliance can be a separate violation. This can result in substantial aggregate penalties for widespread issues.
</Warning>

Unlike GDPR, CCPA provides a limited private right of action allowing consumers to sue directly—but only for data breaches involving non-encrypted or non-redacted personal information due to a business's failure to maintain reasonable security procedures.

***

## CCPA vs. GDPR

While both laws protect consumer privacy, they differ in significant ways:

| Aspect                       | CCPA                                                           | GDPR                                                  |
| ---------------------------- | -------------------------------------------------------------- | ----------------------------------------------------- |
| **Scope**                    | For-profit businesses meeting thresholds, California consumers | Any organization processing EU residents' data        |
| **Approach**                 | Opt-out model (consumers must request restrictions)            | Opt-in model (requires legal basis before processing) |
| **Personal data definition** | Includes household and device data                             | Focuses on identified individuals                     |
| **Enforcement**              | State agency and Attorney General                              | National Data Protection Authorities                  |
| **Private right of action**  | Limited to data breaches                                       | Generally not available                               |
| **Penalties**                | Per-violation penalties                                        | Percentage of global revenue                          |

***

## Related content

<CardGroup>
  <Card title="Managing Data Subject Requests" icon="user-shield" href="/guides/compliance/managing-dsrs">
    Submit opt-out and deletion requests in Narrative
  </Card>

  <Card title="GDPR" icon="shield-check" href="/concepts/compliance/gdpr">
    Compare with the EU's privacy regulation
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Privacy-preserving data collaboration techniques
  </Card>

  <Card title="Security Model" icon="shield-halved" href="/concepts/architecture/security-model">
    How Narrative protects data throughout the platform
  </Card>
</CardGroup>


# GDPR
Source: https://docs.narrative.io/concepts/compliance/gdpr

Understanding the General Data Protection Regulation and its requirements for personal data processing

The General Data Protection Regulation (GDPR) is a comprehensive privacy law that governs how organizations collect, process, and store personal data of individuals in the European Union. Since its enforcement in May 2018, GDPR has become the global benchmark for data protection legislation and influences privacy regulations worldwide.

## Who GDPR applies to

GDPR has broad territorial scope. It applies to your organization if:

### Establishment in the EU

You have any establishment (office, subsidiary, or branch) in the EU, regardless of whether the data processing takes place there.

### Offering goods or services to EU residents

You offer goods or services to individuals in the EU, even if your organization is based elsewhere. This includes free services if you're targeting EU users.

### Monitoring behavior of EU residents

You monitor the behavior of individuals in the EU—such as tracking website visitors or profiling users for advertising purposes.

<Info>
  GDPR applies based on where the individuals are located, not their citizenship. An American tourist in Paris has GDPR protections while in the EU.
</Info>

***

## Key definitions

Understanding GDPR requires familiarity with its specific terminology:

| Term                | Definition                                                                                                                                                           |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Personal data**   | Any information relating to an identified or identifiable natural person. This includes names, email addresses, IP addresses, location data, and online identifiers. |
| **Data subject**    | The individual whose personal data is being processed.                                                                                                               |
| **Data controller** | The entity that determines the purposes and means of processing personal data—essentially, the decision-maker about what data to collect and why.                    |
| **Data processor**  | An entity that processes personal data on behalf of a controller. Processors act on instructions from controllers.                                                   |
| **Processing**      | Any operation performed on personal data, including collection, storage, use, transmission, and deletion.                                                            |

<Tip>
  Organizations often act as both controllers and processors depending on the context. When you decide what customer data to collect, you're a controller. When you process data on behalf of a client according to their instructions, you're a processor.
</Tip>

***

## Core principles

GDPR establishes seven principles that govern all personal data processing:

### Lawfulness, fairness, and transparency

Processing must have a valid legal basis, must not be deceptive or harmful, and individuals must be informed about how their data is used.

### Purpose limitation

Data must be collected for specified, explicit, and legitimate purposes. You cannot repurpose data for unrelated uses without additional consent or legal basis.

### Data minimization

Only collect and process data that is necessary for your stated purposes. Avoid collecting data "just in case" it might be useful later.

### Accuracy

Personal data must be accurate and kept up to date. Inaccurate data should be corrected or deleted without delay.

### Storage limitation

Data should be kept only as long as necessary for the purposes it was collected. Establish retention periods and delete data when no longer needed.

### Integrity and confidentiality

Data must be processed securely, with appropriate technical and organizational measures to protect against unauthorized access, loss, or damage.

### Accountability

Organizations must demonstrate compliance with all principles. This means maintaining records, conducting assessments, and being able to prove you're following the rules.

***

## Data subject rights

GDPR grants individuals significant control over their personal data:

### Right to be informed

Individuals must receive clear information about who is processing their data, why, and how—typically through privacy notices.

### Right of access

Individuals can request a copy of their personal data and information about how it's being processed.

### Right to rectification

Individuals can request correction of inaccurate or incomplete personal data.

### Right to erasure ("right to be forgotten")

Individuals can request deletion of their personal data in certain circumstances, such as when the data is no longer necessary or when they withdraw consent.

### Right to restrict processing

Individuals can request that processing be limited while disputes about accuracy or lawfulness are resolved.

### Right to data portability

Individuals can receive their data in a structured, machine-readable format and transfer it to another controller.

### Right to object

Individuals can object to processing based on legitimate interests or for direct marketing purposes.

### Rights related to automated decision-making

Individuals can request human review of significant decisions made solely by automated processing, including profiling.

***

## Legal bases for processing

GDPR requires a valid legal basis for any processing of personal data. The six legal bases are:

| Legal basis              | When to use                                                                                                                                    |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Consent**              | The individual has given clear, informed consent for a specific purpose. Must be freely given, specific, informed, and unambiguous.            |
| **Contract**             | Processing is necessary to fulfill a contract with the individual or to take steps at their request before entering a contract.                |
| **Legal obligation**     | Processing is necessary to comply with a legal requirement.                                                                                    |
| **Vital interests**      | Processing is necessary to protect someone's life. Rarely applicable in commercial contexts.                                                   |
| **Public task**          | Processing is necessary to perform an official function or task in the public interest. Primarily for public authorities.                      |
| **Legitimate interests** | Processing is necessary for your legitimate interests, provided these aren't overridden by the individual's rights. Requires a balancing test. |

<Warning>
  Consent is not always the best legal basis. If you cannot genuinely offer a choice or would continue processing regardless of consent, consider whether contract or legitimate interests is more appropriate.
</Warning>

***

## Compliance requirements

Organizations subject to GDPR must implement various measures:

### Privacy notices

Provide clear, accessible information about your data processing activities at the point of data collection.

### Records of processing

Maintain detailed records of all processing activities, including purposes, data categories, recipients, and retention periods.

### Data Protection Impact Assessments (DPIAs)

Conduct assessments for high-risk processing activities, such as large-scale profiling or processing sensitive data.

### Data Protection Officer (DPO)

Appoint a DPO if you're a public authority, conduct large-scale monitoring, or process sensitive data at scale.

### Data Processing Agreements (DPAs)

Establish written contracts with any processors that handle personal data on your behalf, specifying their obligations.

### Breach notification

Report personal data breaches to supervisory authorities within 72 hours and notify affected individuals when there's high risk to their rights.

### Cross-border transfers

Implement appropriate safeguards when transferring personal data outside the EU, such as Standard Contractual Clauses (SCCs) or binding corporate rules.

***

## Enforcement and penalties

GDPR is enforced by Data Protection Authorities (DPAs) in each EU member state. Penalties for non-compliance can be significant:

* **Lower tier**: Up to 10 million EUR or 2% of global annual turnover for violations of technical requirements
* **Upper tier**: Up to 20 million EUR or 4% of global annual turnover for violations of core principles and data subject rights

Beyond fines, DPAs can issue warnings, reprimands, and orders to cease processing—which can effectively shut down data-dependent business operations.

***

## Related content

<CardGroup>
  <Card title="Managing Data Subject Requests" icon="user-shield" href="/guides/compliance/managing-dsrs">
    Submit opt-out and deletion requests in Narrative
  </Card>

  <Card title="CCPA" icon="scale-balanced" href="/concepts/compliance/ccpa">
    Compare GDPR with California's privacy regulation
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    A GDPR-recognized data protection technique
  </Card>

  <Card title="Security Model" icon="shield-halved" href="/concepts/architecture/security-model">
    How Narrative protects data throughout the platform
  </Card>
</CardGroup>


# Sub-processors
Source: https://docs.narrative.io/concepts/compliance/sub-processors

Third-party service providers that process data on behalf of Narrative

Narrative engages certain sub-processors to assist with delivering our services. These third-party providers help process both Narrative-owned data and customer-owned data in accordance with our data processing agreements and applicable privacy regulations.

<Info>
  **Stay informed of changes**

  [Subscribe to receive notifications](https://share.hsforms.com/1YeBBL96xThWvrI7iE4ZAiA38eec) when we update our list of sub-processors.
</Info>

## Current sub-processors

| Name                      | Data processed                                 | Purpose                          | Location      |
| ------------------------- | ---------------------------------------------- | -------------------------------- | ------------- |
| Amazon Web Services, Inc. | Customer-owned data, Narrative-owned user data | Cloud infrastructure and hosting | United States |
| Stytch, Inc.              | Narrative-owned user data                      | User authentication and OAuth    | United States |

## Related

<CardGroup>
  <Card title="GDPR Compliance" icon="shield-halved" href="/concepts/compliance/gdpr">
    Learn about our GDPR compliance measures
  </Card>

  <Card title="Security Model" icon="lock" href="/concepts/architecture/security-model">
    Understand our security architecture
  </Card>
</CardGroup>


# Audience Studio
Source: https://docs.narrative.io/concepts/data-activation/audience-studio

Build, filter, and deliver audiences to advertising platforms using a guided builder

Audience Studio is the platform's visual audience builder. It provides a guided workflow for selecting a data source, applying filters to segment your audience, choosing connector destinations, and delivering the resulting audience — all without writing NQL by hand.

## How Audience Studio works

Audience Studio builds audiences by combining three things:

1. **A source dataset** — The data you want to segment. This can be your own first-party data (CRM records, transaction data, customer attributes) or third-party data accessed through access rules.
2. **Filters** — Conditions that narrow the source data to the audience you want. Filters use your dataset's Rosetta Stone attribute mappings to provide structured, type-aware controls.
3. **Connectors** — Destinations where the audience will be delivered. Each connector handles format translation, identity matching, and API integration with the destination platform automatically.

The builder generates an NQL query behind the scenes, executes it to create an audience dataset, and sets up connector deliveries — turning what would be a multi-step manual process into a single workflow.

## Builder steps

### 1. Source selection

Choose the dataset that contains your audience data. The dataset must have [Rosetta Stone attribute mappings](/guides/rosetta-stone/mapping-schemas) so Audience Studio can understand which fields are available for filtering and which identifiers are available for delivery.

Only datasets on the currently selected [data plane](/concepts/primitives/data-planes) are shown.

### 2. Connector selection

Choose one or more destination platforms to deliver your audience to. Available connectors include The Trade Desk, Meta, TikTok, Google DV360, Pinterest, Yahoo DSP, Magnite, PubMatic, and Amazon S3.

Audience Studio checks **connector eligibility** automatically — it verifies that your source dataset contains the identifier types required by each connector's collaboration policy. Ineligible connectors are flagged so you know before building the audience whether delivery will work.

### 3. Filters

Apply conditions to narrow your source data. Filters are built from your dataset's mapped attributes, so the available filter options reflect your actual data.

#### Filter types

| Type                        | Controls                                       | Example                         |
| --------------------------- | ---------------------------------------------- | ------------------------------- |
| **Enum** (low cardinality)  | Checkbox list of values                        | Gender: male, female            |
| **Enum** (high cardinality) | Searchable dropdown                            | Country: US, CA, GB             |
| **Boolean**                 | Toggle                                         | Opted in: true / false          |
| **String**                  | Text input with multiple values                | Email domain: gmail.com         |
| **Timestamp**               | Curated ranges, custom lookback, or date range | Last 30 days, last 90 days      |
| **Numeric** (long, double)  | Comparison operators with values               | Age greater than or equal to 25 |

#### Logical operators

Filters support **AND** / **OR** logic. Multiple filters default to AND — all conditions must match. Group filters with OR to match any condition within a group.

#### Frequency filters

Any filter supports an optional **frequency** constraint — for example, "purchased at least 3 times in the last 30 days." Frequency filters use a `QUALIFY` clause in the generated NQL to count distinct occurrences of a field.

#### Null handling

Filters include **Is null** and **Is not null** options that generate correct `IS NULL` / `IS NOT NULL` NQL predicates, allowing you to filter on the presence or absence of a value.

### 4. Finalize

Configure the audience metadata and connector-specific settings:

* **Audience name** (required) — Identifies this audience in the platform and in destination systems
* **Description** — Optional context about the audience's purpose
* **Tags** — Optional labels for organization
* **Refresh schedule** — How often the audience is rebuilt. Choose **one-time** for a static audience or set a recurring schedule (daily, weekly, etc.) to keep the audience current as source data changes
* **Connector quick settings** — Each selected connector has platform-specific fields (advertiser selection, audience naming, historical data options). These are validated inline before creation.

## Forecasting

Before creating an audience, you can **forecast** its size. The forecast runs a preview query against your source data with the current filters applied and returns the expected record count. This helps you verify that your filters produce the right audience size before committing to creation.

Suggested filter values show what percentage of your forecasted audience matches each value, helping you understand the impact of adding or adjusting filters.

## What Audience Studio creates

When you click **Create Audience**, Audience Studio:

1. Generates and executes an NQL query based on your filters
2. Materializes the results as a new dataset tagged as an audience
3. Creates connector deliveries for each selected destination with your configured quick settings
4. If a refresh schedule is set, configures incremental updates using NQL `MERGE` to keep the audience current

The resulting audience dataset appears under **My Audiences** in the navigation and can be managed, cloned, or delivered to additional connectors at any time.

## Cloning audiences

You can clone an existing audience by selecting **Open in Audience Studio** from the audience's action menu. The builder pre-populates with the original audience's source, filters, connectors, and settings, giving you a starting point for creating a similar but distinct audience.

## Related content

<CardGroup>
  <Card title="Building an Audience" icon="hammer" href="/guides/audience-studio/building-an-audience">
    Step-by-step guide to creating your first audience
  </Card>

  <Card title="Look-alike Studio" icon="users-viewfinder" href="/concepts/data-activation/lookalike-studio">
    Expand a seed audience by scoring a larger population
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Strategies for organizing data for activation
  </Card>

  <Card title="Data Activation" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    How connectors deliver data to destination platforms
  </Card>

  <Card title="Connector Reference" icon="plug" href="/reference/connectors/index">
    Technical details for each destination connector
  </Card>
</CardGroup>


# Look-alike Studio
Source: https://docs.narrative.io/concepts/data-activation/lookalike-studio

Expand a seed audience by scoring a larger population against the seed using a guided builder

Look-alike Studio is the platform's visual look-alike audience builder. It takes a small **seed** dataset of users you already value (converters, high-LTV customers, opted-in subscribers) and scores a larger **population** dataset to find users who resemble the seed — producing a new audience of likely-to-convert users without writing NQL by hand.

## How Look-alike Studio works

A look-alike audience is built from three inputs:

1. **A seed dataset** — Users who exhibit the behavior you want to find more of. Typically small (thousands to low millions of records).
2. **A population dataset** — The larger universe to score against. Must share a join-key identifier with the seed (for example, both datasets map `sha256_hashed_email`).
3. **Shared attributes** — Categorical or continuous fields present on both datasets that describe each user. These are the features used to score similarity.

The builder generates a multi-stage NQL workflow that computes per-attribute weights from the seed, scores every population user against the seed distribution, and materializes the highest-scoring users as a new audience dataset.

## Scoring model

Look-alike Studio uses a Naive Bayes model with Gaussian terms for continuous attributes:

* **Categorical attributes** (country, device type, product category) contribute log-likelihood weights based on how much more frequently each value appears in the seed versus the population.
* **Continuous attributes** (age, lifetime value, days since last visit) contribute Gaussian density terms parameterized by the seed's mean and variance for each attribute.
* **Attribute weights** down-weight low-information attributes — an attribute that has the same distribution in the seed and the population contributes little signal.

Each population user receives a single composite score between 0 and 1 representing how seed-like they are.

## Builder steps

### 1. Seed selection

Choose the dataset of users you want to find more of. The dataset must have [Rosetta Stone attribute mappings](/guides/rosetta-stone/mapping-schemas) so the builder can identify join keys and shared features.

Only datasets on the currently selected [data plane](/concepts/primitives/data-planes) are shown. Pipeline intermediates and other audiences are hidden from the picker.

### 2. Population selection

Choose the larger dataset to score against. The population must share at least one **join-key identifier attribute** with the seed (such as `sha256_hashed_email` or `narrative_id`) so the builder can identify which population users overlap with seed users and exclude or include them appropriately.

If the seed and population do not share a join-key attribute, the dataset is shown as ineligible.

### 3. Attribute selection

Look-alike Studio classifies every attribute that exists on both datasets into one of three roles:

| Role            | Description                                                   | Used for scoring?                          |
| --------------- | ------------------------------------------------------------- | ------------------------------------------ |
| **Identity**    | Identifiers like hashed email, narrative\_id, MAID            | No — used only to join seed and population |
| **Categorical** | Low-to-medium cardinality fields like country, device, gender | Yes — Naive Bayes                          |
| **Continuous**  | Numeric fields like age, LTV, frequency                       | Yes — Gaussian density                     |

You select which classified attributes to include as scoring features. More attributes is not always better — irrelevant or correlated attributes add noise. Start with 3–8 attributes that you believe describe what makes the seed distinctive.

### 4. Output configuration

Choose how the look-alike audience is sized:

* **Limit by size** — Return the top-N highest-scoring population users (for example, the top 100,000).
* **Limit by score** — Return every population user whose score exceeds a threshold (for example, score ≥ 0.5).

You also choose whether to include the original seed users in the output:

* **New users only** — Only population users not already in the seed are returned. Use when you want net-new reach.
* **New + original seed users** — Seed users are added to the output. Use when you want a single combined audience to activate.

### 5. Finalize

Configure the output dataset metadata:

* **Audience name** (required) — Identifies the audience in the platform and downstream systems.
* **Description** — Optional context.
* **Tags** — Optional labels. The system automatically applies the `_nio_audience` and `_nio_lookalike` tags so the audience appears under **My Audiences** and is recognizable as look-alike-derived.

Name uniqueness is enforced against all existing datasets, including hidden interactive results and look-alike pipeline intermediates.

## What Look-alike Studio creates

When you click **Create Look-alike Audience**, the builder:

1. Assembles the seed, population, attribute selections, and output config into a multi-stage workflow YAML.
2. Submits the workflow via the workflow engine with `trigger_immediately`.
3. Materializes around a dozen intermediate views (canonical feature combos, per-attribute distributions, attribute weights, scored candidates) tagged `_nio_lookalike_intermediate`. These are hidden from the standard dataset catalog so only the final audience appears under **My Audiences**.
4. Materializes a final audience dataset with the `_nio_audience` and `_nio_lookalike` tags, your display name, description, and any user-supplied tags applied as part of the `CREATE MATERIALIZED VIEW` so the audience is born fully tagged even if you leave mid-run.

The run surfaces a dismissible progress dialog — closing it does not cancel the workflow, which continues server-side. When the run completes, the audience opens in the standard audience metadata drawer and is available for delivery through any compatible [connector](/reference/connectors/index).

## Resuming or restarting

* **Dismiss progress, keep running** — Closing the progress dialog leaves the workflow running. The audience appears under **My Audiences** when it finishes.
* **Retry on error** — A failed run shows an error dialog with a **Retry** action that resubmits the same workflow.
* **Start Over** — Cancels polling and resets all builder state so you can build a different look-alike audience.

## Related content

<CardGroup>
  <Card title="Building a look-alike audience" icon="hammer" href="/guides/audience-studio/building-a-lookalike-audience">
    Step-by-step guide to creating your first look-alike audience
  </Card>

  <Card title="Audience Studio" icon="users" href="/concepts/data-activation/audience-studio">
    Filter-based audience builder for direct segmentation
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Strategies for organizing data for activation
  </Card>

  <Card title="Connector Reference" icon="plug" href="/reference/connectors/index">
    Technical details for each destination connector
  </Card>
</CardGroup>


# Data Activation
Source: https://docs.narrative.io/concepts/data-activation/overview

Deliver audience data from Narrative to advertising and marketing platforms

Data activation is the process of delivering audience data from Narrative to external platforms—such as demand-side platforms (DSPs), social networks, and analytics tools—where you can take action on that data. Connectors are the mechanism that make this possible.

## What connectors do

A connector is a pre-built integration that connects Narrative to an external destination. Connectors handle the complexities of delivering data to each platform:

* **Format translation**: Converts your data into the format required by the destination
* **Identity matching**: Maps your identifiers to the platform's identity space
* **API integration**: Manages authentication, rate limits, and delivery protocols
* **Audience management**: Creates and updates audiences or segments in the destination

When you deliver data through a connector, you don't need to build custom integrations or understand each platform's technical requirements. The connector handles all of this automatically.

## How activation works

```mermaid theme={null}
flowchart LR
    A[Dataset] --> B[NQL Query]
    B --> C[Connector]
    C --> D[Destination Platform]

    style A fill:#e1e8f0
    style B fill:#e1e8f0
    style C fill:#1438F5,color:#fff
    style D fill:#e1e8f0
```

1. **Prepare your data**: Structure your audience data in one or more [datasets](/concepts/primitives/datasets) with the required identifiers (typically hashed emails or phone numbers)

2. **Normalize with Rosetta Stone**: Map your data fields to [Rosetta Stone](/concepts/rosetta-stone/overview) attributes so the connector knows which fields contain deliverable identifiers

3. **Configure the connector**: Install the connector for your destination and provide the necessary credentials (API keys, account IDs, etc.)

4. **Deliver your audience**: Use NQL to select the records you want to deliver, or deliver entire datasets directly

## Identifier requirements

Each destination requires specific identifier types for matching. Connectors automatically transform your data to meet these requirements, but your source data must include at least one supported identifier type.

| Identifier Type              | Common Destinations                        |
| ---------------------------- | ------------------------------------------ |
| SHA256 hashed email          | Google DV360, Meta, The Trade Desk, TikTok |
| SHA256 hashed phone          | Google DV360, Meta, TikTok                 |
| Mobile advertising ID (MAID) | The Trade Desk, DSPs                       |
| UID2                         | The Trade Desk (auto-generated from email) |

<Note>
  If your data lacks these identifiers but you want to activate it as an audience, contact your Narrative account representative for assistance with identity resolution options.
</Note>

For detailed requirements on preparing identifiers, see [Hashing PII for Upload](/guides/ingestion/hashing-pii).

***

## Available destinations

Narrative provides connectors to major advertising and data platforms:

<CardGroup>
  <Card title="The Trade Desk" icon="bullhorn" href="/reference/connectors/the-trade-desk">
    Deliver audiences to TTD's first-party or third-party marketplace
  </Card>

  <Card title="Meta Custom Audiences" icon="meta" href="/reference/connectors/meta">
    Push custom audiences to Facebook and Instagram
  </Card>

  <Card title="Meta Conversions API" icon="meta" href="/reference/connectors/meta-conversions-api">
    Send conversion events for ad attribution
  </Card>

  <Card title="Google DV360" icon="google" href="/reference/connectors/google-dv360">
    Activate audiences in Display & Video 360
  </Card>

  <Card title="Amazon S3" icon="aws" href="/reference/connectors/amazon-s3">
    Deliver data files to your S3 bucket
  </Card>
</CardGroup>

See [Connector Reference](/reference/connectors/index) for the complete list of available connectors.

***

## Activation patterns

There are several ways to structure your data for activation, depending on your use case:

### Single-purpose datasets

Create a separate dataset for each audience you want to activate. This approach is straightforward when you have distinct, pre-defined audiences.

**Best for**: Organizations that build and maintain audiences outside of Narrative, then upload them for delivery.

### Master dataset with audience segments

Maintain a single dataset containing all your audience data, with columns that identify which audience each record belongs to. Use NQL to filter and deliver specific segments.

**Best for**: Organizations managing many audiences that share the same underlying data structure.

### Raw data with dynamic segmentation

Upload your complete customer data with all available attributes, then use NQL to create audiences on-the-fly based on any combination of attributes.

**Best for**: Organizations that want maximum flexibility to create new audiences without re-uploading data.

<Info>
  For detailed guidance on implementing these patterns, see [Structuring Audiences for Activation](/guides/activation/audience-strategies).
</Info>

***

## What happens after delivery

When you deliver data through a connector:

1. **The connector processes your data**: Identifiers are formatted and, if needed, translated (e.g., hashed email to UID2)

2. **Data is sent to the destination**: The connector uses the platform's API to create or update audiences

3. **The platform matches identifiers**: The destination matches your identifiers against its user base. Match rates vary by platform and identifier type (typically 30-70%)

4. **Audiences become targetable**: Once processed, your audiences are available for targeting in campaigns on that platform

<Warning>
  Match rates depend on the destination platform's user base and your data quality. Not every identifier will match. Monitor your connector delivery reports to understand match rates for your audiences.
</Warning>

***

## Related content

<CardGroup>
  <Card title="Audience Studio" icon="users" href="/concepts/data-activation/audience-studio">
    Build, filter, and deliver audiences using a guided visual builder
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Strategies for organizing data for activation
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Connect offline data to online identifiers
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for upload and activation
  </Card>

  <Card title="Connector Reference" icon="plug" href="/reference/connectors/index">
    Technical details for each connector
  </Card>

  <Card title="App Invites" icon="envelope-open-text" href="/concepts/apps/app-invites">
    How invite links onboard third parties to connect accounts
  </Card>
</CardGroup>


# Data Collaboration Platforms
Source: https://docs.narrative.io/concepts/data-collaboration-platforms

How the convergence of big data infrastructure, specialized talent, and data scarcity led to a new category of software for acquiring and sharing data

A Data Collaboration Platform is software that enables organizations to discover, acquire, and share data with external partners. The category emerged in the mid-2010s as a response to a specific problem: organizations had invested heavily in data infrastructure and analytics talent, but lacked access to the external data they needed to realize the full value of those investments.

## The infrastructure foundation

The 2000s saw a transformation in how organizations store and process data. Open-source projects like Hadoop, Spark, and Kafka—many originating from Silicon Valley companies solving their own scaling challenges—made it possible to work with datasets that would have overwhelmed traditional databases.

Initially, only large enterprises could deploy and operate this infrastructure. The expertise required was scarce and expensive. But cloud platforms changed this dynamic. Services like Amazon Web Services, Google Cloud Platform, and Microsoft Azure abstracted away the operational complexity, allowing mid-sized companies to access the same capabilities without building data centers or hiring specialized operations teams.

By the early 2010s, the infrastructure barrier had largely fallen. Organizations of many sizes could store petabytes of data and run complex analytical workloads. The question shifted from "can we handle this much data?" to "what can we do with it?"

## The talent evolution

Powerful infrastructure created demand for people who could use it. Traditional database administrators understood relational databases and SQL, but the new paradigm required different skills: distributed systems, statistical modeling, and programming in languages like Python and R.

Organizations began hiring data scientists, data engineers, and analytics professionals. These roles brought capabilities that hadn't existed in most companies: predictive modeling, machine learning, and the ability to extract insights from unstructured data. The job title "data scientist" barely existed before 2010; by the mid-2010s, it was one of the most sought-after positions in technology.

This talent could do things with data that simply weren't possible before—identifying patterns in customer behavior, predicting equipment failures, optimizing supply chains in real time. But their effectiveness depended entirely on having access to the right data.

## The data acquisition gap

Infrastructure and talent, it turned out, were necessary but not sufficient. Organizations discovered that their internal data—however large—rarely contained everything they needed for sophisticated analytics.

A retailer might have detailed transaction data but lack information about customer behavior outside their stores. A financial services company might have comprehensive records of their own customers but no visibility into market-wide trends. A healthcare organization might have clinical data but miss the social determinants that influence patient outcomes.

The need for external data wasn't new. What changed was the scale and complexity of what organizations wanted to do with it. Machine learning models require training data from multiple sources. Predictive analytics work better with more signals. The same infrastructure and talent investments that enabled sophisticated analytics also created demand for data that no single organization could generate internally.

Acquiring external data, however, was difficult. The process typically involved:

* **Discovery** — Finding organizations that had relevant data, often through personal networks or industry connections
* **Negotiation** — Working through legal and commercial terms, frequently taking months
* **Technical integration** — Building custom pipelines to ingest data in whatever format the provider used
* **Ongoing maintenance** — Managing schema changes, delivery schedules, and data quality issues

Each data partnership was a bespoke project. Organizations that needed data from multiple sources faced the same integration challenges repeatedly.

## How Data Collaboration Platforms address this

Data Collaboration Platforms emerged to systematize what had been ad-hoc. Rather than negotiating individual data partnerships, organizations can discover and acquire data through a common interface.

The core capabilities of a Data Collaboration Platform typically include:

**Centralized discovery.** A catalog of available datasets that organizations can browse and evaluate, replacing informal networks and one-off outreach.

**Standardized formats.** Common schemas and data formats that reduce the integration burden. Instead of building custom pipelines for each data source, organizations work with consistent structures.

**Programmatic access.** APIs and query interfaces that allow data acquisition to be automated and integrated into existing workflows, rather than relying on manual file transfers.

**Access control and governance.** Mechanisms for data providers to control how their data is used, enabling collaboration that wouldn't happen if providers had to trust recipients with unrestricted access.

**Privacy-preserving techniques.** Capabilities like [data pseudonymization](/concepts/security/data-pseudonymization) that allow organizations to collaborate on sensitive data while protecting individual privacy.

The result is that acquiring external data looks less like a series of custom projects and more like querying a database. Organizations can move from identifying a data need to having that data in their analytics pipeline in days rather than months.

***

## Related content

<CardGroup>
  <Card title="Rosetta Stone" icon="language" href="/concepts/rosetta-stone/overview">
    How Narrative normalizes schemas across different data sources and formats
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    How Narrative orchestrates data discovery, access control, and collaboration
  </Card>
</CardGroup>


# IP Addresses
Source: https://docs.narrative.io/concepts/data-formats/ip-addresses

Understanding IP address formats, types, and their role in data collaboration

IP addresses are unique identifiers for devices communicating over networks. Standing for Internet Protocol, they adhere to standardized formats that enable data routing across the internet. In data collaboration, IP addresses provide signals for geographic targeting, fraud detection, and network-level analytics.

## IP address formats

### IPv4

The most common format, IPv4 addresses consist of four octets (8-bit numbers) separated by periods. Each octet ranges from 0 to 255.

```
192.168.1.1
74.125.224.72
```

**Characteristics:**

* 32-bit address space
* \~4.3 billion possible addresses
* Running out of available addresses globally

### IPv6

Developed to expand the address pool, IPv6 uses a 128-bit address format with eight groups of four hexadecimal digits separated by colons.

```
2001:0db8:85a3:0000:0000:8a2e:0370:7334
```

**Characteristics:**

* 128-bit address space
* \~340 undecillion possible addresses
* Growing adoption but not yet universal

<Tip>
  IPv6 addresses can be abbreviated by omitting leading zeros and collapsing consecutive zero groups with `::`. The address above could be written as `2001:db8:85a3::8a2e:370:7334`.
</Tip>

***

## Types of IP addresses

### Public vs. private

| Type        | Description                                          | Example Ranges                             |
| ----------- | ---------------------------------------------------- | ------------------------------------------ |
| **Public**  | Globally routable, assigned by ISPs                  | Most addresses outside reserved ranges     |
| **Private** | Used within local networks, not routable on internet | `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x` |

Public IP addresses identify devices on the open internet and are the primary type used in data collaboration. Private addresses are used within homes and organizations and are not visible externally.

### Static vs. dynamic

| Type        | Description                            | Use Cases                       |
| ----------- | -------------------------------------- | ------------------------------- |
| **Static**  | Remains constant over time             | Servers, business connections   |
| **Dynamic** | Changes periodically, assigned by DHCP | Residential connections, mobile |

Most consumer internet connections use dynamic IPs, which can change daily or when the router restarts. This affects data collaboration because the same IP address might represent different users over time.

### Dedicated vs. shared

| Type          | Description                       | Context                                        |
| ------------- | --------------------------------- | ---------------------------------------------- |
| **Dedicated** | Single device or organization     | Direct server access                           |
| **Shared**    | Multiple users behind the same IP | NAT, VPNs, corporate networks, mobile carriers |

Carrier-grade NAT (CGNAT) and VPNs mean that many users may share the same public IP address, making IP-based identification less precise.

***

## IP addresses in data collaboration

### Geographic signals

IP addresses can be mapped to approximate geographic locations:

* **Country:** High accuracy
* **Region/State:** Generally reliable
* **City:** Moderate accuracy
* **Postal code:** Less reliable
* **Street level:** Not reliable

IP geolocation is useful for:

* Regulatory compliance (geo-restrictions)
* Regional targeting
* Fraud detection (location mismatches)

<Warning>
  IP geolocation is approximate and can be inaccurate, especially for mobile users and VPN users. Never rely on IP location for critical decisions requiring precise location.
</Warning>

### Fraud and security signals

IP addresses provide useful signals for fraud detection:

* Known proxy/VPN IP ranges
* Data center vs. residential IPs
* Bot traffic identification
* Rate limiting and abuse detection

### Privacy considerations

IP addresses are considered personal data under GDPR and may be subject to other privacy regulations:

* They can identify households or organizations
* Combined with other data, they may identify individuals
* Collection requires appropriate legal basis

***

## Working with IP addresses

### Normalization

When matching IP data across sources:

* Validate format (proper IPv4 or IPv6 structure)
* Handle leading zeros consistently
* Consider canonicalizing IPv6 addresses

### Storage

| Format         | Storage Size | Notes                    |
| -------------- | ------------ | ------------------------ |
| String         | Variable     | Human-readable, flexible |
| Integer (IPv4) | 4 bytes      | Efficient for IPv4 only  |
| Binary (IPv6)  | 16 bytes     | Supports both versions   |

### Matching limitations

IP addresses are session-level identifiers, not user-level:

* Dynamic IPs change over time
* Shared IPs represent multiple users
* Mobile IPs change as users move between networks

For user-level matching, IP addresses are typically combined with other signals or used as one factor in probabilistic matching.

***

## Related content

<CardGroup>
  <Card title="Location Data" icon="location-dot" href="/concepts/data-formats/location-data">
    Geographic data formats and polygons
  </Card>

  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Overview of identifier categories
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Privacy protection for identifiers
  </Card>
</CardGroup>


# Location Data
Source: https://docs.narrative.io/concepts/data-formats/location-data

Understanding location polygons, geospatial formats, and how to work with geographic data

Location data adds geographic context to records, enabling use cases like proximity targeting, foot traffic analysis, and location-based audience building. At its core, location data maps coordinates to meaningful places—transforming raw latitude/longitude pairs into actionable insights.

## Why location context matters

Raw location data typically consists of latitude, longitude, and timestamp—coordinates that have limited meaning in isolation. The latitude `40.74576` and longitude `-73.9848` tells you little on its own, but when mapped to "Narrative Headquarters in New York City," the data becomes actionable.

Location context enables:

* **Proximity targeting:** Reach users near specific locations
* **Foot traffic analysis:** Understand visitation patterns
* **Competitive intelligence:** Analyze visits to competitor locations
* **Attribution:** Connect ad exposure to physical store visits

***

## Location polygons

A location polygon is a shape that represents a place, point of interest, or geographic boundary using geospatial coordinates. Polygons are defined by the coordinates of their vertices (corner points), connected by edges.

### What polygons represent

Polygons can represent locations at any scale:

* Individual buildings or storefronts
* City blocks or neighborhoods
* Entire cities or regions
* Country borders

### Coordinate systems

Most location data uses the **WGS84** (World Geodetic System 1984) coordinate reference system, which represents locations as latitude/longitude pairs:

* **Latitude:** -90 to 90 (north/south of equator)
* **Longitude:** -180 to 180 (east/west of prime meridian)

<Note>
  Coordinate order varies by format. GeoJSON uses longitude-first (`[lon, lat]`), while many other systems use latitude-first (`lat, lon`). Always verify the expected order when working with location data.
</Note>

***

## Location data formats

### GeoJSON

[GeoJSON](https://en.wikipedia.org/wiki/GeoJSON) is a JSON-based standard for encoding geographic data. It's human-readable, easy to parse in most programming languages, and widely supported.

```json theme={null}
{
  "type": "Polygon",
  "coordinates": [[
    [-122.389946, 37.7786],
    [-122.389095, 37.7786],
    [-122.389095, 37.7793],
    [-122.389946, 37.7793],
    [-122.389946, 37.7786]
  ]]
}
```

**Narrative accepts:**

* `.json`, `.geojson`, and `.parquet` files
* `Polygon` and `MultiPolygon` geometries
* Does **not** accept GeoJSON embedded in CSV

<Tip>
  Validate your GeoJSON using [GeoJSONLint](https://geojsonlint.com/) before upload.
</Tip>

### Well-Known Text (WKT)

[WKT](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) is a text-based format for representing geometry. It's more compact than GeoJSON but less human-readable.

```
POLYGON((-122.389946 37.7786, -122.389095 37.7786, -122.389095 37.7793, -122.389946 37.7793, -122.389946 37.7786))
```

**Narrative accepts:**

* WKT in CSV format
* Does **not** accept WKT in JSON format

### Shapefile

[Shapefiles](https://en.wikipedia.org/wiki/Shapefile) are a common GIS format, but they require specialized libraries to read and write. They consist of multiple files (`.shp`, `.shx`, `.dbf`) that must be kept together.

### Point-radius

For locations where exact boundaries aren't known, point-radius defines a center point and a radius (typically in meters). While not technically a polygon, many systems convert point-radius to polygons by calculating vertices around the circle.

```
-122.389521, 37.7789, 100
```

(longitude, latitude, radius in meters)

**Narrative accepts:**

* Point-radius in `.txt`, `.json`, `.parquet`, and `.csv` formats

***

## Supported file formats

| Format           | File Types                          | Notes                            |
| ---------------- | ----------------------------------- | -------------------------------- |
| **GeoJSON**      | `.json`, `.geojson`, `.parquet`     | Polygon and MultiPolygon only    |
| **WKT**          | `.csv`                              | WKT column with polygon geometry |
| **Point-radius** | `.txt`, `.json`, `.parquet`, `.csv` | lon, lat, radius                 |

***

## Coordinate order

A common source of errors is mixing up longitude and latitude order. Different formats use different conventions:

| Format      | Order                   | Example                  |
| ----------- | ----------------------- | ------------------------ |
| GeoJSON     | `[longitude, latitude]` | `[-122.389, 37.778]`     |
| WKT         | `longitude latitude`    | `POINT(-122.389 37.778)` |
| Google Maps | `latitude, longitude`   | `37.778, -122.389`       |

For a comprehensive reference, see [lonlat.info](https://macwright.com/lonlat/).

***

## Creating location data

If you need to define custom polygons:

1. **Drawing tools:** Use [Bounding Box Tool](https://boundingbox.klokantech.com/) to draw shapes and export in various formats
2. **GIS software:** Tools like QGIS (free) can create and edit complex geometries
3. **Existing datasets:** Many POI datasets are available commercially or through open sources

***

## Related content

<CardGroup>
  <Card title="IP Addresses" icon="network-wired" href="/concepts/data-formats/ip-addresses">
    Network-based location signals
  </Card>

  <Card title="Unix Time" icon="clock" href="/concepts/data-formats/unix-time">
    Timestamp formats for location data
  </Card>
</CardGroup>


# Unix Time
Source: https://docs.narrative.io/concepts/data-formats/unix-time

Understanding Unix timestamps and how Narrative uses them for data collaboration

Unix time represents timestamps as the number of seconds elapsed since January 1st, 1970 at 00:00:00 UTC, known as the **Unix epoch**. This method simplifies parsing and utilizing time across various systems by representing it as a single integer.

Narrative's Data Streaming Platform uses Unix time in **milliseconds** for all timestamp fields, providing millisecond-level precision for data collaboration.

## Unix Epoch

The Unix epoch—January 1st, 1970 at 00:00:00 UTC—was chosen by early Unix engineers as a convenient and uniform starting point for representing time. All Unix timestamps are calculated relative to this moment.

## Examples

| Date / Time                  | Unix Time (seconds) |
| ---------------------------- | ------------------- |
| January 1st, 1970 00:00:00   | 0                   |
| June 6th, 1983 00:00:00      | 423705600           |
| June 6th, 1983 16:00:00      | 423763200           |
| September 9th, 2001 01:46:40 | 1000000000          |
| July 20th, 1969 20:17:40     | -14182940           |

<Note>
  Dates before 1970 are represented as negative numbers, counting seconds backward to the Unix epoch.
</Note>

## Limitations

### Year 2038 Problem

Systems that store Unix time as a signed 32-bit integer will encounter errors after **03:14:07 UTC on January 19, 2038**, when timestamps exceed the maximum 32-bit value. This is similar to the Y2K issue. Modern systems using 64-bit integers are not affected.

### Leap Seconds

Unix time assumes each day has exactly 86,400 seconds and does not account for leap seconds. This means Unix time can drift slightly from true UTC time, though the difference is negligible for most applications.

## Variations

### Unix Time with Decimals

For sub-second precision, Unix time can include decimal places. For example, `1609459200.5` represents a moment 500 milliseconds after midnight on January 1st, 2021.

### Unix Time in Milliseconds

Narrative uses this format, representing time as the number of **milliseconds** since the Unix epoch. This provides greater precision for timestamp data—multiply standard Unix time by 1,000 to convert.

For example:

* Unix time (seconds): `1609459200`
* Unix time (milliseconds): `1609459200000`

## Additional Resources

* [Wikipedia: Year 2038 Problem](https://en.wikipedia.org/wiki/Year_2038_problem)
* [Wikipedia: Unix time](https://en.wikipedia.org/wiki/Unix_time)
* [Epoch Converter](https://www.epochconverter.com/) for practical conversions between Unix time and standard dates


# Addressability Expansion
Source: https://docs.narrative.io/concepts/identifiers/addressability

Improving media activation reach by appending additional identifiers to existing identity graph records

Addressability expansion appends additional identifiers to existing records in your identity graph to improve match rates in downstream activation platforms. Unlike [graph enrichment](/concepts/identifiers/graph-enrichment), which changes the graph's structure by adding edges between nodes, addressability expansion adds more "handles" to existing nodes—giving activation platforms more ways to find your customers.

## What addressability expansion is

Every activation platform has its own identifier ecosystem. The Trade Desk works with UID2 and MAIDs. Meta matches on hashed emails and phone numbers. CTV platforms use device-level identifiers. If your CRM only contains email addresses, your match rate on a MAID-based platform will be low—not because your targeting is wrong, but because you lack the right identifiers to reach your audience there.

Addressability expansion solves this by appending additional identifier types to your existing records. A customer who was previously reachable only via email becomes reachable via email, MAID, hashed phone, and postal address.

## How it works

Consider a direct-to-consumer brand with a CRM containing one email per customer. Their current match rates:

| Platform       | Identifier accepted | Match rate |
| -------------- | ------------------- | ---------- |
| Meta           | Hashed email        | 62%        |
| The Trade Desk | UID2, MAID          | 15%        |
| CTV platforms  | MAID, IP address    | 8%         |

After addressability expansion with a provider that appends MAIDs, hashed phones, and additional email addresses:

| Platform       | Identifiers available       | Match rate |
| -------------- | --------------------------- | ---------- |
| Meta           | Hashed email + hashed phone | 78%        |
| The Trade Desk | UID2 + MAID                 | 51%        |
| CTV platforms  | MAID + IP address           | 34%        |

The brand's audience segments did not change. The same customers are targeted with the same logic. But those customers are now reachable on more platforms.

## What it does NOT do

Addressability expansion has clear boundaries:

* **Does not change segmentation.** Your audience definitions stay the same. If a customer was in a segment before expansion, they remain in it. If they were not, expansion does not add them.
* **Does not improve targeting quality.** You are reaching the same people with the same intent. The improvement is in reach, not precision.
* **Does not resolve identity.** Expansion does not merge separate profiles or discover that two records are the same person. That is [graph enrichment](/concepts/identifiers/graph-enrichment).

The value of addressability expansion is straightforward: higher match rates on activation platforms, which means more of your intended audience actually sees your campaigns.

## What to look for in providers

When evaluating addressability expansion providers, focus on activation outcomes:

| Criterion                       | What to measure                                                   | Why it matters                                                |
| ------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- |
| **Identifier coverage by type** | Volume of MAIDs, hashed emails, hashed phones, etc.               | Determines which platforms benefit from expansion             |
| **Match rate lift by platform** | Before/after match rates per destination                          | The direct measure of value                                   |
| **Freshness and churn rate**    | How recently identifiers were observed, how quickly they go stale | Stale identifiers waste spend on unreachable users            |
| **Identity type breadth**       | Person-level vs. household-level identifiers                      | Different activation strategies require different granularity |
| **Cost per incremental match**  | Total cost divided by net new matched users                       | The efficiency metric for comparing providers                 |

## Addressability expansion in Narrative

Narrative's platform supports addressability expansion through Rosetta Stone matching and connector-based activation.

### Appending identifiers via Rosetta Stone

Use NQL to join your first-party data against identity providers and append additional identifier types:

```sql theme={null}
SELECT
    seed.internal_user_id,
    seed."_rosetta_stone"."unique_identifier"."value" AS seed_id,
    seed."_rosetta_stone"."unique_identifier"."type" AS seed_id_type,
    provider."_rosetta_stone"."unique_identifier"."value" AS appended_id,
    provider."_rosetta_stone"."unique_identifier"."type" AS appended_id_type
FROM company_data."crm_seed" seed
INNER JOIN external_provider."identity_data" provider
    ON seed."_rosetta_stone"."unique_identifier"."value" = provider."_rosetta_stone"."unique_identifier"."value"
    AND seed."_rosetta_stone"."unique_identifier"."type" = provider."_rosetta_stone"."unique_identifier"."type"
WHERE provider."_rosetta_stone"."unique_identifier"."type" IN ('maid', 'sha256_phone')
```

### Measuring match rate lift

After expanding your identifiers, use [connectors](/reference/connectors/index) to activate audiences on destination platforms. Compare match rates before and after expansion to quantify the lift each provider delivers.

***

## Related content

<CardGroup>
  <Card title="Identity Graphs" icon="diagram-project" href="/concepts/identifiers/identity-graph">
    The foundational concepts behind identity graph structure
  </Card>

  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    The complementary approach: improving graph structure and resolution
  </Card>

  <Card title="Data Activation" icon="rocket" href="/concepts/data-activation/overview">
    Delivering audiences to destination platforms
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Translating offline data for digital activation
  </Card>
</CardGroup>


# Data Onboarding
Source: https://docs.narrative.io/concepts/identifiers/data-onboarding

Translating offline customer data for digital activation

Data onboarding is the process of taking offline customer data and connecting it to online identifiers for digital activation. It bridges the gap between CRM systems, point-of-sale data, and loyalty programs with the digital advertising ecosystem.

## Why data onboarding matters

Many companies have rich customer data from offline interactions:

* Purchase history from retail stores
* Loyalty program memberships
* Call center interactions
* Direct mail campaigns

But digital advertising platforms work with online identifiers—cookies, mobile ad IDs, and universal IDs. Without onboarding, this valuable first-party data remains unusable for digital marketing.

Data onboarding unlocks:

* **Digital reach for offline customers:** Find your store customers online
* **Personalization at scale:** Apply offline insights to digital experiences
* **Closed-loop measurement:** Connect digital campaigns to offline conversions

***

## How data onboarding works

### 1. Prepare and anonymize offline data

Data onboarding starts with your offline customer file—typically including emails, phone numbers, and customer attributes.

Before any matching occurs, personally identifiable information (PII) must be pseudonymized:

* Hash email addresses and phone numbers
* Remove or mask direct identifiers (names, addresses)
* Retain only the attributes needed for activation

<Note>
  Narrative requires PII to be hashed before upload. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

### 2. Match to online identifiers

Pseudonymized customer data is matched to online identifiers through identity resolution:

| Offline Identifier | Matched To                    |
| ------------------ | ----------------------------- |
| Hashed email       | Mobile ad IDs, cookies, UID2  |
| Hashed phone       | Mobile ad IDs, UID2           |
| Postal address     | Device graphs (probabilistic) |

Matching can be:

* **Deterministic:** Direct link observed (e.g., user logged in with email on mobile app)
* **Probabilistic:** Inferred from signals (e.g., shared household IP address)

### 3. Activate in digital channels

Once matched, customer segments can be activated across digital platforms:

* Programmatic display and video
* Social media advertising
* Connected TV
* Retail media networks

***

## Benefits of data onboarding

### Boost media reach

By appending online identifiers to your customer file, you gain additional touchpoints to reach customers across devices and channels—not just where they gave you their email.

### Improve personalization

Bring offline insights online. A customer who purchased in-store can receive relevant digital ads, while someone who returned a product can be excluded from promotions.

### Enable omni-channel measurement

Combining offline and online data provides a complete view of customer journeys. Measure how digital campaigns drive store visits, or how in-store experiences influence online behavior.

### Power advanced analytics

Onboarded data enables analysis across the full customer journey. Understand how offline and online touchpoints interact and influence each other, rather than analyzing channels in silos.

***

## Challenges in data onboarding

### Privacy compliance

Offline customer records often contain PII that's regulated under GDPR, CCPA, and other privacy laws. Mitigate risk by:

* Hashing all identifiers before sharing
* Maintaining appropriate consent records
* Working with compliant onboarding partners

### Match rates

Not all offline customers will match to online identifiers. Typical match rates range from 30-70% depending on:

* Data quality and recency
* Identifier types available
* Target platform coverage

### Data freshness

Customer data changes over time. Email addresses become invalid, people move, devices are replaced. Regular refresh of onboarded data maintains accuracy.

### Cost and complexity

Traditional onboarding services can be expensive and require long-term commitments. They may also charge for records that don't match, making ROI difficult to predict.

***

## Data onboarding in Narrative

Narrative simplifies data onboarding with transparent matching and flexible activation:

### Self-service upload

Upload your hashed customer file directly to the platform without intermediaries.

### Transparent matching

See which identity providers match your data and at what rates. Choose which matches to activate based on quality and coverage.

### Flexible activation

Activate matched audiences to multiple destinations:

* The Trade Desk (with automatic UID2 generation)
* Other DSPs and ad platforms
* Analytics environments

### Pay for matches

Only pay for records that successfully match to online identifiers.

***

## Related content

<CardGroup>
  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    How identifiers are connected across channels
  </Card>

  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Types of identifiers used in onboarding
  </Card>

  <Card title="UID2" icon="shield-check" href="/concepts/identifiers/uid2">
    Privacy-preserving identifier for activation
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Hashing and privacy protection
  </Card>

  <Card title="Hashing PII for Upload" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare data for onboarding
  </Card>
</CardGroup>


# Graph Enrichment
Source: https://docs.narrative.io/concepts/identifiers/graph-enrichment

Strengthening identity graph structure with third-party linkage data for improved person-level and household-level resolution

Graph enrichment uses third-party identity data to create or strengthen connections between nodes in your identity graph. Unlike [addressability expansion](/concepts/identifiers/addressability), which appends identifiers to existing records, graph enrichment changes the graph's structure itself—adding edges that connect previously isolated nodes into unified profiles.

## What graph enrichment is

Your first-party identity graph reflects the linkages you can directly observe: login events, CRM matches, transaction data. But your observation is limited to your own touchpoints. Graph enrichment fills the gaps by incorporating linkages observed by third-party providers across their own data ecosystems.

The result is a more connected graph with fewer fragmented profiles and better person-level or household-level resolution.

## How it works

Consider a retail company with this first-party data:

* Customer A: hashed email + iPhone IDFA (observed via app login)
* Customer A: hashed email + home address (observed via shipping)

Their first-party graph has a connected component with three nodes: email, IDFA, and address.

A third-party identity provider observes that the same email is also linked to an Android GAID and a second hashed email (a work address). After graph enrichment:

* Customer A's component now includes five nodes: personal email, work email, IDFA, GAID, and home address
* The retailer can now reach Customer A on Android devices and recognize them when they use their work email

The graph's structure changed. New edges were added, and the connected component expanded.

## Effects on the graph

Graph enrichment can improve your identity graph in several ways:

**Person-level resolution.** Connecting identifiers that belong to the same individual across devices and channels. A customer who appears as three separate profiles (email-only, MAID-only, cookie-only) becomes one resolved profile.

**Household-level resolution.** Linking identifiers across individuals within a household. Shared addresses, shared IP ranges, and shared device usage can connect household members when that level of resolution is appropriate for your use case.

**Waterfall matching flexibility.** A richer graph gives you more identifiers per person, which means more opportunities to match against any given activation platform. If one identifier type fails to match, another may succeed.

**Longitudinal consistency.** When identifiers change over time (cookies expire, MAIDs reset), a well-enriched graph maintains continuity through other persistent linkages.

## What to look for in providers

When evaluating graph enrichment providers, focus on these criteria:

| Criterion                         | What to measure                                        | Why it matters                                           |
| --------------------------------- | ------------------------------------------------------ | -------------------------------------------------------- |
| **Linkage depth**                 | Average edges per node                                 | More edges per node means richer resolution              |
| **Determinism rate**              | Percentage of deterministic vs. probabilistic linkages | Deterministic linkages are more reliable                 |
| **Confidence score quality**      | Distribution and calibration of confidence scores      | Well-calibrated scores enable meaningful thresholds      |
| **Update cadence**                | How frequently linkages are refreshed                  | Stale linkages degrade graph quality over time           |
| **Incremental component overlap** | How many of your existing components gain new edges    | Measures actual incremental value vs. redundant coverage |

## Risks

Graph enrichment carries higher stakes than addressability expansion. A bad identifier append reduces match rates—an inconvenience. A bad linkage merges two distinct people into a single profile—a structural corruption that propagates through every downstream use case.

Protect against this by:

* **Setting confidence thresholds.** Only incorporate linkages above a minimum confidence score. Start conservative and relax over time as you validate quality.
* **Monitoring component size distribution.** A sudden increase in very large components may indicate over-linking. Healthy graphs follow a predictable size distribution.
* **Evaluating incrementally.** Test a provider's linkages against a subset of your graph before full integration. Measure resolution improvement against a known truth set.

***

## Graph enrichment in Narrative

Narrative's platform supports graph enrichment evaluation and execution through Rosetta Stone and NQL.

### Evaluating provider linkage coverage

Use NQL to measure how much incremental coverage a provider adds to your existing graph:

```sql theme={null}
SELECT
    provider."_rosetta_stone"."unique_identifier"."type" AS id_type,
    COUNT(DISTINCT provider."_rosetta_stone"."unique_identifier"."value") AS provider_ids,
    COUNT(DISTINCT CASE
        WHEN seed."_rosetta_stone"."unique_identifier"."value" IS NOT NULL
        THEN provider."_rosetta_stone"."unique_identifier"."value"
    END) AS overlapping_ids
FROM external_provider."identity_linkages" provider
LEFT JOIN company_data."first_party_graph" seed
    ON provider."_rosetta_stone"."unique_identifier"."value" = seed."_rosetta_stone"."unique_identifier"."value"
    AND provider."_rosetta_stone"."unique_identifier"."type" = seed."_rosetta_stone"."unique_identifier"."type"
GROUP BY provider."_rosetta_stone"."unique_identifier"."type"
```

### Measuring resolution improvement

Compare connected component metrics before and after incorporating a provider's linkages to quantify the resolution improvement they deliver.

***

## Related content

<CardGroup>
  <Card title="Identity Graphs" icon="diagram-project" href="/concepts/identifiers/identity-graph">
    The foundational concepts behind identity graph structure
  </Card>

  <Card title="Addressability Expansion" icon="bullseye-arrow" href="/concepts/identifiers/addressability">
    The complementary approach: improving activation reach without changing graph structure
  </Card>

  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    How individual identifier linkages work
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Translating offline data for digital activation
  </Card>
</CardGroup>


# Graph Studio
Source: https://docs.narrative.io/concepts/identifiers/graph-studio

Build identity graphs from your first-party and third-party data using a guided, visual builder

Graph Studio is the platform's identity graph building tool. It provides two builders — **Edge Builder** and **Graph Builder** — that work together to transform your raw data into a connected identity graph.

## How it works

Building an identity graph is a two-step process:

1. **Define edges** — Tell the platform which identifiers in your data should be used to connect records. Two records that share the same identifier value (like the same email address) are linked together.

2. **Build the graph** — The platform runs a connected components algorithm over your edges to discover which records belong to the same person or household, resolving transitive connections across multiple hops.

## Edges

An edge connects a **source record** to a **target identifier**. The Edge Builder creates an edges dataset by combining your data sources with the identity attributes you choose as connection points.

### Key concepts

| Term               | Description                                                                              | Example                                            |
| ------------------ | ---------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **Source ID type** | A label you choose to identify which system a record comes from                          | `CRM`, `Website`, `Partner`                        |
| **Source ID**      | The field that uniquely identifies each record in that system                            | `CUSTOMER_ID`                                      |
| **Target ID type** | The category of identifier used as a connection point — always a Rosetta Stone attribute | `normalized_email`, `clear_text_e164_phone_number` |
| **Target ID**      | The actual identifier value for a given record, derived from the attribute mapping       | `michaelscott@gmail.com`, `+15705551234`           |

Each source record produces one edge per target ID group. When two records from any source share the same target ID value, the graph recognizes them as connected.

### Target ID groups

Target IDs are organized into **groups**. Each group defines one type of connection. A group can contain a single attribute or multiple attributes — when a group has multiple attributes, **all values in the group must match** for two records to be connected.

For example:

* A group with just **normalized email** connects any two records sharing the same email — high confidence, since email is typically unique to a person
* A group with **phone number + first name** requires both values to match, which is more precise than phone alone — useful when a phone number might be shared across a household

You can define multiple target ID groups to give the graph different ways to find connections. The algorithm considers all groups when resolving identities.

### Data sources

The Edge Builder accepts two types of sources:

* **Datasets** (first-party) — Your own data, [mapped to Rosetta Stone attributes](/guides/rosetta-stone/mapping-schemas)
* **Access rules** (third-party) — Data shared with you by other companies. Third-party sources introduce connections that your first-party data cannot see on its own.

## Graph

The Graph Builder takes one or more edges datasets and runs a **Label Connected Components** algorithm. It follows connections between records — including transitive chains — and groups every connected record into a single identity.

### Algorithm parameters

| Parameter                | Default | Description                                                                                                  |
| ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------ |
| **Max Component Size**   | 100     | Caps how many records can merge into one identity. Prevents over-connection.                                 |
| **Max Iterations**       | 10      | How many passes the algorithm makes to resolve transitive chains.                                            |
| **Max Degree Threshold** | 100     | Excludes nodes with too many connections (e.g., shared corporate emails) to avoid merging unrelated records. |

The defaults work well for most use cases.

### Output

The graph produces a dataset where each record is assigned a **component ID** — all records with the same component ID belong to the same resolved identity. You can join this back to your original data for analytics, segmentation, and activation.

### Automation

You can run the graph build once or set a **refresh schedule** to keep it current as source data changes. You can also optionally **encode** identifiers in the output using your company's encryption material.

## Source eligibility

Not every dataset or access rule appears in the Graph Studio source pickers. Each builder only lists sources that have been **prepared for the step you're on** — the Edge Builder lists sources that are ready to become edges, and the Graph Builder lists the edges datasets that are ready to be resolved into a graph. If a source you expect is missing, it hasn't met the requirement for that list.

Under the hood, eligibility is driven by dataset and access-rule **tags** and Rosetta Stone **attribute mappings**:

| Builder           | Source list                | Requirement                                                                                                                                        |
| ----------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Edge Builder**  | First-party datasets       | Dataset carries the `_nio_ci_components` tag                                                                                                       |
| **Edge Builder**  | Third-party access rules   | Access rule carries the `_nio_ci_components` tag                                                                                                   |
| **Graph Builder** | First-party input datasets | Dataset is mapped to the `graph_edge` Rosetta Stone attribute (this is what an edges dataset produced by the Edge Builder looks like)              |
| **Graph Builder** | Third-party access rules   | Access rule carries both the `_nio_ci_components` tag and a `_nio_ci_source_id_type:<name>` tag (for example `_nio_ci_source_id_type:CUSTOMER_ID`) |

<Note>
  The `_nio_ci_*` tags are managed on the dataset or access rule itself. First-party data becomes edge-eligible when it's tagged `_nio_ci_components`; the Edge Builder's output edges datasets are mapped to `graph_edge`, which is what makes them appear in the Graph Builder. If a source is missing from a picker, check that it carries the tag or mapping listed above.
</Note>

## Data plane support

Graph Studio runs on both **Snowflake** and **AWS** data planes. Both the Edge Builder and the Graph Builder are available in either environment — the Graph Builder uses the platform's [`LabelConnectedComponents` workflow task](/reference/workflows/tasks) for step 2, which is cross-platform.

## Related content

<CardGroup>
  <Card title="Identity Graphs" icon="circle-nodes" href="/concepts/identifiers/identity-graph">
    How connected components and graph structure unify identifiers
  </Card>

  <Card title="Building an Identity Graph" icon="hammer" href="/guides/graph-studio/building-an-identity-graph">
    Step-by-step guide to creating your first graph
  </Card>

  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    Strengthen graph structure with third-party linkage data
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Map your data to Rosetta Stone attributes
  </Card>
</CardGroup>


# ID Mapping
Source: https://docs.narrative.io/concepts/identifiers/id-mapping

Connecting identifiers across devices, channels, and platforms

ID mapping connects different identifiers to the same user, enabling cross-device targeting, measurement, and analytics. When a data provider links a hashed email to a mobile advertising ID, or a cookie to a device identifier, they create an ID mapping that bridges previously siloed data.

## Why ID mapping matters

Modern consumers interact with brands across multiple touchpoints:

* Mobile apps (identified by MAID)
* Websites on desktop (identified by cookies)
* Email communications (identified by email address)
* Connected TV (identified by device graphs)

Without ID mapping, each touchpoint exists in isolation. A retailer might show the same ad to a customer on mobile, desktop, and CTV without recognizing it's the same person—wasting budget and creating a poor experience.

ID mapping solves this by creating links between identifiers, enabling:

* **Unified targeting:** Reach the same person across devices
* **Frequency capping:** Limit ad exposure across channels
* **Attribution:** Connect conversions to the right touchpoints
* **Audience expansion:** Find customers in new channels

***

## Common ID linkages

### Hashed Email to MAID

The most valuable and deterministic mapping connects hashed emails to mobile advertising IDs.

**How it works:** When a user logs into a mobile app with their email address, the app developer can link the MAID (from the device) to the hashed email (from the login). This creates a deterministic, high-confidence link.

**Use cases:**

* Target email subscribers in mobile apps
* Extend CRM audiences to mobile advertising
* Measure mobile conversions against email campaigns

### Cookie to MAID

Connects web browsing behavior to mobile device identity.

**How it works:**

* Cookie syncing through pixel fires on mobile web
* SDK-based matching when users interact on both web and app
* Probabilistic matching based on shared signals (IP, location, behavior)

**Use cases:**

* Retarget website visitors in mobile apps
* Build unified user profiles across web and app
* Measure cross-platform campaign performance

### Cookie to Hashed Email

Links browser-based identity to email-based identity.

**How it works:** When users log in on websites, the site can link their cookie to their hashed email. This creates a durable identifier that persists even when cookies are cleared.

**Use cases:**

* Build persistent identity across browser sessions
* Connect anonymous browsing to known customers
* Enable email-based audience activation

***

## ID mapping quality

Not all ID mappings are equal. Consider these quality factors:

### Deterministic vs. probabilistic

| Type              | Description                                               | Confidence |
| ----------------- | --------------------------------------------------------- | ---------- |
| **Deterministic** | Direct observation of link (e.g., same user logged in)    | High       |
| **Probabilistic** | Inferred from signals (e.g., shared IP, similar behavior) | Variable   |

Deterministic mappings are more accurate but harder to scale. Probabilistic mappings offer broader reach but may include false matches.

### Recency

ID mappings can become stale:

* Users reset their MAIDs
* Cookies expire or are cleared
* Email addresses change

More recent mappings are generally more reliable.

### Source quality

The method of collection matters:

* **First-party login data:** Highest quality
* **SDK integrations:** High quality
* **Cookie syncing:** Medium quality
* **Probabilistic inference:** Variable quality

***

## ID mapping in Narrative

Narrative enables ID mapping through several mechanisms:

### Standardized identifiers

When ingesting data, Narrative normalizes identifier formats to enable matching across sources:

* Consistent casing for MAIDs
* Standardized hashing for emails and phones
* Narrative Cookie IDs for cross-supplier cookie matching

### Cross-supplier matching

Different data suppliers use different cookie namespaces. Narrative creates a standardized cookie ID that links to each supplier's cookies, enabling matching across suppliers.

### Query-time joining

Use NQL to join datasets on shared identifiers:

```sql theme={null}
SELECT
  a.hashed_email,
  b.maid
FROM company_a."email_data" a
JOIN company_b."maid_data" b
  ON a.narrative_id = b.narrative_id
```

***

## Privacy considerations

ID mapping must respect privacy regulations and user expectations:

* **Consent:** Ensure mappings are created with appropriate user consent
* **Pseudonymization:** Always hash PII before creating mappings
* **Data minimization:** Only create mappings needed for specific use cases
* **User controls:** Honor opt-outs across all linked identifiers

***

## Related content

<CardGroup>
  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Categories of identifiers used in mapping
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Connect offline data to online identifiers
  </Card>

  <Card title="UID2" icon="shield-check" href="/concepts/identifiers/uid2">
    Privacy-preserving universal identifier
  </Card>

  <Card title="Narrative ID" icon="fingerprint" href="/concepts/security/narrative-id">
    Secure cross-partner matching
  </Card>
</CardGroup>


# Identifier Types
Source: https://docs.narrative.io/concepts/identifiers/id-types

Understanding the categories of identifiers used in data collaboration

Most data records in Narrative's platform have one or more identifiers as attributes. Identifiers act as unique representations of underlying concepts—typically the device, browser, or user where data was collected. Since an identifier can represent different concepts, all identifiers include a type designation so you know what the identifier represents.

## Why identifier types matter

When you query data, you need to know what kind of identifier you're working with:

* A **cookie** represents a browser session on a specific domain
* A **mobile ad ID** represents a mobile device
* A **hashed email** represents a person across devices

Understanding identifier types helps you:

* Match data across the right dimensions
* Estimate reach and overlap correctly
* Apply appropriate privacy controls

***

## Identifier categories

### Mobile Ad IDs (MAIDs)

Mobile advertising identifiers are device-level identifiers provided by mobile operating systems. They're accessible through platform APIs and persist until the user resets them.

| Platform       | ID Type                   | Format                                                 |
| -------------- | ------------------------- | ------------------------------------------------------ |
| Apple iOS      | `idfa`                    | Uppercase UUID: `918F1D4F-D195-4A8B-AF47-44683FE11DB9` |
| Google Android | `adid` (also called GAID) | Lowercase UUID: `3f097372-f01e-4b64-984c-395ae5828ee6` |

<Note>
  Mobile ad ID availability has decreased significantly due to privacy changes. Apple's App Tracking Transparency (ATT) requires user opt-in, and many users decline.
</Note>

**Related:** [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids) for a deeper dive into IDFA and GAID.

***

### Cookies

When data is collected in a web browser context (desktop or mobile web), the identifier is typically a cookie. Cookies are storage mechanisms tied to a specific domain—a cookie set by one website cannot be read by another.

Because cookies are domain-specific, Narrative doesn't have a single "cookie" identifier type. Instead, each data provider has their own cookie namespace. Cookie identifiers are useful for:

* Retargeting within a publisher's properties
* Frequency capping across a provider's inventory
* Cross-device matching when linked to other identifiers

<Warning>
  Third-party cookies are being deprecated in major browsers. Chrome has announced plans to phase them out, and Safari and Firefox already block them by default.
</Warning>

***

### Hashed Emails

Email addresses are personally identifiable information (PII) and cannot be shared in raw form. To enable matching while preserving privacy, emails are hashed using one-way cryptographic functions before being shared.

| ID Type        | Hash Algorithm | Output Length     |
| -------------- | -------------- | ----------------- |
| `md5_email`    | MD5            | 32 hex characters |
| `sha1_email`   | SHA-1          | 40 hex characters |
| `sha256_email` | SHA-256        | 64 hex characters |

**Important formatting rules:**

1. Convert to lowercase before hashing
2. Remove leading/trailing whitespace
3. Apply the hash algorithm

```
Email: JohnDoe@Gmail.com
Formatted: johndoe@gmail.com
SHA-256: 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
```

Hashed emails are valuable because:

* They persist across devices (people use the same email everywhere)
* They're deterministic (same email always produces the same hash)
* They can't be reversed to reveal the original email

**Related:** [Data Pseudonymization](/concepts/security/data-pseudonymization) for more on hashing techniques.

***

### Hashed Phone Numbers

Similar to emails, phone numbers are hashed to enable matching while protecting PII.

| ID Type        | Hash Algorithm | Format                  |
| -------------- | -------------- | ----------------------- |
| `md5_phone`    | MD5            | E.164 format, then hash |
| `sha1_phone`   | SHA-1          | E.164 format, then hash |
| `sha256_phone` | SHA-256        | E.164 format, then hash |

Phone numbers should be normalized to E.164 format before hashing:

* Include country code
* Remove spaces, dashes, parentheses
* Example: `+14155551234`

***

### Universal IDs

Emerging universal identifier systems aim to provide privacy-preserving, cross-platform identity without relying on cookies or device IDs.

| ID Type  | Description                                                     |
| -------- | --------------------------------------------------------------- |
| `uid2`   | Unified ID 2.0 - generated from hashed emails, supports opt-out |
| `rampid` | LiveRamp's identity resolution identifier                       |
| `id5`    | ID5's probabilistic and deterministic identity solution         |

**Related:** [UID2](/concepts/identifiers/uid2) for details on Unified ID 2.0.

***

## Choosing the right identifier type

| Use Case                   | Recommended Identifier Types      |
| -------------------------- | --------------------------------- |
| Cross-device targeting     | Hashed email, UID2                |
| Mobile app targeting       | IDFA, GAID                        |
| Web retargeting            | First-party cookies, hashed email |
| Offline-to-online matching | Hashed email, hashed phone        |
| Privacy-first campaigns    | UID2, first-party data            |

***

## Related content

<CardGroup>
  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    Connect identifiers across devices and channels
  </Card>

  <Card title="Mobile Ad IDs" icon="mobile" href="/concepts/identifiers/mobile-ad-ids">
    Deep dive into IDFA and GAID
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How hashing protects identifiers
  </Card>

  <Card title="Hashing PII for Upload" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Step-by-step guide to formatting and hashing
  </Card>
</CardGroup>


# Identity Graphs
Source: https://docs.narrative.io/concepts/identifiers/identity-graph

How connected components and graph structure unify identifiers into person-level and household-level profiles

An identity graph is a data structure that connects identifiers—emails, device IDs, phone numbers, cookies—across devices, channels, and platforms into unified profiles. Each cluster of connected identifiers (a [connected component](/reference/glossary#connected-component)) represents an individual or household.

Organizations use identity graphs to move from fragmented, device-level data to a coherent view of their customers. The quality of that graph directly determines the quality of downstream analytics, segmentation, and activation.

## What is an identity graph

An identity graph consists of three elements:

* **Nodes** are individual identifiers: a hashed email, a mobile advertising ID (MAID), a cookie, a phone number, a postal address
* **Edges** are linkages between identifiers, representing evidence that two identifiers belong to the same person or household
* **Connected components** are clusters of nodes where every identifier is reachable from every other through some path of edges—each component represents a resolved identity

For example, a single customer might generate nodes for their work email, personal email, iPhone IDFA, Android GAID, and home address. As the graph discovers linkages between these identifiers, they merge into a single connected component.

## Why graph quality matters

Graph quality is a balance between two failure modes:

| Problem             | Cause                                | Consequence                                                                       |
| ------------------- | ------------------------------------ | --------------------------------------------------------------------------------- |
| **Under-connected** | Too few or too conservative linkages | Fragmented user profiles, duplicated outreach, inflated audience counts           |
| **Over-connected**  | Too many or too liberal linkages     | Merged distinct individuals, corrupted targeting, wasted spend on wrong audiences |

Under-connected graphs treat the same person as multiple people. Over-connected graphs treat multiple people as one. Both degrade every downstream use case—segmentation, frequency capping, attribution, and personalization.

## How the graph grows

Identity graphs typically build in layers:

1. **First-party data as foundation.** Login events, CRM records, and transaction data create high-confidence, deterministic linkages. A customer who logs into your app with their email on their iPhone creates a direct edge between that hashed email and that device's IDFA.

2. **Third-party data adding edges.** External identity providers contribute additional linkages that your first-party data cannot observe. A provider might link your customer's email to a second device ID or a postal address that you have never seen.

3. **Deterministic vs. probabilistic linkages.** Deterministic linkages come from direct observation (same user logged in on two devices). Probabilistic linkages are inferred from signals like shared IP addresses, co-location patterns, or behavioral similarity. Deterministic linkages are more reliable but harder to scale; probabilistic linkages offer broader reach but carry higher false-match risk.

## Two approaches to using third-party identity data

When organizations purchase third-party identity data, the data serves one of two distinct purposes:

|                               | Graph enrichment                                     | Addressability expansion                                           |
| ----------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------ |
| **Primary benefit**           | Improved identity resolution                         | Improved media activation reach                                    |
| **Effect on graph structure** | Adds or strengthens edges between nodes              | Appends identifiers to existing nodes                              |
| **Impact on segmentation**    | Can change segment composition                       | Does not change segment composition                                |
| **Impact on match rates**     | Indirect (better resolution enables better matching) | Direct (more identifiers per record increases match rates)         |
| **Risk profile**              | Higher (bad linkages corrupt graph structure)        | Lower (bad appends reduce match rates but don't corrupt structure) |

Understanding which approach you need—or whether you need both—is the first decision in any identity data strategy.

<CardGroup>
  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    Strengthen your identity graph's structure with third-party linkage data
  </Card>

  <Card title="Addressability Expansion" icon="bullseye-arrow" href="/concepts/identifiers/addressability">
    Improve downstream activation reach by appending additional identifiers
  </Card>

  <Card title="Evaluating Identity Data Providers" icon="scale-balanced" href="/cookbooks/use-cases/identity-provider-evaluation">
    A structured framework for choosing and measuring identity data providers
  </Card>
</CardGroup>

***

## Identity graphs in Narrative

Narrative's platform connects to identity graph concepts in several ways:

### Graph Studio

[Graph Studio](/concepts/identifiers/graph-studio) is the platform's visual identity graph builder. It provides a guided workflow for defining edges between your data sources and running connected components algorithms to resolve identities. You can build graphs from first-party data alone, or combine first-party and third-party sources for stronger resolution. See the [building an identity graph](/guides/graph-studio/building-an-identity-graph) guide to get started.

### Rosetta Stone and unique identifiers

Rosetta Stone's `unique_identifier` attribute provides a standardized way to match identifiers across suppliers. When you join on `unique_identifier.value` and `unique_identifier.type`, Narrative handles format normalization—consistent MAID casing, standardized email hashing—so that identifiers from different sources match correctly.

### Cross-supplier identity resolution

Different data suppliers contribute different identity linkages. Narrative enables you to evaluate and combine these linkages through NQL queries, comparing coverage, overlap, and incremental contribution across providers before committing to a purchase.

### Narrative ID

[Narrative ID](/concepts/security/narrative-id) provides a privacy-preserving mechanism for cross-partner matching. It enables identity resolution across organizations without exposing raw identifiers, maintaining graph connectivity while protecting PII.

***

## Related content

<CardGroup>
  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    How individual identifier linkages work
  </Card>

  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Categories of identifiers used in data collaboration
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Translating offline data for digital activation
  </Card>

  <Card title="Data Activation" icon="rocket" href="/concepts/data-activation/overview">
    Delivering resolved audiences to destination platforms
  </Card>
</CardGroup>


# Mobile Advertising IDs
Source: https://docs.narrative.io/concepts/identifiers/mobile-ad-ids

Understanding IDFA, GAID, and how mobile device identifiers work

Mobile advertising identifiers (MAIDs) are unique pseudo-anonymous identifiers tied to mobile devices. Both iOS and Android provide APIs that allow developers to collect data linked to these identifiers, enabling advertising, analytics, and measurement without exposing personally identifiable information.

## How MAIDs work

Mobile operating systems generate a unique identifier for each device that app developers can access through platform APIs. This identifier:

* Is pseudo-anonymous (doesn't directly identify a person)
* Persists across app installs and sessions
* Can be reset or disabled by the user
* Is accessible to any app with appropriate permissions

MAIDs were designed as a privacy-conscious alternative to hardware identifiers like IMEI or MAC address, which cannot be reset and could be used for persistent tracking without user control.

***

## Platform implementations

### Apple iOS: IDFA

Apple's Identifier for Advertisers (IDFA) is available on iOS devices through the [ASIdentifierManager](https://developer.apple.com/documentation/adsupport/asidentifiermanager) API.

**Format:** Uppercase UUID with hyphens

```
918F1D4F-D195-4A8B-AF47-44683FE11DB9
```

**Key characteristics:**

* 32 hexadecimal characters separated by hyphens
* Uppercase by convention
* Same across all apps on the device (when available)

<Warning>
  Since iOS 14.5, Apple's App Tracking Transparency (ATT) framework requires apps to request user permission before accessing the IDFA. Many users decline, significantly reducing IDFA availability.
</Warning>

### Google Android: GAID

Google's Advertising ID (GAID, also called Ad ID or AAID) is available on Android devices through [Google Play Services](https://support.google.com/googleplay/android-developer/answer/6048248).

**Format:** Lowercase UUID with hyphens

```
3f097372-f01e-4b64-984c-395ae5828ee6
```

**Key characteristics:**

* 32 hexadecimal characters separated by hyphens
* Lowercase by convention
* Requires Google Play Services (not available on some devices)

***

## Privacy considerations

MAIDs were designed with privacy in mind, providing several user controls:

### User reset

Both platforms allow users to reset their advertising identifier at any time, generating a new ID and breaking the connection to previously collected data.

### Opt-out mechanisms

Users can limit ad tracking or disable personalized advertising entirely:

* **iOS:** Settings > Privacy > Tracking > Allow Apps to Request to Track
* **Android:** Settings > Privacy > Ads > Delete advertising ID

When users opt out, the MAID may return as all zeros or be unavailable entirely.

### Declining availability

The mobile advertising ecosystem has shifted significantly:

* **Apple ATT:** Only \~25-30% of iOS users opt in to tracking
* **Android:** Google has announced plans to introduce similar privacy controls

This declining availability has driven adoption of alternative identifiers like [UID2](/concepts/identifiers/uid2) and increased focus on first-party data strategies.

***

## Using MAIDs in data collaboration

### Data collection context

MAIDs are typically collected in scenarios where apps have user permission:

* SDK integrations with user consent
* Real-time bidding environments where MAIDs are passed in bid requests
* Publisher-direct integrations

### Matching considerations

When matching data using MAIDs:

* **Normalize format:** Ensure consistent casing (uppercase for IDFA, lowercase for GAID)
* **Handle opt-outs:** Filter out all-zero identifiers
* **Consider recency:** Older MAIDs may have been reset

### Cross-device identity

MAIDs identify devices, not people. A person with multiple devices will have different MAIDs on each. To build cross-device identity, MAIDs are often linked to:

* Hashed emails (from app logins)
* Other device identifiers
* Universal IDs like UID2

**Related:** [ID Mapping](/concepts/identifiers/id-mapping) for connecting MAIDs to other identifier types.

***

## Related content

<CardGroup>
  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Overview of all identifier categories
  </Card>

  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    Connect MAIDs to other identifiers
  </Card>

  <Card title="UID2" icon="shield-check" href="/concepts/identifiers/uid2">
    Privacy-preserving alternative identifier
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How hashing protects identifiers
  </Card>
</CardGroup>


# Identifiers in Data Collaboration
Source: https://docs.narrative.io/concepts/identifiers/overview

Understanding how identifiers enable privacy-preserving data matching across organizations

Data collaboration depends on identifiers—the keys that connect records across different organizations and systems. When two companies want to find overlapping customers or match behavioral data to a customer file, identifiers make that connection possible.

But identifiers in data collaboration come with unique challenges: privacy regulations require protection of personal information, different organizations use different identifier systems, and the fragmented digital landscape means a single person might be represented by dozens of different identifiers across devices and channels.

## Why identifiers matter

Consider a retail company that wants to target its existing customers with digital advertising. They have email addresses from their CRM, but ad platforms work with mobile device IDs and cookies. Without identifier mapping, their customer data is siloed and unusable for digital activation.

Data collaboration platforms solve this by:

* **Pseudonymizing** identifiers (hashing emails) so they can be matched without exposing PII
* **Mapping** between identifier types (connecting hashed emails to mobile ad IDs)
* **Standardizing** identifier formats so data from different sources can be matched

## The identifier ecosystem

The digital advertising and data collaboration ecosystem uses several categories of identifiers:

| Category          | Examples                                 | Persistence           | Privacy Status   |
| ----------------- | ---------------------------------------- | --------------------- | ---------------- |
| **Device IDs**    | IDFA (iOS), GAID (Android)               | Resettable by user    | Pseudo-anonymous |
| **Browser IDs**   | Third-party cookies, first-party cookies | Session or persistent | Pseudo-anonymous |
| **Hashed PII**    | MD5/SHA-256 hashed emails and phones     | Permanent             | Pseudonymized    |
| **Universal IDs** | UID2, RampID, ID5                        | Varies                | Pseudonymized    |

Each identifier type serves different use cases and has different trade-offs in terms of reach, accuracy, and privacy.

## Identifier challenges

### Cross-device identity

A single consumer might use a smartphone, tablet, laptop, and smart TV—each with different identifiers. Connecting these identifiers to understand that they represent the same person is a core challenge in data collaboration.

### Privacy and consent

Privacy regulations (GDPR, CCPA) and platform policies (Apple's ATT, Google's Privacy Sandbox) increasingly restrict how identifiers can be collected and used. Organizations must balance data utility with privacy compliance.

### Identifier deprecation

The digital ecosystem is in flux. Third-party cookies are being phased out, mobile ad ID availability is declining, and new privacy-preserving identifiers are emerging. Data strategies must adapt to this changing landscape.

***

## Related content

<CardGroup>
  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Categories of identifiers used in data collaboration
  </Card>

  <Card title="Mobile Ad IDs" icon="mobile" href="/concepts/identifiers/mobile-ad-ids">
    Understanding IDFA, GAID, and mobile device identifiers
  </Card>

  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    Connecting identifiers across devices and channels
  </Card>

  <Card title="UID2" icon="shield-check" href="/concepts/identifiers/uid2">
    The Unified ID 2.0 privacy-preserving identifier
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Translating offline data for digital activation
  </Card>

  <Card title="Identity Graphs" icon="diagram-project" href="/concepts/identifiers/identity-graph">
    How connected components unify identifiers into person-level profiles
  </Card>

  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    Strengthening graph structure with third-party linkage data
  </Card>

  <Card title="Addressability Expansion" icon="bullseye-arrow" href="/concepts/identifiers/addressability">
    Improving activation reach by appending additional identifiers
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How hashing protects identifiers
  </Card>
</CardGroup>


# UID2 (Unified ID 2.0)
Source: https://docs.narrative.io/concepts/identifiers/uid2

A privacy-preserving identifier for cross-platform data collaboration

Unified ID 2.0 (UID2) is an open-source identifier framework designed to enable privacy-conscious data collaboration in a post-cookie world. Developed by the Identity Consortium with The Trade Desk as a primary operator, UID2 provides a deterministic, encrypted identifier derived from email addresses or phone numbers.

## Why UID2 exists

The digital advertising ecosystem faces a identity crisis:

* Third-party cookies are being deprecated
* Mobile ad ID availability is declining
* Privacy regulations require user consent and control

UID2 addresses these challenges by providing:

* **Privacy by design:** Users can opt out at any time
* **Transparency:** Open-source framework with clear governance
* **Interoperability:** Works across platforms and partners
* **Determinism:** Same input always produces the same output

***

## How UID2 works

### Token generation

UID2 tokens are generated from personally identifiable information (PII) through a multi-step process:

1. **Input:** Email address or phone number
2. **Normalization:** Standardize format (lowercase, trim whitespace)
3. **Hashing:** Apply SHA-256 hash
4. **Encryption:** Encrypt with rotating keys
5. **Output:** UID2 token

The resulting token is:

* Deterministic (same email always produces the same UID2)
* Encrypted (raw hash is not exposed)
* Refreshable (tokens can be regenerated with new keys)

### Token types

| Token Type     | Description                             | Use Case                         |
| -------------- | --------------------------------------- | -------------------------------- |
| **UID2 Token** | Encrypted, rotatable identifier         | Real-time bidding, ad serving    |
| **Raw UID2**   | Underlying identifier before encryption | Secure storage, internal systems |

### Key rotation

UID2 tokens are encrypted with keys that rotate periodically. This means:

* Tokens eventually expire and must be refreshed
* Compromised tokens become invalid
* Users can effectively reset their identity by regenerating tokens

***

## Privacy features

### User opt-out

UID2 includes a centralized opt-out mechanism. When users opt out:

* Their UID2 is added to a global opt-out list
* All participants must check this list before using UID2s
* The opt-out persists across all participating companies

### Transparency

The UID2 framework is open-source and governed by an independent consortium, providing:

* Clear documentation of how identifiers are generated
* Public audit of participating companies
* User-facing explanations of how data is used

### No cross-party tracking without consent

UID2 is generated from first-party data (emails collected with consent). It cannot be used to track users who haven't shared their information with a participating company.

***

## UID2 in Narrative

Narrative supports UID2 through automatic generation and native integration with The Trade Desk.

### Automatic generation

When you upload data with supported identifier types, Narrative can automatically generate UID2 tokens:

**Supported input attributes:**

* Hashed Email (any hash algorithm)
* SHA-256 Hashed Email
* Phone Number
* SHA-256 Hashed Phone Number

### Trade Desk integration

The Narrative TTD Connector enables seamless UID2 workflows:

1. Upload data with email or phone identifiers
2. Narrative generates UID2 tokens automatically
3. Activate audiences directly in The Trade Desk

### Existing UID2s

If your data already contains UID2 tokens (from advertising partners or other sources), Narrative's TTD Connector processes them directly, automatically mapping the UID2 field.

***

## UID2 vs. other identifiers

| Aspect               | UID2                | Hashed Email  | Cookies          | MAIDs            |
| -------------------- | ------------------- | ------------- | ---------------- | ---------------- |
| **Persistence**      | Until opt-out       | Permanent     | Session/limited  | Until reset      |
| **Cross-device**     | Yes                 | Yes           | No               | No               |
| **User control**     | Centralized opt-out | None          | Browser settings | Device settings  |
| **Privacy status**   | Enhanced            | Pseudonymized | Trackable        | Pseudo-anonymous |
| **Future viability** | High                | High          | Declining        | Declining        |

***

## Getting started with UID2

To use UID2 in Narrative:

1. Ensure your data includes email or phone identifiers
2. Hash identifiers before upload if not already hashed
3. Configure the TTD Connector for your destination
4. Narrative handles UID2 generation automatically

***

## Additional resources

* [The Trade Desk UID2 Initiative](https://www.thetradedesk.com/us/about-us/industry-initiatives/unified-id-solution-2-0)
* [UID2 Documentation](https://unifiedid.com/docs/intro)

***

## Related content

<CardGroup>
  <Card title="Identifier Types" icon="tags" href="/concepts/identifiers/id-types">
    Overview of all identifier categories
  </Card>

  <Card title="ID Mapping" icon="link" href="/concepts/identifiers/id-mapping">
    Connect identifiers across platforms
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How hashing protects identifiers
  </Card>

  <Card title="Hashing PII for Upload" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare your data for UID2 generation
  </Card>
</CardGroup>


# Concepts
Source: https://docs.narrative.io/concepts/index

Understand how Narrative I/O works—the architecture, primitives, and design decisions behind the platform

The Concepts section explains the *why* behind Narrative I/O. Rather than step-by-step instructions, you'll find explanations of how the platform works, why it's designed the way it is, and how different pieces fit together.

## Core Primitives

The fundamental building blocks of the platform.

<CardGroup>
  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and is processed
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    How Narrative orchestrates data collaboration
  </Card>

  <Card title="Execution Context" icon="crosshairs" href="/concepts/primitives/execution-context">
    How data plane, compute pool, database, and schema determine where operations run
  </Card>

  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    How compute resources are allocated for query execution
  </Card>

  <Card title="Dataset Statistics" icon="chart-column" href="/concepts/primitives/dataset-statistics">
    Column-level metrics that describe your data
  </Card>
</CardGroup>

## Rosetta Stone

Narrative's approach to schema normalization and data mapping.

<CardGroup>
  <Card title="Overview" icon="language" href="/concepts/rosetta-stone/overview">
    What Rosetta Stone is and why it matters
  </Card>

  <Card title="How It Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    The mechanics of automatic schema mapping
  </Card>

  <Card title="Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    How data gets normalized across sources
  </Card>
</CardGroup>

## NQL

The design and philosophy behind Narrative Query Language.

<CardGroup>
  <Card title="Design Philosophy" icon="compass-drafting" href="/concepts/nql/design-philosophy">
    Why NQL exists and how it differs from traditional approaches
  </Card>

  <Card title="SQL Comparison" icon="code-compare" href="/concepts/nql/sql-comparison">
    How NQL relates to SQL and when to use each
  </Card>

  <Card title="Type System" icon="shapes" href="/concepts/nql/type-system">
    How NQL handles data types and type safety
  </Card>

  <Card title="JOIN Performance" icon="chart-line" href="/concepts/nql/join-performance">
    Why certain JOIN patterns cause performance problems
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Pre-computed query results for performance and caching
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    How NQL optimizes materialized view refreshes
  </Card>
</CardGroup>

## Identifiers

How identifiers connect records across organizations and systems.

<CardGroup>
  <Card title="Identity Graphs" icon="diagram-project" href="/concepts/identifiers/identity-graph">
    How connected components unify identifiers into person-level profiles
  </Card>

  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    Strengthening graph structure with third-party linkage data
  </Card>

  <Card title="Addressability Expansion" icon="bullseye-arrow" href="/concepts/identifiers/addressability">
    Improving activation reach by appending additional identifiers
  </Card>
</CardGroup>

## Architecture

How the platform is built and why.

<CardGroup>
  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries are compiled, optimized, and executed
  </Card>

  <Card title="Data Flow" icon="arrows-split-up-and-left" href="/concepts/architecture/data-flow">
    How data moves through the platform
  </Card>

  <Card title="Sample Data" icon="flask" href="/concepts/architecture/sample-data">
    How samples cross to the control plane
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    How Narrative protects your data
  </Card>
</CardGroup>

## Workflows

Automating multi-step data pipelines.

<CardGroup>
  <Card title="Workflow Orchestration" icon="diagram-project" href="/concepts/workflows/workflow-orchestration">
    How workflows automate sequential data operations and relate to the jobs system
  </Card>
</CardGroup>

## Webhooks

Push-based notifications for job state changes.

<CardGroup>
  <Card title="Webhooks" icon="bell" href="/concepts/webhooks/overview">
    How webhooks deliver notifications when jobs change state
  </Card>
</CardGroup>

## MCP Server for Agents

Connect AI agents to Narrative.

<CardGroup>
  <Card title="MCP Server for Agents" icon="robot" href="/concepts/mcp/overview">
    How the Narrative MCP server lets AI agents interact with your data
  </Card>

  <Card title="Skills" icon="sparkles" href="/concepts/mcp/skills">
    Pre-built, harness-agnostic workflows that orchestrate the MCP server
  </Card>
</CardGroup>

## Model Inference

AI-powered operations within your data plane.

<CardGroup>
  <Card title="Overview" icon="brain" href="/concepts/model-inference/overview">
    How AI inference works within your infrastructure
  </Card>

  <Card title="Data Privacy" icon="shield" href="/concepts/model-inference/data-privacy">
    How inference keeps data secure in your data plane
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/concepts/model-inference/structured-output">
    How JSON Schema ensures predictable responses
  </Card>
</CardGroup>

## Apps

Third-party onboarding and account connection.

<CardGroup>
  <Card title="App Invites" icon="envelope-open-text" href="/concepts/apps/app-invites">
    How invite links enable third parties to connect accounts without Narrative authentication
  </Card>
</CardGroup>

## Security

Enterprise security and access management.

<CardGroup>
  <Card title="Single Sign-On (SSO)" icon="shield-keyhole" href="/concepts/security/sso">
    How SSO works with Narrative and why it matters for enterprise security
  </Card>

  <Card title="API Key Permissions" icon="lock-keyhole" href="/concepts/security/api-key-permissions">
    How scoped permissions protect your resources and enforce least privilege
  </Card>
</CardGroup>

## Compliance

Privacy regulations and data protection requirements.

<CardGroup>
  <Card title="GDPR" icon="shield-check" href="/concepts/compliance/gdpr">
    The EU's General Data Protection Regulation and its requirements
  </Card>

  <Card title="CCPA" icon="scale-balanced" href="/concepts/compliance/ccpa">
    California's privacy law and consumer rights framework
  </Card>
</CardGroup>


# Data Collaboration MCP Server
Source: https://docs.narrative.io/concepts/mcp/overview

How the Narrative MCP server lets AI agents interact with your data

The Narrative MCP Server for Agents is a remote [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) server that lets AI agents like Claude explore datasets, kick off queries, refresh materialized views, and manage your Narrative account — all from a single conversation. Instead of switching between a chat window and the Narrative UI, you tell the agent what you need and it operates the platform on your behalf.

## What is MCP?

MCP is an open protocol that standardizes how AI applications connect to external tools and data sources. It defines a universal interface so that any MCP-compatible AI client — Claude Code, Claude Desktop, Cursor, Windsurf, and others — can discover and call tools exposed by an MCP server.

The protocol handles:

* **Tool discovery.** The client asks the server what tools are available and receives their names, descriptions, and parameter schemas.
* **Authentication.** OAuth-based flows let the server verify your identity without you sharing API keys with the AI client.
* **Tool execution.** The client calls a tool on your behalf, and the server returns structured results that the AI can reason about.

## Why use an AI assistant with Narrative?

Working with data collaboration platforms often involves switching between multiple screens — searching for datasets, inspecting schemas, writing queries, monitoring jobs. An AI assistant connected via MCP can compress these multi-step workflows into a conversational loop:

* **Explore datasets naturally.** Ask "what datasets do I have about mobile identifiers?" instead of navigating through the UI. The assistant searches, describes schemas, and summarizes what it finds.
* **Write and validate NQL iteratively.** Describe what you want in plain language, and the assistant drafts NQL, validates it against your schema, and fixes errors before submitting — all without you writing a query from scratch.
* **Operate on datasets without context-switching.** Trigger samples, refresh materialized views, and recalculate statistics through conversation. The assistant polls job status and reports when operations complete.
* **Work across companies.** If you belong to multiple companies, the assistant can switch context on your behalf, making cross-tenant operations seamless.

## How it works

The Narrative MCP server sits between your AI client and the Narrative API:

1. **You connect once.** Add the server URL to your AI client's MCP configuration. The first time the assistant calls a tool, a browser window opens for you to log in with your Narrative credentials.
2. **The assistant discovers tools.** On connection, the client fetches the list of available tools — dataset operations, NQL execution, attribute search, and more.
3. **Tools call the Narrative API.** When the assistant uses a tool, the MCP server translates the request into the appropriate Narrative API calls, using your authenticated session.
4. **Results flow back as context.** The server returns structured responses that the assistant incorporates into the conversation, so it can chain operations (e.g., describe a dataset, then write a query against it).

The server accepts two kinds of bearer credential, so the same endpoint works for interactive and programmatic clients:

* **OAuth login** — Interactive MCP clients (Claude Code, Claude Desktop, Cursor, Windsurf) open a browser-based login, and the server exchanges the resulting credential for a scoped Narrative access token. Your session persists across tool calls, so you only log in once per session.
* **Narrative API token** — Server-to-server callers and scripted clients can send a long-lived Narrative API token directly in the `Authorization: Bearer` header, skipping the browser flow. The token's existing user, company, and permission scope apply. See [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant#narrative-api-token-for-programmatic-clients) for setup details.

## Available tools

The server organizes its tools into these categories:

| Category               | What you can do                                                                                       |
| ---------------------- | ----------------------------------------------------------------------------------------------------- |
| **Context**            | Inspect the current session, list your companies, switch the active company, search companies by name |
| **Datasets**           | Search and describe datasets, inspect column statistics, configure stats collection                   |
| **Dataset operations** | Request samples, refresh materialized views, recalculate statistics                                   |
| **Attributes**         | Search and describe Rosetta Stone attributes                                                          |
| **Access rules**       | Discover and inspect the access rules your company owns or has been granted                           |
| **Mappings**           | Create Rosetta Stone mappings from a dataset to an attribute                                          |
| **NQL**                | Validate queries and submit them for execution                                                        |
| **Jobs**               | Search and describe jobs to track asynchronous work — NQL runs, samples, materialized-view refreshes  |
| **Data planes**        | Discover available data planes and their compute pools                                                |
| **Workflows**          | Create, trigger, describe, and follow runs of multi-step workflows                                    |

See the [MCP server reference](/reference/integrations/mcp-server) for the complete tool catalog with parameters and examples.

## Other Narrative MCP servers

Narrative publishes a small suite of MCP servers alongside the Data Collaboration server. You can add any of them to your client independently:

| Server                                                       | URL                             | Auth  | What it does                                                      |
| ------------------------------------------------------------ | ------------------------------- | ----- | ----------------------------------------------------------------- |
| [Data Collaboration](/reference/integrations/mcp-server)     | `https://mcp.narrative.io/mcp`  | OAuth | Query datasets, run NQL, manage workflows                         |
| [Knowledge Base](/reference/integrations/mcp-knowledge-base) | `https://docs.narrative.io/mcp` | None  | Search the Narrative docs and answer questions with citations     |
| [Agent Feedback](/reference/integrations/mcp-agent-feedback) | `https://narrative.support/mcp` | None  | Submit structured product feedback and bug reports from a session |

All three are discoverable in the Narrative product under **Marketplace > MCP Servers**.

## Getting started

To connect your AI assistant to Narrative, follow the setup guide for your client:

<CardGroup>
  <Card title="Connecting to the Narrative MCP Server" icon="plug" href="/guides/mcp/connecting-an-ai-assistant">
    Step-by-step setup for Claude Code, Claude Desktop, Cursor, and other MCP clients
  </Card>

  <Card title="Tool Reference" icon="book" href="/reference/integrations/mcp-server">
    Complete catalog of available tools with parameters and usage patterns
  </Card>
</CardGroup>


# Skills
Source: https://docs.narrative.io/concepts/mcp/skills

Pre-built skills that automate common Narrative workflows on top of the MCP server

Narrative publishes **Skills** — pre-built workflows that orchestrate the [Data Collaboration MCP Server](/concepts/mcp/overview) to accomplish specific tasks. Where the MCP server exposes raw tools (search datasets, validate NQL, describe attributes), a skill packages those tools into an opinionated procedure for a higher-level goal, like "draft and run an NQL query" or "generate a Rosetta Stone mapping for this dataset."

A skill is harness-agnostic — it's a portable instruction file, not logic tied to a particular AI assistant. You browse and install skills from the [Skills catalog](https://app.narrative.io/marketplace/skills) in the Narrative marketplace. Narrative publishes them in two forms from the public [`narrative-io/narrative-skills-marketplace`](https://github.com/narrative-io/narrative-skills-marketplace) repository: as a [`skills` CLI](https://agentskills.io) marketplace that installs into any spec-compliant agent (`npx skills add narrative-io/narrative-skills-marketplace`), and as [Claude Code plugins](https://docs.claude.com/en/docs/claude-code/plugins) you install with `/plugin`.

## How skills relate to the MCP server

Skills are clients of the MCP server, not a replacement for it:

* **The MCP server** exposes platform operations as discoverable tools (`narrative_datasets_search`, `narrative_nql_validate`, `narrative_attributes_describe`, …). Any MCP-compatible client can call them.
* **A skill** is a markdown instruction file (`SKILL.md`) that tells an AI assistant how to combine those tools — in what order, with what guardrails, and when to ask for approval — to complete a specific task.

This means skills inherit MCP authentication, scoping, and audit trails. When a skill calls a tool, it runs as the authenticated user against the active company, just like any other MCP call.

## Plugins and runtimes

Each skill belongs to a **plugin** — a versioned bundle published to the Claude Code marketplace. Narrative ships three plugins today:

| Plugin               | Skills                                                                                                                                      | Focus                                                                  |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `narrative-common`   | Find Attribute, Write NQL, Design Analysis, Create Workflow, Generate Rosetta Stone Mappings, Apply Rosetta Stone Mappings, Profile Dataset | General-purpose authoring, querying, profiling, and Rosetta Stone work |
| `narrative-identity` | Generate Identity Graph, Triage Pre-Graph Data, Generate Match Report                                                                       | Identity resolution, graph building, and partner overlap               |
| `narrative-audience` | Create Lookalike Audience                                                                                                                   | Audience expansion and look-alike modeling                             |

Skills also declare a **runtime** that describes how they work:

* **Prompt only** — the skill is pure instruction. The assistant follows the procedure using MCP tools and never writes or submits code on your behalf.
* **Prompt + code** — the skill produces or submits artifacts (NQL queries, workflow YAML, mapping definitions) in addition to following the procedure. These skills are always approval-gated.

## The catalog

Available skills include:

<CardGroup>
  <Card title="Find Attribute" icon="magnifying-glass">
    Resolve a fuzzy description or schema shape to the canonical Rosetta Stone attribute ID.
  </Card>

  <Card title="Write NQL" icon="code">
    Draft, validate, and optionally run an NQL query against a Narrative dataset.
  </Card>

  <Card title="Design Analysis" icon="compass-drafting">
    Scope a fuzzy analytical question into a structured brief of query specifications.
  </Card>

  <Card title="Create Workflow" icon="diagram-project">
    Author and submit a Narrative workflow from a natural-language intent.
  </Card>

  <Card title="Generate Rosetta Stone Mappings" icon="wand-magic-sparkles">
    Generate, evaluate, and improve Rosetta Stone attribute mappings for a dataset.
  </Card>

  <Card title="Apply Rosetta Stone Mappings" icon="check-double">
    Apply generated mappings to a dataset via a one-shot, approval-gated workflow.
  </Card>

  <Card title="Generate Identity Graph" icon="share-nodes">
    Compose and submit an identity-graph workflow that unions edge sources and labels components.
  </Card>

  <Card title="Triage Pre-Graph Data" icon="shield-halved">
    Audit a dataset or access rule for bad edges before an identity-graph build.
  </Card>

  <Card title="Profile Dataset" icon="chart-column">
    Produce a coverage and quality profile of a dataset or access rule: fill rate, cardinality, ranges, and quality flags.
  </Card>

  <Card title="Generate Match Report" icon="arrows-left-right">
    Compare an owned dataset to a partner's marketplace data: overlap, match counts, and enrichment coverage.
  </Card>

  <Card title="Create Lookalike Audience" icon="users-viewfinder">
    Score a candidate population against a seed and materialize the closest matches as a new audience.
  </Card>
</CardGroup>

The Skills catalog page lists every published skill with its plugin, runtime, example prompts, accepted arguments, dependencies, and install commands for both the `skills` CLI and Claude Code.

## Arguments

Most skills expose a slash-command form with explicit arguments. The skill's detail page shows an **Arguments** table listing each flag, whether it's required, its default, and what it does. For example, `/write-nql` accepts `--dataset`, `--run`, `--limit`, and a free-text tail, so you can scope a single invocation:

```text theme={null}
/write-nql --dataset 123 --run --limit 50 distinct users in the last 30 days
```

When invoked without arguments, a skill walks you through its flow interactively.

## Dependencies

Skills can depend on other skills. For example, **Generate Identity Graph** delegates attribute lookup to **Find Attribute** rather than reimplementing it. When you install a plugin, every skill in that plugin becomes available — so dependencies within the same plugin are automatically satisfied. Cross-plugin dependencies are called out on the skill's detail page.

## Approval gating

Skills with the `prompt+code` runtime never modify your Narrative account without explicit approval. They draft the artifact (an NQL query, a workflow specification, a mapping), explain what it does, and wait for you to say "go" before calling a tool that submits work. This matters most for skills that generate workflows or write mappings, where the cost of an unintended run is highest.

## Next steps

<CardGroup>
  <Card title="Install a skill" icon="download" href="/guides/mcp/installing-skills">
    Add the Narrative marketplace with the `skills` CLI or as a Claude Code plugin.
  </Card>

  <Card title="Connect Claude Code to the MCP server" icon="plug" href="/guides/mcp/connecting-an-ai-assistant">
    Skills require an authenticated MCP connection. Start here if you haven't connected yet.
  </Card>

  <Card title="Browse the Skills catalog" icon="grid" href="https://app.narrative.io/marketplace/skills">
    See every published skill with example prompts and install commands.
  </Card>

  <Card title="MCP server tool reference" icon="book" href="/reference/integrations/mcp-server">
    The underlying tools that skills orchestrate.
  </Card>
</CardGroup>


# Data Privacy in Model Inference
Source: https://docs.narrative.io/concepts/model-inference/data-privacy

How Model Inference keeps your data secure within your infrastructure

Model Inference is designed with data privacy as a foundational principle. Unlike traditional AI services that require sending data to external providers, Model Inference runs entirely within your [data plane](/concepts/primitives/data-planes), ensuring your data never leaves your controlled infrastructure.

## The privacy challenge with external AI

When using external AI APIs (OpenAI API, Anthropic API, etc.), your data flows to third-party servers:

```mermaid theme={null}
sequenceDiagram
    participant App as Your Application
    participant API as External AI API
    participant Provider as AI Provider Servers

    App->>API: Send data for inference
    API->>Provider: Data processed externally
    Note over Provider: Data potentially logged,<br/>retained, or used for training
    Provider->>API: Return response
    API->>App: Response
```

This creates several concerns:

* **Data residency**: Data leaves your infrastructure and jurisdiction
* **Third-party access**: Provider can potentially access your data
* **Compliance complexity**: Must evaluate provider's data handling practices
* **Retention policies**: Provider may retain data for improvement or debugging

## How Model Inference protects your data

Model Inference eliminates these concerns by hosting models within your data plane:

```mermaid theme={null}
sequenceDiagram
    participant App as Your Application
    participant CP as Control Plane
    participant DP as Your Data Plane
    participant Model as Hosted LLM

    App->>CP: Submit inference request
    Note over CP: Only job metadata<br/>(no raw data)
    CP->>DP: Route job to data plane
    DP->>Model: Local inference
    Note over Model: Model runs in<br/>your infrastructure
    Model->>DP: Response
    DP->>CP: Job completion status
    CP->>App: Structured output
```

### Key privacy guarantees

| Aspect                  | Guarantee                                                    |
| ----------------------- | ------------------------------------------------------------ |
| **Data location**       | Data never leaves your data plane                            |
| **No external calls**   | No network requests to Anthropic, OpenAI, or other providers |
| **Your infrastructure** | Models run on compute within your environment                |
| **No provider logging** | AI providers have no access to your data                     |
| **Full control**        | You control data access, retention, and deletion             |

## Compliance implications

Model Inference simplifies compliance with data protection regulations:

### GDPR considerations

* **Data minimization**: Only necessary data is processed
* **Storage limitation**: You control retention within your data plane
* **Data transfers**: No cross-border transfers to AI providers
* **Processor agreements**: No need for DPAs with AI providers for inference

### CCPA considerations

* **Service provider status**: AI providers are not service providers for your data
* **Sale of data**: No data is shared with third parties
* **Right to delete**: Full control over data deletion

### Industry regulations

For industries with strict data handling requirements (healthcare, finance, government), Model Inference enables AI capabilities without the compliance burden of external AI services:

* **HIPAA**: PHI never leaves your controlled environment
* **PCI DSS**: Payment data stays within your secure perimeter
* **FedRAMP**: Data remains in authorized boundaries

## Audit trail

All inference jobs are tracked through Narrative's standard job system:

* Job creation timestamp
* Data plane where inference ran
* Model used
* Token usage metrics
* Job completion status

This provides a complete audit trail without exposing the actual data processed.

## What the control plane sees

The [control plane](/concepts/primitives/control-plane) only handles:

* Job routing and coordination
* Metadata about requests (model choice, configuration)
* Job status updates
* Token usage statistics

The control plane never receives or processes:

* Your prompt content
* Your data
* The model's responses

## Comparison with external AI

| Aspect               | External AI APIs              | Model Inference           |
| -------------------- | ----------------------------- | ------------------------- |
| Data location        | Provider's servers            | Your data plane           |
| Network traffic      | Data sent externally          | Local only                |
| Provider data access | Yes (per their policies)      | No                        |
| Compliance burden    | High (must evaluate provider) | Low (your infrastructure) |
| Audit complexity     | Must rely on provider logs    | Full control              |
| Data retention       | Provider-controlled           | You control               |

## Best practices

1. **Use appropriate models**: Don't send more context than necessary
2. **Review prompts**: Ensure prompts don't unnecessarily include sensitive data
3. **Monitor usage**: Track inference jobs through the job system
4. **Set retention policies**: Configure data plane retention appropriately

## Related content

<CardGroup>
  <Card title="Model Inference Overview" icon="brain" href="/concepts/model-inference/overview">
    How Model Inference works
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understanding data plane architecture
  </Card>

  <Card title="Security Model" icon="lock" href="/concepts/architecture/security-model">
    Platform security architecture
  </Card>

  <Card title="GDPR" icon="shield" href="/concepts/compliance/gdpr">
    GDPR compliance
  </Card>
</CardGroup>


# Model Inference
Source: https://docs.narrative.io/concepts/model-inference/overview

Run AI-powered operations within your data plane using hosted LLMs

Model Inference enables you to run large language model (LLM) operations directly within your [data plane](/concepts/primitives/data-planes). Unlike traditional AI APIs that require sending data to external services, Model Inference keeps your data secure within your own infrastructure while still leveraging powerful AI capabilities.

## Why Model Inference matters

When working with sensitive data, sending it to external AI services creates compliance and security risks. Model Inference solves this by hosting models within your data plane infrastructure:

| Traditional AI APIs                  | Model Inference                   |
| ------------------------------------ | --------------------------------- |
| Data sent to external servers        | Data stays in your infrastructure |
| Subject to third-party data policies | You control data residency        |
| Network latency to external services | Local execution within data plane |
| Compliance complexity                | Simplified compliance posture     |

This architecture enables AI-powered features—like auto-generating dataset descriptions or translating natural language to technical formats—without compromising your data governance.

## How Model Inference works

Model Inference operates through Narrative's [job queue](/reference/glossary#job-queue), following the same pattern as other asynchronous operations:

1. **Submit request**: Your application sends an inference request specifying the model, messages, and output schema
2. **Job creation**: The [control plane](/concepts/primitives/control-plane) creates an inference job and queues it
3. **Local execution**: The data plane [operator](/reference/glossary#operator-data-plane) picks up the job and runs inference locally
4. **Structured response**: Results are returned in a predictable format defined by your JSON Schema

```mermaid theme={null}
sequenceDiagram
    participant App as Your Application
    participant CP as Control Plane
    participant DP as Data Plane
    participant Model as Hosted LLM

    App->>CP: Submit inference request
    CP->>CP: Create inference job
    CP-->>App: Return job ID
    DP->>CP: Poll for jobs
    CP->>DP: Return inference job
    DP->>Model: Execute inference
    Model->>DP: Return completion
    DP->>CP: Report results
    App->>CP: Poll job status
    CP-->>App: Return structured output
```

## Key capabilities

### Supported models

Model Inference supports models from multiple providers, all hosted within your data plane:

| Provider  | Models                                               |
| --------- | ---------------------------------------------------- |
| Anthropic | Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5 |
| OpenAI    | GPT-4.1, o4-mini, GPT-oss-120b                       |

For detailed model specifications, see the official documentation from [Anthropic](https://docs.anthropic.com/en/docs/about-claude/models) and [OpenAI](https://platform.openai.com/docs/models).

### Structured output

Every inference request includes a JSON Schema that defines the expected response format. The model is constrained to return valid JSON matching your schema, making responses predictable and easy to parse programmatically.

```typescript theme={null}
const inferenceConfig = {
  output_format_schema: {
    type: 'object',
    properties: {
      summary: { type: 'string' },
      confidence: { type: 'number', minimum: 0, maximum: 1 },
      categories: {
        type: 'array',
        items: { type: 'string' }
      }
    },
    required: ['summary', 'confidence']
  }
};
```

This guarantees you receive a response with exactly the fields you expect, in the types you specify.

### Conversation context

Inference requests support multi-turn conversations through the messages array. Each message
has a role (`system`, `user`, or `assistant`) and a `content` array of content blocks. The
dominant block type is `text`; agent-loop flows additionally emit `tool_use` (the model
requesting a tool call) and `tool_result` (the response) blocks.

```typescript theme={null}
const messages = [
  {
    role: 'system',
    content: [{ type: 'text', text: 'You are a data classification assistant.' }]
  },
  {
    role: 'user',
    content: [{ type: 'text', text: 'Classify this dataset based on its columns...' }]
  }
];
```

<Note>
  **Legacy `{role, text}` shape is still accepted.** Older integrations that send
  `{ role: 'user', text: 'hello' }` continue to work — the API auto-canonicalizes the
  request into a single `text` content block. New code should use the content-block shape;
  responses always emit it.
</Note>

### Configuration options

Fine-tune model behavior with inference configuration parameters:

| Parameter        | Description                                  | Default       |
| ---------------- | -------------------------------------------- | ------------- |
| `max_tokens`     | Maximum tokens in the response               | Model default |
| `temperature`    | Randomness (0 = deterministic, 1 = creative) | Model default |
| `top_p`          | Nucleus sampling parameter                   | Model default |
| `stop_sequences` | Strings that stop generation                 | None          |

## Common use cases

Model Inference powers AI features throughout the Narrative platform:

* **Dataset descriptions**: Automatically generate human-readable descriptions from dataset metadata and samples
* **Schedule translation**: Convert natural language schedules ("every weekday at 9am") to CRON expressions
* **Data classification**: Categorize records based on content analysis
* **Schema suggestions**: Recommend Rosetta Stone attribute mappings based on column names and sample data

## Data privacy

Because inference runs within your data plane:

* **No external API calls**: Data is never sent to Anthropic, OpenAI, or any external service
* **Your infrastructure**: Models run on compute resources within your data plane
* **Compliance-friendly**: Simplifies GDPR, CCPA, and other regulatory requirements
* **Audit trail**: All inference jobs are logged through the standard job system

For more details, see [Data Privacy in Model Inference](/concepts/model-inference/data-privacy).

## Related content

<CardGroup>
  <Card title="Running Model Inference" icon="play" href="/guides/sdk/running-model-inference">
    Submit inference requests using the SDK
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/concepts/model-inference/structured-output">
    How JSON Schema ensures predictable responses
  </Card>

  <Card title="Supported Models" icon="microchip" href="/reference/model-inference/supported-models">
    Available models and their capabilities
  </Card>

  <Card title="Data Privacy" icon="shield" href="/concepts/model-inference/data-privacy">
    How inference keeps data secure
  </Card>
</CardGroup>


# Structured Output
Source: https://docs.narrative.io/concepts/model-inference/structured-output

How JSON Schema ensures predictable, typed responses from Model Inference

Structured output is a core capability of Model Inference that guarantees the model returns data in a predictable, machine-readable format. By providing a JSON Schema, you constrain the model's response to match your exact specification.

## The problem with unstructured output

Traditional LLM interactions return free-form text:

```
User: Classify this review as positive, negative, or neutral.
Model: Based on my analysis, I believe this review expresses a positive sentiment. The customer seems satisfied with their purchase.
```

This creates challenges:

* **Parsing complexity**: Must extract the actual classification from prose
* **Inconsistent formats**: Response structure varies between calls
* **Error-prone**: Regex or string matching can fail on edge cases
* **No type safety**: Can't validate response structure at compile time

## How structured output works

With structured output, you define the exact response format:

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
    confidence: { type: 'number', minimum: 0, maximum: 1 }
  },
  required: ['sentiment', 'confidence']
};
```

The model returns exactly what you specify:

```json theme={null}
{
  "sentiment": "positive",
  "confidence": 0.92
}
```

## Why structured output matters

### 1. Reliable automation

Structured output enables reliable automation pipelines:

```typescript theme={null}
// Process results immediately without parsing
const result = job.result.structured_output;
await database.insert({
  category: result.category,
  confidence: result.confidence,
  tags: result.tags
});
```

### 2. Type safety

With TypeScript, you get compile-time type checking:

```typescript theme={null}
interface ClassificationResult {
  category: 'retail' | 'finance' | 'healthcare';
  confidence: number;
}

const result = job.result as ModelInferenceRunResult<ClassificationResult>;
// TypeScript knows result.structured_output.category is a string
// and result.structured_output.confidence is a number
```

### 3. Schema validation

The model is constrained to produce valid JSON matching your schema:

* Required fields are always present
* Types match your specification
* Enum values are restricted to defined options
* Numeric constraints are enforced

### 4. Consistent integration

Every response has the same structure, making integration predictable:

| Without Schema            | With Schema              |
| ------------------------- | ------------------------ |
| Parse response text       | Direct property access   |
| Handle format variations  | Consistent structure     |
| Runtime validation needed | Schema-enforced validity |
| Type casting required     | Native types             |

## JSON Schema capabilities

Model Inference supports standard JSON Schema features:

### Basic types

```json theme={null}
{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "count": { "type": "integer" },
    "score": { "type": "number" },
    "active": { "type": "boolean" }
  }
}
```

### Constraints

```json theme={null}
{
  "type": "object",
  "properties": {
    "rating": {
      "type": "integer",
      "minimum": 1,
      "maximum": 5
    },
    "category": {
      "type": "string",
      "enum": ["A", "B", "C"]
    }
  }
}
```

### Nested structures

```json theme={null}
{
  "type": "object",
  "properties": {
    "analysis": {
      "type": "object",
      "properties": {
        "summary": { "type": "string" },
        "details": { "type": "string" }
      }
    },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  }
}
```

## Design principles

### 1. Define what you need

Only include fields your application will use:

```typescript theme={null}
// Good: Focused schema
const schema = {
  type: 'object',
  properties: {
    category: { type: 'string', enum: ['spam', 'not_spam'] },
    confidence: { type: 'number' }
  },
  required: ['category', 'confidence']
};

// Avoid: Overly broad schema
const schema = {
  type: 'object',
  properties: {
    category: { type: 'string' },
    confidence: { type: 'number' },
    reasoning: { type: 'string' },
    alternatives: { type: 'array' },
    metadata: { type: 'object' }
    // ... many unused fields
  }
};
```

### 2. Use enums for known values

Constrain categorical outputs to valid options:

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    priority: {
      type: 'string',
      enum: ['low', 'medium', 'high', 'critical']
    }
  }
};
```

### 3. Set appropriate bounds

Define numeric ranges when applicable:

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    confidence: {
      type: 'number',
      minimum: 0,
      maximum: 1,
      description: 'Confidence score from 0 (uncertain) to 1 (certain)'
    }
  }
};
```

### 4. Add descriptions

Help the model understand field semantics:

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    sentiment: {
      type: 'string',
      enum: ['positive', 'negative', 'neutral'],
      description: 'Overall emotional tone: positive for satisfaction, negative for complaints, neutral for factual'
    }
  }
};
```

## Common patterns

### Classification

```json theme={null}
{
  "type": "object",
  "properties": {
    "label": { "type": "string", "enum": ["class_a", "class_b", "class_c"] },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
    "reasoning": { "type": "string" }
  },
  "required": ["label", "confidence"]
}
```

### Extraction

```json theme={null}
{
  "type": "object",
  "properties": {
    "entities": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "value": { "type": "string" },
          "type": { "type": "string" }
        },
        "required": ["value", "type"]
      }
    }
  },
  "required": ["entities"]
}
```

### Transformation

```json theme={null}
{
  "type": "object",
  "properties": {
    "input_format": { "type": "string" },
    "output_format": { "type": "string" },
    "transformed_value": { "type": "string" }
  },
  "required": ["transformed_value"]
}
```

## Related content

<CardGroup>
  <Card title="Using Structured Output" icon="code" href="/guides/sdk/structured-inference-output">
    Practical guide to schema definition
  </Card>

  <Card title="JSON Schema Reference" icon="book" href="/reference/model-inference/json-schema-reference">
    Supported schema features
  </Card>

  <Card title="Model Inference Overview" icon="brain" href="/concepts/model-inference/overview">
    How inference works
  </Card>

  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Submit inference requests
  </Card>
</CardGroup>


# Chunking
Source: https://docs.narrative.io/concepts/nql/chunking

How NQL splits large queries into time-based segments for stability and cost efficiency

Chunking is an execution strategy that processes large datasets in smaller, time-bounded segments. It improves stability and cost efficiency without changing your query syntax—you write normal NQL, and the platform decides when and how to split the work.

## Why chunking matters

Very large datasets—from tens of gigabytes to hundreds of terabytes—present challenges for single-shot query execution:

**Stability.** Long-running scans can fail due to resource pressure, network timeouts, or cluster issues. A failure 4 hours into a 5-hour query means starting over from scratch.

**Cost control.** When a query fails late in execution, you've already consumed compute resources with nothing to show for it. The later the failure, the greater the waste.

**Scalability.** Single-pass processing doesn't scale predictably. A query that works for 10GB may fail at 100GB or behave differently at 1TB.

Chunking addresses these problems by capping the work performed in any single step.

## How chunking works

When NQL determines a query is eligible for chunked execution:

1. **Evaluation.** The query planner analyzes the query structure to determine if chunking can apply without changing results.

2. **Time-based splitting.** The platform divides the scan into ranges based on a last-modified timestamp column (typically `nio_last_modified_at` or `last_modified_at`).

3. **Independent execution.** Each chunk runs as a separate operation. If one chunk fails, only that chunk needs to retry—not the entire query.

4. **Result combination.** Results from all chunks are combined to produce the final output, semantically equivalent to running the query as a single operation.

This happens automatically. There are no user-facing configuration options or syntax changes required.

## Eligible queries

Chunking applies to queries that primarily scan a single dataset with standard projections and filters. These patterns work well:

* **Simple scans** with filters and projections from `company_data` datasets
* **Queries against `narrative.rosetta_stone`** with standard selection patterns
* **Materialized views** that read from a single source with straightforward transformations

### Example: A chunkable query

This materialized view is a good candidate for chunking:

```sql theme={null}
CREATE MATERIALIZED VIEW "large_user_enrichment"
  REFRESH_SCHEDULE = '@weekly'
  WRITE_MODE = 'overwrite'
AS
SELECT
  user_id,
  email.value AS email_sha256,
  country_code,
  last_seen_at
FROM company_data."large_source_dataset"
WHERE last_seen_at >= CURRENT_TIMESTAMP - INTERVAL '180' DAY
```

The query shape—a scan with filters and projections from a single dataset—makes it eligible. The platform may split work by last-modified ranges to improve reliability on very large inputs.

## Queries not eligible for chunking

To preserve correctness and keep the execution model predictable, certain query patterns are not chunked:

**Complex joins.** Multi-table joins, especially when join semantics span large time ranges, require seeing all matching rows together. Chunking could produce incorrect results.

**Unions and set operations.** Operations like `UNION`, `INTERSECT`, and `EXCEPT` combine results from multiple queries in ways that don't partition cleanly by time.

**Global aggregations.** Operations like `SUM`, `AVG`, or `COUNT` over the entire dataset need all rows in a single pass to produce correct results. (Group-by aggregations within chunks may still work.)

**Sorting and windowing.** `ORDER BY` across the full result set and window functions that depend on complete data can't be chunked without changing semantics.

**Table functions.** Functions that change scan behavior may not be compatible with time-based partitioning.

<Note>
  Queries that include these operations still run—they just execute as a single operation rather than being split into chunks.
</Note>

## Result consistency

Chunked execution aims to produce results equivalent to a single-pass run. In practice:

* **Results are materially the same** for typical analytical workloads
* **Minor differences** can occur at chunk boundaries due to processing order or concurrent updates to source data
* **Deterministic queries** over stable data produce identical results regardless of chunking

For most use cases, you won't notice whether a query was chunked or not—which is the point.

## Relationship to other optimizations

Chunking complements other NQL execution strategies:

**[Incremental View Maintenance](/concepts/nql/incremental-view-maintenance)** processes only changed data since the last refresh. Chunking handles the initial scan of large datasets; IVM handles ongoing updates efficiently.

**[Query Processing](/concepts/architecture/query-processing)** describes the full query lifecycle. Chunking is one optimization the query planner may apply during the optimization stage.

**[Materialized Views](/concepts/nql/materialized-views)** benefit from chunking when refreshing over very large source datasets. The refresh becomes more reliable without any changes to the view definition.

***

## Related content

<CardGroup>
  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Pre-computed query results that benefit from chunked refresh
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    Update views by processing only changed data
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries are compiled, optimized, and executed
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to creating materialized views
  </Card>
</CardGroup>


# NQL Design Philosophy
Source: https://docs.narrative.io/concepts/nql/design-philosophy

Why Narrative created a purpose-built query language for data collaboration

NQL (Narrative Query Language) is an interpreted language designed specifically for data collaboration. Rather than running directly against a database, NQL is parsed by the [control plane](/concepts/primitives/control-plane) and transpiled into native SQL for whatever database powers your [data plane](/concepts/primitives/data-planes)—whether that's Snowflake, Spark, or another supported engine.

## Why not just use SQL?

SQL was designed for querying a single database. Data collaboration introduces challenges that standard SQL doesn't address:

**Multiple database dialects.** Organizations store data in different systems—Snowflake, Spark, BigQuery, and others. Each has its own SQL dialect with subtle differences in date functions, string handling, NULL semantics, and syntax. A query written for Snowflake won't run on Spark without modification.

**Cross-organization permissions.** When querying data from multiple organizations, permissions must be checked before execution. Standard SQL has no concept of cross-organization [access rules](/concepts/primitives/access-rules) or field-level controls.

**Schema normalization.** Data from different sources uses different column names and formats. [Rosetta Stone](/concepts/rosetta-stone/overview) handles this normalization, but it requires a layer between your query and the underlying database.

**Optimization across data planes.** Queries that span multiple data planes need coordination that a single-database SQL engine can't provide.

NQL addresses these challenges by providing an abstraction layer that handles dialect differences, permission enforcement, and schema normalization transparently.

## NQL as an interpreted language

When you submit an NQL query, it doesn't execute directly. Instead, the control plane:

1. **Parses** the NQL syntax and validates the query structure
2. **Resolves** dataset references and applies Rosetta Stone mappings
3. **Checks** your permissions against access rules and field-level controls
4. **Optimizes** the execution plan for your target data plane(s)
5. **Transpiles** the query into the native SQL dialect of your data plane

This interpretation happens at query time, which means the same NQL query can run against different database systems without modification.

## Transpilation: One query, many dialects

Transpilation is the process of converting NQL into equivalent SQL for a specific database. Consider a simple date comparison:

<CodeGroup>
  ```sql NQL theme={null}
  SELECT user_id, email, event_date
  FROM my_dataset
  WHERE event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
  ```

  ```sql Snowflake SQL theme={null}
  SELECT user_id, email, event_date
  FROM my_dataset
  WHERE event_date >= DATEADD(DAY, -7, CURRENT_DATE())
  ```

  ```sql Spark SQL theme={null}
  SELECT user_id, email, event_date
  FROM my_dataset
  WHERE event_date >= DATE_SUB(CURRENT_DATE(), 7)
  ```
</CodeGroup>

The NQL query is identical regardless of where your data lives. The control plane transpiles it to the correct syntax for your data plane's database engine.

This abstraction extends beyond date functions to:

* **String operations** — Functions like `CONCAT`, `SUBSTRING`, and `TRIM` have different syntax across databases
* **NULL handling** — How NULLs behave in comparisons and aggregations varies by dialect
* **Type casting** — Explicit and implicit type conversions differ between systems
* **Window functions** — Partitioning and ordering syntax has dialect-specific nuances

## Benefits of the abstraction layer

Building NQL as an interpreted, transpiled language provides several advantages:

**Portability.** Your queries work regardless of which database powers your data plane. If you migrate from Snowflake to Spark, your NQL queries continue to work without modification.

**Governance at compile time.** Permissions are checked before the query reaches your data. If you don't have access to a dataset or field, the query fails at compilation—not after scanning millions of rows.

**Rosetta Stone integration.** Schema normalization happens during query compilation. You query standardized attribute names, and the transpiler maps them to the actual column names in each dataset.

**Cross-organization coordination.** For queries that span multiple data planes, the control plane can coordinate execution and optimize data movement.

**Consistent behavior.** NQL provides consistent semantics regardless of the underlying database. A query behaves the same way whether it runs on Snowflake or Spark.

**Explicit column selection.** NQL requires you to list columns explicitly rather than using wildcards like `SELECT *`. This design choice enables better cost transparency, governance compliance, and query optimization—particularly for [Rosetta Stone](/concepts/rosetta-stone/overview) queries where wildcards would expand across many underlying tables. See [Explicit Column Selection](/nql/general/explicit-columns) for details.

## Trade-offs

Every abstraction has costs. NQL's design involves these trade-offs:

**Dialect-specific features unavailable.** Some database-specific features and optimizations aren't exposed through NQL. If you need a Snowflake-specific function that NQL doesn't support, you can't use it directly.

**Compilation overhead.** Query interpretation adds latency before execution begins. For very simple queries, this overhead may be noticeable—though for most analytical workloads, execution time dominates.

**Debugging complexity.** When a query fails, you're working with NQL syntax while the underlying error may come from the transpiled SQL. Error messages bridge this gap, but the indirection adds complexity.

For most data collaboration use cases, these trade-offs are worthwhile. The ability to query across organizations, databases, and schemas with a consistent language outweighs the loss of dialect-specific features.

***

## Related content

<CardGroup>
  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How NQL queries are compiled, dispatched, and executed across data planes
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    The orchestration layer that handles NQL compilation and coordination
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and where transpiled queries execute
  </Card>

  <Card title="Rosetta Stone" icon="language" href="/concepts/rosetta-stone/overview">
    How Narrative normalizes schemas across organizations
  </Card>
</CardGroup>


# Incremental View Maintenance
Source: https://docs.narrative.io/concepts/nql/incremental-view-maintenance

How NQL optimizes materialized view refreshes by updating only changed data

Incremental View Maintenance (IVM) is an optimization technique that updates materialized views by processing only the changes in underlying data, rather than recomputing the entire result from scratch.

## Why IVM matters

Consider a materialized view over a billion-row dataset. If only 10,000 rows changed since the last refresh, a full refresh would reprocess all billion rows to incorporate those changes. IVM processes just the 10,000 changed rows and merges them into the existing result.

The benefits scale with dataset size:

**Performance.** IVM can reduce refresh time from hours to minutes or seconds. When only a small fraction of data changes between refreshes, the savings are dramatic.

**Resource efficiency.** Processing less data means lower compute costs. For views that refresh frequently, IVM can significantly reduce your infrastructure spend.

**Faster refresh cycles.** Because incremental refreshes are faster, you can refresh more frequently without proportionally increasing costs. This enables near-real-time analytics for workloads that would otherwise require daily batches.

## How IVM works

IVM tracks changes to underlying data and applies those changes to the materialized view result:

### Change detection

The system identifies which rows in the source data have been inserted, updated, or deleted since the last refresh. This is typically done by tracking timestamps or using change data capture mechanisms.

### Delta computation

Rather than running the full query, IVM computes only how the result should change based on the detected changes. For an aggregation like `SUM(amount) GROUP BY category`, an inserted row adds to the sum for its category; a deleted row subtracts from it.

### Merge strategies

The computed deltas are merged into the existing materialized view:

* **Insertions** add new rows or update aggregations
* **Deletions** remove rows or adjust aggregations
* **Updates** are typically handled as a deletion followed by an insertion

The result is mathematically equivalent to a full refresh, but computed much more efficiently.

<Tip>
  For explicit control over merge behavior—such as matching on specific keys and deciding whether to update or insert—see [Incremental Upserts with MERGE ON](/guides/nql/incremental-upserts).
</Tip>

## When IVM applies

Not all queries can be incrementally maintained. IVM works well for certain patterns:

**Simple projections and filters.** Selecting columns and filtering rows is straightforward to maintain incrementally—new rows that pass the filter are added, deleted rows are removed.

**Group-by aggregations.** Common aggregations like `COUNT`, `SUM`, `AVG`, `MIN`, and `MAX` can often be maintained incrementally. The system tracks partial aggregates per group and updates them as data changes.

**Inner joins.** When the join keys don't change, new matching pairs can be identified and added without reprocessing existing matches.

### Requirements for IVM

For IVM to work effectively, certain conditions must be met:

* **Deterministic change tracking**: The system must reliably identify what changed
* **Associative operations**: Aggregations must be computable from partial results
* **Stable keys**: Join and group keys should not change for existing rows

## When full refresh is needed

Some scenarios require recomputing the entire view:

**Schema changes.** If you alter the view definition, underlying tables, or query structure, a full refresh ensures consistency.

**Complex aggregations.** Operations like `MEDIAN`, `PERCENTILE`, or custom window functions may not support incremental computation because they depend on the entire dataset, not just deltas.

**Historical corrections.** When backfilling or correcting historical data that spans the entire view, a full refresh may be simpler and more reliable than computing complex deltas.

**Non-monotonic changes.** If changes invalidate previously computed results in non-obvious ways (like changing a dimension value that affects many groups), full refresh avoids subtle correctness issues.

## IVM and refresh scheduling

IVM complements scheduled refreshes. Consider this pattern:

| Refresh frequency | Without IVM                  | With IVM                       |
| ----------------- | ---------------------------- | ------------------------------ |
| Daily             | Processes full dataset daily | Processes only daily changes   |
| Hourly            | 24x daily compute            | Only new hourly data each time |
| Every 15 minutes  | Very expensive               | Efficient, near-real-time      |

With IVM, you can often increase refresh frequency without proportionally increasing costs, because each refresh processes only a fraction of the data.

## Trade-offs

IVM isn't free—it introduces some complexity:

**Change tracking overhead.** The system must track what changed, which has some storage and processing cost.

**Query restrictions.** Not all query patterns support incremental maintenance. Complex queries may fall back to full refresh.

**Correctness complexity.** Incremental computation must produce results identical to full refresh. For complex queries, this requires careful implementation.

For most analytical workloads with regular data updates, IVM provides significant benefits. For small datasets or infrequent refreshes, the overhead may not be worthwhile.

***

## Related content

<CardGroup>
  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    When and why to use materialized views
  </Card>

  <Card title="Chunking" icon="puzzle-piece" href="/concepts/nql/chunking">
    How large queries are split into time-based segments
  </Card>

  <Card title="Incremental Upserts" icon="code-merge" href="/guides/nql/incremental-upserts">
    Use MERGE ON for deduplication and updates
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to creating materialized views
  </Card>
</CardGroup>


# Understanding JOIN Performance
Source: https://docs.narrative.io/concepts/nql/join-performance

How query planners optimize JOIN operations and why certain patterns cause problems

Query performance is critical when working with large datasets in Narrative's Data Collaboration Platform. Understanding how query planners handle JOIN operations helps you write queries that execute efficiently at scale.

## How query planners optimize JOINs

When you write a JOIN in NQL, the query planner analyzes the join condition and chooses an execution strategy. For simple equality conditions like `A.key = B.key`, the planner can use highly optimized algorithms.

### Hash joins

The most efficient join algorithm for equality conditions is the hash join:

1. The planner builds a hash table from the smaller table's join keys
2. It scans the larger table and probes the hash table for each key
3. Matches are found in near-constant time (O(1) per lookup)

Hash joins work because a single equality condition provides a clear, unambiguous way to partition and match data. The hash function maps each key to exactly one bucket, making lookups extremely fast.

### Index utilization

When tables have indexes on join columns, the planner can use index lookups instead of full scans. Like hash joins, index-based strategies require predictable, single-condition lookups to work effectively.

### What makes a join condition optimizable

A join condition is optimizable when the planner can:

* Build a hash table or index on one side
* Probe that structure efficiently from the other side
* Match rows without examining every possible pair

Single-key equality joins (`A.key = B.key`) satisfy all these requirements. The planner knows exactly how to partition the data and find matches efficiently.

## Why OR conditions prevent optimization

When you introduce OR into a join condition, the optimization breaks down:

```sql theme={null}
ON A.user_id = B.user_id OR A.email = B.email
```

The query planner cannot create a single hash table that handles both conditions simultaneously. The OR means a row can match through either path, but the paths are independent—there's no single key to hash.

### Query planner limitations

Most query planners treat OR joins as a single complex condition rather than recognizing they could be split into two efficient joins. The planner isn't sophisticated enough to automatically:

* Run two separate joins and merge results
* Build multiple hash tables and probe them in parallel
* Recognize that UNION would produce the same result more efficiently

Instead, the planner falls back to less efficient strategies.

### What happens internally

When faced with an OR join, the query planner typically resorts to:

**Nested loop joins**: The planner iterates through one table and, for each row, scans the other table checking both conditions. This approach has O(n × m) complexity—for tables with millions of rows, this becomes computationally expensive.

**Spool operations**: Some planners create temporary storage structures (spools) to materialize intermediate results. In extreme cases, these spools can grow to enormous sizes. One documented example showed a spool containing 7 billion rows (112 GB) for what should have been a straightforward join.

**Constant scans and multiple passes**: The planner may create multiple internal join passes or constant scan operations to handle the OR logic, adding overhead with each pass.

### The exponential cost problem

The fundamental issue is that an OR join isn't really one join—it's conceptually two joins that need to be merged. When the planner treats it as one operation, the cost compounds. Instead of `cost(join1) + cost(join2)`, you get something closer to `cost(join1) × cost(join2)` due to the nested evaluation required.

## Performance impact in practice

The theoretical problems translate to dramatic real-world performance differences.

### Orders of magnitude slower

Queries that should complete in seconds can run for hours with OR in the join:

| Scenario           | With OR join | Without OR join | Improvement |
| ------------------ | ------------ | --------------- | ----------- |
| Two 10k-row tables | \~4 hours    | \~2 minutes     | 120x faster |
| Simple match query | 4 min 37 sec | 12 seconds      | 23x faster  |
| Large dataset join | 3 hr 55 min  | under 2 min     | 118x faster |

These aren't edge cases—they represent typical results when OR joins encounter non-trivial data volumes.

### Resource consumption

Beyond time, OR joins consume disproportionate resources:

* **Memory**: Spool operations and intermediate results can consume gigabytes
* **CPU**: Nested loop evaluation keeps processors busy with redundant work
* **I/O**: Multiple passes over data increase disk reads dramatically

In a shared platform environment, a single poorly-optimized query can impact other users' workloads.

### When queries never finish

For sufficiently large tables, OR joins may never complete. The exponential cost growth means that doubling the data size more than doubles the execution time. What works on test data may be completely impractical on production volumes.

## Patterns that enable optimization

Understanding what works helps you write efficient queries.

### Single-key equality: the optimal path

The ideal join condition is a simple equality on one column:

```sql theme={null}
ON A.key = B.key
```

This pattern enables hash joins, index utilization, and predictable performance as data scales.

### How UNNEST transforms OR into single-key joins

The UNNEST technique works by restructuring your data so that multiple possible keys become multiple rows, each with a single key:

```sql theme={null}
-- Before: Two keys per row, requiring OR
Purchases: {user_id: 123, email: "a@b.com", ...}

-- After UNNEST: Two rows, each with one key
Purchases_flat: {key: 123, ...}
Purchases_flat: {key: "a@b.com", ...}
```

Now the join uses a single `key` column, and the planner can apply its efficient algorithms. You've done the work of separating the conditions that the planner couldn't do automatically.

### How UNION separates work efficiently

The UNION approach explicitly runs two optimized joins and combines results:

```sql theme={null}
(SELECT ... JOIN ON condition1)
UNION
(SELECT ... JOIN ON condition2)
```

Each individual join is efficient. The UNION handles deduplication. The total cost is `cost(join1) + cost(join2) + cost(dedup)`—linear, not exponential.

## Related content

<CardGroup>
  <Card title="Avoid OR in JOIN Clauses" icon="ban" href="/guides/nql/query-optimization/avoid-or-in-join">
    Step-by-step guide to restructuring OR conditions
  </Card>

  <Card title="Query Optimization" icon="gauge-high" href="/guides/nql/query-optimization">
    More techniques for writing efficient queries
  </Card>

  <Card title="NQL Design Philosophy" icon="compass-drafting" href="/concepts/nql/design-philosophy">
    Why NQL is designed the way it is
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries are compiled, optimized, and executed
  </Card>
</CardGroup>


# Materialized Views
Source: https://docs.narrative.io/concepts/nql/materialized-views

How materialized views work in NQL and when to use them

A materialized view is a pre-computed query result stored as a dataset. Unlike a regular query that executes each time you run it, a materialized view stores the results physically, giving you the speed of reading a table with the flexibility of a query.

## Why materialized views exist

Materialized views solve three common problems in data collaboration:

**Query performance.** Complex queries with joins, aggregations, or filters can take significant time to execute. When you need to run the same query repeatedly—for dashboards, reports, or API responses—a materialized view lets you compute once and read many times.

**Data caching for collaboration.** When sharing data with partners, you often want to provide a curated, pre-processed dataset rather than raw tables. Materialized views let you define exactly what data is shared and keep it updated automatically.

**Scheduled refresh for near-real-time data.** By refreshing on a schedule, materialized views provide data that's fresh enough for most analytical workloads without the cost of real-time processing.

## Stored snapshot vs. dynamic retrieval

Understanding this distinction is key to using materialized views effectively:

| Aspect         | Regular query               | Materialized view         |
| -------------- | --------------------------- | ------------------------- |
| Execution      | Runs every time you query   | Runs only on refresh      |
| Data freshness | Always current              | As fresh as last refresh  |
| Speed          | Depends on query complexity | Fast reads (like a table) |
| Storage        | None                        | Stores full result set    |
| Cost           | Compute on every read       | Compute on refresh only   |

A regular query is like making a phone call—you get live information but it takes time to connect. A materialized view is like receiving a daily newsletter—the information is only as fresh as the last edition, but it's instantly available.

## When to use materialized views

Materialized views are well-suited for:

**Expensive aggregations that don't change frequently.** If you're computing daily summaries, monthly rollups, or other aggregations over large datasets, a materialized view can run this computation once per day instead of on every request.

**Cross-organization queries.** Queries that join your data with partner data or the Rosetta Stone can be expensive. Materialize the results to avoid repeated cross-organization computation.

**Dashboard and reporting data.** Dashboards often run the same queries many times as users refresh pages. A materialized view provides consistent, fast responses for analytical workloads.

**Data products for partners.** When sharing curated datasets with collaborators, materialized views let you define exactly what's included and refresh it automatically.

## When not to use materialized views

Materialized views aren't always the right choice:

**Constantly changing data.** If your underlying data changes every few seconds and you need up-to-the-second accuracy, the refresh overhead of a materialized view may not be worthwhile.

**Queries that are already fast.** Simple queries against small tables don't benefit from materialization. The added complexity and storage cost isn't justified.

**Storage-constrained environments.** Materialized views duplicate data. If storage costs are a concern, consider whether the performance benefits outweigh the storage requirements.

**Ad-hoc analysis.** For one-time queries or exploratory analysis, just run the query directly. Materialized views are for queries you'll run repeatedly.

## Refresh strategies

Materialized views must be refreshed to reflect changes in underlying data. NQL supports several approaches:

**Scheduled refresh.** Set a CRON expression or use presets like `@daily` or `@hourly` to refresh automatically. This is the most common approach for production workloads.

**Manual refresh.** Trigger a refresh via API when you know the underlying data has changed. Useful for event-driven workflows or testing.

**Incremental refresh.** For large datasets, [Incremental View Maintenance](/concepts/nql/incremental-view-maintenance) can update only the changed portions of the view rather than recomputing everything. This dramatically reduces refresh time for views over large datasets with relatively small changes.

### Choosing a refresh schedule

Consider these factors:

* **How fresh does the data need to be?** If daily is acceptable, don't pay for hourly refreshes.
* **How expensive is the underlying query?** Frequent refreshes of expensive queries increase costs.
* **When do underlying datasets update?** Schedule refreshes after upstream data is available.

## How materialized views relate to datasets

When you create a materialized view, the result becomes a dataset in Narrative I/O. This means:

* The view appears in your dataset catalog
* You can apply access controls and sharing rules
* Partners can query it like any other dataset
* You can see statistics, row counts, and schema information
* The view inherits the governance and compliance features of the platform

This makes materialized views a powerful tool for data products—you define the query once, and the platform handles keeping it fresh and making it available to authorized consumers.

## View datasets

A view dataset is a lightweight alternative to a materialized view. Instead of physically storing query results, a view dataset stores only the NQL query definition. When another query references a view dataset, the NQL is inlined and re-evaluated at query time—similar to a database view.

### When to use view datasets

View datasets are useful when:

* **Data freshness matters more than speed.** Because the query is re-evaluated every time, results always reflect the latest underlying data without waiting for a refresh.
* **You want to avoid data duplication.** View datasets don't store a copy of the results, so there's no additional storage cost.
* **You need a reusable query abstraction.** Define a complex query once and reference it by name in other queries.

### View dataset limitations

View datasets have several restrictions compared to materialized views:

| Feature                | Materialized view             | View dataset                            |
| ---------------------- | ----------------------------- | --------------------------------------- |
| Stores query results   | Yes                           | No                                      |
| Refresh schedule       | Supported                     | Not applicable                          |
| `MERGE ON` (upsert)    | Supported                     | Not supported                           |
| `PARTITIONED_BY`       | Supported                     | Not supported                           |
| Chunking strategy      | Supported                     | Not supported                           |
| `DELTA` syntax         | Supported                     | Not supported                           |
| Access rules           | Can be created                | Cannot be created                       |
| Connectors             | Can be attached               | Cannot be attached                      |
| Rosetta Stone mappings | Derived via attribute lineage | Uses explicit mappings from the dataset |

### Creating a view dataset

You create a view dataset by passing `create_as_view: true` when executing a query through the [SDK](/guides/sdk/executing-queries#creating-a-view-dataset) or API. The resulting dataset appears in your dataset catalog like any other dataset, but its NQL is inlined at compile time rather than being materialized into a physical table.

<Info>
  View datasets resolve their dependencies recursively up to a maximum depth of 5 levels. If a view dataset references another view dataset, the platform inlines each layer until it reaches physical datasets.
</Info>

## Attribute lineage

Attribute lineage refers to how Rosetta Stone attribute mappings are automatically preserved when creating materialized views. When source datasets already have attribute mappings, manually re-mapping these attributes for every derived view is redundant. The attribute lineage feature automates this preservation, maintaining data consistency across your data pipeline.

### When attribute mappings persist

Attribute mappings are automatically preserved in three scenarios:

**Direct selection of unaltered attributes.** When you select a Rosetta Stone attribute directly from a dataset or access rule without modifying its output through data functions, the attribute mapping persists. Aliasing the attribute (e.g., `SELECT email AS user_email`) does not count as altering the data—the mapping is still preserved.

**Selection from an existing materialized view.** Selecting attributes from an already-created materialized view preserves the attribute mappings. Since the source view has already maintained the integrity of its mappings, the new view inherits them.

**Comprehensive selection of unmanipulated fields.** When you select all fields or columns that constitute an attribute mapping without manipulating any of these elements, the resulting materialized view retains the original mappings.

### How lineage is implemented

The system uses several mechanisms to track and preserve attribute mappings:

**Implicit usage-based remapping.** The system automatically remaps attributes based on their usage within dataset properties. For example, if a dataset includes a `Raw Email` attribute mapped through a `value` property, creating a materialized view that selects the `value` property implicitly remaps the `Raw Email` attribute to the new view.

**Identity mappings for dataset attributes.** When attributes are queried from specific datasets using the `_rosetta_stone` namespace, the resulting materialized view inherits mappings to all properties of the selected attribute, regardless of what mappings exist in the source dataset.

**Identity mappings for rosetta\_stone attributes.** Similarly, querying attributes directly from a `rosetta_stone` table and creating a materialized view results in the view inheriting mappings to all properties of the queried attribute.

### Derived mapping properties

When mappings are inherited, certain properties are inferred or copied from the original:

| Property   | Behavior                                                                        |
| ---------- | ------------------------------------------------------------------------------- |
| **Scope**  | Set to `GLOBAL` for `rosetta_stone` attributes; otherwise inherited from source |
| **Source** | Typically set to `SYSTEM` for derived mappings                                  |
| **Tags**   | Copied from the original mapping                                                |
| **Status** | Inferred based on context                                                       |

### Handling multiple datasets

When multiple datasets contribute to a materialized view and have overlapping attribute mappings, a prioritization logic determines which mapping is preserved. The system typically favors the first dataset encountered in the query. This ensures deterministic behavior when the same attribute appears in multiple source datasets.

***

## Related content

<CardGroup>
  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to creating your first materialized view
  </Card>

  <Card title="Materialized View Syntax" icon="list-check" href="/nql/commands/create-materialized-view">
    Complete reference for all CREATE MATERIALIZED VIEW options
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    How NQL optimizes refreshes by updating only changed data
  </Card>

  <Card title="Chunking" icon="puzzle-piece" href="/concepts/nql/chunking">
    How large queries are split into time-based segments for stability
  </Card>
</CardGroup>


# Query Templates
Source: https://docs.narrative.io/concepts/nql/query-templates

Reusable query patterns with configurable placeholders

A query template is a reusable NQL pattern with placeholders that users fill in at execution time. Instead of writing a new query from scratch, users select a template, provide values for the variable parts, and run it. Templates make complex queries accessible to users who don't need to understand the underlying NQL syntax.

## Why query templates exist

Query templates solve three common challenges in data collaboration:

**Ease of customization.** Not everyone who needs data is comfortable writing NQL. Templates let you build sophisticated queries once and expose only the parameters that matter—date ranges, regions, thresholds—so others can customize without touching the query logic.

**Reusability.** When you find yourself writing similar queries repeatedly with minor variations, a template captures the pattern. Change the date range or filter criteria without rewriting the joins, aggregations, or transformations each time.

**Complexity management.** Large queries with multiple joins, conditional logic, and aggregations can be difficult to understand and modify. Templates hide this complexity behind a simple interface of named placeholders, reducing the cognitive load for users who just need to run the query.

## Templates vs. saved queries

Both templates and saved queries let you reuse NQL, but they serve different purposes:

| Aspect       | Saved query                          | Query template                           |
| ------------ | ------------------------------------ | ---------------------------------------- |
| Variability  | Static—runs the same way every time  | Dynamic—placeholders filled at execution |
| Audience     | Query author who wants quick access  | Anyone who needs customized results      |
| Modification | Edit the full NQL to change behavior | Fill in placeholders without editing NQL |
| Use case     | Personal shortcuts                   | Shared, parameterized queries            |

A saved query is like a bookmark—it gets you back to the same place quickly. A template is like a form—it captures a structure that different people can fill in differently.

## Placeholder types

Templates use placeholders to mark the variable parts of a query. Each placeholder has a type that determines what kind of value it accepts:

**Literal values.** Replace constants like strings, numbers, booleans, or dates. Use these for parameters like date ranges, price thresholds, or status filters.

**Columns.** Replace column names in SELECT or GROUP BY clauses. Use these when users need to choose which fields to include in results.

**Filters.** Replace entire WHERE conditions. Use these when the filtering logic itself varies, not just the filter values.

**Output fields.** Replace parts of the SELECT list. Use these when the shape of the output depends on user choice.

For syntax details on each placeholder type, see the [Query Template Syntax](/nql/commands/query-templates) reference.

## When to use templates

Templates work well for:

**Standard reports with variable parameters.** A monthly sales report where the user picks the month, region, and product category. The query structure stays the same; only the filter values change.

**Queries shared with partners.** When collaborators need to run queries against your data with their own parameters, templates let them customize without accessing the full query logic.

**Recurring patterns with variable inputs.** Audience building, cohort analysis, or data exports that follow a consistent pattern but need different inputs each time.

**Materialized views with configurable parameters.** Templates can define materialized views, letting users create scheduled data products with customized filters or aggregations.

## When not to use templates

Templates aren't always the right tool:

**One-time exploratory queries.** If you're investigating data ad-hoc and won't rerun the query, just write it directly.

**Simple queries with no variable parts.** If every aspect of the query is fixed, a saved query is simpler than a template with no placeholders.

**When full NQL flexibility is needed.** Templates constrain what users can change. If the use case requires modifying joins, adding columns, or restructuring the query, direct NQL access is more appropriate.

***

## Related content

<CardGroup>
  <Card title="Using Query Templates" icon="play" href="/guides/nql/using-query-templates">
    Step-by-step guide to finding, using, and creating templates
  </Card>

  <Card title="Query Template Syntax" icon="code" href="/nql/commands/query-templates">
    Complete reference for placeholder syntax and options
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Pre-compute query results for faster access
  </Card>

  <Card title="NQL Syntax" icon="terminal" href="/nql/general/syntax">
    Core NQL syntax reference
  </Card>
</CardGroup>


# NQL vs SQL
Source: https://docs.narrative.io/concepts/nql/sql-comparison

Understanding how NQL extends and differs from standard SQL

If you know SQL, you already know most of NQL. This page explains what's the same, what's different, and what NQL adds for data collaboration scenarios.

## What stays the same

NQL uses standard SQL syntax for core query operations. If you've written SQL queries before, these patterns work exactly as you'd expect:

### SELECT statements

```sql theme={null}
SELECT
  column1,
  column2,
  UPPER(column3) AS transformed
FROM table_reference
WHERE condition
ORDER BY column1 DESC
LIMIT 100
```

### Aggregations and grouping

```sql theme={null}
SELECT
  category,
  COUNT(1) AS count,
  SUM(amount) AS total,
  AVG(score) AS average
FROM table_reference
GROUP BY category
HAVING COUNT(1) > 10
```

### Joins

```sql theme={null}
SELECT t1.column1, t2.column2
FROM table1 t1
INNER JOIN table2 t2 ON t1.id = t2.foreign_id
LEFT JOIN table3 t3 ON t1.id = t3.ref_id
```

### Subqueries and CTEs

```sql theme={null}
WITH summary AS (
  SELECT category, SUM(amount) AS total
  FROM sales
  GROUP BY category
)
SELECT category, total FROM summary WHERE total > 1000
```

### INSERT, UPDATE, and DELETE

NQL supports the standard DML statements. `INSERT` accepts both `VALUES` and `SELECT` forms; `UPDATE` and `DELETE` take an optional `WHERE` clause:

```sql theme={null}
INSERT INTO company_data."123" (user_id, email)
VALUES (42, 'alice@example.com')

INSERT INTO company_data."456" (user_id, email)
SELECT user_id, email FROM company_data."123" WHERE is_active = TRUE

UPDATE company_data."123" SET status = 'archived' WHERE user_id = 42

DELETE FROM company_data."123" WHERE email IS NULL
```

See [INSERT](/nql/commands/insert), [UPDATE](/nql/commands/update), and [DELETE](/nql/commands/delete) for full references.

### Standard functions

Most SQL functions work as expected:

* String: `UPPER`, `LOWER`, `CONCAT`, `SUBSTRING`, `TRIM`
* Numeric: `ABS`, `ROUND`, `FLOOR`, `CEIL`
* Date: `CURRENT_DATE`, `DATE_TRUNC`, `EXTRACT`
* Aggregate: `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`

***

## What's different

### Table references use dataset IDs or names

In traditional SQL, you reference tables by name. In NQL, you reference datasets within the `company_data` schema by either their numeric ID or their dataset name:

<CodeGroup>
  ```sql Standard SQL theme={null}
  SELECT user_id, email FROM users
  SELECT event_type, timestamp FROM analytics.events
  ```

  ```sql NQL (by name) theme={null}
  SELECT user_id, email FROM company_data.my_users
  SELECT event_type, timestamp FROM company_data.web_events
  ```

  ```sql NQL (by ID) theme={null}
  SELECT user_id, email FROM company_data."123"
  SELECT event_type, timestamp FROM company_data."456"
  ```
</CodeGroup>

Numeric IDs must be quoted because they're numeric. Dataset names don't need quoting unless the name is a reserved keyword or contains special characters. The `company_data` schema contains your datasets.

<Note>
  Queries built with Data Studio's Query Builder render the FROM clause using dataset names by default, so the NQL editor view shows readable references like `company_data.my_users`. Numeric IDs are still valid everywhere, and Narrative I/O uses them internally where a stable identifier is required — for example, the NQL persisted with an [access rule](/concepts/primitives/access-rules) is always ID-qualified, since dataset names can be reused after a dataset is archived.
</Note>

### Special data sources

NQL provides access to shared resources that don't exist in traditional databases:

| Source                        | Purpose                             |
| ----------------------------- | ----------------------------------- |
| `narrative.rosetta_stone`     | Identity resolution across datasets |
| `provider_slug."access_rule"` | Data shared via access rules        |

```sql theme={null}
-- Identity resolution data
SELECT unique_id, event_timestamp
FROM narrative.rosetta_stone

-- Data from another company's access rule
SELECT user_id, activity_type, calories
FROM partner_company."fitness_data"
```

### Price filtering

Every dataset includes a `_price_cpm_usd` column representing the cost per 1,000 rows. This doesn't exist in traditional databases:

```sql theme={null}
-- Only get data priced at $1 or less per 1000 rows
SELECT user_id, email, created_at
FROM company_data."123"
WHERE _price_cpm_usd <= 1.00
```

***

## What NQL adds

### Budget controls

NQL includes budget clauses to control data spending—a concept that doesn't exist in traditional SQL:

```sql theme={null}
CREATE MATERIALIZED VIEW "my_data"
AS (SELECT user_id, email, created_at FROM company_data."123")
BUDGET 50 USD

-- Recurring budget limits
LIMIT 100 USD PER CALENDAR_MONTH
LIMIT 10 USD PER CALENDAR_DAY
```

### Materialized views with options

NQL's `CREATE MATERIALIZED VIEW` includes scheduling, partitioning, and metadata options beyond what most databases offer:

```sql theme={null}
CREATE MATERIALIZED VIEW "daily_summary"
REFRESH_SCHEDULE = '@daily'
DISPLAY_NAME = 'Daily Summary Report'
DESCRIPTION = 'Aggregated daily metrics'
EXPIRE = 'P30D'
TAGS = ('analytics', 'daily')
PARTITIONED_BY event_date DAY
AS (
  SELECT date, COUNT(1) as events
  FROM company_data."123"
  GROUP BY date
)
BUDGET 25 USD
```

### QUALIFY clause

While some databases support `QUALIFY`, it's not part of standard SQL. NQL includes it for filtering on window function results:

```sql theme={null}
-- Keep only the most recent record per user
SELECT user_id, email, last_seen
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

Without `QUALIFY`, you'd need a subquery:

```sql theme={null}
-- Equivalent without QUALIFY (more verbose)
SELECT user_id, email, last_seen
FROM (
  SELECT
    user_id, email, last_seen,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY last_seen DESC) AS rn
  FROM company_data."123"
) sub
WHERE rn = 1
```

### DELTA tables

Query only changed records since the last execution—useful for incremental processing:

```sql theme={null}
SELECT user_id, email, updated_at
FROM DELTA(TABLE company_data."123")
```

This capability requires the platform to track changes, which isn't standard SQL functionality.

### Rosetta Stone integration

Access identity resolution through special columns:

```sql theme={null}
SELECT
  company_data."1"._rosetta_stone.unique_id,
  company_data."1".user_attribute
FROM company_data."1"
```

### Narrative-specific functions

Functions designed for data collaboration scenarios:

```sql theme={null}
SELECT
  NORMALIZE_EMAIL(email) AS clean_email,
  NORMALIZE_PHONE('E164', phone, 'US') AS clean_phone,
  HASH(user_id, email) AS identity_hash,
  UNIVERSE_SAMPLE(user_id, 0.1) AS in_sample
FROM company_data."123"
```

***

## SQL features not supported

Some SQL features aren't available in NQL:

| Feature                | Status          | Alternative                                              |
| ---------------------- | --------------- | -------------------------------------------------------- |
| `SELECT *`, `COUNT(*)` | Not supported   | [List columns explicitly](/nql/general/explicit-columns) |
| `CREATE TABLE`         | Not supported   | Use `CREATE MATERIALIZED VIEW`                           |
| `CREATE INDEX`         | Not supported   | Partitioning provides similar benefits                   |
| Stored procedures      | Not supported   | —                                                        |
| Transactions           | Not supported   | Each query is atomic                                     |
| `UNION`                | Limited support | Some restrictions apply                                  |

***

## Dialect differences

NQL is transpiled to different database engines (Snowflake, Spark). The transpiler handles dialect differences, but awareness helps when debugging:

### Date functions

<CodeGroup>
  ```sql NQL theme={null}
  -- NQL uses consistent syntax
  WHERE event_date >= CURRENT_DATE - INTERVAL '30' DAY
  ```

  ```sql Snowflake (transpiled) theme={null}
  WHERE event_date >= DATEADD(DAY, -30, CURRENT_DATE())
  ```

  ```sql Spark (transpiled) theme={null}
  WHERE event_date >= DATE_SUB(CURRENT_DATE(), 30)
  ```
</CodeGroup>

### NULL handling

NQL follows standard SQL NULL semantics, but the underlying engine may have subtle differences. The transpiler normalizes most cases.

***

## Migration tips

### Coming from PostgreSQL/MySQL

1. Reference datasets through the `company_data` schema, by name or ID: `users` → `company_data.users` or `company_data."123"`
2. Add budget clauses to materialized views
3. Consider `_price_cpm_usd` filtering for cost control

### Coming from Snowflake/BigQuery

1. Dataset references live under the `company_data` schema and can use the dataset name or numeric ID
2. Most functions work the same
3. `QUALIFY` is fully supported
4. Window functions work as expected

### Coming from Spark SQL

1. Similar syntax for complex types (arrays, structs, maps)
2. `UNNEST` and `LATERAL` joins work as expected
3. UDFs are replaced with Narrative-specific functions

***

## Related content

<CardGroup>
  <Card title="NQL Design Philosophy" icon="compass-drafting" href="/concepts/nql/design-philosophy">
    Why Narrative created a purpose-built query language
  </Card>

  <Card title="NQL Syntax Reference" icon="code" href="/nql/general/syntax">
    Complete query structure and grammar
  </Card>

  <Card title="Write Your First Query" icon="play" href="/getting-started/first-nql-query">
    Hands-on tutorial to get started
  </Card>

  <Card title="Functions Reference" icon="function" href="/nql/functions/index">
    All available functions
  </Card>
</CardGroup>


# Type System
Source: https://docs.narrative.io/concepts/nql/type-system

How NQL handles types across different database engines

NQL's type system bridges the gap between your queries and multiple underlying database engines. Understanding how types work helps you write portable queries and avoid common pitfalls.

## Why NQL needs its own type system

Different database engines handle types differently:

* **Snowflake** uses `NUMBER`, `VARCHAR`, `VARIANT` for semi-structured data
* **Spark** uses `LONG`, `STRING`, and nested types with specific semantics
* **BigQuery** uses `INT64`, `STRING`, and `STRUCT`/`ARRAY` with its own rules

NQL provides a consistent type system that the control plane maps to the appropriate native types during transpilation. You write one query, and it works correctly regardless of where your data lives.

## Primitive types

NQL's primitive types map to standard database types across all supported engines:

| NQL Type             | Snowflake       | Spark       | Description           |
| -------------------- | --------------- | ----------- | --------------------- |
| `STRING` / `VARCHAR` | `VARCHAR`       | `STRING`    | Variable-length text  |
| `BOOLEAN`            | `BOOLEAN`       | `BOOLEAN`   | True/false            |
| `LONG` / `BIGINT`    | `NUMBER`        | `BIGINT`    | 64-bit integer        |
| `DOUBLE`             | `FLOAT`         | `DOUBLE`    | 64-bit floating point |
| `DECIMAL`            | `NUMBER`        | `DECIMAL`   | Arbitrary precision   |
| `DATE`               | `DATE`          | `DATE`      | Calendar date         |
| `TIMESTAMP`          | `TIMESTAMP_NTZ` | `TIMESTAMP` | Date and time         |

### When precision matters

For financial or measurement data where exact precision is required, use `DECIMAL` instead of `DOUBLE`:

```sql theme={null}
-- Floating-point: may have rounding errors
SELECT 0.1 + 0.2 AS float_result  -- Could be 0.30000000000000004

-- Decimal: exact precision
SELECT CAST(0.1 AS DECIMAL) + CAST(0.2 AS DECIMAL) AS decimal_result  -- Exactly 0.3
```

***

## Complex types

Data collaboration often involves hierarchical and nested data structures. NQL supports three complex types that work consistently across engines.

### Arrays

Arrays store ordered collections of same-type elements. They're useful for:

* Lists of identifiers
* Tags or categories
* Time-series values

```sql theme={null}
-- Creating arrays
SELECT ARRAY('email', 'phone', 'cookie') AS id_types

-- Accessing elements (0-based indexing)
SELECT identifiers[0] AS first_id FROM company_data."123"

-- Getting array size
SELECT SIZE(tags) AS tag_count FROM company_data."123"
```

**Why arrays exist**: Many identity resolution scenarios involve multiple identifiers per record. Arrays let you store and query these without flattening to separate rows.

### Structs

Structs group named fields with potentially different types—like a row within a row. They're useful for:

* Nested attributes
* Composite identifiers
* Grouped metadata

```sql theme={null}
-- Creating structs
SELECT NAMED_STRUCT('type', 'email', 'value', 'user@example.com') AS identifier

-- Accessing fields
SELECT person.name, person.age FROM company_data."123"

-- Deeply nested access
SELECT user.address.city FROM company_data."123"
```

**Why structs exist**: Real-world data is hierarchical. An identity might have a type and value; an address might have street, city, and postal code. Structs preserve this structure without requiring separate columns.

### Maps

Maps store key-value pairs where all keys share a type and all values share a type. They're useful for:

* Dynamic properties
* Metadata with variable keys
* Key-value attributes

```sql theme={null}
-- Accessing map values
SELECT ELEMENT_AT(properties, 'source') AS source FROM company_data."123"
```

**Why maps exist**: Some data has dynamic or sparse attributes. Rather than creating columns for every possible property, maps store only the properties that exist for each record.

***

## Nested types

Complex types can be nested to represent hierarchical data:

```sql theme={null}
-- Array of structs (common for identifiers)
ARRAY<STRUCT<type:STRING, value:STRING, timestamp:LONG>>

-- Struct with nested array
STRUCT<name:STRING, tags:ARRAY<STRING>>

-- Map with struct values
MAP<STRING, STRUCT<count:LONG, last_seen:TIMESTAMP>>
```

### Accessing nested data

Use dot notation for struct fields and bracket notation for arrays:

```sql theme={null}
SELECT
  -- Array of structs
  identifiers[0].type AS first_type,
  identifiers[0].value AS first_value,

  -- Nested struct
  user.preferences.language AS user_language,

  -- Map within struct
  ELEMENT_AT(user.metadata, 'source') AS source
FROM company_data."123"
```

***

## Type coercion

NQL automatically converts types in certain contexts to reduce the need for explicit casting.

### Implicit coercion

| Context              | Behavior                                         |
| -------------------- | ------------------------------------------------ |
| Numeric comparisons  | Smaller types promoted to larger (LONG → DOUBLE) |
| String concatenation | Non-strings converted to strings                 |
| Boolean context      | Some conversions allowed                         |

```sql theme={null}
-- Numeric promotion
SELECT 1 + 1.5  -- LONG + DOUBLE = DOUBLE

-- String concatenation
SELECT 'User #' || 123  -- Number converted to string
```

### When to use explicit CAST

Use `CAST` when:

* Converting between incompatible types
* Ensuring specific precision
* Documenting intent clearly

```sql theme={null}
-- Parse string as number
WHERE CAST(string_column AS LONG) > 100

-- Ensure decimal precision
SELECT CAST(amount AS DECIMAL) * CAST(rate AS DECIMAL)

-- Convert timestamp to date
SELECT CAST(event_timestamp AS DATE) AS event_date
```

***

## NULL handling

NULL represents missing or unknown data. Understanding NULL behavior prevents subtle bugs.

### NULL in comparisons

NULL is not equal to anything, including itself:

```sql theme={null}
-- These all return NULL, not TRUE or FALSE
SELECT NULL = NULL     -- NULL
SELECT NULL <> NULL    -- NULL
SELECT NULL = 'value'  -- NULL
```

Use `IS NULL` and `IS NOT NULL` for NULL checks:

```sql theme={null}
WHERE email IS NOT NULL
WHERE phone IS NULL
```

### NULL in operations

Operations involving NULL typically return NULL:

```sql theme={null}
SELECT NULL + 1           -- NULL
SELECT CONCAT('Hi ', NULL) -- NULL (in most engines)
```

### NULL-safe comparisons

Use `IS NOT DISTINCT FROM` for NULL-safe equality:

```sql theme={null}
-- Returns TRUE when both are NULL
WHERE a IS NOT DISTINCT FROM b
```

This is especially useful in MERGE conditions:

```sql theme={null}
MERGE ON target.id IS NOT DISTINCT FROM source.id
```

***

## Type inference

NQL infers types from context when possible:

```sql theme={null}
-- Literal type inference
SELECT 'text'           -- STRING
SELECT 123              -- LONG
SELECT 123.45           -- DOUBLE
SELECT true             -- BOOLEAN
SELECT DATE '2024-01-01'  -- DATE
```

### ARRAY type inference

Array literals infer element type from contents:

```sql theme={null}
SELECT ARRAY(1, 2, 3)         -- ARRAY<LONG>
SELECT ARRAY('a', 'b', 'c')   -- ARRAY<STRING>
```

***

## Cross-engine considerations

While NQL abstracts type differences, some edge cases exist:

### Timestamp precision

Different engines support different timestamp precision. NQL uses millisecond precision as the common denominator.

### String collation

String comparison and sorting may differ slightly between engines. For consistent behavior, normalize strings (lowercase, trim) before comparison.

### Numeric overflow

Large numbers may overflow differently across engines. For critical calculations, consider using DECIMAL with explicit precision.

***

## Troubleshooting type errors

### UnsupportedTypeError

This error occurs when an operation doesn't support the given types:

```sql theme={null}
-- Error: Can't compare STRING to LONG directly
WHERE string_column = 123
```

**Fix**: Use explicit CAST:

```sql theme={null}
WHERE CAST(string_column AS LONG) = 123
-- or
WHERE string_column = CAST(123 AS STRING)
```

### Type mismatch in arrays

All array elements must have the same type:

```sql theme={null}
-- Error: Mixed types in array
SELECT ARRAY(1, 'two', 3)
```

**Fix**: Ensure consistent types or cast explicitly:

```sql theme={null}
SELECT ARRAY(CAST(1 AS STRING), 'two', CAST(3 AS STRING))
```

### NULL type resolution

NULL by itself has no type. In some contexts, you need to cast NULL:

```sql theme={null}
-- May cause issues in some contexts
SELECT NULL

-- Explicit typed NULL
SELECT CAST(NULL AS STRING)
```

***

## Related content

<CardGroup>
  <Card title="Data Types Reference" icon="shapes" href="/nql/data-types">
    Complete type reference with syntax
  </Card>

  <Card title="NQL vs SQL" icon="code-compare" href="/concepts/nql/sql-comparison">
    How NQL types compare to standard SQL
  </Card>

  <Card title="Functions Reference" icon="function" href="/nql/functions/index">
    Type conversion and manipulation functions
  </Card>

  <Card title="Troubleshooting" icon="bug" href="/guides/nql/troubleshooting">
    Common type-related errors and solutions
  </Card>
</CardGroup>


# Access Rules
Source: https://docs.narrative.io/concepts/primitives/access-rules

Control who can access your data, what they can query, and what it costs

An access rule is a set of business rules that define how other organizations can access data within the Narrative platform. Access rules are a core primitive that give data owners fine-grained governance over their datasets.

## What access rules control

Access rules let you define three key aspects of data access:

**Who can access your data.** Specify which organizations can run queries against your datasets. You can grant access to specific companies, make data available to all platform participants, or restrict access to a closed group of partners.

**What data they can access.** Control which records and fields are available through filtering conditions. You might expose only certain date ranges, geographic regions, or record types to different partners.

**How much access costs.** Set pricing for data access on a per-record basis. Prices can vary by record type, buyer, or any other criteria you define. For non-commercial collaborations, set the price to zero.

## Why access rules exist

Without access rules, datasets in Narrative cannot be queried by other organizations. This is by design—data remains private by default, and you must explicitly create access rules to enable collaboration.

This approach ensures:

* **Data owners stay in control.** No one can access your data without your explicit permission
* **Flexible monetization.** Charge different rates for different data or different buyers
* **Governance compliance.** Audit exactly who has access to what data and under what terms

## How access rules work

Access rules are defined using NQL (Narrative Query Language) and are enforced at query execution time. When another organization queries data you own:

1. **Rule matching** — The control plane identifies which access rules apply to the requested data
2. **Permission verification** — The query is checked against the access rule's constraints
3. **Price calculation** — If the rule includes pricing, costs are calculated based on matching records
4. **Query execution** — Only records that pass the access rule's filters are included in results

```
┌─────────────────────────────────────────────────────────────────┐
│                       Query Submitted                           │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Access Rules Evaluated                          │
│   • Which rules match the requested datasets?                   │
│   • Does the requester have permission?                          │
│   • What are the pricing terms?                                  │
└───────────────────────────────┬─────────────────────────────────┘
                                │
                   ┌────────────┴────────────┐
                   │                         │
                   ▼                         ▼
        ┌──────────────────┐      ┌──────────────────┐
        │  Rules Match     │      │  No Match        │
        │  Query Executes  │      │  Query Fails     │
        └──────────────────┘      └──────────────────┘
```

## Common use cases

### Make data available to a specific partner

Grant a single organization access to your dataset:

```sql theme={null}
-- Access rule granting Partner Corp access to customer segments
SELECT
    segment_id,
    segment_name,
    audience_size
FROM company."customer_segments"
```

Configure the rule to be visible only to the partner's organization, and set pricing (or `$0` for a data share).

### Set different prices for different data

Create multiple access rules with different pricing tiers:

```sql theme={null}
-- Premium access rule: recent data at higher price
SELECT
    user_id,
    event_type,
    event_timestamp
FROM company."user_events"
WHERE event_timestamp > CURRENT_DATE - INTERVAL '30' DAY
```

```sql theme={null}
-- Standard access rule: older data at lower price
SELECT
    user_id,
    event_type,
    event_timestamp
FROM company."user_events"
WHERE event_timestamp <= CURRENT_DATE - INTERVAL '30' DAY
```

### Restrict fields for certain buyers

Expose different columns to different partners by creating separate access rules:

```sql theme={null}
-- Full access for premium partners
SELECT
    user_id,
    email,
    purchase_history,
    lifetime_value
FROM company."customers"
```

```sql theme={null}
-- Limited access for standard partners (no PII)
SELECT
    user_id,
    purchase_history
FROM company."customers"
```

### Enable non-commercial data sharing

Share data at no cost for research, partnerships, or data exchanges:

```sql theme={null}
SELECT
    anonymized_id,
    survey_responses,
    demographic_category
FROM company."research_data"
```

Set the price to `$0 CPM` when creating the rule.

## Attribute mappings

When a dataset has [Rosetta Stone mappings](/concepts/rosetta-stone/how-it-works), the API returns which normalized attributes each access rule exposes. The `mappings` array on each access rule response tells you:

* **Which attributes are mapped** — The Rosetta Stone attribute ID and name
* **Which properties are exposed** — The dot-separated paths of attribute properties that the rule's NQL query selects

For example, if a dataset column `idValue` is mapped to the `identity.value` property of an `identity` attribute, and the access rule's NQL selects `idValue`, the response includes:

```json theme={null}
{
  "mappings": [
    {
      "attribute_id": 42,
      "attribute_name": "identity",
      "properties": ["id.value"]
    }
  ]
}
```

Only properties that are both mapped **and** selected by the access rule's NQL query appear in the response. If a column is mapped to an attribute but the access rule does not select that column, the attribute is excluded. Scalar attributes (attributes without sub-properties) return an empty `properties` list.

<Note>
  Attribute mappings are filtered by the caller's permissions. You only see mappings for attributes your organization has viewing access to.
</Note>

## Access rules and datasets

Every dataset can have multiple access rules, and a single access rule can reference multiple datasets. This flexibility supports complex governance scenarios:

* **One dataset, many rules** — Different pricing tiers, different partner access levels
* **Many datasets, one rule** — Bundle related datasets for partners who need comprehensive access

<Note>
  If a dataset has no access rules, it cannot be queried by other organizations. You always retain access to your own datasets regardless of access rules.
</Note>

<Warning>
  Access rules cannot be created for [view datasets](/concepts/nql/materialized-views#view-datasets). Because a view dataset stores only a query definition rather than physical data, access must be controlled on the underlying datasets that the view references.
</Warning>

## Pricing with access rules

Access rule pricing uses **CPM (cost per thousand)**—the cost per 1,000 records. When a query matches an access rule with pricing:

1. The number of matching records is counted
2. The total cost is calculated: `(matching_records / 1000) × CPM`
3. The buyer is charged and the data owner receives payment

Pricing is enforced at the data plane level, meaning costs are calculated based on actual query results, not estimates.

## Related content

<CardGroup>
  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and access rules are enforced
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    How Narrative orchestrates permissions and query coordination
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    The broader security architecture including access controls
  </Card>

  <Card title="How Rosetta Stone Works" icon="wand-magic-sparkles" href="/concepts/rosetta-stone/how-it-works">
    Understand the attribute mappings returned by access rules
  </Card>

  <Card title="Access Rules API" icon="code" href="/api-reference/access-rules/get-an-access-rule-by-id-v2">
    Manage access rules programmatically
  </Card>

  <Card title="Name Conflict Errors" icon="clone" href="/reference/errors/access-rule-name-conflict">
    Resolve HTTP 409 errors when a name is already in use
  </Card>
</CardGroup>


# Compute Pools
Source: https://docs.narrative.io/concepts/primitives/compute-pools

Understand how compute pools control resource allocation for query execution within a data plane

A compute pool determines the compute resources allocated to process your queries within a [data plane](/concepts/primitives/data-planes). When you execute a query, the compute pool controls how much processing power is available and whether those resources are shared with other users or dedicated to your workload.

Compute pools are one of the four dimensions of your [execution context](/concepts/primitives/execution-context), alongside data plane, database, and schema.

## Compute pool types

### Dedicated

Dedicated compute pools provide isolated resources reserved for your workloads. Your queries don't compete with other users for processing power, which results in more predictable performance.

Use dedicated compute pools when:

* Running production workloads where performance consistency matters
* Processing large or complex queries that need guaranteed resources
* Operating time-sensitive pipelines where latency must stay predictable

### Shared

Shared compute pools use pooled resources across multiple users. This is more cost-effective but means your query performance may vary depending on current platform load.

Use shared compute pools when:

* Running exploratory queries or ad-hoc analysis
* Developing and testing queries before promoting to production
* Working with smaller datasets where performance variability is acceptable

### Snowflake warehouse

On Snowflake-based data planes, each compute pool maps to a Snowflake virtual warehouse. When you [register warehouses](/guides/data-planes/snowflake-native-app-installation) through the Snowflake Native App, each warehouse becomes a compute pool on your data plane. You can register multiple warehouses to separate workloads—for example, a smaller warehouse for exploratory queries and a larger one for production pipelines.

Each Snowflake compute pool has a collaboration policy that controls which companies can use it, and one pool can be designated as the default for the data plane.

### AWS EMR

On AWS-based data planes, compute pools map to Amazon EMR Spark clusters that the data plane operator provisions, reuses across jobs targeting the same pool, and terminates when idle. Each pool has a configured size that determines the cluster's worker memory budget and vCPU count. The operator schedules NQL jobs—including `materialize-view`, `nql-forecast`, `datasets_sample`, and `datasets_execute_dml`—onto the appropriate cluster based on the compute pool selected for the workload.

Sizes mirror Snowflake's warehouse vocabulary (`x_small` through `6x_large`) and target the **same worker memory budget** as the equivalent Snowflake warehouse, so a workload that fits in a Snowflake warehouse of size N gets equivalent RAM on the EMR cluster of size N. vCPU counts won't match Snowflake because EMR uses memory-optimized instances (8 GiB per vCPU), so the same memory budget delivers fewer vCPUs.

| Size       | Worker memory | vCPU (max) | Notes                                                  |
| ---------- | ------------- | ---------- | ------------------------------------------------------ |
| `x_small`  | \~32 GiB      | \~4        | Fixed size; rounded up from Snowflake's 16 GiB minimum |
| `small`    | \~32 GiB      | \~4        | Fixed size                                             |
| `medium`   | \~64 GiB      | \~8        | Fixed size                                             |
| `large`    | \~128 GiB     | \~16       | Fixed size                                             |
| `x_large`  | \~256 GiB     | \~32       | EMR Managed Scaling                                    |
| `2x_large` | \~512 GiB     | \~64       | EMR Managed Scaling                                    |
| `3x_large` | \~1 TiB       | \~128      | EMR Managed Scaling                                    |
| `4x_large` | \~2 TiB       | \~256      | EMR Managed Scaling                                    |
| `5x_large` | \~4 TiB       | \~512      | EMR Managed Scaling                                    |
| `6x_large` | \~8 TiB       | \~1024     | EMR Managed Scaling                                    |

Sizes `x_large` and above use EMR Managed Scaling: the cluster boots small and expands toward the maximum based on YARN load, then contracts back down when idle. Sizes `large` and below run at a fixed instance count.

#### Instance-storage (`_storage`) variants

Every size has a sibling `_storage` variant — `x_small_storage`, `small_storage`, `medium_storage`, `large_storage`, `x_large_storage`, `2x_large_storage`, `3x_large_storage`, `4x_large_storage`, `5x_large_storage`, `6x_large_storage` — with the **same worker memory budget and node counts** as its base size, but with core and task nodes running on Graviton instances with local NVMe SSD (`r7gd`/`r8gd`) instead of EBS-backed families. The master node stays on `m6g` (it does no shuffle).

Use a `_storage` variant when a job spills a large volume of shuffle or scratch data to disk and fails on an EBS-backed pool with `No space left on device`. Typical culprits are wide joins, large `GROUP BY` / `DISTINCT` aggregations, and materialized-view refreshes over very wide datasets. For jobs that are CPU- or memory-bound rather than shuffle-bound, stick with the base size — the local-NVMe families cost more per node.

You switch between the base and `_storage` tiers the same way you change any other size: update the pool's `provider.size` via [`PATCH /data-planes/{id}/compute-pools/{poolId}`](/api-reference/compute-pools/update-a-compute-pool), or create a new pool sized for the workload and pin it on the affected job or dataset.

#### Idle and job execution timeouts

EMR-backed pools expose two optional tunables that control cluster lifetime and per-job runtime. Both are set on the `aws_emr` provider when you [create](/api-reference/compute-pools/create-a-compute-pool) or [update](/api-reference/compute-pools/update-a-compute-pool) a pool, and both are validated server-side — invalid values return HTTP 400.

| Field                           | Range                           | Default on create  | What it controls                                                                                                                                                    |
| ------------------------------- | ------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `idle_timeout_seconds`          | `-1`, or `60`–`604800` (7 days) | `900` (15 minutes) | How long an EMR cluster sits idle before auto-termination. `-1` disables idle-termination entirely; the cluster runs until it is explicitly terminated or recycled. |
| `job_execution_timeout_seconds` | `60`–`604800` (7 days)          | `14400` (4 hours)  | Maximum time a single job's EMR step may stay in `RUNNING`. When exceeded, the operator cancels the step and fails the job with an explanatory message.             |

When either field is omitted (or set to `null`) on create **or update**, the API fills in the default above before validating and persisting the pool. Explicit values — including the `-1` disabled-idle sentinel — are always preserved. This matters when you update the pool's `size` (for example, switching to a `_storage` variant): the `provider` block is replaced wholesale rather than merged, so a size-only update that omits the timeout fields still gets the standard 4h/15m defaults instead of silently dropping them. Set `idle_timeout_seconds` to `-1` when you want a long-lived cluster — for example, a pool dedicated to back-to-back batch jobs where boot latency dominates run time.

<Note>
  The shared Narrative compute pool is configured with a 1-hour job execution timeout and is intended for small jobs. Pools you provision for yourself get the 4-hour default on create, and can set their own (longer or shorter) limit.
</Note>

## Which compute pools are available

The compute pool options available to you depend on your data plane's underlying provider:

| Provider                   | Available compute pools    | Notes                                                           |
| -------------------------- | -------------------------- | --------------------------------------------------------------- |
| **Snowflake**              | Snowflake warehouse        | One compute pool per registered warehouse                       |
| **Narrative (shared AWS)** | Dedicated, Shared          | Choose based on workload requirements                           |
| **Customer AWS**           | AWS EMR, Dedicated, Shared | EMR-backed pools support sized Spark clusters for NQL workloads |

You select your compute pool through the [context selector](/guides/platform/using-the-context-selector) in the platform's top navigation.

## Default compute pool resolution

When a job is created without an explicit compute pool, the platform resolves one through a **four-level fallback chain**. Resolution happens at job-creation time, so a job's `compute_pool_id` is fixed before it lands in `Pending`. The first level present wins.

| Level                                   | Source                                                                               | When it applies                                                                                                                                                                                                                                                                          |
| --------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. **Job-specific**                     | The `computePoolId` passed in the request body (or a workflow task input)            | A specific job needs a non-default pool                                                                                                                                                                                                                                                  |
| 2. **Dataset default**                  | `dataset.computePoolConfig.defaultComputePoolId`                                     | Dataset-scoped operations (refresh, sample, execute-DML, stats) when the request didn't pin a pool                                                                                                                                                                                       |
| 3. **Company default (per data plane)** | `companies.compute_pool_config.by_data_plane[<dataPlaneId>].default_compute_pool_id` | The company has a catch-all default for the job's data plane and nothing earlier supplied a pool. Scoped per data plane because a company can own pools across multiple data planes. Covers the wider surface (model training, model inference, healthcheck) that has no dataset analog. |
| 4. **Data plane default**               | `dataPlane.defaultComputePoolId`                                                     | Catch-all when nothing else resolved                                                                                                                                                                                                                                                     |

A company admin can set the level-3 default for a given data plane with:

```bash theme={null}
PUT /company/{companyId}/data-planes/{dataPlaneId}/default-compute-pool/{computePoolId}
```

and clear it with:

```bash theme={null}
DELETE /company/{companyId}/data-planes/{dataPlaneId}/default-compute-pool
```

Both endpoints require the **Company Info** write permission and return `204 No Content` on success.

<Tip>
  Use a **dataset default** for "all my materialize-view refreshes on this dataset run on a `large` pool", and a **company default** for the broader set of operations that have no dataset to attach to — model inference, healthchecks, or non-dataset NQL workloads. The two configs are intentionally separate so each can grow operation-specific knobs without polluting the other.
</Tip>

## When to use each type

| Scenario                          | Recommended pool    | Why                                                     |
| --------------------------------- | ------------------- | ------------------------------------------------------- |
| Production data pipelines         | Dedicated           | Predictable performance, no resource contention         |
| Ad-hoc data exploration           | Shared              | Cost-effective for variable, low-priority workloads     |
| Testing queries before production | Shared              | Saves dedicated resources for production use            |
| Time-sensitive audience builds    | Dedicated           | Guaranteed resources ensure timely completion           |
| Snowflake data planes             | Snowflake warehouse | Register one or more warehouses sized for your workload |

## Archiving compute pools

Deleting a compute pool is a soft delete: the pool's status changes to `archived`, the record stays in place, and it is filtered out of list responses but still resolvable by id. Archival behavior diverges by provider because each integration has different operational realities:

| Reference path                                    | Snowflake data planes                                                                                                                                              | AWS data planes                                                                          |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Pool is the data plane default                    | The platform re-elects another active pool as the new default (or clears the default if none remain), then archives the pool                                       | The archive is rejected; you must clear or change the data plane default before retrying |
| Pool is set as a dataset default                  | The platform clears the default on every affected dataset, then archives the pool. Affected datasets fall through to the rest of the chain on their next job       | The archive is rejected; you must clear the dataset defaults before retrying             |
| Pool is set as a company default (per data plane) | The platform clears the per-data-plane entry on every affected company, then archives the pool. Companies fall through to the data plane default on their next job | The archive is rejected; you must clear the company defaults before retrying             |
| Pool has in-flight `Pending` or `Running` jobs    | Archive succeeds; jobs are handled at runtime (see below)                                                                                                          | Archive succeeds; jobs are handled at runtime (see below)                                |

Snowflake users frequently mutate their warehouse list outside the platform — through the Snowflake UI, permissions, or infrastructure as code. When a backing warehouse disappears, the platform has to track reality, so the archive must succeed and dependent references are updated automatically. AWS compute pools are wholly managed inside the platform, so the API rejects an archive that would leave a dangling default and forces an explicit decision.

### Runtime impact on in-flight jobs

Neither archive flow blocks in-flight jobs at archive time. Instead, the data plane operator fails any job whose compute pool resolves to a non-`active` status on its next polling iteration. The job reports as failed with an actionable message similar to:

```text theme={null}
compute pool '<id>' could not be resolved (it may have been archived or does not exist).
Resubmit the job with an active compute pool.
```

This is uniform across `Pending`, `Running`, and `PendingCancellation` jobs. Jobs in `PendingCancellation` are failed rather than cancelled because the missing pool is the actionable signal — a successful cancellation report would mask the root cause.

See the [Archive a compute pool](/guides/data-planes/archive-compute-pool) guide for the API workflow and the order in which to clear references.

<Note>
  On AWS data planes, the first compute pool you create is **not** automatically promoted to the data plane default. This is the permanent behavior: jobs targeting an AWS data plane must either pin an explicit `computePoolId`, resolve to a dataset-level default, or have a data plane default set explicitly via `PUT /data-planes/{id}/default-compute-pool/{poolId}`. The intent is that AWS workloads make a deliberate choice about where they run rather than implicitly routing through a "first pool wins" default.

  On Snowflake data planes, the first compute pool you create is currently auto-promoted to the data plane default for backward compatibility while the Snowflake-side migration off implicit defaults is in flight. This behavior is temporary and will be removed once every Snowflake workload pins its compute pool explicitly.
</Note>

<Info>
  Every newly registered company is provisioned a private, x-small AWS EMR compute pool on the Narrative data plane and that pool is set as the company-level default for the same data plane. Jobs that don't pin a pool explicitly resolve to this default through the pool resolution chain (job → dataset → company → data plane). You can rename, archive, or replace the default at any time.
</Info>

### `external_id` is optional for `aws_emr` providers

When creating a compute pool with the `aws_emr` provider via `POST /data-planes/{id}/compute-pools`, you can omit `external_id` from the request body — the server fills in the trivial `{"type": "aws_emr"}` payload automatically. Supplying it explicitly still works, and the type is validated against the provider.

For the `snowflake_warehouse` provider, `external_id` is still required because it carries the warehouse name and alias that the platform uses to dispatch queries to the correct Snowflake warehouse.

## How compute pools relate to the SDK

When executing queries through the [TypeScript SDK](/guides/sdk/executing-queries), the `execution_cluster` parameter maps to the compute pool concept:

```typescript theme={null}
const result = await api.executeNql({
  nql: 'SELECT _nio_id, _nio_updated_at FROM company_data."my_dataset" LIMIT 100',
  data_plane_id: null,
  execution_cluster: { type: 'dedicated' },
});
```

The `execution_cluster.type` accepts `'dedicated'` or `'shared'`, corresponding directly to the Dedicated and Shared compute pool types. If omitted, the data plane's default compute pool is used.

For Snowflake-based data planes, omitting `execution_cluster` uses the data plane's default compute pool (the warehouse you've designated as default).

## Related content

<CardGroup>
  <Card title="Execution Context" icon="crosshairs" href="/concepts/primitives/execution-context">
    How data plane, compute pool, database, and schema work together
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and is processed
  </Card>

  <Card title="Executing NQL Queries" icon="terminal" href="/guides/sdk/executing-queries">
    Run queries programmatically with the TypeScript SDK
  </Card>

  <Card title="Migrate to Compute Pools" icon="arrow-right" href="/guides/data-planes/migrate-to-compute-pools">
    Transition from a single Snowflake warehouse to compute pools
  </Card>

  <Card title="Archive a Compute Pool" icon="box-archive" href="/guides/data-planes/archive-compute-pool">
    Safely retire a compute pool and clear its references
  </Card>
</CardGroup>


# Control Plane
Source: https://docs.narrative.io/concepts/primitives/control-plane

Understand how Narrative orchestrates data collaboration across data planes

The control plane is Narrative's centralized orchestration layer. It manages metadata, enforces permissions, coordinates queries, and enables collaboration between organizations—without ever touching the raw data itself.

## What the control plane manages

The control plane is responsible for everything *about* data, rather than the data itself:

**Metadata and catalogs.** The control plane maintains the catalog of all datasets, their schemas, and their attributes. When you browse available data or search for datasets, you're querying the control plane.

**Identity and permissions.** User authentication, organization membership, and [access rules](/concepts/primitives/access-rules) all live in the control plane. When you grant another organization access to your data, that permission is recorded and enforced by the control plane.

**Query compilation and dispatch.** When you write an NQL query, the control plane parses it, validates your permissions, optimizes the execution plan, and [transpiles](/reference/glossary#transpilation) it to the native SQL dialect of your target [data plane](/concepts/primitives/data-planes). The compiled query is then placed in a [job queue](/reference/glossary#job-queue), where data plane operators pick it up for execution. See [Query Processing](/concepts/architecture/query-processing) for a detailed breakdown of this flow.

**Collaboration coordination.** The control plane enables organizations to discover and transact with each other. It maintains the marketplace, processes access requests, and tracks collaboration activity.

## Why separate control from data?

The separation between control plane and data plane is a deliberate architectural choice:

**Privacy by design.** The control plane orchestrates operations without accessing raw data. This means Narrative's infrastructure never sees your actual data when you host your own data plane—only the metadata needed to coordinate operations.

**Multi-cloud collaboration.** Organizations store data in different environments (AWS, Snowflake, on-premises). A centralized control plane can coordinate across all of these without requiring data to move between them.

**Centralized governance.** While data can be distributed across many data planes, permissions and access policies are managed in one place. This simplifies governance and provides a single audit point for who can access what.

## Single control plane, multiple data planes

Narrative operates a single control plane that coordinates across all data planes in the ecosystem. This architecture enables:

* **Cross-organization collaboration** — Organizations with data in different data planes can still collaborate, with the control plane handling the coordination
* **Consistent experience** — Regardless of where data lives, the interface for discovering, querying, and governing data remains the same
* **Unified permissions** — Access controls are managed centrally, even when data spans multiple data planes

```
┌─────────────────────────────────────────────────────┐
│                   Control Plane                      │
│  (Metadata, Permissions, Query Coordination)         │
└──────────┬──────────────────┬───────────────────────┘
           │                  │
           ▼                  ▼
┌──────────────────┐  ┌──────────────────┐
│   Data Plane A   │  │   Data Plane B   │
│  (Your AWS)      │  │  (Narrative)     │
└──────────────────┘  └──────────────────┘
```

## Related content

<CardGroup>
  <Card title="Access Rules" icon="key" href="/concepts/primitives/access-rules">
    Control who can access your data and at what price
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Learn about where your data lives and is processed
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries are compiled, transpiled, and executed
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    Understand how Narrative protects your data
  </Card>
</CardGroup>


# Data Planes
Source: https://docs.narrative.io/concepts/primitives/data-planes

Understand where your data lives and is processed in the Narrative platform

A data plane is the infrastructure where your data physically resides and where queries execute. When you run a query or register a dataset in Narrative, the data plane determines where that data lives and where computation happens.

## Why data planes exist

Narrative separates the *control plane* (which manages metadata, permissions, and orchestration) from the *data plane* (which stores and processes actual data). This separation exists for three reasons:

**Data residency requirements.** Organizations often have strict requirements about where data can be stored—whether for regulatory compliance, data sovereignty, or internal policy. By separating the data plane, Narrative enables you to keep data in your own infrastructure while still participating in data collaboration.

**Performance optimization.** Queries execute directly in the data plane, close to where the data lives. This avoids the latency and cost of moving large datasets across networks.

**Security boundaries.** The control plane never sees your raw data. It only manages metadata and coordinates operations, while the actual data remains within the data plane's security perimeter.

## How data planes relate to the control plane

The [control plane](/concepts/primitives/control-plane) is Narrative's centralized service that orchestrates all data collaboration activities. It maintains the catalog of available datasets, enforces access rules, and coordinates queries across data planes.

When you execute a query:

1. The control plane validates your permissions and [transpiles](/reference/glossary#transpilation) your NQL to the native SQL dialect for the target data plane
2. The control plane creates a job containing the compiled SQL and adds it to a [job queue](/reference/glossary#job-queue)
3. The data plane's [operator](#the-operator-component) polls the job queue and picks up the compiled query
4. The operator executes the native SQL against the data plane's query engine (Snowflake, Spark, etc.)
5. Results flow back through the control plane to you

This architecture means you can collaborate with organizations whose data lives in completely different data planes—the control plane handles the coordination. See [Query Processing](/concepts/architecture/query-processing) for a detailed breakdown of this flow.

## Hosting options

Narrative supports two hosting models for data planes:

### Narrative-hosted data planes

Narrative operates a shared data plane that provides a turnkey experience. Your data is stored in Narrative's infrastructure, and you benefit from managed operations without needing to provision or maintain infrastructure.

This option works well when:

* You want to get started quickly without infrastructure setup
* You don't have strict data residency requirements
* You prefer a fully managed experience

### Customer-hosted data planes

You can run a data plane in your own infrastructure—whether that's your AWS account, Snowflake environment, or another supported platform. Narrative's control plane coordinates with your data plane, but your data never leaves your infrastructure.

This option works well when:

* Regulatory or compliance requirements mandate data residency
* You want to leverage existing infrastructure investments
* You need to keep data within your organization's security perimeter
* You want to process data closer to where it's already stored

<Note>
  Customer-hosted data planes require additional setup and configuration. Contact your Narrative representative to discuss options for your infrastructure.
</Note>

## What belongs to a data plane

Several objects in Narrative are scoped to a specific data plane:

* **[Datasets](/concepts/primitives/datasets)** — The actual data you've registered lives in a data plane
* **Queries** — Query execution happens within a data plane
* **[Access rules](/concepts/primitives/access-rules)** — Rules that govern data access are enforced at the data plane level
* **Jobs** — Each job executes against a specific data plane; the Jobs list shows only jobs for the active data plane
* **[Workflows](/guides/workflows/workflow-orchestration)** — Workflows are tied to the data plane their tasks run on; the Workflows list shows only workflows for the active data plane

When working in the Narrative platform, you'll use the [context selector](/guides/platform/using-the-context-selector) to choose which data plane (and other execution settings) are active. See [Execution Context](/concepts/primitives/execution-context) for details on how data plane selection combines with compute pool, database, and schema to form your full execution context.

## The operator component

Customer-hosted data planes include an operator—a software component that bridges the control plane and your data infrastructure. The operator:

* **Polls the control plane's job queue** for work targeting your data plane
* **Executes compiled SQL** against your native database engine (Snowflake, Spark, etc.)
* **Reports results and status** back to the control plane

The operator runs within your infrastructure, meaning it has access to your data but the control plane does not. This **pull-based design** ensures:

* Your data plane can be behind a firewall with no inbound connections
* The control plane never needs database credentials or direct access
* You control the compute resources allocated to query execution

The operator handles various types of jobs beyond queries, including materialized view refreshes and dataset operations. See [Job Types](/reference/architecture/job-types) for a complete reference.

## Related content

<CardGroup>
  <Card title="Datasets" icon="table" href="/concepts/primitives/datasets">
    Understand the datasets that live within data planes
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries flow from submission through execution
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    Learn how Narrative orchestrates data collaboration across data planes
  </Card>

  <Card title="Job Types" icon="list-check" href="/reference/architecture/job-types">
    Reference for the different types of jobs operators execute
  </Card>

  <Card title="Execution Context" icon="crosshairs" href="/concepts/primitives/execution-context">
    How data plane, compute pool, database, and schema work together
  </Card>

  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    How compute resources are allocated for query execution
  </Card>
</CardGroup>


# Dataset Retention Policies
Source: https://docs.narrative.io/concepts/primitives/dataset-retention-policies

Understand how retention policies manage your data lifecycle and automatic cleanup

A retention policy is a set of rules that governs how long data within a dataset is kept before automatic deletion. Retention policies help organizations manage storage costs, comply with data governance requirements, and ensure data is only stored for as long as necessary.

## Why retention policies matter

**Cost management.** Data storage incurs ongoing costs. Retention policies automatically remove data you no longer need, preventing unbounded storage growth.

**Compliance.** Many regulations (GDPR, CCPA, industry-specific rules) require organizations to delete data after a certain period. Retention policies automate this requirement.

**Data hygiene.** Stale data can lead to incorrect analysis or decisions. Retention policies ensure your datasets contain only current, relevant information.

## How retention policies work

Retention policies are applied at the dataset level. Each dataset can have one or more policies that Narrative evaluates on a configurable schedule to determine which data to remove.

Each policy specifies:

* **A policy class** that determines *what* gets deleted (rows, snapshots, or the entire table)
* **An interval** that determines *when* deletion occurs, expressed as an [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) (for example, `P30D` for 30 days)
* **An enabled flag** that controls whether the policy is actively enforced

You can combine multiple policy classes on a single dataset. For example, you might use a Row TTL policy for routine data expiration alongside a Table TTL policy as a safety net to drop the dataset entirely if it becomes stale.

<Note>
  Retention policy behavior varies by [data plane](/concepts/primitives/data-planes). See [Data plane differences](#data-plane-differences) below for details.
</Note>

## Policy classes

Narrative supports three policy classes, each targeting a different level of the data lifecycle.

### Row TTL

Performs **row-level hard deletes** based on a timestamp column in your data. Row TTL evaluates each row individually and deletes rows whose timestamp exceeds the retention interval.

Row TTL is supported on both **AWS-based** and **Snowflake-based** data planes. On AWS data planes, row deletions are executed against the dataset's Iceberg table; physical file reclamation happens later when Iceberg snapshots expire.

Row TTL uses an **event time clock**—a [Rosetta Stone](/concepts/rosetta-stone/overview) attribute that maps to a timestamp column in your dataset. If you don't specify a clock, the system defaults to the `nio_last_modified` attribute mapping when one exists. If no default mapping is available, the request fails.

Row TTL is useful when:

* You need fine-grained control over which rows expire
* Your dataset contains rows with different ages that shouldn't all expire together
* You want to enforce retention based on when events occurred, not when data was ingested

### Snapshot TTL

Deletes old [snapshots](/reference/glossary#snapshot) based on snapshot age—the time since data was ingested into the dataset. This is the original retention model and operates at the Iceberg snapshot level.

Snapshot TTL is **only available on AWS-based data planes** that use Iceberg storage.

Within a Snapshot TTL policy, you specify a **retention value** that controls behavior:

| Value                 | Behavior                                                                           |
| --------------------- | ---------------------------------------------------------------------------------- |
| **Time-based**        | Delete snapshots older than a specified duration (for example, `P90D` for 90 days) |
| **Retain everything** | Keep all snapshots indefinitely until manually deleted                             |
| **Expire everything** | Remove all snapshots immediately                                                   |

Common ISO 8601 duration examples:

| Duration | Meaning  |
| -------- | -------- |
| `P30D`   | 30 days  |
| `P90D`   | 90 days  |
| `P6M`    | 6 months |
| `P1Y`    | 1 year   |

### Table TTL

Drops the **entire dataset** when the table age exceeds the retention interval. This is the most aggressive policy class—instead of removing individual snapshots or rows, it deletes the dataset object itself.

Table TTL supports three clock types that determine how table age is measured:

| Clock                | Measures age from                                                         |
| -------------------- | ------------------------------------------------------------------------- |
| **`created_at`**     | When the dataset was created                                              |
| **`max_event_time`** | The most recent event timestamp in the data (requires a column reference) |
| **`static_time`**    | A fixed timestamp you provide (requires a column reference)               |

Table TTL is useful for:

* Temporary or staging datasets that should be automatically cleaned up
* Datasets with a defined useful lifespan
* Enforcing hard data deletion deadlines for compliance

<Warning>
  Table TTL permanently deletes the dataset, not just its data. Use this policy class with care—once the dataset is dropped, it cannot be recovered.
</Warning>

### Choosing the right policy class

| Scenario                                                               | Recommended policy  |
| ---------------------------------------------------------------------- | ------------------- |
| Delete individual rows that have aged out based on event timestamps    | Row TTL             |
| Remove old ingestion batches based on when they arrived (Iceberg only) | Snapshot TTL        |
| Drop a temporary or staging dataset after a fixed period               | Table TTL           |
| Combine routine row expiration with a hard dataset deadline            | Row TTL + Table TTL |

## Retention evaluation schedule

Retention policies are not evaluated continuously. Instead, Narrative evaluates them on a configurable schedule. You can set the evaluation schedule when configuring your retention policies through the API.

Between evaluation runs, data that has exceeded its retention period remains in the dataset until the next evaluation occurs.

## Retention and dataset deletion

When you delete a dataset, the retention policy determines what happens to the underlying data:

**On AWS-based data planes**, Narrative applies a default 30-day retention period before permanent deletion. This grace period protects against accidental deletions and provides an opportunity to restore the dataset if needed.

**On Snowflake-based data planes**, data is deleted immediately when you delete a dataset. There is no grace period.

<Warning>
  Understand which data plane your dataset resides in before deleting. Snowflake-based data planes do not offer a recovery window.
</Warning>

## Data plane differences

Retention policy behavior differs depending on where your data is stored:

| Behavior                      | AWS Data Planes | Snowflake Data Planes |
| ----------------------------- | --------------- | --------------------- |
| Default deletion grace period | 30 days         | None (immediate)      |
| Row TTL                       | Supported       | Supported             |
| Table TTL                     | Supported       | Supported             |
| Snapshot TTL                  | Supported       | Not supported         |

Row TTL and Table TTL work across both data plane types. Snapshot TTL operates on Iceberg snapshots, which are specific to AWS-based data planes. On AWS, Row TTL deletes rows from the dataset's Iceberg table; the underlying files are reclaimed when Iceberg snapshots expire.

## Retention policies for materialized views

[Materialized views](/concepts/nql/materialized-views) support retention policies through the `EXPIRE` clause in NQL. When creating or modifying a materialized view, you can specify how long data should be retained:

```sql theme={null}
CREATE MATERIALIZED VIEW "my_view"
EXPIRE = 'P90D'
AS
SELECT
    user_id,
    event_type,
    event_timestamp
FROM my_dataset
```

For complete syntax details, see the [EXPIRE clause reference](/nql/commands/create-materialized-view#expire).

## Choosing a retention period

Consider these factors when setting retention policies:

**Regulatory requirements.** Check what your compliance obligations require. Some regulations mandate maximum retention periods, while others require minimum retention.

**Business needs.** How far back does your analysis typically need to go? Set retention to cover your longest reasonable lookback period plus a buffer.

**Cost sensitivity.** Longer retention means higher storage costs. Balance completeness against budget constraints.

**Data refresh patterns.** If you're regularly refreshing data (for example, daily snapshots), you may not need to retain old versions for long.

<Tip>
  Start with a longer retention period and reduce it based on observed usage patterns. It's easier to shorten retention than to recover deleted data.
</Tip>

## Related content

<CardGroup>
  <Card title="Configuring Retention Policies" icon="sliders" href="/guides/platform/configuring-retention-policies">
    Step-by-step guide for adding and managing retention policies from the UI
  </Card>

  <Card title="Datasets" icon="table" href="/concepts/primitives/datasets">
    Understand how datasets store and organize your data
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Learn where your data physically resides
  </Card>

  <Card title="EXPIRE Clause Reference" icon="list-check" href="/nql/commands/create-materialized-view#expire">
    Complete syntax for setting retention in materialized views
  </Card>

  <Card title="Upsert Retention Policies API" icon="code" href="/api-reference/datasets/upsert-retention-policies-for-a-dataset">
    API reference for configuring retention policies
  </Card>
</CardGroup>


# Dataset Statistics
Source: https://docs.narrative.io/concepts/primitives/dataset-statistics

Understand how column-level metrics describe your data and power platform features

Dataset statistics are column-level metrics computed over the contents of a [dataset](/concepts/primitives/datasets). They provide a quantitative profile of each column—counts, bounds, distributions, and quality indicators—so you can understand what your data looks like without querying it directly.

## Why statistics matter

**Data quality assessment.** Statistics give you an immediate picture of data health. You can see how many values are null, whether numeric ranges look reasonable, and how complete each column is—all without writing a query.

**Schema understanding.** When working with a new dataset, statistics help you understand the shape of the data. Cardinality tells you whether a column has a few categories or millions of unique values. Histograms show you how values are distributed.

**Query context.** Statistics surface in the platform UI to help you make better decisions. For example, when filtering on a field in Data Studio, histogram statistics populate a dropdown with the field's actual values, so you can select from real data rather than guessing.

## Where statistics appear in the platform

Statistics are visible in two places:

* **Dataset details page** — shows column-level statistics for the full dataset
* **Dataset sample page** — shows statistics computed over the sample

They also power UI features elsewhere. In Data Studio, filtering on a field that has histogram statistics displays the histogram values in a dropdown for easy selection.

## Configuring what gets computed

You can control which statistics are computed, set refresh schedules, and add per-field overrides directly from the dataset details page. The configuration supports three levels of specificity: global defaults that apply to all fields, namespace-scoped overrides for dataset columns or Rosetta Stone attributes, and individual field-level overrides.

For step-by-step instructions, see the [configuring dataset statistics](/guides/platform/configuring-dataset-statistics) guide. For the full configuration schema and API details, see the [reference](/reference/architecture/dataset-statistics).

## What gets computed

The platform computes 12 statistics per column, grouped into three categories.

### Counts

Metrics that describe how many values exist and their uniqueness.

| Statistic             | What it tells you                                                                   |
| --------------------- | ----------------------------------------------------------------------------------- |
| `valueCount`          | Total number of non-null values                                                     |
| `nullValueCount`      | Number of null values                                                               |
| `nanValueCount`       | Number of NaN (not-a-number) values—applies only to floating-point columns          |
| `approxCountDistinct` | Approximate number of unique values, using a probabilistic algorithm for efficiency |
| `countDistinct`       | Exact number of unique values                                                       |

### Bounds and distribution

Metrics that describe how values are distributed across the column.

| Statistic           | What it tells you                                                |
| ------------------- | ---------------------------------------------------------------- |
| `lowerBound`        | Minimum value in the column                                      |
| `upperBound`        | Maximum value in the column                                      |
| `histogram`         | Frequency distribution of values across distinct buckets         |
| `mean`              | Average value—applies only to numeric columns                    |
| `standardDeviation` | Spread of values around the mean—applies only to numeric columns |

### Storage and quality

Metrics that describe the physical footprint and overall completeness of the column.

| Statistic           | What it tells you                                                            |
| ------------------- | ---------------------------------------------------------------------------- |
| `columnStoredBytes` | Bytes of storage consumed by the column                                      |
| `completeness`      | Ratio of non-null values to total rows, expressed as a value between 0 and 1 |

## How data types affect available statistics

Not all statistics apply to all data types. Numeric columns support the full set of 12 statistics, while non-numeric types like strings and booleans don't have `mean` or `standardDeviation`. Complex types like arrays and objects only support basic counts and storage metrics.

For the full mapping of which statistics are available for each data type, see the [type compatibility matrix](/reference/architecture/dataset-statistics#type-compatibility-matrix) in the reference.

## Related content

<CardGroup>
  <Card title="Datasets" icon="table" href="/concepts/primitives/datasets">
    How datasets store, organize, and protect your data
  </Card>

  <Card title="Configuring Dataset Statistics" icon="gear" href="/guides/platform/configuring-dataset-statistics">
    Set up statistics through the platform UI
  </Card>

  <Card title="Dataset Statistics Reference" icon="chart-column" href="/reference/architecture/dataset-statistics">
    Full reference for each statistic and type compatibility matrix
  </Card>

  <Card title="Managing Datasets" icon="code" href="/guides/sdk/managing-datasets">
    Create and manage datasets with the SDK
  </Card>

  <Card title="Sample Data" icon="flask" href="/concepts/architecture/sample-data">
    How samples cross to the control plane
  </Card>
</CardGroup>


# Datasets
Source: https://docs.narrative.io/concepts/primitives/datasets

Understand how datasets store, organize, and protect your data in Narrative

A dataset is a structured collection of data registered in Narrative. Think of it as a table in a database—it has a defined schema, holds rows of data, and can be queried, shared, and collaborated on. Datasets are one of the core primitives in the Narrative platform.

## What datasets are for

Datasets serve as the primary way to bring data into Narrative and make it available for collaboration:

**Storage and organization.** Datasets provide a structured container for your data. Each dataset has a defined schema that specifies what fields exist, their data types, and how they should be validated.

**Querying.** Once data is in a dataset, you can query it using [NQL](/concepts/nql/design-philosophy). Datasets are the foundation of all query operations in Narrative.

**Collaboration.** Through [access rules](/concepts/primitives/access-rules), you can grant other organizations permission to query your datasets—enabling data sharing, monetization, or joint analysis.

## How datasets are structured

### Schema

Every dataset has a schema that acts as its structural blueprint. The schema defines:

* **Field names** — The columns that exist in the dataset
* **Field types** — The data type for each field (string, number, timestamp, etc.)
* **Descriptions** — Documentation explaining what each field contains
* **Validations** — Rules that ensure data integrity when records are added

```typescript theme={null}
// Example schema definition
{
  properties: {
    customer_id: { type: 'string' },
    event_type: { type: 'string' },
    event_timestamp: { type: 'timestamptz' },
    event_value: { type: 'double' },
  }
}
```

<Note>
  Schemas are designed to be stable. While you can add new fields to a schema, changing or removing existing fields requires careful consideration to avoid breaking queries or integrations that depend on them.
</Note>

### Records and snapshots

Data in a dataset is organized into records (rows) and snapshots:

* **Records** are individual data entries that conform to the dataset's schema
* **Snapshots** represent a point-in-time collection of files that were ingested together

When you upload data, the ingestion process validates each record against the schema and adds it to the dataset as part of a new snapshot.

## Adding data to datasets

Datasets support multiple ways to add data:

**Append mode.** New data is added alongside existing data. Use this for event-style data where each upload contains new records.

**Overwrite mode.** New data replaces existing data. Use this when you want to refresh the entire dataset with an updated version.

For procedural details on uploading data, see [Uploading Data](/guides/sdk/uploading-data).

## Retention policies

Datasets can have [retention policies](/concepts/primitives/dataset-retention-policies) that automatically manage data lifecycle. A retention policy defines how long data is kept before automatic deletion, helping you manage storage costs and comply with data governance requirements.

Common retention configurations include:

* **Time-based retention** — Automatically remove data older than a specified period (e.g., 90 days, 1 year)
* **Retain everything** — Keep all data indefinitely until manually deleted

For details on how retention policies work, including differences between data planes and how to configure them, see [Dataset Retention Policies](/concepts/primitives/dataset-retention-policies).

## Ownership and access

### Single-company ownership

Every dataset is owned by exactly one company. The owner has full control over:

* The dataset's schema and configuration
* Who can access the data and under what terms
* Whether to archive or delete the dataset

This ownership model ensures clear accountability and prevents ambiguity about who controls sensitive data.

### Access through access rules

By default, a dataset is private to its owner. To share data with other organizations, you create [access rules](/concepts/primitives/access-rules) that define:

* Which organizations can query the dataset
* Which fields and records they can access
* What pricing applies (if any)

This separation between ownership and access provides flexibility—you retain full control while selectively enabling collaboration.

## Where datasets live

Datasets are scoped to a specific [data plane](/concepts/primitives/data-planes). The data plane determines:

* Where the data physically resides (Narrative-hosted or your own infrastructure)
* Which query engine processes queries against the dataset
* What data residency and compliance requirements are met

When you create a dataset, you specify which data plane it belongs to. The [control plane](/concepts/primitives/control-plane) maintains metadata about the dataset—its schema, access rules, and statistics—while the actual data remains in the data plane.

## View datasets

A view dataset is a dataset backed by an NQL query rather than uploaded data. When you execute a query with the `create_as_view` option, the result is stored as a view dataset. Unlike a [materialized view](/concepts/nql/materialized-views), a view dataset does not refresh on a schedule — the stored NQL is inlined and re-evaluated at query time whenever the view dataset is referenced.

View datasets are useful when you want a reusable, queryable subset of your data without duplicating it into a separate physical dataset. The underlying NQL query can reference other datasets (including other view datasets), and the platform resolves those dependencies automatically.

### Restrictions

View datasets have specific restrictions compared to regular datasets:

* **No access rules.** You cannot create [access rules](/concepts/primitives/access-rules) on a view dataset. If you need to share the data with other organizations, create a regular dataset or materialized view instead.
* **No connections.** You cannot create [connections](/concepts/data-activation/overview) to deliver a view dataset to an external platform.
* **No forecasting.** Query cost forecasting is not available for queries against view datasets.

For the full feature comparison with materialized views — including unsupported NQL features like `MERGE ON`, `PARTITIONED_BY`, and chunking strategies — see [view dataset limitations](/concepts/nql/materialized-views#view-dataset-limitations).

### Creating a view dataset

Use the `create_as_view` option when executing an NQL query through the SDK:

```typescript theme={null}
const result = await api.executeNql({
  nql: 'SELECT user_id, email, event_type FROM company_data."my_dataset" WHERE event_type = \'purchase\'',
  data_plane_id: null,
  create_as_view: true,
});
```

For details on executing queries with this option, see [Executing NQL Queries](/guides/sdk/executing-queries#creating-a-view-dataset).

## Datasets, materialized views, and view datasets

Narrative supports three types of data containers:

| Type                  | Source                   | Data storage                    | Updates                               |
| --------------------- | ------------------------ | ------------------------------- | ------------------------------------- |
| **Dataset**           | External data you upload | Physical table                  | Manual uploads or automated ingestion |
| **Materialized view** | NQL query results        | Physical table (cached results) | Automatic refresh on schedule         |
| **View dataset**      | NQL query definition     | No physical storage             | Re-evaluated at query time            |

[Materialized views](/concepts/nql/materialized-views) are created from NQL queries and automatically refresh their contents. Regular datasets require you to explicitly add data through uploads or ingestion. View datasets store only the NQL query definition and re-evaluate it at query time — see [View datasets](#view-datasets) above.

## Related content

<CardGroup>
  <Card title="Retention Policies" icon="clock-rotate-left" href="/concepts/primitives/dataset-retention-policies">
    Configure automatic data lifecycle management
  </Card>

  <Card title="Access Rules" icon="key" href="/concepts/primitives/access-rules">
    Control who can query your datasets and at what price
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understand where your datasets physically reside
  </Card>

  <Card title="Dataset Statistics" icon="chart-column" href="/concepts/primitives/dataset-statistics">
    Column-level metrics computed over your dataset contents
  </Card>

  <Card title="Managing Datasets" icon="gear" href="/guides/sdk/managing-datasets">
    Create and manage datasets with the SDK
  </Card>

  <Card title="Name Conflict Errors" icon="clone" href="/reference/errors/access-rule-name-conflict">
    Resolve HTTP 409 errors when a dataset name is already in use
  </Card>

  <Card title="Schema Incompatibility Errors" icon="plug-circle-exclamation" href="/reference/errors/dataset-schema-connector-incompatible">
    Resolve HTTP 400 errors when a dataset schema doesn't match a connector
  </Card>
</CardGroup>


# Execution Context
Source: https://docs.narrative.io/concepts/primitives/execution-context

Understand how data plane, compute pool, database, and schema combine to determine where and how platform operations run

Execution context is the combination of four settings—data plane, compute pool, database, and schema—that determines where and how your platform operations run. Every query you execute, every dataset you browse, and every audience you build runs within a specific execution context.

Think of it as answering four questions at once: *Where does my data live?* (data plane), *What resources should process it?* (compute pool), *Which database?* (database), and *Which schema?* (schema).

## The four dimensions

| Dimension        | What it controls                                             | Example values                                  |
| ---------------- | ------------------------------------------------------------ | ----------------------------------------------- |
| **Data plane**   | Where your data physically resides and where queries execute | Narrative DCP, Customer Snowflake, Customer AWS |
| **Compute pool** | The compute resources allocated to process your queries      | Dedicated, Shared, Default                      |
| **Database**     | The logical database within the data plane                   | `DEFAULT`                                       |
| **Schema**       | The schema within the database that scopes visible objects   | `DEFAULT`                                       |

Each dimension narrows the scope of the next. Selecting a data plane determines which compute pools are available. Selecting a database determines which schemas are available.

## How context affects the platform

Your execution context influences what you see and how operations behave across the platform:

| Feature              | How context applies                                                                    |
| -------------------- | -------------------------------------------------------------------------------------- |
| **NQL queries**      | Queries execute in the selected data plane using the selected compute pool's resources |
| **Dataset browsing** | Only datasets registered in the selected data plane are visible                        |
| **Data Studio**      | Query editor and results operate within the current context                            |
| **Audiences**        | Audience creation and activation use the selected data plane and compute resources     |

If you switch your data plane, you'll see a different set of datasets, and any queries you run will execute against different infrastructure.

## Supported providers

Execution context options vary depending on your data plane's underlying provider:

| Provider                   | Description                                       | Available compute pools         |
| -------------------------- | ------------------------------------------------- | ------------------------------- |
| **Snowflake**              | Customer-hosted Snowflake data planes             | Registered Snowflake warehouses |
| **Narrative (shared AWS)** | Narrative's managed data collaboration platform   | Dedicated, Shared               |
| **Customer AWS**           | Customer-hosted data planes on AWS infrastructure | Dedicated, Shared               |

See [Compute Pools](/concepts/primitives/compute-pools) for details on what each compute pool type offers.

## Default behavior

When you first load the platform, context auto-selects based on what's available to you:

* **Single data plane** — Auto-selected. No action needed.
* **Multiple data planes** — Defaults to the Narrative DCP (data collaboration platform).
* **Compute pool** — Defaults to the data plane's default compute pool. For Snowflake, this is the warehouse you've designated as default. For Narrative and Customer AWS, this is Shared.
* **Database and schema** — Default to `DEFAULT`.

For most users with a single data plane, the execution context is set automatically and requires no interaction.

## Context persistence

Your execution context is saved to your browser's local storage. When you return to the platform in a later session, your previous selections are restored automatically. This means you don't need to re-select your context each time you log in.

<Note>
  Because context is stored per-browser, using a different browser or clearing local storage will reset your selections to the defaults.
</Note>

## Database and schema

The database and schema dimensions currently default to `DEFAULT` for all data planes. These dimensions exist to support future multi-database and multi-schema configurations, where organizations may need to scope operations to specific databases or schemas within a data plane.

For now, you can leave these at their default values.

## Related content

<CardGroup>
  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and is processed
  </Card>

  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    How compute resources are allocated for query execution
  </Card>

  <Card title="Using the Context Selector" icon="hand-pointer" href="/guides/platform/using-the-context-selector">
    Step-by-step guide to changing your execution context
  </Card>
</CardGroup>


# Confidence Scoring
Source: https://docs.narrative.io/concepts/rosetta-stone/confidence-scoring

How AI evaluates mapping quality and generates confidence scores

When you create mappings in Rosetta Stone, how do you know they're correct? A mapping might look reasonable but produce unexpected results with certain data values. Confidence scoring uses AI to evaluate your mappings and identify potential issues before they affect your data.

## What is a confidence score?

A confidence score is an AI-generated rating from 0-100% that indicates how likely a mapping is to produce correct results. Higher scores mean the AI found strong evidence that the mapping is accurate; lower scores indicate potential issues that warrant review.

Confidence scores differ from manual validation in important ways:

| Approach           | What it checks                   | When it runs  |
| ------------------ | -------------------------------- | ------------- |
| Manual validation  | Specific test cases you define   | On demand     |
| Confidence scoring | Pattern analysis across all data | Automatically |

Manual validation confirms mappings work for known cases. Confidence scoring identifies issues you might not have anticipated—unusual data patterns, edge cases, or transformation logic that could fail under certain conditions.

## How scores are calculated

The AI evaluates each mapping by analyzing multiple factors:

**Column name analysis**
The AI examines whether the source column name semantically matches the target attribute. A column named `email_address` mapping to the `raw_email` attribute scores higher than a column named `field_7`.

**Data sample inspection**
The system samples actual data values and checks whether they match expected patterns for the target attribute. If the `hl7_gender` attribute expects values like `male` or `female`, but the source column contains `1` and `2`, the AI will flag this unless there's an appropriate transformation.

**Transformation logic evaluation**
When a mapping includes a transformation expression, the AI analyzes whether the logic correctly handles the conversion. It looks for:

* Missing case handling (what happens to unexpected values?)
* Type mismatches (is a string being cast to a number correctly?)
* Edge cases (null values, empty strings, special characters)

**Historical patterns**
The AI learns from mappings across the platform. If similar columns from other datasets map to the same attribute using a particular transformation, it uses this pattern to inform scoring.

## Understanding score ranges

Confidence scores fall into three tiers:

| Score range | Classification    | Recommended action                        |
| ----------- | ----------------- | ----------------------------------------- |
| 80-100%     | High confidence   | Generally reliable; spot-check if desired |
| 50-79%      | Medium confidence | Review the mapping and AI feedback        |
| 0-49%       | Low confidence    | Manual review required before use         |

**High confidence (80-100%)** indicates strong alignment between the source column and target attribute. The AI found consistent patterns, appropriate transformations, and no significant edge cases. These mappings are likely correct, though you should still validate critical data flows.

**Medium confidence (50-79%)** suggests the mapping is probably correct but has characteristics worth reviewing. Common reasons include:

* Partial transformation coverage (handles most but not all values)
* Column names that don't clearly indicate content
* Data patterns that vary from typical examples

**Low confidence (0-49%)** means the AI identified significant concerns. The mapping might still be correct, but requires human verification. Low scores often result from:

* Missing or incomplete transformations
* Source data that doesn't match expected patterns
* Ambiguous column names with multiple possible interpretations

## Evaluation vs. suggestions

Confidence scoring powers two distinct workflows:

**Mapping evaluation** analyzes your existing mappings. Run an evaluation to:

* Assess the overall quality of your normalizations
* Identify specific mappings that need attention
* Get AI-generated explanations for quality issues

**Mapping suggestions** proposes new mappings for columns that aren't yet normalized. The AI examines unmapped columns and recommends:

* Which attribute each column should map to
* What transformation expression to use
* Sample output showing before/after values

Both workflows use the same underlying AI analysis, but evaluation focuses on improving what exists while suggestions help you expand your normalizations.

## The confidence gradient

In the Normalized Datasets interface, confidence is visualized as a gradient bar showing the distribution of your mappings across quality tiers:

| Color  | Meaning                    |
| ------ | -------------------------- |
| Green  | High confidence mappings   |
| Yellow | Medium confidence mappings |
| Red    | Low confidence mappings    |
| Gray   | Not yet scored             |

This visualization helps you quickly assess the overall health of a dataset's normalizations. A dataset with mostly green indicates well-configured mappings; significant red suggests the dataset needs attention.

## When to re-evaluate

Confidence scores can become stale when:

* You modify transformation expressions
* The source data changes significantly
* New data samples reveal patterns not present in the original evaluation

Re-run evaluations periodically, especially after schema changes or when you notice unexpected query results. The AI will incorporate new data samples and may adjust scores based on current conditions.

## Related content

<CardGroup>
  <Card title="Managing Evaluations" icon="gauge" href="/guides/rosetta-stone/managing-evaluations">
    Run evaluations and interpret results
  </Card>

  <Card title="Accepting AI Suggestions" icon="wand-magic-sparkles" href="/guides/rosetta-stone/accepting-ai-suggestions">
    Review and accept AI-generated mapping recommendations
  </Card>

  <Card title="Normalized Datasets UI" icon="table-columns" href="/reference/rosetta-stone/normalized-datasets-ui">
    Reference for the Normalized Datasets interface
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Manual validation techniques for mappings
  </Card>
</CardGroup>


# How Rosetta Stone Works
Source: https://docs.narrative.io/concepts/rosetta-stone/how-it-works

The mechanics of attributes, mappings, and automatic schema normalization

Rosetta Stone normalizes data through two core primitives: **attributes** and **mappings**. Attributes define the common schema—what the normalized data looks like. Mappings translate each provider's native columns into that common schema.

## The two core primitives

### Attributes

An attribute is a standardized field definition in the common schema. Each attribute specifies:

| Property        | Description                                                                               |
| --------------- | ----------------------------------------------------------------------------------------- |
| **Name**        | A unique identifier (e.g., `hl7_gender`, `event_timestamp`)                               |
| **Description** | Human-readable explanation of what the attribute represents                               |
| **Type**        | The data type: `string`, `long`, `double`, `boolean`, `timestamptz`, `object`, or `array` |
| **Validations** | Rules that data must satisfy (as an array of validation strings)                          |

**Example: Unique Identifier**

The `unique_identifier` attribute captures identity data from various sources. It's defined as an object with three properties:

```json theme={null}
{
  "id": 100,
  "name": "unique_identifier",
  "type": "object",
  "properties": {
    "type": {
      "type": "string"
    },
    "value": {
      "type": "string"
    },
    "context": {
      "type": "string"
    }
  },
  "required": ["type", "value"]
}
```

This structure allows different identity types to be represented consistently, whether the source data contains mobile advertising IDs, hashed emails, or cookie identifiers.

**Example: Gender**

The `hl7_gender` attribute normalizes gender data using the HL7 standard. It's a string type with restricted enum values:

```json theme={null}
{
  "id": 200,
  "name": "hl7_gender",
  "type": "string",
  "enum": ["male", "female", "other", "unknown"],
  "description": "Gender using HL7 administrative gender codes"
}
```

### Mappings

A mapping connects a specific column in a dataset to an attribute. Each mapping includes:

| Property             | Description                                  |
| -------------------- | -------------------------------------------- |
| **Source column**    | The column in the provider's dataset         |
| **Target attribute** | The Rosetta Stone attribute to map to        |
| **Transformation**   | An optional expression to convert the data   |
| **Dataset**          | The specific dataset this mapping applies to |

**Example: Gender mapping**

A provider stores gender as `"M"` or `"F"` in a column called `sex`. The mapping would be:

```json theme={null}
{
  "source_column": "sex",
  "target_attribute": "hl7_gender",
  "transformation": "CASE WHEN sex = 'M' THEN 'male' WHEN sex = 'F' THEN 'female' ELSE 'unknown' END"
}
```

The transformation expression converts the provider's format to the standard enum values.

## The normalization pipeline

Rosetta Stone normalizes data through a three-stage pipeline:

### Stage 1: Schema inference

When data is uploaded to Narrative, the system analyzes it to understand its structure:

1. **Column detection**: Identifies column names and data types
2. **Pattern recognition**: Detects common patterns (dates, identifiers, categorical data)
3. **Attribute suggestion**: Uses machine learning to suggest which attributes each column maps to

This automated analysis provides a starting point—suggestions that can be accepted, modified, or rejected.

### Stage 2: Mapping creation

Mappings are created through a combination of machine learning and human curation:

1. **Auto-generated mappings**: The system proposes mappings based on schema inference
2. **Human review**: Data owners review suggestions and refine as needed
3. **Transformation definition**: Complex mappings include transformation expressions

This iterative process ensures accuracy while minimizing manual effort. As more data flows through the system, the ML models improve their suggestions.

### AI-assisted quality evaluation

After mappings are created, you can use AI to evaluate their quality and suggest improvements:

1. **Confidence scoring**: AI analyzes each mapping and assigns a confidence score (0-100%) indicating how likely the mapping is to produce correct results
2. **Issue identification**: The system highlights potential problems with transformations, such as missing case handling or type mismatches
3. **Suggestion generation**: AI proposes new mappings for columns that aren't yet normalized

This ongoing refinement helps maintain data quality as source schemas evolve or new data patterns emerge. See [Confidence Scoring](/concepts/rosetta-stone/confidence-scoring) for details on how the AI evaluation works.

### Stage 3: Query-time translation

When you query the `narrative.rosetta_stone` table:

1. **Query analysis**: The system identifies which attributes you're requesting
2. **Dataset discovery**: Finds all datasets with mappings for those attributes
3. **Query translation**: Rewrites your query for each dataset's native schema
4. **Execution**: Runs the translated queries against source data
5. **Normalization**: Applies transformations and unions results
6. **Return**: Delivers data in the consistent, normalized format

This happens transparently—you write one query, and the platform handles the complexity of accessing multiple heterogeneous sources.

### Multiple mappings to the same attribute

When a dataset has multiple columns mapped to the same attribute, the normalization process:

1. **Evaluates each mapping** independently
2. **Collects results** into an array
3. **Expands the array** to produce one output row per value
4. **Filters NULL values** during expansion

This enables maximum data utilization—if a provider captures email addresses in multiple columns, queries receive all available values without needing to know the source schema structure.

For example, if a dataset has `email_1` and `email_2` both mapped to `raw_email`, a query selecting that attribute returns one row per non-null email address. See [Multiple columns mapped to the same attribute](/guides/rosetta-stone/edge-cases#multiple-columns-mapped-to-the-same-attribute) for detailed examples and when to use alternatives like `COALESCE`.

## Normalization examples

### Date normalization

Different providers store dates in various formats:

| Provider   | Column       | Sample value           |
| ---------- | ------------ | ---------------------- |
| Provider A | `event_date` | `01/15/2024`           |
| Provider B | `timestamp`  | `2024-01-15T14:30:00Z` |
| Provider C | `dt`         | `15-Jan-2024`          |

All three map to the `event_timestamp` attribute. The mappings include transformations that parse each format and output ISO 8601:

```sql theme={null}
-- Provider A mapping transformation
TO_TIMESTAMP(event_date, 'MM/DD/YYYY')

-- Provider B mapping transformation
timestamp  -- Already ISO 8601, no transformation needed

-- Provider C mapping transformation
TO_TIMESTAMP(dt, 'DD-Mon-YYYY')
```

When you query `event_timestamp`, you receive consistent ISO 8601 timestamps regardless of source.

### Gender normalization

Providers represent gender in many ways:

| Provider   | Column        | Values               |
| ---------- | ------------- | -------------------- |
| Provider A | `gender`      | `"male"`, `"female"` |
| Provider B | `sex`         | `"M"`, `"F"`         |
| Provider C | `gender_code` | `1`, `2`, `0`        |
| Provider D | `gndr`        | `"m"`, `"f"`, `"nb"` |

Each mapping includes a transformation to the `hl7_gender` enum:

```sql theme={null}
-- Provider A: Already matches, minimal transformation
LOWER(gender)

-- Provider B: Letter codes to words
CASE sex WHEN 'M' THEN 'male' WHEN 'F' THEN 'female' ELSE 'unknown' END

-- Provider C: Numeric codes to words
CASE gender_code WHEN 1 THEN 'male' WHEN 2 THEN 'female' ELSE 'unknown' END

-- Provider D: Handle non-binary
CASE LOWER(gndr) WHEN 'm' THEN 'male' WHEN 'f' THEN 'female' WHEN 'nb' THEN 'other' ELSE 'unknown' END
```

## Validation and quality

Mappings aren't just translations—they're quality gates. Each mapping can enforce validations:

**Type checking**: Ensures values can be cast to the target type

**Enum validation**: Confirms values match allowed enum members

**Range checking**: Verifies numeric values fall within acceptable bounds

**Pattern matching**: Validates strings match expected formats (e.g., email patterns)

When data fails validation, the system can:

* Reject the record
* Map to a default value (like `unknown` for invalid gender)
* Flag the record for review

This ensures the normalized view contains clean, consistent data.

## Related content

<CardGroup>
  <Card title="Rosetta Stone Overview" icon="language" href="/concepts/rosetta-stone/overview">
    Understand the business problem Rosetta Stone solves
  </Card>

  <Card title="The Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    Deep dive into the type system and attribute hierarchy
  </Card>

  <Card title="Confidence Scoring" icon="gauge-high" href="/concepts/rosetta-stone/confidence-scoring">
    How AI evaluates mapping quality
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Step-by-step guide to creating mappings
  </Card>
</CardGroup>


# The Normalization Model
Source: https://docs.narrative.io/concepts/rosetta-stone/normalization-model

How Rosetta Stone structures attributes, types, and validations

Rosetta Stone's normalization model defines how data is structured, typed, and validated across the platform. Understanding this model helps you work effectively with attributes and create accurate mappings.

## The attribute hierarchy

Attributes exist at different scopes within the platform:

### Global attributes

Global attributes are available to all organizations on the platform. They represent standardized concepts that are commonly used across data collaboration scenarios:

* **Identity attributes**: `email_sha256`, `phone_sha256`, `unique_identifier`
* **Demographic attributes**: `hl7_gender`, `age`, `birth_year`
* **Temporal attributes**: `event_timestamp`, `nio_last_modified`
* **Geographic attributes**: `country_code`, `postal_code`, `geo_coordinates`

Global attributes are maintained by Narrative and follow industry standards where applicable (e.g., HL7 for healthcare, ISO for country codes).

### Organization attributes

Organizations can create custom attributes for concepts specific to their domain or use case. Organization attributes:

* Are visible only within your organization (and to partners you explicitly share with)
* Can extend or specialize global attributes
* Follow the same type system and validation rules

### Attribute composition

Attributes can reference other attributes, enabling complex data structures:

```json theme={null}
{
  "id": 150,
  "name": "hashed_identity",
  "type": "object",
  "properties": {
    "identifier": {
      "$ref": 100
    },
    "hash_algorithm": {
      "type": "string",
      "enum": ["sha256", "md5", "sha1"]
    }
  },
  "required": ["identifier", "hash_algorithm"]
}
```

In this example, `"$ref": 100` references the `unique_identifier` attribute (ID 100). This allows reuse of standardized definitions while building domain-specific structures.

## The type system

Attributes use a type system that supports both primitive and complex data.

### Primitive types

| Type          | Description                            | Example values                  |
| ------------- | -------------------------------------- | ------------------------------- |
| `string`      | Text data of variable length           | `"hello"`, `"user@example.com"` |
| `long`        | Whole numbers (64-bit integer)         | `42`, `-17`, `0`                |
| `double`      | Decimal numbers (64-bit float)         | `3.14`, `-0.001`, `1000.0`      |
| `boolean`     | True/false values                      | `true`, `false`                 |
| `timestamptz` | Date and time with timezone (ISO 8601) | `2024-01-15T14:30:00Z`          |

### Enum type

Enums restrict string values to a predefined set. An enum is a `string` type with an `enum` property:

```json theme={null}
{
  "id": 200,
  "name": "hl7_gender",
  "type": "string",
  "enum": ["male", "female", "other", "unknown"],
  "description": "Gender using HL7 administrative gender codes"
}
```

When mapping to an enum attribute, transformations must produce one of the allowed values. Unrecognized values typically map to a fallback like `unknown`.

### Object type

Objects group related fields into a single attribute using `properties`:

```json theme={null}
{
  "id": 401,
  "name": "geo_coordinates",
  "type": "object",
  "properties": {
    "latitude": {
      "type": "double"
    },
    "longitude": {
      "type": "double"
    },
    "accuracy_meters": {
      "type": "double"
    }
  },
  "required": ["latitude", "longitude"]
}
```

The `required` array specifies which properties must be present. When mapping to an object, the transformation must produce all required fields:

```sql theme={null}
STRUCT(
  CAST(lat AS DOUBLE) AS latitude,
  CAST(lon AS DOUBLE) AS longitude,
  CAST(accuracy AS DOUBLE) AS accuracy_meters
)
```

### Array type

Arrays contain multiple values of the same type, specified with `items`:

```json theme={null}
{
  "id": 500,
  "name": "interest_categories",
  "type": "array",
  "items": {
    "type": "string"
  },
  "description": "Categories of user interests"
}
```

Source data might be comma-separated:

```sql theme={null}
SPLIT(interests, ',')
```

### Reference type

References link to other attribute definitions using `$ref` with the numeric attribute ID:

```json theme={null}
{
  "id": 600,
  "name": "purchase_event",
  "type": "object",
  "properties": {
    "timestamp": {
      "$ref": 300
    },
    "user": {
      "$ref": 100
    },
    "amount": {
      "type": "double"
    }
  },
  "required": ["timestamp", "user", "amount"]
}
```

In this example, `$ref: 300` references the `event_timestamp` attribute and `$ref: 100` references the `unique_identifier` attribute. References inherit the type, validations, and semantics of the referenced attribute.

## Validations

Validations enforce data quality at the attribute level. Validations are NQL expressions where `$this` represents the value being validated. These expressions are injected into compiled NQL queries to filter out invalid values.

### Required fields in objects

For object types, the `required` array specifies which properties must be present:

```json theme={null}
{
  "id": 700,
  "name": "person",
  "type": "object",
  "properties": {
    "first_name": { "type": "string" },
    "middle_name": { "type": "string" },
    "last_name": { "type": "string" }
  },
  "required": ["first_name", "last_name"]
}
```

In this example, `middle_name` is optional because it's not in the `required` array.

### Range constraints

Numeric fields can have minimum and maximum bounds using comparison operators with `$this`:

```json theme={null}
{
  "id": 201,
  "name": "age",
  "type": "long",
  "validations": ["$this >= 0", "$this <= 150"]
}
```

Values outside the range are rejected.

### String length

Use the `LENGTH()` function to constrain string length:

```json theme={null}
{
  "id": 400,
  "name": "country_code",
  "type": "string",
  "validations": ["LENGTH($this) = 2"]
}
```

### Pattern matching

Use `LIKE` for pattern validation or combine with length checks:

```json theme={null}
{
  "id": 101,
  "name": "email_sha256",
  "type": "string",
  "is_join_key": true,
  "validations": ["LENGTH($this) = 64"]
}
```

### Combining validations

Multiple validations can be combined in a single array:

```json theme={null}
{
  "id": 800,
  "name": "date_range",
  "type": "object",
  "properties": {
    "start_date": { "type": "timestamptz" },
    "end_date": { "type": "timestamptz" }
  },
  "required": ["start_date", "end_date"],
  "validations": ["end_date >= start_date"]
}
```

## Schema presets

Schema presets are curated collections of attributes designed for common use cases.

### Available presets

| Preset       | Description               | Key attributes                                      |
| ------------ | ------------------------- | --------------------------------------------------- |
| Demographics | Consumer demographic data | `hl7_gender`, `age`, `birth_year`, `country_code`   |
| Identity     | User identification       | `unique_identifier`, `email_sha256`, `phone_sha256` |
| Events       | Timestamped occurrences   | `event_timestamp`, `event_type`, `event_properties` |
| Location     | Geographic data           | `geo_coordinates`, `country_code`, `postal_code`    |
| Marketing    | Campaign and engagement   | `campaign_id`, `channel`, `conversion_timestamp`    |

### Using a preset

When configuring a new dataset, you can select a preset to automatically suggest mappings for standard attributes. This accelerates the mapping process for common data types.

### Creating custom presets

Organizations can create private presets that bundle:

* A set of attributes (global or organization-specific)
* Default transformation templates
* Validation rules

Custom presets standardize mapping across teams and datasets.

## The narrative.rosetta\_stone table

The `narrative.rosetta_stone` table is a virtual table that provides unified access to all normalized data.

### How it works

When you query `narrative.rosetta_stone`:

1. The query planner identifies which attributes you're selecting
2. It finds all datasets with active mappings for those attributes
3. It rewrites queries for each dataset using the mapping transformations
4. Results are unioned and returned in the normalized format

### Querying the table

```sql theme={null}
SELECT
    unique_identifier,
    hl7_gender,
    event_timestamp
FROM narrative.rosetta_stone
WHERE
    event_timestamp > CURRENT_TIMESTAMP - INTERVAL '30' DAY
    AND hl7_gender.gender = 'female'
```

This query automatically accesses all datasets with mappings for these three attributes, regardless of how the source data is structured.

## Scoping your queries

When querying Rosetta Stone, you can control which data sources are included by choosing an appropriate scope level.

### Global scope

Use `narrative.rosetta_stone` when you want to query all normalized data available to you:

* Includes data from all companies that have shared data with you
* Best for broad analysis across your entire data ecosystem
* Returns the widest possible result set

### Company scope

Use `company_data._rosetta_stone` or `<company_slug>._rosetta_stone` when you want data from a specific company:

* Limits results to datasets owned by that company
* Useful when you need data from a specific partner or your own organization
* Reduces the search space for faster queries

### Dataset scope

Use `company_data.<dataset>._rosetta_stone` when you need to work with a specific dataset:

* Required when combining normalized attributes with non-normalized columns from the same dataset
* Gives you precise control over exactly which data is queried
* Enables joining Rosetta Stone attributes back to source-specific columns

### Choosing the right scope

| I want to...                                          | Use this scope                                |
| ----------------------------------------------------- | --------------------------------------------- |
| Query all available normalized data                   | Global (`narrative.rosetta_stone`)            |
| Query data from my company only                       | Company (`company_data._rosetta_stone`)       |
| Query data from a specific partner                    | Company (`partner_slug._rosetta_stone`)       |
| Join normalized and original columns from one dataset | Dataset (`company_data."123"._rosetta_stone`) |
| Control exactly which datasets contribute to results  | Dataset                                       |

For complete syntax reference, see [Rosetta Stone table references](/nql/general/syntax#rosetta-stone-table-references).

### System columns

The table includes system columns for traceability:

| Column                   | Description                 |
| ------------------------ | --------------------------- |
| `_nio_source_dataset_id` | ID of the source dataset    |
| `_nio_source_row_id`     | Original row identifier     |
| `_nio_mapping_version`   | Version of the mapping used |

## Data quality implications

Normalization through Rosetta Stone improves data quality in several ways:

**Consistency**: All data adheres to the same type system and validations, regardless of source.

**Completeness**: Missing or malformed data is flagged during mapping, enabling targeted data quality improvements.

**Comparability**: Data from different sources can be meaningfully combined because it shares a common semantic model.

**Traceability**: System columns maintain lineage back to source data.

### Handling validation failures

When data fails validation during mapping, the system can:

1. **Reject**: Exclude the record from the normalized view
2. **Default**: Map to a default value (e.g., `unknown` for invalid enums)
3. **Flag**: Include the record but mark it for review

The behavior is configured per mapping.

## Related content

<CardGroup>
  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand the normalization pipeline
  </Card>

  <Card title="Attribute Types Reference" icon="shapes" href="/reference/rosetta-stone/attribute-types">
    Complete type system documentation
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Create mappings for your data
  </Card>

  <Card title="Creating Normalized Attributes" icon="pen-ruler" href="/guides/rosetta-stone/creating-normalized-attributes">
    Best practices for designing attributes
  </Card>
</CardGroup>


# Rosetta Stone Overview
Source: https://docs.narrative.io/concepts/rosetta-stone/overview

How Narrative enables data collaboration across organizations with different schemas

In the landscape of data management, every organization collects and stores data in its own "language"—a format shaped by internal systems, legacy decisions, and unique business requirements. This diversity creates a significant barrier to data collaboration: before two organizations can work together, someone must decode and translate each party's data into a common format.

Rosetta Stone eliminates this barrier.

## The data collaboration problem

Consider a simple field like gender. Three different data providers might store this information as:

| Provider   | Column name   | Values               |
| ---------- | ------------- | -------------------- |
| Provider A | `gender`      | `"male"`, `"female"` |
| Provider B | `sex`         | `"M"`, `"F"`         |
| Provider C | `gender_code` | `1`, `2`             |

Without normalization, consuming data from all three providers requires:

* Understanding each provider's schema
* Writing custom ETL logic for each integration
* Maintaining these transformations as schemas evolve
* Repeating this work for every new partnership

This approach doesn't scale. As organizations add more data partners, the complexity grows exponentially.

## What Rosetta Stone is

Rosetta Stone is Narrative's universal translator for data collaboration. It takes data from providers in their native format and normalizes it to a common schema—enabling you to query, combine, and analyze data from multiple sources without writing custom integrations.

The system works through two core primitives:

* **Attributes**: Standardized field definitions that form the common schema (like `hl7_gender` with values `male`, `female`, `other`, `unknown`)
* **Mappings**: Translations that connect each provider's columns to the appropriate attributes, including any necessary transformations

When you query through Rosetta Stone, the platform automatically translates your request to each provider's native schema, executes the query, and returns normalized results.

## Real-world example: Weather data

Imagine you need weather data from multiple providers for a logistics optimization project. Each provider structures their data differently:

**Provider A (European):**

* Temperature in Celsius
* Timestamps in `DD/MM/YYYY HH:mm` format
* Wind speed in kilometers per hour

**Provider B (US-based):**

* Temperature in Fahrenheit
* Timestamps in `MM-DD-YYYY h:mm AM/PM` format
* Wind speed in miles per hour

**Provider C (Scientific):**

* Temperature in Kelvin
* Timestamps in ISO 8601 format
* Wind speed in meters per second

Without Rosetta Stone, you'd need to:

1. Learn each provider's schema
2. Write conversion logic for each field
3. Handle edge cases (what does Provider A do with negative temperatures?)
4. Maintain this code as providers update their formats

With Rosetta Stone, you simply query the normalized attributes (`temperature_celsius`, `event_timestamp`, `wind_speed_kph`), and the platform handles all translations automatically.

## The query planner analogy

Rosetta Stone functions like a sophisticated query planner that operates across organizational boundaries. When you request data:

1. You specify what you need using standardized attribute names
2. Rosetta Stone identifies which providers have the relevant data
3. The system translates your query to each provider's native format
4. Results are normalized and returned in a consistent structure

This abstraction means you can add new data providers without changing your queries—the platform handles the translation layer.

## Key benefits

**No ETL pipelines per partnership**
Traditional data collaboration requires building custom integrations for each partner. Rosetta Stone eliminates this by providing a universal translation layer.

**Schema changes handled transparently**
When a provider updates their schema, Narrative updates the mappings. Your queries continue working without modification.

**Query once, access many**
A single query against Rosetta Stone can retrieve and combine data from dozens of providers, each with their own schemas.

**Data quality enforcement**
Mappings include validations that ensure data meets quality standards before it enters the normalized view—catching issues like out-of-range values or malformed dates.

## Related content

<CardGroup>
  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand attributes, mappings, and the normalization pipeline
  </Card>

  <Card title="The Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    Explore the type system and data quality enforcement
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Learn how to create mappings for your data
  </Card>

  <Card title="Normalize Your Data" icon="graduation-cap" href="/getting-started/normalize-data">
    Hands-on tutorial for normalizing your first dataset
  </Card>
</CardGroup>


# API Key Permissions
Source: https://docs.narrative.io/concepts/security/api-key-permissions

Understand how scoped permissions control what API keys can do and protect your resources

API key permissions control what each key can do within the Narrative platform. By assigning specific permissions to each key, you limit its capabilities to only what's needed—reducing risk and making your integrations easier to reason about.

## Why scope API key permissions

**Principle of least privilege.** A key that can only read datasets cannot accidentally modify or delete them. Scoping permissions limits the blast radius of a compromised or misconfigured key to only the resources it was granted access to.

**Operational clarity.** When each key has a clear, narrow purpose, it's easy to understand what a key is for just by looking at its permissions. A key with `Read Datasets` and `Read Queries` is obviously for analytics, not for data ingestion or workflow management.

**Auditability.** Scoped keys make audit logs more meaningful. When an action occurs, the key's permissions tell you not just *who* performed it, but what category of operation was intended—helping you quickly identify unexpected behavior.

## How permissions work

Every API key permission is a combination of two things:

* **Access level** — What the key can do: **Read** (view and list resources) or **Write** (create, update, and delete resources)
* **Resource** — Which functional area of the platform the permission applies to, such as Datasets, Connections, or Jobs

For example, a key with `Read Datasets` permission can view dataset metadata, samples, and statistics, but cannot create new datasets or modify existing ones. Adding `Write Datasets` would grant the ability to create, update, and delete datasets as well.

## How permissions compose

API keys can hold multiple permissions. Permissions are **additive**—each permission grants additional capabilities, and there are no deny rules. A key's effective access is the union of all its assigned permissions.

If a key attempts an operation that requires a permission it doesn't have, the API returns a `401 Unauthorized` response. This applies even if the key has other permissions—each resource and access level is checked independently.

## Choosing the right scope

When deciding which permissions to assign to a key, consider:

* **Think about the key's purpose.** What specific tasks will this key perform? Start with the minimum permissions needed for those tasks and add more only if required.
* **Separate concerns across keys.** Use different keys for different systems or workflows. A key for your analytics dashboard shouldn't also have permissions to manage workflows or modify datasets.
* **Review permissions periodically.** As your integration evolves, the permissions a key needs may change. Audit your keys regularly and remove permissions that are no longer necessary.

For a complete list of all available resources and their access levels, see the [Permissions Reference](/reference/security/permissions).

## Related content

<CardGroup>
  <Card title="Permissions Reference" icon="table" href="/reference/security/permissions">
    All API key permission resources and their available access levels
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys with scoped permissions
  </Card>

  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    How Narrative protects your data through architecture and access controls
  </Card>

  <Card title="SDK Authentication" icon="code" href="/guides/sdk/authentication">
    Configure API key authentication for the TypeScript SDK
  </Card>
</CardGroup>


# Data Pseudonymization
Source: https://docs.narrative.io/concepts/security/data-pseudonymization

How Narrative uses hashed identifiers to enable privacy-preserving data collaboration

Data collaboration often involves matching records across different organizations—connecting a customer in your database to their activity in a partner's dataset. But sharing raw email addresses or phone numbers creates privacy and compliance risks. Pseudonymization solves this problem by replacing identifiable information with hashed values that enable matching without exposing the underlying data.

## Why pseudonymization matters

When organizations collaborate on data, they need a way to identify common records without sharing sensitive information. Consider two companies that want to find overlapping customers:

* **Company A** has email addresses for their customers
* **Company B** has email addresses for their customers
* Both want to find the intersection without revealing their full customer lists

By hashing email addresses before sharing, both companies can find matches (identical hashes indicate identical emails) without either party seeing the other's raw data.

### Regulatory compliance

Privacy regulations like GDPR, CCPA, and HIPAA recognize pseudonymization as a data protection technique. While pseudonymized data is still considered personal data under these regulations, it receives favorable treatment because:

* The data cannot be attributed to a specific individual without additional information
* It reduces risk if the data is exposed or breached
* It demonstrates a commitment to data minimization principles

### Reduced breach impact

If hashed data is exposed, attackers cannot directly use it to contact or identify individuals. While hashing is not encryption and determined attackers could attempt to reverse common values, it significantly raises the barrier compared to raw PII.

***

## How hashing works

A hash function is a one-way mathematical transformation that converts input data into a fixed-length string of characters. The key properties that make hashing useful for pseudonymization:

### Deterministic

The same input always produces the same output. This is essential for matching—if two organizations hash the same email address, they get identical hashes.

```
johndoe@gmail.com → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
johndoe@gmail.com → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3  (same hash)
```

### One-way (non-reversible)

You cannot mathematically reverse a hash to recover the original input. There is no "unhash" function.

```
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3 → ???  (cannot reverse)
```

### Collision-resistant

It is computationally infeasible to find two different inputs that produce the same hash. This ensures that matching hashes truly indicate matching inputs.

### Sensitive to input changes

Even tiny changes to the input produce completely different hashes. This is why pre-formatting (lowercase, whitespace removal) is critical—`JohnDoe@gmail.com` and `johndoe@gmail.com` produce entirely different hashes.

```
johndoe@gmail.com  → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
JohnDoe@gmail.com  → 8f5a8e5e9a8f5e8a9e5f8a5e9a8f5e8a9e5f8a5e9a8f5e8a9e5f8a5e9a8f5e8a  (completely different)
```

***

## Supported hash algorithms

Narrative supports three widely-used hash algorithms. Each produces a different output length and has different characteristics:

| Algorithm   | Output                 | Status      | Notes                                                                                                                |
| ----------- | ---------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------- |
| **MD5**     | 128-bit (32 hex chars) | Legacy      | Fast, widely supported. Cryptographically broken but acceptable for non-security use cases like identifier matching. |
| **SHA-1**   | 160-bit (40 hex chars) | Legacy      | More secure than MD5. Also cryptographically deprecated but acceptable for identifier matching.                      |
| **SHA-256** | 256-bit (64 hex chars) | Recommended | Part of the SHA-2 family. Currently considered secure and is the preferred choice for new implementations.           |

<Tip>
  For maximum compatibility when matching across datasets, generate all three hash formats. Different organizations may have standardized on different algorithms.
</Tip>

***

## Pseudonymization vs. anonymization

These terms are often confused but have important distinctions:

| Aspect                  | Pseudonymization                     | Anonymization                        |
| ----------------------- | ------------------------------------ | ------------------------------------ |
| **Reversibility**       | Possible with additional information | Irreversible by design               |
| **Matching capability** | Yes—same input produces same output  | No—cannot match records              |
| **Regulatory status**   | Still personal data (GDPR)           | Not personal data                    |
| **Use case**            | Data collaboration, matching         | Aggregate analytics, public datasets |

Hashing is a pseudonymization technique, not anonymization. The hashed value still represents a specific individual—you just cannot determine who without additional context or resources.

### Why not anonymize?

True anonymization would prevent record matching entirely, defeating the purpose of data collaboration. Pseudonymization strikes a balance: it protects individual privacy while enabling the business value of finding common records across datasets.

***

## Security considerations

While hashing provides meaningful privacy protection, it's important to understand its limitations:

### Dictionary attacks

Common email addresses (like `john@gmail.com`) could theoretically be discovered by hashing a dictionary of likely values and comparing. This is why:

* Narrative never exposes raw hash values to unauthorized parties
* Access controls govern who can query hashed identifiers
* The platform's security model adds additional protections

### Not a substitute for access controls

Pseudonymization reduces risk but doesn't eliminate the need for proper data governance. Always combine hashing with:

* Role-based access controls
* Query logging and auditing
* Data sharing agreements

***

## Related content

<CardGroup>
  <Card title="Narrative ID" icon="fingerprint" href="/concepts/security/narrative-id">
    Per-partner pseudonymous identifiers for secure collaboration
  </Card>

  <Card title="Hashing PII for Upload" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Step-by-step guide to formatting and hashing your data
  </Card>

  <Card title="Security Model" icon="shield-halved" href="/concepts/architecture/security-model">
    How Narrative protects data throughout the platform
  </Card>
</CardGroup>


# Hashing
Source: https://docs.narrative.io/concepts/security/hashing

Understanding cryptographic hash functions and their role in data protection

Hashing is the process of transforming input data into a fixed-length output using a cryptographic hash function. In data collaboration, hashing enables organizations to share and match data without exposing sensitive information like email addresses or phone numbers.

## Why hashing matters for data collaboration

When organizations want to find overlapping customers or match records across datasets, they need a common key. But sharing raw email addresses or other personally identifiable information (PII) creates privacy and compliance risks.

Hashing solves this by:

* Converting PII into non-reversible pseudonyms
* Enabling deterministic matching (same input = same output)
* Allowing data collaboration without exposing underlying identifiers

***

## Properties of hash functions

Hash functions have specific properties that make them useful for data protection:

### Deterministic (consistent)

The same input always produces the same output, regardless of when or where the hash is computed. This enables matching—if two organizations hash the same email address, they get identical hashes.

```
johndoe@gmail.com → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
johndoe@gmail.com → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3  (always same)
```

### One-way (irreversible)

You cannot mathematically reverse a hash to recover the original input. There is no "unhash" function. This protects the underlying PII.

```
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3 → ???  (cannot reverse)
```

### Collision-resistant

Different inputs produce different outputs. While theoretical collisions exist (the output space is finite), well-designed hash functions make them computationally infeasible to find.

### Computationally efficient

Hashing billions of records is practical. The algorithms are designed to be fast while maintaining security properties.

### Sensitive to input

Even tiny changes produce completely different outputs. This is why input normalization matters—`johndoe@gmail.com` and `JohnDoe@Gmail.com` produce entirely different hashes.

***

## Common hash algorithms

Three hash algorithms are commonly used for pseudonymization in data collaboration:

| Algorithm   | Output Length | Status      | Example Output                                                     |
| ----------- | ------------- | ----------- | ------------------------------------------------------------------ |
| **MD5**     | 32 hex chars  | Legacy      | `0c036b871e3c66c1724f68fd007c4718`                                 |
| **SHA-1**   | 40 hex chars  | Legacy      | `fafe72dfca878eb2084fb7478a44d279f7895b9b`                         |
| **SHA-256** | 64 hex chars  | Recommended | `ed2e59c337d01185f388a4e9334d6f2e5cb29652f222afe4b692582d2e1c3fce` |

<Tip>
  For maximum interoperability, generate all three hash formats when preparing data for collaboration. Different partners may have standardized on different algorithms.
</Tip>

### Algorithm considerations

* **MD5 and SHA-1** are cryptographically broken for security purposes but remain acceptable for identifier matching where collision attacks aren't a concern
* **SHA-256** (part of the SHA-2 family) is currently considered secure and is recommended for new implementations
* All three produce deterministic outputs suitable for matching

***

## Input normalization

Hash functions are sensitive to their input—any difference produces a completely different output. Before hashing, normalize your data:

### Case sensitivity

Standardize to lowercase for case-insensitive identifiers like email addresses:

```
MD5("narrative")  → a936fba0061bf85904722500de008e51
MD5("NARRATIVE")  → 2402a932631e086f028721943b7350c8  (different!)
MD5("Narrative")  → 0c036b871e3c66c1724f68fd007c4718  (different!)
```

### Whitespace

Remove leading and trailing whitespace:

```
MD5(" johndoe@gmail.com ")  → different from MD5("johndoe@gmail.com")
```

### Phone number formatting

Normalize to E.164 format before hashing:

```
Original: (415) 555-1234
Normalized: +14155551234
Then hash: SHA256("+14155551234")
```

***

## Security considerations

### Dictionary attacks

While hash functions are irreversible, attackers can create dictionaries mapping common inputs to their hash outputs. If someone has a dictionary of all possible email hashes, they can look up your hash to find the original email.

This is most concerning for inputs with limited possibilities:

* **Phone numbers:** Only \~10 billion 10-digit numbers—feasible to hash them all
* **Common emails:** Popular email patterns could be pre-computed
* **Short strings:** Limited character combinations

<Warning>
  Hashing alone may not provide sufficient protection for high-value identifiers with limited input spaces. Consider additional protections like Narrative ID for sensitive use cases.
</Warning>

### Mitigation strategies

Narrative's platform adds protections beyond simple hashing:

* Access controls limit who can query hashed identifiers
* [Narrative ID](/concepts/security/narrative-id) provides per-partner encoding
* Query logging enables auditing of data access

### Not a substitute for access controls

Hashing reduces risk but doesn't eliminate the need for proper data governance. Always combine hashing with:

* Role-based access controls
* Query logging and auditing
* Data sharing agreements

***

## Related content

<CardGroup>
  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Broader context for pseudonymization techniques
  </Card>

  <Card title="PII" icon="id-card" href="/concepts/security/pii">
    Understanding personally identifiable information
  </Card>

  <Card title="Narrative ID" icon="fingerprint" href="/concepts/security/narrative-id">
    Enhanced identifier protection
  </Card>

  <Card title="Hashing PII for Upload" icon="upload" href="/guides/ingestion/hashing-pii">
    Step-by-step hashing guide
  </Card>
</CardGroup>


# Narrative ID
Source: https://docs.narrative.io/concepts/security/narrative-id

Per-partner pseudonymous identifiers for privacy-safe data collaboration

Data collaboration requires matching records across organizations—finding the same customer in both your database and a partner's dataset. But sharing raw identifiers creates privacy risks, and even standard hashing has limitations: if two companies hash the same email with SHA-256, they produce identical hashes that could be correlated across partnerships. Narrative ID solves this by creating partner-specific pseudonymous identifiers that enable matching without exposing underlying data or allowing cross-partner correlation.

## Why Narrative ID matters

### Beyond simple hashing

Traditional pseudonymization through hashing converts identifiers into fixed outputs:

```
johndoe@gmail.com → 06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
```

This hash is the same everywhere. If Company A and Company B both hash their customer emails, they can match records—but so could any third party with access to both datasets. The hash becomes a de facto identifier.

Narrative ID takes a different approach: the same identifier produces **different** encoded values for each partner.

```
johndoe@gmail.com → (encoded for Partner A) → NID_a8f3e2b1c9d4...
johndoe@gmail.com → (encoded for Partner B) → NID_7c2d9e5f1a8b...
```

These two Narrative IDs represent the same person but cannot be correlated without going through Narrative's translation system. This provides an additional layer of privacy protection beyond standard pseudonymization.

### Key benefits

| Aspect                           | Standard Hashing                        | Narrative ID                                            |
| -------------------------------- | --------------------------------------- | ------------------------------------------------------- |
| **Cross-partner correlation**    | Possible (same hash everywhere)         | Prevented (per-partner encoding)                        |
| **Translation between partners** | Not possible                            | Supported via Narrative platform                        |
| **Identifier exposure risk**     | Hash could leak to unauthorized parties | Encoded IDs only meaningful within their encoding space |
| **Matching capability**          | Direct hash comparison                  | Translation-based matching                              |

***

## How Narrative ID works

### Secure encoding methodology

When a clear text identifier (such as an email address or hashed email) is encoded into a Narrative ID, the encoding process transforms it into a value that:

* **Conceals the original data**: The encoded value reveals nothing about the underlying identifier
* **Is deterministic within an encoding space**: The same identifier always produces the same Narrative ID for a given partner

```
Clear text identifier → Secure encoding with partner key → Narrative ID
```

### Per-partner encoding spaces

Each partner on the Narrative platform has their own **encoding space**—a unique cryptographic context for generating Narrative IDs. When an identifier is encoded for a specific partner, the result is tied to that partner's encoding space.

This means:

* **Partner A's Narrative ID** for `johndoe@gmail.com` differs from **Partner B's Narrative ID** for the same email
* Neither partner can determine the other's Narrative ID without translation
* A Narrative ID only has meaning within its encoding space

### Translation between partners

Narrative's software provides the ability to translate Narrative IDs from one partner's encoding space to another. This enables secure collaboration:

1. **Partner A** encodes their customer identifiers into their encoding space
2. **Partner A** shares data with Narrative IDs (not raw identifiers)
3. **Narrative** translates the Narrative IDs to **Partner B's** encoding space
4. **Partner B** can now match against their own Narrative IDs

The translation happens within the Narrative platform—either in the Narrative cloud or in a customer-hosted data plane—ensuring that clear text identifiers are never exposed during the collaboration.

```sql theme={null}
-- Translate Narrative IDs from Partner A's space to Partner B's space
SELECT
  NARRATIVE_ID_TRANSLATE(partner_a_narrative_id, 'partner_b_key_id') AS partner_b_narrative_id,
  other_columns
FROM partner_a_data
```

***

## Key concepts

### Encoding spaces

An encoding space is the cryptographic namespace associated with a specific partner. Think of it as a private room where identifiers are transformed into Narrative IDs using that partner's unique encoding key.

Key properties:

* **Isolated**: Each partner's encoding space is independent
* **Consistent**: The same identifier always maps to the same Narrative ID within an encoding space
* **Non-transferable**: A Narrative ID from one encoding space cannot be directly used in another

### Match tables

A **match table** is a pre-generated lookup table that pairs Narrative IDs with their corresponding clear text identifiers. Match tables enable partners to use Narrative IDs independently of the Narrative platform for certain workflows.

Use cases for match tables:

* Offline matching processes
* Integration with systems that cannot connect to the Narrative platform directly
* Performance optimization for high-volume lookups

Match tables are generated within the secure environment and maintain the privacy guarantees of the Narrative ID system.

***

## Security considerations

### What Narrative ID protects against

* **Identifier exposure**: Raw identifiers (emails, phone numbers) are never shared between partners
* **Cross-partner correlation**: Without translation through Narrative, partners cannot determine if they share common records
* **Unauthorized access**: Even if Narrative IDs are intercepted, they reveal nothing about the underlying individuals

### Defense in depth

Narrative ID is one layer in a comprehensive security approach:

* **Access controls** govern who can query and translate Narrative IDs
* **Audit logging** tracks all encoding and translation operations
* **Data plane isolation** keeps raw data within controlled environments

### Not a substitute for governance

While Narrative ID provides strong privacy protections, organizations should still maintain:

* Data sharing agreements with partners
* Clear policies on what data can be collaborated on
* Compliance documentation for regulatory requirements

***

## Related content

<CardGroup>
  <Card title="Using Narrative ID" icon="code" href="/guides/collaboration/using-narrative-id">
    Step-by-step guide to encoding and translating identifiers
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How hashing protects privacy in data collaboration
  </Card>

  <Card title="NQL Functions" icon="function" href="/nql/functions/index#narrative-specific-functions">
    NARRATIVE\_ID\_ENCODE and NARRATIVE\_ID\_TRANSLATE reference
  </Card>

  <Card title="Security Model" icon="shield-halved" href="/concepts/architecture/security-model">
    Platform-wide security architecture
  </Card>
</CardGroup>


# Personally Identifiable Information (PII)
Source: https://docs.narrative.io/concepts/security/pii

Understanding PII, regulatory definitions, and how to handle it in data collaboration

Personally Identifiable Information (PII) is data that can be used to identify a specific individual, either on its own or when combined with other information. As data collection has become ubiquitous through digital technologies, understanding and properly handling PII has become essential for compliance and trust.

## What is PII?

The National Institute of Standards and Technology (NIST) defines PII as:

> Information which can be used to distinguish or trace the identity of an individual (e.g., name, social security number, biometric records, etc.) alone, or when combined with other personal or identifying information which is linked or linkable to a specific individual (e.g., date and place of birth, mother's maiden name, etc.).

The key concept is **identifiability**—whether data can be linked back to a specific person.

***

## Common examples of PII

While no comprehensive list exists, these attributes are universally recognized as PII:

### Direct identifiers

Data that identifies an individual on its own:

* Full name
* Email address
* Phone number
* Social Security number
* Passport or driver's license number
* Physical address

### Indirect identifiers

Data that can identify individuals when combined:

* Date of birth
* Place of birth
* Gender
* Race or ethnicity
* Employer
* Job title

### Digital identifiers

Context-dependent identifiers from online activity:

* IP addresses
* Device identifiers (IDFA, GAID)
* Cookie IDs
* Precise geolocation

<Note>
  Whether something constitutes PII can depend on context. An IP address might not identify an individual in isolation, but combined with other data, it could. Err on the side of caution when handling potentially identifying data.
</Note>

***

## Regulatory definitions

Privacy regulations worldwide define personal data similarly but with important nuances.

### GDPR (European Union)

The General Data Protection Regulation, effective since 2018, is the most comprehensive privacy framework globally:

> 'Personal data' means **any information relating to an identified or identifiable natural person** ('data subject'); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person.

Key points:

* Broad definition includes online identifiers
* Covers indirect identification
* Applies to EU residents regardless of where data is processed

### CCPA (California)

The California Consumer Privacy Act, effective 2020 and amended by CPRA:

> "Personal information" means **information that identifies, relates to, describes, is capable of being associated with, or could reasonably be linked, directly or indirectly**, with a particular consumer or household.

Key points:

* Includes household-level data
* Covers information "capable of being associated" with individuals
* Applies to California residents

### Other regulations

Many jurisdictions have enacted similar laws:

* Australia's Privacy Act of 1988
* UK Data Protection Act / UK GDPR
* Switzerland's Federal Act on Data Protection
* Brazil's LGPD
* Canada's PIPEDA

Each law has nuances—consult legal counsel for specific requirements.

***

## PII in data collaboration

### The challenge

Data collaboration often requires matching records across organizations—finding common customers, enriching profiles, or measuring campaign effectiveness. These use cases typically need identifiers to match on, but sharing raw PII creates legal and reputational risk.

### The solution: pseudonymization

[Pseudonymization](/concepts/security/data-pseudonymization) transforms PII into non-identifying values while preserving the ability to match records. The most common technique is [hashing](/concepts/security/hashing), which creates a deterministic but irreversible transformation.

| Original Record           | Pseudonymized Record    |
| ------------------------- | ----------------------- |
| `email: john@example.com` | `hashed_email: 5ab6...` |
| `phone: +14155551234`     | `hashed_phone: 8f2a...` |
| `age: 36`                 | `age: 36`               |
| `gender: male`            | `gender: male`          |

The pseudonymized record:

* Can still be matched to other datasets
* Doesn't expose the underlying PII
* Reduces compliance risk

### Narrative's approach

Narrative requires PII to be hashed before upload:

* Email addresses must be hashed (MD5, SHA-1, or SHA-256)
* Phone numbers must be hashed after E.164 normalization
* Raw PII is not accepted in the platform

This ensures that data collaboration can occur without PII exposure.

***

## Best practices

### Minimize collection

Only collect PII that's necessary for your specific use case. More data means more risk.

### Pseudonymize early

Hash identifiers as early as possible in your data pipeline—ideally at the point of collection.

### Maintain consent records

Document the legal basis for processing personal data, whether consent, legitimate interest, or contractual necessity.

### Implement access controls

Limit who can access PII and pseudonymized data within your organization.

### Plan for data subject requests

Be prepared to respond to access, deletion, and correction requests required by privacy regulations.

***

## Related content

<CardGroup>
  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    How pseudonymization protects PII
  </Card>

  <Card title="Hashing" icon="hashtag" href="/concepts/security/hashing">
    Technical details of hash functions
  </Card>

  <Card title="GDPR" icon="globe-europe" href="/concepts/compliance/gdpr">
    European data protection regulation
  </Card>

  <Card title="CCPA" icon="flag-usa" href="/concepts/compliance/ccpa">
    California privacy law
  </Card>

  <Card title="Hashing PII for Upload" icon="upload" href="/guides/ingestion/hashing-pii">
    Prepare your data for the platform
  </Card>
</CardGroup>


# Single Sign-On (SSO)
Source: https://docs.narrative.io/concepts/security/sso

Understand how SSO works with Narrative I/O and why it matters for enterprise security

Single Sign-On (SSO) allows your team to access Narrative using your organization's existing identity provider. Instead of managing separate login credentials, your team authenticates through the same system they use for other enterprise applications.

## Why use SSO

Organizations choose SSO for three main reasons:

**Simplified access management.** When employees join your organization, granting Narrative access is as simple as adding them to the appropriate group in your identity provider. When they leave, removing their IdP access immediately revokes their Narrative access. No separate password resets or account deactivations required.

**Enhanced security.** SSO integrates with your existing security policies, including multi-factor authentication requirements enforced by your identity provider. Your security team maintains centralized control over authentication standards across all enterprise applications, including Narrative.

**Streamlined user experience.** Your team uses one set of credentials for everything. No more password fatigue from managing application-specific logins—just a consistent authentication experience that matches how they access other enterprise tools.

## How SAML 2.0 authentication works

Narrative uses SAML 2.0, the industry standard for enterprise single sign-on. SAML (Security Assertion Markup Language) enables secure communication between your identity provider and Narrative.

When a user attempts to access Narrative:

1. **User initiates login** — They enter your organization's slug on the Narrative login page
2. **Redirect to your IdP** — Narrative redirects them to your identity provider (Okta, Azure AD, etc.)
3. **User authenticates** — They log in using your organization's credentials and MFA requirements
4. **Access granted** — Your IdP confirms their identity, and Narrative grants access

This flow ensures your identity provider remains the single source of truth for user authentication. Narrative never sees or stores your users' passwords.

## Supported identity providers

Narrative supports any SAML 2.0-compliant identity provider. We provide specific configuration guidance for:

* **Okta** — Full support with detailed attribute mapping
* **Microsoft Azure AD (Entra ID)** — Full support with detailed attribute mapping
* **OneLogin** — Full support with detailed attribute mapping

Using a different provider? Our support team can assist with configuration for any SAML-compliant IdP.

## Key features

| Feature                       | Description                                               |
| ----------------------------- | --------------------------------------------------------- |
| **SAML 2.0**                  | Industry-standard secure authentication protocol          |
| **Just-in-time provisioning** | Automatically create Narrative accounts on first login    |
| **SSO enforcement**           | Optionally require SSO for all users—no password fallback |
| **Role-based access**         | Assign admin or standard roles to organization members    |
| **Self-service admin portal** | Your admins manage SSO configuration and members directly |

## Availability

SSO is available to enterprise customers with qualifying contracts. Contact your Narrative account manager to confirm eligibility and provision your organization.

## Related content

<CardGroup>
  <Card title="SSO Configuration" icon="shield-keyhole" href="/account-settings/sso">
    Step-by-step instructions for configuring SSO with your identity provider
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Manage API keys for programmatic access to Narrative
  </Card>
</CardGroup>


# Webhooks
Source: https://docs.narrative.io/concepts/webhooks/overview

How webhooks deliver push-based notifications for job state changes

A webhook subscription is a registration that tells Narrative to send an HTTP POST request to your URL whenever a [job](/reference/architecture/job-types) changes state. Instead of polling the jobs API to check whether something finished, you receive a notification the moment it happens.

## Why webhooks exist

Many Narrative operations are asynchronous. When you create a materialized view, refresh a dataset, or run an inference job, the platform returns a job ID and processes the work in the background. To know when the job finishes, you have two options:

**Polling** means repeatedly calling the jobs API to check the current state. This works, but it's wasteful—most requests return the same "still running" response—and introduces latency between when the job completes and when your system finds out.

**Webhooks** invert the flow. You register a URL once, and Narrative pushes an event to you the moment a job transitions to a state you care about. No wasted API calls, no delay.

Webhooks are especially valuable when you need to:

* **Trigger downstream workflows** — start a data pipeline step as soon as the previous job completes
* **Alert on failures** — notify your team immediately when a job fails, rather than discovering it during the next manual check
* **Integrate with external systems** — forward job status to Slack, PagerDuty, or your own orchestration layer

## How job webhooks work

When you create a webhook subscription, you specify:

1. **A URL** — the HTTPS endpoint that receives event payloads
2. **Filters** (optional) — which jobs you care about, by job ID, job type, or job state

When a job transitions to a matching state, Narrative sends an HTTP POST request to your URL with a JSON payload describing the event.

### Filtering

Filters let you control which events trigger a notification. You can combine them to match exactly the events you need:

| Filter      | Effect                                                                    |
| ----------- | ------------------------------------------------------------------------- |
| `job_ids`   | Only notify for specific job IDs                                          |
| `job_types` | Only notify for specific job types (e.g., `materialize-view`, `forecast`) |
| `states`    | Only notify for specific state transitions (e.g., `completed`, `failed`)  |

If you omit all filters, the subscription fires for every job state change in your company.

## How delivery works

```mermaid theme={null}
sequenceDiagram
    participant J as Job System
    participant W as Webhook Service
    participant Y as Your Endpoint

    J->>W: Job state changed
    W->>W: Match against subscriptions
    W->>Y: HTTP POST with event payload
    Y->>W: 200 OK
```

1. A job transitions to a new state (e.g., `running` to `completed`)
2. The webhook service checks all active subscriptions for that company
3. For each matching subscription, it sends an HTTP POST to the registered URL
4. Your endpoint should respond with a `2xx` status code to acknowledge receipt

Each event includes an [idempotency key](/reference/glossary#idempotency-key) so your endpoint can detect and discard duplicate deliveries.

## Webhook lifecycle

Subscriptions have two statuses:

| Status     | Description                                                |
| ---------- | ---------------------------------------------------------- |
| `active`   | The subscription is live and will receive events           |
| `archived` | The subscription is disabled and no longer receives events |

When you no longer need a subscription, archive it with a `DELETE` request. Archived subscriptions are retained for audit purposes but stop generating events.

## Security

Each subscription is assigned a **shared secret** (a UUID) when it's created. This secret is returned in the creation response and should be stored securely. Use it to verify that incoming webhook requests genuinely originated from Narrative rather than a third party.

All webhook endpoints must use **HTTPS** to ensure payloads are encrypted in transit.

<Note>
  Narrative also supports **app webhooks** for application-level events. App webhooks use a different subscription type (`webhook_subscription_app`) and are scoped to a specific application rather than job state changes. This documentation focuses on job webhooks, which are the most common use case.
</Note>

***

## Related content

<CardGroup>
  <Card title="Subscribing to Notifications" icon="bell" href="/guides/webhooks/subscribing-to-notifications">
    Step-by-step guide to creating and managing webhook subscriptions
  </Card>

  <Card title="Webhook Event Reference" icon="list-check" href="/reference/webhooks/event-reference">
    Complete field reference for webhook requests and responses
  </Card>

  <Card title="Tracking Job Status" icon="spinner" href="/guides/sdk/tracking-jobs">
    Poll-based alternative for monitoring job progress
  </Card>

  <Card title="Job Types" icon="list" href="/reference/architecture/job-types">
    All job types that can trigger webhook events
  </Card>
</CardGroup>


# Workflow Orchestration
Source: https://docs.narrative.io/concepts/workflows/workflow-orchestration

How workflows automate multi-step data pipelines and relate to the jobs system

A workflow is a declarative pipeline that executes multiple data operations in sequence. Instead of manually triggering each step and waiting for it to finish before starting the next, you define the full pipeline once and let the system handle orchestration.

## Why workflows exist

Workflows solve problems that arise when data pipelines involve multiple dependent steps:

**Manual orchestration is error-prone.** If you need to create a materialized view, wait for it to finish, then create a second view that reads from the first, doing this manually introduces risk. You might trigger the second step before the first completes, or forget a step entirely.

**Complex pipelines need coordination.** As pipelines grow beyond two or three steps, tracking which step is running, what's finished, and what's next becomes increasingly difficult—especially when multiple team members are involved.

**Repeatable processes should be automated.** If you're running the same sequence of operations daily or weekly, defining it once as a workflow eliminates repetitive manual work and ensures consistency.

## Workflows, tasks, and jobs

Understanding the relationship between these three concepts is key to working with the system effectively.

### The three layers

**Workflows** define *what* should happen—a named, versioned sequence of operations. A workflow is a blueprint, not an execution. You create a workflow once and trigger it many times.

**Tasks** are the individual steps within a workflow. Each task calls a specific operation (like creating a materialized view or refreshing one) with specific parameters. Tasks execute sequentially in the order they're defined.

**Jobs** are how tasks actually execute. The workflow system is an orchestration layer built *on top of* the existing [jobs system](/concepts/architecture/query-processing). When a task runs, it creates a job—the same kind of job that's created when you run a query or refresh a view manually.

### How execution flows

When you trigger a workflow, here's what happens:

1. The system creates a **run**—a specific execution instance with its own `run_id`
2. The first task starts and creates a **job** in the jobs system
3. The orchestrator waits for that job to complete
4. Once the job succeeds, the next task starts and creates its own job
5. This continues until all tasks complete or one fails

```mermaid theme={null}
flowchart LR
  subgraph Workflow
    direction LR
    T1("Task 1<br/><small>job_id_1</small>") --> T2("Task 2<br/><small>job_id_2</small>") --> T3("Task 3<br/><small>job_id_3</small>")
  end
```

Each task creates its own independent job. This means you can monitor execution at two levels:

* **Workflow level** — check the overall run status to see if the pipeline succeeded or failed
* **Job level** — inspect individual jobs for detailed timing, errors, and output

<Note>
  Jobs created by workflows are automatically tagged with `workflow_enqueued`. On the **Jobs** page, you can filter by the **Workflow Enqueued** status or search by `workflow_id` or `workflow_run_id` to distinguish workflow-generated jobs from manually triggered ones.
</Note>

## Task dependencies and data passing

Tasks in a workflow execute sequentially—each task waits for the previous one to complete before starting. This sequential model is simple but powerful: later tasks can reference data created by earlier ones.

For example, a workflow might:

1. Create a materialized view called `active_users`
2. Create a second materialized view that queries `company_data.active_users`

The workflow guarantees that step 1 finishes (and the dataset exists) before step 2 begins. Without workflows, you'd need to poll the jobs API and build this coordination logic yourself.

### Referencing earlier outputs

When a task creates a dataset (via `CreateMaterializedViewIfNotExists`), subsequent tasks can reference it using the fully qualified path `company_data.<dataset_name>`. The workflow's sequential execution guarantees the dataset exists by the time later tasks need it.

### Task output and workflow context

Beyond dataset references, tasks produce structured JSON output after execution — dataset IDs, snapshot IDs, row counts, and other metadata. This output enables more flexible data passing between tasks.

The **export** mechanism captures task output into a **workflow context** (`$context`), an object that accumulates data as the workflow progresses. Variable expressions (`${…}`) then inject context values or previous task output into subsequent task parameters.

For full syntax details, see the [Specification Syntax](/reference/workflows/specification-syntax#task-output) reference.

### Two approaches to data passing

Workflows support two ways to pass data between tasks:

**Dataset references** — When a task creates a dataset, later tasks query it by name (`company_data.<dataset_name>`). This is the simplest approach and works well when tasks consume each other's data directly via NQL.

**Output and context** — When tasks need to pass metadata (dataset IDs, row counts, status information) rather than query data, use `export` and `${…}` variable expressions. This is useful for logging, conditional DML operations, or any case where a downstream task needs a value produced by an upstream task.

Most workflows use dataset references. Output and context become valuable when you need to pass specific values — like inserting a dataset ID into a log table or using a row count in a subsequent operation.

## Error handling

Workflows use **fail-fast** behavior. If any task fails:

1. The workflow immediately stops execution
2. Remaining tasks are skipped
3. The run status becomes `failed`

This design prevents cascading failures—if step 1 fails to create a dataset, there's no point in running step 2 which depends on it.

There are no automatic retries. When a task fails, you need to investigate the cause, fix the underlying issue, and trigger a new run. You can inspect the failed task's job for detailed error information.

## Supported operations

Workflows currently support these operations:

| Task                                   | What it does                                                 |
| -------------------------------------- | ------------------------------------------------------------ |
| `CreateMaterializedViewIfNotExists`    | Creates a new materialized view from an NQL query            |
| `RefreshMaterializedView`              | Refreshes an existing materialized view with the latest data |
| `ExecuteDml`                           | Executes a DML statement (INSERT, UPDATE, DELETE)            |
| `RunModelInference`                    | Runs a model inference job                                   |
| `CreateRosettaStoneMappingsIfNotExist` | Creates Rosetta Stone attribute mappings for a dataset       |

These operations cover the most common pipeline patterns: building datasets, keeping them fresh, modifying data, running AI inference, and managing schema mappings. Direct `ExecuteNQL` (arbitrary SELECT queries) is not supported—use `CreateMaterializedViewIfNotExists` to execute a query and persist the results.

For full parameter details on each task, see the [Task Reference](/reference/workflows/tasks).

## Scheduling

Workflows can be triggered manually via the API or the platform UI, or run automatically on a schedule. Scheduled workflows use cron expressions to define when they run—hourly, daily, weekly, or on any custom cadence.

Scheduling is particularly valuable for refresh pipelines. Rather than remembering to refresh your materialized views every morning, a scheduled workflow handles it automatically and ensures the correct refresh order.

## Current limitations

The workflow system is designed for sequential, deterministic pipelines. It intentionally keeps things simple:

* **Sequential execution only** — tasks run one at a time, in order. There is no parallel task execution.
* **No conditional logic** — every task in the workflow runs. Tasks can pass data via context, but there is no if/else branching based on output values.
* **No loops** — you cannot iterate over a list of items or repeat tasks.
* **No automatic retries** — failed tasks must be investigated and rerun manually.

These constraints keep the system predictable and easy to reason about. For most data pipeline use cases—creating datasets, refreshing views, running DML—sequential execution with fail-fast error handling is sufficient.

***

## Related content

<CardGroup>
  <Card title="Automating Multi-Step Pipelines" icon="diagram-project" href="/guides/workflows/workflow-orchestration">
    Step-by-step guide to creating, managing, and running workflows
  </Card>

  <Card title="Task Reference" icon="rectangle-terminal" href="/reference/workflows/tasks">
    Complete reference for all supported workflow tasks and parameters
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    How materialized views work—the most common operation in workflows
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    The compute environment where workflow tasks execute
  </Card>
</CardGroup>


# Cookbooks
Source: https://docs.narrative.io/cookbooks/index

Complete, working examples for real-world use cases

Cookbooks provide complete, tested solutions for common scenarios. Each recipe includes all the code and configuration you need, along with explanations of what it accomplishes and when to use it.

## NQL Patterns

Reusable query patterns for common data tasks.

<CardGroup>
  <Card title="Common Queries" icon="star" href="/cookbooks/nql/common-queries">
    Frequently used query patterns and templates
  </Card>

  <Card title="Performance Patterns" icon="gauge-high" href="/cookbooks/nql/performance-patterns">
    Optimized patterns for large-scale data processing
  </Card>

  <Card title="Complex Joins" icon="code-merge" href="/cookbooks/nql/complex-joins">
    Multi-way joins and advanced data combination techniques
  </Card>

  <Card title="AI Data Enrichment" icon="wand-magic-sparkles" href="/cookbooks/nql/ai-enrichment">
    Enrich records with AI-generated attributes using AI\_COMPLETE
  </Card>
</CardGroup>

## Model Inference

AI-powered data processing patterns.

<CardGroup>
  <Card title="Data Enrichment" icon="wand-magic-sparkles" href="/cookbooks/sdk/model-inference-automation">
    Build a pipeline that enriches records using AI inference
  </Card>

  <Card title="Batch Processing" icon="layer-group" href="/cookbooks/sdk/batch-inference-patterns">
    Process large volumes of data efficiently
  </Card>

  <Card title="Auto-generate Descriptions" icon="file-lines" href="/cookbooks/use-cases/ai-dataset-descriptions">
    Generate dataset documentation with AI
  </Card>

  <Card title="Natural Language to CRON" icon="clock" href="/cookbooks/use-cases/natural-language-to-cron">
    Convert schedules to CRON expressions
  </Card>
</CardGroup>

## Use Cases

End-to-end solutions for specific industries and scenarios.

<CardGroup>
  <Card title="Retail Collaboration" icon="store" href="/cookbooks/use-cases/retail-collaboration">
    Share and analyze retail data across partners
  </Card>

  <Card title="Demographic Enrichment" icon="users" href="/cookbooks/use-cases/demographic-enrichment">
    Enrich customer lists with age and gender data
  </Card>

  <Card title="Evaluating Identity Providers" icon="scale-balanced" href="/cookbooks/use-cases/identity-provider-evaluation">
    A structured framework for choosing and measuring identity data providers
  </Card>
</CardGroup>

## Contributing recipes

Have a useful pattern or integration to share? Cookbooks are meant to grow with real-world usage. Contact us to contribute your own recipes.


# Data Enrichment with AI_COMPLETE
Source: https://docs.narrative.io/cookbooks/nql/ai-enrichment

Enrich records with AI-generated attributes using in-query LLM inference

This cookbook demonstrates how to use the `AI_COMPLETE` function to enrich records with AI-generated structured attributes directly within NQL. The entire inference pipeline runs inside your Snowflake data plane—your data never leaves your infrastructure.

<Note>
  `AI_COMPLETE` is available on **Snowflake data planes only**.
</Note>

## What you will build

A materialized view that takes raw customer support tickets and enriches each record with:

* A standardized issue category
* A priority level
* A sentiment score
* Extracted product mentions

The same pattern applies to any enrichment task: product classification, lead scoring, content tagging, entity extraction, and more.

## Prerequisites

* A [Snowflake data plane](/concepts/primitives/data-planes) with AI model access configured (see setup below)
* A dataset with text content to enrich
* Familiarity with [NQL syntax](/nql/general/syntax) and [materialized views](/concepts/nql/materialized-views)

### One-time Snowflake setup

`AI_COMPLETE` runs through [Snowflake Cortex](https://docs.snowflake.com/en/guides-overview-ai-features), so the customer Snowflake account must grant Cortex access to the Narrative application and allow cross-region model invocation. A Snowflake account admin runs these statements once:

```sql theme={null}
-- Allow Cortex to route requests to models hosted in any region
ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION';

-- Grant the Cortex role to the Narrative application
GRANT DATABASE ROLE snowflake.cortex_user TO APPLICATION <narrative_application_name>;
```

Replace `<narrative_application_name>` with the name of the Narrative Snowflake Native App installed in the account (for example, `NARRATIVE_DATA_COLLABORATION_HMI`).

<Warning>
  The `GRANT DATABASE ROLE ... TO APPLICATION` statement is a temporary workaround required by current Snowflake Native App permissioning. Snowflake may change how Native Apps access Cortex in the future, at which point this setup step may change. Contact Narrative support if `AI_COMPLETE` queries begin failing with permission errors.
</Warning>

<Note>
  `CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION'` lets Cortex invoke models that are hosted outside the account's home region. Narrative recommends `ANY_REGION` because model availability varies by region. If your compliance posture requires region pinning, consult [Snowflake's cross-region inference documentation](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cross-region-inference) for the available alternatives.
</Note>

To confirm the grant succeeded, run a minimal query with a literal prompt:

```sql theme={null}
CREATE MATERIALIZED VIEW "test_ai_complete" AS
WITH test_prompt AS (SELECT 'respond with ok' AS prompt)
SELECT
  AI_COMPLETE(
    'openai-gpt-5-nano',
    test_prompt.prompt,
    '{}',
    '{"type": "json", "schema": {"type": "object", "properties": {"reply": {"type": "string"}}, "required": ["reply"], "additionalProperties": false}}',
    TRUE
  ) AS model_response
FROM test_prompt
```

If this query fails with a privileges error, re-run the grant statements above.

## Step 1: Define the output schema

Before writing NQL, design the JSON Schema that describes the structured attributes you want the model to produce. This schema constrains the model's output, guaranteeing predictable, typed results.

```json theme={null}
{
  "type": "json",
  "schema": {
    "type": "object",
    "properties": {
      "category": {
        "type": "string",
        "description": "The primary issue category",
        "enum": [
          "billing",
          "technical",
          "account_access",
          "feature_request",
          "general_inquiry"
        ]
      },
      "priority": {
        "type": "string",
        "description": "Urgency level based on the issue description",
        "enum": ["low", "medium", "high", "critical"]
      },
      "sentiment": {
        "type": "number",
        "description": "Customer sentiment from -1 (very negative) to 1 (very positive)"
      },
      "product_mentioned": {
        "type": "string",
        "description": "The specific product or feature referenced, or 'none' if not applicable"
      }
    },
    "required": ["category", "priority", "sentiment", "product_mentioned"],
    "additionalProperties": false
  }
}
```

**Key schema design decisions:**

| Decision                            | Why                                                          |
| ----------------------------------- | ------------------------------------------------------------ |
| `enum` on `category` and `priority` | Restricts output to known values, preventing free-form drift |
| `number` for `sentiment`            | Enables numeric filtering and aggregation downstream         |
| `additionalProperties: false`       | Prevents the model from adding unexpected fields             |
| `description` on each field         | Guides the model's interpretation of what each field means   |

<Tip>
  Keep schemas focused on the fields your downstream queries will actually use. Every additional field increases token usage and inference cost.
</Tip>

## Step 2: Build prompt strings in a CTE

`AI_COMPLETE` requires its `prompt` parameter to be a column reference, not a string literal. Use a Common Table Expression (CTE) to assemble prompts from your source columns.

```sql theme={null}
WITH prompts AS (
  SELECT
    company_data."support_tickets".ticket_id,
    company_data."support_tickets".customer_id,
    company_data."support_tickets".created_at,
    'Analyze the following customer support ticket and extract structured attributes.'
      || ' Ticket subject: ' || company_data."support_tickets".subject
      || ' Ticket body: ' || company_data."support_tickets".body
    AS prompt_text
  FROM company_data."support_tickets"
  WHERE company_data."support_tickets".created_at > CURRENT_TIMESTAMP - INTERVAL '7' DAY
)
SELECT prompts.ticket_id, prompts.prompt_text
FROM prompts
```

**Tips for building effective prompts:**

* Prefix each field with a label (e.g., `' Ticket subject: '`) so the model can distinguish between data elements
* Filter rows before calling `AI_COMPLETE` to avoid unnecessary inference costs
* Use `COALESCE` to handle nulls: `COALESCE(company_data."support_tickets".subject, 'No subject')`

## Step 3: Call AI\_COMPLETE

Pass the prompt column to `AI_COMPLETE` along with your schema. The response is a JSON string that you parse in the next step.

```sql theme={null}
WITH prompts AS (
  SELECT
    company_data."support_tickets".ticket_id,
    company_data."support_tickets".customer_id,
    company_data."support_tickets".created_at,
    'Analyze the following customer support ticket and extract structured attributes.'
      || ' Ticket subject: ' || company_data."support_tickets".subject
      || ' Ticket body: ' || company_data."support_tickets".body
    AS prompt_text
  FROM company_data."support_tickets"
  WHERE company_data."support_tickets".created_at > CURRENT_TIMESTAMP - INTERVAL '7' DAY
),
enriched AS (
  SELECT
    prompts.ticket_id,
    prompts.customer_id,
    prompts.created_at,
    AI_COMPLETE(
      'openai-gpt-5',
      prompts.prompt_text,
      '{"temperature": 0}',
      '{"type": "json", "schema": {"type": "object", "properties": {"category": {"type": "string", "description": "The primary issue category", "enum": ["billing", "technical", "account_access", "feature_request", "general_inquiry"]}, "priority": {"type": "string", "description": "Urgency level", "enum": ["low", "medium", "high", "critical"]}, "sentiment": {"type": "number", "description": "Sentiment from -1 (very negative) to 1 (very positive)"}, "product_mentioned": {"type": "string", "description": "Product or feature referenced, or none"}}, "required": ["category", "priority", "sentiment", "product_mentioned"], "additionalProperties": false}}',
      TRUE
    ) AS ai_response
  FROM prompts
)
SELECT enriched.ticket_id, enriched.ai_response
FROM enriched
```

**Parameter choices:**

| Parameter          | Value                  | Reason                                                               |
| ------------------ | ---------------------- | -------------------------------------------------------------------- |
| `model`            | `'openai-gpt-5'`       | Choose the model that best fits your accuracy and cost requirements  |
| `model_parameters` | `'{"temperature": 0}'` | Temperature 0 produces deterministic, consistent classifications     |
| `show_details`     | `TRUE`                 | Returns the full `structured_output` array for reliable JSON parsing |

## Step 4: Parse the response into columns

Use `PARSE_JSON` and path notation to extract individual fields from the AI response into typed columns.

```sql theme={null}
SELECT
  enriched.ticket_id,
  enriched.customer_id,
  enriched.created_at,
  CAST(
    PARSE_JSON(enriched.ai_response)['structured_output'][0]['raw_message']['category'] AS STRING
  ) AS category,
  CAST(
    PARSE_JSON(enriched.ai_response)['structured_output'][0]['raw_message']['priority'] AS STRING
  ) AS priority,
  CAST(
    PARSE_JSON(enriched.ai_response)['structured_output'][0]['raw_message']['sentiment'] AS DOUBLE
  ) AS sentiment,
  CAST(
    PARSE_JSON(enriched.ai_response)['structured_output'][0]['raw_message']['product_mentioned'] AS STRING
  ) AS product_mentioned
FROM enriched
```

The path `['structured_output'][0]['raw_message']` navigates to the model's schema-validated output. Each field within `raw_message` corresponds to a property in your `response_format` schema.

## Complete example

Here is the full query wrapped as a materialized view that refreshes weekly:

```sql theme={null}
CREATE MATERIALIZED VIEW "support_tickets_enriched"
DISPLAY_NAME = 'Support Tickets - AI Enriched'
DESCRIPTION = 'Tickets enriched with AI-classified category, priority, sentiment, and product mentions'
TAGS = ('ai_enrichment', 'support', 'classification')
WRITE_MODE = 'overwrite'
REFRESH_SCHEDULE = '@weekly'
AS
WITH prompts AS (
  SELECT
    company_data."support_tickets".ticket_id,
    company_data."support_tickets".customer_id,
    company_data."support_tickets".created_at,
    'Analyze the following customer support ticket and extract structured attributes.'
      || ' Ticket subject: ' || COALESCE(company_data."support_tickets".subject, 'No subject')
      || ' Ticket body: ' || COALESCE(company_data."support_tickets".body, 'No body')
    AS prompt_text
  FROM company_data."support_tickets"
  WHERE company_data."support_tickets".created_at > CURRENT_TIMESTAMP - INTERVAL '30' DAY
),
ai_results AS (
  SELECT
    prompts.ticket_id,
    prompts.customer_id,
    prompts.created_at,
    AI_COMPLETE(
      'openai-gpt-5',
      prompts.prompt_text,
      '{"temperature": 0}',
      '{"type": "json", "schema": {"type": "object", "properties": {"category": {"type": "string", "description": "The primary issue category", "enum": ["billing", "technical", "account_access", "feature_request", "general_inquiry"]}, "priority": {"type": "string", "description": "Urgency level", "enum": ["low", "medium", "high", "critical"]}, "sentiment": {"type": "number", "description": "Sentiment from -1 (very negative) to 1 (very positive)"}, "product_mentioned": {"type": "string", "description": "Product or feature referenced, or none"}}, "required": ["category", "priority", "sentiment", "product_mentioned"], "additionalProperties": false}}',
      TRUE
    ) AS ai_response
  FROM prompts
)
SELECT
  ai_results.ticket_id,
  ai_results.customer_id,
  ai_results.created_at,
  CAST(
    PARSE_JSON(ai_results.ai_response)['structured_output'][0]['raw_message']['category'] AS STRING
  ) AS category,
  CAST(
    PARSE_JSON(ai_results.ai_response)['structured_output'][0]['raw_message']['priority'] AS STRING
  ) AS priority,
  CAST(
    PARSE_JSON(ai_results.ai_response)['structured_output'][0]['raw_message']['sentiment'] AS DOUBLE
  ) AS sentiment,
  CAST(
    PARSE_JSON(ai_results.ai_response)['structured_output'][0]['raw_message']['product_mentioned'] AS STRING
  ) AS product_mentioned
FROM ai_results
```

## Adapting this pattern

The CTE → AI\_COMPLETE → PARSE\_JSON pattern works for any enrichment task. Here are variations for common scenarios.

### Product classification

Classify products into a taxonomy from their name and description:

```sql theme={null}
WITH product_prompts AS (
  SELECT
    company_data."products".product_id,
    'Classify this product into a category.'
      || ' Brand: ' || COALESCE(company_data."products".brand_name, 'Unknown')
      || ' Product name: ' || company_data."products".product_name
      || ' Description: ' || COALESCE(company_data."products".description, 'No description')
    AS prompt_text
  FROM company_data."products"
),
classified AS (
  SELECT
    product_prompts.product_id,
    AI_COMPLETE(
      'openai-gpt-5',
      product_prompts.prompt_text,
      '{"temperature": 0}',
      '{"type": "json", "schema": {"type": "object", "properties": {"primary_category": {"type": "string", "description": "Top-level product category"}, "sub_category": {"type": "string", "description": "More specific product category"}, "confidence": {"type": "number", "description": "Classification confidence from 0 to 1"}}, "required": ["primary_category", "sub_category", "confidence"], "additionalProperties": false}}',
      TRUE
    ) AS ai_response
  FROM product_prompts
)
SELECT
  classified.product_id,
  CAST(
    PARSE_JSON(classified.ai_response)['structured_output'][0]['raw_message']['primary_category'] AS STRING
  ) AS primary_category,
  CAST(
    PARSE_JSON(classified.ai_response)['structured_output'][0]['raw_message']['sub_category'] AS STRING
  ) AS sub_category,
  CAST(
    PARSE_JSON(classified.ai_response)['structured_output'][0]['raw_message']['confidence'] AS DOUBLE
  ) AS confidence
FROM classified
```

### Entity extraction

Extract structured entities from free-text fields:

```sql theme={null}
WITH text_prompts AS (
  SELECT
    company_data."articles".article_id,
    'Extract key entities from this text.'
      || ' Text: ' || company_data."articles".body
    AS prompt_text
  FROM company_data."articles"
),
extracted AS (
  SELECT
    text_prompts.article_id,
    AI_COMPLETE(
      'openai-gpt-5',
      text_prompts.prompt_text,
      '{"temperature": 0}',
      '{"type": "json", "schema": {"type": "object", "properties": {"people": {"type": "array", "items": {"type": "string"}, "description": "Names of people mentioned"}, "organizations": {"type": "array", "items": {"type": "string"}, "description": "Organization names mentioned"}, "topics": {"type": "array", "items": {"type": "string"}, "description": "Key topics discussed"}}, "required": ["people", "organizations", "topics"], "additionalProperties": false}}',
      TRUE
    ) AS ai_response
  FROM text_prompts
)
SELECT
  extracted.article_id,
  PARSE_JSON(extracted.ai_response)['structured_output'][0]['raw_message']['people'] AS people,
  PARSE_JSON(extracted.ai_response)['structured_output'][0]['raw_message']['organizations'] AS organizations,
  PARSE_JSON(extracted.ai_response)['structured_output'][0]['raw_message']['topics'] AS topics
FROM extracted
```

## Best practices

**Prompt design:**

* Label each data element clearly (e.g., `' Brand: '`, `' Description: '`)
* Include a brief instruction at the start of the prompt describing the task
* Use `COALESCE` to handle nullable columns and avoid sending nulls to the model

**Schema design:**

* Use `enum` to constrain categorical outputs to known values
* Set `additionalProperties: false` to prevent unexpected fields
* Add `description` to each property to guide the model
* Use `number` types for scores you want to filter or aggregate

**Performance and cost:**

* Filter rows with `WHERE` before the `AI_COMPLETE` call to reduce inference volume
* Use `FETCH NEXT N ROWS ONLY` to limit batch sizes during development and testing
* Set `temperature` to `0` for deterministic, reproducible results
* Keep schemas focused—fewer fields means fewer tokens per response

***

## Related content

<CardGroup>
  <Card title="AI_COMPLETE Reference" icon="brain" href="/nql/functions/ai-functions">
    Function syntax and parameter details
  </Card>

  <Card title="Model Inference" icon="microchip" href="/concepts/model-inference/overview">
    How AI inference works within your data plane
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/concepts/model-inference/structured-output">
    Designing JSON schemas for AI responses
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Automating queries with scheduled materialized views
  </Card>
</CardGroup>


# Common Query Patterns
Source: https://docs.narrative.io/cookbooks/nql/common-queries

Copy-paste NQL patterns for everyday data tasks

This cookbook contains ready-to-use NQL query patterns for common data tasks. Copy these patterns and adapt them to your datasets.

***

## Deduplication

### Keep the most recent record per user

```sql theme={null}
SELECT
  user_id,
  email,
  last_seen,
  attributes
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

**How it works:** `ROW_NUMBER()` assigns a sequence number within each user partition, ordered by `last_seen` descending. `QUALIFY` filters to keep only the first row (most recent) for each user.

### Keep the first occurrence

```sql theme={null}
SELECT
  user_id,
  email,
  created_at
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY email
  ORDER BY created_at ASC
) = 1
```

### Deduplicate by multiple columns

```sql theme={null}
SELECT
  user_id,
  email,
  phone,
  last_updated
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY email, phone
  ORDER BY last_updated DESC
) = 1
```

***

## Date range filtering

### Last N days

```sql theme={null}
SELECT user_id, email, event_timestamp
FROM company_data."123"
WHERE event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

### Specific date range

```sql theme={null}
SELECT user_id, email, event_date
FROM company_data."123"
WHERE event_date BETWEEN '2024-01-01' AND '2024-03-31'
```

### Current month

```sql theme={null}
SELECT user_id, email, event_date
FROM company_data."123"
WHERE event_date >= DATE_TRUNC('month', CURRENT_DATE)
```

### Previous month

```sql theme={null}
SELECT user_id, email, event_date
FROM company_data."123"
WHERE
  event_date >= DATE_TRUNC('month', CURRENT_DATE - INTERVAL '1' MONTH)
  AND event_date < DATE_TRUNC('month', CURRENT_DATE)
```

### Rolling 12 months

```sql theme={null}
SELECT user_id, email, event_timestamp
FROM company_data."123"
WHERE event_timestamp >= CURRENT_DATE - INTERVAL '12' MONTH
```

***

## Price-constrained queries

### Maximum price per 1000 rows

```sql theme={null}
SELECT user_id, email, category
FROM company_data."123"
WHERE
  category = 'premium'
  AND _price_cpm_usd <= 1.00
```

### Tiered pricing selection

```sql theme={null}
SELECT
  user_id,
  email,
  category,
  _price_cpm_usd,
  CASE
    WHEN _price_cpm_usd <= 0.50 THEN 'budget'
    WHEN _price_cpm_usd <= 1.00 THEN 'standard'
    ELSE 'premium'
  END AS price_tier
FROM company_data."123"
WHERE _price_cpm_usd <= 2.00
```

***

## Aggregations

### Count by category

```sql theme={null}
SELECT
  category,
  COUNT(1) AS count,
  APPROX_COUNT_DISTINCT(user_id) AS unique_users
FROM company_data."123"
GROUP BY category
ORDER BY count DESC
```

### Daily aggregation

```sql theme={null}
SELECT
  DATE_TRUNC('day', event_timestamp) AS event_date,
  COUNT(1) AS events,
  APPROX_COUNT_DISTINCT(user_id) AS users
FROM company_data."123"
WHERE event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
GROUP BY DATE_TRUNC('day', event_timestamp)
ORDER BY event_date
```

### Monthly aggregation

```sql theme={null}
SELECT
  DATE_TRUNC('month', event_date) AS month,
  SUM(amount) AS total_amount,
  AVG(amount) AS avg_amount,
  COUNT(1) AS transaction_count
FROM company_data."123"
GROUP BY DATE_TRUNC('month', event_date)
ORDER BY month
```

### Top N by category

```sql theme={null}
SELECT
  category,
  product_name,
  revenue
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY category
  ORDER BY revenue DESC
) <= 10
```

***

## Identity lookups

### Rosetta Stone unique IDs

```sql theme={null}
SELECT
  unique_id,
  event_timestamp
FROM narrative.rosetta_stone
WHERE
  event_timestamp >= CURRENT_DATE - INTERVAL '90' DAY
```

### Filter by identifier type

```sql theme={null}
SELECT
  unique_id.b."value" AS identifier_value
FROM narrative.rosetta_stone
WHERE
  unique_id.b."type" = 'email'
  AND event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

### Exclude specific identifier types

```sql theme={null}
SELECT
  unique_id.b."type",
  unique_id.b."value"
FROM narrative.rosetta_stone
WHERE
  unique_id.b."type" NOT IN ('cookie', 'tdid')
```

### Access Rosetta Stone from dataset

```sql theme={null}
SELECT
  company_data."123"._rosetta_stone.unique_id,
  company_data."123".user_attribute
FROM company_data."123"
```

### Company-scoped Rosetta Stone

Query normalized data from a specific company only:

```sql theme={null}
SELECT
  company_data._rosetta_stone.unique_id,
  company_data._rosetta_stone.event_timestamp
FROM company_data._rosetta_stone
WHERE company_data._rosetta_stone.event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

**How it works:** Using `company_data._rosetta_stone` limits the query to datasets owned by your company. Replace `company_data` with a partner's company slug to query their shared data.

### Combine normalized and original columns

Join Rosetta Stone attributes with non-normalized dataset columns:

```sql theme={null}
SELECT
  company_data."123"._rosetta_stone.unique_id,
  company_data."123"._rosetta_stone.hl7_gender,
  company_data."123".custom_segment,
  company_data."123".loyalty_tier
FROM company_data."123"._rosetta_stone
WHERE company_data."123"._rosetta_stone.event_timestamp >= CURRENT_DATE - INTERVAL '90' DAY
```

**How it works:** Dataset-scoped access (`company_data."123"._rosetta_stone`) lets you select both normalized Rosetta Stone attributes and original dataset columns in the same query.

***

## Data quality checks

### Count nulls per column

```sql theme={null}
SELECT
  COUNT(1) AS total_rows,
  COUNT(email) AS has_email,
  COUNT(1) - COUNT(email) AS missing_email,
  COUNT(phone) AS has_phone,
  COUNT(1) - COUNT(phone) AS missing_phone
FROM company_data."123"
```

### Null percentage

```sql theme={null}
SELECT
  ROUND(100.0 * COUNT(email) / COUNT(1), 2) AS email_fill_rate,
  ROUND(100.0 * COUNT(phone) / COUNT(1), 2) AS phone_fill_rate,
  ROUND(100.0 * COUNT(address) / COUNT(1), 2) AS address_fill_rate
FROM company_data."123"
```

### Find duplicates

```sql theme={null}
SELECT
  email,
  COUNT(1) AS occurrence_count
FROM company_data."123"
GROUP BY email
HAVING COUNT(1) > 1
ORDER BY occurrence_count DESC
LIMIT 100 ROWS
```

### Value distribution

```sql theme={null}
SELECT
  status,
  COUNT(1) AS count,
  ROUND(100.0 * COUNT(1) / SUM(COUNT(1)) OVER (), 2) AS percentage
FROM company_data."123"
GROUP BY status
ORDER BY count DESC
```

***

## String operations

### Email domain extraction

```sql theme={null}
SELECT
  email,
  REGEXP_EXTRACT(email, '@(.+)$') AS domain
FROM company_data."123"
WHERE email IS NOT NULL
```

### Email normalization

```sql theme={null}
SELECT
  email,
  NORMALIZE_EMAIL(email) AS normalized_email
FROM company_data."123"
```

### Phone normalization

```sql theme={null}
SELECT
  phone,
  NORMALIZE_PHONE('E164', phone, 'US') AS e164_phone
FROM company_data."123"
WHERE phone IS NOT NULL
```

### Name standardization

```sql theme={null}
SELECT
  UPPER(TRIM(first_name)) AS first_name,
  UPPER(TRIM(last_name)) AS last_name,
  CONCAT(UPPER(TRIM(first_name)), ' ', UPPER(TRIM(last_name))) AS full_name
FROM company_data."123"
```

***

## Array operations

### Expand array to rows

```sql theme={null}
SELECT
  t.user_id,
  tag
FROM company_data."123" t, UNNEST(t.tags) AS tag
```

### Count array elements

```sql theme={null}
SELECT
  user_id,
  SIZE(tags) AS tag_count,
  SIZE(identifiers) AS identifier_count
FROM company_data."123"
```

### Access first array element

```sql theme={null}
SELECT
  user_id,
  identifiers[0].type AS primary_id_type,
  identifiers[0].value AS primary_id_value
FROM company_data."123"
```

### Filter by array contents

```sql theme={null}
SELECT user_id, email, tags
FROM company_data."123"
WHERE SIZE(tags) > 0
  AND tags[0] IN ('premium', 'vip')
```

***

## Sampling

### Deterministic sample

```sql theme={null}
SELECT user_id, email, created_at
FROM company_data."123"
WHERE UNIVERSE_SAMPLE(user_id, 0.1)  -- 10% sample
```

### Random sample with LIMIT

```sql theme={null}
SELECT user_id, email, created_at
FROM company_data."123"
ORDER BY DETERMINISTIC_RAND(user_id)
LIMIT 1000 ROWS
```

***

## Materialized view patterns

### Basic view with budget

```sql theme={null}
CREATE MATERIALIZED VIEW "active_users"
AS (
  SELECT
    user_id,
    email,
    last_seen
  FROM company_data."123"
  WHERE
    last_seen >= CURRENT_DATE - INTERVAL '30' DAY
    AND _price_cpm_usd <= 1.00
)
BUDGET 50 USD
```

### Daily refresh with deduplication

```sql theme={null}
CREATE MATERIALIZED VIEW "daily_users"
REFRESH_SCHEDULE = '@daily'
PARTITIONED_BY event_date DAY
AS (
  SELECT
    user_id,
    email,
    DATE_TRUNC('day', last_seen) AS event_date
  FROM company_data."123"
  WHERE last_seen >= CURRENT_DATE - INTERVAL '7' DAY
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY user_id
    ORDER BY last_seen DESC
  ) = 1
)
BUDGET 25 USD PER CALENDAR_DAY
```

***

## Related content

<CardGroup>
  <Card title="Performance Patterns" icon="bolt" href="/cookbooks/nql/performance-patterns">
    Optimization recipes for faster queries
  </Card>

  <Card title="Complex Joins" icon="link" href="/cookbooks/nql/complex-joins">
    Advanced join patterns
  </Card>

  <Card title="Functions Reference" icon="function" href="/nql/functions/index">
    All available functions
  </Card>

  <Card title="Filtering Guide" icon="filter" href="/guides/nql/filtering-transforming">
    Complete filtering tutorial
  </Card>
</CardGroup>


# Complex Join Patterns
Source: https://docs.narrative.io/cookbooks/nql/complex-joins

Advanced join recipes for multi-dataset scenarios

This cookbook contains advanced join patterns for combining data across multiple datasets. These recipes address common data collaboration scenarios.

***

## Multi-dataset joins

### Three-way join

Combine customers, orders, and products:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  o.order_id,
  o.order_date,
  p.product_name,
  p.category,
  o.quantity * p.unit_price AS line_total
FROM company_data."100" c
INNER JOIN company_data."200" o
  ON c.customer_id = o.customer_id
INNER JOIN company_data."300" p
  ON o.product_id = p.product_id
WHERE
  o.order_date >= CURRENT_DATE - INTERVAL '30' DAY
```

### Chain of related entities

Follow relationships through multiple tables:

```sql theme={null}
WITH customer_orders AS (
  SELECT
    c.customer_id,
    c.name AS customer_name,
    c.segment,
    o.order_id,
    o.order_date
  FROM company_data."100" c
  INNER JOIN company_data."200" o
    ON c.customer_id = o.customer_id
  WHERE o.order_date >= CURRENT_DATE - INTERVAL '90' DAY
),
order_products AS (
  SELECT
    co.*,
    p.product_id,
    p.product_name,
    p.category
  FROM customer_orders co
  INNER JOIN company_data."300" op
    ON co.order_id = op.order_id
  INNER JOIN company_data."400" p
    ON op.product_id = p.product_id
)
SELECT
  customer_id,
  customer_name,
  segment,
  category,
  APPROX_COUNT_DISTINCT(product_id) AS unique_products,
  COUNT(1) AS total_purchases
FROM order_products
GROUP BY customer_id, customer_name, segment, category
```

***

## Identity-based joins with Rosetta Stone

### Join datasets via resolved identities

```sql theme={null}
SELECT
  d1.user_attribute AS attribute_from_dataset1,
  d2.user_attribute AS attribute_from_dataset2
FROM company_data."123" d1
INNER JOIN company_data."456" d2
  ON d1._rosetta_stone.unique_id = d2._rosetta_stone.unique_id
WHERE
  d1._price_cpm_usd <= 1.00
  AND d2._price_cpm_usd <= 1.00
```

### Enrich with Rosetta Stone data

```sql theme={null}
SELECT
  d.user_id,
  d.email,
  rs.unique_id,
  rs.event_timestamp AS identity_timestamp
FROM company_data."123" d
LEFT JOIN narrative.rosetta_stone rs
  ON d._rosetta_stone.unique_id = rs.unique_id
WHERE
  rs.event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

### Match by identifier type

```sql theme={null}
SELECT
  d1.email,
  d2.phone,
  d1.last_seen
FROM company_data."123" d1
INNER JOIN company_data."456" d2
  ON d1._rosetta_stone.unique_id = d2._rosetta_stone.unique_id
WHERE
  d1._rosetta_stone.unique_id.b."type" = 'email'
```

***

## Temporal joins

### Event-session matching

Match events to sessions based on timestamp:

```sql theme={null}
SELECT
  e.event_id,
  e.event_type,
  e.event_timestamp,
  s.session_id,
  s.session_start,
  s.session_end
FROM company_data."100" e
INNER JOIN company_data."200" s
  ON e.user_id = s.user_id
  AND e.event_timestamp BETWEEN s.session_start AND s.session_end
```

### Point-in-time lookup

Get the state of a record at a specific point in time:

```sql theme={null}
SELECT
  t.transaction_id,
  t.transaction_date,
  t.amount,
  p.price AS price_at_transaction
FROM company_data."100" t
INNER JOIN (
  SELECT
    product_id,
    price,
    effective_date,
    COALESCE(
      LEAD(effective_date) OVER (PARTITION BY product_id ORDER BY effective_date),
      DATE '9999-12-31'
    ) AS end_date
  FROM company_data."200"
) p
  ON t.product_id = p.product_id
  AND t.transaction_date >= p.effective_date
  AND t.transaction_date < p.end_date
```

### Time-windowed join

Join events that occurred within a time window:

```sql theme={null}
SELECT
  a.event_id AS first_event,
  b.event_id AS follow_up_event,
  a.user_id,
  a.event_timestamp AS first_timestamp,
  b.event_timestamp AS follow_up_timestamp
FROM company_data."100" a
INNER JOIN company_data."100" b
  ON a.user_id = b.user_id
  AND b.event_timestamp > a.event_timestamp
  AND b.event_timestamp <= a.event_timestamp + INTERVAL '24' HOUR
WHERE
  a.event_type = 'page_view'
  AND b.event_type = 'purchase'
```

***

## Self-joins

### Find related records within same dataset

```sql theme={null}
SELECT
  e1.employee_id,
  e1.name AS employee_name,
  e1.department,
  e2.name AS manager_name
FROM company_data."123" e1
LEFT JOIN company_data."123" e2
  ON e1.manager_id = e2.employee_id
```

### Compare records over time

```sql theme={null}
SELECT
  curr.user_id,
  curr.metric_date,
  curr.value AS current_value,
  prev.value AS previous_value,
  curr.value - prev.value AS change
FROM company_data."123" curr
LEFT JOIN company_data."123" prev
  ON curr.user_id = prev.user_id
  AND prev.metric_date = curr.metric_date - INTERVAL '1' DAY
WHERE
  curr.metric_date = CURRENT_DATE
```

### Find duplicates

```sql theme={null}
SELECT
  a.record_id AS record_1,
  b.record_id AS record_2,
  a.email
FROM company_data."123" a
INNER JOIN company_data."123" b
  ON a.email = b.email
  AND a.record_id < b.record_id
```

***

## Anti-joins and exclusions

### Find records without matches

Find customers who haven't made purchases:

```sql theme={null}
SELECT
  c.customer_id,
  c.email,
  c.signup_date
FROM company_data."100" c
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."200" o
  WHERE c.customer_id = o.customer_id
)
```

### Find new records not in reference

Find users not in suppression list:

```sql theme={null}
SELECT
  u.user_id,
  u.email
FROM company_data."100" u
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."200" suppression
  WHERE u.email = suppression.email
)
```

### Exclude recent interactions

Find users without recent activity:

```sql theme={null}
SELECT
  u.user_id,
  u.email,
  u.last_login
FROM company_data."100" u
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."200" a
  WHERE u.user_id = a.user_id
    AND a.activity_date >= CURRENT_DATE - INTERVAL '90' DAY
)
```

***

## Enrichment patterns

### Latest record enrichment

Enrich with the most recent related record:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  latest_order.order_id,
  latest_order.order_date,
  latest_order.amount
FROM company_data."100" c
LEFT JOIN (
  SELECT
    customer_id,
    order_id,
    order_date,
    amount
  FROM company_data."200"
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY customer_id
    ORDER BY order_date DESC
  ) = 1
) latest_order ON c.customer_id = latest_order.customer_id
```

### Aggregated enrichment

Enrich with summary statistics:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  c.segment,
  COALESCE(stats.order_count, 0) AS order_count,
  COALESCE(stats.total_spent, 0) AS total_spent,
  COALESCE(stats.avg_order, 0) AS avg_order_value
FROM company_data."100" c
LEFT JOIN (
  SELECT
    customer_id,
    COUNT(1) AS order_count,
    SUM(amount) AS total_spent,
    AVG(amount) AS avg_order
  FROM company_data."200"
  GROUP BY customer_id
) stats ON c.customer_id = stats.customer_id
```

### Multiple attribute enrichment

Enrich from multiple sources:

```sql theme={null}
WITH customer_base AS (
  SELECT customer_id, name, email
  FROM company_data."100"
),
order_summary AS (
  SELECT
    customer_id,
    COUNT(1) AS orders,
    SUM(amount) AS total
  FROM company_data."200"
  GROUP BY customer_id
),
engagement_summary AS (
  SELECT
    customer_id,
    COUNT(1) AS interactions,
    MAX(interaction_date) AS last_interaction
  FROM company_data."300"
  GROUP BY customer_id
)
SELECT
  c.customer_id,
  c.name,
  c.email,
  COALESCE(o.orders, 0) AS order_count,
  COALESCE(o.total, 0) AS total_spent,
  COALESCE(e.interactions, 0) AS engagement_count,
  e.last_interaction
FROM customer_base c
LEFT JOIN order_summary o ON c.customer_id = o.customer_id
LEFT JOIN engagement_summary e ON c.customer_id = e.customer_id
```

***

## Cross-dataset deduplication

### Deduplicate across datasets

Merge records from multiple datasets, keeping the most complete:

```sql theme={null}
WITH all_records AS (
  SELECT
    email,
    name,
    phone,
    'dataset_1' AS source,
    updated_at
  FROM company_data."100"
  UNION ALL
  SELECT
    email,
    name,
    phone,
    'dataset_2' AS source,
    updated_at
  FROM company_data."200"
),
ranked AS (
  SELECT
    email,
    name,
    phone,
    source,
    updated_at,
    ROW_NUMBER() OVER (
      PARTITION BY email
      ORDER BY
        CASE WHEN name IS NOT NULL THEN 1 ELSE 2 END,
        CASE WHEN phone IS NOT NULL THEN 1 ELSE 2 END,
        updated_at DESC
    ) AS rn
  FROM all_records
)
SELECT email, name, phone, source
FROM ranked
WHERE rn = 1
```

***

## Materialized view with joins

### Create enriched view

```sql theme={null}
CREATE MATERIALIZED VIEW "enriched_customers"
REFRESH_SCHEDULE = '@daily'
AS (
  SELECT
    c.customer_id,
    c.email,
    c.segment,
    COALESCE(metrics.order_count, 0) AS lifetime_orders,
    COALESCE(metrics.total_value, 0) AS lifetime_value,
    metrics.last_order_date
  FROM company_data."100" c
  LEFT JOIN (
    SELECT
      customer_id,
      COUNT(1) AS order_count,
      SUM(amount) AS total_value,
      MAX(order_date) AS last_order_date
    FROM company_data."200"
    GROUP BY customer_id
  ) metrics ON c.customer_id = metrics.customer_id
  WHERE
    c._price_cpm_usd <= 1.00
)
BUDGET 100 USD PER CALENDAR_DAY
```

***

## Related content

<CardGroup>
  <Card title="Joining Datasets Guide" icon="link" href="/guides/nql/joining-datasets">
    Complete guide to join operations
  </Card>

  <Card title="Join Performance" icon="gauge-high" href="/concepts/nql/join-performance">
    Understanding join performance
  </Card>

  <Card title="Common Queries" icon="copy" href="/cookbooks/nql/common-queries">
    Basic query patterns
  </Card>

  <Card title="Performance Patterns" icon="bolt" href="/cookbooks/nql/performance-patterns">
    Query optimization recipes
  </Card>
</CardGroup>


# Geospatial POI Joins
Source: https://docs.narrative.io/cookbooks/nql/geospatial-poi-joins

Join datasets on point of interest geometries using ST_Intersects

Geospatial joins let you match records based on geographic overlap rather than shared keys. This is useful for foot traffic analysis, proximity targeting, and visitation attribution---any scenario where you need to find which observations fall within a set of points of interest.

***

## POI polygon-to-polygon join

Use `ST_Intersects` to join two datasets that both have POI geometry columns. This finds all records from one dataset whose geographic polygon overlaps with polygons in another dataset.

```sql theme={null}
SELECT
  ls."_rosetta_stone"."geographic_location"."latitude",
  ls."_rosetta_stone"."geographic_location"."longitude",
  ls."_rosetta_stone"."mobile_id_unique_identifier"."value" AS maid_value,
  ls."_rosetta_stone"."event_timestamp"
FROM
  company_data."observations" ls
  INNER JOIN company_data."target_locations" t
    ON ST_Intersects(
      t."_rosetta_stone"."geographic_poi"."poi",
      ls."_rosetta_stone"."geographic_poi"."poi"
    )
WHERE
  ls."_rosetta_stone"."geographic_location"."latitude" IS NOT NULL
  AND ls."_rosetta_stone"."geographic_location"."longitude" IS NOT NULL
  AND ls."_rosetta_stone"."mobile_id_unique_identifier"."value" IS NOT NULL
  AND ls."_rosetta_stone"."event_timestamp" BETWEEN '2026-02-01' AND '2026-04-01'
  AND ls."_rosetta_stone"."nio_last_modified_at" >= '2026-02-01'
```

Key points:

* Both datasets must have a `geographic_poi.poi` attribute mapped through Rosetta Stone
* `ST_Intersects` returns true when the two geometries share any area in common
* Select non-geometry columns only---latitude, longitude, timestamps, and identifiers

***

## Point-to-polygon join

When one dataset has raw coordinates instead of POI polygons, use `ST_SafePoint` and `ST_Circle` to create a geometry on the fly, then join with `ST_Intersects`.

```sql theme={null}
SELECT
  obs."_rosetta_stone"."mobile_id_unique_identifier"."value" AS maid_value,
  obs."_rosetta_stone"."event_timestamp",
  pois.name AS poi_name
FROM company_data."observations" obs
INNER JOIN company_data."poi_locations" pois
  ON ST_Intersects(
    pois."_rosetta_stone"."geographic_poi"."poi",
    ST_Circle(
      ST_SafePoint(
        obs."_rosetta_stone"."geographic_location"."longitude",
        obs."_rosetta_stone"."geographic_location"."latitude"
      ),
      50
    )
  )
WHERE
  obs."_rosetta_stone"."event_timestamp" >= CURRENT_DATE - INTERVAL '30' DAY
  AND obs."_rosetta_stone"."nio_last_modified_at" >= CURRENT_DATE - INTERVAL '30' DAY
```

`ST_Circle` takes a point and a radius in meters. In this example, each observation is expanded into a 50-meter circle and checked for overlap with the POI polygons.

***

## Combining geospatial and temporal filters

Most geospatial queries also need time-based filtering. Always include both an `event_timestamp` range and a `nio_last_modified_at` filter to scope the data appropriately.

```sql theme={null}
SELECT
  ls."_rosetta_stone"."geographic_location"."latitude",
  ls."_rosetta_stone"."geographic_location"."longitude",
  ls."_rosetta_stone"."mobile_id_unique_identifier"."value" AS maid_value,
  ls."_rosetta_stone"."event_timestamp"
FROM
  company_data."observations" ls
  INNER JOIN company_data."target_locations" t
    ON ST_Intersects(
      t."_rosetta_stone"."geographic_poi"."poi",
      ls."_rosetta_stone"."geographic_poi"."poi"
    )
WHERE
  ls."_rosetta_stone"."event_timestamp" BETWEEN '2026-02-01' AND '2026-04-01'
  AND ls."_rosetta_stone"."nio_last_modified_at" >= '2026-02-01'
```

* `event_timestamp` filters the actual observation time
* `nio_last_modified_at` ensures query efficiency by limiting the data scan window

***

## Common pitfalls

### GEOMETRY type cannot be in SELECT

Geometry columns cannot be returned in query results. Use them only in JOIN conditions or WHERE clauses.

**This fails:**

```sql theme={null}
SELECT
  ls."_rosetta_stone"."geographic_poi"."poi" AS poi_geometry
FROM company_data."observations" ls
```

**Instead, select non-geometry columns** like latitude, longitude, or identifiers. See [Unsupported Type Errors](/guides/nql/troubleshooting/unsupported-type-error) for details.

### Argument order in ST\_Intersects

Place the reference/POI dataset geometry as the **first** argument and the observation geometry as the **second** argument. Reversing the order can cause internal errors.

```sql theme={null}
-- Correct: reference geometry first
ON ST_Intersects(
  t."_rosetta_stone"."geographic_poi"."poi",
  ls."_rosetta_stone"."geographic_poi"."poi"
)

-- Incorrect: observation geometry first (may cause errors)
ON ST_Intersects(
  ls."_rosetta_stone"."geographic_poi"."poi",
  t."_rosetta_stone"."geographic_poi"."poi"
)
```

### Always use \_rosetta\_stone

Access columns through the `_rosetta_stone` path. Queries that omit `_rosetta_stone` may appear to work in some cases, but this is not a supported pattern and will produce unreliable results.

```sql theme={null}
-- Correct
ls."_rosetta_stone"."geographic_poi"."poi"

-- Incorrect: missing _rosetta_stone
ls."geographic_poi"."poi"
```

***

## Related content

<CardGroup>
  <Card title="Location Data" icon="location-dot" href="/concepts/data-formats/location-data">
    Understanding location polygons and geospatial formats
  </Card>

  <Card title="Geospatial Functions" icon="map-location-dot" href="/nql/functions/geospatial-functions">
    ST\_SafePoint, ST\_Circle, ST\_Intersects, and more
  </Card>

  <Card title="Complex Join Patterns" icon="diagram-project" href="/cookbooks/nql/complex-joins">
    Advanced join recipes for multi-dataset scenarios
  </Card>

  <Card title="Unsupported Type Errors" icon="ban" href="/guides/nql/troubleshooting/unsupported-type-error">
    Resolving GEOMETRY type errors in SELECT statements
  </Card>
</CardGroup>


# Performance Patterns
Source: https://docs.narrative.io/cookbooks/nql/performance-patterns

Optimization recipes for faster, more efficient NQL queries

This cookbook contains patterns for optimizing NQL query performance. Apply these techniques to reduce execution time and costs.

***

## Filter early and aggressively

### Apply filters before joins

Filter data in subqueries or CTEs before joining to reduce the amount of data processed:

```sql theme={null}
-- Better: Filter before joining
WITH active_users AS (
  SELECT user_id, email
  FROM company_data."100"
  WHERE
    status = 'active'
    AND created_at >= CURRENT_DATE - INTERVAL '90' DAY
),
recent_orders AS (
  SELECT user_id, order_id, amount
  FROM company_data."200"
  WHERE order_date >= CURRENT_DATE - INTERVAL '30' DAY
)
SELECT
  u.user_id,
  u.email,
  o.order_id,
  o.amount
FROM active_users u
INNER JOIN recent_orders o ON u.user_id = o.user_id
```

```sql theme={null}
-- Avoid: Filtering after joining
SELECT
  u.user_id,
  u.email,
  o.order_id,
  o.amount
FROM company_data."100" u
INNER JOIN company_data."200" o ON u.user_id = o.user_id
WHERE
  u.status = 'active'
  AND u.created_at >= CURRENT_DATE - INTERVAL '90' DAY
  AND o.order_date >= CURRENT_DATE - INTERVAL '30' DAY
```

### Use price filtering early

Add `_price_cpm_usd` filters to limit data costs:

```sql theme={null}
SELECT user_id, email, category, event_date
FROM company_data."123"
WHERE
  _price_cpm_usd <= 1.00  -- Apply price filter early
  AND category = 'target'
  AND event_date >= CURRENT_DATE - INTERVAL '30' DAY
```

***

## Use QUALIFY instead of subqueries

### Deduplication with QUALIFY

QUALIFY is more efficient than subquery-based deduplication:

```sql theme={null}
-- Better: QUALIFY
SELECT user_id, email, last_seen
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

```sql theme={null}
-- Avoid: Subquery approach
SELECT user_id, email, last_seen
FROM (
  SELECT
    user_id,
    email,
    last_seen,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY last_seen DESC) AS rn
  FROM company_data."123"
) sub
WHERE rn = 1
```

### Combined filtering with QUALIFY

Apply multiple window conditions efficiently:

```sql theme={null}
SELECT
  user_id,
  email,
  category
FROM company_data."123"
QUALIFY
  ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC) = 1
  AND SUM(1) OVER (PARTITION BY category) >= 100  -- Only categories with 100+ rows
```

***

## Optimize joins

### Use fully qualified column names

Explicit column references help the query optimizer:

```sql theme={null}
SELECT
  company_data."100".user_id,
  company_data."100".email,
  company_data."200".order_id
FROM company_data."100"
INNER JOIN company_data."200"
  ON company_data."100".user_id = company_data."200".customer_id
```

### Avoid OR in join conditions

OR conditions prevent efficient join algorithms:

```sql theme={null}
-- Avoid: OR in join condition
SELECT t1.id, t1.value, t2.related_value
FROM company_data."100" t1
INNER JOIN company_data."200" t2
  ON t1.id = t2.id OR t1.alt_id = t2.alt_id
```

```sql theme={null}
-- Better: Use COALESCE
SELECT t1.id, t1.value, t2.related_value
FROM company_data."100" t1
INNER JOIN company_data."200" t2
  ON COALESCE(t1.id, t1.alt_id) = COALESCE(t2.id, t2.alt_id)
```

```sql theme={null}
-- Or: Union separate queries
SELECT t1.id, t1.value, t2.related_value
FROM company_data."100" t1
INNER JOIN company_data."200" t2 ON t1.id = t2.id
UNION
SELECT t1.id, t1.value, t2.related_value
FROM company_data."100" t1
INNER JOIN company_data."200" t2 ON t1.alt_id = t2.alt_id
WHERE t1.id IS NULL OR t2.id IS NULL
```

### Aggregate before joining

Pre-aggregate large tables to reduce join complexity:

```sql theme={null}
-- Better: Aggregate first
SELECT
  c.customer_id,
  c.name,
  order_stats.total_orders,
  order_stats.total_amount
FROM company_data."100" c
INNER JOIN (
  SELECT
    customer_id,
    COUNT(1) AS total_orders,
    SUM(amount) AS total_amount
  FROM company_data."200"
  GROUP BY customer_id
) order_stats ON c.customer_id = order_stats.customer_id
```

***

## Partitioning strategies

### Partition by date for time-series data

```sql theme={null}
CREATE MATERIALIZED VIEW "daily_events"
PARTITIONED_BY event_date DAY
AS (
  SELECT
    user_id,
    event_type,
    DATE_TRUNC('day', event_timestamp) AS event_date
  FROM company_data."123"
  WHERE event_timestamp >= CURRENT_DATE - INTERVAL '90' DAY
)
BUDGET 100 USD
```

### Monthly partitions for longer retention

```sql theme={null}
CREATE MATERIALIZED VIEW "monthly_summary"
PARTITIONED_BY report_month MONTH
AS (
  SELECT
    DATE_TRUNC('month', event_date) AS report_month,
    category,
    COUNT(1) AS event_count,
    SUM(amount) AS total_amount
  FROM company_data."123"
  GROUP BY DATE_TRUNC('month', event_date), category
)
BUDGET 50 USD
```

***

## Budget management

### Set appropriate budget limits

Match budget to expected data volume:

```sql theme={null}
-- Daily budget for daily refreshes
CREATE MATERIALIZED VIEW "daily_data"
REFRESH_SCHEDULE = '@daily'
AS (SELECT user_id, email, event_date FROM company_data."123" WHERE event_date = CURRENT_DATE - INTERVAL '1' DAY)
LIMIT 10 USD PER CALENDAR_DAY

-- Monthly budget for larger datasets
CREATE MATERIALIZED VIEW "monthly_data"
REFRESH_SCHEDULE = '@monthly'
AS (SELECT user_id, email, created_at FROM company_data."123")
LIMIT 500 USD PER CALENDAR_MONTH
```

### Combine price and date filters

Maximize value by filtering on both price and recency:

```sql theme={null}
SELECT user_id, email, event_timestamp
FROM company_data."123"
WHERE
  _price_cpm_usd <= 0.75
  AND event_timestamp >= CURRENT_DATE - INTERVAL '14' DAY
```

***

## Efficient aggregations

### Use APPROX\_COUNT\_DISTINCT instead of COUNT(DISTINCT ...)

`APPROX_COUNT_DISTINCT` is significantly faster and cheaper than `COUNT(DISTINCT ...)`. It returns exact results for low-cardinality columns and near-exact results at scale. Use it for any query that doesn't require a guaranteed exact count.

```sql theme={null}
-- Better: Use APPROX_COUNT_DISTINCT for unique counts
SELECT
  category,
  APPROX_COUNT_DISTINCT(user_id) AS unique_users
FROM company_data."123"
GROUP BY category
```

```sql theme={null}
-- Avoid: COUNT(DISTINCT ...) is slower and more expensive
SELECT
  category,
  COUNT(DISTINCT user_id) AS unique_users
FROM company_data."123"
GROUP BY category
```

Reserve `COUNT(DISTINCT ...)` for cases where an exact count drives business logic—for example, threshold comparisons in `HAVING` or `CASE WHEN` clauses:

```sql theme={null}
-- Exact count is appropriate here because the threshold matters
SELECT category, COUNT(DISTINCT user_id) AS unique_users
FROM company_data."123"
GROUP BY category
HAVING COUNT(DISTINCT user_id) >= 5
```

### Pre-aggregate for common queries

Create summary views for frequently queried aggregations:

```sql theme={null}
CREATE MATERIALIZED VIEW "daily_category_stats"
REFRESH_SCHEDULE = '@daily'
AS (
  SELECT
    DATE_TRUNC('day', event_timestamp) AS event_date,
    category,
    COUNT(1) AS events,
    APPROX_COUNT_DISTINCT(user_id) AS users,
    SUM(amount) AS total
  FROM company_data."123"
  WHERE event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
  GROUP BY DATE_TRUNC('day', event_timestamp), category
)
BUDGET 25 USD PER CALENDAR_DAY
```

***

## Limit result sets

### Always use LIMIT during exploration

```sql theme={null}
-- While exploring data
SELECT user_id, email, category
FROM company_data."123"
WHERE category = 'test'
LIMIT 100 ROWS

-- Or with offset for pagination
LIMIT 100 OFFSET 100 ROWS
```

***

## Efficient date handling

### Use DATE\_TRUNC for grouping

```sql theme={null}
SELECT
  DATE_TRUNC('day', event_timestamp) AS day,
  COUNT(1) AS events
FROM company_data."123"
GROUP BY DATE_TRUNC('day', event_timestamp)
```

### Avoid functions on indexed columns in WHERE

```sql theme={null}
-- Better: Compare directly
WHERE event_date >= '2024-01-01'

-- Avoid: Function on column
WHERE EXTRACT(YEAR FROM event_date) = 2024
```

***

## CTEs for complex queries

### Use CTEs for readability and potential optimization

```sql theme={null}
WITH
  base_data AS (
    SELECT user_id, event_date, amount
    FROM company_data."123"
    WHERE
      event_date >= CURRENT_DATE - INTERVAL '30' DAY
      AND _price_cpm_usd <= 1.00
  ),
  user_metrics AS (
    SELECT
      user_id,
      COUNT(1) AS event_count,
      SUM(amount) AS total_amount
    FROM base_data
    GROUP BY user_id
  ),
  ranked_users AS (
    SELECT
      user_id,
      event_count,
      total_amount,
      RANK() OVER (ORDER BY total_amount DESC) AS rank
    FROM user_metrics
  )
SELECT user_id, event_count, total_amount, rank
FROM ranked_users
WHERE rank <= 100
```

***

## Anti-patterns to avoid

### Use explicit column lists (wildcards not supported)

NQL requires explicit column lists—`SELECT *` and `COUNT(*)` are not supported. See [Explicit Column Selection](/nql/general/explicit-columns) for why this design choice improves cost transparency, governance, and query optimization.

```sql theme={null}
-- Not supported in NQL
SELECT * FROM company_data."123"

-- Required: Select only needed columns
SELECT user_id, email, created_at FROM company_data."123"

-- Not supported: COUNT(*)
SELECT COUNT(*) FROM company_data."123"

-- Required: Use COUNT(1) for row counts
SELECT COUNT(1) FROM company_data."123"
```

### Don't use LIKE with leading wildcards when possible

```sql theme={null}
-- Slow: Leading wildcard
WHERE email LIKE '%@gmail.com'

-- Better: Use REGEXP_EXTRACT or suffix column
WHERE REGEXP_EXTRACT(email, '@(.+)$') = 'gmail.com'
```

### Don't nest too many subqueries

```sql theme={null}
-- Avoid: Deeply nested subqueries
SELECT user_id, email FROM (
  SELECT user_id, email FROM (
    SELECT user_id, email FROM (
      SELECT user_id, email FROM company_data."123"
    ) a
  ) b
) c

-- Better: Use CTEs
WITH step1 AS (SELECT user_id, email FROM company_data."123"),
     step2 AS (SELECT user_id, email FROM step1),
     step3 AS (SELECT user_id, email FROM step2)
SELECT user_id, email FROM step3
```

***

## Related content

<CardGroup>
  <Card title="Query Optimization Guide" icon="gauge-high" href="/guides/nql/query-optimization">
    Detailed optimization techniques
  </Card>

  <Card title="Join Performance" icon="link" href="/concepts/nql/join-performance">
    Understanding join performance implications
  </Card>

  <Card title="Common Queries" icon="copy" href="/cookbooks/nql/common-queries">
    Copy-paste query patterns
  </Card>

  <Card title="Complex Joins" icon="diagram-project" href="/cookbooks/nql/complex-joins">
    Advanced join recipes
  </Card>
</CardGroup>


# Batch Processing with Model Inference
Source: https://docs.narrative.io/cookbooks/sdk/batch-inference-patterns

Process large volumes of data efficiently using inference

This cookbook demonstrates patterns for efficiently processing large datasets with Model Inference. You'll learn chunking strategies, parallel execution, rate limiting, error recovery, and result aggregation.

## What this recipe accomplishes

* Process large datasets in manageable chunks
* Execute inference requests in parallel
* Handle rate limits and backoff
* Recover from failures without losing progress
* Aggregate and report results

## Prerequisites

* SDK installed and configured (see [Authentication](/guides/sdk/authentication))
* A data plane ID where inference will run
* Understanding of [Running Model Inference](/guides/sdk/running-model-inference)

## Complete example

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

// Types
interface BatchItem {
  id: string;
  content: string;
}

interface ProcessedItem {
  id: string;
  result: unknown;
  success: boolean;
  error?: string;
  processingTime: number;
}

interface BatchProgress {
  total: number;
  processed: number;
  succeeded: number;
  failed: number;
  startTime: number;
}

// Configuration
const CONFIG = {
  dataPlaneId: process.env.DATA_PLANE_ID!,
  model: 'anthropic.claude-haiku-4.5' as const, // Use faster model for batch
  concurrency: 5, // Parallel requests
  chunkSize: 50, // Items per chunk
  maxRetries: 3,
  retryDelayMs: 2000,
  pollIntervalMs: 2000,
  jobTimeoutMs: 60000,
};

// Initialize API
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY!,
});

// Output schema (customize for your use case)
const outputSchema = {
  type: 'object',
  properties: {
    category: { type: 'string' },
    score: { type: 'number', minimum: 0, maximum: 1 },
    summary: { type: 'string', maxLength: 100 },
  },
  required: ['category', 'score'],
};

/**
 * Process a single item with retries
 */
async function processItem(
  item: BatchItem,
  retryCount = 0
): Promise<ProcessedItem> {
  const startTime = Date.now();

  try {
    // Submit inference request
    const job = await api.runModelInference({
      data_plane_id: CONFIG.dataPlaneId,
      model: CONFIG.model,
      messages: [
        {
          role: 'user',
          content: [
            { type: 'text', text: `Process this item:\n\nID: ${item.id}\nContent: ${item.content}` }
          ]
        },
      ],
      inference_config: {
        output_format_schema: outputSchema,
        temperature: 0.2,
      },
      tags: ['batch-processing', `item-${item.id}`],
    });

    // Wait for completion
    const result = await waitForJob(job.id);

    return {
      id: item.id,
      result: result,
      success: true,
      processingTime: Date.now() - startTime,
    };
  } catch (error) {
    // Retry on transient errors
    if (retryCount < CONFIG.maxRetries && isRetryableError(error)) {
      const delay = CONFIG.retryDelayMs * Math.pow(2, retryCount);
      console.log(`Retrying item ${item.id} in ${delay}ms (attempt ${retryCount + 1})`);
      await sleep(delay);
      return processItem(item, retryCount + 1);
    }

    return {
      id: item.id,
      result: null,
      success: false,
      error: error instanceof Error ? error.message : 'Unknown error',
      processingTime: Date.now() - startTime,
    };
  }
}

/**
 * Wait for job completion with timeout
 */
async function waitForJob(jobId: string): Promise<unknown> {
  const startTime = Date.now();

  while (Date.now() - startTime < CONFIG.jobTimeoutMs) {
    const job = await api.getJob(jobId);

    if (job.state === 'completed' && job.result) {
      return job.result.structured_output;
    }

    if (job.state === 'failed') {
      throw new Error(`Job failed: ${JSON.stringify(job.failures)}`);
    }

    await sleep(CONFIG.pollIntervalMs);
  }

  throw new Error(`Job ${jobId} timed out`);
}

/**
 * Check if error is retryable
 */
function isRetryableError(error: unknown): boolean {
  if (error instanceof Error) {
    // Rate limit or temporary failures
    return (
      error.message.includes('rate limit') ||
      error.message.includes('timeout') ||
      error.message.includes('503') ||
      error.message.includes('429')
    );
  }
  return false;
}

/**
 * Process items with controlled concurrency
 */
async function processWithConcurrency(
  items: BatchItem[],
  concurrency: number,
  onProgress?: (progress: BatchProgress) => void
): Promise<ProcessedItem[]> {
  const results: ProcessedItem[] = [];
  const progress: BatchProgress = {
    total: items.length,
    processed: 0,
    succeeded: 0,
    failed: 0,
    startTime: Date.now(),
  };

  // Process in chunks
  for (let i = 0; i < items.length; i += concurrency) {
    const chunk = items.slice(i, i + concurrency);

    // Process chunk in parallel
    const chunkResults = await Promise.all(
      chunk.map(item => processItem(item))
    );

    // Update results and progress
    for (const result of chunkResults) {
      results.push(result);
      progress.processed++;
      if (result.success) {
        progress.succeeded++;
      } else {
        progress.failed++;
      }
    }

    // Report progress
    if (onProgress) {
      onProgress(progress);
    }

    // Rate limiting between chunks
    if (i + concurrency < items.length) {
      await sleep(500);
    }
  }

  return results;
}

/**
 * Chunk an array into smaller arrays
 */
function chunkArray<T>(array: T[], size: number): T[][] {
  const chunks: T[][] = [];
  for (let i = 0; i < array.length; i += size) {
    chunks.push(array.slice(i, i + size));
  }
  return chunks;
}

/**
 * Sleep helper
 */
function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

/**
 * Format duration
 */
function formatDuration(ms: number): string {
  if (ms < 1000) return `${ms}ms`;
  if (ms < 60000) return `${(ms / 1000).toFixed(1)}s`;
  return `${(ms / 60000).toFixed(1)}m`;
}

/**
 * Main batch processing function
 */
async function processBatch(items: BatchItem[]): Promise<void> {
  console.log(`\n=== Batch Processing ===`);
  console.log(`Items: ${items.length}`);
  console.log(`Concurrency: ${CONFIG.concurrency}`);
  console.log(`Model: ${CONFIG.model}\n`);

  const startTime = Date.now();

  // Process with progress reporting
  const results = await processWithConcurrency(
    items,
    CONFIG.concurrency,
    (progress) => {
      const elapsed = Date.now() - progress.startTime;
      const rate = progress.processed / (elapsed / 1000);
      const eta = progress.processed > 0
        ? (progress.total - progress.processed) / rate
        : 0;

      process.stdout.write(
        `\rProgress: ${progress.processed}/${progress.total} ` +
        `(${progress.succeeded} ok, ${progress.failed} failed) ` +
        `Rate: ${rate.toFixed(1)}/s ETA: ${formatDuration(eta * 1000)}  `
      );
    }
  );

  console.log('\n');

  // Aggregate results
  const totalTime = Date.now() - startTime;
  const succeeded = results.filter(r => r.success);
  const failed = results.filter(r => !r.success);
  const avgTime = results.reduce((sum, r) => sum + r.processingTime, 0) / results.length;

  console.log('=== Results ===');
  console.log(`Total time: ${formatDuration(totalTime)}`);
  console.log(`Processed: ${results.length}`);
  console.log(`Succeeded: ${succeeded.length} (${(succeeded.length / results.length * 100).toFixed(1)}%)`);
  console.log(`Failed: ${failed.length}`);
  console.log(`Avg processing time: ${formatDuration(avgTime)}`);
  console.log(`Throughput: ${(results.length / (totalTime / 1000)).toFixed(2)} items/sec`);

  if (failed.length > 0) {
    console.log('\nFailed items:');
    failed.slice(0, 5).forEach(f => {
      console.log(`  - ${f.id}: ${f.error}`);
    });
    if (failed.length > 5) {
      console.log(`  ... and ${failed.length - 5} more`);
    }
  }

  // Return or store results as needed
  return;
}

/**
 * Checkpoint-based processing for very large batches
 */
async function processWithCheckpoints(
  items: BatchItem[],
  checkpointFile: string
): Promise<void> {
  const fs = await import('fs').then(m => m.promises);

  // Load checkpoint if exists
  let processedIds: Set<string> = new Set();
  try {
    const checkpoint = await fs.readFile(checkpointFile, 'utf-8');
    processedIds = new Set(JSON.parse(checkpoint));
    console.log(`Resuming from checkpoint: ${processedIds.size} items already processed`);
  } catch {
    console.log('Starting fresh (no checkpoint found)');
  }

  // Filter already processed
  const remaining = items.filter(item => !processedIds.has(item.id));
  console.log(`Remaining items to process: ${remaining.length}`);

  // Process in chunks with checkpoints
  const chunks = chunkArray(remaining, CONFIG.chunkSize);

  for (let i = 0; i < chunks.length; i++) {
    console.log(`\nProcessing chunk ${i + 1}/${chunks.length}...`);

    const results = await processWithConcurrency(chunks[i], CONFIG.concurrency);

    // Update checkpoint
    results.forEach(r => processedIds.add(r.id));
    await fs.writeFile(checkpointFile, JSON.stringify([...processedIds]));

    console.log(`Checkpoint saved: ${processedIds.size} items`);
  }
}

// Example usage
async function main(): Promise<void> {
  // Generate sample items
  const sampleItems: BatchItem[] = Array.from({ length: 100 }, (_, i) => ({
    id: `item-${i + 1}`,
    content: `Sample content for item ${i + 1}`,
  }));

  await processBatch(sampleItems);
}

main().catch(console.error);
```

## Key patterns

### Controlled concurrency

Limit parallel requests to avoid overwhelming the system:

```typescript theme={null}
// Process in parallel batches
for (let i = 0; i < items.length; i += concurrency) {
  const chunk = items.slice(i, i + concurrency);
  await Promise.all(chunk.map(processItem));
}
```

### Exponential backoff

Retry failed requests with increasing delays:

```typescript theme={null}
const delay = baseDelay * Math.pow(2, retryCount);
await sleep(delay);
```

### Progress tracking

Report progress for visibility into long-running jobs:

```typescript theme={null}
onProgress({
  processed: count,
  total: items.length,
  rate: count / elapsedSeconds,
});
```

### Checkpoint recovery

Save progress for resumable processing:

```typescript theme={null}
// Save after each chunk
await fs.writeFile(checkpointFile, JSON.stringify([...processedIds]));

// Resume from checkpoint
const checkpoint = await fs.readFile(checkpointFile, 'utf-8');
const processedIds = new Set(JSON.parse(checkpoint));
```

## Performance tuning

| Parameter        | Effect                 | Recommendation                                 |
| ---------------- | ---------------------- | ---------------------------------------------- |
| `concurrency`    | Parallel requests      | Start with 5, increase if system handles it    |
| `chunkSize`      | Items per checkpoint   | 50-100 for good checkpoint granularity         |
| `model`          | Speed vs capability    | Use Haiku for simple tasks, Sonnet for complex |
| `pollIntervalMs` | Status check frequency | 2000ms balances responsiveness and API calls   |

## Related content

<CardGroup>
  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Inference fundamentals
  </Card>

  <Card title="Choosing Models" icon="scale-balanced" href="/guides/sdk/choosing-inference-models">
    Model selection guide
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor job status
  </Card>

  <Card title="Data Enrichment" icon="wand-magic-sparkles" href="/cookbooks/sdk/model-inference-automation">
    Enrichment pipeline example
  </Card>
</CardGroup>


# Automating Data Enrichment with Model Inference
Source: https://docs.narrative.io/cookbooks/sdk/model-inference-automation

Build a pipeline that enriches dataset records using AI inference

This cookbook demonstrates how to build an automated data enrichment pipeline using Model Inference. You'll learn to fetch records from a dataset, run inference to classify or enrich each record, and handle results at scale.

## What this recipe accomplishes

* Fetch sample records from a dataset
* Run AI inference to enrich each record with classifications
* Handle inference results with proper error handling
* Process records in batches for efficiency

## Prerequisites

* SDK installed and configured (see [Authentication](/guides/sdk/authentication))
* A dataset with records to enrich
* A data plane ID where inference will run

## Complete example

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

// Types for our enrichment
interface RecordToEnrich {
  id: string;
  text: string;
  [key: string]: unknown;
}

interface EnrichmentResult {
  category: 'retail' | 'finance' | 'healthcare' | 'technology' | 'other';
  confidence: number;
  tags: string[];
  summary: string;
}

interface EnrichedRecord extends RecordToEnrich {
  enrichment: EnrichmentResult;
  enriched_at: string;
}

// Configuration
const CONFIG = {
  dataPlaneId: process.env.DATA_PLANE_ID!,
  datasetId: parseInt(process.env.DATASET_ID!),
  model: 'anthropic.claude-sonnet-4.5' as const,
  batchSize: 10,
  maxRetries: 3,
};

// Initialize API
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY!,
});

// Schema for the enrichment output
const enrichmentSchema = {
  type: 'object',
  properties: {
    category: {
      type: 'string',
      enum: ['retail', 'finance', 'healthcare', 'technology', 'other'],
      description: 'Primary category for this record',
    },
    confidence: {
      type: 'number',
      minimum: 0,
      maximum: 1,
      description: 'Confidence score for the classification',
    },
    tags: {
      type: 'array',
      items: { type: 'string' },
      maxItems: 5,
      description: 'Relevant tags for this record',
    },
    summary: {
      type: 'string',
      maxLength: 200,
      description: 'Brief summary of the record content',
    },
  },
  required: ['category', 'confidence', 'tags', 'summary'],
};

/**
 * Wait for an inference job to complete
 */
async function waitForInference(jobId: string, maxWaitMs = 60000): Promise<EnrichmentResult | null> {
  const startTime = Date.now();
  const pollInterval = 2000;

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    if (job.state === 'completed' && job.result) {
      return job.result.structured_output as EnrichmentResult;
    }

    if (job.state === 'failed') {
      console.error('Inference job failed:', job.failures);
      return null;
    }

    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`Job ${jobId} timed out`);
}

/**
 * Enrich a single record using inference
 */
async function enrichRecord(record: RecordToEnrich): Promise<EnrichedRecord | null> {
  const prompt = `Analyze and categorize this record:

ID: ${record.id}
Content: ${record.text}

Classify into one of: retail, finance, healthcare, technology, or other.
Extract relevant tags and provide a brief summary.`;

  try {
    const job = await api.runModelInference({
      data_plane_id: CONFIG.dataPlaneId,
      model: CONFIG.model,
      messages: [
        {
          role: 'system',
          content: [
            {
              type: 'text',
              text: 'You are a data classification assistant. Analyze records and provide structured classifications.',
            },
          ],
        },
        {
          role: 'user',
          content: [{ type: 'text', text: prompt }],
        },
      ],
      inference_config: {
        output_format_schema: enrichmentSchema,
        temperature: 0.3, // Lower temperature for consistent classifications
      },
      tags: ['enrichment', `dataset-${CONFIG.datasetId}`],
    });

    const result = await waitForInference(job.id);

    if (result) {
      return {
        ...record,
        enrichment: result,
        enriched_at: new Date().toISOString(),
      };
    }

    return null;
  } catch (error) {
    console.error(`Failed to enrich record ${record.id}:`, error);
    return null;
  }
}

/**
 * Process records in batches with rate limiting
 */
async function processBatch(records: RecordToEnrich[]): Promise<EnrichedRecord[]> {
  const results: EnrichedRecord[] = [];

  // Process in parallel within the batch
  const promises = records.map(async (record) => {
    const enriched = await enrichRecord(record);
    if (enriched) {
      results.push(enriched);
    }
    return enriched;
  });

  await Promise.all(promises);
  return results;
}

/**
 * Main enrichment pipeline
 */
async function runEnrichmentPipeline(): Promise<void> {
  console.log('Starting enrichment pipeline...');

  // Step 1: Fetch records from the dataset
  console.log(`Fetching sample from dataset ${CONFIG.datasetId}...`);
  const sample = await api.getDatasetSample(CONFIG.datasetId, 100);
  const records = sample.records as RecordToEnrich[];

  console.log(`Found ${records.length} records to process`);

  // Step 2: Process in batches
  const allEnriched: EnrichedRecord[] = [];

  for (let i = 0; i < records.length; i += CONFIG.batchSize) {
    const batch = records.slice(i, i + CONFIG.batchSize);
    const batchNumber = Math.floor(i / CONFIG.batchSize) + 1;
    const totalBatches = Math.ceil(records.length / CONFIG.batchSize);

    console.log(`Processing batch ${batchNumber}/${totalBatches}...`);

    const enriched = await processBatch(batch);
    allEnriched.push(...enriched);

    console.log(`  Enriched ${enriched.length}/${batch.length} records`);

    // Rate limiting between batches
    if (i + CONFIG.batchSize < records.length) {
      await new Promise(resolve => setTimeout(resolve, 1000));
    }
  }

  // Step 3: Report results
  console.log('\n=== Enrichment Complete ===');
  console.log(`Total records processed: ${records.length}`);
  console.log(`Successfully enriched: ${allEnriched.length}`);
  console.log(`Failed: ${records.length - allEnriched.length}`);

  // Category distribution
  const categoryCount = allEnriched.reduce((acc, r) => {
    acc[r.enrichment.category] = (acc[r.enrichment.category] || 0) + 1;
    return acc;
  }, {} as Record<string, number>);

  console.log('\nCategory distribution:');
  Object.entries(categoryCount)
    .sort((a, b) => b[1] - a[1])
    .forEach(([category, count]) => {
      console.log(`  ${category}: ${count}`);
    });

  // Average confidence
  const avgConfidence = allEnriched.reduce((sum, r) => sum + r.enrichment.confidence, 0) / allEnriched.length;
  console.log(`\nAverage confidence: ${(avgConfidence * 100).toFixed(1)}%`);
}

// Run the pipeline
runEnrichmentPipeline()
  .then(() => console.log('\nPipeline finished successfully'))
  .catch((error) => {
    console.error('Pipeline failed:', error);
    process.exit(1);
  });
```

## How it works

1. **Fetch records**: The pipeline retrieves sample records from your dataset using `getDatasetSample()`

2. **Enrich with inference**: Each record is sent to Model Inference with a prompt asking for classification and tagging

3. **Structured output**: The JSON Schema ensures every response includes category, confidence, tags, and summary

4. **Batch processing**: Records are processed in parallel batches with rate limiting between batches

5. **Error handling**: Failed enrichments are tracked and reported without stopping the pipeline

## Customizing the recipe

### Different enrichment tasks

Change the schema and prompt for your use case:

```typescript theme={null}
// Sentiment analysis
const sentimentSchema = {
  type: 'object',
  properties: {
    sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
    score: { type: 'number', minimum: -1, maximum: 1 },
    key_phrases: { type: 'array', items: { type: 'string' } }
  },
  required: ['sentiment', 'score']
};
```

### Writing results back

Store enriched records in a new dataset:

```typescript theme={null}
// Create output dataset
const outputDataset = await api.createDataset({
  name: 'enriched_records',
  display_name: 'Enriched Records',
  schema: { /* schema matching EnrichedRecord */ }
});

// Write enriched data
// ... upload enriched records
```

## Related content

<CardGroup>
  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Inference fundamentals
  </Card>

  <Card title="Batch Processing" icon="layer-group" href="/cookbooks/sdk/batch-inference-patterns">
    Advanced batch patterns
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor job status
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets">
    Dataset operations
  </Card>
</CardGroup>


# Automating Data Workflows
Source: https://docs.narrative.io/cookbooks/sdk/typescript-automation

Build an automated data pipeline using the TypeScript SDK

This recipe demonstrates a complete automation workflow: uploading data, waiting for ingestion, running a query, and exporting results.

## What this recipe accomplishes

* Upload a CSV file to a dataset
* Monitor the ingestion job until completion
* Execute an NQL query against the ingested data
* Export results to a local file

## Prerequisites

* Node.js 18 or later
* TypeScript configured in your project
* A Narrative API key with read/write permissions
* An existing dataset to upload data to

## Complete example

Create a file called `data-pipeline.ts`:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

// Configuration
const DATASET_ID = 12345;  // Replace with your dataset ID
const INPUT_FILE = './data/customer_events.csv';
const OUTPUT_FILE = './output/aggregated_events.csv';

// Initialize the SDK
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

/**
 * Wait for a job to complete with exponential backoff
 */
async function waitForJob(jobId: string, maxWaitMs = 300000): Promise<void> {
  const startTime = Date.now();
  let pollInterval = 2000;
  const maxInterval = 30000;

  console.log(`Waiting for job ${jobId}...`);

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    console.log(`  Status: ${job.state}`);

    if (job.state === 'completed') {
      console.log('Job completed successfully');
      return;
    }

    if (job.state === 'failed') {
      throw new Error(`Job failed: ${JSON.stringify(job.failures)}`);
    }

    await new Promise(resolve => setTimeout(resolve, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, maxInterval);
  }

  throw new Error(`Job ${jobId} timed out after ${maxWaitMs}ms`);
}

/**
 * Upload a file to a dataset
 */
async function uploadFile(datasetId: number, filePath: string): Promise<string> {
  const fileName = path.basename(filePath);
  const fileContent = fs.readFileSync(filePath);

  console.log(`Uploading ${fileName}...`);

  // Get upload URL
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);

  // Upload to S3
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);

  // Confirm upload
  await api.confirmUpload(datasetId, uploadUrl.upload_id);

  console.log(`Upload confirmed: ${uploadUrl.upload_id}`);
  return uploadUrl.upload_id;
}

/**
 * Run an aggregation query
 */
async function runQuery(datasetId: number): Promise<any[]> {
  console.log('Running aggregation query...');

  const result = await api.executeNql({
    nql: `
      SELECT
        sha256_hashed_email,
        COUNT(1) as event_count,
        MAX(event_timestamp) as last_seen,
        SUM(event_value) as total_value
      FROM company_data."dataset_${datasetId}"
      WHERE event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
      GROUP BY sha256_hashed_email
      ORDER BY event_count DESC
      LIMIT 10000
    `,
    data_plane_id: null,
  });

  if (result.state === 'failed') {
    throw new Error(`Query failed: ${JSON.stringify(result.failures)}`);
  }

  console.log(`Query completed: ${result.result.rows} rows`);

  // For this example, we'll return mock data since actual row data
  // would come from the result or a subsequent data retrieval
  return [];
}

/**
 * Export results to CSV
 */
function exportToCsv(data: any[], outputPath: string): void {
  if (data.length === 0) {
    console.log('No data to export');
    return;
  }

  const headers = Object.keys(data[0]);
  const rows = data.map(row =>
    headers.map(h => String(row[h] ?? '')).join(',')
  );

  const csv = [headers.join(','), ...rows].join('\n');

  // Ensure output directory exists
  const outputDir = path.dirname(outputPath);
  if (!fs.existsSync(outputDir)) {
    fs.mkdirSync(outputDir, { recursive: true });
  }

  fs.writeFileSync(outputPath, csv);
  console.log(`Exported ${data.length} rows to ${outputPath}`);
}

/**
 * Main pipeline function
 */
async function runPipeline() {
  console.log('Starting data pipeline...\n');

  try {
    // Step 1: Upload data
    console.log('=== Step 1: Upload Data ===');
    if (fs.existsSync(INPUT_FILE)) {
      const uploadId = await uploadFile(DATASET_ID, INPUT_FILE);

      // Wait for ingestion (if there's an associated job)
      // Note: You may need to track the ingestion job separately
      console.log('Upload complete\n');
    } else {
      console.log(`Skipping upload: ${INPUT_FILE} not found\n`);
    }

    // Step 2: Run query
    console.log('=== Step 2: Run Query ===');
    const queryResults = await runQuery(DATASET_ID);
    console.log('Query complete\n');

    // Step 3: Export results
    console.log('=== Step 3: Export Results ===');
    exportToCsv(queryResults, OUTPUT_FILE);
    console.log('Export complete\n');

    console.log('Pipeline finished successfully!');

  } catch (error) {
    console.error('Pipeline failed:', error);
    process.exit(1);
  }
}

// Run the pipeline
runPipeline();
```

## Running the pipeline

Set your environment variable and run:

```bash theme={null}
export NARRATIVE_API_KEY="your-api-key-here"
npx ts-node data-pipeline.ts
```

## Scheduling with cron

To run automatically, add to crontab:

```bash theme={null}
# Run daily at 2 AM
0 2 * * * cd /path/to/project && NARRATIVE_API_KEY="your-key" npx ts-node data-pipeline.ts >> /var/log/pipeline.log 2>&1
```

Or use Node.js scheduling:

```typescript theme={null}
import cron from 'node-cron';

// Run every day at 2 AM
cron.schedule('0 2 * * *', () => {
  runPipeline();
});
```

## Adding error notifications

Send alerts on failure:

```typescript theme={null}
async function runPipelineWithNotification() {
  try {
    await runPipeline();
    await sendNotification('Pipeline completed successfully');
  } catch (error) {
    await sendNotification(`Pipeline failed: ${error.message}`);
    throw error;
  }
}

async function sendNotification(message: string) {
  // Example: Send to Slack
  await fetch(process.env.SLACK_WEBHOOK_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text: message }),
  });
}
```

## Parallel processing

Process multiple datasets concurrently:

```typescript theme={null}
async function processMultipleDatasets(datasetIds: number[]) {
  const results = await Promise.allSettled(
    datasetIds.map(async (id) => {
      console.log(`Processing dataset ${id}...`);
      const data = await runQuery(id);
      exportToCsv(data, `./output/dataset_${id}.csv`);
      return { id, rows: data.length };
    })
  );

  const succeeded = results.filter(r => r.status === 'fulfilled');
  const failed = results.filter(r => r.status === 'rejected');

  console.log(`Completed: ${succeeded.length}/${results.length}`);
  if (failed.length > 0) {
    console.error('Failed datasets:', failed);
  }
}
```

## Handling incremental updates

Track last run time for incremental processing:

```typescript theme={null}
const STATE_FILE = './state/last_run.json';

function getLastRunTime(): string | null {
  try {
    const state = JSON.parse(fs.readFileSync(STATE_FILE, 'utf8'));
    return state.lastRun;
  } catch {
    return null;
  }
}

function saveLastRunTime(timestamp: string): void {
  const dir = path.dirname(STATE_FILE);
  if (!fs.existsSync(dir)) {
    fs.mkdirSync(dir, { recursive: true });
  }
  fs.writeFileSync(STATE_FILE, JSON.stringify({ lastRun: timestamp }));
}

async function runIncrementalQuery(datasetId: number) {
  const lastRun = getLastRunTime();
  const now = new Date().toISOString();

  const whereClause = lastRun
    ? `WHERE event_timestamp > '${lastRun}'`
    : '';

  const result = await api.executeNql({
    nql: `
      SELECT user_id, email, event_timestamp, event_value
      FROM company_data."dataset_${datasetId}"
      ${whereClause}
      LIMIT 100000
    `,
    data_plane_id: null,
  });

  saveLastRunTime(now);
  return result;
}
```

## Related content

<CardGroup>
  <Card title="SDK Quickstart" icon="rocket" href="/getting-started/sdk-quickstart">
    Get started with the SDK
  </Card>

  <Card title="Uploading Data" icon="upload" href="/guides/sdk/uploading-data">
    Detailed upload guide
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor async operations
  </Card>

  <Card title="Executing Queries" icon="terminal" href="/guides/sdk/executing-queries">
    Query patterns and best practices
  </Card>
</CardGroup>


# Auto-generating Dataset Descriptions
Source: https://docs.narrative.io/cookbooks/use-cases/ai-dataset-descriptions

Use Model Inference to automatically generate human-readable dataset descriptions

This cookbook demonstrates how to use Model Inference to automatically generate descriptions for datasets based on their metadata and sample data. This is useful for documenting datasets, improving discoverability, and maintaining data catalogs.

## What this recipe accomplishes

* Fetch dataset metadata and column information
* Analyze sample data to understand content patterns
* Generate a comprehensive, human-readable description
* Update the dataset with the generated description

## Prerequisites

* SDK installed and configured (see [Authentication](/guides/sdk/authentication))
* A dataset that needs a description
* A data plane ID where inference will run

## Complete example

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

// Types for the generated description
interface GeneratedDescription {
  title: string;
  summary: string;
  column_descriptions: Array<{
    column_name: string;
    data_type: string;
    description: string;
    sample_values: string;
  }>;
  use_cases: string[];
  data_quality_notes: string[];
}

// Configuration
const CONFIG = {
  dataPlaneId: process.env.DATA_PLANE_ID!,
  datasetId: parseInt(process.env.DATASET_ID!),
  model: 'anthropic.claude-sonnet-4.5' as const,
};

// Initialize API
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY!,
});

// Schema for the description output
const descriptionSchema = {
  type: 'object',
  properties: {
    title: {
      type: 'string',
      maxLength: 100,
      description: 'A clear, descriptive title for the dataset',
    },
    summary: {
      type: 'string',
      maxLength: 500,
      description: 'A comprehensive summary of what this dataset contains and its purpose',
    },
    column_descriptions: {
      type: 'array',
      items: {
        type: 'object',
        properties: {
          column_name: { type: 'string' },
          data_type: { type: 'string' },
          description: { type: 'string', maxLength: 200 },
          sample_values: { type: 'string', maxLength: 100 },
        },
        required: ['column_name', 'data_type', 'description'],
      },
      description: 'Description of each column in the dataset',
    },
    use_cases: {
      type: 'array',
      items: { type: 'string' },
      maxItems: 5,
      description: 'Potential use cases for this dataset',
    },
    data_quality_notes: {
      type: 'array',
      items: { type: 'string' },
      maxItems: 3,
      description: 'Notes about data quality or considerations',
    },
  },
  required: ['title', 'summary', 'column_descriptions', 'use_cases'],
};

/**
 * Wait for inference job to complete
 */
async function waitForInference(jobId: string): Promise<GeneratedDescription | null> {
  const maxWaitMs = 90000; // 90 seconds for potentially long analysis
  const startTime = Date.now();
  const pollInterval = 3000;

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    if (job.state === 'completed' && job.result) {
      return job.result.structured_output as GeneratedDescription;
    }

    if (job.state === 'failed') {
      console.error('Inference job failed:', job.failures);
      return null;
    }

    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`Job ${jobId} timed out`);
}

/**
 * Format sample data for the prompt
 */
function formatSampleForPrompt(
  sample: Record<string, unknown>[],
  maxRows = 5
): string {
  if (sample.length === 0) return 'No sample data available';

  const rows = sample.slice(0, maxRows);
  const columns = Object.keys(rows[0]);

  let formatted = 'Sample rows:\n';
  formatted += columns.join(' | ') + '\n';
  formatted += columns.map(() => '---').join(' | ') + '\n';

  rows.forEach(row => {
    formatted += columns.map(col => {
      const val = row[col];
      const str = val === null ? 'NULL' : String(val);
      return str.length > 30 ? str.substring(0, 27) + '...' : str;
    }).join(' | ') + '\n';
  });

  return formatted;
}

/**
 * Generate description for a dataset
 */
async function generateDatasetDescription(
  datasetId: number
): Promise<GeneratedDescription | null> {
  console.log(`Fetching metadata for dataset ${datasetId}...`);

  // Step 1: Get dataset metadata
  const dataset = await api.getDataset(datasetId);

  // Step 2: Get sample data
  console.log('Fetching sample data...');
  const sample = await api.getDatasetSample(datasetId, 20);
  const sampleRecords = sample.records as Record<string, unknown>[];

  // Step 3: Extract schema information
  const schemaInfo = dataset.schema?.properties
    ? Object.entries(dataset.schema.properties).map(([name, prop]: [string, any]) => ({
        name,
        type: prop.type || 'unknown',
        description: prop.description || '',
      }))
    : [];

  // Step 4: Build the prompt
  const prompt = `Analyze this dataset and generate a comprehensive description.

Dataset Name: ${dataset.name}
Display Name: ${dataset.display_name || 'Not set'}
Current Description: ${dataset.description || 'None'}

Schema (${schemaInfo.length} columns):
${schemaInfo.map(col => `- ${col.name} (${col.type}): ${col.description || 'No description'}`).join('\n')}

${formatSampleForPrompt(sampleRecords)}

Total rows in sample: ${sampleRecords.length}

Based on this information:
1. Generate a clear, descriptive title
2. Write a comprehensive summary explaining what this dataset contains
3. Describe each column based on its name, type, and sample values
4. Suggest potential use cases
5. Note any data quality observations`;

  console.log('Running inference to generate description...');

  // Step 5: Run inference
  const job = await api.runModelInference({
    data_plane_id: CONFIG.dataPlaneId,
    model: CONFIG.model,
    messages: [
      {
        role: 'system',
        content: [
          {
            type: 'text',
            text: `You are a data documentation expert. Analyze datasets and generate clear,
accurate descriptions that help users understand the data. Be specific and factual,
basing your analysis on the actual schema and sample data provided.`,
          },
        ],
      },
      {
        role: 'user',
        content: [{ type: 'text', text: prompt }],
      },
    ],
    inference_config: {
      output_format_schema: descriptionSchema,
      max_tokens: 2000,
      temperature: 0.3,
    },
    tags: ['description-generation', `dataset-${datasetId}`],
  });

  console.log(`Inference job created: ${job.id}`);
  return await waitForInference(job.id);
}

/**
 * Update dataset with generated description
 */
async function updateDatasetDescription(
  datasetId: number,
  description: GeneratedDescription
): Promise<void> {
  // Format the full description
  const fullDescription = `${description.summary}

## Columns

${description.column_descriptions.map(col =>
  `**${col.column_name}** (${col.data_type}): ${col.description}${col.sample_values ? ` Examples: ${col.sample_values}` : ''}`
).join('\n\n')}

## Use Cases

${description.use_cases.map(uc => `- ${uc}`).join('\n')}

${description.data_quality_notes?.length ? `
## Data Quality Notes

${description.data_quality_notes.map(note => `- ${note}`).join('\n')}` : ''}`;

  console.log('Updating dataset with generated description...');

  await api.updateDataset({
    dataset_id: datasetId,
    display_name: description.title,
    description: fullDescription,
  });

  console.log('Dataset updated successfully');
}

/**
 * Main function
 */
async function main(): Promise<void> {
  console.log('=== Dataset Description Generator ===\n');

  const description = await generateDatasetDescription(CONFIG.datasetId);

  if (!description) {
    console.error('Failed to generate description');
    process.exit(1);
  }

  console.log('\n=== Generated Description ===\n');
  console.log(`Title: ${description.title}`);
  console.log(`\nSummary:\n${description.summary}`);
  console.log(`\nColumns:`);
  description.column_descriptions.forEach(col => {
    console.log(`  - ${col.column_name}: ${col.description}`);
  });
  console.log(`\nUse Cases:`);
  description.use_cases.forEach(uc => console.log(`  - ${uc}`));

  // Optionally update the dataset
  const shouldUpdate = process.env.UPDATE_DATASET === 'true';
  if (shouldUpdate) {
    await updateDatasetDescription(CONFIG.datasetId, description);
  } else {
    console.log('\nSet UPDATE_DATASET=true to apply this description to the dataset');
  }
}

// Run
main()
  .then(() => console.log('\nDone'))
  .catch((error) => {
    console.error('Error:', error);
    process.exit(1);
  });
```

## How it works

1. **Fetch metadata**: The script retrieves the dataset's schema and existing metadata

2. **Get sample data**: A sample of records helps the model understand actual data patterns

3. **Build context**: Schema information and sample data are formatted into a comprehensive prompt

4. **Generate description**: Model Inference analyzes the data and generates structured documentation

5. **Update dataset**: The generated description can be applied back to the dataset

## Variations

### Batch processing multiple datasets

```typescript theme={null}
async function describeAllDatasets(): Promise<void> {
  const datasets = await api.getDatasets();

  for (const dataset of datasets.records) {
    if (!dataset.description || dataset.description.length < 50) {
      console.log(`Processing: ${dataset.name}`);
      const description = await generateDatasetDescription(dataset.id);
      if (description) {
        await updateDatasetDescription(dataset.id, description);
      }
    }
  }
}
```

### Description quality scoring

```typescript theme={null}
const qualitySchema = {
  type: 'object',
  properties: {
    completeness_score: { type: 'number', minimum: 0, maximum: 1 },
    clarity_score: { type: 'number', minimum: 0, maximum: 1 },
    improvements: { type: 'array', items: { type: 'string' } }
  },
  required: ['completeness_score', 'clarity_score']
};
```

## Related content

<CardGroup>
  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Inference fundamentals
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets">
    Dataset operations
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Define response schemas
  </Card>

  <Card title="Model Inference Overview" icon="brain" href="/concepts/model-inference/overview">
    How inference works
  </Card>
</CardGroup>


# Demographic Enrichment
Source: https://docs.narrative.io/cookbooks/use-cases/demographic-enrichment

Enrich customer lists with age and gender data using identity matching and automated refresh

A mobile gaming company has rapidly grown to 2 million users but knows very little about who those users are. Since the game doesn't require account registration, they lack basic demographic information needed for effective ad targeting and audience insights. Through Narrative, they can enrich their user base with age and gender data from identity providers, then automate monthly refreshes for new users.

## What you will build

This cookbook walks through creating a demographic enrichment pipeline that:

1. Uploads a seed dataset of mobile advertising IDs (MAIDs) from your customer base
2. Matches those MAIDs against demographic data providers using Rosetta Stone
3. Enriches records with raw age values (not pre-bucketed segments) and gender
4. Automates recurring enrichment for new users added each month

The result is a continuously refreshed customer dataset with demographic attributes that you own outright and can use across any platform.

## Prerequisites

Before starting, ensure you have:

* A dataset of customer identifiers (MAIDs, hashed emails, or other supported identifier types)
* Access to demographic data through data collaboration or marketplace providers
* Familiarity with [NQL syntax](/nql/general/syntax)
* Understanding of [materialized views](/concepts/nql/materialized-views)
* Understanding of [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids)

***

## Step 1: Upload your seed dataset

First, upload your customer identifier file to Narrative. This dataset serves as the seed list you want to enrich.

Your dataset should include at minimum:

* A unique identifier column (MAID, hashed email, etc.)
* An identifier type column if you have mixed identifier types

Example schema for a MAID-based seed file:

| Column             | Type   | Description                                              |
| ------------------ | ------ | -------------------------------------------------------- |
| `maid`             | string | Mobile advertising ID (IDFA or GAID)                     |
| `maid_type`        | string | Identifier type: `idfa` or `gaid`                        |
| `internal_user_id` | string | Your internal user ID for joining back to source systems |
| `first_seen_date`  | date   | When the user first appeared in your system              |

<Note>
  Upload your seed dataset through the Narrative UI or [SDK](/guides/sdk/uploading-data). Map the identifier column to the `unique_identifier` Rosetta Stone attribute for identity matching.
</Note>

### Normalize identifiers with Rosetta Stone

Ensure your identifier column is mapped to Rosetta Stone's `unique_identifier` attribute:

```sql theme={null}
-- Verify your dataset has Rosetta Stone mappings
SELECT
    seed."_rosetta_stone"."unique_identifier"."value" AS maid,
    seed."_rosetta_stone"."unique_identifier"."type" AS maid_type,
    seed.internal_user_id
FROM company_data."your_seed_dataset" seed
LIMIT 10
```

***

## Step 2: Identify available demographic sources

Query the Rosetta Stone to understand which demographic attributes are available across your data collaborations:

```sql theme={null}
-- Preview available demographic data
SELECT
    rs."unique_identifier"."type" AS id_type,
    rs."demographics"."age" AS age,
    rs."demographics"."gender" AS gender,
    COUNT(1) AS record_count
FROM narrative.rosetta_stone rs
WHERE
    rs."demographics"."age" IS NOT NULL
    OR rs."demographics"."gender" IS NOT NULL
GROUP BY
    rs."unique_identifier"."type",
    rs."demographics"."age",
    rs."demographics"."gender"
LIMIT 100
```

This shows you the identifier types and demographic coverage across available providers.

***

## Step 3: Match and enrich with demographics

Create a materialized view that joins your seed dataset with demographic data through Rosetta Stone identity matching.

```sql theme={null}
CREATE MATERIALIZED VIEW "customers_demographics_enriched"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Customer Demographics - Enriched'
DESCRIPTION = 'Seed customer list enriched with age and gender from data providers'
TAGS = ('enrichment', 'demographics', 'audience')
WRITE_MODE = 'overwrite'
AS
SELECT
    seed.internal_user_id,
    seed."_rosetta_stone"."unique_identifier"."value" AS maid,
    seed."_rosetta_stone"."unique_identifier"."type" AS maid_type,
    seed.first_seen_date,
    demo."demographics"."age" AS age,
    demo."demographics"."gender" AS gender,
    demo."_ni_event_timestamp" AS demo_observed_at
FROM company_data."your_seed_dataset" seed
INNER JOIN narrative.rosetta_stone demo
    ON seed."_rosetta_stone"."unique_identifier"."value" = demo."unique_identifier"."value"
    AND seed."_rosetta_stone"."unique_identifier"."type" = demo."unique_identifier"."type"
WHERE
    demo."demographics"."age" IS NOT NULL
    AND demo."demographics"."age" BETWEEN 13 AND 100
    AND demo."event_timestamp" > CURRENT_TIMESTAMP - INTERVAL '180' DAY
```

**Key patterns used:**

| Pattern                                                | Purpose                                           |
| ------------------------------------------------------ | ------------------------------------------------- |
| `INNER JOIN narrative.rosetta_stone`                   | Matches seed identifiers to demographic records   |
| `unique_identifier.value` and `unique_identifier.type` | Ensures exact identifier matching                 |
| `demographics.age BETWEEN 13 AND 100`                  | Filters to valid age ranges                       |
| `INTERVAL '180' DAY`                                   | Uses recent demographic observations for accuracy |
| `REFRESH_SCHEDULE = '@monthly'`                        | Automates recurring enrichment                    |

<Tip>
  Unlike pre-bucketed audience segments (e.g., "25-34"), Rosetta Stone provides raw age values. This gives you flexibility to create your own segments, perform precise analytics, or build custom targeting strategies.
</Tip>

***

## Step 4: Handle multiple demographic observations

A single identifier may have demographic data from multiple sources or observation times. To get the most recent or most common value:

### Most recent observation

```sql theme={null}
CREATE MATERIALIZED VIEW "customers_demographics_latest"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Customer Demographics - Latest Observation'
DESCRIPTION = 'Enriched demographics using most recent observation per customer'
TAGS = ('enrichment', 'demographics', 'deduplicated')
WRITE_MODE = 'overwrite'
AS
SELECT
    internal_user_id,
    maid,
    maid_type,
    first_seen_date,
    age,
    gender,
    demo_observed_at
FROM (
    SELECT
        seed.internal_user_id,
        seed."_rosetta_stone"."unique_identifier"."value" AS maid,
        seed."_rosetta_stone"."unique_identifier"."type" AS maid_type,
        seed.first_seen_date,
        demo."demographics"."age" AS age,
        demo."demographics"."gender" AS gender,
        demo."_ni_event_timestamp" AS demo_observed_at,
        ROW_NUMBER() OVER (
            PARTITION BY seed.internal_user_id
            ORDER BY demo."_ni_event_timestamp" DESC
        ) AS row_num
    FROM company_data."your_seed_dataset" seed
    INNER JOIN narrative.rosetta_stone demo
        ON seed."_rosetta_stone"."unique_identifier"."value" = demo."unique_identifier"."value"
        AND seed."_rosetta_stone"."unique_identifier"."type" = demo."unique_identifier"."type"
    WHERE
        demo."demographics"."age" IS NOT NULL
        AND demo."demographics"."age" BETWEEN 13 AND 100
)
WHERE row_num = 1
```

### Consensus across sources

When you want the most common demographic value across multiple providers:

```sql theme={null}
CREATE MATERIALIZED VIEW "customers_demographics_consensus"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Customer Demographics - Consensus'
DESCRIPTION = 'Enriched demographics using most frequently observed values'
TAGS = ('enrichment', 'demographics', 'consensus')
WRITE_MODE = 'overwrite'
AS
WITH age_consensus AS (
    SELECT
        seed.internal_user_id,
        demo."demographics"."age" AS age,
        COUNT(1) AS age_count,
        ROW_NUMBER() OVER (
            PARTITION BY seed.internal_user_id
            ORDER BY COUNT(1) DESC
        ) AS age_rank
    FROM company_data."your_seed_dataset" seed
    INNER JOIN narrative.rosetta_stone demo
        ON seed."_rosetta_stone"."unique_identifier"."value" = demo."unique_identifier"."value"
        AND seed."_rosetta_stone"."unique_identifier"."type" = demo."unique_identifier"."type"
    WHERE demo."demographics"."age" IS NOT NULL
    GROUP BY seed.internal_user_id, demo."demographics"."age"
),
gender_consensus AS (
    SELECT
        seed.internal_user_id,
        demo."demographics"."gender" AS gender,
        COUNT(1) AS gender_count,
        ROW_NUMBER() OVER (
            PARTITION BY seed.internal_user_id
            ORDER BY COUNT(1) DESC
        ) AS gender_rank
    FROM company_data."your_seed_dataset" seed
    INNER JOIN narrative.rosetta_stone demo
        ON seed."_rosetta_stone"."unique_identifier"."value" = demo."unique_identifier"."value"
        AND seed."_rosetta_stone"."unique_identifier"."type" = demo."unique_identifier"."type"
    WHERE demo."demographics"."gender" IS NOT NULL
    GROUP BY seed.internal_user_id, demo."demographics"."gender"
)
SELECT
    seed.internal_user_id,
    seed."_rosetta_stone"."unique_identifier"."value" AS maid,
    seed."_rosetta_stone"."unique_identifier"."type" AS maid_type,
    ac.age,
    gc.gender
FROM company_data."your_seed_dataset" seed
LEFT JOIN age_consensus ac
    ON seed.internal_user_id = ac.internal_user_id
    AND ac.age_rank = 1
LEFT JOIN gender_consensus gc
    ON seed.internal_user_id = gc.internal_user_id
    AND gc.gender_rank = 1
```

***

## Step 5: Automate incremental enrichment

For growing customer bases, create an incremental view that only processes new users added since the last refresh.

### Create an incremental seed view

First, identify new users not yet in your enriched dataset:

```sql theme={null}
CREATE MATERIALIZED VIEW "customers_new_since_last_enrichment"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'New Customers - Pending Enrichment'
DESCRIPTION = 'Customers added since last demographic enrichment'
TAGS = ('enrichment', 'incremental', 'staging')
WRITE_MODE = 'overwrite'
AS
SELECT
    seed.internal_user_id,
    seed."_rosetta_stone"."unique_identifier"."value" AS maid,
    seed."_rosetta_stone"."unique_identifier"."type" AS maid_type,
    seed.first_seen_date
FROM company_data."your_seed_dataset" seed
WHERE NOT EXISTS (
    SELECT 1
    FROM company_data."customers_demographics_enriched" enriched
    WHERE seed.internal_user_id = enriched.internal_user_id
)
```

### Enrich and merge new users

```sql theme={null}
CREATE MATERIALIZED VIEW "customers_demographics_incremental"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Customer Demographics - Incremental Update'
DESCRIPTION = 'Demographic enrichment for newly added customers'
TAGS = ('enrichment', 'demographics', 'incremental')
WRITE_MODE = 'append'
AS
SELECT
    new_users.internal_user_id,
    new_users.maid,
    new_users.maid_type,
    new_users.first_seen_date,
    demo."demographics"."age" AS age,
    demo."demographics"."gender" AS gender,
    demo."_ni_event_timestamp" AS demo_observed_at
FROM company_data."customers_new_since_last_enrichment" new_users
INNER JOIN narrative.rosetta_stone demo
    ON new_users.maid = demo."unique_identifier"."value"
    AND new_users.maid_type = demo."unique_identifier"."type"
WHERE
    demo."demographics"."age" IS NOT NULL
    AND demo."demographics"."age" BETWEEN 13 AND 100
```

<Note>
  Use `WRITE_MODE = 'append'` for incremental views that add new records to an existing dataset without overwriting historical data.
</Note>

***

## Step 6: Create audience segments

With enriched demographics, create targetable audience segments:

### Age-based segments

```sql theme={null}
CREATE MATERIALIZED VIEW "audience_segment_age_buckets"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Audience Segments - Age Buckets'
DESCRIPTION = 'Customers segmented by age ranges for targeting'
TAGS = ('audience', 'segmentation', 'age')
WRITE_MODE = 'overwrite'
AS
SELECT
    internal_user_id,
    maid,
    maid_type,
    age,
    gender,
    CASE
        WHEN age BETWEEN 18 AND 24 THEN '18-24'
        WHEN age BETWEEN 25 AND 34 THEN '25-34'
        WHEN age BETWEEN 35 AND 44 THEN '35-44'
        WHEN age BETWEEN 45 AND 54 THEN '45-54'
        WHEN age >= 55 THEN '55+'
        ELSE 'unknown'
    END AS age_segment
FROM company_data."customers_demographics_enriched"
WHERE age IS NOT NULL
```

### Gender-specific segments

```sql theme={null}
CREATE MATERIALIZED VIEW "audience_female_25_44"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Audience - Female 25-44'
DESCRIPTION = 'Female customers aged 25-44'
TAGS = ('audience', 'segmentation', 'female', '25-44')
WRITE_MODE = 'overwrite'
AS
SELECT
    internal_user_id,
    maid,
    maid_type
FROM company_data."customers_demographics_enriched"
WHERE
    gender = 'female'
    AND age BETWEEN 25 AND 44
```

***

## Summary

You have built a demographic enrichment pipeline with these components:

| View                                  | Purpose                                    |
| ------------------------------------- | ------------------------------------------ |
| `customers_demographics_enriched`     | Base enrichment with age and gender        |
| `customers_demographics_latest`       | Deduplicated using most recent observation |
| `customers_demographics_consensus`    | Deduplicated using most common values      |
| `customers_new_since_last_enrichment` | Identifies users pending enrichment        |
| `customers_demographics_incremental`  | Processes only new users                   |
| `audience_segment_age_buckets`        | Age-based targeting segments               |

### Use cases

* **Media targeting:** Reach specific demographic segments with relevant creative
* **Audience analytics:** Understand the composition of your customer base
* **Lookalike modeling:** Build seed audiences based on demographic profiles
* **Personalization:** Tailor app experiences or content by age group
* **Attribution:** Analyze conversion rates across demographic segments

### Key advantages

<CardGroup>
  <Card title="Raw values, not buckets" icon="hashtag">
    Receive exact age values instead of pre-defined segments, giving you flexibility to create custom ranges
  </Card>

  <Card title="Omni-use license" icon="infinity">
    Data is enriched once and can be used across any platform or use case without per-use fees
  </Card>

  <Card title="Automated refresh" icon="arrows-rotate">
    Monthly schedules keep your enriched data current as new users join
  </Card>

  <Card title="Transparent sourcing" icon="eye">
    See which providers contribute demographic data and evaluate match quality
  </Card>
</CardGroup>

### Important considerations

<Warning>
  **MAID availability:** iOS App Tracking Transparency and Android privacy changes have reduced MAID availability. Consider supplementing with hashed email or other identifier types for broader coverage. See [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids) for details.
</Warning>

<Note>
  **Privacy compliance:** Demographic data may be subject to GDPR, CCPA, and other privacy regulations depending on source and use case. Ensure your enrichment workflow complies with applicable requirements. See [CCPA compliance](/concepts/compliance/ccpa) and [GDPR compliance](/concepts/compliance/gdpr) for details.
</Note>

***

## Related content

<CardGroup>
  <Card title="Joining Datasets" icon="link" href="/guides/nql/joining-datasets">
    Learn JOIN patterns for enrichment queries
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to automated views
  </Card>

  <Card title="Mobile Ad IDs" icon="mobile" href="/concepts/identifiers/mobile-ad-ids">
    Understanding IDFA, GAID, and privacy considerations
  </Card>

  <Card title="Data Onboarding" icon="upload" href="/concepts/identifiers/data-onboarding">
    Connecting offline data to digital identifiers
  </Card>
</CardGroup>


# Evaluating Identity Data Providers
Source: https://docs.narrative.io/cookbooks/use-cases/identity-provider-evaluation

A structured framework for choosing between graph enrichment and addressability expansion providers and measuring their incremental value

An organization maturing its identity strategy faces a common challenge: dozens of third-party identity providers offer overlapping capabilities, and it is not always clear which providers deliver incremental value versus redundant coverage. This cookbook provides a structured evaluation process for choosing between [graph enrichment](/concepts/identifiers/graph-enrichment) and [addressability expansion](/concepts/identifiers/addressability) providers and measuring their contribution.

## What this cookbook covers

This cookbook walks through a five-step evaluation process:

1. Define whether you need graph enrichment, addressability expansion, or both
2. Baseline your current identity graph and match rates
3. Select providers for proof-of-concept testing
4. Measure each provider's incremental contribution
5. Evaluate commercial terms and build a layered strategy

The result is a repeatable framework you can apply each time you evaluate a new provider or renegotiate an existing relationship.

## Prerequisites

Before starting, ensure you have:

* Familiarity with [identity graph concepts](/concepts/identifiers/identity-graph)
* Understanding of [graph enrichment](/concepts/identifiers/graph-enrichment) and [addressability expansion](/concepts/identifiers/addressability)
* A first-party dataset with identifier mappings in Narrative
* Familiarity with [NQL syntax](/nql/general/syntax)
* Understanding of [materialized views](/concepts/nql/materialized-views)

***

## Step 1: Define your primary objective

Start by identifying which problem you are solving:

**Choose graph enrichment if:**

* Your audience segments feel fragmented—the same customers appear as multiple profiles
* Your frequency capping is ineffective because you cannot recognize users across devices
* Your attribution is unreliable because conversion paths break across identifier boundaries
* You need to improve person-level or household-level resolution before activation

**Choose addressability expansion if:**

* Your segmentation is sound but match rates on activation platforms are low
* You have strong first-party identity but limited identifier type diversity
* You are expanding to new activation channels (CTV, programmatic audio, digital out-of-home) that use identifier types you do not currently have
* You need immediate lift in campaign reach without changing your targeting logic

**Choose both if:**

* You have fragmented profiles (graph enrichment problem) AND low platform match rates (addressability problem)
* You are building identity infrastructure for the first time and need both resolution and reach

Most organizations eventually need both, but the sequencing matters. Graph enrichment typically comes first because better resolution improves the accuracy of everything downstream, including addressability expansion.

***

## Step 2: Baseline your current state

Before evaluating any provider, measure where you stand today.

### For graph enrichment: measure graph structure

```sql theme={null}
-- Count connected component sizes in your first-party graph
SELECT
    component_size,
    COUNT(1) AS component_count,
    SUM(component_size) AS total_identifiers
FROM (
    SELECT
        seed.internal_user_id,
        COUNT(DISTINCT seed."_rosetta_stone"."unique_identifier"."value") AS component_size
    FROM company_data."first_party_graph" seed
    GROUP BY seed.internal_user_id
)
GROUP BY component_size
ORDER BY component_size
```

Key metrics to record:

| Metric                           | What it tells you                                               |
| -------------------------------- | --------------------------------------------------------------- |
| **Singleton percentage**         | Share of profiles with only one identifier—these are unresolved |
| **Average component size**       | How many identifiers per resolved profile                       |
| **Deterministic coverage**       | Share of linkages based on direct observation vs. inference     |
| **Identifier type distribution** | Which identifier types you have and which you lack              |

### For addressability expansion: measure match rates

```sql theme={null}
-- Count identifiers by type in your first-party data
SELECT
    seed."_rosetta_stone"."unique_identifier"."type" AS id_type,
    COUNT(DISTINCT seed."_rosetta_stone"."unique_identifier"."value") AS unique_ids,
    COUNT(DISTINCT seed.internal_user_id) AS unique_users
FROM company_data."crm_seed" seed
GROUP BY seed."_rosetta_stone"."unique_identifier"."type"
```

Record your current match rates on each activation platform you use. These become the baseline against which you measure provider lift.

***

## Step 3: Select providers for proof of concept

### For graph enrichment providers

Evaluate candidates on:

* **Linkage methodology.** Deterministic linkages from login events are more reliable than probabilistic linkages from behavioral signals. Understand each provider's methodology mix.
* **Update cadence.** Identity data has a shelf life. Providers that refresh linkages monthly are more valuable than those refreshing quarterly.
* **Incremental coverage.** A provider whose linkages overlap 90% with your existing graph adds less value than one with 40% overlap and 60% net new edges.

### For addressability expansion providers

Evaluate candidates on:

* **Identifier type coverage.** If you need MAIDs for programmatic and hashed phones for social, ensure the provider covers both.
* **Platform-specific match rates.** Ask providers for benchmark match rates on the specific platforms you activate on.
* **Freshness guarantees.** Appended identifiers that are six months old will have significantly lower match rates than those observed within the last 30 days.

### How to evaluate incremental vs. overlapping coverage

Use NQL to measure overlap between a candidate provider and your existing data:

```sql theme={null}
-- Measure identifier overlap with a candidate provider
SELECT
    provider."_rosetta_stone"."unique_identifier"."type" AS id_type,
    COUNT(DISTINCT provider."_rosetta_stone"."unique_identifier"."value") AS provider_total,
    COUNT(DISTINCT CASE
        WHEN seed."_rosetta_stone"."unique_identifier"."value" IS NOT NULL
        THEN provider."_rosetta_stone"."unique_identifier"."value"
    END) AS overlapping,
    COUNT(DISTINCT CASE
        WHEN seed."_rosetta_stone"."unique_identifier"."value" IS NULL
        THEN provider."_rosetta_stone"."unique_identifier"."value"
    END) AS net_new
FROM external_provider."candidate_identity_data" provider
LEFT JOIN company_data."first_party_graph" seed
    ON provider."_rosetta_stone"."unique_identifier"."value" = seed."_rosetta_stone"."unique_identifier"."value"
    AND provider."_rosetta_stone"."unique_identifier"."type" = seed."_rosetta_stone"."unique_identifier"."type"
GROUP BY provider."_rosetta_stone"."unique_identifier"."type"
```

***

## Step 4: Measure incremental contribution

Run a time-boxed proof of concept with each shortlisted provider and measure specific outcomes.

### For graph enrichment: measure structural improvement

| Metric                    | How to calculate                                           | What good looks like                                   |
| ------------------------- | ---------------------------------------------------------- | ------------------------------------------------------ |
| **Net new edges**         | Count linkages the provider adds that you did not have     | Higher is better, but quality matters more than volume |
| **Component size change** | Compare average component size before and after            | Modest, consistent growth (not sudden large merges)    |
| **Singleton reduction**   | Change in percentage of single-identifier profiles         | Direct measure of resolution improvement               |
| **Cross-device reach**    | Percentage of profiles with identifiers on 2+ device types | Measures practical multi-device capability             |

### For addressability expansion: measure activation lift

| Metric                            | How to calculate                                                      | What good looks like                        |
| --------------------------------- | --------------------------------------------------------------------- | ------------------------------------------- |
| **Match rate lift by platform**   | (New match rate - baseline match rate) / baseline match rate          | Varies by platform; 20%+ lift is meaningful |
| **Incremental addressable users** | Users matchable after expansion minus users matchable before          | The absolute reach improvement              |
| **Cost per incremental match**    | Provider cost / incremental matched users                             | Compare across providers for efficiency     |
| **Identifier freshness**          | Percentage of appended identifiers observed within last 30/60/90 days | Fresher identifiers match at higher rates   |

### Measuring before and after in Narrative

Create materialized views that capture your baseline and post-enrichment states:

```sql theme={null}
-- Baseline: unique users matchable by identifier type
CREATE MATERIALIZED VIEW "identity_baseline"
DISPLAY_NAME = 'Identity Baseline - Pre-Enrichment'
DESCRIPTION = 'Baseline identifier coverage before provider enrichment'
TAGS = ('identity', 'evaluation', 'baseline')
WRITE_MODE = 'overwrite'
AS
SELECT
    seed."_rosetta_stone"."unique_identifier"."type" AS id_type,
    COUNT(DISTINCT seed.internal_user_id) AS unique_users,
    COUNT(DISTINCT seed."_rosetta_stone"."unique_identifier"."value") AS unique_ids
FROM company_data."crm_seed" seed
GROUP BY seed."_rosetta_stone"."unique_identifier"."type"
```

```sql theme={null}
-- Post-enrichment: unique users matchable after provider data is incorporated
CREATE MATERIALIZED VIEW "identity_post_enrichment"
DISPLAY_NAME = 'Identity Post-Enrichment - Provider A'
DESCRIPTION = 'Identifier coverage after incorporating Provider A data'
TAGS = ('identity', 'evaluation', 'post-enrichment')
WRITE_MODE = 'overwrite'
AS
SELECT
    combined_id_type AS id_type,
    COUNT(DISTINCT internal_user_id) AS unique_users,
    COUNT(DISTINCT combined_id_value) AS unique_ids
FROM (
    SELECT
        seed.internal_user_id,
        seed."_rosetta_stone"."unique_identifier"."type" AS combined_id_type,
        seed."_rosetta_stone"."unique_identifier"."value" AS combined_id_value
    FROM company_data."crm_seed" seed
    UNION
    SELECT
        seed.internal_user_id,
        provider."_rosetta_stone"."unique_identifier"."type" AS combined_id_type,
        provider."_rosetta_stone"."unique_identifier"."value" AS combined_id_value
    FROM company_data."crm_seed" seed
    INNER JOIN external_provider."provider_a_data" provider
        ON seed."_rosetta_stone"."unique_identifier"."value" = provider."_rosetta_stone"."unique_identifier"."value"
        AND seed."_rosetta_stone"."unique_identifier"."type" = provider."_rosetta_stone"."unique_identifier"."type"
)
GROUP BY combined_id_type
```

***

## Step 5: Evaluate commercial terms

Identity data pricing varies significantly. Consider:

| Factor                    | Graph enrichment considerations                                                                       | Addressability expansion considerations                                     |
| ------------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Pricing model**         | Flat licensing or per-linkage pricing; understand cost per incremental edge                           | CPM-based or per-record pricing; understand cost per incremental match      |
| **Use case restrictions** | Some providers restrict derived-data usage or require separate licensing for analytics vs. activation | Some providers restrict which platforms you can activate on                 |
| **Refresh costs**         | Ongoing costs for linkage updates; stale linkages degrade value                                       | Ongoing costs for identifier refresh; churned identifiers waste spend       |
| **Data sovereignty**      | Where linkage data is stored and processed; relevant for GDPR/CCPA compliance                         | Whether appended identifiers can be stored or must be used in-flight        |
| **Pass-through rights**   | Whether you can use enriched profiles with downstream partners                                        | Whether expanded identifiers can be shared with agency or platform partners |

***

## The layered identity strategy

Most mature identity strategies follow a phased approach:

### Phase 1: Graph foundation

Build your first-party graph with deterministic linkages from login events, CRM data, and transaction records. This is your highest-confidence data and forms the foundation for everything else.

### Phase 2: Addressability expansion

Once your graph foundation is solid, expand addressability to improve activation reach. This delivers immediate, measurable ROI through higher match rates and broader campaign reach.

### Phase 3: Graph enrichment

With addressability established, invest in graph enrichment to improve structural resolution. Better resolution improves segmentation accuracy, frequency capping, and attribution—benefits that compound over time.

### Phase 4: Continuous measurement

Identity is not a one-time project. Establish ongoing measurement of graph quality, match rates, and provider contribution. Re-evaluate providers annually as the identity landscape shifts with privacy regulations, platform changes, and new identifier standards.

***

## Summary

| Decision point               | Key question                                                    | Primary metric                                          |
| ---------------------------- | --------------------------------------------------------------- | ------------------------------------------------------- |
| **Enrichment vs. expansion** | Is your problem fragmented profiles or low match rates?         | Singleton % vs. platform match rates                    |
| **Provider selection**       | Does this provider add incremental value or redundant coverage? | Net new edges or net new matched users                  |
| **Quality validation**       | Are new linkages or identifiers reliable?                       | Component size distribution or identifier freshness     |
| **Commercial fit**           | Does the pricing align with the incremental value delivered?    | Cost per incremental edge or cost per incremental match |

***

## Related content

<CardGroup>
  <Card title="Identity Graphs" icon="diagram-project" href="/concepts/identifiers/identity-graph">
    Core concepts behind identity graph structure
  </Card>

  <Card title="Graph Enrichment" icon="circle-nodes" href="/concepts/identifiers/graph-enrichment">
    Strengthening graph structure with third-party linkage data
  </Card>

  <Card title="Addressability Expansion" icon="bullseye-arrow" href="/concepts/identifiers/addressability">
    Improving activation reach by appending identifiers
  </Card>

  <Card title="Data Activation" icon="rocket" href="/concepts/data-activation/overview">
    Delivering audiences to destination platforms
  </Card>

  <Card title="Demographic Enrichment" icon="users" href="/cookbooks/use-cases/demographic-enrichment">
    A related pattern: enriching records with demographic attributes
  </Card>
</CardGroup>


# IP Address to MAID Enrichment
Source: https://docs.narrative.io/cookbooks/use-cases/maid-enrichment

Resolve IP addresses to mobile advertising IDs using Rosetta Stone for cross-device targeting and mobile campaign activation

A company has server logs or web analytics data containing IP addresses, but needs mobile advertising IDs (MAIDs) to activate audiences on mobile channels. By joining their IP dataset against Rosetta Stone, they can resolve IPs to MAIDs observed on the same network, rank them by observation frequency, and build targetable mobile audiences—all without querying individual data providers.

## What you will build

This cookbook walks through creating an IP-to-MAID enrichment pipeline that:

1. Uploads a dataset of IP addresses from your server logs or web analytics
2. Previews MAID data availability by type (IDFA vs GAID) across Rosetta Stone
3. Matches IPs to MAIDs using an identity join against Rosetta Stone
4. Ranks MAIDs by observation frequency to surface the most reliable matches
5. Automates recurring enrichment with a materialized view

The result is a continuously refreshed dataset that maps your IP addresses to mobile advertising IDs for cross-device targeting.

## Prerequisites

Before starting, ensure you have:

* A dataset of IP addresses uploaded to Narrative
* Familiarity with [NQL syntax](/nql/general/syntax)
* Understanding of [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids) (IDFA, GAID)
* Understanding of [IP addresses](/concepts/data-formats/ip-addresses) and their limitations
* Understanding of [materialized views](/concepts/nql/materialized-views)

***

## Step 1: Upload your IP address dataset

Upload your IP address file to Narrative. This dataset serves as the seed list you want to enrich with MAIDs.

Your dataset should include at minimum:

| Column        | Type      | Description                             |
| ------------- | --------- | --------------------------------------- |
| `ip`          | string    | IPv4 or IPv6 address                    |
| `observed_at` | timestamp | When the IP was observed in your system |

You may also include additional context columns (session ID, page URL, etc.) that you want to carry through to the enriched output.

<Note>
  Upload your dataset through the Narrative UI or [SDK](/guides/sdk/uploading-data). The `ip` column does not need a Rosetta Stone mapping—you will join it directly against the `"ip_address"` attribute in Rosetta Stone.
</Note>

***

## Step 2: Preview IP-to-MAID data availability

Before running the full enrichment, query Rosetta Stone to understand how much MAID coverage is available by identifier type:

```sql theme={null}
SELECT
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    APPROX_COUNT_DISTINCT(rs."ip_address") AS unique_ips,
    APPROX_COUNT_DISTINCT(rs."mobile_id_unique_identifier"."value") AS unique_maids
FROM narrative.rosetta_stone rs
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
GROUP BY rs."mobile_id_unique_identifier"."type"
```

This shows you the relative volume of IDFA vs GAID data and helps set expectations for match rates.

<Tip>
  GAID (Android) typically has higher volume than IDFA (iOS) due to Apple's App Tracking Transparency framework. See [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids) for details on availability trends.
</Tip>

***

## Step 3: Match IPs to MAIDs

Join your IP dataset against Rosetta Stone to find MAIDs observed on the same IP addresses. This is the core enrichment query:

```sql theme={null}
SELECT
    cd."ip",
    rs."mobile_id_unique_identifier"."value" AS maid,
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    rs."iso_3166_1_country" AS country_code,
    rs."event_timestamp" AS observed_at
FROM narrative.rosetta_stone rs
INNER JOIN company_data."your_ip_dataset" cd
    ON rs."ip_address" = cd."ip"
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
```

**Key patterns used:**

| Pattern                                       | Purpose                                         |
| --------------------------------------------- | ----------------------------------------------- |
| `INNER JOIN company_data`                     | Matches only IPs present in your seed dataset   |
| `mobile_id_unique_identifier.value` / `.type` | Extracts MAID value and type (IDFA or GAID)     |
| `ip_address`                                  | Rosetta Stone's normalized IP address attribute |
| `INTERVAL '45' DAY`                           | Limits to recent observations for freshness     |

<Note>
  A single IP address may resolve to many MAIDs—especially shared or commercial IPs. Step 4 shows how to rank and limit results to the most reliable matches.
</Note>

***

## Step 4: Rank MAIDs by observation frequency

A single IP can map to many devices, and not all associations are equally reliable. Use `QUALIFY` with `ROW_NUMBER` and a nested `APPROX_COUNT_DISTINCT` window function to keep only the top N most-frequently-observed MAIDs per IP:

```sql theme={null}
SELECT
    cd."ip",
    rs."mobile_id_unique_identifier"."value" AS maid,
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    rs."iso_3166_1_country" AS country_code
FROM narrative.rosetta_stone rs
INNER JOIN company_data."your_ip_dataset" cd
    ON rs."ip_address" = cd."ip"
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY rs."iso_3166_1_country", cd."ip"
    ORDER BY APPROX_COUNT_DISTINCT(rs."event_timestamp") OVER (
        PARTITION BY rs."iso_3166_1_country", cd."ip",
                     rs."mobile_id_unique_identifier"."value",
                     rs."mobile_id_unique_identifier"."type"
    ) DESC
) <= 3
```

**How the nested window functions work:**

1. The **inner window** (`APPROX_COUNT_DISTINCT ... OVER (PARTITION BY country, ip, maid, maid_type)`) counts distinct observation timestamps for each IP-MAID pair. MAIDs seen more often on an IP address get a higher count.
2. The **outer window** (`ROW_NUMBER ... OVER (PARTITION BY country, ip ORDER BY ... DESC)`) ranks MAIDs within each country + IP group by that observation count, highest first.
3. `QUALIFY ... <= 3` keeps only the top 3 most-frequently-observed MAIDs per IP per country.

This approach prioritizes devices that consistently appear on an IP address, filtering out transient connections.

<Tip>
  Adjust the `<= 3` threshold based on your use case. For residential IPs, 1-3 MAIDs is typical. For commercial or CGNAT IPs shared across many users, consider filtering these out entirely or increasing the threshold.
</Tip>

***

## Step 5: Add country filtering and enrichment stats

Extend the query with geographic filtering and a `maid_count` column that shows the total unique MAIDs found per IP before ranking. This helps you assess match quality:

```sql theme={null}
SELECT
    cd."ip",
    rs."mobile_id_unique_identifier"."value" AS maid,
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    rs."iso_3166_1_country" AS country_code,
    APPROX_COUNT_DISTINCT(rs."mobile_id_unique_identifier"."value") OVER (
        PARTITION BY cd."ip"
    ) AS maid_count
FROM narrative.rosetta_stone rs
INNER JOIN company_data."your_ip_dataset" cd
    ON rs."ip_address" = cd."ip"
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
    AND rs."iso_3166_1_country" IN ('US', 'CA', 'GB', 'AU')
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY rs."iso_3166_1_country", cd."ip"
    ORDER BY APPROX_COUNT_DISTINCT(rs."event_timestamp") OVER (
        PARTITION BY rs."iso_3166_1_country", cd."ip",
                     rs."mobile_id_unique_identifier"."value",
                     rs."mobile_id_unique_identifier"."type"
    ) DESC
) <= 3
```

The `maid_count` column reveals IP quality:

* **Low count (1-5):** Likely residential IPs with a small number of devices—high-confidence matches
* **High count (50+):** Likely commercial, VPN, or CGNAT IPs—consider excluding these from activation

### MAID type breakdown variation

To see the split between IDFA and GAID per IP, replace the `maid_count` window with a type-specific count:

```sql theme={null}
SELECT
    cd."ip",
    rs."mobile_id_unique_identifier"."value" AS maid,
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    rs."iso_3166_1_country" AS country_code,
    APPROX_COUNT_DISTINCT(
        CASE WHEN rs."mobile_id_unique_identifier"."type" = 'idfa' THEN rs."mobile_id_unique_identifier"."value" END
    ) OVER (PARTITION BY cd."ip") AS idfa_count,
    APPROX_COUNT_DISTINCT(
        CASE WHEN rs."mobile_id_unique_identifier"."type" = 'gaid' THEN rs."mobile_id_unique_identifier"."value" END
    ) OVER (PARTITION BY cd."ip") AS gaid_count
FROM narrative.rosetta_stone rs
INNER JOIN company_data."your_ip_dataset" cd
    ON rs."ip_address" = cd."ip"
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
    AND rs."iso_3166_1_country" IN ('US', 'CA', 'GB', 'AU')
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY rs."iso_3166_1_country", cd."ip"
    ORDER BY APPROX_COUNT_DISTINCT(rs."event_timestamp") OVER (
        PARTITION BY rs."iso_3166_1_country", cd."ip",
                     rs."mobile_id_unique_identifier"."value",
                     rs."mobile_id_unique_identifier"."type"
    ) DESC
) <= 3
```

***

## Step 6: Automate with a materialized view

Wrap the final query in a materialized view that refreshes automatically:

```sql theme={null}
CREATE MATERIALIZED VIEW "ip_to_maid_enriched"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'IP to MAID Enrichment'
DESCRIPTION = 'IP addresses enriched with top mobile ad IDs ranked by observation frequency'
TAGS = ('enrichment', 'ip', 'maid', 'cross-device')
WRITE_MODE = 'overwrite'
AS
SELECT
    cd."ip",
    rs."mobile_id_unique_identifier"."value" AS maid,
    rs."mobile_id_unique_identifier"."type" AS maid_type,
    rs."iso_3166_1_country" AS country_code,
    APPROX_COUNT_DISTINCT(rs."mobile_id_unique_identifier"."value") OVER (
        PARTITION BY cd."ip"
    ) AS maid_count
FROM narrative.rosetta_stone rs
INNER JOIN company_data."your_ip_dataset" cd
    ON rs."ip_address" = cd."ip"
WHERE
    rs."mobile_id_unique_identifier"."value" IS NOT NULL
    AND rs."ip_address" IS NOT NULL
    AND rs."event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '45' DAY
    AND rs."iso_3166_1_country" IN ('US', 'CA', 'GB', 'AU')
QUALIFY ROW_NUMBER() OVER (
    PARTITION BY rs."iso_3166_1_country", cd."ip"
    ORDER BY APPROX_COUNT_DISTINCT(rs."event_timestamp") OVER (
        PARTITION BY rs."iso_3166_1_country", cd."ip",
                     rs."mobile_id_unique_identifier"."value",
                     rs."mobile_id_unique_identifier"."type"
    ) DESC
) <= 3
```

<Note>
  Adjust the `REFRESH_SCHEDULE` based on how frequently your IP dataset changes. `@weekly` works well for server logs that update regularly. Use `@monthly` if your IP list is relatively static.
</Note>

***

## Summary

You have built an IP-to-MAID enrichment pipeline with these components:

| Query / View              | Purpose                                                                |
| ------------------------- | ---------------------------------------------------------------------- |
| Data availability preview | Assess MAID volume and type coverage before enrichment                 |
| IP-to-MAID join           | Core matching of IPs to mobile ad IDs via Rosetta Stone                |
| Frequency-ranked results  | Top N MAIDs per IP based on observation count                          |
| `ip_to_maid_enriched`     | Automated materialized view with country filtering and quality metrics |

### Use cases

* **Cross-device targeting:** Reach users on mobile who were previously only identifiable by IP
* **Mobile campaign activation:** Build MAID-based audiences from web traffic data for DSP activation
* **Device graph enrichment:** Augment your identity graph with IP-to-MAID linkages
* **Measurement and attribution:** Connect web sessions (IP) to mobile app activity (MAID)

### Key advantages

<CardGroup>
  <Card title="Simplified sourcing" icon="wand-magic-sparkles">
    Rosetta Stone queries all available data providers automatically—no need to identify and UNION across individual datasets
  </Card>

  <Card title="Frequency-based ranking" icon="arrow-down-1-9">
    Nested window functions surface the most reliable IP-MAID associations, filtering out transient connections
  </Card>

  <Card title="Automated refresh" icon="arrows-rotate">
    Weekly materialized views keep your MAID mappings current as new observations arrive
  </Card>

  <Card title="Quality signals" icon="chart-simple">
    The `maid_count` column helps identify commercial or shared IPs that may produce low-quality matches
  </Card>
</CardGroup>

### Important considerations

<Warning>
  **MAID availability:** iOS App Tracking Transparency and Android privacy changes have reduced MAID availability. GAID volume is significantly higher than IDFA. Consider supplementing with hashed email or other identifier types for broader iOS coverage. See [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids) for details.
</Warning>

<Note>
  **IP addresses as personal data:** Under GDPR, IP addresses are considered personal data. Ensure your enrichment workflow complies with applicable privacy regulations, particularly when processing IPs from EU users. See [GDPR compliance](/concepts/compliance/gdpr) and [CCPA compliance](/concepts/compliance/ccpa) for details.
</Note>

***

## Related content

<CardGroup>
  <Card title="Mobile Ad IDs" icon="mobile" href="/concepts/identifiers/mobile-ad-ids">
    Understanding IDFA, GAID, and privacy considerations
  </Card>

  <Card title="IP Addresses" icon="globe" href="/concepts/data-formats/ip-addresses">
    IP address types, limitations, and best practices
  </Card>

  <Card title="Joining Datasets" icon="link" href="/guides/nql/joining-datasets">
    Learn JOIN patterns for enrichment queries
  </Card>

  <Card title="Performance Patterns" icon="gauge-high" href="/cookbooks/nql/performance-patterns">
    Window functions, QUALIFY, and other advanced NQL patterns
  </Card>
</CardGroup>


# Converting Natural Language to CRON
Source: https://docs.narrative.io/cookbooks/use-cases/natural-language-to-cron

Use Model Inference to translate human-readable schedules into CRON expressions

This cookbook demonstrates how to use Model Inference to convert natural language schedule descriptions into valid CRON expressions. This is useful for materialized view refresh scheduling, report generation, and any feature where users need to specify schedules without knowing CRON syntax.

## What this recipe accomplishes

* Accept natural language schedule descriptions
* Convert them to valid CRON expressions
* Provide human-readable confirmation of the schedule
* Handle ambiguous inputs with clarification

## Prerequisites

* SDK installed and configured (see [Authentication](/guides/sdk/authentication))
* A data plane ID where inference will run

## Complete example

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

// Types for CRON conversion
interface CronConversionResult {
  cron_expression: string;
  human_readable: string;
  timezone_note: string;
  next_runs: string[];
  confidence: number;
  ambiguity_warning?: string;
}

// Configuration
const CONFIG = {
  dataPlaneId: process.env.DATA_PLANE_ID!,
  model: 'anthropic.claude-sonnet-4.5' as const,
};

// Initialize API
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY!,
});

// Schema for CRON conversion output
const cronSchema = {
  type: 'object',
  properties: {
    cron_expression: {
      type: 'string',
      description: 'Valid CRON expression (5 fields: minute hour day month weekday)',
    },
    human_readable: {
      type: 'string',
      description: 'Human-readable description of when this schedule runs',
    },
    timezone_note: {
      type: 'string',
      description: 'Note about timezone assumptions',
    },
    next_runs: {
      type: 'array',
      items: { type: 'string' },
      minItems: 3,
      maxItems: 5,
      description: 'Next 3-5 execution times in ISO format',
    },
    confidence: {
      type: 'number',
      minimum: 0,
      maximum: 1,
      description: 'Confidence in the interpretation (0-1)',
    },
    ambiguity_warning: {
      type: 'string',
      description: 'Warning if the input was ambiguous',
    },
  },
  required: ['cron_expression', 'human_readable', 'timezone_note', 'next_runs', 'confidence'],
};

/**
 * Wait for inference job to complete
 */
async function waitForInference(jobId: string): Promise<CronConversionResult | null> {
  const maxWaitMs = 30000;
  const startTime = Date.now();
  const pollInterval = 2000;

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    if (job.state === 'completed' && job.result) {
      return job.result.structured_output as CronConversionResult;
    }

    if (job.state === 'failed') {
      console.error('Inference job failed:', job.failures);
      return null;
    }

    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`Job ${jobId} timed out`);
}

/**
 * Convert natural language to CRON expression
 */
async function convertToCron(
  naturalLanguage: string,
  timezone = 'UTC'
): Promise<CronConversionResult | null> {
  const currentDate = new Date().toISOString();

  const prompt = `Convert this schedule description to a CRON expression:

"${naturalLanguage}"

Current date/time: ${currentDate}
Assumed timezone: ${timezone}

Requirements:
1. Use standard 5-field CRON format: minute hour day-of-month month day-of-week
2. Use * for "every", specific numbers for fixed times
3. For "weekdays", use 1-5. For "weekends", use 0,6
4. Calculate the next 3-5 run times based on the current date
5. If the input is ambiguous, note this and make a reasonable assumption

Common patterns:
- "every day at 9am" -> 0 9 * * *
- "every Monday at 8am" -> 0 8 * * 1
- "every hour" -> 0 * * * *
- "every 15 minutes" -> */15 * * * *
- "first of every month at midnight" -> 0 0 1 * *
- "weekdays at 6pm" -> 0 18 * * 1-5`;

  const job = await api.runModelInference({
    data_plane_id: CONFIG.dataPlaneId,
    model: CONFIG.model,
    messages: [
      {
        role: 'system',
        content: [
          {
            type: 'text',
            text: `You are an expert at CRON expressions. Convert natural language schedule
descriptions into valid CRON expressions. Be precise and handle edge cases carefully.
Always use 5-field CRON format (minute hour day month weekday).`,
          },
        ],
      },
      {
        role: 'user',
        content: [{ type: 'text', text: prompt }],
      },
    ],
    inference_config: {
      output_format_schema: cronSchema,
      temperature: 0.1, // Low temperature for consistent, precise output
    },
    tags: ['cron-conversion'],
  });

  return await waitForInference(job.id);
}

/**
 * Validate a CRON expression format
 */
function validateCronFormat(cron: string): { valid: boolean; error?: string } {
  const fields = cron.trim().split(/\s+/);

  if (fields.length !== 5) {
    return { valid: false, error: `Expected 5 fields, got ${fields.length}` };
  }

  const patterns = [
    /^(\*|[0-5]?\d)(\/(([0-5]?\d)))?$|^(\*|[0-5]?\d)(,([0-5]?\d))*$|^([0-5]?\d)-([0-5]?\d)$/, // minute
    /^(\*|[01]?\d|2[0-3])(\/(([01]?\d|2[0-3])))?$|^(\*|[01]?\d|2[0-3])(,([01]?\d|2[0-3]))*$|^([01]?\d|2[0-3])-([01]?\d|2[0-3])$/, // hour
    /^(\*|\?|[1-9]|[12]\d|3[01])$/, // day of month
    /^(\*|[1-9]|1[0-2])$/, // month
    /^(\*|\?|[0-6])(,[0-6])*$|^[0-6]-[0-6]$/, // day of week
  ];

  for (let i = 0; i < 5; i++) {
    // Basic check - just verify non-empty for now
    if (!fields[i] || fields[i].length === 0) {
      return { valid: false, error: `Field ${i + 1} is empty` };
    }
  }

  return { valid: true };
}

/**
 * Interactive CRON converter
 */
async function interactiveConverter(): Promise<void> {
  const testSchedules = [
    'every day at 9am',
    'every Monday and Thursday at 2:30pm',
    'first of every month at midnight',
    'every 15 minutes during business hours on weekdays',
    'every Sunday at 6pm',
    'twice a day at 8am and 8pm',
  ];

  console.log('=== Natural Language to CRON Converter ===\n');

  for (const schedule of testSchedules) {
    console.log(`Input: "${schedule}"`);

    const result = await convertToCron(schedule);

    if (result) {
      const validation = validateCronFormat(result.cron_expression);

      console.log(`CRON:  ${result.cron_expression}`);
      console.log(`Means: ${result.human_readable}`);
      console.log(`Valid: ${validation.valid ? 'Yes' : `No - ${validation.error}`}`);
      console.log(`Confidence: ${(result.confidence * 100).toFixed(0)}%`);

      if (result.ambiguity_warning) {
        console.log(`Warning: ${result.ambiguity_warning}`);
      }

      console.log('Next runs:');
      result.next_runs.forEach(run => console.log(`  - ${run}`));
    } else {
      console.log('Failed to convert');
    }

    console.log('');
  }
}

/**
 * Use with materialized view scheduling
 */
async function scheduleViewRefresh(
  datasetId: number,
  scheduleDescription: string
): Promise<void> {
  console.log(`Setting refresh schedule for dataset ${datasetId}`);
  console.log(`Schedule: "${scheduleDescription}"`);

  const result = await convertToCron(scheduleDescription);

  if (!result) {
    throw new Error('Failed to convert schedule');
  }

  const validation = validateCronFormat(result.cron_expression);
  if (!validation.valid) {
    throw new Error(`Invalid CRON expression: ${validation.error}`);
  }

  if (result.confidence < 0.8) {
    console.warn(`Low confidence (${(result.confidence * 100).toFixed(0)}%) - please verify:`);
    console.warn(`  CRON: ${result.cron_expression}`);
    console.warn(`  Means: ${result.human_readable}`);
    if (result.ambiguity_warning) {
      console.warn(`  Note: ${result.ambiguity_warning}`);
    }
  }

  // Here you would apply the CRON to the materialized view
  // await api.updateMaterializedViewSchedule(datasetId, result.cron_expression);

  console.log(`Schedule set: ${result.cron_expression}`);
  console.log(`This means: ${result.human_readable}`);
}

// Run the interactive converter
interactiveConverter()
  .then(() => console.log('Done'))
  .catch((error) => {
    console.error('Error:', error);
    process.exit(1);
  });
```

## How it works

1. **Natural language input**: User provides a schedule description in plain English

2. **Context building**: The prompt includes the current date and examples of CRON patterns

3. **Inference**: Model Inference converts the description to a structured response

4. **Validation**: The CRON expression is validated for correct format

5. **Confirmation**: Human-readable description and next run times provide verification

## Common schedule patterns

| Natural Language           | CRON Expression |
| -------------------------- | --------------- |
| Every day at 9am           | `0 9 * * *`     |
| Every Monday at 8am        | `0 8 * * 1`     |
| Every hour                 | `0 * * * *`     |
| Every 15 minutes           | `*/15 * * * *`  |
| First of month at midnight | `0 0 1 * *`     |
| Weekdays at 6pm            | `0 18 * * 1-5`  |
| Every Sunday at noon       | `0 12 * * 0`    |

## Handling ambiguity

The model includes confidence scores and warnings for ambiguous inputs:

```typescript theme={null}
if (result.confidence < 0.8) {
  // Ask user for confirmation
  console.log('Please confirm this interpretation:');
  console.log(`  "${scheduleDescription}" -> ${result.cron_expression}`);
  console.log(`  Which means: ${result.human_readable}`);
}

if (result.ambiguity_warning) {
  // Show the assumption made
  console.log(`Note: ${result.ambiguity_warning}`);
}
```

## Related content

<CardGroup>
  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Inference fundamentals
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Define response schemas
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Schedule view refreshes
  </Card>

  <Card title="Creating Views" icon="table" href="/guides/nql/creating-materialized-views">
    Create and manage views
  </Card>
</CardGroup>


# Retail Competitive Intelligence
Source: https://docs.narrative.io/cookbooks/use-cases/retail-collaboration

Build audience segments from competitor store visitors using location data, frequency analysis, and purchase history enrichment

A national retailer is expanding to the West Coast and wants to understand their potential customers before opening new stores. Through Narrative, they can identify shoppers at competitor locations, segment them by loyalty, and enrich their profiles with purchase behavior—all without having any first-party data in those markets.

## What you will build

This cookbook walks through creating a competitive intelligence pipeline that:

1. Identifies mobile device IDs (MAIDs) observed at competitor store locations in California, Nevada, and Oregon
2. Segments visitors into loyalty tiers based on visit frequency (loyal, casual, one-time)
3. Enriches loyal competitor shoppers with CPG purchase history (groceries, personal care, household goods)

The result is targetable audience segments for conquest marketing campaigns.

## Prerequisites

Before starting, ensure you have:

* Access to location data containing `geo_coordinates` and `unique_identifier` (MAID) attributes
* Access to purchase transaction data with category information
* Familiarity with [NQL syntax](/nql/general/syntax)
* Understanding of [materialized views](/concepts/nql/materialized-views)

## Step 1: Define competitor store locations

First, create a reference dataset of competitor store locations. This dataset contains the latitude/longitude coordinates for each store, which you will use as geofence centers.

```sql theme={null}
CREATE MATERIALIZED VIEW "competitor_store_locations"
REFRESH_SCHEDULE = '@monthly'
DISPLAY_NAME = 'Competitor Store Locations - West Coast'
DESCRIPTION = 'Geofenced competitor retail locations in CA, NV, OR'
TAGS = ('competitive_intelligence', 'location', 'west_coast')
WRITE_MODE = 'overwrite'
AS
SELECT
    store_id,
    brand_name,
    latitude,
    longitude,
    region AS state
FROM company_data.store_directory
WHERE
    brand_name IN ('CompetitorA', 'CompetitorB', 'CompetitorC')
    AND region IN ('CA', 'NV', 'OR')
```

<Note>
  Replace `company_data.store_directory` with your actual store location dataset. You can also upload a CSV of competitor locations to create this reference table.
</Note>

## Step 2: Identify competitor store visitors

Query the Rosetta Stone table to find mobile advertising IDs (IDFA/GAID) observed within proximity of competitor store coordinates over the past 90 days.

```sql theme={null}
CREATE MATERIALIZED VIEW "competitor_store_visitors"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'Competitor Store Visitors - Raw'
DESCRIPTION = 'MAIDs observed at competitor retail locations in past 90 days'
TAGS = ('competitive_intelligence', 'location', 'audience')
WRITE_MODE = 'overwrite'
PARTITIONED_BY event_date DAY
AS
SELECT
    rs."unique_identifier"."value" AS maid,
    rs."unique_identifier"."type" AS maid_type,
    stores.store_id,
    stores.brand_name,
    stores.state,
    CAST(rs."event_timestamp" AS TIMESTAMP) AS visit_timestamp,
    DATE(rs."event_timestamp") AS event_date
FROM narrative.rosetta_stone rs
JOIN company_data."competitor_store_locations" stores
    ON ABS(rs."geo_coordinates"."latitude" - stores.latitude) < 0.002
    AND ABS(rs."geo_coordinates"."longitude" - stores.longitude) < 0.002
WHERE
    rs."event_timestamp" > CURRENT_TIMESTAMP - INTERVAL '90' DAY
    AND rs."unique_identifier"."type" IN ('idfa', 'gaid')
    AND rs."unique_identifier"."value" IS NOT NULL
    AND rs."geo_coordinates"."latitude" IS NOT NULL
    AND rs."region" IN ('CA', 'NV', 'OR')
```

**Key patterns used:**

| Pattern                                      | Purpose                                            |
| -------------------------------------------- | -------------------------------------------------- |
| `unique_identifier.value`                    | Extracts the MAID value from the identifier object |
| `unique_identifier.type IN ('idfa', 'gaid')` | Filters to mobile advertising IDs only             |
| `ABS(latitude - stores.latitude) < 0.002`    | Proximity match (\~150-200 meters geofence)        |
| `INTERVAL '90' DAY`                          | Rolling 90-day lookback window                     |
| `PARTITIONED_BY event_date DAY`              | Optimizes queries filtering by date                |

<Tip>
  The proximity threshold of 0.002 degrees equals approximately 150-200 meters depending on latitude. Adjust this value based on store size and urban density. Larger stores or rural areas may warrant a larger radius.
</Tip>

## Step 3: Segment by visit frequency

Aggregate the visitor data to calculate how many times each MAID visited competitor stores, then segment into loyalty tiers.

```sql theme={null}
CREATE MATERIALIZED VIEW "competitor_visitors_segmented"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'Competitor Visitors - Loyalty Segments'
DESCRIPTION = 'MAIDs segmented by visit frequency: loyal (4+), casual (2-3), one-time (1)'
TAGS = ('competitive_intelligence', 'audience', 'segmentation')
WRITE_MODE = 'overwrite'
AS
SELECT
    maid,
    maid_type,
    COUNT(DISTINCT visit_timestamp) AS total_visits,
    COUNT(DISTINCT store_id) AS unique_stores_visited,
    COUNT(DISTINCT brand_name) AS unique_brands_visited,
    MIN(visit_timestamp) AS first_visit,
    MAX(visit_timestamp) AS last_visit,
    CASE
        WHEN COUNT(DISTINCT visit_timestamp) >= 4 THEN 'loyal'
        WHEN COUNT(DISTINCT visit_timestamp) BETWEEN 2 AND 3 THEN 'casual'
        ELSE 'one_time'
    END AS loyalty_segment
FROM company_data."competitor_store_visitors"
GROUP BY maid, maid_type
```

To create separate audiences for activation, you can also build segment-specific views:

### Loyal shoppers (4+ visits)

```sql theme={null}
CREATE MATERIALIZED VIEW "loyal_competitor_shoppers"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'Loyal Competitor Shoppers'
DESCRIPTION = 'MAIDs with 4+ visits to competitor stores'
TAGS = ('competitive_intelligence', 'audience', 'loyal')
WRITE_MODE = 'overwrite'
AS
SELECT
    maid,
    maid_type,
    COUNT(DISTINCT visit_timestamp) AS total_visits,
    MAX(visit_timestamp) AS last_visit
FROM company_data."competitor_store_visitors"
GROUP BY maid, maid_type
HAVING COUNT(DISTINCT visit_timestamp) >= 4
```

### Non-loyal shoppers (1-2 visits)

```sql theme={null}
CREATE MATERIALIZED VIEW "nonloyal_competitor_shoppers"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'Non-Loyal Competitor Shoppers'
DESCRIPTION = 'MAIDs with 1-2 visits to competitor stores'
TAGS = ('competitive_intelligence', 'audience', 'nonloyal')
WRITE_MODE = 'overwrite'
AS
SELECT
    maid,
    maid_type,
    COUNT(DISTINCT visit_timestamp) AS total_visits,
    MAX(visit_timestamp) AS last_visit
FROM company_data."competitor_store_visitors"
GROUP BY maid, maid_type
HAVING COUNT(DISTINCT visit_timestamp) <= 2
```

## Step 4: Enrich with purchase history

Join the loyal segment with purchase transaction data to understand CPG category affinity. This creates richer profiles for targeting.

```sql theme={null}
CREATE MATERIALIZED VIEW "loyal_shoppers_enriched"
REFRESH_SCHEDULE = '@weekly'
DISPLAY_NAME = 'Loyal Competitor Shoppers - Purchase Enriched'
DESCRIPTION = 'Loyal competitor shoppers with CPG purchase category flags'
TAGS = ('competitive_intelligence', 'audience', 'enriched')
WRITE_MODE = 'overwrite'
AS
SELECT
    loyal.maid,
    loyal.maid_type,
    loyal.total_visits,
    loyal.last_visit AS last_store_visit,
    MAX(CASE WHEN purchases.purchase_category = 'groceries' THEN 1 ELSE 0 END) AS has_groceries,
    MAX(CASE WHEN purchases.purchase_category = 'personal_care' THEN 1 ELSE 0 END) AS has_personal_care,
    MAX(CASE WHEN purchases.purchase_category = 'household_goods' THEN 1 ELSE 0 END) AS has_household_goods,
    COUNT(DISTINCT purchases.transaction_id) AS total_purchases,
    SUM(purchases.transaction_amount) AS total_spend
FROM company_data."loyal_competitor_shoppers" loyal
LEFT JOIN company_data."purchase_transactions" purchases
    ON loyal.maid = purchases."unique_identifier"."value"
    AND purchases."event_timestamp" > CURRENT_TIMESTAMP - INTERVAL '180' DAY
GROUP BY
    loyal.maid,
    loyal.maid_type,
    loyal.total_visits,
    loyal.last_visit
```

This query:

* Joins loyal shoppers with purchase data over a 180-day window
* Pivots purchase categories into boolean flags for easy filtering
* Calculates aggregate purchase metrics (count and total spend)
* Uses `LEFT JOIN` to retain all loyal shoppers even if no purchase data exists

## Summary

You have built a competitive intelligence pipeline with four materialized views:

| View                            | Purpose                                         |
| ------------------------------- | ----------------------------------------------- |
| `competitor_store_locations`    | Reference data for competitor store geofences   |
| `competitor_store_visitors`     | Raw visitor log with MAIDs and visit timestamps |
| `competitor_visitors_segmented` | Visitors segmented by loyalty tier              |
| `loyal_shoppers_enriched`       | Loyal shoppers enriched with purchase behavior  |

### Use cases

* **Conquest campaigns**: Target loyal competitor shoppers with promotional offers to drive store switching
* **Lookalike modeling**: Use enriched profiles as seed audiences for prospecting
* **Store placement analysis**: Understand which competitor locations draw specific shopper profiles
* **Assortment planning**: Align product mix with purchase category preferences of local shoppers

### Important considerations

<Warning>
  **MAID availability**: iOS App Tracking Transparency and Android privacy changes have reduced MAID availability. Expect smaller audience sizes compared to historical baselines.
</Warning>

<Note>
  **Privacy compliance**: Location-based audiences may be subject to CCPA (California) and other privacy regulations. Ensure your use case complies with applicable data protection requirements. See [CCPA compliance](/concepts/compliance/ccpa) for details.
</Note>

***

## Related content

<CardGroup>
  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to creating and managing materialized views
  </Card>

  <Card title="Materialized View Syntax" icon="list-check" href="/nql/commands/create-materialized-view">
    Complete reference for REFRESH\_SCHEDULE, PARTITIONED\_BY, and other options
  </Card>

  <Card title="JOIN Performance" icon="chart-line" href="/concepts/nql/join-performance">
    Optimize spatial and identity joins for large datasets
  </Card>
</CardGroup>


# Create an Account
Source: https://docs.narrative.io/getting-started/create-account

Sign up for Narrative I/O and set up your workspace

In this tutorial, you'll create your Narrative I/O account and complete the registration process. By the end, you'll be ready to start uploading data.

## Prerequisites

* A valid email address, or
* A Google, GitHub, or LinkedIn account for single sign-on

## What you'll learn

* How to sign up for Narrative I/O
* How to verify your email via magic link (for email signups)
* How to complete your profile

## Steps

<Steps>
  <Step title="Go to the sign-up page">
    Navigate to [app.narrative.io](https://app.narrative.io) to access the sign-up page.

    You have two options to create your account:

    * **Email**: Enter your email address and click **Continue with email**
    * **Single sign-on**: Click **Continue with Google**, **Continue with GitHub**, or **Continue with LinkedIn**

    <Frame>
      <img alt="Narrative sign up page showing email field and SSO options for Google, GitHub, and LinkedIn" />
    </Frame>

    <Note>
      If you sign up with an SSO provider, skip the next step and proceed directly to completing your profile.
    </Note>
  </Step>

  <Step title="Verify your email">
    If you signed up with an email address, you'll see a confirmation that a magic link was sent to your inbox.

    <Frame>
      <img alt="Check your email confirmation page with quick links to open Gmail, Yahoo, or Outlook" />
    </Frame>

    Open your email and find the message from Narrative. Click **Continue** to verify your email address.

    <Frame>
      <img alt="Magic link email with Continue button to verify your account" />
    </Frame>

    <Warning>
      The magic link expires in 1 hour. If it expires, return to the sign-up page and request a new link.
    </Warning>
  </Step>

  <Step title="Complete your profile">
    Enter your name and the company you work for, then click **Complete Registration**.

    <Frame>
      <img alt="Profile completion form with fields for name and company" />
    </Frame>
  </Step>

  <Step title="Accept Terms of Service">
    Review the Terms of Service. Check the box to confirm you've read and agree to the terms, then click **Agree**.

    <Frame>
      <img alt="Terms of Service agreement page" />
    </Frame>
  </Step>

  <Step title="Log in to your account">
    You'll see a success message confirming your account was created. Click **Login** to access Narrative I/O.

    <Frame>
      <img alt="Success page confirming account registration" />
    </Frame>

    For future logins with an email account, you'll receive a magic link each time—no password required.
  </Step>
</Steps>

## What you accomplished

* Created a Narrative I/O account using email or SSO
* Verified your email address (if using email signup)
* Completed your profile and accepted the Terms of Service

## Troubleshooting

| Issue              | Solution                                                                                         |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| Magic link expired | Return to [app.narrative.io](https://app.narrative.io), enter your email, and request a new link |
| Email not received | Check your spam folder; add Narrative to your safe senders list                                  |
| SSO login fails    | Ensure you're using the correct provider account; try a different browser                        |

## Next steps

<CardGroup>
  <Card title="Upload Your Data" icon="upload" href="/getting-started/upload-data">
    Add your first dataset to Narrative I/O
  </Card>
</CardGroup>


# Evaluate Your Mappings
Source: https://docs.narrative.io/getting-started/evaluate-mappings

Use AI to assess and improve your schema mappings

In this tutorial, you'll use AI-powered evaluation to assess your mapping quality and refine your normalizations.

<Info>
  This tutorial is part of the First Steps series. Complete [Normalize with Rosetta Stone](/getting-started/normalize-data) before starting.

  This step is optional for getting started. Evaluating mappings is more commonly part of a production workflow when you want to refine and improve your normalizations over time.
</Info>

<iframe title="Review and refine transformations in a normalized dataset" />

## Prerequisites

* A Narrative I/O account
* At least one dataset with active Rosetta Stone mappings

## Steps

<Steps>
  <Step title="Confirm your workspace">
    Use the workspace dropdown to switch organizations if needed. This keeps your normalized data organized by account.
  </Step>

  <Step title="Open Normalized Datasets">
    Navigate to **Normalized Datasets** to review and manage your standardized tables.
  </Step>

  <Step title="Select your dataset">
    Select the dataset you want to inspect. You can review its mappings, quality signals, and transformation setup.
  </Step>

  <Step title="Open the Actions menu">
    Open the **Actions** menu to access evaluation and update options for this dataset.
  </Step>

  <Step title="Run Evaluate Mappings">
    Choose **Evaluate Mappings** to validate how columns map to standardized attributes and assess confidence levels.
  </Step>

  <Step title="Review evaluation results">
    Click **Evaluate Mappings** to refresh scoring and identify any fields that need attention.
  </Step>

  <Step title="Check column normalization status">
    Review the column normalization status to see which columns have been analyzed and which still need review.
  </Step>

  <Step title="Use AI to auto-analyze">
    Use the **AI** button to auto-analyze column normalizations and speed up the review process.
  </Step>

  <Step title="Review an expression">
    Open an expression to review the exact logic producing the output. This helps you understand how source values are being transformed.
  </Step>

  <Step title="Edit expressions as needed">
    Edit expressions to refine how values are parsed and formatted for consistency. You can adjust logic for timestamps, addresses, or any other field.
  </Step>

  <Step title="View all transformations">
    View all transformations to quickly audit every normalized field and its confidence score.
  </Step>
</Steps>

## Next steps

<CardGroup>
  <Card title="Managing Evaluations" icon="gauge" href="/guides/rosetta-stone/managing-evaluations">
    Deep dive into evaluation workflows
  </Card>

  <Card title="Accepting AI Suggestions" icon="wand-magic-sparkles" href="/guides/rosetta-stone/accepting-ai-suggestions">
    Advanced suggestion review techniques
  </Card>

  <Card title="Confidence Scoring" icon="gauge-high" href="/concepts/rosetta-stone/confidence-scoring">
    Understand how scores are calculated
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Manual testing techniques
  </Card>
</CardGroup>


# Write Your First Query
Source: https://docs.narrative.io/getting-started/first-nql-query

Create a filtered dataset using Query Builder in Data Studio

In this tutorial, you'll use Query Builder to create a filtered dataset from your data. Query Builder provides a visual interface for selecting data sources, choosing output fields, and applying filters—no code required.

<iframe title="Create a filtered dataset in Data Studio using Query Builder" />

<Info>
  This tutorial is part of the First Steps series. Complete [Normalize with Rosetta Stone](/getting-started/normalize-data) before starting.
</Info>

## Prerequisites

* A Narrative I/O account
* At least one dataset in My Data

## What you'll learn

* How to access Data Studio
* How to use Query Builder to select data sources
* How to choose output fields
* How to add filters to narrow your data
* How to create and save a dataset

***

## Steps

<Steps>
  <Step title="Open Data Studio">
    From the navigation menu, go to **My Data** to view your available datasets. Then navigate to **Data Studio** to start creating a new dataset with Query Builder.
  </Step>

  <Step title="Add a data source">
    In Query Builder, click **Add** under **Sources** to choose where your dataset should pull data from. Select **My Data** as your source, then pick the dataset you want to work with. This dataset becomes the foundation for your filtered output.
  </Step>

  <Step title="Configure output fields">
    Click **Add** under **Output** to control which columns appear in the final dataset. Click **Add field** to select each column you want to include. Choose the fields that are relevant to your use case—these might include identifiers, timestamps, or any attributes you need for analysis.

    After selecting your fields, click **Continue** to confirm your output configuration.
  </Step>

  <Step title="Add a filter">
    Click **Add** under **Filter** to narrow your dataset to specific records. Choose a field to filter on, then select an operator (such as "Is greater than" or "Equals") and provide a value.

    For date fields, you can use the calendar picker to select a specific cutoff date. Click **Add** to apply the filter condition, then confirm to finalize your filter logic.
  </Step>

  <Step title="Create the dataset">
    Once your sources, outputs, and filters are configured, click **Create dataset**. Enter a unique name that makes your dataset easy to find and reference later. Click **Save** to generate the dataset from your query configuration.

    Your new filtered dataset is now available in My Data for use in analysis, exports, or collaborations.
  </Step>
</Steps>

***

## Next steps

<CardGroup>
  <Card title="Filtering Data" icon="filter" href="/guides/nql/filtering-transforming">
    Learn to filter and transform your data
  </Card>

  <Card title="NQL Syntax Reference" icon="code" href="/nql/general/syntax">
    Explore the complete query syntax
  </Card>

  <Card title="Create a Materialized View" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Save query results for recurring use
  </Card>
</CardGroup>


# Normalize with Rosetta Stone
Source: https://docs.narrative.io/getting-started/normalize-data

Map your data to a standard schema for seamless collaboration

In this tutorial, you'll use Rosetta Stone to normalize your data. Normalization maps your unique schema to Narrative's standard format, enabling queries across datasets from different sources.

<Info>
  This tutorial is part of the First Steps series. Complete [Upload Your Data](/getting-started/upload-data) before starting.
</Info>

<iframe title="Normalize a dataset with suggested attribute mappings" />

## Prerequisites

* A Narrative I/O account
* At least one uploaded dataset (from the previous tutorial)

## Steps

<Steps>
  <Step title="Confirm your workspace">
    Open the workspace menu to confirm you're normalizing data in the right account.
  </Step>

  <Step title="Go to Normalized Datasets">
    Navigate to **Normalized Datasets** to manage the standardized datasets used across your workflows.
  </Step>

  <Step title="Select your dataset">
    Select the dataset you want to review and improve its attribute mappings.
  </Step>

  <Step title="Switch to the Normalize tab">
    Go to the **Normalize** tab to generate and apply suggested column-to-attribute mappings.
  </Step>

  <Step title="Start the analysis">
    Click **Start analysis** to get AI-suggested normalization strategies for your fields.
  </Step>

  <Step title="Review suggested expressions">
    Open expression mappings to verify how your data will be transformed. For example, address parts might be combined into a postal code, or values like "M" might be standardized to "male".
  </Step>

  <Step title="Accept individual mappings">
    Click **Accept** to apply mappings that look correct. This ensures downstream workflows work from consistent fields.
  </Step>

  <Step title="Review high-confidence mappings">
    Use **View** to quickly review all high-confidence mappings before applying them at scale.
  </Step>

  <Step title="Accept all high-confidence mappings">
    Click **Accept all high confidence** to apply the best matches and save manual review time.
  </Step>
</Steps>

## Next steps

<CardGroup>
  <Card title="Write Your First Query" icon="terminal" href="/getting-started/first-nql-query">
    Query your normalized data with NQL
  </Card>

  <Card title="How Rosetta Stone Works" icon="lightbulb" href="/concepts/rosetta-stone/how-it-works">
    Understand the normalization model in depth
  </Card>

  <Card title="Mapping Schemas Guide" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Advanced mapping techniques and API usage
  </Card>

  <Card title="Edge Cases" icon="triangle-exclamation" href="/guides/rosetta-stone/edge-cases">
    Handle complex mapping scenarios
  </Card>
</CardGroup>


# Upload Your Data
Source: https://docs.narrative.io/getting-started/upload-data

Add your first dataset to Narrative I/O

In this tutorial, you'll upload your first dataset to Narrative I/O. By the end, your data will be ready for normalization and querying.

<iframe title="Upload a CSV dataset and review its schema and sample records" />

<Info>
  This tutorial is part of the First Steps series. Complete [Create an Account](/getting-started/create-account) before starting.
</Info>

## Prerequisites

* A Narrative I/O account
* A CSV file to upload

<Tip>
  Don't have a dataset handy? Download this [sample CSV of top domains](https://github.com/fffaraz/datasets/blob/master/top500Domains.csv) to follow along.
</Tip>

## What you'll learn

* How to navigate to the Datasets page
* How to upload a CSV file
* How to verify your dataset was created successfully

## Steps

<Steps>
  <Step title="Navigate to Datasets">
    In the left sidebar, click **My Data**, then select **Datasets**.

    This takes you to [https://app.narrative.io/platform/my-data/datasets](https://app.narrative.io/platform/my-data/datasets), where you can see all your existing datasets and their statuses.
  </Step>

  <Step title="Start a new dataset">
    Click **New Dataset** to begin the upload process.

    You'll land on the dataset creation page at [https://app.narrative.io/platform/my-data/new-dataset](https://app.narrative.io/platform/my-data/new-dataset). This is where you'll upload your file and configure your dataset.
  </Step>

  <Step title="Upload your file">
    Upload your CSV file using one of these methods:

    * **Drag and drop**: Drag your file directly onto the upload area
    * **File picker**: Click the upload area to browse and select your file

    Once you select a file, the platform begins processing:

    * The file uploads to Narrative's servers
    * The system infers your schema from the data
    * Initial validation checks run automatically

    <Tip>
      Narrative also supports JSON files. For connecting external data sources like cloud storage or databases, see the [Connectors reference](/reference/connectors).
    </Tip>
  </Step>

  <Step title="Review your dataset">
    After a successful upload, you're redirected to your dataset's overview page.

    The URL follows this pattern: `https://app.narrative.io/platform/my-data/dataset/{id}#overview`

    On this page you'll see:

    * **Dataset name and metadata**: Basic information about your upload
    * **Status indicator**: Confirms the dataset was created successfully
    * **Navigation tabs**: Access to Overview, Schema, Records, and Mappings

    Your dataset now exists in Narrative and is ready for the next step: normalization.
  </Step>
</Steps>

## Verify it worked

To confirm your upload succeeded:

1. Navigate back to **My Data** → **Datasets**
2. Find your new dataset in the list
3. Check that the status shows the upload is complete
4. Click into the dataset and select the **Records** tab to preview your data

You should see your CSV data displayed with the columns the system inferred from your file.

## What you accomplished

* Navigated to the dataset management interface
* Uploaded a CSV file as a new dataset
* Verified the dataset was created and is ready for normalization

## Troubleshooting

| Issue                     | Solution                                                        |
| ------------------------- | --------------------------------------------------------------- |
| Upload fails or times out | Check file size limits; try a smaller file or split into chunks |
| File format not supported | Ensure your file is a valid CSV or JSON format                  |
| Dataset shows 0 records   | Verify the file has data rows, not just headers                 |
| Schema looks incorrect    | Check that your CSV uses consistent delimiters and quoting      |

## Next steps

<CardGroup>
  <Card title="Normalize with Rosetta Stone" icon="language" href="/getting-started/normalize-data">
    Map your data to Narrative's standard schema
  </Card>

  <Card title="Connect External Sources" icon="plug" href="/reference/connectors">
    Set up connectors for cloud storage and databases
  </Card>
</CardGroup>


# Structuring Audiences for Activation
Source: https://docs.narrative.io/guides/activation/audience-strategies

Strategies for organizing your datasets to deliver audiences to advertising platforms

Before delivering audience data to a connector, you need to structure your data in datasets. This guide covers three common strategies for organizing your data, with recommendations for when to use each approach.

## Prerequisites

Before you begin:

* You have data with deliverable identifiers (hashed email, hashed phone, or mobile ad IDs)
* Your identifiers are properly formatted and hashed (see [Hashing PII for Upload](/guides/ingestion/hashing-pii))
* You understand which identifiers your destination requires (see [Data Activation Overview](/concepts/data-activation/overview))

***

## Strategy 1: One dataset per audience

Create a separate dataset for each audience you want to deliver. The dataset name and description correspond directly to the audience.

### Dataset structure

| Field                 | Description                                    |
| --------------------- | ---------------------------------------------- |
| `sha256_hashed_email` | The hashed identifier for each audience member |

Example data:

```csv theme={null}
sha256_hashed_email
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3
a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd
b2c3d4e5f6789012345678901234567890123456789012345678901234bcde
```

### When to use this strategy

<AccordionGroup>
  <Accordion title="You build audiences outside Narrative">
    If your audience creation happens in external systems (CRM, CDP, marketing automation), then export each audience as a separate file for upload.
  </Accordion>

  <Accordion title="You use API automation for dataset creation">
    If you have automated pipelines that create datasets via the [SDK](/guides/sdk/managing-datasets), creating a new dataset per audience is straightforward.
  </Accordion>

  <Accordion title="Audiences are static or infrequently updated">
    If audiences don't change often, managing them as separate datasets keeps things simple.
  </Accordion>
</AccordionGroup>

### Advantages

* **Simplicity**: Each dataset maps directly to one audience
* **Clear ownership**: Easy to track which dataset powers which audience
* **Selective updates**: Update only the audiences that have changed

### Considerations

* Requires creating a new dataset for each new audience
* Can become difficult to manage with hundreds of audiences
* Duplicated records across datasets if users belong to multiple audiences

***

## Strategy 2: Master dataset with audience identifiers

Maintain a single dataset containing all audience members, with columns that identify which audience each record belongs to.

### Dataset structure

| Field                 | Description                        |
| --------------------- | ---------------------------------- |
| `sha256_hashed_email` | The hashed identifier              |
| `audience_id`         | Unique identifier for the audience |
| `audience_name`       | Human-readable audience name       |

Example data:

```csv theme={null}
sha256_hashed_email,audience_id,audience_name
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3,1001,high_value_customers
a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd,1001,high_value_customers
b2c3d4e5f6789012345678901234567890123456789012345678901234bcde,1002,recent_purchasers
c3d4e5f6789012345678901234567890123456789012345678901234cdef,1002,recent_purchasers
```

### Delivering specific audiences

Use NQL to filter for a specific audience when delivering:

```sql theme={null}
SELECT
    sha256_hashed_email
FROM
    company_123.master_audiences
WHERE
    audience_id = '1001'
```

### When to use this strategy

<AccordionGroup>
  <Accordion title="You manage many related audiences">
    If you have dozens or hundreds of audiences that share similar structures, a single dataset is easier to maintain.
  </Accordion>

  <Accordion title="You want simplified data updates">
    Append new rows to a single dataset instead of managing uploads across many datasets.
  </Accordion>

  <Accordion title="You need to add new audiences without creating new datasets">
    Adding a new audience is as simple as appending rows with a new `audience_id`.
  </Accordion>
</AccordionGroup>

### Advantages

* **Centralized management**: One dataset to maintain
* **Easy audience additions**: New audiences are just new rows
* **Efficient updates**: Append new data without managing multiple datasets

### Considerations

* Requires NQL filtering to extract specific audiences
* Larger dataset size may affect query performance
* Need to maintain unique audience IDs

***

## Strategy 3: Raw data with dynamic audience creation

Upload a master dataset with all available customer attributes, then use NQL to define audiences dynamically based on any attribute combination.

### Dataset structure

| Field                        | Description                  |
| ---------------------------- | ---------------------------- |
| `sha256_hashed_email`        | The hashed identifier        |
| `age`                        | Customer age                 |
| `gender`                     | Customer gender              |
| `estimated_household_income` | Income range or value        |
| `last_purchase_date`         | Date of most recent purchase |
| `lifetime_value`             | Total customer value         |

Example data:

```csv theme={null}
sha256_hashed_email,age,gender,estimated_household_income,last_purchase_date,lifetime_value
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3,34,male,85000,2024-01-15,1250.00
a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd,28,female,52000,2024-02-20,890.00
b2c3d4e5f6789012345678901234567890123456789012345678901234bcde,45,female,120000,2024-01-05,3200.00
```

### Creating audiences with NQL

Define audiences on-the-fly using any attribute combination:

**High-income females:**

```sql theme={null}
SELECT
    sha256_hashed_email
FROM
    company_123.customer_data
WHERE
    gender = 'female'
    AND estimated_household_income >= 75000
```

**Recent high-value customers:**

```sql theme={null}
SELECT
    sha256_hashed_email
FROM
    company_123.customer_data
WHERE
    last_purchase_date >= '2024-01-01'
    AND lifetime_value >= 1000
```

**Age-based segments:**

```sql theme={null}
SELECT
    sha256_hashed_email
FROM
    company_123.customer_data
WHERE
    age BETWEEN 25 AND 44
```

### When to use this strategy

<AccordionGroup>
  <Accordion title="You want maximum flexibility">
    Create any audience segment without re-uploading data or pre-defining audiences.
  </Accordion>

  <Accordion title="You prefer minimal data preprocessing">
    Send data to Narrative with minimal transformation and let the platform handle segmentation.
  </Accordion>

  <Accordion title="Your audience definitions change frequently">
    Quickly iterate on audience criteria without modifying source data.
  </Accordion>
</AccordionGroup>

### Advantages

* **Maximum flexibility**: Define any audience criteria without changing data
* **Minimal preprocessing**: Upload raw data, segment later
* **Rapid iteration**: Test new audience definitions instantly

### Considerations

* Requires familiarity with NQL for audience creation
* Complex queries may take longer to execute on large datasets
* Attribute data must be normalized consistently

***

## Choosing a strategy

| Factor                   | Strategy 1 (Per Audience) | Strategy 2 (Master + IDs) | Strategy 3 (Raw Data) |
| ------------------------ | ------------------------- | ------------------------- | --------------------- |
| **Number of audiences**  | Few (1-20)                | Many (20-100+)            | Variable/unknown      |
| **Audience definitions** | Fixed                     | Fixed                     | Dynamic               |
| **Data preprocessing**   | Done externally           | Partial                   | Minimal               |
| **NQL required**         | No                        | Basic filtering           | Advanced queries      |
| **Flexibility**          | Low                       | Medium                    | High                  |

<Info>
  You can combine strategies. For example, use Strategy 3 for your core customer data while maintaining Strategy 1 datasets for specialized audiences from external sources.
</Info>

***

## Next steps

After structuring your data:

1. **[Normalize your data](/guides/rosetta-stone/mapping-schemas)**: Map your fields to Rosetta Stone attributes so connectors recognize your identifiers

2. **Configure your connector**: Install and configure the connector for your destination platform

3. **Deliver your audiences**: Use the connector to push your data to the destination

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data to destinations
  </Card>

  <Card title="Datasets" icon="table" href="/concepts/primitives/datasets">
    Learn about dataset structure and management
  </Card>

  <Card title="Filtering & Transforming" icon="filter" href="/guides/nql/filtering-transforming">
    NQL basics for filtering audience data
  </Card>

  <Card title="Managing Datasets" icon="gear" href="/guides/sdk/managing-datasets">
    Create and manage datasets with the SDK
  </Card>
</CardGroup>


# Building a look-alike audience
Source: https://docs.narrative.io/guides/audience-studio/building-a-lookalike-audience

Expand a seed audience by scoring a larger population for users that resemble it, using Look-alike Studio

This guide walks through building a look-alike audience in Look-alike Studio — from picking a seed audience and population dataset to selecting similarity attributes, sizing the output, and finalizing the audience.

<Info>
  A look-alike audience is a new audience derived from an existing **seed audience** by scoring a larger **population dataset** and keeping the users that most resemble the seed. Use it to prospect beyond your known customers without leaving the platform.
</Info>

## Prerequisites

* An existing seed audience — typically a dataset created with [Audience Studio](/guides/audience-studio/building-an-audience) — with at least one identity attribute (such as `sha256_hashed_email` or `maid`) mapped via [Rosetta Stone](/guides/rosetta-stone/mapping-schemas)
* A separate population dataset on the same data plane, also with at least one identity attribute mapped via Rosetta Stone
* Both datasets must share at least one identity attribute so seed users can be matched against the population

<Note>
  The seed and population must be **different datasets**. Scoring an audience against itself would only return the seed users back.
</Note>

## Step 1: Pick a seed audience

<Steps>
  <Step title="Open Look-alike Studio">
    Navigate to **My Audiences** in the sidebar, then open **Look-alike Studio**.
  </Step>

  <Step title="Select a seed audience">
    Choose the existing audience the look-alike model will learn from. Only audiences on the currently selected data plane are shown.

    Audiences without identity attributes appear under **Not available** — they have no join keys, so their users can't be matched against a population. Map an identity attribute via Rosetta Stone to use them as a seed.

    If you don't have any audiences yet, create one in [Audience Studio](/guides/audience-studio/building-an-audience) first.
  </Step>
</Steps>

## Step 2: Pick a population dataset

The population is the pool of candidate users the model scores against the seed. Users in the population who most resemble the seed become the look-alike audience.

<Steps>
  <Step title="Select a dataset">
    Choose a dataset of candidate users on the same data plane as the seed.

    The seed audience itself is disabled — it can't double as the population. Datasets without identity attributes are also disabled, since seed users couldn't be matched within them.
  </Step>
</Steps>

## Step 3: Choose similarity attributes

Look-alike Studio classifies the population dataset's fields into:

* **Identity attributes** — Used as join keys to match seed users within the population. Required on both sides.
* **Feature attributes** — Used to measure similarity between users. You choose which features the model considers.
* **Metadata** — Non-predictive fields excluded from scoring.

<Steps>
  <Step title="Confirm join keys">
    Look-alike Studio shows the identity attributes available on the seed and population. If either side has no identity attributes, scoring isn't possible — return to Rosetta Stone and map at least one shared identifier on both datasets.
  </Step>

  <Step title="Select feature attributes">
    Pick the features the model should use to compare users — for example, age, interests, purchase categories, or geography. More features can improve quality but increase compute cost.

    Use the search field to filter the attribute list by name or path.
  </Step>
</Steps>

<Tip>
  Choose features that distinguish your seed users from the general population. Universally common attributes add noise without improving similarity scoring.
</Tip>

## Step 4: Configure the output

Decide how large the look-alike audience should be and whether the original seed users are kept.

<Steps>
  <Step title="Pick a sizing mode">
    Choose one:

    * **Limit by size** — Keep the top *N* highest-scoring users from the population. Defaults to **10,000**.
    * **Limit by score** — Keep every user whose similarity score meets or exceeds the threshold (0–100%). Defaults to **50%**.

    Sizing by score produces a variable-size audience that grows or shrinks with the population's similarity distribution. Sizing by size produces a fixed-size audience regardless of distribution.
  </Step>

  <Step title="Choose whether to include seed users">
    Pick one:

    * **New users only** — The look-alike audience contains only matched population users. Use this for prospecting.
    * **New + original seed users** — Adds the seed users back into the output. Use this when you want a single audience that combines known customers with prospects.
  </Step>
</Steps>

## Step 5: Finalize the audience

<Steps>
  <Step title="Name the audience">
    Enter:

    * **Display name** — The human-readable name shown throughout the platform
    * **Unique name** — Auto-slugified from the display name and deduped against existing dataset names. Shown read-only.
    * **Description** — Optional context about the audience's purpose
    * **Tags** — Optional labels for organization
  </Step>

  <Step title="Review the configuration summary">
    The summary card recaps every choice you've made:

    * Seed audience
    * Population dataset
    * Similarity attributes selected
    * Output limit (size or score threshold)
    * Whether seed users are included

    If anything is wrong, return to the relevant step from the builder sidebar.
  </Step>

  <Step title="Save details">
    Click **Save details** to lock in the finalize step. The **Create** button becomes available once every step is complete.
  </Step>

  <Step title="Create the look-alike audience">
    Click **Create** to:

    1. Run the look-alike scoring query against the population
    2. Materialize the top-scoring users (and optionally the seed) as a new audience dataset
    3. Add the dataset to **My Audiences**

    The new audience can then be delivered to connectors from [Audience Studio](/guides/audience-studio/building-an-audience).
  </Step>
</Steps>

## After creation

A look-alike audience is a regular dataset, so you can:

* **Activate it** — Open it in [Audience Studio](/guides/audience-studio/building-an-audience) to deliver it to connectors
* **Tag or describe it** — Edit metadata from the dataset's detail view
* **Use it as a seed for another look-alike** — Chain models when you want to expand iteratively

## Related content

<CardGroup>
  <Card title="Audience Studio" icon="users" href="/concepts/data-activation/audience-studio">
    Build and activate audiences from your data
  </Card>

  <Card title="Building an audience" icon="user-plus" href="/guides/audience-studio/building-an-audience">
    Create the seed audiences look-alike models learn from
  </Card>

  <Card title="Rosetta Stone mappings" icon="language" href="/guides/rosetta-stone/mapping-schemas">
    Map identity and feature attributes so datasets can be matched and scored
  </Card>

  <Card title="Audience strategies" icon="sitemap" href="/guides/activation/audience-strategies">
    Structure underlying data for effective activation
  </Card>
</CardGroup>


# Building an audience
Source: https://docs.narrative.io/guides/audience-studio/building-an-audience

Create a targeted audience from your data and deliver it to advertising platforms using Audience Studio

This guide walks through building an audience in Audience Studio — from selecting a data source and applying filters to choosing delivery destinations and creating the audience.

<Info>
  For a conceptual overview of how Audience Studio works, see [Audience Studio](/concepts/data-activation/audience-studio). For strategies on how to structure your underlying data, see [Structuring Audiences for Activation](/guides/activation/audience-strategies).
</Info>

## Prerequisites

* A dataset ingested into the platform with [Rosetta Stone attribute mappings](/guides/rosetta-stone/mapping-schemas), including at least one deliverable identifier (such as `sha256_hashed_email` or `sha256_hashed_phone`)
* At least one [connector](/reference/connectors/index) installed with an active profile, if you want to deliver the audience to an external platform

## Step 1: Select a source

<Steps>
  <Step title="Open Audience Studio">
    Navigate to **My Audiences** in the sidebar, then click **Create Audience** or **Open Audience Studio**.
  </Step>

  <Step title="Choose a dataset">
    Select the dataset that contains the data you want to segment. Only datasets on the currently selected data plane are shown.

    The dataset must have Rosetta Stone attribute mappings — these determine which fields are available for filtering and which identifiers can be delivered to connectors.
  </Step>
</Steps>

## Step 2: Choose connectors

Select one or more destination platforms where you want to deliver the audience.

<Steps>
  <Step title="Review connector eligibility">
    Audience Studio checks whether your source dataset contains the identifier types required by each connector. Eligible connectors are highlighted; ineligible ones show a warning explaining which identifiers are missing.
  </Step>

  <Step title="Toggle destinations on">
    Enable the connectors you want to deliver to. You can select multiple destinations — the audience will be delivered to all of them.

    Common destinations include [The Trade Desk](/reference/connectors/the-trade-desk), [Meta](/reference/connectors/meta), [TikTok](/reference/connectors/tiktok), [Google DV360](/reference/connectors/google-dv360), [Pinterest](/reference/connectors/pinterest), and [Amazon S3](/reference/connectors/amazon-s3).
  </Step>
</Steps>

## Step 3: Apply filters

Filters narrow your source data to the audience you want. The available filter controls are determined by your dataset's Rosetta Stone attribute mappings.

<Steps>
  <Step title="Add a filter">
    Click **Add Filter** and select the attribute you want to filter on. The filter attribute dropdown groups options into **Rosetta Stone** attributes and **Dataset Columns** for clarity.
  </Step>

  <Step title="Configure the filter">
    Depending on the attribute type, you will see different controls:

    * **Enum fields** show a checkbox list or searchable dropdown of values
    * **Numeric fields** show a comparison operator selector and a value input
    * **Timestamp fields** offer curated ranges (last 7/30/90 days), custom lookback, or a date range picker
    * **Boolean fields** show a toggle
    * **Null checks** — select **Is null** or **Is not null** to filter on the presence or absence of a value
  </Step>

  <Step title="Combine multiple filters">
    Add additional filters as needed. By default, filters are combined with **AND** logic — all conditions must match. To use **OR** logic (any condition matches), group filters together.

    You can also add **frequency** constraints to any filter — for example, "purchased at least 3 times."
  </Step>

  <Step title="Forecast the audience size">
    Click **Calculate** (or enable auto-calculate) to preview how many records match your current filters. The forecast runs a preview query without creating the audience, helping you verify the size before committing.

    Suggested filter values show what percentage of your forecasted audience matches each option, helping you understand the impact of each filter.
  </Step>
</Steps>

## Step 4: Finalize and create

<Steps>
  <Step title="Set audience metadata">
    Configure the audience details:

    * **Name** (required) — Identifies the audience in the platform and destination systems
    * **Description** — Optional context about the audience's purpose
    * **Tags** — Optional labels for organization
    * **Refresh schedule** — Choose **One-time** for a static audience, or set a recurring schedule (daily, weekly, etc.) to keep the audience current as source data changes
  </Step>

  <Step title="Configure connector settings">
    Each selected connector shows its **quick settings** panel. These are platform-specific fields required for delivery — for example:

    * **The Trade Desk**: Audience name, advertiser selection, historical data delivery
    * **Meta**: Ad account selection
    * **TikTok**: Business Center and ad account selection

    Fill in the required fields for each connector. A green checkmark indicates the connector is fully configured and validated.
  </Step>

  <Step title="Create the audience">
    Click **Create Audience**. Audience Studio:

    1. Generates and executes an NQL query based on your filters
    2. Materializes the results as a new audience dataset
    3. Creates connector deliveries for each destination
    4. If a refresh schedule is set, configures incremental updates to keep the audience current

    The audience appears under **My Audiences** in the sidebar.
  </Step>
</Steps>

## After creation

Once your audience is created, you can:

* **Monitor delivery** — Check connector delivery status from the audience's Connections tab
* **Clone the audience** — Select **Open in Audience Studio** from the action menu to create a new audience pre-populated with the same source, filters, and settings
* **Add more connectors** — Deliver the same audience to additional destinations from the Connections tab
* **Adjust the refresh schedule** — Change how often the audience is rebuilt

## Related content

<CardGroup>
  <Card title="Audience Studio" icon="users" href="/concepts/data-activation/audience-studio">
    Conceptual overview of filter types, forecasting, and cloning
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Strategies for organizing data for activation
  </Card>

  <Card title="Connector Reference" icon="plug" href="/reference/connectors/index">
    Technical details and identifier requirements for each destination
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for upload and activation
  </Card>
</CardGroup>


# Using Narrative ID
Source: https://docs.narrative.io/guides/collaboration/using-narrative-id

Encode and translate identifiers for privacy-safe data collaboration

This guide walks through the practical steps for working with Narrative IDs—encoding your identifiers and translating them for partner collaboration.

## Prerequisites

Before working with Narrative IDs, ensure you have:

* Access to the Narrative platform with appropriate permissions
* An encoding key ID for your organization's encoding space
* Partner encoding key IDs if you need to translate for collaboration

***

## Encoding identifiers

Encoding converts a clear text identifier (such as an email or hashed email) into a Narrative ID within your organization's encoding space.

### When to encode

Encode identifiers when you want to:

* Share data with partners without exposing raw identifiers
* Create privacy-safe join keys for cross-organization matching
* Prepare data for collaboration workflows

### Using NARRATIVE\_ID\_ENCODE

The `NARRATIVE_ID_ENCODE` function creates a Narrative ID from a clear text identifier.

**Syntax:**

```sql theme={null}
NARRATIVE_ID_ENCODE(raw_identifier, key_id)
```

**Parameters:**

| Parameter        | Type   | Description                                                            |
| ---------------- | ------ | ---------------------------------------------------------------------- |
| `raw_identifier` | string | The clear text identifier to encode (email, hashed email, phone, etc.) |
| `key_id`         | string | Your organization's encoding key ID                                    |

**Example: Encoding email addresses**

```sql theme={null}
SELECT
  NARRATIVE_ID_ENCODE(LOWER(TRIM(email)), 'your_org_key_id') AS narrative_id,
  event_timestamp,
  event_type
FROM your_company."customer_events"
```

**Example: Encoding already-hashed identifiers**

If your data already contains hashed emails (SHA-256), you can still encode them:

```sql theme={null}
SELECT
  NARRATIVE_ID_ENCODE(email_sha256, 'your_org_key_id') AS narrative_id,
  purchase_amount,
  purchase_date
FROM your_company."transactions"
```

<Tip>
  Always normalize identifiers before encoding. For emails, use `LOWER(TRIM(email))` to ensure consistent matching.
</Tip>

### Creating a dataset with Narrative IDs

To prepare a dataset for sharing, encode the identifiers during a materialized view creation:

```sql theme={null}
CREATE MATERIALIZED VIEW your_company."customer_data_encoded" AS
SELECT
  NARRATIVE_ID_ENCODE(LOWER(TRIM(email)), 'your_org_key_id') AS narrative_id,
  age_bracket,
  region,
  interest_categories,
  event_timestamp
FROM your_company."raw_customer_data"
```

***

## Translating between partners

Translation converts a Narrative ID from one partner's encoding space to another, enabling cross-organization matching without exposing the underlying identifiers.

### When to translate

Translate Narrative IDs when you need to:

* Share data with a specific partner for matching
* Query across datasets from different partners
* Enable a partner to join your data with their own

### Using NARRATIVE\_ID\_TRANSLATE

The `NARRATIVE_ID_TRANSLATE` function converts a Narrative ID from its current encoding space to a target partner's encoding space.

**Syntax:**

```sql theme={null}
NARRATIVE_ID_TRANSLATE(narrative_id, target_key_id)
```

**Parameters:**

| Parameter       | Type   | Description                                                 |
| --------------- | ------ | ----------------------------------------------------------- |
| `narrative_id`  | string | An existing Narrative ID to translate                       |
| `target_key_id` | string | The encoding key ID for the target partner's encoding space |

**Example: Translating for a partner**

```sql theme={null}
SELECT
  NARRATIVE_ID_TRANSLATE(narrative_id, 'partner_b_key_id') AS partner_b_narrative_id,
  demographic_segment,
  activity_score
FROM your_company."audience_segments"
```

### Partner collaboration workflow

Here's a typical workflow for sharing data with a partner:

<Steps>
  <Step title="Encode your identifiers">
    Create Narrative IDs in your encoding space:

    ```sql theme={null}
    SELECT
      NARRATIVE_ID_ENCODE(email_sha256, 'your_org_key_id') AS narrative_id,
      customer_segment,
      lifetime_value
    FROM your_company."customers"
    ```
  </Step>

  <Step title="Translate for your partner">
    Convert your Narrative IDs to the partner's encoding space:

    ```sql theme={null}
    SELECT
      NARRATIVE_ID_TRANSLATE(narrative_id, 'partner_key_id') AS partner_narrative_id,
      customer_segment,
      lifetime_value
    FROM your_company."customers_encoded"
    ```
  </Step>

  <Step title="Share the translated data">
    The partner receives data with Narrative IDs in their encoding space. They can now join against their own data using their Narrative IDs.
  </Step>
</Steps>

### Joining across partner datasets

When querying data that spans multiple partners, use translation in the join condition:

```sql theme={null}
SELECT
  a.campaign_name,
  a.impressions,
  b.conversion_value
FROM partner_a."ad_impressions" a
JOIN partner_b."conversions" b
  ON NARRATIVE_ID_TRANSLATE(a.narrative_id, 'partner_b_key_id') = b.narrative_id
WHERE a.event_timestamp >= '2024-01-01'
```

***

## Working with match tables

Match tables are pre-generated lookup tables that pair Narrative IDs with clear text identifiers, enabling offline workflows.

### What match tables provide

* A static mapping between your clear text identifiers and their Narrative IDs
* The ability to work with Narrative IDs outside the Narrative platform
* Faster lookups for high-volume processing

### Generating a match table

Create a match table by selecting both the raw identifier and its encoded Narrative ID:

```sql theme={null}
CREATE MATERIALIZED VIEW your_company."identity_match_table" AS
SELECT DISTINCT
  email_sha256 AS clear_text_identifier,
  NARRATIVE_ID_ENCODE(email_sha256, 'your_org_key_id') AS narrative_id
FROM your_company."customer_identities"
```

### Using match tables

Once generated, you can export the match table for use in external systems:

```sql theme={null}
SELECT
  clear_text_identifier,
  narrative_id
FROM your_company."identity_match_table"
```

<Warning>
  Match tables contain mappings between clear text identifiers and Narrative IDs. Handle them with the same security precautions as any sensitive data.
</Warning>

***

## Best practices

### Normalization before encoding

Always normalize identifiers before encoding to ensure consistent matching:

| Identifier Type | Normalization                                           |
| --------------- | ------------------------------------------------------- |
| Email           | `LOWER(TRIM(email))`                                    |
| Phone           | E.164 format or `NORMALIZE_PHONE()`                     |
| Hashed email    | Already normalized (ensure consistent hashing upstream) |

### Performance considerations

* **Encode once, translate as needed**: Encode your raw identifiers once and store the Narrative IDs. Translate only when preparing data for a specific partner
* **Use materialized views**: For frequently accessed data, encode identifiers in a materialized view rather than at query time
* **Batch translations**: When sharing with multiple partners, consider creating partner-specific views

### Encoding vs. hashing

| Use Case                    | Approach                                   |
| --------------------------- | ------------------------------------------ |
| General pseudonymization    | Use standard hashing (SHA-256)             |
| Cross-partner collaboration | Use Narrative ID encoding                  |
| Internal data processing    | Either, depending on downstream use        |
| Regulatory compliance       | Narrative ID provides additional isolation |

***

## Troubleshooting

| Issue                        | Possible Cause                          | Solution                                                    |
| ---------------------------- | --------------------------------------- | ----------------------------------------------------------- |
| No matches after translation | Identifiers not normalized consistently | Ensure both partners normalize the same way before encoding |
| Invalid key\_id error        | Incorrect encoding key ID               | Verify the key ID with your Narrative administrator         |
| Translation returns NULL     | Source Narrative ID is invalid          | Check that the source was properly encoded                  |

***

## Related content

<CardGroup>
  <Card title="Narrative ID Concepts" icon="lightbulb" href="/concepts/security/narrative-id">
    Understand how Narrative ID works
  </Card>

  <Card title="NQL Functions Reference" icon="function" href="/nql/functions/index#narrative-specific-functions">
    Complete function syntax and parameters
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Hashing fundamentals for privacy
  </Card>
</CardGroup>


# Managing Data Subject Requests
Source: https://docs.narrative.io/guides/compliance/managing-dsrs

Submit and process opt-out and deletion requests to comply with GDPR, CCPA, and other privacy regulations

When you receive a data subject request (DSR) from an individual exercising their privacy rights, you must ensure that request is honored across all systems where their data exists—including any data you've shared with partners through Narrative. This guide shows you how to submit DSRs to the platform so that opted-out identifiers are automatically excluded from queries and propagated to downstream data collaborators.

<Note>
  For background on GDPR data subject rights and CCPA consumer rights, see [GDPR](/concepts/compliance/gdpr) and [CCPA](/concepts/compliance/ccpa).
</Note>

## Prerequisites

Before you begin, ensure you have:

* A dataset uploaded to Narrative
* The identifier(s) from the DSR (hashed email, MAID, cookie, or other unique identifier)
* Understanding of the request type (opt-out or data erasure)

***

## How DSR management works

When you submit a DSR to Narrative, the platform takes two automatic actions:

1. **Blocks future transactions**: The identifier is anti-joined from all NQL queries in your account, preventing the data subject's information from being queried, returned, or filtered on by any party.

2. **Propagates to data buyers**: If you've previously shared data containing this identifier with other organizations, Narrative notifies those buyers so they can comply with the request wherever they've already received the data.

<Info>
  As a data owner, regulatory frameworks like CCPA and GDPR hold you responsible for complying with privacy requests. Narrative's DSR management helps you fulfill your obligations to both the data subject and your downstream data collaborators.
</Info>

***

## Submitting a DSR

To register a privacy request with Narrative, create a dataset containing the opted-out identifiers and map it to the **Data Privacy Request Identifier** Rosetta Stone attribute.

### Step 1: Prepare your DSR data

Create a dataset with two fields:

| Field            | Description                                                                                                                       | Example values                         |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| **Identifier**   | The unique identifier from the DSR. Can be any identifier type: hashed email, mobile advertising ID (MAID), cookie, or custom ID. | `a1b2c3d4e5f6...` (SHA-256 email hash) |
| **Request type** | The type of privacy request received.                                                                                             | `opt_out` or `data_erasure`            |

Example CSV:

```csv theme={null}
identifier,request_type
06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3,opt_out
e1e8d3e4a336d4f9dc63b70a534ff10834471556,data_erasure
abc123def456,opt_out
```

### Step 2: Upload and create the dataset

<Steps>
  <Step title="Navigate to Datasets">
    In the Narrative UI, go to **Datasets** and click **Create Dataset**.
  </Step>

  <Step title="Upload your DSR file">
    Upload the CSV file containing your privacy request identifiers.
  </Step>

  <Step title="Define the schema">
    Map your columns to appropriate data types:

    * `identifier`: String
    * `request_type`: String (enum: `opt_out`, `data_erasure`)
  </Step>

  <Step title="Save and activate">
    Save the dataset and ensure it's activated.
  </Step>
</Steps>

### Step 3: Map to the Data Privacy Request Identifier attribute

The critical step is mapping your dataset to the **Data Privacy Request Identifier** Rosetta Stone attribute. This mapping tells the platform that identifiers in this dataset represent privacy requests that should be excluded from queries.

<Steps>
  <Step title="Open the Rosetta Stone tab">
    Navigate to your DSR dataset and click the **Rosetta Stone** tab.
  </Step>

  <Step title="Create a mapping">
    Click **Add Mapping** and select the `identifier` column.
  </Step>

  <Step title="Select the target attribute">
    Search for and select **Data Privacy Request Identifier** (attribute ID 232).
  </Step>

  <Step title="Save the mapping">
    Click **Create** to activate the mapping.
  </Step>
</Steps>

<Warning>
  The mapping to the Data Privacy Request Identifier attribute is what triggers the privacy enforcement. Without this mapping, your DSR dataset will be treated as regular data.
</Warning>

Once mapped, the platform automatically:

* Anti-joins these identifiers from all NQL queries in your account
* Propagates the requests to any organization that has previously purchased data containing these identifiers from you

***

## Setting a retention policy

For optimal data management and compliance, set a 30-day retention policy on datasets containing DSR identifiers. This approach:

* Promotes timely data deletion
* Aligns with many privacy regulation timelines (GDPR requires response within one month; CCPA within 45 days)
* Prevents indefinite accumulation of DSR records

<Tip>
  Always tailor your retention policy to the specific regulations of your jurisdiction. Consult your legal team for guidance on appropriate retention periods.
</Tip>

***

## For data buyers

If you subscribe to data containing user identifiers from other organizations, Narrative automatically subscribes you to receive relevant DSRs from those suppliers.

### Where to find DSRs

Privacy requests passed to you appear in **My Data** under **Purchased Data**. These requests include:

* The identifier that was opted out or requested for deletion
* The request type (`opt_out` or `data_erasure`)
* The supplier who originated the request

### Your compliance responsibility

When you receive a DSR from a supplier:

1. **Review the request**: Identify the identifier and request type
2. **Locate the data**: Find any records in your systems that match the identifier
3. **Take appropriate action**:
   * For `opt_out`: Remove the identifier from marketing communications and targeted advertising
   * For `data_erasure`: Delete all personal data associated with the identifier
4. **Document compliance**: Maintain records of your response for audit purposes

<Warning>
  You are responsible for complying with DSRs in systems where you have already exfiltrated data from Narrative. The platform blocks future queries, but you must handle data that has already been exported.
</Warning>

***

## Request types

Narrative supports two types of privacy requests:

| Request type     | Value          | Description                                                    | Typical use case                                  |
| ---------------- | -------------- | -------------------------------------------------------------- | ------------------------------------------------- |
| **Opt-out**      | `opt_out`      | Data subject requests removal from marketing or advertising    | CCPA "Do Not Sell" requests, GDPR right to object |
| **Data erasure** | `data_erasure` | Data subject requests complete deletion of their personal data | GDPR right to erasure, CCPA deletion requests     |

Both request types result in the identifier being excluded from future queries. The distinction helps you and your data collaborators take the appropriate compliance action.

***

## Troubleshooting

| Issue                                       | Cause                                                           | Solution                                                        |
| ------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- |
| Identifier still appearing in query results | Dataset not mapped to Data Privacy Request Identifier attribute | Verify the Rosetta Stone mapping is active                      |
| DSR not propagating to buyers               | Mapping was created after data was already transacted           | Contact support—the platform will backfill notifications        |
| Invalid request type error                  | Request type value doesn't match expected enum                  | Use exactly `opt_out` or `data_erasure` (lowercase, underscore) |
| Buyers not receiving DSRs                   | No prior transactions on the identifier                         | DSRs only propagate when there's a transaction history          |

***

## Related content

<CardGroup>
  <Card title="GDPR" icon="shield-check" href="/concepts/compliance/gdpr">
    Understand GDPR data subject rights
  </Card>

  <Card title="CCPA" icon="scale-balanced" href="/concepts/compliance/ccpa">
    Understand CCPA consumer rights
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Learn how to create Rosetta Stone mappings
  </Card>

  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Privacy-preserving data collaboration techniques
  </Card>
</CardGroup>


# Ad Account Eligibility for Custom Audiences
Source: https://docs.narrative.io/guides/connectors/meta/ad-account-access

Resolve eligibility warnings shown when selecting a Meta ad account for Custom Audience delivery

When you select an ad account while creating a Meta Connector profile, Narrative checks whether the account is eligible for Custom Audience delivery. Accounts that may not be eligible are flagged with a warning, but you can still select them and complete profile creation.

<Info>
  You can proceed with an ad account that shows an eligibility warning. If you have already confirmed eligibility outside of Narrative — for example, by accepting the Terms of Service in another browser session — delivery may still succeed. If the underlying issue is not resolved on the Meta side, deliveries to that ad account will fail.
</Info>

This page covers the eligibility warnings you may see and how to resolve the underlying issue on Meta's side so that deliveries succeed reliably.

## Custom Audience Terms of Service not accepted

**Warning shown:** *The business associated with the ad account must accept the Custom Audience Terms of Service.*

**Cause:** The Business Manager that owns the ad account hasn't accepted Meta's Custom Audiences Terms of Service.

The warning includes a **Learn more** link that opens the Terms of Service page for that specific ad account.

<Steps>
  <Step title="Open the Terms of Service link">
    Click **Learn more** on the warning, or open the URL directly:

    ```
    https://business.facebook.com/ads/manage/customaudiences/tos/?act=YOUR_ACCOUNT_ID
    ```

    Replace `YOUR_ACCOUNT_ID` with your Meta ad account ID (numbers only, without the `act_` prefix).
  </Step>

  <Step title="Accept the Terms of Service">
    Review and accept the Terms of Service. A Business Manager admin must perform this step.
  </Step>

  <Step title="Refresh the connector">
    Return to the Narrative Meta Connector and re-authenticate, or reload the ad account list. The eligibility warning should clear.
  </Step>
</Steps>

## Ad account is not associated with a business

**Warning shown:** *Only ad accounts associated with businesses are eligible for use with custom audiences.*

**Cause:** The ad account is a personal ad account that has not been claimed by a Meta Business Manager. Custom Audiences require a Business Manager.

<Steps>
  <Step title="Add the ad account to a Business Manager">
    In [Meta Business Suite](https://business.facebook.com/), open **Business Settings** > **Ad Accounts** and add or claim the ad account.
  </Step>

  <Step title="Accept the Custom Audience Terms of Service">
    Once the ad account is associated with a business, accept the Custom Audience Terms of Service for it (see the section above).
  </Step>

  <Step title="Re-authenticate the profile">
    Return to Narrative and re-authenticate the Meta Connector profile so it picks up the updated business association.
  </Step>
</Steps>

<Note>
  Ads datasets are only available for profiles whose ad account is associated with a Business. Profiles whose ad account is not associated with a Business will not show an ads datasets table on the profile details page.
</Note>

## Ad account does not support custom audiences

**Warning shown:** *This ad account does not support custom audiences.*

**Cause:** Meta has flagged the ad account as ineligible for Custom Audiences for a reason that is not specific to Terms of Service or business association. Common causes include:

* The Business Manager has not completed business verification.
* Your role on the ad account is below **Advertiser** or **Ad Account Admin**.
* The ad account is restricted, disabled, or under review by Meta.

### Verify your role permissions

You must have at least **Advertiser** or **Ad Account Admin** role on the ad account to create Custom Audiences.

<Steps>
  <Step title="Contact your Business Manager admin">
    Ask them to verify your role on the ad account.
  </Step>

  <Step title="Request elevated permissions">
    If you don't have sufficient access, request **Advertiser** or **Ad Account Admin** permissions.
  </Step>

  <Step title="Update permissions in Business Manager">
    In Meta Business Manager, roles can be updated under **Ad Accounts** > select your ad account > **People** > edit user permissions.
  </Step>
</Steps>

### Complete business verification

<Steps>
  <Step title="Access Business Verification">
    Log into Meta Business Manager and navigate to **Business Settings** > **Security Center** > **Business Verification**.
  </Step>

  <Step title="Complete the verification process">
    Follow Meta's prompts to verify your business. This process may take several days.
  </Step>

  <Step title="Re-authenticate the profile">
    Once verification is complete, re-authenticate the Meta Connector profile.
  </Step>
</Steps>

<Note>
  Business verification is a Meta requirement, not a Narrative requirement. Learn more in [Meta's Business Help Center](https://www.facebook.com/business/help/2058515294227817).
</Note>

### Confirm ad account association

<Steps>
  <Step title="Confirm the correct Business Manager">
    Verify that you're logged into the correct Meta Business Manager account.
  </Step>

  <Step title="Check ad account assignment">
    In Business Manager, go to **Business Settings** > **Ad Accounts** and confirm that the ad account is listed and assigned to the correct Business Manager.
  </Step>

  <Step title="Add or request access if missing">
    If the ad account is missing, request access from its owner or add it to your Business Manager.
  </Step>
</Steps>

## Additional troubleshooting

If the warning persists after resolving the underlying issue:

* **Re-authenticate the profile** — Eligibility flags are read at authentication time. Re-authenticate to refresh them.
* **Clear browser cache** — Clear your browser cache and cookies, then try again.
* **Try a different browser** — Some browser extensions may interfere with the Meta authentication flow.
* **Check Meta status** — Verify that Meta's systems are operational at [Meta Status](https://metastatus.com/).
* **Wait and retry** — Eligibility changes on Meta's side can take several minutes to propagate.

## Getting support

If you continue to experience issues, contact [Narrative support](mailto:support@narrative.io) with:

* Your Meta ad account ID
* The Business Manager ID you're using
* Screenshots of the eligibility warning
* Confirmation of the steps you've already tried

***

## Related content

<CardGroup>
  <Card title="Meta Connector" icon="meta" href="/reference/connectors/meta">
    Reference documentation for the Meta Connector
  </Card>

  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for Meta
  </Card>
</CardGroup>


# Archive a Compute Pool
Source: https://docs.narrative.io/guides/data-planes/archive-compute-pool

Retire a compute pool safely by clearing its references and understanding the runtime impact on in-flight jobs

Archiving a compute pool removes it from selection without permanently deleting it. The archive flow behaves differently on Snowflake and AWS data planes, and in-flight jobs that still reference the pool are failed by the operator on their next iteration. This guide walks through the order in which to clear references so the archive succeeds and your jobs report meaningful errors.

## Prerequisites

* A Narrative API token with **Compute Pools** Read and Write permissions ([manage API keys](/account-settings/api-keys))
* The `computePoolId` of the pool you want to archive
* For AWS data planes: knowledge of any datasets or companies that pin this pool as a default

## When archive rejects

On AWS data planes, the API rejects the archive when the pool is still:

* The **data plane default** — the level-4 fallback when no other level supplied a pool
* A **dataset default** — pinned to one or more datasets as their default compute pool (level 2)
* A **company default** — pinned by one or more companies as their per-data-plane default (level 3)

When this happens, the API returns a `CannotArchiveDefault` error describing every reference that would dangle. Clear those references first, then retry.

On Snowflake data planes, all three reference paths are reconciled automatically: the platform re-elects another active pool as the new data plane default (or clears it if none remain) and clears the dataset and company defaults on every affected entity before archiving. You can skip the manual steps below and call the archive endpoint directly.

<Info>
  Snowflake users frequently mutate their warehouse list outside the platform. The Snowflake archive flow is permissive so the platform can track reality when a backing warehouse disappears. See [Compute Pools › Archiving compute pools](/concepts/primitives/compute-pools#archiving-compute-pools) for the rationale.
</Info>

## Archive on an AWS data plane

<Steps>
  <Step title="Reassign or clear the data plane default">
    If the compute pool you want to archive is the data plane default, update the data plane to point at a different active compute pool, or clear the default entirely. After this step, jobs that do not specify a `computePoolId` either resolve to the new default or fail with an actionable error at job creation.
  </Step>

  <Step title="Clear the pool from every dataset that pins it">
    Update each affected dataset's `computePoolConfig.defaultComputePoolId` to a different pool or to `null`. Datasets whose default is cleared fall through to the rest of the resolution chain on their next job.

    <Tip>
      Dataset defaults are a user-specific choice. The platform does not auto-pick a replacement on AWS data planes — you decide whether to repoint each dataset or let it fall through.
    </Tip>
  </Step>

  <Step title="Clear the pool from every company that pins it as a per-data-plane default">
    If the pool is set as a company's default for this data plane, clear or repoint it. A company admin clears the default with:

    ```bash theme={null}
    curl -X DELETE \
      -H "Authorization: Bearer $NARRATIVE_API_TOKEN" \
      https://api.narrative.io/company/<companyId>/data-planes/<dataPlaneId>/default-compute-pool
    ```

    or repoints it to a different active pool with:

    ```bash theme={null}
    curl -X PUT \
      -H "Authorization: Bearer $NARRATIVE_API_TOKEN" \
      https://api.narrative.io/company/<companyId>/data-planes/<dataPlaneId>/default-compute-pool/<newPoolId>
    ```

    Both endpoints return `204 No Content` on success and require the **Company Info** write permission.
  </Step>

  <Step title="Call the archive endpoint">
    Send a `DELETE` request to `/compute-pools/{id}`. The pool's status changes to `archived` and it stops appearing in list responses.

    ```bash theme={null}
    curl -X DELETE \
      -H "Authorization: Bearer $NARRATIVE_API_TOKEN" \
      https://api.narrative.io/compute-pools/<id>
    ```
  </Step>

  <Step title="Decide what to do with in-flight jobs">
    The data plane operator fails any job whose compute pool resolves to a non-`active` status on its next polling iteration. If you do not want in-flight jobs to fail, wait until they reach a terminal state before archiving, or resubmit them against a different active pool first.
  </Step>
</Steps>

## What happens to in-flight jobs

After archive, the operator treats the pool as unresolvable for every `Pending`, `Running`, and `PendingCancellation` job that referenced it. The job is reported as failed with a message similar to:

```text theme={null}
compute pool '<id>' could not be resolved (it may have been archived or does not exist).
Resubmit the job with an active compute pool.
```

Jobs in `PendingCancellation` are failed rather than reported as cancelled. The missing pool is the actionable signal users need to see — a successful cancellation report would hide the underlying cause. From the user's perspective the job is terminal either way, so failing it is the safe and informative choice.

## Verifying the archive

The pool's record stays in the database, so `GET /compute-pools/{id}` still returns it with `status: archived`. The pool is filtered out of `GET /compute-pools` list responses by default. Job creation requests that try to resolve to an archived pool fail validation at the API layer; existing jobs that resolved to it at submission time fail at runtime as described above.

## Related content

<CardGroup>
  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    How compute pools work and why archive behavior diverges by provider
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where data lives and how the operator processes jobs
  </Card>

  <Card title="Migrate to Compute Pools" icon="arrow-right" href="/guides/data-planes/migrate-to-compute-pools">
    Transition from a single Snowflake warehouse to compute pools
  </Card>

  <Card title="API keys" icon="key" href="/account-settings/api-keys">
    Manage API tokens with the permissions needed to archive pools
  </Card>
</CardGroup>


# Migrate to Compute Pools
Source: https://docs.narrative.io/guides/data-planes/migrate-to-compute-pools

Map your Snowflake virtual warehouses to Narrative Compute Pools to use multiple warehouses for different job types

If your Snowflake data plane was set up before compute pools were available, it uses a single virtual warehouse for all workloads. Migrating to compute pools lets you register one or more Snowflake warehouses, giving you control over which warehouse handles each type of work.

<iframe title="Configure Snowflake Warehouse Access for Narrative Data Collaboration" />

## Why migrate

With a single default warehouse, every query—exploratory, production, audience builds—competes for the same resources. Compute pools solve this by letting you assign different warehouses to different workloads:

* **Right-size compute for the job.** Use a small warehouse for quick lookups and a large one for heavy processing, rather than over-provisioning a single warehouse for all cases.
* **Isolate workloads.** Prevent an expensive production pipeline from slowing down ad-hoc exploration, or vice versa.
* **Control costs.** Smaller warehouses for lightweight work cost less per credit-second than running everything on a large warehouse.
* **Set collaboration policies.** Control which companies in your data plane can use which warehouses, so shared environments stay predictable.

## Prerequisites

* ACCOUNTADMIN role or equivalent in your Snowflake account
* Access to the Narrative Snowflake Native App in your account
* One or more Snowflake warehouses you want to register as compute pools
* A Narrative API token with **Compute Pools** Read and Write permissions ([manage API keys](/account-settings/api-keys))

## Migration steps

<Steps>
  <Step title="Register warehouses in the native app">
    1. Open the Narrative Data Collaboration app configuration screen in Snowflake.
    2. Expand the **Warehouse Management** section.
    3. Click **Select Warehouses to Register** and select the warehouses you want to use as compute pools.

    Then, grant the app usage on each selected warehouse by running the following in a Snowflake worksheet:

    ```sql theme={null}
    GRANT USAGE ON WAREHOUSE <warehouse_name> TO APPLICATION NARRATIVE_DATA_COLLABORATION;
    ```

    For example, to register two warehouses:

    ```sql theme={null}
    GRANT USAGE ON WAREHOUSE ANALYTICS_SMALL TO APPLICATION NARRATIVE_DATA_COLLABORATION;
    GRANT USAGE ON WAREHOUSE PRODUCTION_XLARGE TO APPLICATION NARRATIVE_DATA_COLLABORATION;
    ```

    Each granted warehouse is automatically registered as a compute pool on your data plane. You can grant access to your existing warehouse plus any additional warehouses you want to add.
  </Step>

  <Step title="Verify compute pools appear">
    In the [Narrative Platform UI](https://app.narrative.io/platform/), navigate to **Settings > Data Planes**, select your Snowflake data plane, and open the **Compute Pools** tab. You should see each registered warehouse listed as a compute pool.
  </Step>

  <Step title="Set a default compute pool">
    From the Compute Pools tab, open the action menu (`···`) on the warehouse you want as your default and select **Set as default**. This warehouse is used whenever a query doesn't specify a compute pool.
  </Step>

  <Step title="Select compute pools in the context selector">
    The [context selector](/guides/platform/using-the-context-selector) now shows your registered compute pools in the second column. Select the appropriate warehouse for your current workload. The previous static "DEFAULT" label is replaced by your registered pools.
  </Step>
</Steps>

## What changes after migration

| Before                                                | After                                                               |
| ----------------------------------------------------- | ------------------------------------------------------------------- |
| Single warehouse for all workloads                    | One or more warehouses, each as a named compute pool                |
| No choice in the context selector                     | Context selector populates with your registered warehouses          |
| Warehouse configuration managed entirely in Snowflake | Basic settings visible and editable from the data plane detail page |

The legacy single-warehouse configuration is removed once you complete the migration. All queries going forward use the compute pool model.

## Related content

<CardGroup>
  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    Understand compute pool types and when to use each
  </Card>

  <Card title="Using the Context Selector" icon="hand-pointer" href="/guides/platform/using-the-context-selector">
    Choose your compute pool and other execution context settings
  </Card>

  <Card title="Snowflake Native App Installation" icon="snowflake" href="/guides/data-planes/snowflake-native-app-installation">
    Full installation guide for new Snowflake data planes
  </Card>
</CardGroup>


# Install the Snowflake Native App
Source: https://docs.narrative.io/guides/data-planes/snowflake-native-app-installation

Set up a customer-hosted data plane in your Snowflake environment

The Snowflake Native App installs a customer-hosted [data plane](/concepts/primitives/data-planes) directly in your Snowflake environment. Once installed, it operates as a data plane operator—polling Narrative's control plane for jobs and executing queries against your Snowflake data without it ever leaving your account.

## Prerequisites

* Snowflake ACCOUNTADMIN role or equivalent privileges

* A Narrative account ([create one here](/getting-started/create-account))

* A Narrative API token with the following permissions ([manage API keys](/account-settings/api-keys)):

  | Permission      | Access level   |
  | --------------- | -------------- |
  | Compute Pools   | Read and Write |
  | Data Planes     | Read and Write |
  | Data Plane Logs | Read and Write |
  | Datasets        | Read and Write |
  | Jobs            | Read and Write |
  | Models          | Read           |

* A Snowflake virtual warehouse

* Access to the [Narrative Data Collaboration listing](https://app.snowflake.com/marketplace/listing/GZTSZI0HICH/narrative-inc-narrative-data-collaboration) on the Snowflake Marketplace

* At least one Snowflake table or view to register as a dataset

## Installation steps

<Steps>
  <Step title="Install the Snowflake Native App">
    Find and install the [Narrative Data Collaboration](https://app.snowflake.com/marketplace/listing/GZTSZI0HICH/narrative-inc-narrative-data-collaboration) app from the Snowflake Marketplace. Click **Get** and follow the prompts to install it in your Snowflake account.
  </Step>

  <Step title="Grant privileges">
    Click **Grant Privileges** to grant the app the following privileges:

    * `EXECUTE TASK`
    * `CREATE COMPUTE POOL`
    * `BIND SERVICE ENDPOINT`

    These allow the app to schedule jobs, provision compute resources, and communicate with Narrative's control plane.
  </Step>

  <Step title="Register compute warehouses">
    In the app configuration screen, expand **Warehouse Management** and click **Select Warehouses to Register**. Select the warehouses you want to use as compute pools.

    Then, grant the app usage on each selected warehouse by running the following in a Snowflake worksheet:

    ```sql theme={null}
    GRANT USAGE ON WAREHOUSE <warehouse_name> TO APPLICATION NARRATIVE_DATA_COLLABORATION;
    ```

    Each granted warehouse is registered as a [compute pool](/concepts/primitives/compute-pools) on your data plane, giving you control over which warehouse handles each workload. You can register additional warehouses later and manage them from the **Compute Pools** tab on the data plane detail page.

    <Tip>
      Start with a smaller warehouse and scale up based on your data volume and query complexity.
    </Tip>
  </Step>

  <Step title="Configure external access integration">
    1. Click **Setup External Access Integration**.
    2. Click **Next**.
    3. Paste your Narrative API token under **Secret value**.

    This allows the app to authenticate with Narrative's control plane.
  </Step>

  <Step title="Select a source table or view">
    Select the Snowflake table or view you want to register as a dataset in Narrative.

    <Note>
      Narrative is granted **read-only** access to the selected object. You can change or add additional tables later.
    </Note>
  </Step>

  <Step title="Name your dataset">
    Set the **DATASET UNIQUE NAME** for your registered dataset.

    <Note>
      Dataset names must use uppercase alphanumeric characters and underscores only (e.g., `MY_DATASET_NAME`).
    </Note>
  </Step>

  <Step title="Upload sample data (optional)">
    Upload a sample of your data for display in the Narrative UI. This helps collaborators understand the structure and content of your dataset. You can skip this step and upload sample data later from within the platform.
  </Step>

  <Step title="Upload statistics (optional)">
    Upload dataset statistics for display in the Narrative UI. Statistics help collaborators evaluate your dataset. You can skip this step and upload statistics later from within the platform.
  </Step>
</Steps>

## What happens next

The app now operates as a data plane [operator](/concepts/primitives/data-planes)—it continuously polls Narrative's control plane for pending jobs and executes NQL queries directly in your Snowflake environment. Your data stays within your Snowflake security perimeter throughout this process.

Your registered warehouses appear as compute pools on the data plane detail page. You can set a default compute pool, add or remove warehouses, and configure collaboration policies from the **Compute Pools** tab. To choose which compute pool runs a specific query, use the [context selector](/guides/platform/using-the-context-selector).

<Note>
  If your data plane was set up before compute pools were available, see [Migrate to Compute Pools](/guides/data-planes/migrate-to-compute-pools) to transition from the legacy single-warehouse model.
</Note>

## Recommended: Enable AI features

To enable optional Narrative features that use advanced AI models hosted by Snowflake Cortex (such as Ask Rosetta), run the following grants after the app is installed:

```sql theme={null}
GRANT DATABASE ROLE snowflake.cortex_user TO APPLICATION NARRATIVE_DATA_COLLABORATION;

ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION';
```

The first grant allows the app to call Cortex AI models. The second enables cross-region Cortex access, which is required if your Snowflake account is in a region where certain models are not natively available.

***

## Related content

<CardGroup>
  <Card title="Snowflake Native App Reference" icon="snowflake" href="/reference/integrations/snowflake-native-app">
    Capabilities, requirements, and configuration details
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understand how data planes and operators work
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Manage API tokens and permissions
  </Card>

  <Card title="Execution Context" icon="circle-play" href="/concepts/primitives/execution-context">
    Learn how queries are executed across data planes
  </Card>
</CardGroup>


# Building a marketplace match report
Source: https://docs.narrative.io/guides/graph-studio/building-a-marketplace-match-report

Configure and generate a Marketplace Match Report to evaluate a third-party supplier against your first-party identity data

A [Marketplace Match Report](/guides/graph-studio/viewing-marketplace-match-reports) measures how well a supplier's data overlaps with — and enriches — your first-party identity data. This guide walks through the five-step builder that produces a report.

<Info>
  Marketplace Match Reports run on Snowflake data planes. The builder generates a workflow that materializes the report as a dataset; the dataset appears in the Reports listing once the workflow's final step completes.
</Info>

## Prerequisites

* A first-party **edge dataset** built with [Graph Studio](/guides/graph-studio/building-an-identity-graph). The match report uses these edges as its starting inventory of identifiers.
* An [access rule](/concepts/primitives/access-rules) granting you access to a supplier dataset that has been mapped to Rosetta Stone identifier attributes and (optionally) demographic attributes.
* A Snowflake data plane selected in the data plane switcher.

## Open the builder

Navigate to **My Data > Reports** and click **New report**. The builder loads with a five-step left rail:

1. **Source dataset** — your first-party edge dataset.
2. **Supplier dataset** — the third-party access rule you are evaluating.
3. **Match identifiers** — which identifier types to match on.
4. **Supplier enrichment** — which identifiers and demographic attributes to pull from the supplier for matched people.
5. **Finalize** — name and schedule the report.

You can move forward only when the current step is valid. You can return to any earlier step at any time; downstream selections are preserved when they remain valid.

## Step 1: Select a source dataset

Pick the edge dataset that holds the customers you want to evaluate. Only datasets eligible to act as a graph source — those with Rosetta Stone identifier mappings and statistics that the report can read — appear in the picker.

<Tip>
  A match report requires **exactly one** source dataset. Selecting a different dataset replaces the current selection rather than adding to it.
</Tip>

## Step 2: Select a supplier dataset

Choose the access rule for the supplier you want to evaluate. Only access rules that the report builder can join against your source dataset appear in the list. The builder shows an inline message when a supplier is ineligible — typically because it has no overlapping identifier types with your source.

You can select **one** supplier access rule per report. To compare multiple suppliers, build one report per supplier.

## Step 3: Choose match identifiers

Match identifiers determine which identifier types the report uses to match your people to the supplier. The grid lists every identifier type your source dataset has in common with the supplier; all are selected by default.

Deselect a type to exclude it from matching. The match rate the report reports is the share of your people who matched on **any** selected identifier type, so a broader selection generally raises the match rate but may include lower-confidence matches.

<Note>
  You must keep at least one identifier type selected to continue.
</Note>

## Step 4: Configure supplier enrichment

Supplier enrichment defines what the report pulls from the supplier for the people who matched in step 3. Two selections live on this step:

* **New identifiers** — additional identifier types the supplier contributes for your matched people (e.g., a hashed email the supplier has that you do not). These drive the **Enrichment** card and the **ID Enrichment** KPI.
* **Demographic and behavioral attributes** — categorical attributes from the supplier (e.g., gender, education level, income range). Each attribute you select becomes a panel in the [Demographic & Behavioral Enrichment](/guides/graph-studio/viewing-marketplace-match-reports#demographic--behavioral-enrichment) section of the rendered report.

Only attributes the supplier has mapped to Rosetta Stone are available. Selecting nothing is allowed — the report will still produce match metrics and identifier enrichment without demographic panels.

## Step 5: Finalize the report

Name the report, optionally describe it, and choose a refresh schedule.

<Steps>
  <Step title="Set the unique name">
    The unique name is the NQL-safe identifier for the generated dataset. The builder uppercases the name on blur and surfaces a conflict warning if another dataset already uses the same name.
  </Step>

  <Step title="Set the display name and description">
    The display name is what readers see in the Reports listing and at the top of the rendered report. If you leave the display name blank, the builder copies the unique name into it when you finish editing.
  </Step>

  <Step title="Choose a refresh schedule">
    The default is **Daily**. Match reports are [materialized views](/concepts/nql/materialized-views) under the hood, so the schedule controls how often the underlying workflow re-runs and the report's numbers update.
  </Step>

  <Step title="Review the workflow preview">
    The collapsible workflow preview shows the multi-step workflow the builder will submit — including the intermediate datasets it creates and the final materialized view that backs the report. Use this to sanity-check what will run on your data plane.
  </Step>

  <Step title="Submit">
    Click **Build report**. The builder submits the workflow and returns you to the Reports listing.
  </Step>
</Steps>

## After submission

The report appears in the Reports listing immediately. While the producing workflow runs, the row shows the workflow's run state (pending, running, or failed) and is not yet clickable. Once the workflow's final step has materialized the dataset, the row becomes a link to the rendered report.

<Tip>
  If the workflow fails, open the run from the listing to see step-level errors. Re-submitting the builder with the same unique name will conflict with the failed workflow's dataset — change the unique name or archive the failed dataset before retrying.
</Tip>

## Related

* [Viewing a marketplace match report](/guides/graph-studio/viewing-marketplace-match-reports) — how to read the rendered report once the workflow finishes
* [Building an identity graph](/guides/graph-studio/building-an-identity-graph) — produce the edge dataset that feeds the report's source step
* [Access rules](/concepts/primitives/access-rules) — how supplier datasets become available to the builder


# Building an identity graph
Source: https://docs.narrative.io/guides/graph-studio/building-an-identity-graph

Create an identity graph from your first-party data using Graph Studio

This guide walks through building an identity graph from a CRM dataset using Graph Studio. By the end, you will have resolved duplicate customer records into unified identities based on shared emails and phone numbers.

<Info>
  Graph Studio runs on both Snowflake and AWS data planes. For a conceptual overview, see [Graph Studio](/concepts/identifiers/graph-studio).
</Info>

## Prerequisites

* A dataset with [Rosetta Stone attribute mappings](/guides/rosetta-stone/mapping-schemas) for a unique identifier and at least one identity attribute (e.g., normalized email, phone number)
* The dataset must be eligible as a Graph Studio source — the Edge Builder only lists datasets tagged for Graph Studio, and the Graph Builder only lists edges datasets. See [Source eligibility](/concepts/identifiers/graph-studio#source-eligibility) if a source is missing from a picker.
* A Snowflake or AWS data plane

## Example dataset

This guide uses a CRM dataset called `OFFICE_CRM`:

| CUSTOMER\_ID | FIRST\_NAME | LAST\_NAME | EMAIL                                                                     | PHONE          |
| ------------ | ----------- | ---------- | ------------------------------------------------------------------------- | -------------- |
| CRM-001      | Michael     | Scott      | [michael.scott+rewards@gmail.com](mailto:michael.scott+rewards@gmail.com) | (570) 555-1234 |
| CRM-002      | Dwight      | Schrute    | [Dwight.Schrute@gmail.com](mailto:Dwight.Schrute@gmail.com)               | 15705552345    |
| CRM-003      | Jim         | Halpert    | [j.im.halpert@gmail.com](mailto:j.im.halpert@gmail.com)                   | 5705553456     |
| CRM-004      | Michael     | Scott      | [michael.scott@dundermifflin.com](mailto:michael.scott@dundermifflin.com) | (570) 555-9876 |
| CRM-005      | Pam         | Beesly     | [pam.beesly@gmail.com](mailto:pam.beesly@gmail.com)                       | (570) 555-4567 |
| CRM-006      | Michael     | Scott      | [michael.scott+rewards@gmail.com](mailto:michael.scott+rewards@gmail.com) | (570) 555-9876 |

Michael Scott appears three times with different email and phone combinations. The goal is to resolve all three records into a single identity.

The dataset is mapped to Rosetta Stone attributes for **Unique Identifier** (using `CUSTOMER_ID`), **Normalized Email**, **Clear Text E.164 Phone Number**, and **Person Name**.

## Step 1: Build edges

Edges define how records connect to each other through shared identifiers. Navigate to **My Data > Graph Studio** and select the **Edge builder** tab.

<Steps>
  <Step title="Add a source dataset">
    Click **Select Sources** and choose your dataset. Set a **source ID type** (a label like `OFFICE_CRM` that identifies this system) and choose the **source ID** field (`CUSTOMER_ID`).
  </Step>

  <Step title="Choose target IDs">
    Target IDs are the Rosetta Stone attributes that serve as connection points. When two records share the same target ID value, the graph connects them.

    Target IDs are grouped — each group acts as a single connection type. For this example, add two target ID groups:

    1. **Normalized Email** — connects records that share the same email address
    2. **Clear Text E.164 Phone Number + Person Name > first\_name** — connects records that share both the same phone number and first name. Combining these into one group means both values must match for a connection, which is more precise than matching on phone alone.
  </Step>

  <Step title="Finalize and build">
    Name the edge dataset (e.g., `office_crm_edges`) and click **Build Edges**.
  </Step>
</Steps>

## Step 2: Build the graph

Switch to the **Graph builder** tab. This takes your edges and runs a connected components algorithm to discover which records belong to the same person.

<Steps>
  <Step title="Select input sources">
    Click **Select Sources** and choose the edges dataset you just created.
  </Step>

  <Step title="Review algorithm parameters">
    The defaults work well for most use cases. You can adjust **max component size** (caps how many records can merge into one identity), **max iterations**, and **max degree threshold** (excludes overly-connected nodes like shared corporate emails) if needed later.

    Optionally, use **Exclusive attributes** to pick one or more target ID types that must hold a single value per identity (for example, an SSN token or exact date of birth). After the connected-components pass converges, any component whose vertices disagree on a declared attribute is split apart, targeting known overmerge cases without dropping legitimate connections. Only first-party target ID types present on your input edges are selectable; leaving the field empty preserves the default behavior.
  </Step>

  <Step title="Finalize and build">
    Name the graph (e.g., `office_crm_graph`), choose a refresh schedule, and click **Build Graph +**.
  </Step>
</Steps>

## Results

The graph resolves the six CRM records into four identities:

| Identity   | Records                   | How they connected                                                                                                                |
| ---------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Identity 1 | CRM-001, CRM-004, CRM-006 | All three Michael Scott records — CRM-001 and CRM-006 share the same email; CRM-004 and CRM-006 share the same phone + first name |
| Identity 2 | CRM-002                   | Dwight Schrute                                                                                                                    |
| Identity 3 | CRM-003                   | Jim Halpert                                                                                                                       |
| Identity 4 | CRM-005                   | Pam Beesly                                                                                                                        |

Michael Scott's three records are resolved into a single identity even though no two records share both the same email and phone — the graph follows transitive connections across shared values to link them together.

Dwight, Jim, and Pam remain as separate identities because they have no overlapping identifiers with other records in this dataset.

## Next steps

* **Add third-party data** — Include an access rule as an additional source in the Edge Builder to introduce connections your first-party data cannot see on its own. See [Graph Enrichment](/concepts/identifiers/graph-enrichment).
* **Set up recurring builds** — Use a refresh schedule to keep your graph current as new data arrives.


# Viewing a marketplace match report
Source: https://docs.narrative.io/guides/graph-studio/viewing-marketplace-match-reports

Understand how to read and interpret the Marketplace Match Report to evaluate supplier overlap and enrichment potential

The Marketplace Match Report measures how your identity data connects with a third-party supplier's data. It quantifies three things: how many of your customers the supplier can identify, what new identifiers the supplier adds, and what demographic attributes the supplier has for your matched customers.

This guide explains how to read a rendered report. To configure and generate a new report, see [Building a marketplace match report](/guides/graph-studio/building-a-marketplace-match-report).

Navigate to **My Data > Reports** and click a completed report to view it.

<Info>
  All counts in the report use approximate distinct counting (HyperLogLog) with roughly 1% relative error. Use these numbers for decision support and supplier evaluation, not for billing or contractual purposes.
</Info>

## Report info

The card at the top of the report displays metadata about the underlying dataset:

| Field            | Description                                                            |
| ---------------- | ---------------------------------------------------------------------- |
| **Dataset Name** | The generated dataset that contains the report data                    |
| **Created**      | When the report was generated                                          |
| **Description**  | The supplier dataset and enrichment attributes included in this report |

You can share a report by copying its link. Recipients see a **Loaded from shared link** badge when they open a shared report.

## Identity Enrichment

This section visualizes the end-to-end flow from your starting identifiers through matching to the new identifiers gained. It reads left to right across three cards.

### Input (Customer Identifiers)

The pink card on the left shows your starting inventory:

| Metric                        | What it means                                                                                       |
| ----------------------------- | --------------------------------------------------------------------------------------------------- |
| **Identifier counts by type** | The number of unique identifiers you have for each type (e.g., E164 Phone Number, Normalized Email) |
| **Total IDs**                 | The sum of unique identifiers across all types                                                      |
| **Unique People**             | The number of distinct person profiles in your data, after deduplication across identifier types    |
| **IDs per person**            | Total IDs divided by Unique People — how many identifiers you have per person on average            |

### Resolution (Identity Matching)

The purple card in the center shows which of your identifiers matched the supplier:

| Metric                        | What it means                                                                                                                                                                    |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Identifier counts by type** | The number of unique identifiers that matched, broken out by type. Each type also shows its **match rate** — the percentage of your identifiers of that type that found a match. |
| **Total Matched**             | The sum of matched identifiers across all types                                                                                                                                  |
| **Unique People**             | The number of your distinct person profiles that had at least one identifier match                                                                                               |

### Enrichment (Additional Data)

The green card on the right shows what the supplier adds for your matched people:

| Metric                            | What it means                                                                                                                                                                        |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **New identifier counts by type** | The number of net-new unique identifiers the supplier contributes, broken out by type. These are identifiers the supplier has for your matched people that you did not already have. |
| **New IDs**                       | The total net-new identifiers gained across all types                                                                                                                                |

## Identifier Type Summary

Below the flow cards, a per-identifier-type breakdown consolidates the numbers into one view:

| Column                | Definition                                                                                |
| --------------------- | ----------------------------------------------------------------------------------------- |
| **Input (unique)**    | Your unique identifiers of this type (from the Input card)                                |
| **Matched (unique)**  | How many of your identifiers of this type matched the supplier (from the Resolution card) |
| **New (unique)**      | Net-new identifiers of this type contributed by the supplier (from the Enrichment card)   |
| **Total Identifiers** | Input + New — the total identifiers of this type you would have after enrichment          |

## Key Performance Metrics

Three headline KPIs summarize the report. Each card shows the metric value and the formula components beneath it.

### Match Rate

The percentage of your people that the supplier can identify.

**Formula:** Matched Unique People ÷ Input Unique People × 100

A higher match rate means the supplier covers more of your customer base. In the example report, 156M matched out of 232M input people yields a 67% match rate.

### ID Enrichment

How many times over the supplier multiplies your total identifier count.

**Formula:** Total IDs after enrichment ÷ Input Total IDs

A value of 2.1x means the supplier roughly doubles the number of identifiers available for activation. This measures the channel-expansion potential of the supplier's data.

### IDs Per Person

The average number of identifiers per person after enrichment.

**Formula:** Total IDs after enrichment ÷ Input Unique People

Higher density means more ways to reach each person across channels. In the example report, 798M total IDs across 232M people yields 3.4 IDs per person.

## Demographic & Behavioral Enrichment

This section shows the distribution of demographic and behavioral attributes that the supplier has for your matched people. Each attribute the report was configured with gets its own panel.

Every panel header displays:

* The **attribute name** (e.g., `hl7 gender.gender`, `Education Level`, `household income range.income_range`)
* **Coverage**: how many of your matched people have a value for this attribute, shown as both a count and a percentage (e.g., "156M of 232M people (67% match rate)")

Within each panel, a horizontal bar chart shows the top values for that attribute ranked by person count. Each bar displays:

* The **value label** (e.g., "female", "Completed College", "$100,000 - $149,999")
* The **person count** — how many matched people have this value
* The **percentage** — that count as a share of all matched people with a value for this attribute

Use these distributions to evaluate whether the supplier's view of your customers is consistent with what you already know. Unexpected skews in gender, age, or income distributions may indicate data-quality issues worth investigating before activating.

## Report Metadata

The collapsible **Report Metadata** section at the bottom of the page contains technical details about the report dataset and its generation.


# Guides
Source: https://docs.narrative.io/guides/index

Step-by-step instructions for common tasks in Narrative I/O

Guides help you accomplish specific tasks. Unlike tutorials, they assume you have basic familiarity with the platform and focus on getting things done efficiently.

## Platform

Configure datasets and manage platform settings.

<CardGroup>
  <Card title="Configuring Dataset Statistics" icon="chart-column" href="/guides/platform/configuring-dataset-statistics">
    Choose which column-level metrics to compute and set refresh schedules
  </Card>

  <Card title="Configuring Retention Policies" icon="clock-rotate-left" href="/guides/platform/configuring-retention-policies">
    Add, edit, and manage retention policies from the dataset details page
  </Card>
</CardGroup>

## Data Ingestion

Get your data into Narrative I/O.

<CardGroup>
  <Card title="Hashing PII for Upload" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare email addresses and phone numbers by hashing them
  </Card>
</CardGroup>

## Querying with NQL

Write effective queries to filter, transform, and analyze data.

<CardGroup>
  <Card title="Filtering & Transforming" icon="filter" href="/guides/nql/filtering-transforming">
    Select, filter, and transform data with NQL
  </Card>

  <Card title="Joining Datasets" icon="code-merge" href="/guides/nql/joining-datasets">
    Combine data from multiple datasets
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Cache expensive query results for better performance
  </Card>

  <Card title="Query Optimization" icon="gauge-high" href="/guides/nql/query-optimization">
    Techniques for writing faster, more efficient queries
  </Card>
</CardGroup>

## Rosetta Stone

Map and normalize schemas across data sources.

<CardGroup>
  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Create mappings between your schema and Rosetta Stone
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Test and verify your schema mappings
  </Card>

  <Card title="Edge Cases" icon="triangle-exclamation" href="/guides/rosetta-stone/edge-cases">
    Handle complex mapping scenarios
  </Card>
</CardGroup>

## Audience Studio

Build and deliver targeted audiences.

<CardGroup>
  <Card title="Building an Audience" icon="users" href="/guides/audience-studio/building-an-audience">
    Select data, apply filters, choose destinations, and create an audience
  </Card>
</CardGroup>

## Graph Studio

Build identity graphs from your data.

<CardGroup>
  <Card title="Building an Identity Graph" icon="diagram-project" href="/guides/graph-studio/building-an-identity-graph">
    Create an identity graph from first-party data using the Edge Builder and Graph Builder
  </Card>
</CardGroup>

## Connectors

Troubleshoot and configure data delivery to external platforms.

<CardGroup>
  <Card title="Meta: Ad Account Eligibility" icon="meta" href="/guides/connectors/meta/ad-account-access">
    Resolve eligibility warnings when selecting ad accounts for Custom Audience delivery
  </Card>
</CardGroup>

## Compliance

Manage privacy requests and regulatory obligations.

<CardGroup>
  <Card title="Managing Data Subject Requests" icon="user-shield" href="/guides/compliance/managing-dsrs">
    Submit and process opt-out and deletion requests
  </Card>
</CardGroup>

## Webhooks

Receive push notifications for job events.

<CardGroup>
  <Card title="Subscribing to Notifications" icon="bell" href="/guides/webhooks/subscribing-to-notifications">
    Create and manage webhook subscriptions for job state changes
  </Card>
</CardGroup>

## Account Settings

Manage your account and API access.

<CardGroup>
  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys for programmatic access
  </Card>

  <Card title="SSO Configuration" icon="shield-keyhole" href="/account-settings/sso">
    Configure single sign-on for your organization
  </Card>
</CardGroup>


# Hashing PII for Upload
Source: https://docs.narrative.io/guides/ingestion/hashing-pii

Prepare email addresses and phone numbers for upload by hashing them with MD5, SHA-1, and SHA-256

All personally identifiable information (PII) uploaded to Narrative must be pseudonymized using cryptographic hash functions. This guide walks you through formatting and hashing email addresses and phone numbers before uploading them to the platform.

<Note>
  To understand why Narrative requires hashed identifiers rather than raw PII, see [Data Pseudonymization](/concepts/security/data-pseudonymization).
</Note>

## Prerequisites

Before you begin, ensure you have:

* Your ID list saved as a single-column CSV file
* Access to one of the following: Google Sheets, a command line terminal, or the Touchless PII Hasher app

## Choose your method

| Method                                                                | Best for                        | Requirements     |
| --------------------------------------------------------------------- | ------------------------------- | ---------------- |
| [Touchless PII Hasher](https://www.narrative.io/touchless-pii-hasher) | Quick, one-off hashing          | Web browser only |
| Google Sheets                                                         | Small lists (under 10,000 rows) | Google account   |
| Command line                                                          | Large lists, automation         | Terminal access  |

***

## Hashing email addresses

### Pre-formatting requirements

Hash functions are case-sensitive and whitespace-sensitive. Before hashing, you must normalize your email addresses:

| Rule                                    | Example                     | Result              |
| --------------------------------------- | --------------------------- | ------------------- |
| Lowercase all text                      | `JohnDoe@Gmail.COM`         | `johndoe@gmail.com` |
| Remove leading/trailing whitespace      | `  john@example.com  `      | `john@example.com`  |
| Remove `+` aliases and everything after | `john+newsletter@gmail.com` | `john@gmail.com`    |
| Remove extra periods in username        | `j.o.h.n@gmail.com`         | `john@gmail.com`    |

<Warning>
  Skipping pre-formatting will result in different hash values, preventing matches with other datasets. A single extra space or uppercase letter produces a completely different hash.
</Warning>

### Supported hash algorithms

Narrative supports three hash algorithms. For maximum correspondence with other datasets, we recommend generating all three:

| Algorithm | Output length | Example hash of `johndoe@gmail.com`                                |
| --------- | ------------- | ------------------------------------------------------------------ |
| MD5       | 32 characters | `29a1df4646cb3417c19994a59a3e022a`                                 |
| SHA-1     | 40 characters | `e1e8d3e4a336d4f9dc63b70a534ff10834471556`                         |
| SHA-256   | 64 characters | `06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3` |

### Method 1: Touchless PII Hasher (fastest)

The [Touchless PII Hasher](https://www.narrative.io/touchless-pii-hasher) is a free web app that handles formatting and hashing automatically. Your data never leaves your browser.

<Steps>
  <Step title="Open the app">
    Navigate to [narrative.io/touchless-pii-hasher](https://www.narrative.io/touchless-pii-hasher).
  </Step>

  <Step title="Upload your file">
    Drag and drop your CSV file or click to browse.
  </Step>

  <Step title="Download the results">
    The app automatically formats and hashes your data. Download the resulting file.
  </Step>
</Steps>

### Method 2: Google Sheets (small lists)

For lists under 10,000 rows, use Narrative's [Hashing Functions spreadsheet](https://docs.google.com/spreadsheets/d/1JZrNMcLC4PXrBPWfpL4JHdBPKUngZmW33hC7U1NrLBo/copy) which automatically handles pre-formatting and all three hash algorithms.

<Steps>
  <Step title="Make a copy of the template">
    Open the [Hashing Functions spreadsheet](https://docs.google.com/spreadsheets/d/1JZrNMcLC4PXrBPWfpL4JHdBPKUngZmW33hC7U1NrLBo/copy) and click **Make a copy** when prompted.
  </Step>

  <Step title="Paste your email list">
    Copy your list of emails and paste them starting in cell **A4**.
  </Step>

  <Step title="Wait for processing">
    The spreadsheet may take a few minutes to compute, depending on list size.
  </Step>

  <Step title="Download the results">
    Navigate to the tab titled **DOWNLOAD\_THIS\_SHEET\_AS\_.CSV\_FILE**. Go to **File > Download > Comma-separated values (.csv, current sheet)**.
  </Step>
</Steps>

### Method 3: Command line (large lists)

For large lists or automation, use the command line. Narrative provides a hashing script, or you can use standard Unix tools.

<Tabs>
  <Tab title="Mac/Linux">
    **Option A: Use Narrative's hashing script**

    <Steps>
      <Step title="Download the script">
        Download [hashingFunction.py](https://gist.github.com/narrative-io/example-hashing-script) and save it to your `~/bin` directory.
      </Step>

      <Step title="Make it executable">
        ```bash theme={null}
        chmod +x ~/bin/hashingFunction.py
        ```
      </Step>

      <Step title="Run the script">
        ```bash theme={null}
        ~/bin/hashingFunction.py < input.csv > output.csv
        ```
      </Step>
    </Steps>

    **Option B: Use standard Unix tools**

    For a quick one-liner that handles pre-formatting and generates SHA-256 hashes:

    ```bash theme={null}
    cat emails.csv | tr '[:upper:]' '[:lower:]' | sed 's/+[^@]*//g' | \
      while read email; do echo -n "$email" | shasum -a 256 | cut -d' ' -f1; done > hashed.csv
    ```

    To generate all three hash formats:

    ```bash theme={null}
    cat emails.csv | tr '[:upper:]' '[:lower:]' | sed 's/+[^@]*//g' | \
      while read email; do
        echo "$(echo -n "$email" | md5),$(echo -n "$email" | shasum -a 1 | cut -d' ' -f1),$(echo -n "$email" | shasum -a 256 | cut -d' ' -f1)"
      done > hashed.csv
    ```
  </Tab>

  <Tab title="Windows (PowerShell)">
    Use PowerShell to hash your email list:

    ```powershell theme={null}
    Get-Content emails.csv | ForEach-Object {
        $email = $_.Trim().ToLower() -replace '\+[^@]*', ''
        $bytes = [System.Text.Encoding]::UTF8.GetBytes($email)

        $md5 = [System.Security.Cryptography.MD5]::Create()
        $sha1 = [System.Security.Cryptography.SHA1]::Create()
        $sha256 = [System.Security.Cryptography.SHA256]::Create()

        $md5Hash = [BitConverter]::ToString($md5.ComputeHash($bytes)) -replace '-', ''
        $sha1Hash = [BitConverter]::ToString($sha1.ComputeHash($bytes)) -replace '-', ''
        $sha256Hash = [BitConverter]::ToString($sha256.ComputeHash($bytes)) -replace '-', ''

        "$sha256Hash,$md5Hash,$sha1Hash".ToLower()
    } | Out-File -Encoding UTF8 hashed.csv
    ```
  </Tab>
</Tabs>

<Warning>
  Never direct output to the same file as input. This will delete your data:

  ```bash theme={null}
  # DO NOT DO THIS
  hashingFunction.py < emails.csv > emails.csv  # Data will be lost!
  ```
</Warning>

***

## Hashing phone numbers

Phone numbers follow a similar process, but with different pre-formatting rules.

### Pre-formatting requirements

| Rule                                       | Example             | Result        |
| ------------------------------------------ | ------------------- | ------------- |
| Remove all non-numeric characters          | `+1 (555) 123-4567` | `15551234567` |
| Include country code                       | `555-123-4567` (US) | `15551234567` |
| Remove leading zeros (except country code) | `0555-123-4567`     | `15551234567` |

### Using Google Sheets for phone numbers

Use Narrative's [Phone Hashing Functions spreadsheet](https://docs.google.com/spreadsheets/d/1example-phone-sheet/copy) which automatically strips non-numeric characters before hashing.

Follow the same steps as email hashing above.

### Using command line for phone numbers

<Tabs>
  <Tab title="Mac/Linux">
    ```bash theme={null}
    cat phones.csv | tr -cd '0-9\n' | \
      while read phone; do
        echo "$(echo -n "$phone" | md5),$(echo -n "$phone" | shasum -a 1 | cut -d' ' -f1),$(echo -n "$phone" | shasum -a 256 | cut -d' ' -f1)"
      done > hashed_phones.csv
    ```
  </Tab>

  <Tab title="Windows (PowerShell)">
    ```powershell theme={null}
    Get-Content phones.csv | ForEach-Object {
        $phone = $_ -replace '[^0-9]', ''
        $bytes = [System.Text.Encoding]::UTF8.GetBytes($phone)

        $md5 = [System.Security.Cryptography.MD5]::Create()
        $sha1 = [System.Security.Cryptography.SHA1]::Create()
        $sha256 = [System.Security.Cryptography.SHA256]::Create()

        $md5Hash = [BitConverter]::ToString($md5.ComputeHash($bytes)) -replace '-', ''
        $sha1Hash = [BitConverter]::ToString($sha1.ComputeHash($bytes)) -replace '-', ''
        $sha256Hash = [BitConverter]::ToString($sha256.ComputeHash($bytes)) -replace '-', ''

        "$sha256Hash,$md5Hash,$sha1Hash".ToLower()
    } | Out-File -Encoding UTF8 hashed_phones.csv
    ```
  </Tab>
</Tabs>

***

## Verify your output

Before uploading, verify your hashing is correct using this test data:

**Test email:** `johndoe@gmail.com`

| Algorithm | Expected hash                                                      |
| --------- | ------------------------------------------------------------------ |
| SHA-256   | `06a240d11cc201676da976f7b49341181fd180da37cbe40a77432c0a366c80c3` |
| MD5       | `29a1df4646cb3417c19994a59a3e022a`                                 |
| SHA-1     | `e1e8d3e4a336d4f9dc63b70a534ff10834471556`                         |

If your hashes match, your implementation is correct.

***

## Uploading your hashed list

Once your list is hashed:

1. Ensure your file is in CSV format
2. Navigate to your dataset in Narrative
3. Upload the hashed file

***

## Troubleshooting

| Issue                                 | Cause                      | Solution                                                      |
| ------------------------------------- | -------------------------- | ------------------------------------------------------------- |
| Hashes don't match expected values    | Pre-formatting not applied | Verify lowercase, whitespace removal, and alias stripping     |
| Google Sheets is slow or unresponsive | List too large             | Use command line method for lists over 10,000 rows            |
| Different hashes for same email       | Inconsistent formatting    | Ensure all emails go through the same pre-formatting pipeline |
| Command not found (Mac/Linux)         | Script not in PATH         | Use full path to script or add `~/bin` to PATH                |
| PowerShell security error             | Execution policy           | Run `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`     |

***

## Related content

<CardGroup>
  <Card title="Data Pseudonymization" icon="user-secret" href="/concepts/security/data-pseudonymization">
    Understand why Narrative requires hashed identifiers
  </Card>

  <Card title="Touchless PII Hasher" icon="wand-magic-sparkles" href="https://www.narrative.io/touchless-pii-hasher">
    Free web app for quick, secure hashing
  </Card>

  <Card title="Security Model" icon="shield-halved" href="/concepts/architecture/security-model">
    Learn how Narrative protects your data
  </Card>
</CardGroup>


# Connecting to the Narrative MCP Server
Source: https://docs.narrative.io/guides/mcp/connecting-an-ai-assistant

Set up your MCP client to interact with Narrative through an AI agent

This guide walks you through connecting an MCP-compatible AI assistant to the Narrative platform. Once connected, your assistant can search datasets, run NQL queries, manage materialized views, and more — all from within your conversation.

## Prerequisites

* A Narrative account with access to at least one company
* An MCP-compatible AI client installed (see supported clients below)

## Server URL

All clients connect to the same endpoint:

```
https://mcp.narrative.io/mcp
```

## Setup by client

<Tabs>
  <Tab title="Claude Code">
    Run the following command in your terminal:

    ```bash theme={null}
    claude mcp add --transport http narrative https://mcp.narrative.io/mcp
    ```

    This registers the Narrative MCP server in your Claude Code configuration. The next time Claude calls a Narrative tool, a browser window opens for you to authenticate.

    To verify the server is registered:

    ```bash theme={null}
    claude mcp list
    ```

    You should see `narrative` in the output with the server URL.
  </Tab>

  <Tab title="Claude Desktop">
    Claude Desktop connects to remote MCP servers through its built-in **Connectors** UI. The `claude_desktop_config.json` file only supports stdio-based servers, so don't add Narrative there.

    1. Open **Settings** (gear icon, or `⌘,`).
    2. In the left nav, select **Connectors**.
    3. Scroll to the bottom and click **Add custom connector**.
    4. Enter the following:
       * **Name:** `Narrative`
       * **Remote MCP server URL:** `https://mcp.narrative.io/mcp`
    5. Click **Add**. A browser window opens for OAuth — sign in with your Narrative credentials.
    6. Return to Claude Desktop. The connector should show **Connected**.

    The Narrative tools appear in the tools menu (hammer icon) at the bottom of the chat input.

    <Note>
      If you don't see **Add custom connector**, update Claude Desktop — native remote MCP support requires a recent version.
    </Note>
  </Tab>

  <Tab title="Cursor">
    1. Open **Settings** > **MCP**.
    2. Click **+ Add new MCP server**.
    3. Enter the following:
       * **Name:** `narrative`
       * **Type:** `http`
       * **URL:** `https://mcp.narrative.io/mcp`
    4. Click **Add** and confirm the server appears in the list.

    Narrative tools are available in Cursor's Agent mode.
  </Tab>

  <Tab title="Windsurf">
    1. Open the **Command Palette** and search for **MCP Configuration**.
    2. Add the Narrative server to your configuration:

    ```json theme={null}
    {
      "mcpServers": {
        "narrative": {
          "type": "http",
          "url": "https://mcp.narrative.io/mcp"
        }
      }
    }
    ```

    3. Save and restart Windsurf.
  </Tab>

  <Tab title="Other clients">
    Any MCP-compatible client that supports HTTP transport can connect using:

    * **Transport:** HTTP (Streamable HTTP)
    * **URL:** `https://mcp.narrative.io/mcp`

    Refer to your client's documentation for how to add a remote MCP server.
  </Tab>
</Tabs>

## Authenticate

The MCP server accepts two kinds of bearer credential in the `Authorization` header:

* **OAuth login (interactive clients).** A standard OAuth flow exchanges your Narrative login for a scoped session token. This is what Claude Code, Claude Desktop, Cursor, and Windsurf use by default.
* **Narrative API token (programmatic clients).** Long-lived API tokens are accepted directly, which lets server-to-server callers and scripted MCP clients connect without a browser-based login.

The server distinguishes the two by token shape — JWTs (three dot-separated segments) follow the OAuth path, and opaque tokens are treated as Narrative API tokens.

### OAuth (default for interactive clients)

The first time your assistant calls a Narrative tool, the MCP server initiates an OAuth login:

1. A browser window opens with the Narrative login page.
2. Sign in with your Narrative credentials.
3. Return to your AI client — the session is now authenticated.

Your session persists across tool calls. You only need to log in once per session.

### Narrative API token (for programmatic clients)

If your MCP client supports a static bearer token, you can authenticate with a [Narrative API key](/account-settings/api-keys) instead of OAuth. Set the `Authorization` header on every MCP request:

```http theme={null}
Authorization: Bearer <your-narrative-api-token>
```

The server validates the token, resolves the associated user and default company, and reuses the same token for downstream Narrative API calls. The token's existing permissions and company access apply.

<Tip>
  Use API token auth for headless workloads — CI jobs, agent orchestrators, and other server-to-server callers that can't complete an interactive OAuth flow.
</Tip>

<Note>
  If you belong to multiple companies, the server defaults to your primary company. Ask the assistant to switch companies during the conversation — for example, "switch to Acme Corp" — and it will use the `narrative_context_set_company` tool.
</Note>

## Verify the connection

After setup, test the connection by asking your assistant something like:

> "What companies do I have access to on Narrative?"

The assistant should call `narrative_context_get_companies` and return a list of your companies. If you see results, the connection is working.

## Troubleshooting

### Browser window doesn't open

Some terminal environments block browser launches. If the OAuth login window doesn't appear:

* Check that your default browser is configured correctly.
* Try opening the server URL directly in a browser to confirm network access: `https://mcp.narrative.io/mcp`

### Tools not appearing

If your AI client doesn't show Narrative tools:

* Verify the server URL is exactly `https://mcp.narrative.io/mcp` (no trailing slash).
* Restart your AI client after adding the server configuration.
* Check your client's MCP logs for connection errors.

### Session expired

If tool calls start failing with authentication errors, your session may have expired. Ask the assistant to retry — the server will prompt you to log in again.

## What's next

<CardGroup>
  <Card title="Data Collaboration MCP Server" icon="lightbulb" href="/concepts/mcp/overview">
    Understand how the MCP server works and what it enables
  </Card>

  <Card title="Tool Reference" icon="book" href="/reference/integrations/mcp-server">
    Browse the complete catalog of available tools
  </Card>

  <Card title="NQL Guide" icon="terminal" href="/guides/nql/filtering-transforming">
    Learn NQL basics to get more from query tools
  </Card>

  <Card title="Dataset Statistics" icon="chart-column" href="/guides/platform/configuring-dataset-statistics">
    Configure column statistics that the MCP server can read and trigger
  </Card>
</CardGroup>


# Installing skills
Source: https://docs.narrative.io/guides/mcp/installing-skills

Install Narrative skills into Claude Code or any agentskills.io-compliant agent

[Skills](/concepts/mcp/skills) are pre-built workflows that orchestrate Narrative's MCP server to complete specific tasks — drafting NQL queries, generating Rosetta Stone mappings, building identity graphs, and more. Skills are harness-agnostic: you can install them with the standalone `skills` CLI into any spec-compliant agent, or as Claude Code plugins.

## Prerequisites

* An agent runtime that loads skills. Either:
  * Any [agentskills.io](https://agentskills.io)-compliant agent (used with the `skills` CLI), or
  * Claude Code, updated to a version that supports the `/plugin` command.
* The Narrative MCP server connected to that agent. See [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant) if you haven't set this up yet.
* A Narrative account with access to the active company you want the skills to operate against.

<Info>
  Skills run inside your agent and call the MCP server on your behalf. Without an active MCP connection, skills can't reach your Narrative data.
</Info>

## Install skills

Each skill's detail page in the [Skills catalog](https://app.narrative.io/marketplace/skills) shows an **Installation** picker with the two options below. Pick whichever matches your agent.

<Tabs>
  <Tab title="skills CLI">
    The standalone [`skills` CLI](https://agentskills.io) installs a skill's rendered `SKILL.md` into any spec-compliant agent. No repo clone, no Bun runtime, no Claude Code required.

    <Steps>
      <Step title="Add the Narrative skills marketplace">
        Run the install command from any skill's detail page, or globally for the whole marketplace:

        ```text theme={null}
        npx skills add narrative-io/narrative-skills-marketplace
        ```

        The CLI fetches the marketplace and installs every available skill into your active agent's skills directory.
      </Step>

      <Step title="Verify the skill is loaded">
        Open your agent and ask it to describe the skills it has available. You should see entries such as **Find Attribute**, **Write NQL**, and **Design Analysis**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Claude Code">
    Narrative also publishes its skills as Claude Code plugins from the public [`narrative-io/narrative-skills-marketplace`](https://github.com/narrative-io/narrative-skills-marketplace) repository. You add the marketplace once, then install whichever plugins you want.

    <Steps>
      <Step title="Add the Narrative skills marketplace">
        In a Claude Code session, run:

        ```text theme={null}
        /plugin marketplace add narrative-io/narrative-skills-marketplace
        ```

        Claude Code fetches the marketplace manifest and lists the available plugins. You only need to do this once.
      </Step>

      <Step title="Install the plugin">
        Install the plugin that contains the skill you want. For general-purpose authoring, querying, and Rosetta Stone work:

        ```text theme={null}
        /plugin install narrative-common@narrative-skills-marketplace
        ```

        For identity resolution and partner overlap work:

        ```text theme={null}
        /plugin install narrative-identity@narrative-skills-marketplace
        ```

        For audience expansion and look-alike modeling:

        ```text theme={null}
        /plugin install narrative-audience@narrative-skills-marketplace
        ```

        Installing a plugin makes every skill it contains available in the current Claude Code session.
      </Step>

      <Step title="Verify the skill is loaded">
        In a Claude Code chat, ask Claude to describe its available skills. You should see entries for each skill in the plugin you installed (for example, **Find Attribute**, **Write NQL**, **Design Analysis**).
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Use a skill

Once a plugin is installed, you invoke its skills with natural-language prompts. Claude routes the request to the matching skill and follows its instructions.

For example, with the `narrative-common` plugin installed:

```text theme={null}
Find the Rosetta Stone attribute for a SHA-256 hashed email.
```

Claude calls the **Find Attribute** skill, which uses the `narrative_attributes_search` and `narrative_attributes_describe` MCP tools to resolve the description to a canonical attribute ID (`sha256_hashed_email`).

Each skill's detail page in the [Skills catalog](https://app.narrative.io/marketplace/skills) lists example prompts, accepted arguments, and the use cases each one addresses.

### Slash commands and arguments

Many skills also expose a slash-command form with explicit arguments. For example, **Write NQL** accepts `--dataset`, `--run`, and `--limit`:

```text theme={null}
/write-nql --dataset 123 --run --limit 50 distinct users in the last 30 days
```

The slash-command form is useful for scripting, scoping a skill to a specific dataset, or chaining skills together. See each skill's **Arguments** table on its detail page for the full list of flags and defaults.

<Tip>
  Skills that produce artifacts — NQL queries, workflow specifications, mapping definitions — always show you the draft and wait for explicit approval before submitting it. You can edit the draft or ask Claude to revise it before saying "go."
</Tip>

## Where skills run

Skills run on your machine inside the agent you installed them into. They reach Narrative through the MCP server using the OAuth credential you established when you connected the agent to the server. Skills never bypass MCP — every action they take is a tool call you can inspect in the agent's transcript.

This also means skills inherit MCP's company scoping: they operate against whichever company is active in your MCP session. Use the `narrative_context_set_company` tool (or ask the agent to switch companies) before invoking a skill if you need to target a different account.

## Update or remove a skill

If you used the **skills CLI**, rerun the add command to pick up newer versions, or use the CLI's remove command to uninstall:

```text theme={null}
npx skills add narrative-io/narrative-skills-marketplace
```

If you used **Claude Code**, manage plugins with `/plugin`:

```text theme={null}
/plugin update narrative-common@narrative-skills-marketplace
/plugin uninstall narrative-common@narrative-skills-marketplace
```

The skill version shown on each catalog detail page matches the semver in the skill's `SKILL.md` frontmatter. Reinstall to pick up newer versions.

## Next steps

<CardGroup>
  <Card title="Skills concepts" icon="sparkles" href="/concepts/mcp/skills">
    How skills relate to the MCP server, plugins, and runtimes.
  </Card>

  <Card title="MCP tool reference" icon="book" href="/reference/integrations/mcp-server">
    The underlying tools that skills call.
  </Card>

  <Card title="Connect to the MCP server" icon="plug" href="/guides/mcp/connecting-an-ai-assistant">
    Set up the OAuth connection skills depend on.
  </Card>

  <Card title="Browse the Skills catalog" icon="grid" href="https://app.narrative.io/marketplace/skills">
    Every published skill with example prompts and install commands.
  </Card>
</CardGroup>


# Creating Materialized Views
Source: https://docs.narrative.io/guides/nql/creating-materialized-views

Pre-compute and cache query results for improved performance

If you're running the same expensive query repeatedly—for dashboards, reports, or data sharing—a materialized view lets you compute once and read many times. This guide walks through creating and managing materialized views in NQL.

## Prerequisites

Before creating a materialized view, you should have:

* Basic familiarity with [NQL syntax](/nql/general/syntax)
* A dataset to query (your own or one you have access to)
* Understanding of what [materialized views](/concepts/nql/materialized-views) are and when to use them

## Basic syntax

A materialized view is created with the `CREATE MATERIALIZED VIEW` statement:

```sql theme={null}
CREATE MATERIALIZED VIEW "view_name"
AS
<your_query>
```

The query can be any valid NQL SELECT statement. The result is stored as a dataset that you can query like any other table.

## Creating your first materialized view

Let's create a simple materialized view that aggregates user counts by region.

### Step 1: Write and test your query

Start by writing the query you want to materialize. Run it to verify it returns the expected results:

```sql theme={null}
SELECT
    region,
    COUNT(1) as user_count,
    APPROX_COUNT_DISTINCT(company_id) as company_count
FROM user_data
WHERE status = 'active'
GROUP BY region
```

### Step 2: Wrap it in CREATE MATERIALIZED VIEW

Once your query works, add the `CREATE MATERIALIZED VIEW` wrapper:

```sql theme={null}
CREATE MATERIALIZED VIEW "active_users_by_region"
AS
SELECT
    region,
    COUNT(1) as user_count,
    APPROX_COUNT_DISTINCT(company_id) as company_count
FROM user_data
WHERE status = 'active'
GROUP BY region
```

### Step 3: Execute the statement

Run the statement to create the view. The query executes immediately and stores the results.

### Step 4: Query the materialized view

Once created, query it like any other table:

```sql theme={null}
SELECT region, user_count, company_count
FROM active_users_by_region
WHERE user_count > 1000
```

## Adding metadata options

You can configure how the view behaves using metadata options. The most common are:

### Setting a refresh schedule

To keep your view updated automatically:

```sql theme={null}
CREATE MATERIALIZED VIEW "daily_sales_summary"
REFRESH_SCHEDULE = '@daily'
AS
SELECT
    DATE(order_timestamp) as order_date,
    COUNT(1) as order_count,
    SUM(total_amount) as total_revenue
FROM orders
GROUP BY DATE(order_timestamp)
```

Available schedules include `@hourly`, `@daily`, `@weekly`, `@monthly`, and custom CRON expressions.

### Adding a display name and description

Make your view easier to find in the UI:

```sql theme={null}
CREATE MATERIALIZED VIEW "monthly_revenue"
DISPLAY_NAME = 'Monthly Revenue Report'
DESCRIPTION = 'Total revenue aggregated by month, refreshed daily'
REFRESH_SCHEDULE = '@daily'
AS
SELECT
    DATE_TRUNC('month', order_timestamp) as month,
    SUM(total_amount) as revenue
FROM orders
GROUP BY DATE_TRUNC('month', order_timestamp)
```

### Partitioning for performance

For time-series data, partition by date to improve query performance:

```sql theme={null}
CREATE MATERIALIZED VIEW "event_metrics"
REFRESH_SCHEDULE = '@hourly'
PARTITIONED_BY event_date DAY
AS
SELECT
    DATE(event_timestamp) as event_date,
    event_type,
    COUNT(1) as event_count
FROM events
GROUP BY DATE(event_timestamp), event_type
```

<Tip>
  For complete details on all available options, see the [Materialized View Syntax Reference](/nql/commands/create-materialized-view).
</Tip>

## Refreshing materialized views

Views refresh according to their schedule. You can also trigger manual refreshes via the API when you need immediate updates.

### Scheduled refresh

Use the `REFRESH_SCHEDULE` option to set automatic refresh intervals:

| Schedule        | When it runs    |
| --------------- | --------------- |
| `@hourly`       | Every hour      |
| `@daily`        | Once per day    |
| `@weekly`       | Once per week   |
| `@monthly`      | Once per month  |
| CRON expression | Custom schedule |

### Custom CRON schedules

For more control, use a CRON expression:

```sql theme={null}
-- Refresh every Monday at 3:15 PM UTC
REFRESH_SCHEDULE = '15 15 * * 1'

-- Refresh at 2 AM on the first day of each month
REFRESH_SCHEDULE = '0 2 1 * *'
```

### Incremental refresh

For large datasets, NQL can refresh incrementally by processing only changed data. This happens automatically when the query pattern supports it. To learn more, see [Incremental View Maintenance](/concepts/nql/incremental-view-maintenance).

### Editing the refresh schedule and compute pool from the UI

You can change a materialized view's refresh cadence and the [compute pool](/concepts/primitives/compute-pools) that runs its scheduled refreshes without rewriting the view. From the dataset's **Overview** screen, click the pencil icon next to the **Refresh Schedule** row:

* **Refresh schedule** — pick a preset (`@hourly`, `@daily`, `@weekly`, `@monthly`) or enter a custom CRON expression.
* **Compute pool** — select which compute pool runs the scheduled refresh. The dropdown lists only **active, company-owned** pools on the dataset's [data plane](/concepts/primitives/data-planes), matching the pools you can pick in the [Context Selector](/guides/platform/using-the-context-selector).

The selected pool persists with the dataset and is reused for every subsequent scheduled refresh until you change it.

## Write modes

Control how refreshes update the view data:

### Overwrite mode (default)

Each refresh replaces the entire view with fresh results:

```sql theme={null}
CREATE MATERIALIZED VIEW "current_inventory"
WRITE_MODE = 'overwrite'
AS
SELECT product_id, SUM(quantity) as stock_level
FROM inventory
GROUP BY product_id
```

### Append mode

Each refresh adds new data to the existing view:

```sql theme={null}
CREATE MATERIALIZED VIEW "event_log"
WRITE_MODE = 'append'
AS
SELECT event_id, user_id, event_type, event_timestamp
FROM events
WHERE event_timestamp > CURRENT_TIMESTAMP - INTERVAL 1 HOUR
```

<Warning>
  When using append mode, ensure your query includes filters to avoid inserting duplicate records on each refresh.
</Warning>

<Tip>
  Need to update existing records instead of just appending? Use [MERGE ON](/guides/nql/incremental-upserts) to match incoming rows against existing data and decide whether to update or insert.
</Tip>

## Example: Complete materialized view

Here's a complete example combining multiple options:

```sql theme={null}
CREATE MATERIALIZED VIEW "female_audience"
REFRESH_SCHEDULE = '@daily'
DISPLAY_NAME = 'Female Audience'
DESCRIPTION = 'Audience containing demographic of female users.'
EXPIRE = 'retain_everything'
TAGS = ( '_nio_materialized_view', 'user_engagement', 'metrics' )
WRITE_MODE = 'overwrite'
EXTENDED_STATS = 'all'
PARTITIONED_BY event_date DAY
AS
SELECT
    "unique_identifier",
    "hl7_gender",
    "age"
FROM narrative.rosetta_stone
WHERE
    "hl7_gender"."gender" = 'female'
    AND "event_timestamp" > CURRENT_TIMESTAMP - INTERVAL 160 DAYS
    AND rosetta_stone._price_cpm_usd <= 2.00
```

This creates a view that:

* Refreshes daily with fresh data
* Stores 160 days of female user demographics
* Partitions by day for efficient date-range queries
* Computes extended statistics for query optimization
* Retains all data indefinitely

***

## Related content

<CardGroup>
  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Understand when and why to use materialized views
  </Card>

  <Card title="Materialized View Syntax" icon="list-check" href="/nql/commands/create-materialized-view">
    Complete reference for all available options
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    How NQL optimizes refreshes for large datasets
  </Card>

  <Card title="Query Optimization" icon="gauge-high" href="/guides/nql/query-optimization">
    Techniques for writing faster, more efficient queries
  </Card>
</CardGroup>


# Filtering and Transforming Data
Source: https://docs.narrative.io/guides/nql/filtering-transforming

Select columns, filter rows, and transform data with NQL

This guide covers the fundamentals of querying data with NQL: selecting columns, filtering rows with WHERE clauses, and transforming data using expressions and functions.

## Prerequisites

* A Narrative I/O account with at least one dataset
* Basic familiarity with SQL syntax

## What you'll learn

* How to select specific columns and use aliases
* How to filter data with comparison and logical operators
* How to transform data using expressions and functions
* How to work with complex types (arrays, structs, maps)

***

## Selecting columns

### Basic column selection

Select specific columns from a dataset:

```sql theme={null}
SELECT
  user_id,
  email,
  created_at
FROM company_data."123"
```

### Using column aliases

Rename columns in your output with `AS`:

```sql theme={null}
SELECT
  user_id AS id,
  email AS contact_email,
  created_at AS signup_date
FROM company_data."123"
```

### Fully qualified column names

When joining datasets or for clarity, use fully qualified names:

```sql theme={null}
SELECT
  company_data."123".user_id,
  company_data."123".email
FROM company_data."123"
```

<Tip>
  Always use fully qualified column names in joins to avoid ambiguity.
</Tip>

<Note>
  NQL requires explicit column lists. Wildcards (`SELECT *`) are not supported. See [Explicit Column Selection](/nql/general/explicit-columns) for why and how to list columns explicitly.
</Note>

### Select distinct values

Remove duplicates with `DISTINCT`:

```sql theme={null}
SELECT DISTINCT category
FROM company_data."123"
```

***

## Filtering with WHERE

### Comparison filters

Filter rows using comparison operators:

```sql theme={null}
SELECT user_id, email, age, status
FROM company_data."123"
WHERE
  age >= 18
  AND age < 65
  AND status = 'active'
```

| Operator | Meaning               |
| -------- | --------------------- |
| `=`      | Equal to              |
| `<>`     | Not equal to          |
| `<`      | Less than             |
| `>`      | Greater than          |
| `<=`     | Less than or equal    |
| `>=`     | Greater than or equal |

### Combining conditions

Use `AND`, `OR`, and `NOT` to combine conditions:

```sql theme={null}
SELECT user_id, email, status, region
FROM company_data."123"
WHERE
  (status = 'active' OR status = 'pending')
  AND NOT is_test_account
  AND region = 'US'
```

Use parentheses to control evaluation order. `AND` is evaluated before `OR`.

### IN and NOT IN

Check membership in a list:

```sql theme={null}
SELECT user_id, category, source, created_at
FROM company_data."123"
WHERE
  category IN ('electronics', 'clothing', 'home')
  AND source NOT IN ('test', 'internal')
```

### BETWEEN

Filter within a range (inclusive):

```sql theme={null}
SELECT user_id, email, age, score
FROM company_data."123"
WHERE
  age BETWEEN 25 AND 45
  AND score BETWEEN 80 AND 100
```

### NULL checks

Check for missing values:

```sql theme={null}
SELECT user_id, email, phone
FROM company_data."123"
WHERE
  email IS NOT NULL
  AND phone IS NOT NULL
  AND secondary_email IS NULL
```

<Warning>
  Don't use `= NULL` or `<> NULL`. These always return NULL, not true/false. Use `IS NULL` and `IS NOT NULL`.
</Warning>

### Pattern matching with LIKE

Match string patterns using LIKE wildcards:

```sql theme={null}
SELECT user_id, email, name, code
FROM company_data."123"
WHERE
  email LIKE '%@gmail.com'      -- Ends with @gmail.com
  AND name LIKE 'John%'          -- Starts with John
  AND code LIKE 'A__'            -- A followed by exactly 2 characters
```

| Wildcard | Matches                 |
| -------- | ----------------------- |
| `%`      | Zero or more characters |
| `_`      | Exactly one character   |

For case-insensitive matching, normalize the case first:

```sql theme={null}
WHERE LOWER(email) LIKE '%@gmail.com'
```

***

## Date and time filtering

### Current date comparisons

Filter relative to the current date:

```sql theme={null}
SELECT user_id, event_date, created_at
FROM company_data."123"
WHERE
  -- Last 30 days
  event_date >= CURRENT_DATE - INTERVAL '30' DAY

  -- Last 3 months
  AND created_at >= CURRENT_DATE - INTERVAL '3' MONTH
```

### Date ranges

Filter within specific date ranges:

```sql theme={null}
SELECT user_id, email, event_date
FROM company_data."123"
WHERE
  event_date BETWEEN '2024-01-01' AND '2024-12-31'
```

### Extracting date parts

Filter by specific date components:

```sql theme={null}
SELECT user_id, event_date, event_type
FROM company_data."123"
WHERE
  EXTRACT(YEAR FROM event_date) = 2024
  AND EXTRACT(MONTH FROM event_date) IN (1, 2, 3)
  AND EXTRACT(DAYOFWEEK FROM event_date) = 1  -- Sunday
```

***

## Price filtering

Control costs by filtering on the special `_price_cpm_usd` column:

```sql theme={null}
SELECT user_id, email, category
FROM company_data."123"
WHERE
  category = 'premium'
  AND _price_cpm_usd <= 1.00  -- Max $1 per 1000 rows
```

<Tip>
  Always include price filtering to control data acquisition costs, especially when creating materialized views.
</Tip>

***

## Transforming data

### Expressions in SELECT

Compute values using expressions:

```sql theme={null}
SELECT
  quantity * unit_price AS line_total,
  (quantity * unit_price) * 1.0825 AS total_with_tax,
  first_name || ' ' || last_name AS full_name
FROM company_data."123"
```

### CASE expressions

Apply conditional logic:

```sql theme={null}
SELECT
  user_id,
  CASE
    WHEN lifetime_value > 1000 THEN 'platinum'
    WHEN lifetime_value > 500 THEN 'gold'
    WHEN lifetime_value > 100 THEN 'silver'
    ELSE 'bronze'
  END AS tier
FROM company_data."123"
```

### String transformations

Clean and format text:

```sql theme={null}
SELECT
  UPPER(name) AS uppercase_name,
  LOWER(email) AS lowercase_email,
  TRIM(user_input) AS cleaned_input,
  CONCAT(city, ', ', state) AS location,
  SUBSTRING(phone, 1, 3) AS area_code
FROM company_data."123"
```

### Type casting

Convert between types:

```sql theme={null}
SELECT
  CAST(string_number AS LONG) AS numeric_value,
  CAST(timestamp_value AS DATE) AS date_only,
  CAST(amount AS DECIMAL) AS precise_amount
FROM company_data."123"
```

***

## Working with complex types

### Accessing array elements

Use bracket notation with zero-based indexing:

```sql theme={null}
SELECT
  identifiers[0] AS first_id,
  identifiers[1] AS second_id,
  tags[0] AS primary_tag
FROM company_data."123"
```

### Accessing struct fields

Use dot notation:

```sql theme={null}
SELECT
  person.name,
  person.age,
  address.city,
  address.postal_code
FROM company_data."123"
```

### Nested access

Combine notation for deeply nested data:

```sql theme={null}
SELECT
  -- Array of structs
  identifiers[0].type AS first_id_type,
  identifiers[0].value AS first_id_value,

  -- Struct with nested struct
  user.preferences.language,
  user.preferences.timezone
FROM company_data."123"
```

### Accessing map values

Use `ELEMENT_AT` for maps:

```sql theme={null}
SELECT
  ELEMENT_AT(properties, 'color') AS color,
  ELEMENT_AT(metadata, 'source') AS source
FROM company_data."123"
```

### Expanding arrays with UNNEST

Convert array elements to rows:

```sql theme={null}
SELECT
  t.user_id,
  tag
FROM company_data."123" t, UNNEST(t.tags) AS tag
```

This creates one row per array element. If a user has tags `['a', 'b', 'c']`, the result includes three rows for that user.

***

## Aggregating data

### Basic aggregations

Compute summary statistics:

```sql theme={null}
SELECT
  COUNT(1) AS total_rows,
  APPROX_COUNT_DISTINCT(user_id) AS unique_users,
  SUM(amount) AS total_amount,
  AVG(score) AS average_score,
  MIN(created_at) AS earliest,
  MAX(created_at) AS latest
FROM company_data."123"
```

### Grouping

Aggregate by categories:

```sql theme={null}
SELECT
  category,
  COUNT(1) AS count,
  SUM(amount) AS total,
  AVG(amount) AS average
FROM company_data."123"
GROUP BY category
```

### Filtering aggregated results

Use `HAVING` to filter after aggregation:

```sql theme={null}
SELECT
  category,
  COUNT(1) AS count
FROM company_data."123"
GROUP BY category
HAVING COUNT(1) > 100
```

***

## Deduplication with QUALIFY

Remove duplicates based on a window function:

```sql theme={null}
SELECT
  user_id,
  email,
  last_seen
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

This keeps only the most recent record for each user.

***

## Putting it together

Here's a complete example combining multiple techniques:

```sql theme={null}
SELECT
  user_id,
  LOWER(email) AS normalized_email,
  CASE
    WHEN age < 25 THEN '18-24'
    WHEN age < 35 THEN '25-34'
    WHEN age < 45 THEN '35-44'
    ELSE '45+'
  END AS age_bracket,
  identifiers[0].type AS primary_id_type,
  created_at
FROM company_data."123"
WHERE
  email IS NOT NULL
  AND status = 'active'
  AND created_at >= CURRENT_DATE - INTERVAL '90' DAY
  AND _price_cpm_usd <= 0.50
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY created_at DESC
) = 1
ORDER BY created_at DESC
LIMIT 1000 ROWS
```

***

## Related content

<CardGroup>
  <Card title="Joining Datasets" icon="link" href="/guides/nql/joining-datasets">
    Combine data from multiple sources
  </Card>

  <Card title="NQL Syntax Reference" icon="code" href="/nql/general/syntax">
    Complete syntax documentation
  </Card>

  <Card title="Operators Reference" icon="calculator" href="/nql/operators">
    All comparison and logical operators
  </Card>

  <Card title="Functions Reference" icon="function" href="/nql/functions/index">
    String, date, and transformation functions
  </Card>
</CardGroup>


# Incremental Upserts with MERGE ON
Source: https://docs.narrative.io/guides/nql/incremental-upserts

Update existing records and insert new ones during materialized view refreshes

When building materialized views that refresh periodically, you often need to update existing records rather than simply appending new ones. The `MERGE ON` clause lets you match incoming rows against existing data and decide whether to update or insert—avoiding duplicates while keeping data fresh.

## Prerequisites

Before using MERGE ON, you should have:

* Familiarity with [materialized views](/concepts/nql/materialized-views)
* Understanding of [write modes](/nql/commands/create-materialized-view#write_mode) (append vs overwrite)
* A dataset with stable keys suitable for matching

## The problem

Consider a view that refreshes monthly with recent activity data. With `WRITE_MODE = 'overwrite'`, each refresh replaces all data—losing historical records. With `WRITE_MODE = 'append'`, each refresh adds new rows—but if the same record appears in multiple refreshes, you get duplicates.

MERGE ON solves this by matching rows between the query result and existing view data, then updating matches and inserting non-matches.

## How MERGE ON works

MERGE ON compares each row from your query (the **source**) against existing rows in the materialized view (the **target**). Based on the match condition, it either updates the existing row or inserts a new one.

### Source and target aliases

NQL automatically generates two aliases for use in MERGE ON clauses:

| Alias    | Description                                            |
| -------- | ------------------------------------------------------ |
| `source` | References columns from the incoming query result      |
| `target` | References columns from the existing materialized view |

Use these aliases in your MERGE ON condition and in the WHEN MATCHED/WHEN NOT MATCHED clauses.

## Use case 1: Incremental upsert with time-bounded matching

This example maintains an audience feed that refreshes monthly. The MERGE ON condition restricts updates to rows considered "fresh" (modified within the last 20 days), which can help respect license refresh windows or avoid stale overwrites.

```sql theme={null}
CREATE MATERIALIZED VIEW "ongoing_Feed"
REFRESH_SCHEDULE = '@monthly'
WRITE_MODE = 'append'
AS
SELECT
    "ip_address" AS ip,
    "sha256_hashed_email"."value" AS sha256,
    "sha1_hashed_email"."value" AS sha1,
    "md5_hashed_email"."value" AS md5,
    "mobile_id_unique_identifier"."value" AS maid,
    "mobile_id_unique_identifier"."type" AS maid_type,
    CAST("event_timestamp" AS TIMESTAMP) AS "event_timestamp",
    CURRENT_TIMESTAMP as last_modified_at
FROM
    company_data.audience_feed
WHERE
    "event_timestamp" >= CURRENT_TIMESTAMP - INTERVAL '30' DAY
    AND "sha256_hashed_email"."value" IS NOT NULL
    AND "ip_address" IS NOT NULL
    AND "mobile_id_unique_identifier"."value" IS NOT NULL
MERGE ON target.last_modified_at > CURRENT_TIMESTAMP - INTERVAL '20' DAY
WHEN MATCHED THEN UPDATE SET
  ip = source.ip,
  sha256 = source.sha256,
  sha1 = source.sha1,
  md5 = source.md5,
  maid = source.maid,
  maid_type = source.maid_type,
  event_timestamp = source.event_timestamp,
  last_modified_at = source.last_modified_at
WHEN NOT MATCHED THEN INSERT (
  ip,
  sha256,
  sha1,
  md5,
  maid,
  maid_type,
  event_timestamp,
  last_modified_at
) VALUES (
  source.ip,
  source.sha256,
  source.sha1,
  source.md5,
  source.maid,
  source.maid_type,
  source.event_timestamp,
  source.last_modified_at
);
```

**Key points:**

* The time-bounded condition (`target.last_modified_at > CURRENT_TIMESTAMP - INTERVAL '20' DAY`) is optional but useful for limiting which rows are considered for updates
* `WRITE_MODE = 'append'` is required for incremental merge semantics
* The `last_modified_at` column tracks when each row was last updated

## Use case 2: Enrichment with composite keys

This example enriches internal user data with geographic information from Rosetta Stone. It uses a composite key (user\_id + sha256) with `IS NOT DISTINCT FROM` to handle nullable values safely.

```sql theme={null}
CREATE MATERIALIZED VIEW "email_geo_enrichment"
WRITE_MODE = 'append'
REFRESH_SCHEDULE = '@weekly'
AS (
  SELECT
    ds.user_id,
    rs."sha256_hashed_email"."value" AS sha256,
    rs."geo_country_code" AS country_code,
    rs."geo_region" AS region,
    rs."nio_last_modified_at" AS rs_modified_at,
    CURRENT_TIMESTAMP AS last_modified_at
  FROM company_data."internal_users" ds
  JOIN narrative."rosetta_stone" rs
    ON rs."unique_id"."value" = ds.unique_id
)
MERGE ON
  target.user_id IS NOT DISTINCT FROM source.user_id
  AND target.sha256 IS NOT DISTINCT FROM source.sha256
WHEN MATCHED THEN UPDATE SET
  country_code = source.country_code,
  region = source.region,
  rs_modified_at = source.rs_modified_at,
  last_modified_at = source.last_modified_at
WHEN NOT MATCHED THEN INSERT (
  user_id,
  sha256,
  country_code,
  region,
  rs_modified_at,
  last_modified_at
) VALUES (
  source.user_id,
  source.sha256,
  source.country_code,
  source.region,
  source.rs_modified_at,
  source.last_modified_at
);
```

**Key points:**

* `IS NOT DISTINCT FROM` treats two NULL values as equal, unlike `=` which returns NULL when comparing NULLs
* Composite keys work well when no single column uniquely identifies a record
* The enriched columns (country\_code, region) update on match while the key columns remain stable

## Best practices

| Practice                                       | Why                                                                                       |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Use a stable, unique key for matching          | Non-unique or changing keys cause repeated inserts or updates to wrong rows               |
| Use `IS NOT DISTINCT FROM` for nullable fields | Plain `=` on nullable fields can misclassify matches (NULL = NULL returns NULL, not true) |
| Consider time-bounded conditions               | Scoping to recent data respects license windows and improves performance                  |
| Always use `WRITE_MODE = 'append'`             | Required for incremental merge semantics; overwrite mode replaces all data                |
| Keep MERGE ON conditions minimal               | Overly broad conditions cause frequent rewrites; scope to the minimum key set needed      |

## Common pitfalls

<Warning>
  **Non-unique match keys**

  If your MERGE ON condition matches multiple target rows for a single source row, you may see unexpected updates or duplicate inserts. Ensure your match key (or composite key) uniquely identifies records.
</Warning>

<Warning>
  **Using = on nullable fields**

  Using plain `=` to compare nullable fields can cause matches to be missed:

  ```sql theme={null}
  -- Wrong: will not match when both values are NULL
  MERGE ON target.user_id = source.user_id

  -- Correct: treats NULL = NULL as a match
  MERGE ON target.user_id IS NOT DISTINCT FROM source.user_id
  ```
</Warning>

<Warning>
  **MERGE ON only works in CREATE MATERIALIZED VIEW**

  MERGE ON is a clause within the `CREATE MATERIALIZED VIEW` statement. It cannot be used as a standalone MERGE command in Data Studio. For row-level changes outside a materialized view refresh, use standalone [INSERT](/nql/commands/insert), [UPDATE](/nql/commands/update), or [DELETE](/nql/commands/delete) statements instead.
</Warning>

***

## Related content

<CardGroup>
  <Card title="Materialized View Syntax" icon="list-check" href="/nql/commands/create-materialized-view">
    Complete reference for MERGE ON and other options
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    How NQL optimizes refreshes for large datasets
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    When and why to use materialized views
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Basic guide to creating materialized views
  </Card>
</CardGroup>


# Joining Datasets
Source: https://docs.narrative.io/guides/nql/joining-datasets

Combine data from multiple datasets using JOINs

This guide covers how to combine data from multiple datasets using JOIN operations. JOINs are essential for enriching your data with information from other sources.

## Prerequisites

* A Narrative I/O account with access to multiple datasets
* Understanding of [filtering and transforming data](/guides/nql/filtering-transforming)

## What you'll learn

* How to use INNER JOIN to match records across datasets
* How to use LEFT JOIN for optional matches
* How to chain multiple JOINs
* How to use anti-joins to find unmatched records
* How to join with Rosetta Stone for identity resolution
* Performance considerations for large joins

***

## INNER JOIN

An INNER JOIN returns only rows that have matching values in both datasets.

### Basic syntax

```sql theme={null}
SELECT
  t1.column1,
  t2.column2
FROM company_data."123" t1
INNER JOIN company_data."456" t2
  ON t1.id = t2.id
```

### Example: Enriching customer data

Combine customer information with their orders:

```sql theme={null}
SELECT
  customers.customer_id,
  customers.email,
  orders.order_id,
  orders.order_date,
  orders.total_amount
FROM company_data."100" customers
INNER JOIN company_data."200" orders
  ON customers.customer_id = orders.customer_id
WHERE
  orders.order_date >= CURRENT_DATE - INTERVAL '30' DAY
```

<Note>
  Only customers who have orders (and orders with valid customers) appear in the results. Customers without orders are excluded.
</Note>

### Using table aliases

Aliases make queries more readable and are required when referencing columns:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  o.order_total
FROM company_data."100" AS c
INNER JOIN company_data."200" AS o
  ON c.customer_id = o.customer_id
```

***

## LEFT JOIN

A LEFT JOIN returns all rows from the left table, plus matching rows from the right table. Unmatched rows have NULL values for right-table columns.

### Basic syntax

```sql theme={null}
SELECT
  t1.column1,
  t2.column2
FROM company_data."123" t1
LEFT JOIN company_data."456" t2
  ON t1.id = t2.id
```

### Example: Find customers with or without orders

```sql theme={null}
SELECT
  c.customer_id,
  c.email,
  COALESCE(o.order_count, 0) AS order_count
FROM company_data."100" c
LEFT JOIN (
  SELECT customer_id, COUNT(1) AS order_count
  FROM company_data."200"
  GROUP BY customer_id
) o ON c.customer_id = o.customer_id
```

This returns all customers. Those without orders show `order_count = 0`.

### Filtering LEFT JOIN results

To find records without matches, filter for NULL:

```sql theme={null}
SELECT
  c.customer_id,
  c.email
FROM company_data."100" c
LEFT JOIN company_data."200" o
  ON c.customer_id = o.customer_id
WHERE o.customer_id IS NULL  -- No matching orders
```

***

## Multiple JOINs

Chain JOINs to combine data from three or more datasets:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  o.order_id,
  p.product_name
FROM company_data."100" c
INNER JOIN company_data."200" o
  ON c.customer_id = o.customer_id
INNER JOIN company_data."300" p
  ON o.product_id = p.product_id
WHERE
  o.order_date >= '2024-01-01'
```

### Mixing JOIN types

You can mix INNER and LEFT JOINs:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  o.order_id,
  r.rating
FROM company_data."100" c
INNER JOIN company_data."200" o
  ON c.customer_id = o.customer_id
LEFT JOIN company_data."300" r
  ON o.order_id = r.order_id
```

This returns all customers with orders, optionally including ratings if they exist.

***

## Anti-joins with NOT EXISTS

Find records in one dataset that don't have matches in another:

### Basic anti-join

```sql theme={null}
SELECT
  c.customer_id,
  c.email
FROM company_data."100" c
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."200" o
  WHERE c.customer_id = o.customer_id
)
```

This returns customers who have never placed an order.

### Example: Find inactive users

Find users who haven't logged in recently:

```sql theme={null}
SELECT
  u.user_id,
  u.email,
  u.signup_date
FROM company_data."100" u
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."200" logins
  WHERE u.user_id = logins.user_id
    AND logins.login_date >= CURRENT_DATE - INTERVAL '90' DAY
)
```

***

## Joining with Rosetta Stone

Rosetta Stone enables identity resolution across datasets using normalized identifiers.

### Access Rosetta Stone identifiers

```sql theme={null}
SELECT
  d.user_attribute,
  d._rosetta_stone.unique_id
FROM company_data."123" d
```

### Join datasets via Rosetta Stone

Match records across datasets using resolved identities:

```sql theme={null}
SELECT
  d1.attribute1,
  d2.attribute2
FROM company_data."123" d1
INNER JOIN company_data."456" d2
  ON d1._rosetta_stone.unique_id = d2._rosetta_stone.unique_id
```

### Query Rosetta Stone directly

```sql theme={null}
SELECT
  unique_id,
  event_timestamp
FROM narrative.rosetta_stone
WHERE
  unique_id.b."type" = 'email'
  AND event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

<Note>
  Rosetta Stone queries can be scoped to different levels: global (`narrative.rosetta_stone`), company-specific (`company_data._rosetta_stone`), or dataset-specific (`company_data."123"._rosetta_stone`). See [Rosetta Stone table references](/nql/general/syntax#rosetta-stone-table-references) for complete syntax.
</Note>

***

## Join conditions

### Multiple conditions

Join on multiple columns:

```sql theme={null}
SELECT
  t1.value,
  t2.related_value
FROM company_data."123" t1
INNER JOIN company_data."456" t2
  ON t1.key1 = t2.key1
  AND t1.key2 = t2.key2
```

### Complex conditions

Use expressions in join conditions:

```sql theme={null}
SELECT
  events.event_id,
  sessions.session_id
FROM company_data."123" events
INNER JOIN company_data."456" sessions
  ON events.user_id = sessions.user_id
  AND events.event_timestamp BETWEEN sessions.start_time AND sessions.end_time
```

### NULL-safe joins

Use `IS NOT DISTINCT FROM` for NULL-safe equality:

```sql theme={null}
SELECT
  t1.id,
  t1.nullable_key,
  t2.value,
  t2.nullable_key AS t2_nullable_key
FROM company_data."123" t1
INNER JOIN company_data."456" t2
  ON t1.nullable_key IS NOT DISTINCT FROM t2.nullable_key
```

***

## Self-joins

Join a dataset to itself for comparisons:

```sql theme={null}
SELECT
  e1.employee_id,
  e1.name AS employee_name,
  e2.name AS manager_name
FROM company_data."123" e1
LEFT JOIN company_data."123" e2
  ON e1.manager_id = e2.employee_id
```

***

## Subqueries in JOINs

Join with aggregated or transformed data:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  stats.total_spent,
  stats.order_count
FROM company_data."100" c
INNER JOIN (
  SELECT
    customer_id,
    SUM(total_amount) AS total_spent,
    COUNT(1) AS order_count
  FROM company_data."200"
  GROUP BY customer_id
) stats ON c.customer_id = stats.customer_id
WHERE stats.total_spent > 1000
```

***

## Performance considerations

### Use fully qualified column names

Always qualify column names in joins to avoid ambiguity and improve query parsing:

```sql theme={null}
-- Good: Fully qualified
SELECT
  company_data."123".column1,
  company_data."456".column2
FROM company_data."123"
INNER JOIN company_data."456"
  ON company_data."123".id = company_data."456".id

-- Also good: With aliases
SELECT
  t1.column1,
  t2.column2
FROM company_data."123" t1
INNER JOIN company_data."456" t2
  ON t1.id = t2.id
```

### Filter before joining

Apply filters as early as possible to reduce the data being joined:

```sql theme={null}
-- Better: Filter in subqueries
SELECT
  c.customer_id,
  o.order_id
FROM (
  SELECT customer_id, email, status FROM company_data."100"
  WHERE status = 'active'
) c
INNER JOIN (
  SELECT order_id, customer_id, order_date FROM company_data."200"
  WHERE order_date >= '2024-01-01'
) o ON c.customer_id = o.customer_id

-- Or use CTEs for clarity
WITH active_customers AS (
  SELECT customer_id, email, status FROM company_data."100"
  WHERE status = 'active'
),
recent_orders AS (
  SELECT order_id, customer_id, order_date FROM company_data."200"
  WHERE order_date >= '2024-01-01'
)
SELECT
  c.customer_id,
  o.order_id
FROM active_customers c
INNER JOIN recent_orders o
  ON c.customer_id = o.customer_id
```

### Avoid OR in join conditions

OR conditions in joins can cause performance issues:

```sql theme={null}
-- Avoid
ON t1.id = t2.id OR t1.alt_id = t2.alt_id

-- Better: Use COALESCE or separate queries
ON COALESCE(t1.id, t1.alt_id) = COALESCE(t2.id, t2.alt_id)
```

See [Query Optimization](/guides/nql/query-optimization) for more performance tips.

### Include budget controls

Always include budget controls when joining datasets in materialized views:

```sql theme={null}
CREATE MATERIALIZED VIEW "enriched_customers"
AS (
  SELECT
    c.customer_id,
    c.email,
    o.total_orders
  FROM company_data."100" c
  INNER JOIN (
    SELECT customer_id, COUNT(1) AS total_orders
    FROM company_data."200"
    GROUP BY customer_id
  ) o ON c.customer_id = o.customer_id
)
BUDGET 50 USD
```

***

## Common patterns

### Enrich with latest record

Get the most recent related record for each row:

```sql theme={null}
SELECT
  c.customer_id,
  c.name,
  latest.last_order_date,
  latest.last_order_amount
FROM company_data."100" c
INNER JOIN (
  SELECT
    customer_id,
    order_date AS last_order_date,
    total_amount AS last_order_amount
  FROM company_data."200"
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY customer_id
    ORDER BY order_date DESC
  ) = 1
) latest ON c.customer_id = latest.customer_id
```

### Aggregate before joining

Pre-aggregate to reduce join complexity:

```sql theme={null}
SELECT
  c.customer_id,
  c.segment,
  metrics.total_value,
  metrics.avg_order
FROM company_data."100" c
INNER JOIN (
  SELECT
    customer_id,
    SUM(amount) AS total_value,
    AVG(amount) AS avg_order
  FROM company_data."200"
  GROUP BY customer_id
) metrics ON c.customer_id = metrics.customer_id
```

***

## Related content

<CardGroup>
  <Card title="Join Performance" icon="gauge-high" href="/concepts/nql/join-performance">
    Understanding performance implications
  </Card>

  <Card title="Query Optimization" icon="bolt" href="/guides/nql/query-optimization">
    Tips for faster queries
  </Card>

  <Card title="Complex Joins Cookbook" icon="book" href="/cookbooks/nql/complex-joins">
    Advanced join patterns and examples
  </Card>

  <Card title="Rosetta Stone" icon="language" href="/concepts/rosetta-stone/overview">
    Identity resolution for cross-dataset matching
  </Card>
</CardGroup>


# Query Optimization
Source: https://docs.narrative.io/guides/nql/query-optimization

Write faster, more efficient NQL queries

Efficient queries are essential for working with large datasets in Narrative. This section covers optimization techniques and patterns to help your queries run faster and use fewer resources.

## Prerequisites

Before diving into optimization, you should have:

* Basic familiarity with [NQL syntax](/nql/general/syntax)
* Understanding of [JOIN operations](/guides/nql/joining-datasets)

## Optimization guides

<CardGroup>
  <Card title="Avoid OR in JOIN Clauses" icon="ban" href="/guides/nql/query-optimization/avoid-or-in-join">
    Restructure OR conditions into single-key joins for dramatically better performance
  </Card>
</CardGroup>

## Related content

<CardGroup>
  <Card title="Understanding JOIN Performance" icon="chart-line" href="/concepts/nql/join-performance">
    Learn why certain JOIN patterns cause performance problems
  </Card>

  <Card title="NQL Functions Reference" icon="code" href="/nql/functions/index">
    Complete reference for NQL functions including UNNEST
  </Card>

  <Card title="Joining Datasets" icon="code-merge" href="/guides/nql/joining-datasets">
    Fundamentals of combining data from multiple datasets
  </Card>
</CardGroup>


# Avoid OR in JOIN Clauses
Source: https://docs.narrative.io/guides/nql/query-optimization/avoid-or-in-join

Restructure OR conditions into single-key joins for dramatically better query performance

When joining datasets on multiple possible keys, you might be tempted to use OR in your JOIN condition. This pattern prevents the query planner from optimizing the join, often causing queries to run for hours instead of seconds.

<Warning>
  Using OR in JOIN conditions can make queries 100x slower or more. Always restructure OR conditions into single-key joins using the techniques in this guide.
</Warning>

## The anti-pattern

This query attempts to match records where either `user_id` or `email` matches:

```sql theme={null}
SELECT p.purchase_id, p.amount, u.name, u.email
FROM Purchases p
JOIN Users u
  ON p.user_id = u.id
  OR p.email = u.email;
```

While logically correct, this pattern prevents the query planner from using efficient join algorithms. The planner cannot build a hash table or use indexes effectively when it must evaluate two conditions with OR.

## Solution 1: Array flattening with UNNEST

The recommended approach is to combine your join keys into an array, flatten it with UNNEST, and then join on a single key column.

### How it works

1. **Combine keys into an array** on each side of the join
2. **UNNEST the arrays** to create one row per potential key
3. **JOIN on the single key column** using standard equality

### Example

```sql theme={null}
WITH Purchases_flat AS (
    SELECT
      p.*,
      key
    FROM Purchases p
    CROSS JOIN UNNEST([p.user_id, p.email]) AS t(key)
),
Users_flat AS (
    SELECT
      u.*,
      key
    FROM Users u
    CROSS JOIN UNNEST([u.id, u.email]) AS t(key)
)
SELECT DISTINCT
  pf.purchase_id,
  pf.item,
  pf.amount,
  uf.user_name,
  uf.email
FROM Purchases_flat pf
JOIN Users_flat uf
  ON pf.key = uf.key;
```

This query:

* Creates arrays containing both possible join keys
* Uses UNNEST to expand each row into multiple rows (one per key)
* Joins on a single `key` column, which the planner can optimize efficiently
* Uses `SELECT DISTINCT` to remove duplicate matches

<Tip>
  The UNNEST approach works well when you have more than two possible join keys or when the keys are already stored as arrays.
</Tip>

## Solution 2: Using UNION

For simple cases with exactly two join conditions, UNION provides a cleaner alternative:

```sql theme={null}
SELECT
  p.purchase_id,
  p.item,
  p.amount,
  u.user_name,
  u.email
FROM Purchases p
JOIN Users u ON p.user_id = u.id

UNION

SELECT
  p.purchase_id,
  p.item,
  p.amount,
  u.user_name,
  u.email
FROM Purchases p
JOIN Users u ON p.email = u.email;
```

This approach:

* Runs two optimized single-key joins separately
* Combines the results and automatically removes duplicates with UNION
* May be easier to read for simple two-condition cases

<Note>
  Use `UNION ALL` instead of `UNION` if you want to keep duplicates, but be aware this may produce unexpected results when both conditions match the same record.
</Note>

## Choosing the right approach

| Approach   | Best for                                               | Trade-offs                                                   |
| ---------- | ------------------------------------------------------ | ------------------------------------------------------------ |
| **UNNEST** | Multiple keys, complex queries, keys already in arrays | Requires explicit DISTINCT, slightly more complex syntax     |
| **UNION**  | Exactly two conditions, readability is priority        | Repeats query logic, harder to maintain with many conditions |

Both approaches transform the problematic OR condition into single-key joins that the query planner can optimize using hash joins or indexes.

<Note>
  To understand why OR conditions cause these performance problems, see [Understanding JOIN Performance](/concepts/nql/join-performance).
</Note>

## Related content

<CardGroup>
  <Card title="Understanding JOIN Performance" icon="chart-line" href="/concepts/nql/join-performance">
    Learn why OR in JOINs prevents optimization
  </Card>

  <Card title="Query Optimization" icon="gauge-high" href="/guides/nql/query-optimization">
    More techniques for writing efficient queries
  </Card>

  <Card title="Joining Datasets" icon="code-merge" href="/guides/nql/joining-datasets">
    Fundamentals of combining data from multiple datasets
  </Card>

  <Card title="NQL Functions" icon="code" href="/nql/functions/index">
    Reference for UNNEST and other NQL functions
  </Card>
</CardGroup>


# NQL Troubleshooting
Source: https://docs.narrative.io/guides/nql/troubleshooting

Diagnose and resolve common NQL query errors

When an NQL query fails, the platform returns a structured error response with details about what went wrong. This section helps you diagnose and resolve common errors.

## Understanding error responses

NQL errors return a JSON response with the following structure:

```json theme={null}
{
  "type": "https://www.narrative.io/knowledge-base/...",
  "title": "Error Title",
  "status": 422,
  "detail": "Detailed explanation of what went wrong",
  "instance": "/nql/run",
  "logId": "unique-error-identifier"
}
```

| Field      | Description                                            |
| ---------- | ------------------------------------------------------ |
| `type`     | URL identifying the error type                         |
| `title`    | Brief error summary                                    |
| `status`   | HTTP status code (typically 422 for validation errors) |
| `detail`   | Detailed explanation of the problem                    |
| `instance` | The API endpoint that produced the error               |
| `logId`    | Unique identifier for support requests                 |

<Tip>
  When contacting support about an error, include the `logId` value to help quickly identify your specific issue.
</Tip>

## Common errors

<CardGroup>
  <Card title="Cross-Data Plane Queries" icon="database" href="/guides/nql/troubleshooting/cross-data-plane-queries">
    Resolve errors when queries reference datasets in different data planes
  </Card>

  <Card title="Unsupported Type Errors" icon="ban" href="/guides/nql/troubleshooting/unsupported-type-error">
    Fix type mismatches and unsupported operations in your queries
  </Card>
</CardGroup>

## General troubleshooting steps

1. **Read the error detail** — The `detail` field usually explains exactly what's wrong
2. **Check the instance** — Identifies whether the error occurred during parsing (`/nql/parse`) or execution (`/nql/run`)
3. **Review your query** — Look for the specific element mentioned in the error
4. **Consult the reference** — Check [NQL syntax](/nql/general/syntax) and [data types](/nql/data-types) for supported operations


# Cross-Data Plane Query Errors
Source: https://docs.narrative.io/guides/nql/troubleshooting/cross-data-plane-queries

Resolve errors when queries reference data across multiple data planes

NQL queries execute within a single [data plane](/concepts/primitives/data-planes). If your query references datasets that reside in different data planes, the query will fail.

<Warning>
  NQL cannot join or query across multiple data planes. All datasets referenced in a query must exist within the same data plane.
</Warning>

## Why this happens

A data plane is the infrastructure where your data physically resides and where queries execute. Narrative's architecture separates data planes to enable:

* **Data residency compliance** — Keep data in specific geographic or organizational boundaries
* **Security isolation** — Maintain separate security perimeters for different data sources
* **Infrastructure flexibility** — Run data planes on different platforms (Narrative-hosted, Snowflake, Spark)

Because each data plane is an independent execution environment, the query engine cannot access data across data plane boundaries in a single query.

For a deeper understanding of data plane architecture, see [Data Planes](/concepts/primitives/data-planes).

## Troubleshooting steps

### 1. Identify which datasets are in different data planes

Review your query and identify all datasets referenced. In the Narrative platform:

1. Navigate to the dataset catalog
2. Check the data plane assignment for each dataset in your query
3. Note which datasets belong to different data planes

### 2. Modify your query

Once you've identified the issue, you have two options:

**Option A: Query datasets separately**

Run separate queries against each data plane and combine results outside of NQL:

```sql theme={null}
-- Query 1: Run against Data Plane A
SELECT user_id, email, created_at FROM "companydata"."dataset_in_plane_a" WHERE ...

-- Query 2: Run against Data Plane B
SELECT user_id, email, created_at FROM "companydata"."dataset_in_plane_b" WHERE ...

-- Combine results in your application
```

**Option B: Move data to a common data plane**

If you need to join datasets frequently, consider:

1. Creating a materialized view that copies the necessary data to a single data plane
2. Working with your Narrative representative to consolidate datasets

<Note>
  Data movement between data planes may have compliance implications. Consult your data governance team before moving data.
</Note>

## Related content

<CardGroup>
  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understand where your data lives and is processed
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries flow from submission through execution
  </Card>

  <Card title="Control Plane" icon="sitemap" href="/concepts/primitives/control-plane">
    How Narrative orchestrates across data planes
  </Card>

  <Card title="NQL Troubleshooting" icon="triangle-exclamation" href="/guides/nql/troubleshooting">
    Overview of common NQL errors
  </Card>
</CardGroup>


# Unsupported Type Errors
Source: https://docs.narrative.io/guides/nql/troubleshooting/unsupported-type-error

Resolve errors when NQL queries use unsupported data types or operations

NQL validates data types during query parsing and execution. When a query includes unsupported types or incompatible type operations, the platform returns a 422 error with details about the type issue.

## GEOMETRY type not supported in SELECT

The GEOMETRY type cannot be used directly in SELECT statements. This error occurs when your query attempts to return a geometry column or the result of a geometry function.

### Error response

```json theme={null}
{
  "type": "https://www.narrative.io/knowledge-base/support-troubleshooting/nql/unsupported-type-error",
  "title": "Unsupported Type Error: Unsupported GEOMETRY Type",
  "status": 422,
  "detail": "The NQL query includes the unsupported GEOMETRY type. The request cannot be processed because the GEOMETRY type is not supported in the SELECT.",
  "instance": "/nql/run",
  "logId": "2cf8fef9-9dc2-4438-b6f9-e28c386e48cc"
}
```

### Cause

Geometry functions like `STCIRCLE()` return GEOMETRY types that cannot be serialized in query results. While you can use geometry functions in JOIN conditions and WHERE clauses, you cannot SELECT them directly.

### Solution

Remove the GEOMETRY column from your SELECT statement. Keep geometry operations only in JOIN or WHERE clauses.

**Before (causes error):**

```sql theme={null}
SELECT
  maid,
  STCIRCLE(longitude, latitude, 15) AS geofence  -- GEOMETRY in SELECT
FROM companydata.locations
```

**After (fixed):**

```sql theme={null}
SELECT
  maid,
  longitude,
  latitude
FROM companydata.locations
```

If you need to use geometry for filtering, keep it in the JOIN or WHERE clause:

```sql theme={null}
SELECT
  observations.maid,
  observations.event_timestamp,
  pois.name AS poi_name
FROM companydata.observations
INNER JOIN companydata.pois
  ON STIntersects(
    STSafePoint(observations.longitude, observations.latitude),
    STCIRCLE(pois.longitude, pois.latitude, 15)
  )
```

## String concatenation type mismatch

The `||` operator concatenates strings but requires compatible types on both sides. This error occurs when you try to concatenate incompatible types.

### Error response

```json theme={null}
{
  "type": "https://www.narrative.io/knowledge-base/support-troubleshooting/nql/unsupported-type-error",
  "title": "Unsupported Type Error: String Concatenation",
  "status": 422,
  "detail": "The NQL query encountered an unsupported type error when attempting to concatenate different types using the '||' operator. The '||' operator can only concatenate :string || :string or <EQUIVALENTTYPE> || <EQUIVALENTTYPE> and is not compatible with '<RECORDTYPE:PEEK_DEFAULT(VARCHAR TYPE, VARCHAR VALUE)> || <CHAR(1)>'.",
  "instance": "/nql/parse",
  "logId": "7798e641-5d5e-47d3-a884-f5271e788b6b"
}
```

### Cause

The `||` operator requires both operands to be strings (or equivalent string types). Common causes include:

* Concatenating a complex record type with a string
* Mixing incompatible character types
* Using structured fields that haven't been extracted to simple strings

### Solution

Extract string values from complex types before concatenation, or cast values to compatible string types.

**Before (causes error):**

```sql theme={null}
SELECT
  "hashedemail" || '-' || "mobileiduniqueidentifier" || '-' || "eventtimestamp" AS unique_key
FROM companydata.events
```

**After (fixed):**

If the fields are structured types with nested values, extract the string value first:

```sql theme={null}
SELECT
  "hashedemail"."value" || '-' || "mobileiduniqueidentifier"."value" || '-' || CAST("eventtimestamp" AS VARCHAR) AS unique_key
FROM companydata.events
```

<Tip>
  When concatenating timestamps or other non-string types, use `CAST(field AS VARCHAR)` to convert them to strings first.
</Tip>

## General tips for type errors

1. **Check the `detail` field** — It identifies the specific types that are incompatible
2. **Review structured fields** — Many fields in Narrative are record types with `.value` subfields
3. **Use explicit casts** — When in doubt, cast values to the expected type with `CAST(field AS type)`
4. **Check the instance** — `/nql/parse` errors occur during query validation; `/nql/run` errors occur during execution

## Related content

<CardGroup>
  <Card title="NQL Data Types" icon="list-check" href="/nql/data-types">
    Reference for supported data types
  </Card>

  <Card title="NQL Functions" icon="code" href="/nql/functions/index">
    Reference for built-in functions including geometry operations
  </Card>

  <Card title="NQL Troubleshooting" icon="triangle-exclamation" href="/guides/nql/troubleshooting">
    Overview of common NQL errors
  </Card>

  <Card title="Rosetta Stone Attributes" icon="language" href="/reference/rosetta-stone/attribute-types">
    Understanding structured attribute types
  </Card>
</CardGroup>


# Using Query Templates
Source: https://docs.narrative.io/guides/nql/using-query-templates

Create and use reusable queries with configurable placeholders

Query templates let you build reusable queries that others can customize without editing NQL. This guide walks through finding existing templates, using them, and creating your own.

## Prerequisites

Before working with query templates, you should have:

* Basic familiarity with [NQL syntax](/nql/general/syntax)
* Access to Data Studio
* Understanding of [what query templates are](/concepts/nql/query-templates) and when to use them

## Finding templates

To browse available templates:

1. Navigate to **My Data** in the main navigation
2. Select **Queries**
3. Click **Query Templates**

The template list shows all templates available to your organization. Each entry displays the template name, description, and the placeholders it requires.

## Using an existing template

### Select a template

From the Query Templates list, click on the template you want to use. The template opens showing:

* The template's description and purpose
* A list of placeholders with their names, types, and descriptions
* Which placeholders are required vs. optional

### Fill in placeholders

For each placeholder, enter an appropriate value:

* **Required placeholders** must have values before you can execute the query
* **Optional placeholders** use default values if left empty
* The placeholder type determines what values are valid (dates, numbers, column names, etc.)

<Tip>
  Placeholder descriptions explain what each value controls. Read them carefully to understand how your inputs affect the query results.
</Tip>

### Execute the query

Once all required placeholders are filled:

1. Click **Execute** to run the query
2. Review the results in the query output panel
3. Optionally, save the executed query for future reference

## Creating a template

### Enter Template Mode

To create a new template:

1. Open **Data Studio**
2. Click **Template Mode** in the toolbar

Template Mode enables placeholder insertion and template-specific options.

### Write your base query

Start by writing a working NQL query that produces the results you want. Test it with specific values to verify it works correctly:

```sql theme={null}
SELECT
    region,
    COUNT(1) as user_count,
    SUM(revenue) as total_revenue
FROM sales_data
WHERE
    sale_date >= '2024-01-01'
    AND sale_date < '2024-02-01'
    AND region = 'North America'
GROUP BY region
```

### Add placeholders

Identify the parts of your query that should be variable, then replace them with placeholders using `{{placeholder_name}}` syntax:

```sql theme={null}
SELECT
    region,
    COUNT(1) as user_count,
    SUM(revenue) as total_revenue
FROM sales_data
WHERE
    sale_date >= {{start_date}}
    AND sale_date < {{end_date}}
    AND region = {{target_region}}
GROUP BY region
```

### Define placeholder metadata

For each placeholder, specify:

* **Name**: A clear, descriptive identifier
* **Type**: The data type (string, number, date, etc.)
* **Description**: What this placeholder controls
* **Required**: Whether users must provide a value
* **Default** (optional): A fallback value for optional placeholders

<Tip>
  Good placeholder names are self-documenting. `start_date` is clearer than `d1`. `min_revenue_threshold` is clearer than `thresh`.
</Tip>

### Save the template

1. Click **Save Template**
2. Enter a template name that describes its purpose
3. Add a description explaining what the template does and when to use it
4. Add tags to help others find the template
5. Click **Save**

## Editing templates

To modify an existing template:

1. Navigate to **Queries** → **Query Templates**
2. Click on the template you want to edit
3. Click **Edit**
4. Modify the query or placeholder definitions as needed
5. Save your changes

<Warning>
  Editing a template affects all future uses. If others rely on the template, communicate changes before modifying placeholder names or removing placeholders.
</Warning>

## Example: Date-range report template

Let's create a template for a sales report that users can run for any date range and region.

### Step 1: Write the base query

```sql theme={null}
SELECT
    DATE(sale_timestamp) as sale_date,
    product_category,
    COUNT(1) as transactions,
    SUM(amount) as revenue
FROM sales
WHERE
    sale_timestamp >= '2024-01-01'
    AND sale_timestamp < '2024-02-01'
    AND region = 'EMEA'
GROUP BY DATE(sale_timestamp), product_category
ORDER BY sale_date, revenue DESC
```

### Step 2: Identify variable parts

In this query, three things might change:

* Start date (`2024-01-01`)
* End date (`2024-02-01`)
* Region (`EMEA`)

### Step 3: Add placeholders

```sql theme={null}
SELECT
    DATE(sale_timestamp) as sale_date,
    product_category,
    COUNT(1) as transactions,
    SUM(amount) as revenue
FROM sales
WHERE
    sale_timestamp >= {{start_date}}
    AND sale_timestamp < {{end_date}}
    AND region = {{region}}
GROUP BY DATE(sale_timestamp), product_category
ORDER BY sale_date, revenue DESC
```

### Step 4: Define placeholder metadata

| Placeholder  | Type   | Required | Description                                   |
| ------------ | ------ | -------- | --------------------------------------------- |
| `start_date` | date   | Yes      | Beginning of the reporting period (inclusive) |
| `end_date`   | date   | Yes      | End of the reporting period (exclusive)       |
| `region`     | string | Yes      | Geographic region to filter by                |

### Step 5: Save and test

Save the template as "Regional Sales Report" with a description like "Daily sales metrics by product category for a specified region and date range."

Test with different placeholder values to verify results match expectations.

## Example: Materialized view template

Templates can also create materialized views. This lets users generate scheduled data products with customized parameters.

```sql theme={null}
CREATE MATERIALIZED VIEW {{view_name}}
REFRESH_SCHEDULE = '@daily'
DISPLAY_NAME = {{display_name}}
AS
SELECT
    user_id,
    COUNT(1) as event_count,
    MAX(event_timestamp) as last_activity
FROM events
WHERE
    event_type = {{event_type}}
    AND event_timestamp > CURRENT_TIMESTAMP - INTERVAL {{lookback_days}} DAY
GROUP BY user_id
```

| Placeholder     | Type   | Required | Description                             |
| --------------- | ------ | -------- | --------------------------------------- |
| `view_name`     | string | Yes      | Internal name for the materialized view |
| `display_name`  | string | Yes      | Human-readable name shown in the UI     |
| `event_type`    | string | Yes      | Type of event to aggregate              |
| `lookback_days` | number | Yes      | Number of days of history to include    |

***

## Related content

<CardGroup>
  <Card title="Query Templates" icon="lightbulb" href="/concepts/nql/query-templates">
    Understand when and why to use templates
  </Card>

  <Card title="Query Template Syntax" icon="code" href="/nql/commands/query-templates">
    Complete reference for placeholder syntax
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Pre-compute and cache query results
  </Card>

  <Card title="NQL Syntax" icon="terminal" href="/nql/general/syntax">
    Core NQL syntax reference
  </Card>
</CardGroup>


# Configuring dataset statistics
Source: https://docs.narrative.io/guides/platform/configuring-dataset-statistics

Choose which column-level metrics to compute, set refresh schedules, and add per-field overrides from the dataset details page

Dataset statistics configuration lets you control which column-level metrics are computed for a dataset, how often they refresh, and whether specific fields get different treatment than the defaults. You configure all of this from a side panel on the dataset details page.

<Info>
  This guide covers the UI-based configuration. You can also manage statistics configuration programmatically through the [API](/reference/architecture/dataset-statistics#api-endpoints) or the [TypeScript SDK](/guides/sdk/managing-datasets#getting-dataset-statistics).
</Info>

## Prerequisites

* A dataset you own or have write access to
* The dataset must be on an active [data plane](/concepts/primitives/data-planes)

## Opening the configuration panel

<Steps>
  <Step title="Navigate to the dataset">
    Go to **My Data > Datasets** and select the dataset you want to configure.
  </Step>

  <Step title="Open the configuration drawer">
    You can open the statistics configuration drawer from two places on the dataset details page:

    * **Overview tab** — click the pencil icon on the **Extended Stats** row.
    * **Actions menu** — click **Actions**, then under the **Stats** group select **Configure statistics**.

    If a configuration already exists, the drawer loads it in edit mode. Otherwise, you see the create form with sensible defaults pre-selected.
  </Step>
</Steps>

## Setting default statistics

The **Defaults** section controls which statistics are computed for all eligible fields across the dataset. Select or deselect individual statistics, or use the category-level and global toggles to select groups at once.

Available statistics are organized into five categories:

| Category         | Statistics                                     |
| ---------------- | ---------------------------------------------- |
| **Counts**       | Value count, Null value count, NaN value count |
| **Bounds**       | Lower bound, Upper bound                       |
| **Distinctness** | Approx count distinct, Count distinct          |
| **Distribution** | Histogram, Mean, Standard deviation            |
| **Quality**      | Completeness                                   |

<Warning>
  **Approx count distinct** and **Count distinct** are mutually exclusive — you can enable one or the other, but not both. Approximate counting uses HyperLogLog and is significantly faster on large datasets.
</Warning>

Not all statistics apply to all data types. For example, Mean and Standard deviation only apply to numeric columns (`double` and `long`). The UI automatically filters out incompatible statistics when you configure per-field overrides. See the [type compatibility matrix](/reference/architecture/dataset-statistics#type-compatibility-matrix) for the full mapping.

### Histogram options

When you enable **Histogram**, additional options appear:

| Option       | Description                                                         | Default |
| ------------ | ------------------------------------------------------------------- | ------- |
| **Max bins** | Maximum number of histogram buckets (2–100,000)                     | 50      |
| **Overflow** | How to handle values exceeding the bin count — `none` or `truncate` | none    |

## Setting a refresh schedule

The **Refresh Trigger** section controls when statistics are recomputed. Choose one of three options:

| Trigger       | Behavior                                                                                   |
| ------------- | ------------------------------------------------------------------------------------------ |
| **Manual**    | Statistics are only recomputed when you manually trigger an update from the actions menu   |
| **On update** | Statistics are automatically recomputed whenever new data is ingested into the dataset     |
| **Cron**      | Statistics are recomputed on a UTC cron schedule (e.g., `0 0 * * *` for daily at midnight) |

<Tip>
  For most datasets, a daily cron schedule (`0 0 * * *`) provides a good balance between freshness and compute cost. Use **On update** only for datasets where you need statistics to reflect every data change immediately.
</Tip>

## Adding field overrides

By default, all eligible fields inherit the statistics you set in the Defaults section. You can override this at two levels:

### Namespace scope overrides

Expand the **Dataset Field Overrides** or **Rosetta Stone Overrides** section and enable the scope override toggle. This lets you set different default statistics for all fields within that namespace, overriding the global defaults.

### Per-field overrides

Within each namespace section, click **Add Field Override** to configure statistics for a specific column or attribute. Each override row lets you:

1. Select a field from the dropdown
2. Choose which statistics to compute for that field
3. Optionally configure histogram options specific to that field

<Note>
  For non-primitive fields (objects and arrays), only **Null value count** is available as a self-statistic. Statistics on nested child fields within objects and arrays are not yet configurable from the UI.
</Note>

The inheritance model means a per-field override completely replaces the stat set from the parent level. If you want a field to compute only `value_count` and `histogram`, select just those two — it won't also inherit the defaults. For details on how inheritance works, see the [inheritance model](/reference/architecture/dataset-statistics#inheritance-model) in the reference.

## Saving and deleting

* Click **Save** to create or update the configuration. A confirmation toast appears on success.
* In edit mode, click **Delete Configuration** to remove the configuration entirely. This stops automatic statistics computation for the dataset.

## Manually triggering a statistics update

Separately from the configuration, you can trigger an immediate one-off recomputation. From the dataset actions menu under the **Stats** group, click **Update statistics**. This runs the `datasets_calculate_column_stats` [job](/reference/architecture/job-types) regardless of the refresh trigger setting.

## Related content

<CardGroup>
  <Card title="Dataset Statistics" icon="lightbulb" href="/concepts/primitives/dataset-statistics">
    Why statistics matter and where they appear in the platform
  </Card>

  <Card title="Dataset Statistics Reference" icon="book" href="/reference/architecture/dataset-statistics">
    Full API reference, configuration schema, and validation errors
  </Card>

  <Card title="Managing Datasets (SDK)" icon="code" href="/guides/sdk/managing-datasets">
    Manage datasets programmatically including reading statistics
  </Card>

  <Card title="Job Types" icon="list-check" href="/reference/architecture/job-types">
    Job types including statistics computation
  </Card>
</CardGroup>


# Configuring dataset retention policies
Source: https://docs.narrative.io/guides/platform/configuring-retention-policies

Add, edit, and manage retention policies from the dataset details page to automate data lifecycle cleanup

Retention policies automate data lifecycle management by removing data that has exceeded a specified age. You can configure retention policies directly from the dataset details page using the **Retention** tab.

<Info>
  This guide covers the UI-based configuration. You can also manage retention policies programmatically through the [Upsert Retention Policies API](/api-reference/datasets/upsert-retention-policies-for-a-dataset) or the [TypeScript SDK](/guides/sdk/managing-datasets).
</Info>

## Prerequisites

* A dataset you own or have write access to
* The dataset must be on an active [data plane](/concepts/primitives/data-planes)

## Understanding policy classes

Before configuring a policy, choose the right policy class for your use case. The available classes depend on your data plane:

| Policy class     | What it deletes                                            | Available on                  |
| ---------------- | ---------------------------------------------------------- | ----------------------------- |
| **Row TTL**      | Individual rows whose event timestamp exceeds the interval | AWS and Snowflake data planes |
| **Snapshot TTL** | Old ingestion snapshots based on snapshot age              | AWS data planes               |
| **Table TTL**    | The entire dataset when table age exceeds the interval     | Both                          |

For a detailed explanation of each policy class, see [Dataset retention policies](/concepts/primitives/dataset-retention-policies).

<Warning>
  Deletions on Snowflake-based data planes are immediate with no grace period. AWS-based data planes apply a default 30-day grace period before permanent deletion.
</Warning>

## Adding a retention policy

<Steps>
  <Step title="Navigate to the dataset">
    Go to **My Data > Datasets** and select the dataset you want to configure.
  </Step>

  <Step title="Open the new-policy drawer">
    You can start a new policy from several places on the dataset details page:

    * **Retention tab** — click the **New Policy +** button. If no policies are configured, the tab shows an empty state that prompts you to add your first policy.
    * **Overview tab** — click the pencil icon on the **Data Expiration** row to open the new-policy drawer directly.
    * **Actions menu** — click **Actions**, then under the **Retention** group select **Configure retention policy**.
  </Step>

  <Step title="Start a new policy">
    From the Retention tab, click the **New Policy +** button. You can either:

    * Click the button directly to open the manual configuration drawer
    * Click the dropdown arrow to choose between **Ask Rosetta** (AI-assisted) or **Configure manually**

    The Overview pencil and actions-menu shortcuts open the manual configuration drawer directly.
  </Step>

  <Step title="Configure the policy">
    In the configuration drawer, set the following:

    **Policy class** — Select the type of retention policy. The available options depend on your data plane.

    **Interval** — Choose a preset duration (30, 90, 180, or 365 days) or enter a custom [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) such as `P60D` for 60 days.

    **Class-specific settings:**

    <Tabs>
      <Tab title="Row TTL">
        **Event time clock** — Select the [Rosetta Stone](/concepts/rosetta-stone/overview) attribute that maps to a timestamp column in your dataset. The dropdown shows all `timestamptz` attributes mapped on the dataset. If you don't specify one, the system defaults to `nio_last_modified`.
      </Tab>

      <Tab title="Snapshot TTL">
        **Retention value** — Choose how snapshots are handled:

        * **Time-based** — Delete snapshots older than the interval
        * **Retain everything** — Keep all snapshots indefinitely
        * **Expire everything** — Remove all snapshots on each evaluation
      </Tab>

      <Tab title="Table TTL">
        **Clock type** — Choose how table age is measured:

        * **`created_at`** — Uses when the dataset was created
        * **`max_event_time`** — Uses the most recent event timestamp (requires a column reference)
        * **`static_time`** — Uses a fixed timestamp (requires a column reference)
      </Tab>
    </Tabs>

    **Status** — Toggle the policy on or off. Policies are enabled by default.
  </Step>

  <Step title="Save the policy">
    Click **Add Policy** to save. The policy appears in the retention policies table.
  </Step>
</Steps>

## Using Ask Rosetta to generate a policy

Instead of configuring a policy manually, you can describe what you want in plain language and let Rosetta generate the configuration for you.

<Steps>
  <Step title="Open Ask Rosetta">
    From the **Retention** tab, click the dropdown arrow on the **New Policy +** button and select **Ask Rosetta**.
  </Step>

  <Step title="Describe your policy">
    Enter a plain-language description of the retention behavior you want. For example:

    > Delete individual rows older than 90 days based on when the event occurred. Use the event\_timestamp attribute as the clock. Keep the policy enabled.

    You can also click **Use this prompt** to start with the sample prompt provided.
  </Step>

  <Step title="Generate and review">
    Click **Generate Policy**. Rosetta converts your description into a structured policy configuration and opens the configuration drawer with the values pre-filled. Review the settings and adjust if needed.
  </Step>

  <Step title="Save the policy">
    Click **Add Policy** to save the generated configuration.
  </Step>
</Steps>

<Tip>
  If you prefer to configure manually instead, click the **Manual** button in the Ask Rosetta dialog to switch to the manual configuration drawer.
</Tip>

## Editing a retention policy

To modify an existing policy, click the actions menu (three dots) on the policy row and select **Edit**. The configuration drawer opens in edit mode with the current settings pre-populated. You cannot change the policy class when editing — to switch policy classes, delete the existing policy and create a new one.

## Enabling or disabling a policy

You can toggle a policy on or off without deleting it:

* **From the table** — Click the toggle switch next to the policy class badge
* **From the actions menu** — Click the actions menu and select **Enable** or **Disable**

Disabled policies remain in the configuration but are not evaluated during retention runs.

## Deleting a retention policy

Click the actions menu on the policy row and select **Delete**. Confirm the deletion in the dialog that appears. This removes the policy configuration entirely — it does not undo any deletions that have already occurred.

## Related content

<CardGroup>
  <Card title="Dataset retention policies" icon="clock-rotate-left" href="/concepts/primitives/dataset-retention-policies">
    Understand how retention policies work and when to use each policy class
  </Card>

  <Card title="Data planes" icon="database" href="/concepts/primitives/data-planes">
    Learn where your data physically resides and how it affects retention behavior
  </Card>

  <Card title="EXPIRE clause" icon="list-check" href="/nql/commands/create-materialized-view#expire">
    Set retention on materialized views using NQL
  </Card>

  <Card title="Upsert Retention Policies API" icon="code" href="/api-reference/datasets/upsert-retention-policies-for-a-dataset">
    Manage retention policies programmatically
  </Card>
</CardGroup>


# Using the Context Selector
Source: https://docs.narrative.io/guides/platform/using-the-context-selector

Choose your data plane, compute pool, database, and schema to control where queries execute and what data is visible

The context selector is the pill-shaped button in the platform's top navigation that shows your current [execution context](/concepts/primitives/execution-context). Use it to switch between data planes, choose compute pools, and set database and schema scope.

<iframe title="Using the context selector" />

<Info>
  The context selector replaces the previous "Data Plane" dropdown. It now includes compute pool, database, and schema settings alongside data plane selection.
</Info>

## Prerequisites

* A Narrative account with access to at least one [data plane](/concepts/primitives/data-planes)

## Reading the current context

The context selector displays your current settings as a segmented pill button in the top navigation:

| Segment                      | What it shows                                  | Example       |
| ---------------------------- | ---------------------------------------------- | ------------- |
| **Data plane** (icon + name) | The active data plane where operations execute | Narrative DCP |
| **Compute pool**             | The compute resources used for query execution | Shared        |
| **Database**                 | The active database within the data plane      | DEFAULT       |
| **Schema**                   | The active schema within the database          | DEFAULT       |

## Changing your execution context

<Steps>
  <Step title="Open the context selector">
    Click the pill button in the top navigation to open the context flyout. The flyout displays four columns—one for each dimension of your execution context.
  </Step>

  <Step title="Select a data plane">
    Click a data plane in the first column. This resets the downstream columns (compute pool, database, and schema) since available options depend on the selected data plane.
  </Step>

  <Step title="Select a compute pool">
    Click a compute pool in the second column. The available options depend on your data plane's provider—see [Compute Pools](/concepts/primitives/compute-pools) for details on each type.
  </Step>

  <Step title="Select database and schema">
    Select a database in the third column, then a schema in the fourth column. Selecting the schema confirms your choices and closes the flyout.
  </Step>
</Steps>

## Shortcuts

When downstream columns have only a single option, you can **double-click** a selection to auto-commit the remaining choices and close the flyout immediately. This is common when a data plane has only one compute pool and default database/schema.

## Canceling

Click anywhere outside the flyout to discard your changes. Your previous execution context remains active.

## Common scenarios

### Single data plane user

If your account has access to only one data plane, the context selector auto-selects it for you. You may never need to interact with the context selector—your context is set automatically with sensible defaults.

### Switching compute pools for a production query

If you've been running exploratory queries on a shared compute pool, switch to a dedicated pool before running production workloads:

1. Click the context selector
2. Keep your current data plane selected
3. Select **Dedicated** in the compute pool column
4. Confirm database and schema selections

Your subsequent queries will use dedicated compute resources until you switch back.

### Selecting a Snowflake warehouse for a query

If your Snowflake data plane has multiple registered warehouses (compute pools), you can choose which one to use:

1. Click the context selector
2. Keep your Snowflake data plane selected
3. Select the warehouse you want in the compute pool column
4. Confirm database and schema selections

This is useful when you have warehouses sized for different workloads—for example, a small warehouse for quick lookups and a larger one for heavy data processing.

### Working across multiple data planes

If your organization has multiple data planes (for example, a Narrative-hosted plane and a customer Snowflake instance), use the context selector to switch between them. Remember that datasets, queries, and other objects are scoped to the selected data plane—switching data planes changes what's visible across the platform.

<Note>
  The full context selector is available on desktop. On mobile, a simplified data plane dropdown appears in the navigation menu.
</Note>

## Related content

<CardGroup>
  <Card title="Execution Context" icon="crosshairs" href="/concepts/primitives/execution-context">
    Understand how the four context dimensions work together
  </Card>

  <Card title="Compute Pools" icon="microchip" href="/concepts/primitives/compute-pools">
    Learn about dedicated, shared, and default compute pools
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where your data lives and is processed
  </Card>
</CardGroup>


# Accepting AI Mapping Suggestions
Source: https://docs.narrative.io/guides/rosetta-stone/accepting-ai-suggestions

Review, accept, and reject AI-generated mapping recommendations

This guide covers how to generate AI mapping suggestions and decide which to accept. AI suggestions help you expand your normalizations by proposing mappings for columns that aren't yet mapped to Rosetta Stone attributes.

<Note>
  Before generating suggestions, your dataset should be in the Normalized Datasets interface. If you don't see your dataset there, you may need to create at least one mapping first. See [Mapping Schemas](/guides/rosetta-stone/mapping-schemas).
</Note>

## Why use AI suggestions

Creating mappings manually requires understanding both your source data and the target attribute schema. AI suggestions accelerate this process by:

**Analyzing column content**: The AI examines actual data values, not just column names, to propose appropriate attributes.

**Generating transformations**: Suggestions include complete transformation expressions that handle format conversion.

**Providing confidence ratings**: Each suggestion includes a confidence score so you can prioritize your review.

**Showing reasoning**: The AI explains why it proposed each mapping, helping you evaluate its accuracy.

## Generating suggestions

### Using the UI

1. Navigate to **Rosetta Stone** → **Normalized Datasets**
2. Click on a dataset to open its detail page
3. Switch to the **Normalize** tab
4. Click **Start Analysis**

The AI analyzes unmapped columns in your dataset. This process:

* Samples data from each column
* Compares patterns against known attribute types
* Generates transformation expressions
* Assigns confidence scores

When complete, suggestions appear grouped by confidence level.

## Understanding suggestion cards

Each suggestion card displays:

| Element              | Description                                      |
| -------------------- | ------------------------------------------------ |
| Source column        | The column in your dataset                       |
| Target attribute     | The Rosetta Stone attribute the AI recommends    |
| Confidence score     | 0-100% indicating how certain the AI is          |
| Transformation       | The expression that will convert source values   |
| Reasoning            | Why the AI proposed this mapping                 |
| Before/after samples | Example values showing the transformation effect |

### Confidence groupings

Suggestions are organized into collapsible sections by confidence:

**High confidence (80-100%)**: Strong matches where column content clearly aligns with the attribute. These are generally safe to accept.

**Medium confidence (50-79%)**: Reasonable matches with some uncertainty. Review the reasoning and samples before accepting.

**Low confidence (0-49%)**: Uncertain matches that require careful evaluation. The AI found some alignment but isn't confident.

## Reviewing before/after samples

Before accepting any suggestion, examine the sample transformations:

### What to check

**Value accuracy**: Do the "after" values look correct? Is the transformation producing expected output?

**Edge case handling**: Look for unusual values in the samples. Does the transformation handle them appropriately?

**Null behavior**: If samples include null or empty values, verify they're handled as you expect.

**Format consistency**: Are all output values in the correct format for the attribute?

### Example review

A suggestion to map `gender_code` to `hl7_gender`:

| Before | After       |
| ------ | ----------- |
| `"1"`  | `"male"`    |
| `"2"`  | `"female"`  |
| `"0"`  | `"unknown"` |
| `"3"`  | `"other"`   |
| `null` | `null`      |

Review questions:

* Are the numeric codes mapped correctly? (Verify with your data documentation)
* Is `"0"` → `"unknown"` the right interpretation?
* Should `null` remain `null` or become `"unknown"`?

If the mapping looks correct, accept it. If something seems off, reject it and create a manual mapping instead.

## Accepting suggestions

### Individual acceptance

1. Review the suggestion card
2. Examine the before/after samples
3. Click **Accept**

The suggestion becomes an active mapping immediately. You can verify by switching to the **Evaluate** tab or checking the mapping in your dataset's Rosetta Stone configuration.

### Bulk accepting high-confidence suggestions

When you have many high-confidence suggestions:

1. Review a sample of the high-confidence suggestions to verify quality
2. Click **Accept All High Confidence**
3. Confirm the bulk action

This creates mappings for all suggestions scoring 80% or above. Use this when you've verified that high-confidence suggestions are consistently accurate for your dataset.

<Warning>
  Bulk accept creates multiple mappings at once. While high-confidence suggestions are generally reliable, review at least a few before bulk accepting to ensure the AI understands your data correctly.
</Warning>

## Rejecting suggestions

Reject suggestions that don't fit your use case:

1. Click **Reject** on the suggestion card
2. The suggestion is dismissed

Rejected suggestions won't reappear in future analysis runs for the same column.

### Reasons to reject

**Incorrect attribute**: The AI suggested mapping to the wrong attribute. For example, a phone number column suggested for an email attribute.

**Wrong transformation**: The mapping logic doesn't match your data's semantics. For example, gender codes that use different values than the AI assumed.

**Intentionally unmapped**: You don't want this column normalized, perhaps because it contains internal identifiers or data you'll handle differently.

**Duplicate coverage**: Another mapping already covers this data adequately.

## When AI suggestions don't fit

Sometimes none of the AI suggestions work for a column. This happens when:

* Your data uses non-standard formats or encodings
* Column names don't reflect content
* The data requires complex business logic to map correctly
* No standard attribute matches your data's semantics

### Creating manual mappings instead

When AI suggestions miss the mark:

1. Reject the suggestion (if one was proposed)
2. Navigate to your dataset's **Rosetta Stone** tab
3. Click **Add Mapping**
4. Select the source column and target attribute
5. Write your transformation expression
6. Test and activate the mapping

See [Mapping Schemas](/guides/rosetta-stone/mapping-schemas) for detailed guidance on manual mapping creation.

### Requesting custom attributes

If no existing attribute fits your data:

1. Check the [attribute catalog](/reference/rosetta-stone/attribute-types) to confirm no suitable attribute exists
2. Contact Narrative support to discuss creating a custom attribute
3. Once the attribute exists, create your mapping manually

### Improving future suggestions

AI suggestions improve over time. To help:

* Use descriptive column names when possible
* Ensure data quality in source columns
* Accept accurate suggestions—this trains the system on your data patterns

## Validating accepted suggestions

After accepting suggestions, verify they work correctly:

### Run an evaluation

1. Switch to the **Evaluate** tab
2. Click **Evaluate Mappings**
3. Check that newly created mappings score well

New mappings from accepted suggestions should typically score high. If they score medium or low, review the evaluation feedback.

### Query the normalized data

Test that your mappings produce expected results:

```sql theme={null}
SELECT
    raw_email,
    hl7_gender,
    COUNT(1) as count
FROM narrative.rosetta_stone
WHERE _nio_source_dataset_id = 'your-dataset-id'
GROUP BY raw_email, hl7_gender
LIMIT 100
```

Spot-check that values look correct.

### Monitor over time

As new data flows through your mappings, periodically:

* Re-run evaluations to catch emerging issues
* Check for unexpected null rates
* Verify distribution of enum values

## Global warnings

The AI analysis may produce dataset-level warnings that appear at the top of the suggestions list:

| Warning              | Meaning                                          |
| -------------------- | ------------------------------------------------ |
| High null rate       | Many columns have significant null values        |
| Inconsistent formats | Same column contains multiple data formats       |
| Low coverage         | Few columns could be mapped to known attributes  |
| Duplicate columns    | Multiple columns appear to contain the same data |

Address these warnings to improve overall data quality, independent of individual mapping decisions.

## Best practices

**Review before accepting**: Even high-confidence suggestions deserve a quick review of the samples.

**Accept incrementally**: Accept a few suggestions, validate them, then continue. This catches issues early.

**Document rejections**: When rejecting suggestions, note why so future team members understand your decisions.

**Re-run after changes**: If you modify source data or add columns, generate new suggestions to capture the changes.

**Combine with evaluation**: After accepting suggestions, run an evaluation to verify quality holistically.

## Related content

<CardGroup>
  <Card title="Managing Evaluations" icon="gauge" href="/guides/rosetta-stone/managing-evaluations">
    Assess mapping quality with AI scoring
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Create mappings manually
  </Card>

  <Card title="Confidence Scoring" icon="gauge-high" href="/concepts/rosetta-stone/confidence-scoring">
    Understand how confidence scores work
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Test mappings before production use
  </Card>
</CardGroup>


# Creating Normalized Attributes
Source: https://docs.narrative.io/guides/rosetta-stone/creating-normalized-attributes

Best practices for designing and proposing Rosetta Stone attributes

This guide covers how to design well-formed, reusable normalized attributes for Rosetta Stone. A good attribute is unambiguous, well-documented, and mappable from real-world data sources.

<Note>
  You can create custom attributes directly from the UI using the [Attributes interface](/reference/rosetta-stone/attributes-ui). Rosetta Stone can auto-generate the display name, name, and description for you. For the complete attribute type system, see [Attribute Types](/reference/rosetta-stone/attribute-types) and [The Normalization Model](/concepts/rosetta-stone/normalization-model).
</Note>

## Before you begin

Answer these three questions before proposing a new attribute:

1. **Does this attribute already exist?** Search by aliases, descriptions, and examples---not just names. An attribute called `purchase_channel` might already cover what you plan to name `buy_source`.
2. **Is the meaning unambiguous?** If two reasonable interpretations exist, you need two attributes, not one fuzzy compromise.
3. **Do you have real demand?** At least two concrete use cases from different parties should need this attribute.

If you cannot answer yes to all three, stop and reconsider.

## Naming your attribute

The `name` field uses the pattern `<attribute>[.<qualifier>]`. Names should be specific enough to understand without additional context.

| Good                  | Bad            | Why bad                                  |
| --------------------- | -------------- | ---------------------------------------- |
| `email`               | `address`      | Ambiguous---postal or email?             |
| `email.hashed_sha256` | `hashed_value` | Hashed what?                             |
| `total_amount.gross`  | `total`        | Total of what?                           |
| `purchase_channel`    | `channel`      | Marketing channel? Distribution channel? |
| `birth_year`          | `year`         | Year of what event?                      |

The `display_name` field is the human-readable label shown in UIs. It should be concise but clear---for example, `"Email (SHA-256 Hash)"` for the attribute named `email.hashed_sha256`.

## Writing the description

The description is the most important piece of documentation for an attribute. A good description eliminates ambiguity for anyone creating mappings.

**Bad:**

> Email address

**Good:**

> The electronic mail address associated with an individual, formatted per RFC 5321, representing a point of contact for that person. Does not include addresses associated with businesses, roles, or distribution lists.

Your description should answer:

* **What does this attribute represent?** Define the concept precisely.
* **What does it explicitly NOT represent?** Call out common confusions. If `email` does not include business or role addresses, say so.
* **What does null mean?** Distinguish between unknown (data was not collected), not applicable (concept does not apply to this record), and intentionally blank (value was collected but is empty).

### Join key designation

Set `is_join_key` to `true` if the attribute is suitable for identity resolution or record linkage. Only primitive types can be join keys.

```json theme={null}
{
  "id": 101,
  "name": "email.hashed_sha256",
  "type": "string",
  "is_join_key": true,
  "description": "SHA-256 hash of an individual's email address, used for identity resolution. Lowercase and trimmed before hashing.",
  "validations": ["LENGTH($this) = 64"]
}
```

## Choosing a type

Select the type that most accurately represents the attribute's semantics. Do not default to `string` when a more specific type exists.

| Type          | Use when                                        | Examples                                      |
| ------------- | ----------------------------------------------- | --------------------------------------------- |
| `string`      | Text values, identifiers, codes                 | Names, hashed emails, country codes           |
| `boolean`     | True/false flags                                | Opt-in status, active/inactive                |
| `double`      | Decimal numbers, measurements, currency amounts | Price, latitude, temperature                  |
| `long`        | Integers, counts, whole-number IDs              | Age, row counts, sequence numbers             |
| `timestamptz` | Points in time (always with timezone)           | Event timestamps, signup dates                |
| `array`       | Repeated values of a single type                | Interest categories, identifier lists         |
| `object`      | Nested structure with named properties          | Addresses, coordinates, composite identifiers |

For primitive types, also consider:

* **`enum`**: Use when the attribute has a closed set of allowed values. Add an `enum` array to a `string` type.
* **`validations`**: Use to enforce the canonical format (length constraints, range bounds, patterns).

For arrays, define the element type in `items`. For objects, define the field structure in `properties` and list mandatory fields in `required`.

See the [Attribute Types Reference](/reference/rosetta-stone/attribute-types) for full type definitions and examples.

## Enum governance

When your attribute uses an `enum`, define each value with clear semantics. Every enum value should be a deliberate, documented choice---not an afterthought.

### Example: purchase channel

| Value        | Meaning                                              | Common source mappings                     |
| ------------ | ---------------------------------------------------- | ------------------------------------------ |
| `in_store`   | Purchase at a physical retail location               | "retail", "brick\_and\_mortar", "pos", "1" |
| `online_web` | Purchase via a web browser                           | "web", "ecommerce", "website", "2"         |
| `online_app` | Purchase via a native mobile app                     | "app", "mobile\_app", "mobile", "3"        |
| `other`      | Purchase through a channel not listed above          | Any unrecognized value                     |
| `unknown`    | Purchase channel was not collected or is unavailable | NULL, empty string, "N/A"                  |

### Enum rules

* **Case-sensitive and lowercase.** All enum values use `snake_case`.
* **Always include fallback values.** Use `other` when the source value is known but does not match any defined option. Use `unknown` when the source value is missing or indeterminate.
* **Document when to use `other` vs. `unknown`.** `other` means "we know the value but it doesn't fit our categories." `unknown` means "we don't have the information."

```json theme={null}
{
  "id": 250,
  "name": "purchase_channel",
  "type": "string",
  "enum": ["in_store", "online_web", "online_app", "other", "unknown"],
  "description": "The sales channel through which a purchase was completed. Does not describe marketing attribution or traffic source."
}
```

## Using references

Use `$ref` with a numeric attribute ID when an object property should conform to an existing attribute's definition. References avoid duplicating validation rules and semantics, and changes to the referenced attribute propagate automatically.

```json theme={null}
{
  "id": 600,
  "name": "purchase_event",
  "type": "object",
  "properties": {
    "timestamp": {
      "$ref": 1007
    },
    "amount": {
      "type": "double"
    },
    "channel": {
      "$ref": 250
    }
  },
  "required": ["timestamp", "amount"]
}
```

In this example, `timestamp` inherits the full definition of the `event_timestamp` attribute (ID 1007) and `channel` inherits from `purchase_channel` (ID 250), including their types, validations, and enum constraints.

See [Reference Type](/reference/rosetta-stone/attribute-types#reference-type) for full details.

## Setting collaborators

The `collaborators` property controls which companies can view or map to your attribute. Use it for organization-specific attributes shared with selected partners.

```json theme={null}
{
  "id": 700,
  "name": "loyalty_tier",
  "type": "string",
  "enum": ["bronze", "silver", "gold", "platinum"],
  "description": "Customer loyalty program tier level.",
  "collaborators": [
    { "company_id": 42, "access": "read" },
    { "company_id": 87, "access": "map" }
  ]
}
```

When `collaborators` is omitted, the attribute follows default visibility rules for your organization.

## Documenting beyond the schema

The schema defines structure. Documentation captures intent. Include the following alongside your attribute definition to help others create accurate mappings.

### Good examples

Provide 3--5 real values with context explaining why they belong:

* `"user@example.com"` --- a personal email address (valid)
* `"jane.doe+newsletter@gmail.com"` --- plus-addressed email (valid, the plus suffix is part of the address)
* `"info@company.org"` --- a role address (invalid for this attribute; use a business email attribute instead)

### Anti-examples

Values that do NOT belong and where they should go instead:

* `"not-an-email"` --- fails format validation, should be null
* `"support@company.com"` --- role address, belongs in `business_email` attribute
* `""` (empty string) --- should be null, not empty

### Mapping guidance

Document common source representations and the expected transformations:

* **Email**: Lowercase, trim whitespace, validate against pattern `$this LIKE '%@%.%'`
* **Phone**: Convert to E.164 format (e.g., `+15551234567`), strip parentheses, dashes, and spaces
* **Currency**: Decimal with two-digit precision, strip currency symbols and thousands separators

### Canonical representation

For ambiguous types, specify the expected serialization:

* **Timestamps**: ISO 8601 in UTC (`2024-01-15T14:30:00Z`)
* **Decimals**: Precision expectations (e.g., currency amounts use two decimal places)
* **Booleans**: `true` / `false` (not `1`/`0`, not `"yes"`/`"no"`)

## Validation checklist

Before submitting a new attribute, verify:

* `name` is unambiguous without additional context
* `description` explains what it means AND what it does not mean
* `type` is appropriate (not just defaulting to `string`)
* `enum` values are defined with clear semantics (if applicable)
* `validations` enforce the canonical format (if applicable)
* `is_join_key` is set correctly for identity attributes
* The attribute can be mapped from at least 2--3 real datasets
* No existing attribute already covers this concept

## Recommended governance workflow

The following workflow is a recommended team process for introducing new attributes. It is not enforced by the platform.

<Steps>
  <Step title="Proposal">
    A champion creates a specification using this guide. The spec includes the full attribute definition, description, examples, anti-examples, and mapping guidance.
  </Step>

  <Step title="Triage">
    Review the proposal for duplicates, ambiguity, and overly company-specific scope. If the attribute only makes sense for one organization, it may belong as an organization attribute rather than a shared one.
  </Step>

  <Step title="Draft">
    Create the attribute with limited `collaborators` access. This restricts visibility while the attribute is being validated.
  </Step>

  <Step title="Validation">
    Prove mapping feasibility by mapping the attribute across at least two real datasets. Confirm that the type, enum values, and validations work with actual source data.
  </Step>

  <Step title="Acceptance">
    After production use by multiple parties, expand access. Update the description and mapping guidance based on what you learned during validation.
  </Step>
</Steps>

## Design principles

* **Names are presentation, not identity.** The `id` is immutable; `name` and `display_name` can evolve.
* **Separate semantics from syntax.** `description` captures meaning; `type`, `enum`, and `validations` capture structure.
* **Prefer two specific attributes over one vague one.** Ambiguity is a bug, not a feature.
* **Every attribute is maintenance burden.** Only create what has real, demonstrated demand.
* **Use references to avoid duplication.** If a property should match an existing attribute, use `$ref`.

## Related content

<CardGroup>
  <Card title="Attributes Interface" icon="tags" href="/reference/rosetta-stone/attributes-ui">
    Browse and create attributes in the UI
  </Card>

  <Card title="Attribute Types Reference" icon="shapes" href="/reference/rosetta-stone/attribute-types">
    Complete type system documentation
  </Card>

  <Card title="The Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    How Rosetta Stone structures types and validations
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Map your dataset columns to attributes
  </Card>
</CardGroup>


# Handling Edge Cases
Source: https://docs.narrative.io/guides/rosetta-stone/edge-cases

Solutions for complex mapping scenarios and unusual data patterns

This guide covers solutions for complex mapping scenarios that go beyond simple column-to-attribute translations. Use these patterns when your data doesn't fit neatly into the standard mapping model.

<Note>
  Familiarize yourself with [basic mapping](/guides/rosetta-stone/mapping-schemas) before tackling these advanced scenarios.
</Note>

## Multi-value fields

When a single source column contains multiple values (often comma-separated or delimited).

### Scenario

Your source data has an `interests` column with values like:

* `"sports,music,travel"`
* `"cooking"`
* `"tech,gaming,music,movies"`

### Solution: Split to array

Map to an array attribute using `SPLIT`:

```sql theme={null}
SPLIT(interests, ',')
```

This transforms `"sports,music,travel"` into `["sports", "music", "travel"]`.

### Handling whitespace

If values might have spaces around delimiters:

```sql theme={null}
TRANSFORM(SPLIT(interests, ','), x -> TRIM(x))
```

### Handling empty values

Filter out empty strings that result from trailing delimiters:

```sql theme={null}
FILTER(SPLIT(interests, ','), x -> LENGTH(TRIM(x)) > 0)
```

## Conditional mappings

When different source values require completely different transformation logic.

### Scenario: Different ID types in one column

Your `identifier` column contains mixed ID types:

* `"email:user@example.com"`
* `"phone:5551234567"`
* `"cookie:abc123xyz"`

### Solution: Parse and route

```sql theme={null}
STRUCT(
  CASE
    WHEN identifier LIKE 'email:%' THEN 'email'
    WHEN identifier LIKE 'phone:%' THEN 'phone'
    WHEN identifier LIKE 'cookie:%' THEN 'cookie_id'
    ELSE 'unknown'
  END AS type,
  SUBSTRING(identifier, POSITION(':' IN identifier) + 1) AS value,
  CASE
    WHEN identifier LIKE 'email:%' THEN 'hashed_email'
    WHEN identifier LIKE 'phone:%' THEN 'hashed_phone'
    WHEN identifier LIKE 'cookie:%' THEN 'advertising'
    ELSE 'other'
  END AS context
)
```

### Scenario: Different date formats in one column

Some rows use `MM/DD/YYYY`, others use `YYYY-MM-DD`:

```sql theme={null}
CASE
  WHEN date_col LIKE '____-__-__' THEN
    TO_TIMESTAMP(date_col, 'YYYY-MM-DD')
  WHEN date_col LIKE '__/__/____' THEN
    TO_TIMESTAMP(date_col, 'MM/DD/YYYY')
  ELSE
    NULL
END
```

## Missing or null data

Handling columns that are frequently null or contain placeholder values.

### Default values

Provide a fallback when data is missing:

```sql theme={null}
COALESCE(gender, 'unknown')
```

### Coalescing from multiple columns

Try multiple columns in order:

```sql theme={null}
COALESCE(primary_email, secondary_email, backup_email)
```

### Treating placeholders as null

Your data uses `"N/A"` or `"-"` as placeholders:

```sql theme={null}
CASE
  WHEN UPPER(value) IN ('N/A', 'NA', '-', 'NONE', 'NULL', '') THEN NULL
  ELSE value
END
```

### Conditional defaults

Different defaults based on context:

```sql theme={null}
CASE
  WHEN country = 'US' AND state IS NULL THEN 'unknown_us_state'
  WHEN country != 'US' AND state IS NULL THEN NULL  -- Non-US doesn't need state
  ELSE state
END
```

## Type coercion challenges

When source data types don't cleanly match target types.

### String to numeric with formatting

Source has currency formatting: `"$1,234.56"`

```sql theme={null}
CAST(
  REPLACE(REPLACE(amount, '$', ''), ',', '')
  AS FLOAT
)
```

### Boolean interpretations

Sources represent booleans differently:

| Source values    | Transformation        |
| ---------------- | --------------------- |
| `"yes"/"no"`     | `LOWER(col) = 'yes'`  |
| `"Y"/"N"`        | `UPPER(col) = 'Y'`    |
| `"1"/"0"`        | `col = '1'`           |
| `"true"/"false"` | `LOWER(col) = 'true'` |
| `1/0` (numeric)  | `col = 1`             |

Handle multiple formats:

```sql theme={null}
CASE
  WHEN UPPER(col) IN ('Y', 'YES', 'TRUE', '1') THEN true
  WHEN UPPER(col) IN ('N', 'NO', 'FALSE', '0') THEN false
  ELSE NULL
END
```

### Timestamp with multiple formats

Parse dates that might be in several formats:

```sql theme={null}
COALESCE(
  TRY_TO_TIMESTAMP(date_col, 'YYYY-MM-DD''T''HH24:MI:SS'),
  TRY_TO_TIMESTAMP(date_col, 'MM/DD/YYYY'),
  TRY_TO_TIMESTAMP(date_col, 'DD-Mon-YYYY'),
  TRY_TO_TIMESTAMP(date_col, 'YYYYMMDD')
)
```

The `TRY_TO_TIMESTAMP` function returns null instead of erroring on parse failure, allowing the `COALESCE` to try the next format.

## One-to-many relationships

When a single source column maps to multiple parts of a struct attribute.

### Scenario

Your `full_address` column contains: `"123 Main St, Boston, MA 02101"`

You need to map to a struct with separate fields for street, city, state, and postal code.

### Solution: Parse components

```sql theme={null}
STRUCT(
  TRIM(SPLIT_PART(full_address, ',', 1)) AS street,
  TRIM(SPLIT_PART(full_address, ',', 2)) AS city,
  TRIM(SPLIT_PART(
    SPLIT_PART(full_address, ',', 3), ' ', 1
  )) AS state,
  TRIM(SPLIT_PART(
    SPLIT_PART(full_address, ',', 3), ' ', 2
  )) AS postal_code
)
```

<Warning>
  Parsing unstructured text is error-prone. Consider whether the complexity is worth it, or if the source data can be improved.
</Warning>

## Many-to-one relationships

When multiple source columns combine into a single attribute.

### Scenario: Separate date and time columns

Source has `event_date` and `event_time` as separate columns:

```sql theme={null}
TO_TIMESTAMP(
  CONCAT(event_date, ' ', event_time),
  'YYYY-MM-DD HH24:MI:SS'
)
```

### Scenario: First name and last name

Combine into a full name:

```sql theme={null}
TRIM(CONCAT(COALESCE(first_name, ''), ' ', COALESCE(last_name, '')))
```

### Scenario: Building a struct from multiple columns

Create a `unique_identifier` from separate columns:

```sql theme={null}
STRUCT(
  id_type AS type,
  id_value AS value,
  COALESCE(id_context, 'default') AS context
)
```

## Multiple columns mapped to the same attribute

When multiple source columns map to the same Rosetta Stone attribute, each mapping produces a value. The NQL compiler collects these into an array and automatically expands (unnests) them into separate rows.

### Scenario

Your dataset has multiple email columns:

| user\_id | name  | email\_1                              | email\_2                          |
| -------- | ----- | ------------------------------------- | --------------------------------- |
| 1        | Alice | [foo@bar.com](mailto:foo@bar.com)     | [baz@bar.com](mailto:baz@bar.com) |
| 2        | Bob   | [no@thanks.com](mailto:no@thanks.com) | NULL                              |

Both `email_1` and `email_2` are mapped to the `raw_email` attribute.

### Query behavior

When you query:

```sql theme={null}
SELECT
  user_id,
  rosetta_stone.raw_email
FROM narrative.rosetta_stone
WHERE ...
```

The result contains one row per non-null email:

| user\_id | raw\_email                            |
| -------- | ------------------------------------- |
| 1        | [foo@bar.com](mailto:foo@bar.com)     |
| 1        | [baz@bar.com](mailto:baz@bar.com)     |
| 2        | [no@thanks.com](mailto:no@thanks.com) |

### Why this happens

The NQL compiler:

1. Evaluates each mapping to `raw_email` (both `email_1` and `email_2`)
2. Collects the results into an array: `['foo@bar.com', 'baz@bar.com']`
3. Automatically unnests the array, creating one output row per array element
4. Filters out NULL values during expansion

This behavior enables queries that need all available values for an attribute without requiring you to know the source schema.

### When to use alternatives

**Use COALESCE when you want only one value per row:**

If your source columns represent "primary email" and "backup email" with a preference order, map them using:

```sql theme={null}
COALESCE(primary_email, secondary_email)
```

This returns only the first non-null value instead of expanding to multiple rows.

### Performance considerations

Row expansion multiplies result set size. If a dataset has 5 columns mapped to the same attribute and 1 million source rows, queries could return up to 5 million rows. Consider:

* Filtering early to reduce the row count before expansion
* Using COALESCE if you only need one value per source row
* Being explicit about which specific columns you need

## Version handling

When your source data schema changes over time.

### Scenario: Column renamed in new data

Old data has `sex`, new data has `gender`:

```sql theme={null}
COALESCE(gender, sex)
```

### Scenario: New values added to enum

Old data: `"M"`, `"F"`
New data: `"M"`, `"F"`, `"NB"`, `"O"`

Update your transformation to handle all values:

```sql theme={null}
CASE UPPER(COALESCE(gender, sex))
  WHEN 'M' THEN 'male'
  WHEN 'F' THEN 'female'
  WHEN 'NB' THEN 'other'
  WHEN 'O' THEN 'other'
  ELSE 'unknown'
END
```

### Scenario: Data format changed

Old timestamps in epoch milliseconds, new data in ISO 8601:

```sql theme={null}
CASE
  WHEN timestamp_col ~ '^\d+$' THEN
    -- Numeric: assume epoch milliseconds
    TO_TIMESTAMP(CAST(timestamp_col AS BIGINT) / 1000)
  ELSE
    -- String: assume ISO 8601
    TO_TIMESTAMP(timestamp_col, 'YYYY-MM-DD''T''HH24:MI:SS')
END
```

## Handling arrays of structs

When your data contains nested structures.

### Scenario: Multiple identifiers per row

Source column `identifiers` contains JSON like:

```json theme={null}
[
  {"type": "email", "value": "user@example.com"},
  {"type": "phone", "value": "5551234567"}
]
```

### Solution: Parse JSON array

```sql theme={null}
TRANSFORM(
  JSON_PARSE(identifiers),
  x -> STRUCT(
    x.type AS type,
    x.value AS value,
    'parsed' AS context
  )
)
```

## Handling special characters

When source data contains characters that need escaping or encoding.

### URL encoding

Source has URL-encoded values:

```sql theme={null}
URL_DECODE(encoded_value)
```

### HTML entities

Source has HTML entities like `&amp;`:

```sql theme={null}
REPLACE(
  REPLACE(
    REPLACE(value, '&amp;', '&'),
    '&lt;', '<'
  ),
  '&gt;', '>'
)
```

### Unicode normalization

Ensure consistent unicode representation:

```sql theme={null}
NORMALIZE(unicode_string, 'NFC')
```

## Performance considerations

Complex transformations can impact query performance.

### Avoid expensive operations in hot paths

If a transformation is complex, consider:

* Pre-computing values in a materialized view
* Simplifying the logic where possible
* Breaking into multiple simpler mappings

### Test with realistic data volumes

Edge case handling that works in tests might be slow at scale. Test with production-like data volumes.

## Related content

<CardGroup>
  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Basic mapping creation
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Test your edge case handling
  </Card>

  <Card title="Transformation Functions" icon="function" href="/reference/rosetta-stone/transformation-functions">
    Complete function reference
  </Card>

  <Card title="The Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    Understand attribute types and validations
  </Card>
</CardGroup>


# Managing Mapping Evaluations
Source: https://docs.narrative.io/guides/rosetta-stone/managing-evaluations

Run AI evaluations, filter by confidence, and understand mapping quality

This guide covers how to evaluate your Rosetta Stone mappings using AI-powered confidence scoring. Evaluations help you identify which mappings are working well and which need attention.

<Note>
  Before running evaluations, you'll need a dataset with existing mappings. See [Mapping Schemas](/guides/rosetta-stone/mapping-schemas) to create mappings first.
</Note>

## Why evaluate mappings

Manual validation tests specific cases you define, but it can't anticipate every issue. AI evaluation analyzes your mappings holistically, looking for:

**Transformation gaps**: Cases your transformation doesn't handle, like unexpected input values or edge cases.

**Pattern mismatches**: Source data that doesn't align with the target attribute's expected format.

**Quality indicators**: Signals that suggest a mapping might produce incorrect results under certain conditions.

Evaluations complement manual validation—they surface issues you might not have thought to test for.

## Running an evaluation

### Using the UI

1. Navigate to **Rosetta Stone** → **Normalized Datasets**
2. Click on a dataset to open its detail page
3. On the **Evaluate** tab, click **Evaluate Mappings**

The system analyzes all mappings on the dataset. This process examines column names, samples actual data, and analyzes transformation logic. Depending on the number of mappings, evaluation may take a few moments.

When complete, you'll see:

* A **confidence gradient bar** showing the distribution across quality tiers
* **Confidence level cards** with counts for High, Medium, Low, and Not Scored
* **Individual mapping cards** with scores and AI feedback

## Understanding evaluation results

### The confidence gradient

The gradient bar provides a visual summary of your dataset's mapping quality:

| Color  | Score Range | Meaning                                     |
| ------ | ----------- | ------------------------------------------- |
| Green  | 80-100%     | High confidence—mappings are likely correct |
| Yellow | 50-79%      | Medium confidence—review recommended        |
| Red    | 0-49%       | Low confidence—manual review required       |
| Gray   | Not scored  | Mappings that couldn't be evaluated         |

A dataset with mostly green indicates well-configured mappings. Significant yellow or red sections suggest areas needing attention.

### Confidence level cards

Each card shows the count of mappings in that tier:

**High confidence (80-100%)**: These mappings show strong alignment between source columns and target attributes. The AI found consistent patterns and appropriate transformations. You can generally trust these, though spot-checking critical data flows is still recommended.

**Medium confidence (50-79%)**: The mapping is probably correct but has characteristics worth reviewing. Common reasons include partial transformation coverage or ambiguous column names.

**Low confidence (0-49%)**: The AI identified significant concerns. These mappings require human verification before relying on them.

**Not scored**: Mappings that couldn't be evaluated, typically system-generated mappings for internal columns.

### Individual mapping feedback

Click on any mapping card to see detailed AI feedback:

* **Score breakdown**: Factors contributing to the confidence score
* **Identified issues**: Specific concerns the AI found
* **Recommendations**: Suggested improvements
* **Sample analysis**: How the mapping performs on actual data samples

## Filtering mappings

### By confidence level

Click any confidence level card to filter the mapping list:

1. Click the **Low** card to see only low-confidence mappings
2. Click the **Medium** card to see medium-confidence mappings
3. Click **All** to clear the filter

This helps you focus on mappings that need attention without being distracted by well-functioning ones.

### By mapping type

Filter by how the mapping was created:

| Type    | Description                                          |
| ------- | ---------------------------------------------------- |
| System  | Auto-generated for internal columns (always correct) |
| AI      | Created from AI suggestions                          |
| User    | Manually created by users                            |
| Lineage | Inherited from parent datasets                       |

Click the corresponding overview card to filter. You can combine type and confidence filters—for example, view only low-confidence user mappings.

## Understanding low confidence scores

When a mapping scores low, the AI provides specific feedback. Common issues include:

### Missing transformation cases

The transformation doesn't handle all values in the source data:

```
Issue: Transformation covers 85% of values.
Unhandled values: "N/A", "Not Specified", ""
Recommendation: Add cases for missing values or map to default.
```

**Resolution**: Update the transformation to handle additional cases:

```sql theme={null}
CASE UPPER(status)
  WHEN 'ACTIVE' THEN 'active'
  WHEN 'INACTIVE' THEN 'inactive'
  WHEN 'N/A' THEN 'unknown'
  WHEN 'NOT SPECIFIED' THEN 'unknown'
  WHEN '' THEN 'unknown'
  ELSE 'unknown'
END
```

### Type mismatch concerns

The source data contains values that might not convert correctly:

```
Issue: Source column contains non-numeric values but maps to integer attribute.
Samples failing conversion: "N/A", "TBD", "-"
Recommendation: Add validation or filter invalid values.
```

**Resolution**: Add explicit handling for non-numeric values:

```sql theme={null}
CASE
  WHEN REGEXP_LIKE(value_col, '^[0-9]+$') THEN CAST(value_col AS INTEGER)
  ELSE NULL
END
```

### Ambiguous column mapping

The column name doesn't clearly indicate its content:

```
Issue: Column "field_7" mapped to "email".
Pattern analysis inconclusive—only 60% of values match email format.
Recommendation: Verify this column contains email addresses.
```

**Resolution**: Verify the mapping is correct by examining sample data. If it's wrong, update or remove the mapping.

### Edge case failures

The transformation fails for certain input patterns:

```
Issue: Date parsing fails for some formats.
Successful: "2024-01-15" (ISO format)
Failed: "01/15/2024" (US format), "15-Jan-2024" (text month)
Recommendation: Update transformation to handle multiple date formats.
```

**Resolution**: Use a more flexible date parsing approach or normalize source data before mapping.

## Refreshing stale evaluations

Evaluation results can become outdated when:

* You modify transformation expressions
* The source data changes significantly
* Significant time has passed since the last evaluation

### Recognizing stale evaluations

The UI indicates when results may be stale:

* **Last evaluated** timestamp shows when evaluation ran
* A **Stale** badge appears if mappings changed since evaluation
* Confidence scores may show a warning indicator

### Re-running evaluations

To refresh results:

1. Navigate to the dataset's **Evaluate** tab
2. Click **Evaluate Mappings** to run a new evaluation
3. The new results replace the previous evaluation

Re-evaluate after:

* Modifying any transformation expression
* Accepting or rejecting AI suggestions
* Observing unexpected query results
* Major updates to source data

## Best practices

**Evaluate regularly**: Run evaluations periodically, especially after changes to mappings or source data.

**Address low confidence first**: Focus on low-confidence mappings before medium-confidence ones.

**Don't ignore medium confidence**: While not urgent, medium-confidence mappings may have subtle issues worth investigating.

**Combine with manual validation**: Use AI evaluation to identify potential issues, then validate specific cases manually.

**Document your decisions**: When you verify a low-confidence mapping is actually correct, document why so future reviewers understand.

## Related content

<CardGroup>
  <Card title="Confidence Scoring" icon="gauge-high" href="/concepts/rosetta-stone/confidence-scoring">
    Understand how confidence scores are calculated
  </Card>

  <Card title="Accepting AI Suggestions" icon="wand-magic-sparkles" href="/guides/rosetta-stone/accepting-ai-suggestions">
    Generate and accept AI mapping recommendations
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Manual validation techniques
  </Card>

  <Card title="Normalized Datasets UI" icon="table-columns" href="/reference/rosetta-stone/normalized-datasets-ui">
    Reference for the interface elements
  </Card>
</CardGroup>


# Mapping Schemas
Source: https://docs.narrative.io/guides/rosetta-stone/mapping-schemas

Create mappings between your data schema and Rosetta Stone attributes

This guide covers how to create and manage mappings between your dataset columns and Rosetta Stone attributes. Mappings are the translation layer that enables your data to be queried through the normalized `narrative.rosetta_stone` table.

<Note>
  For background on how mappings work, see [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works).
</Note>

## Prerequisites

* A dataset uploaded to Narrative
* Understanding of your dataset's column semantics
* Familiarity with basic NQL syntax (for transformation expressions)

## Using the UI

### View suggested mappings

1. Navigate to **Datasets** and select your dataset
2. Click the **Rosetta Stone** tab
3. Review the auto-generated mapping suggestions

The UI displays your columns alongside suggested attributes. Each suggestion includes:

* The target attribute name and description
* A confidence score
* A preview of sample data after transformation

### Accept a suggested mapping

For high-confidence suggestions:

1. Review the suggested attribute and sample output
2. Click **Accept** to create the mapping

The mapping becomes active immediately.

### Modify a suggested mapping

If a suggestion is close but not quite right:

1. Click **Edit** on the suggestion
2. Modify the target attribute or transformation expression
3. Click **Save**

### Create a manual mapping

For columns without suggestions or when you need a custom mapping:

1. Click **Add Mapping**
2. Select the source column
3. Search for and select the target attribute
4. Write a transformation expression if needed
5. Click **Create**

### Reject a suggestion

If a suggestion is incorrect:

1. Click the **X** or **Reject** button
2. The suggestion is removed and won't reappear

## Using the API

### List available attributes

Before creating mappings, browse available attributes:

```bash theme={null}
curl -X GET "https://api.narrative.io/v1/attributes" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json"
```

Response:

```json theme={null}
{
  "data": [
    {
      "id": "hl7_gender",
      "name": "hl7_gender",
      "description": "Gender using HL7 administrative gender codes",
      "type": "enum",
      "allowed_values": ["male", "female", "other", "unknown"]
    },
    {
      "id": "event_timestamp",
      "name": "event_timestamp",
      "description": "The timestamp when the event occurred",
      "type": "timestamp"
    }
  ]
}
```

### Get mapping suggestions for a dataset

Request auto-generated mapping suggestions:

```bash theme={null}
curl -X GET "https://api.narrative.io/v1/datasets/{dataset_id}/mappings/suggestions" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json"
```

Response:

```json theme={null}
{
  "data": [
    {
      "source_column": "gender",
      "suggested_attribute": "hl7_gender",
      "confidence": 0.95,
      "suggested_transformation": "LOWER(gender)",
      "sample_output": ["male", "female", "male"]
    },
    {
      "source_column": "event_date",
      "suggested_attribute": "event_timestamp",
      "confidence": 0.87,
      "suggested_transformation": "TO_TIMESTAMP(event_date, 'YYYY-MM-DD')",
      "sample_output": ["2024-01-15T00:00:00Z", "2024-01-16T00:00:00Z"]
    }
  ]
}
```

### Create a mapping

Create a new mapping for a dataset column:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "your-dataset-id",
    "source_column": "sex",
    "target_attribute": "hl7_gender",
    "transformation": "CASE sex WHEN '\''M'\'' THEN '\''male'\'' WHEN '\''F'\'' THEN '\''female'\'' ELSE '\''unknown'\'' END"
  }'
```

Response:

```json theme={null}
{
  "data": {
    "id": "mapping-12345",
    "dataset_id": "your-dataset-id",
    "source_column": "sex",
    "target_attribute": "hl7_gender",
    "transformation": "CASE sex WHEN 'M' THEN 'male' WHEN 'F' THEN 'female' ELSE 'unknown' END",
    "status": "active",
    "created_at": "2024-01-15T10:30:00Z"
  }
}
```

### Accept a system-proposed mapping

Accept a suggestion from the auto-generated mappings:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings/{suggestion_id}/accept" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json"
```

### Update an existing mapping

Modify a mapping's transformation:

```bash theme={null}
curl -X PATCH "https://api.narrative.io/v1/mappings/{mapping_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "transformation": "CASE LOWER(sex) WHEN '\''m'\'' THEN '\''male'\'' WHEN '\''f'\'' THEN '\''female'\'' WHEN '\''nb'\'' THEN '\''other'\'' ELSE '\''unknown'\'' END"
  }'
```

### Delete a mapping

Remove a mapping:

```bash theme={null}
curl -X DELETE "https://api.narrative.io/v1/mappings/{mapping_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

## Mapping types

The Mappings API accepts three mapping `type`s:

* `value_mapping` — a single SQL expression producing a primitive value.
* `object_mapping` — one expression per property of an object attribute.
* `cached_mapping` — joins the source dataset against a precomputed cache dataset at query time. Use this when the transformation is expensive or comes from an external workflow.

For the full schema, validation rules, and constraints (including the company-scope and opt-out restrictions on cached mappings), see [Mapping types](/reference/rosetta-stone/mapping-types).

## Writing transformation expressions

Transformation expressions are NQL expressions that convert source values to the target attribute format.

### Simple column reference

When no transformation is needed:

```sql theme={null}
source_column
```

### Type conversion

Convert a string to an integer:

```sql theme={null}
CAST(string_column AS INTEGER)
```

Convert to timestamp:

```sql theme={null}
TO_TIMESTAMP(date_string, 'YYYY-MM-DD')
```

### Conditional logic

Map discrete values:

```sql theme={null}
CASE source_column
  WHEN 'A' THEN 'value_a'
  WHEN 'B' THEN 'value_b'
  ELSE 'default'
END
```

Map ranges:

```sql theme={null}
CASE
  WHEN age < 18 THEN 'minor'
  WHEN age < 65 THEN 'adult'
  ELSE 'senior'
END
```

### Null handling

Provide a default for nulls:

```sql theme={null}
COALESCE(source_column, 'unknown')
```

Replace specific values with null:

```sql theme={null}
NULLIF(source_column, '')
```

### String manipulation

Normalize case:

```sql theme={null}
LOWER(source_column)
UPPER(source_column)
```

Trim whitespace:

```sql theme={null}
TRIM(source_column)
```

Extract a substring:

```sql theme={null}
SUBSTRING(source_column, 1, 10)
```

### Combining multiple columns

Concatenate values:

```sql theme={null}
CONCAT(first_name, ' ', last_name)
```

Build a struct:

```sql theme={null}
STRUCT(
  'email_sha256' AS type,
  hashed_email AS value,
  'hashed_email' AS context
)
```

## Complete worked example

### Scenario

You have a customer dataset with these columns:

| Column        | Sample values                  |
| ------------- | ------------------------------ |
| `cust_id`     | `"12345"`, `"67890"`           |
| `sex`         | `"M"`, `"F"`, `"X"`            |
| `signup_date` | `"01/15/2024"`, `"02/20/2024"` |
| `email_hash`  | `"a1b2c3..."`, `"d4e5f6..."`   |

### Step 1: Review suggestions

The system suggests:

* `sex` → `hl7_gender` (medium confidence)
* `signup_date` → `event_timestamp` (high confidence)
* `email_hash` → `email_sha256` (high confidence)

### Step 2: Accept high-confidence mappings

Accept the `signup_date` and `email_hash` mappings.

### Step 3: Customize the gender mapping

The suggested transformation doesn't handle `"X"` correctly. Create a custom mapping:

```sql theme={null}
CASE sex
  WHEN 'M' THEN 'male'
  WHEN 'F' THEN 'female'
  WHEN 'X' THEN 'other'
  ELSE 'unknown'
END
```

### Step 4: Create a manual mapping for customer ID

Map `cust_id` to the `unique_identifier` attribute:

```sql theme={null}
STRUCT(
  'customer_id' AS type,
  cust_id AS value,
  'internal' AS context
)
```

### Step 5: Test and activate

Test all mappings, then activate.

## Troubleshooting

| Issue                      | Cause                                              | Solution                              |
| -------------------------- | -------------------------------------------------- | ------------------------------------- |
| "Type mismatch" error      | Transformation output doesn't match attribute type | Use `CAST()` or adjust transformation |
| "Invalid enum value" error | Transformation produces values not in allowed list | Add missing cases to `CASE` statement |
| Null values in output      | Transformation doesn't handle all source values    | Add `ELSE` clause or use `COALESCE`   |
| "Column not found" error   | Typo in column name                                | Verify column name matches exactly    |

## Related content

<CardGroup>
  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand the mechanics of attributes and mappings
  </Card>

  <Card title="Validating Mappings" icon="circle-check" href="/guides/rosetta-stone/validating-mappings">
    Test and verify your mappings
  </Card>

  <Card title="Edge Cases" icon="triangle-exclamation" href="/guides/rosetta-stone/edge-cases">
    Handle complex mapping scenarios
  </Card>

  <Card title="Transformation Functions" icon="function" href="/reference/rosetta-stone/transformation-functions">
    Complete function reference
  </Card>

  <Card title="Mapping Types" icon="shapes" href="/reference/rosetta-stone/mapping-types">
    API reference for `value_mapping`, `object_mapping`, and `cached_mapping`
  </Card>
</CardGroup>


# Validating Mappings
Source: https://docs.narrative.io/guides/rosetta-stone/validating-mappings

Test and verify your schema mappings before production use

This guide covers how to test and validate your Rosetta Stone mappings before activating them. Validation catches errors early, ensuring data quality before your data enters the normalized view.

<Note>
  Before validating, you'll need created mappings. See [Mapping Schemas](/guides/rosetta-stone/mapping-schemas) to create your first mapping.
</Note>

## Why validate mappings

Validation serves several purposes:

**Catch transformation errors**: Identify syntax errors, type mismatches, or logic bugs in transformation expressions before they affect production data.

**Verify edge cases**: Ensure your transformations handle unusual or unexpected values in your source data.

**Preview output**: See exactly how your data will appear after normalization.

**Ensure attribute compliance**: Confirm that transformed values meet attribute validations (enums, ranges, patterns).

## Testing with sample data

### Using the UI

1. Navigate to your dataset's **Rosetta Stone** tab
2. Select one or more mappings
3. Click **Test Mappings**

The test interface displays:

* Sample rows from your source data
* The transformation applied
* The resulting output
* Any validation errors or warnings

\[Screenshot placeholder: Test results showing source data and transformed output]

Review each column to verify the transformation produces expected results.

### Using the API

Test a specific mapping against sample data:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings/{mapping_id}/test" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "sample_size": 100
  }'
```

Response:

```json theme={null}
{
  "data": {
    "mapping_id": "mapping-12345",
    "samples_tested": 100,
    "samples_passed": 97,
    "samples_failed": 3,
    "results": [
      {
        "source_value": "M",
        "transformed_value": "male",
        "validation_status": "passed"
      },
      {
        "source_value": "X",
        "transformed_value": null,
        "validation_status": "failed",
        "error": "Value 'X' not in allowed enum values"
      }
    ],
    "error_summary": {
      "enum_validation_failed": 3
    }
  }
}
```

Test with custom sample data:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings/{mapping_id}/test" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "custom_samples": [
      {"sex": "M"},
      {"sex": "F"},
      {"sex": "X"},
      {"sex": null},
      {"sex": ""}
    ]
  }'
```

This allows you to specifically test edge cases you're concerned about.

## Common validation checks

### Type conversion accuracy

Verify that type conversions work correctly:

| Source type        | Target type     | Test for                            |
| ------------------ | --------------- | ----------------------------------- |
| String → Integer   | Numeric strings | Non-numeric values, decimals        |
| String → Timestamp | Date strings    | Malformed dates, timezone handling  |
| Integer → Float    | Numbers         | Precision loss                      |
| String → Boolean   | Text values     | Case sensitivity, unexpected values |

**Example**: Testing date parsing

```sql theme={null}
-- Transformation
TO_TIMESTAMP(date_col, 'MM/DD/YYYY')
```

Test with:

* Valid dates: `"01/15/2024"`
* Invalid format: `"2024-01-15"` (wrong format)
* Edge cases: `"02/29/2024"` (leap year), `"13/01/2024"` (invalid month)

### Enum value coverage

For enum attributes, ensure all source values map to valid enum members:

1. Get unique values from your source column
2. Trace each through your transformation
3. Verify all outputs are in the allowed values list

**Example**: Testing gender mapping

Source values: `"M"`, `"F"`, `"Male"`, `"Female"`, `"m"`, `"f"`, `""`, `null`

Transformation:

```sql theme={null}
CASE UPPER(gender)
  WHEN 'M' THEN 'male'
  WHEN 'MALE' THEN 'male'
  WHEN 'F' THEN 'female'
  WHEN 'FEMALE' THEN 'female'
  ELSE 'unknown'
END
```

Verify that all variations map correctly.

### Null handling

Test how your transformation handles:

* Explicit nulls
* Empty strings
* Missing values (if applicable)

Ensure nulls are handled intentionally, not accidentally.

### Date and timestamp parsing

Date transformations are common sources of errors. Test:

| Scenario                     | Example                        |
| ---------------------------- | ------------------------------ |
| Standard format              | `"01/15/2024"`                 |
| Different separators         | `"01-15-2024"`, `"01.15.2024"` |
| Year variations              | `"01/15/24"` (two-digit year)  |
| Time zones                   | `"2024-01-15T10:30:00-05:00"`  |
| UTC timestamps               | `"2024-01-15T15:30:00Z"`       |
| Timestamps with milliseconds | `"2024-01-15T15:30:00.123Z"`   |

## Reviewing validation results

### Understanding error messages

Common validation errors and their meanings:

| Error                           | Cause                                    | Solution                                          |
| ------------------------------- | ---------------------------------------- | ------------------------------------------------- |
| "Type mismatch"                 | Output type doesn't match attribute type | Add explicit `CAST()` or fix transformation logic |
| "Invalid enum value"            | Output not in allowed values             | Add case to transformation or map to default      |
| "Validation failed: minimum"    | Numeric value below minimum              | Add range check or filter invalid data            |
| "Validation failed: pattern"    | String doesn't match regex               | Verify format or clean source data                |
| "Null value for required field" | Required field is null                   | Add `COALESCE()` or fix source data               |

### Sample output review

When reviewing test output:

1. **Spot check random samples**: Verify that typical values transform correctly
2. **Focus on edge cases**: Pay extra attention to unusual values
3. **Check for unexpected nulls**: Nulls appearing where values are expected indicate transformation issues
4. **Verify consistency**: Same input should always produce same output

## Iterating on mappings

When validation reveals issues, follow this process:

### 1. Identify the root cause

Is the issue:

* A bug in the transformation expression?
* Unexpected values in source data?
* A mismatch between data and attribute definition?

### 2. Update the mapping

Fix the transformation expression to handle the issue:

```bash theme={null}
curl -X PATCH "https://api.narrative.io/v1/mappings/{mapping_id}" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "transformation": "CASE ... END"
  }'
```

### 3. Re-validate

Run tests again to confirm the fix works and doesn't introduce new issues.

### 4. Test with full sample

Before activating, test against a larger sample to catch less common edge cases:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings/{mapping_id}/test" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "sample_size": 1000
  }'
```

## Promoting to production

Once validation passes:

### Review summary statistics

Check the overall validation results:

* What percentage of samples passed?
* Are failures concentrated in specific value patterns?
* Is the failure rate acceptable for your use case?

### Activate the mapping

Via UI:

1. Click **Activate** on the mapping
2. Confirm the activation

Via API:

```bash theme={null}
curl -X POST "https://api.narrative.io/v1/mappings/{mapping_id}/activate" \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

### Monitor initial results

After activation, query the normalized data to verify it looks correct:

```sql theme={null}
SELECT
    target_attribute,
    COUNT(1) as count
FROM narrative.rosetta_stone
WHERE _nio_source_dataset_id = 'your-dataset-id'
GROUP BY target_attribute
```

This helps catch any issues that weren't apparent in sample testing.

## Best practices

**Test with representative data**: Ensure your test samples include the full variety of values in your dataset.

**Include edge cases deliberately**: Don't rely on random sampling to cover edge cases. Test specific problematic values.

**Document mapping decisions**: Record why certain values map to certain outputs, especially for ambiguous cases.

**Version your mappings**: Keep track of mapping changes so you can roll back if issues arise.

**Test after source data changes**: When your source data format changes, re-validate existing mappings.

## Related content

<CardGroup>
  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Create and manage mappings
  </Card>

  <Card title="Edge Cases" icon="triangle-exclamation" href="/guides/rosetta-stone/edge-cases">
    Handle complex mapping scenarios
  </Card>

  <Card title="Transformation Functions" icon="function" href="/reference/rosetta-stone/transformation-functions">
    Complete function reference
  </Card>

  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand the normalization pipeline
  </Card>
</CardGroup>


# SDK Authentication
Source: https://docs.narrative.io/guides/sdk/authentication

Configure API key authentication for the Narrative TypeScript SDK

The Narrative SDK uses API keys for authentication. This guide covers how to configure authentication securely and manage credentials across different environments.

## Prerequisites

* An API key with appropriate permissions (see [API Keys](/account-settings/api-keys) and [Permissions Reference](/reference/security/permissions))
* The Narrative SDK installed in your project

## Authentication methods

### Environment variable (recommended)

Store your API key in an environment variable to keep it out of source control:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});
```

Set the environment variable before running your application:

<Tabs>
  <Tab title="Mac/Linux">
    ```bash theme={null}
    export NARRATIVE_API_KEY="your-api-key-here"
    node app.js
    ```
  </Tab>

  <Tab title="Windows (PowerShell)">
    ```powershell theme={null}
    $env:NARRATIVE_API_KEY = "your-api-key-here"
    node app.js
    ```
  </Tab>

  <Tab title="Windows (CMD)">
    ```batch theme={null}
    set NARRATIVE_API_KEY=your-api-key-here
    node app.js
    ```
  </Tab>
</Tabs>

### Using a .env file

For local development, use a `.env` file with a package like `dotenv`:

```bash theme={null}
npm install dotenv
```

Create a `.env` file in your project root:

```bash theme={null}
NARRATIVE_API_KEY=your-api-key-here
NARRATIVE_ENVIRONMENT=prod
```

Load the environment variables in your code:

```typescript theme={null}
import 'dotenv/config';
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: process.env.NARRATIVE_ENVIRONMENT || 'prod',
});
```

<Warning>
  Never commit `.env` files to source control. Add `.env` to your `.gitignore` file.
</Warning>

### Direct configuration

For testing or when credentials are managed externally:

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: 'your-api-key-here',
});
```

<Warning>
  Avoid hardcoding API keys in source code. Use environment variables or a secrets manager in production.
</Warning>

## Environment selection

The SDK supports multiple environments:

| Environment | Use case                          |
| ----------- | --------------------------------- |
| `prod`      | Production workloads (default)    |
| `dev`       | Development and testing           |
| Custom URL  | Self-hosted or custom deployments |

```typescript theme={null}
// Production (default)
const prodApi = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

// Development
const devApi = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: 'dev',
});

// Custom URL
const customApi = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: 'https://custom.narrative.example.com/',
});
```

## Verifying authentication

Test your credentials by making a simple API call:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

async function verifyAuth() {
  const api = new NarrativeApi({
    apiKey: process.env.NARRATIVE_API_KEY,
  });

  try {
    const datasets = await api.getDatasets();
    console.log('Authentication successful!');
    console.log(`Found ${datasets.records.length} datasets`);
  } catch (error) {
    if (error.status === 401) {
      console.error('Authentication failed: Invalid API key');
    } else if (error.status === 403) {
      console.error('Authentication failed: Insufficient permissions');
    } else {
      console.error('Unexpected error:', error);
    }
  }
}

verifyAuth();
```

## Updating credentials at runtime

You can change the API key after initialization:

```typescript theme={null}
const api = new NarrativeApi({});

// Set API key later
api.setApiKey('new-api-key');

// Verify the change
console.log('Current key:', api.getApiKey());
```

## Security best practices

| Practice                  | Description                                      |
| ------------------------- | ------------------------------------------------ |
| Use environment variables | Keep credentials out of source code              |
| Rotate keys regularly     | Create new keys and revoke old ones periodically |
| Use minimum permissions   | Grant only the permissions needed for the task   |
| Set key expiration        | Configure expiration dates for all API keys      |
| Monitor key usage         | Review access logs for unusual activity          |

## Troubleshooting

| Issue                    | Cause                        | Solution                                           |
| ------------------------ | ---------------------------- | -------------------------------------------------- |
| `401 Unauthorized`       | Invalid or expired API key   | Verify the key is correct and hasn't expired       |
| `403 Forbidden`          | Insufficient permissions     | Check the key's permissions in Settings > API Keys |
| `undefined` API key      | Environment variable not set | Ensure the variable is exported in your shell      |
| Key not working in CI/CD | Variable not configured      | Add the secret to your CI/CD environment           |

## Related content

<CardGroup>
  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys
  </Card>

  <Card title="Configuration Reference" icon="gear" href="/reference/sdks/typescript/configuration">
    Complete configuration options
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/guides/sdk/error-handling">
    Handle authentication errors gracefully
  </Card>

  <Card title="Permissions Reference" icon="lock-keyhole" href="/reference/security/permissions">
    All available permission resources and access levels
  </Card>
</CardGroup>


# Choosing the Right Model
Source: https://docs.narrative.io/guides/sdk/choosing-inference-models

Select the appropriate model based on task complexity, speed, and cost

Model Inference supports multiple models with different capabilities. This guide helps you choose the right model for your use case.

## Quick selection guide

| Your Priority            | Recommended Model                                                             |
| ------------------------ | ----------------------------------------------------------------------------- |
| **Speed and cost**       | `anthropic.claude-haiku-4.5` or `openai.o4-mini`                              |
| **Balanced performance** | `anthropic.claude-sonnet-4.5` or `anthropic.claude-sonnet-4.6`                |
| **Complex reasoning**    | `anthropic.claude-opus-4.5`, `anthropic.claude-opus-4.6`, or `openai.gpt-4.1` |

## Model comparison

### Anthropic Claude models

| Model             | Speed   | Capability | Best For                                            |
| ----------------- | ------- | ---------- | --------------------------------------------------- |
| Claude Haiku 4.5  | Fastest | Good       | Simple classification, extraction                   |
| Claude Sonnet 4.5 | Fast    | Better     | Analysis, summarization, most tasks                 |
| Claude Sonnet 4.6 | Fast    | Better     | Latest Sonnet generation with improved reasoning    |
| Claude Opus 4.5   | Slower  | Best       | Complex reasoning, nuanced decisions                |
| Claude Opus 4.6   | Slower  | Best       | Latest Opus generation for the most demanding tasks |

### OpenAI models

| Model        | Speed    | Capability | Best For                      |
| ------------ | -------- | ---------- | ----------------------------- |
| o4-mini      | Fastest  | Good       | Quick responses, simple tasks |
| GPT-4.1      | Moderate | Better     | Advanced reasoning            |
| GPT-oss-120b | Moderate | Good       | General-purpose tasks         |

## When to use smaller models

Use Claude Haiku or o4-mini when:

* **Task is straightforward**: Binary classification, simple extraction
* **High volume**: Processing many items where speed matters
* **Cost sensitivity**: Budget constraints require efficiency
* **Latency matters**: User-facing features needing fast response

```typescript theme={null}
// Good use case for Haiku: Simple classification
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-haiku-4.5',
  messages: [
    {
      role: 'user',
      content: [{ type: 'text', text: 'Is this email spam? Subject: "You won $1M!"' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        is_spam: { type: 'boolean' },
        confidence: { type: 'number', minimum: 0, maximum: 1 }
      },
      required: ['is_spam', 'confidence']
    }
  }
});
```

## When to use medium models

Use Claude Sonnet 4.5 when:

* **Task requires understanding**: Content analysis, summarization
* **Balanced needs**: Good quality without excessive cost
* **Most production use cases**: Default choice for typical workflows

```typescript theme={null}
// Good use case for Sonnet: Content analysis
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'system',
      content: [{ type: 'text', text: 'Analyze dataset descriptions for quality and completeness.' }]
    },
    {
      role: 'user',
      content: [{ type: 'text', text: 'Dataset: Customer transactions\nColumns: user_id, amount, date, category' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        summary: { type: 'string' },
        completeness_score: { type: 'number', minimum: 0, maximum: 1 },
        missing_elements: { type: 'array', items: { type: 'string' } },
        suggestions: { type: 'array', items: { type: 'string' } }
      },
      required: ['summary', 'completeness_score']
    }
  }
});
```

## When to use larger models

Use Claude Opus 4.5 or GPT-4.1 when:

* **Complex reasoning required**: Multi-step analysis, nuanced judgment
* **High stakes**: Decisions with significant impact
* **Ambiguous inputs**: Tasks requiring interpretation
* **Quality over speed**: Accuracy is paramount

```typescript theme={null}
// Good use case for Opus: Complex analysis
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-opus-4.5',
  messages: [
    {
      role: 'system',
      content: [{ type: 'text', text: 'You are a data governance expert. Analyze datasets for compliance risks.' }]
    },
    {
      role: 'user',
      content: [{
        type: 'text',
        text: `Analyze this dataset schema for privacy compliance:
          - email (string)
          - phone (string)
          - purchase_history (array)
          - ip_address (string)
          - device_fingerprint (string)`
      }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        risk_level: { type: 'string', enum: ['low', 'medium', 'high', 'critical'] },
        pii_fields: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              field: { type: 'string' },
              pii_type: { type: 'string' },
              risk: { type: 'string' }
            },
            required: ['field', 'pii_type', 'risk']
          }
        },
        compliance_concerns: { type: 'array', items: { type: 'string' } },
        recommendations: { type: 'array', items: { type: 'string' } }
      },
      required: ['risk_level', 'pii_fields', 'compliance_concerns', 'recommendations']
    }
  }
});
```

## Task-based recommendations

### Classification tasks

| Complexity                                  | Recommended Model      |
| ------------------------------------------- | ---------------------- |
| Binary (yes/no, spam/not spam)              | Claude Haiku           |
| Multi-class (3-5 categories)                | Claude Haiku or Sonnet |
| Complex taxonomy (many categories, nuanced) | Claude Sonnet or Opus  |

### Extraction tasks

| Complexity                                | Recommended Model |
| ----------------------------------------- | ----------------- |
| Simple fields (dates, names, numbers)     | Claude Haiku      |
| Structured entities (addresses, products) | Claude Sonnet     |
| Complex relationships (multi-entity)      | Claude Opus       |

### Generation tasks

| Complexity                               | Recommended Model      |
| ---------------------------------------- | ---------------------- |
| Short text (taglines, labels)            | Claude Haiku or Sonnet |
| Medium content (descriptions, summaries) | Claude Sonnet          |
| Long-form (reports, analysis)            | Claude Sonnet or Opus  |

### Transformation tasks

| Complexity                                        | Recommended Model |
| ------------------------------------------------- | ----------------- |
| Format conversion (dates, units)                  | Claude Haiku      |
| Language translation (technical to plain)         | Claude Sonnet     |
| Complex interpretation (natural language to code) | Claude Opus       |

## Testing different models

Try multiple models on sample data to compare quality:

```typescript theme={null}
async function compareModels(prompt: string, schema: object) {
  const models = [
    'anthropic.claude-haiku-4.5',
    'anthropic.claude-sonnet-4.5',
    'anthropic.claude-opus-4.5'
  ];

  const results = await Promise.all(
    models.map(async (model) => {
      const start = Date.now();
      const job = await api.runModelInference({
        data_plane_id: 'dp_your_data_plane_id',
        model,
        messages: [
          { role: 'user', content: [{ type: 'text', text: prompt }] }
        ],
        inference_config: { output_format_schema: schema }
      });

      const result = await waitForJob(job.id);
      const duration = Date.now() - start;

      return {
        model,
        duration,
        tokens: result.result?.usage.total_tokens,
        output: result.result?.structured_output
      };
    })
  );

  console.table(results.map(r => ({
    model: r.model,
    duration_ms: r.duration,
    tokens: r.tokens
  })));

  return results;
}
```

## Best practices

| Practice                 | Description                                             |
| ------------------------ | ------------------------------------------------------- |
| Start small              | Begin with Haiku, upgrade if quality is insufficient    |
| Test on samples          | Compare models on representative data before production |
| Monitor quality          | Track output quality metrics over time                  |
| Balance cost and quality | Don't over-engineer simple tasks                        |
| Consider latency         | User-facing features may need faster models             |

## Related content

<CardGroup>
  <Card title="Supported Models" icon="microchip" href="/reference/model-inference/supported-models">
    Complete model reference
  </Card>

  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Submit inference requests
  </Card>

  <Card title="Model Inference Overview" icon="brain" href="/concepts/model-inference/overview">
    How inference works
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Define response schemas
  </Card>
</CardGroup>


# Error Handling
Source: https://docs.narrative.io/guides/sdk/error-handling

Handle errors gracefully when using the Narrative TypeScript SDK

Robust error handling is essential for building reliable applications with the Narrative SDK. This guide covers common error types and patterns for handling them.

## Prerequisites

* SDK installed and configured (see [SDK Quickstart](/getting-started/sdk-quickstart))

## Error types

### HTTP errors

The SDK throws errors for HTTP response codes indicating failures:

| Status | Name                  | Description                                   |
| ------ | --------------------- | --------------------------------------------- |
| 400    | Bad Request           | Invalid request parameters or malformed query |
| 401    | Unauthorized          | Invalid or missing API key                    |
| 403    | Forbidden             | API key lacks required permissions            |
| 404    | Not Found             | Resource doesn't exist                        |
| 429    | Too Many Requests     | Rate limit exceeded                           |
| 500    | Internal Server Error | Server-side error                             |
| 502    | Bad Gateway           | Upstream service error                        |
| 503    | Service Unavailable   | Service temporarily unavailable               |

### Network errors

Connection issues throw standard JavaScript errors:

* Connection timeout
* DNS resolution failure
* Network unreachable

### Validation errors

Query or data validation failures return detailed error messages:

* NQL syntax errors
* Schema validation failures
* Invalid parameter values

## Basic error handling

Wrap SDK calls in try-catch blocks:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

try {
  const datasets = await api.getDatasets();
  console.log('Datasets:', datasets.records.length);
} catch (error) {
  console.error('Error fetching datasets:', error);
}
```

## Handling specific errors

Check the error status for specific handling:

```typescript theme={null}
try {
  const dataset = await api.getDataset(12345);
  console.log('Dataset:', dataset.name);
} catch (error) {
  if (error.status === 401) {
    console.error('Authentication failed: Check your API key');
  } else if (error.status === 403) {
    console.error('Permission denied: API key lacks access to this dataset');
  } else if (error.status === 404) {
    console.error('Dataset not found');
  } else if (error.status === 429) {
    console.error('Rate limited: Too many requests');
  } else if (error.status >= 500) {
    console.error('Server error: Try again later');
  } else {
    console.error('Unexpected error:', error);
  }
}
```

## Rate limiting

Handle rate limits with exponential backoff:

```typescript theme={null}
async function withRetry<T>(
  operation: () => Promise<T>,
  maxRetries = 5,
  baseDelayMs = 1000
): Promise<T> {
  let lastError: Error | undefined;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      lastError = error as Error;

      // Only retry on rate limits and server errors
      const status = (error as any).status;
      if (status !== 429 && status < 500) {
        throw error;
      }

      if (attempt < maxRetries - 1) {
        const delay = baseDelayMs * Math.pow(2, attempt);
        const jitter = Math.random() * 1000;
        console.log(`Retry ${attempt + 1}/${maxRetries} after ${delay}ms`);
        await new Promise(resolve => setTimeout(resolve, delay + jitter));
      }
    }
  }

  throw lastError;
}

// Usage
const datasets = await withRetry(() => api.getDatasets());
```

## Query error handling

NQL queries can fail for various reasons:

```typescript theme={null}
try {
  const result = await api.executeNql({
    nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
    data_plane_id: null,
  });

  if (result.state === 'failed') {
    console.error('Query failed:', result.failures);
  } else {
    console.log('Query completed:', result.result.rows, 'rows');
  }
} catch (error) {
  if (error.status === 400) {
    // Parse error details for NQL syntax issues
    console.error('Invalid query:', error.message);
  } else {
    console.error('Query error:', error);
  }
}
```

### Validating before execution

Catch syntax errors before running expensive queries:

```typescript theme={null}
async function safeExecuteQuery(nql: string) {
  // First validate the query
  try {
    await api.validateNql({ nql, data_plane_id: null });
  } catch (validationError) {
    console.error('Query validation failed:', validationError);
    throw new Error('Invalid query syntax');
  }

  // Then execute
  return await api.executeNql({ nql, data_plane_id: null });
}
```

## Creating an error wrapper

Build a reusable error handler:

```typescript theme={null}
class NarrativeError extends Error {
  status?: number;
  details?: unknown;

  constructor(message: string, status?: number, details?: unknown) {
    super(message);
    this.name = 'NarrativeError';
    this.status = status;
    this.details = details;
  }

  isRetryable(): boolean {
    return this.status === 429 || (this.status ?? 0) >= 500;
  }

  isAuthError(): boolean {
    return this.status === 401 || this.status === 403;
  }

  isNotFound(): boolean {
    return this.status === 404;
  }
}

async function wrapApiCall<T>(operation: () => Promise<T>): Promise<T> {
  try {
    return await operation();
  } catch (error) {
    const status = (error as any).status;
    const message = (error as any).message || 'Unknown error';
    throw new NarrativeError(message, status, error);
  }
}

// Usage
try {
  const dataset = await wrapApiCall(() => api.getDataset(12345));
} catch (error) {
  if (error instanceof NarrativeError) {
    if (error.isAuthError()) {
      console.error('Authentication issue');
    } else if (error.isNotFound()) {
      console.error('Resource not found');
    } else if (error.isRetryable()) {
      console.error('Temporary error, retry later');
    }
  }
}
```

## Logging errors

Implement structured error logging:

```typescript theme={null}
function logError(context: string, error: unknown) {
  const errorInfo = {
    timestamp: new Date().toISOString(),
    context,
    message: (error as Error).message,
    status: (error as any).status,
    stack: (error as Error).stack,
  };

  console.error(JSON.stringify(errorInfo, null, 2));

  // Send to your logging service
  // await logService.error(errorInfo);
}

try {
  const datasets = await api.getDatasets();
} catch (error) {
  logError('getDatasets', error);
  throw error;
}
```

## Graceful degradation

Provide fallbacks when services are unavailable:

```typescript theme={null}
async function getDatasetsWithFallback() {
  try {
    const response = await api.getDatasets();
    return { source: 'api', data: response.records };
  } catch (error) {
    const status = (error as any).status;

    if (status >= 500 || status === 429) {
      // Return cached data if available
      const cachedData = await getCachedDatasets();
      if (cachedData) {
        console.warn('Using cached datasets due to API error');
        return { source: 'cache', data: cachedData };
      }
    }

    throw error;
  }
}
```

## Best practices

| Practice             | Description                                  |
| -------------------- | -------------------------------------------- |
| Always use try-catch | Wrap all SDK calls in error handlers         |
| Check error status   | Handle different errors appropriately        |
| Implement retries    | Use exponential backoff for transient errors |
| Validate inputs      | Catch errors early with validation           |
| Log errors           | Record errors for debugging and monitoring   |
| Graceful degradation | Provide fallbacks when possible              |

## Troubleshooting

| Issue                   | Cause                              | Solution                            |
| ----------------------- | ---------------------------------- | ----------------------------------- |
| Frequent 401 errors     | API key expired or invalid         | Generate a new API key              |
| Frequent 429 errors     | Too many requests                  | Implement rate limiting and backoff |
| Intermittent 500 errors | Server issues                      | Implement retry logic               |
| Network timeouts        | Slow connection or large responses | Increase timeout or paginate        |

## Related content

<CardGroup>
  <Card title="Authentication" icon="key" href="/guides/sdk/authentication">
    Configure authentication properly
  </Card>

  <Card title="API Keys" icon="gear" href="/account-settings/api-keys">
    Manage API keys and permissions
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Handle async operation errors
  </Card>

  <Card title="SDK Reference" icon="book" href="/reference/sdks/typescript">
    Complete SDK documentation
  </Card>
</CardGroup>


# Executing NQL Queries
Source: https://docs.narrative.io/guides/sdk/executing-queries

Run, validate, and compile NQL queries using the TypeScript SDK

The Narrative SDK provides methods to execute, validate, and compile NQL queries programmatically. This guide covers common query patterns and best practices.

<Note>
  For NQL syntax and language features, see the [NQL Reference](/nql/general/syntax).
</Note>

## Prerequisites

* SDK installed and configured (see [SDK Quickstart](/getting-started/sdk-quickstart))
* Familiarity with [NQL syntax](/nql/general/syntax)
* An API key with query permissions

## Running a query

### Basic execution

Use `executeNql()` to run a query and retrieve results:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

const result = await api.executeNql({
  nql: `
    SELECT _nio_id, _nio_updated_at
    FROM company_data."my_dataset"
    LIMIT 100
  `,
  data_plane_id: null,
});

console.log('Query state:', result.state);
console.log('Rows returned:', result.result.rows);
console.log('Cost:', result.result.cost);
```

### Query input options

The `executeNql()` method accepts the following options:

| Option              | Type             | Required | Description                                                                                                          |
| ------------------- | ---------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `nql`               | `string`         | Yes      | The NQL query to execute                                                                                             |
| `data_plane_id`     | `string \| null` | Yes      | Target data plane ID, or `null` for default                                                                          |
| `execution_cluster` | `object`         | No       | Cluster configuration (`shared` or `dedicated`). See [Compute Pools](/concepts/primitives/compute-pools)             |
| `create_as_view`    | `boolean`        | No       | Create the result as a [view dataset](/concepts/nql/materialized-views#view-datasets) instead of a materialized view |

```typescript theme={null}
const result = await api.executeNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
  execution_cluster: { type: 'dedicated' },
});
```

### Creating a view dataset

Set `create_as_view` to `true` to create a [view dataset](/concepts/nql/materialized-views#view-datasets) instead of a materialized view. A view dataset stores only the NQL query definition—not the query results. The NQL is inlined and re-evaluated at query time whenever other queries reference the view dataset.

```typescript theme={null}
const result = await api.executeNql({
  nql: `
    SELECT user_id, email, event_type
    FROM company_data."customer_events"
    WHERE event_type = 'purchase'
  `,
  data_plane_id: null,
  create_as_view: true,
});
```

<Warning>
  View datasets have several restrictions compared to materialized views. You cannot create access rules, connections, or use features like `MERGE ON`, `PARTITIONED_BY`, or chunking strategies with view datasets. See [view dataset limitations](/concepts/nql/materialized-views#view-dataset-limitations) for the full list.
</Warning>

### Understanding the result

The `NqlResult` object contains execution metadata:

```typescript theme={null}
interface NqlResult {
  id: string;              // Job ID
  state: string;           // 'completed', 'failed', etc.
  created_at: string;      // When the query was submitted
  completed_at: string;    // When execution finished
  result: {
    type: string;
    rows: number;          // Number of rows returned
    cost: number;          // Query cost
    nql_type: string;
  };
  input: {
    nql: string;           // The original query
    compiled_select?: string;
  };
  failures: unknown[];     // Any execution errors
}
```

## Validating a query

Use `validateNql()` to check query syntax without execution:

```typescript theme={null}
const validation = await api.validateNql({
  nql: `
    SELECT _nio_id
    FROM company_data."my_dataset"
    LIMIT 10
  `,
  data_plane_id: null,
});

console.log('Query is valid');
```

If the query has syntax errors, the method throws an error with details about the issue.

## Compiling a query

Use `compileNql()` to see the transpiled SQL without executing:

```typescript theme={null}
const compiled = await api.compileNql({
  nql: `
    SELECT _nio_id, _nio_updated_at
    FROM company_data."my_dataset"
    WHERE _nio_updated_at > CURRENT_DATE - INTERVAL '7' DAY
    LIMIT 100
  `,
  data_plane_id: null,
});

console.log('Transpiled SQL:', compiled.sql);

if (compiled.mappingErrors) {
  console.log('Mapping errors:', compiled.mappingErrors);
}
```

This is useful for:

* Debugging query issues
* Understanding how NQL maps to the underlying SQL dialect
* Verifying Rosetta Stone mappings

## Parsing a query

Use `parseNql()` to get the abstract syntax tree (AST):

```typescript theme={null}
const ast = await api.parseNql({
  nql: 'SELECT _nio_id FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
});

console.log('AST:', JSON.stringify(ast, null, 2));
```

## Retrieving a previous query

If you have a job ID from a previous query, retrieve its results:

```typescript theme={null}
const result = await api.getNqlByJobId('job-id-here');

console.log('Query state:', result.state);
console.log('Rows returned:', result.result.rows);
```

## Query patterns

### Filtering data

```typescript theme={null}
const result = await api.executeNql({
  nql: `
    SELECT sha256_hashed_email, event_type, event_timestamp
    FROM company_data."customer_events"
    WHERE event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
      AND event_type = 'purchase'
    LIMIT 1000
  `,
  data_plane_id: null,
});
```

### Aggregations

```typescript theme={null}
const result = await api.executeNql({
  nql: `
    SELECT
      event_type,
      COUNT(1) as event_count
    FROM company_data."customer_events"
    GROUP BY event_type
    LIMIT 100
  `,
  data_plane_id: null,
});
```

### Using Rosetta Stone attributes

Query normalized data across datasets:

```typescript theme={null}
const result = await api.executeNql({
  nql: `
    SELECT
      hl7_gender,
      unique_id,
      timestamp_utc
    FROM narrative.rosetta_stone
    LIMIT 100
  `,
  data_plane_id: null,
});
```

## Error handling

Wrap query execution in try-catch to handle errors:

```typescript theme={null}
try {
  const result = await api.executeNql({
    nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
    data_plane_id: null,
  });

  if (result.state === 'completed') {
    console.log('Success:', result.result.rows, 'rows');
  } else {
    console.log('Query state:', result.state);
  }
} catch (error) {
  if (error.status === 400) {
    console.error('Invalid query:', error.message);
  } else if (error.status === 403) {
    console.error('Permission denied');
  } else {
    console.error('Query failed:', error);
  }
}
```

## Best practices

| Practice                  | Description                                |
| ------------------------- | ------------------------------------------ |
| Always use LIMIT          | Prevent unbounded result sets              |
| Validate before executing | Use `compileNql()` to catch errors early   |
| Handle errors gracefully  | Wrap calls in try-catch                    |
| Use appropriate timeouts  | Long-running queries may need monitoring   |
| Filter at the source      | Apply WHERE clauses to reduce data scanned |

## Related content

<CardGroup>
  <Card title="NQL Syntax Reference" icon="book" href="/nql/general/syntax">
    Complete NQL language reference
  </Card>

  <Card title="NQL Design Philosophy" icon="lightbulb" href="/concepts/nql/design-philosophy">
    Understanding NQL's approach
  </Card>

  <Card title="Query Optimization" icon="gauge-high" href="/guides/nql/query-optimization">
    Tips for efficient queries
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/guides/sdk/error-handling">
    Handle SDK errors gracefully
  </Card>
</CardGroup>


# SDK Guides
Source: https://docs.narrative.io/guides/sdk/index

Step-by-step instructions for common SDK tasks

These guides help you accomplish specific tasks using the Narrative TypeScript SDK. They assume basic familiarity with the SDK and focus on practical implementation.

<Info>
  New to the SDK? Start with the [SDK Quickstart](/getting-started/sdk-quickstart) tutorial to install and configure the SDK.
</Info>

## Getting started

<CardGroup>
  <Card title="SDK Quickstart" icon="rocket" href="/getting-started/sdk-quickstart">
    Install the SDK and run your first query
  </Card>

  <Card title="Authentication" icon="key" href="/guides/sdk/authentication">
    Configure API key authentication
  </Card>
</CardGroup>

## Data operations

<CardGroup>
  <Card title="Executing Queries" icon="terminal" href="/guides/sdk/executing-queries">
    Run, validate, and compile NQL queries
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets">
    Create, read, update, and delete datasets
  </Card>

  <Card title="Uploading Data" icon="upload" href="/guides/sdk/uploading-data">
    Upload files to datasets
  </Card>
</CardGroup>

## Model Inference

<CardGroup>
  <Card title="Running Model Inference" icon="brain" href="/guides/sdk/running-model-inference">
    Submit inference requests to hosted LLMs
  </Card>

  <Card title="Using Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Define JSON Schema for typed responses
  </Card>

  <Card title="Choosing Models" icon="scale-balanced" href="/guides/sdk/choosing-inference-models">
    Select the right model for your task
  </Card>
</CardGroup>

## Monitoring and reliability

<CardGroup>
  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor asynchronous job status
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/guides/sdk/error-handling">
    Handle errors gracefully
  </Card>
</CardGroup>

## Reference

<CardGroup>
  <Card title="SDK Reference" icon="book" href="/reference/sdks/typescript">
    TypeScript SDK overview and architecture
  </Card>

  <Card title="Configuration" icon="gear" href="/reference/sdks/typescript/configuration">
    Complete configuration options
  </Card>

  <Card title="API Modules" icon="puzzle-piece" href="/reference/sdks/typescript/api-modules">
    All available methods and parameters
  </Card>

  <Card title="REST API" icon="code" href="/api-reference">
    Full REST API documentation
  </Card>
</CardGroup>

## Recipes

<CardGroup>
  <Card title="Automating Data Workflows" icon="robot" href="/cookbooks/sdk/typescript-automation">
    Build a complete automation pipeline
  </Card>
</CardGroup>


# Managing Datasets
Source: https://docs.narrative.io/guides/sdk/managing-datasets

Create, read, update, and delete datasets using the TypeScript SDK

The Narrative SDK provides comprehensive methods for managing datasets programmatically. This guide covers common dataset operations and patterns.

## Prerequisites

* SDK installed and configured (see [SDK Quickstart](/getting-started/sdk-quickstart))
* An API key with dataset permissions

## Listing datasets

Retrieve all datasets accessible to your account:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

const response = await api.getDatasets();

console.log(`Found ${response.records.length} datasets:`);
for (const dataset of response.records) {
  console.log(`- ${dataset.name} (ID: ${dataset.id}, Status: ${dataset.status})`);
}
```

## Getting dataset details

Retrieve details for a specific dataset by ID:

```typescript theme={null}
const dataset = await api.getDataset(12345);

console.log('Name:', dataset.name);
console.log('Description:', dataset.description);
console.log('Status:', dataset.status);
console.log('Created:', dataset.created_at);
console.log('Updated:', dataset.updated_at);
console.log('Tags:', dataset.tags);
```

### Dataset properties

| Property       | Type       | Description                        |
| -------------- | ---------- | ---------------------------------- |
| `id`           | `number`   | Unique dataset identifier          |
| `name`         | `string`   | Dataset name                       |
| `display_name` | `string`   | Human-readable display name        |
| `description`  | `string`   | Dataset description                |
| `status`       | `string`   | `active`, `archived`, or `pending` |
| `schema`       | `object`   | Dataset schema definition          |
| `tags`         | `string[]` | Tags for organization              |
| `created_at`   | `string`   | Creation timestamp                 |
| `updated_at`   | `string`   | Last update timestamp              |
| `write_mode`   | `string`   | `append` or `overwrite`            |

## Creating a dataset

Create a new dataset with schema definition:

```typescript theme={null}
const newDataset = await api.createDataset({
  name: 'customer_events',
  display_name: 'Customer Events',
  description: 'Customer interaction events from our platform',
  schema: {
    properties: {
      customer_id: { type: 'string' },
      event_type: { type: 'string' },
      event_timestamp: { type: 'timestamptz' },
      event_value: { type: 'double' },
    },
  },
  tags: ['customers', 'events'],
});

console.log('Created dataset:', newDataset.id);
```

### Schema property types

| Type          | Description             |
| ------------- | ----------------------- |
| `string`      | Text values             |
| `long`        | Integer numbers         |
| `double`      | Floating-point numbers  |
| `boolean`     | True/false values       |
| `timestamptz` | Timestamp with timezone |
| `object`      | Nested object           |
| `array`       | Array of values         |

## Updating a dataset

Update dataset metadata:

```typescript theme={null}
const updated = await api.updateDataset({
  dataset_id: 12345,
  display_name: 'Customer Events (Updated)',
  description: 'Updated description for customer events',
});

console.log('Updated:', updated.name);
```

## Managing tags

Add tags to a dataset:

```typescript theme={null}
const dataset = await api.addTags(12345, ['production', 'verified']);
console.log('Tags:', dataset.tags);
```

Remove tags from a dataset:

```typescript theme={null}
const dataset = await api.removeTags(12345, ['deprecated']);
console.log('Tags:', dataset.tags);
```

## Getting dataset statistics

Retrieve summary statistics:

```typescript theme={null}
const stats = await api.getStatistics(12345);

console.log('Statistics:', stats.records);
```

Get column-level statistics:

```typescript theme={null}
const columnStats = await api.getColumnStats(12345);

console.log('Column statistics:', columnStats);
```

## Dataset summary

Get a summary with sample data:

```typescript theme={null}
const summary = await api.getDatasetSummary(
  12345,    // dataset ID
  false,    // allRecords - set to true for all records
  100,      // perPage
  0         // offset
);

console.log('Summary:', summary);
```

## Sampling data

Sampling provides a preview of dataset records without downloading the full dataset. When you run an [interactive query](/reference/glossary#interactive-query), results are stored as a dataset in your data plane. To view these results in the UI or API, a sample job retrieves up to 1,000 rows and stores them in the control plane for quick access.

<Note>
  Samples are one of the few cases where actual data leaves your data plane. For governance implications, see [Sample Data](/concepts/architecture/sample-data).
</Note>

Samples are useful for:

* **Viewing query results** — See what your query returned without downloading everything
* **Validating data** — Check data quality before sharing with partners
* **Quick inspection** — Preview dataset contents during development

Request a sample of records:

```typescript theme={null}
// Request a sample (async operation)
const sampleRequest = await api.requestDatasetSample(12345);
console.log('Sample job ID:', sampleRequest.job_id);

// After the job completes, retrieve the sample
const sample = await api.getDatasetSample(12345, 1000);
console.log(`Sample contains ${sample.records.length} records`);
```

Delete a sample:

```typescript theme={null}
await api.deleteDatasetSample(12345);
console.log('Sample deleted');
```

## Working with files

List files in a dataset:

```typescript theme={null}
const files = await api.getDatasetFiles(
  12345,           // dataset ID
  100,             // size (number of snapshots)
  0,               // offset
  '2024-01-01',    // afterTimestamp (optional)
);

for (const snapshot of files.snapshots) {
  console.log(`Snapshot ${snapshot.id}:`, snapshot.files.length, 'files');
}
```

Get a download URL for a specific file:

```typescript theme={null}
const download = await api.getDatasetFileDownloadUrl(
  12345,           // dataset ID
  67890,           // snapshot ID
  'file-id-here'   // file ID
);

console.log('Download URL:', download.download_url);
```

## Materialized views

Refresh a materialized view:

```typescript theme={null}
const job = await api.refreshMaterializedView(12345);
console.log('Refresh job ID:', job.id);
```

Create a refresh schedule:

```typescript theme={null}
await api.createDatasetRefreshSchedule(12345, {
  cron_expression: '0 0 * * *',  // Daily at midnight
  enabled: true,
});
```

Update a refresh schedule:

```typescript theme={null}
await api.updateDatasetRefreshSchedule(12345, {
  cron_expression: '0 */6 * * *',  // Every 6 hours
  enabled: true,
});
```

Delete a refresh schedule:

```typescript theme={null}
await api.deleteDatasetRefreshSchedule(12345);
```

## Retention policy

Update the retention policy:

```typescript theme={null}
const updated = await api.updateRetentionPolicy(12345, {
  type: 'time_based',
  retention_days: 365,
});

console.log('Retention policy updated');
```

## Activating a dataset

Activate a pending dataset:

```typescript theme={null}
await api.activateDataset(12345);
console.log('Dataset activated');
```

## Deleting a dataset

<Warning>
  Deleting a dataset is permanent and cannot be undone. Ensure you have backups if needed.
</Warning>

```typescript theme={null}
await api.deleteDataset(12345);
console.log('Dataset deleted');
```

## Error handling

```typescript theme={null}
try {
  const dataset = await api.getDataset(12345);
  console.log('Dataset:', dataset.name);
} catch (error) {
  if (error.status === 404) {
    console.error('Dataset not found');
  } else if (error.status === 403) {
    console.error('Permission denied');
  } else {
    console.error('Error:', error);
  }
}
```

## Related content

<CardGroup>
  <Card title="Uploading Data" icon="upload" href="/guides/sdk/uploading-data">
    Upload files to datasets
  </Card>

  <Card title="Executing Queries" icon="terminal" href="/guides/sdk/executing-queries">
    Query dataset data
  </Card>

  <Card title="API Modules Reference" icon="puzzle-piece" href="/reference/sdks/typescript/api-modules">
    Complete API documentation
  </Card>
</CardGroup>


# Running Model Inference
Source: https://docs.narrative.io/guides/sdk/running-model-inference

Submit inference requests to hosted LLMs using the TypeScript SDK

Model Inference enables you to run LLM operations within your [data plane](/concepts/primitives/data-planes), keeping data secure while leveraging AI capabilities. This guide covers how to submit inference requests and handle results.

## Prerequisites

* SDK installed and configured (see [Authentication](/guides/sdk/authentication))
* A data plane ID where inference will run
* An API key with appropriate permissions

## Basic inference request

Submit a simple inference request with a system prompt and user message:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'system',
      content: [{ type: 'text', text: 'You are a helpful data assistant.' }]
    },
    {
      role: 'user',
      content: [{ type: 'text', text: 'What are common use cases for customer data collaboration?' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        use_cases: {
          type: 'array',
          items: { type: 'string' }
        }
      },
      required: ['use_cases']
    }
  }
});

console.log('Inference job created:', job.id);
```

## Tracking job completion

Inference jobs are asynchronous. Poll for completion using the job ID:

```typescript theme={null}
async function waitForInference(jobId: string, maxWaitMs = 60000) {
  const startTime = Date.now();
  const pollInterval = 2000;

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    if (job.state === 'completed') {
      return { success: true, result: job.result };
    }

    if (job.state === 'failed') {
      return { success: false, error: job.failures };
    }

    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`Inference job ${jobId} timed out`);
}

// Usage
const result = await waitForInference(job.id);

if (result.success) {
  console.log('Use cases:', result.result.structured_output.use_cases);
  console.log('Tokens used:', result.result.usage.total_tokens);
}
```

For more polling patterns, see [Tracking Job Status](/guides/sdk/tracking-jobs).

## Configuring inference parameters

Fine-tune the model's behavior with configuration options:

```typescript theme={null}
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'user',
      content: [{ type: 'text', text: 'Generate a creative tagline for a data platform.' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        tagline: { type: 'string' },
        tone: { type: 'string', enum: ['professional', 'playful', 'bold'] }
      },
      required: ['tagline', 'tone']
    },
    max_tokens: 200,
    temperature: 0.8,  // Higher for more creative responses
    top_p: 0.9
  }
});
```

| Parameter        | Effect                                          | Typical Values                            |
| ---------------- | ----------------------------------------------- | ----------------------------------------- |
| `temperature`    | Controls randomness. Lower = more deterministic | 0.0-0.3 for factual, 0.7-1.0 for creative |
| `max_tokens`     | Limits response length                          | 100-4000 depending on task                |
| `top_p`          | Nucleus sampling threshold                      | 0.9-1.0 for most cases                    |
| `stop_sequences` | Strings that end generation                     | `["\n\n", "END"]`                         |

## Multi-turn conversations

Include previous messages for context-aware responses:

```typescript theme={null}
const messages = [
  {
    role: 'system',
    content: [{ type: 'text', text: 'You are a data classification assistant.' }]
  },
  {
    role: 'user',
    content: [{ type: 'text', text: 'I have a dataset with email, purchase_date, and amount columns.' }]
  },
  {
    role: 'assistant',
    content: [{ type: 'text', text: 'This appears to be transactional customer data.' }]
  },
  {
    role: 'user',
    content: [{ type: 'text', text: 'What privacy considerations should I be aware of?' }]
  }
];

const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages,
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        considerations: {
          type: 'array',
          items: {
            type: 'object',
            properties: {
              issue: { type: 'string' },
              recommendation: { type: 'string' }
            },
            required: ['issue', 'recommendation']
          }
        }
      },
      required: ['considerations']
    }
  }
});
```

## Tool-use messages (agent loops)

In agent-style flows, the model emits `tool_use` content blocks requesting a tool
call and you reply with a matching `tool_result` block. Both share the same
`tool_use_id` so the model can correlate the request and response across turns.

```typescript theme={null}
const messages = [
  {
    role: 'system',
    content: [{ type: 'text', text: 'You are a dataset discovery agent.' }]
  },
  {
    role: 'user',
    content: [{ type: 'text', text: 'Find datasets containing email addresses.' }]
  },
  {
    role: 'assistant',
    content: [
      {
        type: 'tool_use',
        tool_use_id: 'tooluse_DWXPKZ50JDGib5GmShyUgJ',
        name: 'n-narrative_datasets_describe',
        arguments: { search: 'email' }
      }
    ]
  },
  {
    role: 'user',
    content: [
      {
        type: 'tool_result',
        tool_use_id: 'tooluse_DWXPKZ50JDGib5GmShyUgJ',
        content: [{ type: 'text', text: '[{"id": 123, "name": "company_emails"}]' }],
        is_error: false
      }
    ]
  }
];
```

<Note>
  The legacy `{ role, text: string }` request shape is still accepted for backwards
  compatibility — the API auto-canonicalizes it into a single `text` content block.
  Responses always emit the content-block shape. New integrations should use content
  blocks directly. See [Model Inference API reference](/reference/sdks/typescript/model-inference-api#inferencemessage).
</Note>

## Choosing a model

Select the model based on your task requirements:

| Model                         | Best For                                                   |
| ----------------------------- | ---------------------------------------------------------- |
| `anthropic.claude-haiku-4.5`  | Fast, simple tasks (classification, extraction)            |
| `anthropic.claude-sonnet-4.5` | Balanced tasks (summarization, analysis)                   |
| `anthropic.claude-sonnet-4.6` | Latest balanced model with improved reasoning              |
| `anthropic.claude-opus-4.5`   | Complex reasoning (multi-step analysis, nuanced decisions) |
| `anthropic.claude-opus-4.6`   | Latest most capable model for the most demanding tasks     |
| `openai.gpt-4.1`              | Advanced reasoning tasks                                   |
| `openai.o4-mini`              | Fast, cost-effective tasks                                 |

For detailed guidance, see [Choosing the Right Model](/guides/sdk/choosing-inference-models).

## Typing responses

Use TypeScript generics to type the structured output:

```typescript theme={null}
interface SentimentResult {
  sentiment: 'positive' | 'negative' | 'neutral';
  confidence: number;
  key_phrases: string[];
}

const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-haiku-4.5',
  messages: [
    {
      role: 'user',
      content: [{ type: 'text', text: 'Analyze the sentiment: "This product exceeded my expectations!"' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
        confidence: { type: 'number', minimum: 0, maximum: 1 },
        key_phrases: { type: 'array', items: { type: 'string' } }
      },
      required: ['sentiment', 'confidence', 'key_phrases']
    }
  }
});

// After polling for completion
const result = completedJob.result as ModelInferenceRunResult<SentimentResult>;

// Fully typed access
const sentiment: string = result.structured_output.sentiment;
const confidence: number = result.structured_output.confidence;
```

## Error handling

Handle common inference errors:

```typescript theme={null}
try {
  const job = await api.runModelInference(request);
  const result = await waitForInference(job.id);

  if (!result.success) {
    // Job failed during execution
    console.error('Inference failed:', result.error);
    return;
  }

  // Process successful result
  console.log('Output:', result.result.structured_output);

} catch (error) {
  if (error.status === 400) {
    console.error('Invalid request:', error.message);
    // Check schema, messages, or model
  } else if (error.status === 403) {
    console.error('Access denied to data plane');
  } else if (error.status === 404) {
    console.error('Data plane not found');
  } else {
    console.error('Unexpected error:', error);
  }
}
```

## Best practices

| Practice                   | Description                                               |
| -------------------------- | --------------------------------------------------------- |
| Use specific schemas       | Define precise JSON Schema to get consistent outputs      |
| Choose appropriate models  | Use smaller models for simple tasks to save cost and time |
| Set reasonable max\_tokens | Avoid unnecessarily large values that increase latency    |
| Include system prompts     | Guide model behavior with clear instructions              |
| Handle failures gracefully | Implement retries for transient errors                    |

## Related content

<CardGroup>
  <Card title="Structured Output Guide" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Deep dive into JSON Schema for inference
  </Card>

  <Card title="Choosing Models" icon="scale-balanced" href="/guides/sdk/choosing-inference-models">
    Select the right model for your task
  </Card>

  <Card title="Model Inference API" icon="code" href="/reference/sdks/typescript/model-inference-api">
    Complete API reference
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor job status and handle completion
  </Card>
</CardGroup>


# Using Structured Output
Source: https://docs.narrative.io/guides/sdk/structured-inference-output

Define JSON Schema to get predictable, typed responses from model inference

Structured output ensures that model inference returns data in a predictable format you can parse programmatically. By providing a JSON Schema, you constrain the model to return valid JSON matching your specification.

## Prerequisites

* Familiarity with [Running Model Inference](/guides/sdk/running-model-inference)
* Basic understanding of JSON Schema

## Why structured output matters

Without structured output, LLM responses are free-form text that requires parsing and error handling. With structured output:

| Without Schema          | With Schema                      |
| ----------------------- | -------------------------------- |
| Free-form text response | Guaranteed JSON structure        |
| Manual parsing required | Direct property access           |
| Inconsistent formats    | Consistent field names and types |
| Runtime type errors     | TypeScript type safety           |

## Defining a schema

The `output_format_schema` field accepts a [JSON Schema](https://json-schema.org/) object that defines your expected response structure:

```typescript theme={null}
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'user',
      content: [{ type: 'text', text: 'Classify this text: "Great product, fast shipping!"' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        sentiment: {
          type: 'string',
          enum: ['positive', 'negative', 'neutral']
        },
        confidence: {
          type: 'number',
          minimum: 0,
          maximum: 1
        }
      },
      required: ['sentiment', 'confidence']
    }
  }
});
```

The model will return:

```json theme={null}
{
  "sentiment": "positive",
  "confidence": 0.95
}
```

## Common schema patterns

### Simple object with required fields

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    title: { type: 'string' },
    description: { type: 'string' },
    priority: { type: 'integer', minimum: 1, maximum: 5 }
  },
  required: ['title', 'description']
};
```

### Arrays of items

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    categories: {
      type: 'array',
      items: { type: 'string' },
      minItems: 1,
      maxItems: 5
    }
  },
  required: ['categories']
};
```

### Nested objects

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    analysis: {
      type: 'object',
      properties: {
        summary: { type: 'string' },
        key_points: {
          type: 'array',
          items: { type: 'string' }
        }
      },
      required: ['summary', 'key_points']
    },
    metadata: {
      type: 'object',
      properties: {
        word_count: { type: 'integer' },
        language: { type: 'string' }
      }
    }
  },
  required: ['analysis']
};
```

### Enum constraints

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    category: {
      type: 'string',
      enum: ['retail', 'finance', 'healthcare', 'technology', 'other']
    },
    data_type: {
      type: 'string',
      enum: ['pii', 'aggregated', 'anonymous']
    }
  },
  required: ['category', 'data_type']
};
```

### Array of typed objects

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    entities: {
      type: 'array',
      items: {
        type: 'object',
        properties: {
          name: { type: 'string' },
          type: { type: 'string', enum: ['person', 'organization', 'location'] },
          confidence: { type: 'number' }
        },
        required: ['name', 'type']
      }
    }
  },
  required: ['entities']
};
```

## TypeScript integration

Define TypeScript interfaces that match your schema for type-safe access:

```typescript theme={null}
// Define the interface matching your schema
interface ClassificationResult {
  category: 'retail' | 'finance' | 'healthcare' | 'technology' | 'other';
  confidence: number;
  reasoning: string;
  tags: string[];
}

// Define the schema (must match the interface)
const schema = {
  type: 'object',
  properties: {
    category: {
      type: 'string',
      enum: ['retail', 'finance', 'healthcare', 'technology', 'other']
    },
    confidence: { type: 'number', minimum: 0, maximum: 1 },
    reasoning: { type: 'string' },
    tags: { type: 'array', items: { type: 'string' } }
  },
  required: ['category', 'confidence', 'reasoning', 'tags']
};

// Submit the request
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    { role: 'user', content: [{ type: 'text', text: 'Classify this dataset...' }] }
  ],
  inference_config: { output_format_schema: schema }
});

// After polling for completion, cast the result
import { ModelInferenceRunResult } from '@narrative.io/data-collaboration-sdk-ts';

const result = completedJob.result as ModelInferenceRunResult<ClassificationResult>;

// Now you have full type safety
const category: string = result.structured_output.category;  // 'retail' | 'finance' | ...
const confidence: number = result.structured_output.confidence;
const tags: string[] = result.structured_output.tags;
```

## Handling optional fields

Use `required` array to specify which fields must be present:

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    title: { type: 'string' },           // Required
    subtitle: { type: 'string' },        // Optional
    author: { type: 'string' },          // Optional
    published_date: { type: 'string' }   // Required
  },
  required: ['title', 'published_date']  // Only these are required
};

// TypeScript interface
interface ArticleInfo {
  title: string;
  subtitle?: string;
  author?: string;
  published_date: string;
}
```

## Validating responses

While the model is constrained to the schema, you may want additional runtime validation:

```typescript theme={null}
function validateResult<T>(
  result: ModelInferenceRunResult<T>,
  validate: (output: T) => boolean
): T | null {
  const output = result.structured_output;

  if (!validate(output)) {
    console.error('Validation failed for output:', output);
    return null;
  }

  return output;
}

// Usage
const validated = validateResult(result, (output: ClassificationResult) => {
  return output.confidence >= 0.5 && output.tags.length > 0;
});

if (validated) {
  console.log('Valid classification:', validated.category);
}
```

## Best practices

| Practice                        | Description                             |
| ------------------------------- | --------------------------------------- |
| Keep schemas focused            | Define only the fields you need         |
| Use enums for known values      | Constrain strings to valid options      |
| Set numeric bounds              | Use `minimum`/`maximum` for numbers     |
| Make fields required explicitly | Don't rely on defaults                  |
| Match TypeScript interfaces     | Keep schema and types in sync           |
| Add descriptions                | Help the model understand field purpose |

### Adding descriptions for clarity

```typescript theme={null}
const schema = {
  type: 'object',
  properties: {
    sentiment: {
      type: 'string',
      enum: ['positive', 'negative', 'neutral'],
      description: 'Overall sentiment of the text'
    },
    confidence: {
      type: 'number',
      minimum: 0,
      maximum: 1,
      description: 'Confidence score from 0 (uncertain) to 1 (certain)'
    },
    key_phrases: {
      type: 'array',
      items: { type: 'string' },
      description: 'Most important phrases that influenced the sentiment'
    }
  },
  required: ['sentiment', 'confidence', 'key_phrases']
};
```

## Troubleshooting

| Issue                   | Cause                             | Solution                                |
| ----------------------- | --------------------------------- | --------------------------------------- |
| Missing required fields | Schema mismatch or unclear prompt | Verify required array, improve prompt   |
| Wrong types             | Schema not enforced               | Check property types match expectations |
| Empty arrays            | Model unsure what to include      | Add `minItems` or clearer prompt        |
| Enum violations         | Value not in enum list            | Verify enum covers all possibilities    |

## Related content

<CardGroup>
  <Card title="JSON Schema Reference" icon="book" href="/reference/model-inference/json-schema-reference">
    Supported JSON Schema features
  </Card>

  <Card title="Running Model Inference" icon="play" href="/guides/sdk/running-model-inference">
    Complete inference guide
  </Card>

  <Card title="Model Inference API" icon="code" href="/reference/sdks/typescript/model-inference-api">
    API reference with all types
  </Card>

  <Card title="Structured Output Concepts" icon="lightbulb" href="/concepts/model-inference/structured-output">
    Why structured output matters
  </Card>
</CardGroup>


# Tracking Job Status
Source: https://docs.narrative.io/guides/sdk/tracking-jobs

Monitor asynchronous jobs and handle completion events

Many Narrative operations are asynchronous, returning a job ID that you can use to track progress. This guide covers how to monitor jobs and handle their completion.

<Note>
  For a detailed list of job types, see [Job Types Reference](/reference/architecture/job-types).
</Note>

<Tip>
  Need real-time notifications instead of polling? [Webhook subscriptions](/guides/webhooks/subscribing-to-notifications) push job state changes to your endpoint the moment they happen.
</Tip>

## Prerequisites

* SDK installed and configured (see [SDK Quickstart](/getting-started/sdk-quickstart))
* An API key with appropriate permissions

## Understanding jobs

Operations like file uploads, dataset refreshes, and certain queries run asynchronously. When you start one of these operations, you receive a job ID that you can poll for status updates.

### Job states

| State                  | Description                                      |
| ---------------------- | ------------------------------------------------ |
| `pending`              | Job is queued and waiting to start               |
| `running`              | Job is currently executing                       |
| `completed`            | Job finished successfully                        |
| `pending_cancellation` | Job is marked for cancellation but still running |
| `cancelled`            | Job was cancelled before completion              |
| `failed`               | Job encountered an error                         |

## Getting job status

Retrieve the status of a specific job:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

const job = await api.getJob('job-id-here');

console.log('Job ID:', job.id);
console.log('State:', job.state);
console.log('Created:', job.created_at);
console.log('Updated:', job.updated_at);
```

## Listing jobs

List all jobs, optionally filtered by state, type, or data plane:

```typescript theme={null}
// List all jobs
const allJobs = await api.getJobs();
console.log(`Total jobs: ${allJobs.total_records}`);

// List with pagination
const paginatedJobs = await api.getJobs({
  page: 1,
  per_page: 50,
});

// Filter and process
for (const job of paginatedJobs.records) {
  console.log(`${job.id}: ${job.state} - ${job.type}`);
}
```

The list response includes pagination metadata alongside `records`:

| Field           | Description                                       |
| --------------- | ------------------------------------------------- |
| `records`       | The jobs on this page                             |
| `current_page`  | The 1-indexed page number returned                |
| `total_pages`   | Total number of pages available for this query    |
| `total_records` | Total number of jobs that match the query         |
| `prev_page`     | Previous page number, or `null` on the first page |
| `next_page`     | Next page number, or `null` on the last page      |

<Note>
  `per_page` is capped at `500`. Requests with a larger value are rejected with a `400`.
</Note>

### Filtering by state, type, and tag

The `state`, `type`, and `tag` query parameters are repeatable — pass each one multiple times to match any of several values. This is useful for building active-vs-finished views or scoping to a job family.

```typescript theme={null}
// Only jobs that are currently active
const active = await api.getJobs({
  state: ['pending', 'running', 'pending_cancellation'],
});

// All jobs for a given dataset on a specific data plane
const datasetJobs = await api.getJobs({
  dataset_id: 'dataset-id',
  data_plane_id: 'dp_your_data_plane_id',
});
```

<Note>
  Job responses always include a `data_plane_id`. Jobs that target the default Narrative-managed data plane report its id explicitly rather than returning `null`.
</Note>

## Polling for completion

A common pattern is to poll until a job completes:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function waitForJob(jobId: string, maxWaitMs = 300000) {
  const startTime = Date.now();
  const pollInterval = 5000; // 5 seconds

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    console.log(`Job ${jobId}: ${job.state}`);

    if (job.state === 'completed') {
      return { success: true, job };
    }

    if (job.state === 'failed') {
      return { success: false, job, error: job.failures };
    }

    if (job.state === 'cancelled') {
      return { success: false, job, error: 'Job was cancelled' };
    }

    // Wait before next poll
    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`Job ${jobId} did not complete within ${maxWaitMs}ms`);
}

// Usage
const result = await waitForJob('job-id-here');
if (result.success) {
  console.log('Job completed successfully');
} else {
  console.error('Job failed:', result.error);
}
```

## Exponential backoff

For long-running jobs, use exponential backoff to reduce API calls:

```typescript theme={null}
async function waitForJobWithBackoff(jobId: string, maxWaitMs = 600000) {
  const startTime = Date.now();
  let pollInterval = 1000; // Start with 1 second
  const maxInterval = 30000; // Cap at 30 seconds

  while (Date.now() - startTime < maxWaitMs) {
    const job = await api.getJob(jobId);

    console.log(`Job ${jobId}: ${job.state} (next check in ${pollInterval}ms)`);

    if (job.state === 'completed') {
      return { success: true, job };
    }

    if (job.state === 'failed' || job.state === 'cancelled') {
      return { success: false, job };
    }

    // Wait with exponential backoff
    await new Promise(resolve => setTimeout(resolve, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, maxInterval);
  }

  throw new Error(`Job ${jobId} timed out`);
}
```

## Handling job results

Different job types return different results:

### NQL query jobs

```typescript theme={null}
const queryResult = await api.getNqlByJobId(jobId);

if (queryResult.state === 'completed') {
  console.log('Rows returned:', queryResult.result.rows);
  console.log('Query cost:', queryResult.result.cost);
}
```

### Dataset refresh jobs

```typescript theme={null}
const job = await api.getJob(refreshJobId);

if (job.state === 'completed') {
  console.log('Materialized view refreshed');
  // Fetch updated dataset statistics
  const stats = await api.getStatistics(datasetId);
}
```

### Model inference jobs

```typescript theme={null}
import type { ModelInferenceRunResult } from '@narrative.io/data-collaboration-sdk-ts';

const job = await api.getJob(inferenceJobId);

if (job.state === 'completed' && job.result) {
  const result = job.result as ModelInferenceRunResult;
  console.log('Tokens used:', result.usage.total_tokens);
  console.log('Output:', result.structured_output);
}
```

For inference-specific patterns, see [Running Model Inference](/guides/sdk/running-model-inference).

## Error handling

Handle job failures gracefully:

```typescript theme={null}
async function executeWithRetry(operation: () => Promise<string>, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const jobId = await operation();
      const result = await waitForJob(jobId);

      if (result.success) {
        return result.job;
      }

      console.error(`Attempt ${attempt} failed:`, result.error);

      if (attempt < maxRetries) {
        const backoff = Math.pow(2, attempt) * 1000;
        console.log(`Retrying in ${backoff}ms...`);
        await new Promise(resolve => setTimeout(resolve, backoff));
      }
    } catch (error) {
      console.error(`Attempt ${attempt} error:`, error);

      if (attempt === maxRetries) {
        throw error;
      }
    }
  }

  throw new Error('All retry attempts failed');
}

// Usage
const job = await executeWithRetry(async () => {
  const result = await api.refreshMaterializedView(datasetId);
  return result.id;
});
```

## Batch job monitoring

Monitor multiple jobs concurrently:

```typescript theme={null}
async function waitForAllJobs(jobIds: string[]) {
  const results = await Promise.all(
    jobIds.map(async (jobId) => {
      try {
        const result = await waitForJob(jobId);
        return { jobId, ...result };
      } catch (error) {
        return { jobId, success: false, error };
      }
    })
  );

  const succeeded = results.filter(r => r.success);
  const failed = results.filter(r => !r.success);

  console.log(`Completed: ${succeeded.length}/${results.length}`);
  console.log(`Failed: ${failed.length}/${results.length}`);

  return { results, succeeded, failed };
}

// Usage
const jobIds = ['job-1', 'job-2', 'job-3'];
const { succeeded, failed } = await waitForAllJobs(jobIds);
```

## Best practices

| Practice                | Description                                                       |
| ----------------------- | ----------------------------------------------------------------- |
| Use exponential backoff | Reduce API calls for long-running jobs                            |
| Set reasonable timeouts | Prevent indefinite waiting                                        |
| Log progress            | Track job state changes for debugging                             |
| Handle all states       | Account for all terminal states: completed, failed, and cancelled |
| Implement retries       | Retry failed operations when appropriate                          |

## Troubleshooting

| Issue                 | Cause                                 | Solution                                 |
| --------------------- | ------------------------------------- | ---------------------------------------- |
| Job stays pending     | Queue backlog or resource constraints | Wait longer or contact support           |
| Job fails immediately | Invalid input or permissions          | Check error details and fix input        |
| Timeout waiting       | Job taking longer than expected       | Increase timeout or check job complexity |
| 404 on job ID         | Job expired or invalid ID             | Verify job ID and check retention        |

## Related content

<CardGroup>
  <Card title="Job Types" icon="list" href="/reference/architecture/job-types">
    Reference for all job types
  </Card>

  <Card title="Error Handling" icon="triangle-exclamation" href="/guides/sdk/error-handling">
    Handle SDK errors gracefully
  </Card>

  <Card title="Uploading Data" icon="upload" href="/guides/sdk/uploading-data">
    Upload files and track ingestion
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets">
    Dataset operations that create jobs
  </Card>
</CardGroup>


# Uploading Data
Source: https://docs.narrative.io/guides/sdk/uploading-data

Upload files to Narrative datasets using the TypeScript SDK

The Narrative SDK provides methods to upload data files to your datasets programmatically. This guide covers the upload workflow and best practices.

## Prerequisites

* SDK installed and configured (see [SDK Quickstart](/getting-started/sdk-quickstart))
* A target dataset (see [Managing Datasets](/guides/sdk/managing-datasets))
* An API key with write permissions
* Data file in a supported format (CSV, JSON, Parquet)

## Supported formats

| Format  | Extension  | Notes                                          |
| ------- | ---------- | ---------------------------------------------- |
| CSV     | `.csv`     | Comma-separated values with headers            |
| JSON    | `.json`    | JSON Lines (one object per line) or JSON array |
| Parquet | `.parquet` | Apache Parquet columnar format                 |

## Upload workflow

The upload process involves these steps:

1. **Get an upload URL** - Request a pre-signed URL for uploading
2. **Upload the file** - Send the file to the pre-signed URL
3. **Confirm the upload** - Notify Narrative that the upload is complete
4. **Monitor ingestion** - Track the ingestion job status

## Basic upload

Here's a complete example using Node.js:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadFile(datasetId: number, filePath: string) {
  const fileName = path.basename(filePath);
  const fileContent = fs.readFileSync(filePath);

  // Step 1: Get upload URL
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  console.log('Got upload URL');

  // Step 2: Upload file to S3
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  console.log('File uploaded to S3');

  // Step 3: Confirm upload
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
  console.log('Upload confirmed');

  return uploadUrl.upload_id;
}

// Usage
const uploadId = await uploadFile(12345, './data/customer_events.csv');
console.log('Upload ID:', uploadId);
```

## Ingesting from a URL

If your data is already accessible via URL, you can ingest directly:

```typescript theme={null}
await api.ingestDatasetFile(
  { source_file: 'https://example.com/data/events.csv' },
  12345  // dataset ID
);

console.log('Ingestion started');
```

## Monitoring upload status

After confirming an upload, monitor the ingestion job:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function waitForIngestion(jobId: string) {
  while (true) {
    const job = await api.getJob(jobId);

    console.log(`Status: ${job.state}`);

    if (job.state === 'completed') {
      console.log('Ingestion completed successfully');
      return job;
    }

    if (job.state === 'failed') {
      throw new Error(`Ingestion failed: ${JSON.stringify(job.failures)}`);
    }

    // Wait before checking again
    await new Promise(resolve => setTimeout(resolve, 5000));
  }
}
```

## Uploading multiple files

Upload multiple files to the same dataset:

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadMultipleFiles(datasetId: number, filePaths: string[]) {
  const results = [];

  for (const filePath of filePaths) {
    const fileName = path.basename(filePath);
    const fileContent = fs.readFileSync(filePath);

    console.log(`Uploading ${fileName}...`);

    const uploadUrl = await api.getUploadUrl(datasetId, fileName);
    await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
    await api.confirmUpload(datasetId, uploadUrl.upload_id);

    results.push({
      file: fileName,
      uploadId: uploadUrl.upload_id,
    });

    console.log(`Completed ${fileName}`);
  }

  return results;
}

// Usage
const files = [
  './data/events_2024_01.csv',
  './data/events_2024_02.csv',
  './data/events_2024_03.csv',
];

const results = await uploadMultipleFiles(12345, files);
console.log('Uploaded:', results);
```

## Data preparation

Before uploading, ensure your data meets the dataset schema requirements.

### CSV files

* Include a header row matching schema field names
* Use consistent formatting for dates and timestamps
* Handle special characters properly (escape or quote)

```csv theme={null}
customer_id,event_type,event_timestamp,event_value
cust_123,purchase,2024-01-15T10:30:00Z,99.99
cust_456,view,2024-01-15T11:00:00Z,0
```

### JSON files

Use JSON Lines format (one object per line):

```json theme={null}
{"customer_id": "cust_123", "event_type": "purchase", "event_timestamp": "2024-01-15T10:30:00Z", "event_value": 99.99}
{"customer_id": "cust_456", "event_type": "view", "event_timestamp": "2024-01-15T11:00:00Z", "event_value": 0}
```

### Hashing PII

If your data contains PII, hash it before uploading:

```typescript theme={null}
import crypto from 'crypto';

function hashEmail(email: string): string {
  const normalized = email.toLowerCase().trim();
  return crypto.createHash('sha256').update(normalized).digest('hex');
}

// Before uploading, transform your data
const hashedData = rawData.map(row => ({
  ...row,
  sha256_hashed_email: hashEmail(row.email),
}));
```

<Tip>
  For detailed hashing instructions, see [Hashing PII for Upload](/guides/ingestion/hashing-pii).
</Tip>

## Error handling

```typescript theme={null}
try {
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
} catch (error) {
  if (error.status === 400) {
    console.error('Invalid request:', error.message);
  } else if (error.status === 403) {
    console.error('Permission denied - check API key permissions');
  } else if (error.status === 404) {
    console.error('Dataset not found');
  } else {
    console.error('Upload failed:', error);
  }
}
```

## Best practices

| Practice                | Description                                      |
| ----------------------- | ------------------------------------------------ |
| Validate before upload  | Check file format and schema match               |
| Hash PII                | Never upload unhashed personal data              |
| Use appropriate formats | Parquet for large datasets, CSV/JSON for smaller |
| Monitor jobs            | Track ingestion status to catch failures         |
| Handle errors           | Implement retry logic for transient failures     |

## Troubleshooting

| Issue             | Cause                                   | Solution                              |
| ----------------- | --------------------------------------- | ------------------------------------- |
| Schema mismatch   | File columns don't match dataset schema | Verify column names and types         |
| Upload timeout    | File too large or slow connection       | Try smaller files or increase timeout |
| Permission denied | API key lacks write access              | Check API key permissions             |
| Invalid format    | File format not recognized              | Verify file extension and content     |

## Related content

<CardGroup>
  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare PII for upload
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor upload progress
  </Card>

  <Card title="Managing Datasets" icon="database" href="/guides/sdk/managing-datasets">
    Dataset operations
  </Card>
</CardGroup>


# Secret Sharing
Source: https://docs.narrative.io/guides/tools/secret-sharing

Send sensitive information through one-time-use encrypted links

Secret Sharing lets you send sensitive information—like API keys, passwords, or credentials—to another person through a one-time-use encrypted link. Once the recipient opens the link and views the secret, it is permanently deleted. No one, including you, can view it again.

<iframe title="Share a Secret Securely with a One-Time Link" />

<Note>
  Secret Sharing is a **Tools** feature. Tools features are experimental and may change or be removed in future releases.
</Note>

## How to share a secret

<Steps>
  <Step title="Open Secret Sharing">
    Navigate to **[Tools > Secrets Sharing](https://app.narrative.io/platform/tools/secrets-sharing)** in the platform navigation.
  </Step>

  <Step title="Enter your secret">
    Type or paste your secret into the input field.
  </Step>

  <Step title="Share the secret">
    Click **Share**. A unique retrieval URL is automatically copied to your clipboard.
  </Step>

  <Step title="Send the link">
    Send the retrieval URL to your recipient through whatever channel you prefer (Slack, email, etc.).
  </Step>
</Steps>

<Warning>
  **Do not open the retrieval link yourself.** The secret is destroyed the moment anyone views it—including you.
</Warning>

***

## How to retrieve a secret

<iframe title="Copy a Retrieved Secret for One-Time Use" />

<Steps>
  <Step title="Open the retrieval link">
    Open the retrieval URL you received from the sender.
  </Step>

  <Step title="Copy the secret">
    Your secret is displayed in a read-only field. Click **Copy** to copy it to your clipboard.
  </Step>

  <Step title="Store it safely">
    Save the secret somewhere secure. You will not be able to access this link again.
  </Step>
</Steps>

If the link has already been used, the secret will no longer be available. Ask the sender to create a new link.

***

## How it works

When you share a secret, the following happens behind the scenes:

1. Your secret is sent to the server over HTTPS
2. The server encrypts your secret using **AES-256-GCM** with a key that only the server knows
3. Each secret is encrypted with its own unique random salt and initialization vector
4. The encrypted secret is stored in a database—the plaintext is never saved
5. A unique ID is generated and returned to you as part of the retrieval URL

When someone opens the retrieval link:

1. The server looks up the encrypted secret by its unique ID
2. The secret is **immediately deleted** from the database
3. The server decrypts the secret and returns it to the recipient
4. The secret no longer exists anywhere on the server

***

## Why is this secure?

* **Encrypted at rest** — Your secret is encrypted before it is stored. Even if someone accessed the database directly, they would only see unreadable ciphertext.
* **One-time access** — The secret is permanently deleted from the database the instant it is viewed. There is no way to retrieve it a second time.
* **Unique per secret** — Each secret is encrypted with its own random salt and initialization vector, so even identical secrets produce completely different ciphertext.
* **Server-side encryption** — Encryption and decryption happen on the server. The encryption key never leaves the server environment and is not exposed to the browser.
* **No accounts required** — Anyone with the link can retrieve the secret. No login is needed, which means you can share secrets with people who don't have a Narrative account.

***

## Limitations

<Warning>
  Keep these limitations in mind when using Secret Sharing:

  * **No expiration** — If a secret is never retrieved, it remains stored indefinitely. Share links only with people you expect to use them.
  * **No access log** — There is no way to confirm who retrieved the secret or when. You only know that once the link stops working, someone has viewed it.
  * **Single viewer** — Only one person can ever view the secret. If you need to share the same secret with multiple people, create a separate link for each recipient.
  * **Designed for short secrets** — The feature is intended for keys and passwords, not large documents.
</Warning>

***

## Common use cases

* Sharing API keys or tokens with a teammate
* Sending a database password to a contractor
* Passing credentials to a support engineer during troubleshooting
* Any time you need to send sensitive text and don't want it sitting in a chat history or email thread

***

## Related content

<CardGroup>
  <Card title="Security Model" icon="shield" href="/concepts/architecture/security-model">
    How Narrative protects your data
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys for the Narrative API
  </Card>
</CardGroup>


# Subscribing to Notifications
Source: https://docs.narrative.io/guides/webhooks/subscribing-to-notifications

Create and manage webhook subscriptions to receive push notifications for job state changes

Webhook subscriptions let you receive HTTP POST notifications when jobs change state, eliminating the need to poll the jobs API. This guide walks through creating, receiving, and managing webhook subscriptions.

<Info>
  For background on how webhooks work and when to use them, see [Webhooks](/concepts/webhooks/overview). For complete API details, see the [Webhooks API Reference](/api-reference/webhooks/list-webhook-subscriptions).
</Info>

## Prerequisites

* An API token with Webhooks `read` and `write` [permissions](/reference/security/permissions)
* An HTTPS endpoint ready to receive POST requests
* Familiarity with [job types](/reference/architecture/job-types) and [job states](/guides/sdk/tracking-jobs#job-states)

## What you'll learn

* How to create a job webhook subscription with filters
* How to receive and verify webhook events
* How to list, inspect, and archive subscriptions

***

## Creating a job webhook subscription

Use the `POST /webhooks` endpoint to create a subscription. Set the `type` to `webhook_subscription_jobs` and provide your endpoint URL.

<Steps>
  ### Choose your filters

  Decide which job events you want to receive. You can filter by job IDs, job types, job states, or any combination.

  | Scenario                         | Filters                                                    |
  | -------------------------------- | ---------------------------------------------------------- |
  | All completed materialized views | `job_types: ["materialize-view"]`, `states: ["completed"]` |
  | Track a specific job             | `job_ids: ["<job-id>"]`                                    |
  | All failed jobs                  | `states: ["failed"]`                                       |
  | All state changes (no filter)    | Omit `job_ids`, `job_types`, and `states`                  |

  ### Create the subscription

  ```bash theme={null}
  curl -X POST https://api.narrative.io/webhooks \
    -H "Authorization: Bearer $NIO_API_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "type": "webhook_subscription_jobs",
      "url": "https://your-app.example.com/webhooks/narrative",
      "name": "completed_matviews",
      "job_types": ["materialize-view"],
      "states": ["completed", "failed"]
    }'
  ```

  ### Store the secret

  The response includes a `secret` field — a UUID used to verify that incoming requests are from Narrative. Store this value securely.

  ```json theme={null}
  {
    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "secret": "8b6dba8c-9a8b-4a3a-8469-9dc7a33c3e17",
    "status": "active",
    "companyId": 42,
    "jobTypes": ["materialize-view"],
    "jobStates": ["completed", "failed"],
    "url": "https://your-app.example.com/webhooks/narrative",
    "createdAt": "2025-04-29T16:12:45.123456",
    "updatedAt": "2025-04-29T16:12:45.123456"
  }
  ```
</Steps>

***

## Receiving events

When a job matches your subscription's filters, Narrative sends an HTTP POST to your URL with a JSON payload.

### Example event payload

```json theme={null}
{
  "context": {
    "companyId": "42"
  },
  "eventTimestamp": "2025-04-29T16:15:30.456789",
  "eventType": {
    "value": "job_state_change"
  },
  "idempotencyKey": {
    "value": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  },
  "payload": {
    "value": {
      "jobId": "d35a12c1-7a2f-46b9-8d60-9e0a2a84c205",
      "jobType": "materialize-view",
      "state": "completed"
    }
  }
}
```

Your endpoint should return a `2xx` status code to acknowledge receipt.

***

## Handling idempotency

Each event includes an `idempotencyKey` — a unique identifier for that specific delivery. If Narrative delivers the same event more than once (for example, due to a network retry), the idempotency key remains the same.

Use the idempotency key to detect duplicates:

1. When you receive an event, check if you've already processed that `idempotencyKey`
2. If yes, return `200 OK` without processing again
3. If no, process the event and store the key

***

## Managing subscriptions

### List all subscriptions

```bash theme={null}
curl https://api.narrative.io/webhooks \
  -H "Authorization: Bearer $NIO_API_TOKEN"
```

Returns an array of all webhook subscriptions for your company, including both `active` and `archived` subscriptions.

### Get a specific subscription

```bash theme={null}
curl https://api.narrative.io/webhooks/{webhook_subscription_id} \
  -H "Authorization: Bearer $NIO_API_TOKEN"
```

### Archive a subscription

Archiving stops event delivery but retains the subscription record.

```bash theme={null}
curl -X DELETE https://api.narrative.io/webhooks/{webhook_subscription_id} \
  -H "Authorization: Bearer $NIO_API_TOKEN"
```

<Warning>
  Archiving is permanent — there is no way to reactivate an archived subscription. Create a new subscription if you need to resume notifications.
</Warning>

***

## Best practices

| Practice                  | Why                                                                                                                     |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Respond quickly**       | Return a `2xx` response immediately, then process the event asynchronously. Slow responses may cause delivery timeouts. |
| **Store the secret**      | Save the `secret` from the creation response. You cannot retrieve it later.                                             |
| **Implement idempotency** | Use the `idempotencyKey` to deduplicate events in case of retries.                                                      |
| **Use HTTPS**             | Webhook URLs must use HTTPS to protect payloads in transit.                                                             |
| **Filter narrowly**       | Subscribe only to the job types and states you need. This reduces noise and processing overhead.                        |

***

## Related content

<CardGroup>
  <Card title="Webhooks" icon="bell" href="/concepts/webhooks/overview">
    How webhooks work and when to use them
  </Card>

  <Card title="Webhook Event Reference" icon="list-check" href="/reference/webhooks/event-reference">
    Complete field reference for subscription requests and responses
  </Card>

  <Card title="Tracking Job Status" icon="spinner" href="/guides/sdk/tracking-jobs">
    Poll-based alternative for monitoring job progress
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API tokens for webhook access
  </Card>
</CardGroup>


# Automating Multi-Step Pipelines
Source: https://docs.narrative.io/guides/workflows/workflow-orchestration

Define and run multi-step data workflows that execute automatically based on task dependencies

If you're running multi-step data operations manually—creating datasets, refreshing views, then building derived tables—workflows let you define the entire pipeline once and execute it automatically. Tasks run in sequence, each waiting for the previous one to finish before starting.

<Info>
  For a deeper understanding of how workflows relate to the jobs system, see [Workflow Orchestration](/concepts/workflows/workflow-orchestration). For complete API details, see the [Workflows API Reference](/api-reference/workflows/list-workflows).
</Info>

## Prerequisites

* An API token with workflow permissions (`read` and `write` access to `workflows`)
* A [data plane](/reference/glossary) configured for your account
* Familiarity with [NQL syntax](/nql/general/syntax) and [materialized views](/concepts/nql/materialized-views)

## What you'll learn

* How to view and manage workflows in the UI
* How to define a workflow specification in YAML
* How to create and trigger workflows via the API
* How to chain multiple tasks with dependencies
* How to pass data between tasks using export and variable expressions
* How to schedule workflows for automatic execution
* How to monitor workflow runs and troubleshoot failures

***

## Managing workflows in the UI

You can view and manage your workflows from **Settings > Workflows** in the platform sidebar. The workflows page displays all your workflows in a table with the following information:

* **Name** — the workflow's name (links to the detail page)
* **Status** — whether the workflow is Active or Archived
* **Data Plane** — which data plane the workflow runs on
* **Created** — when the workflow was created

<Info>
  The workflows list is scoped to the data plane currently selected in the [context selector](/guides/platform/using-the-context-selector). Switch data planes from the context selector to see workflows that run on a different data plane. Status filter chip counts reflect only the workflows visible for the selected data plane.
</Info>

### Workflow actions

From the workflows table or the detail page, you can perform these actions on each workflow:

* **Trigger** — manually start a new run of an active workflow
* **Archive** — deactivate a workflow so it no longer runs on its schedule
* **Copy ID** — copy the workflow ID to your clipboard

### Viewing workflow details

Click any workflow name to open its detail page, which has two tabs:

* **Definition** — displays the full YAML specification with syntax highlighting
* **Runs** — shows the run history with status, start time, and duration for each execution

<Note>
  Workflow creation and editing are currently available through the API only. Use the UI to monitor, trigger, and archive workflows you've created via the API.
</Note>

***

## Defining a workflow

Workflows use the [Serverless Workflow DSL](https://github.com/serverlessworkflow/specification/blob/main/dsl-reference.md) in YAML format. A workflow is a list of tasks that execute sequentially in the order you define them.

### Minimal example

Here's the simplest possible workflow—a single task that creates a materialized view:

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: analytics
  name: daily-user-summary
  version: '1.0.0'
do:
  - createUserSummary:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: |
          CREATE MATERIALIZED VIEW user_event_summary AS
          SELECT
            user_id,
            COUNT(1) as event_count
          FROM events
          GROUP BY user_id
```

Every workflow has two sections:

| Section    | Purpose                                            |
| ---------- | -------------------------------------------------- |
| `document` | Metadata—DSL version, namespace, name, and version |
| `do`       | Ordered list of tasks to execute                   |

Each task has a unique name (the key), a `call` specifying which task to run, and `with` providing the task's parameters.

### Supported tasks

Workflows support these operations:

| Task                                   | Description                                                                                                               |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `CreateMaterializedViewIfNotExists`    | Creates a new materialized view from an NQL query                                                                         |
| `RefreshMaterializedView`              | Refreshes an existing materialized view with the latest data                                                              |
| `ExecuteDml`                           | Executes a DML statement ([INSERT](/nql/commands/insert), [UPDATE](/nql/commands/update), [DELETE](/nql/commands/delete)) |
| `RunModelInference`                    | Runs a model inference job                                                                                                |
| `CreateRosettaStoneMappingsIfNotExist` | Creates Rosetta Stone attribute mappings for a dataset                                                                    |

<Note>
  Direct `ExecuteNQL` is not supported. To execute a SELECT query and persist results, use `CreateMaterializedViewIfNotExists`.
</Note>

***

## Creating and running a workflow

### Step 1: Create the workflow

Send your YAML specification to the API:

```bash theme={null}
curl -X POST https://api.narrative.io/workflows \
  -H "Authorization: Bearer $NIO_API_TOKEN" \
  -H "Content-Type: application/x-yaml" \
  -d @my-workflow.yaml
```

The response includes a `workflow_id` that you'll use for all subsequent operations.

### Step 2: Trigger a run

Execute the workflow on a specific data plane:

```bash theme={null}
curl -X POST https://api.narrative.io/workflows/{workflow_id}/trigger \
  -H "Authorization: Bearer $NIO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"data_plane_id": 123}'
```

Each trigger creates a new **run** with its own `run_id`. The workflow's tasks execute sequentially—each task waits for the previous one to complete before starting.

### Step 3: Check the status

Monitor the run to see how it's progressing:

```bash theme={null}
curl https://api.narrative.io/workflows/{workflow_id}/runs \
  -H "Authorization: Bearer $NIO_API_TOKEN"
```

<Tip>
  Each task in a workflow creates its own job in the underlying jobs system. Jobs created by workflows are automatically tagged with `workflow_enqueued`, making them easy to filter when checking job history. On the **Jobs** page, select **Workflow Enqueued** in the Status filter or paste a `workflow_id` or `workflow_run_id` into the search bar to scope the table to a single workflow or run.
</Tip>

***

## Chaining dependent tasks

The real power of workflows is chaining operations that depend on each other. Tasks run in order, so later tasks can reference datasets created by earlier ones.

### Example: Create a source dataset, then derive from it

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: analytics
  name: user-comparison-workflow
  version: '1.0.0'
do:
  - createSourceData:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: |
          CREATE MATERIALIZED VIEW active_users_source AS
          SELECT
            id,
            name,
            email
          FROM users
          WHERE active = true
  - createDerivedView:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: |
          CREATE MATERIALIZED VIEW active_users_with_names AS
          SELECT
            id,
            name
          FROM company_data.active_users_source
          WHERE name IS NOT NULL
```

The second task references `company_data.active_users_source`—the dataset created by the first task. The workflow ensures the first task completes before the second one starts.

<Warning>
  When referencing datasets created by earlier tasks, use the fully qualified path: `company_data.dataset_name`.
</Warning>

### Example: Refresh multiple views in sequence

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: etl
  name: daily-refresh
  version: '1.0.0'
do:
  - refreshSource:
      call: RefreshMaterializedView
      with:
        datasetName: raw_events
  - refreshAggregates:
      call: RefreshMaterializedView
      with:
        datasetName: event_aggregates
```

This ensures `raw_events` is refreshed before `event_aggregates`, which may depend on it.

***

## Passing data between tasks

Beyond referencing datasets by name, workflows can pass structured data between tasks using **export** and **variable expressions**. This is useful when a downstream task needs metadata from an upstream task — like a dataset ID, row count, or status value.

<Info>
  For the complete syntax reference, see [Task output](/reference/workflows/specification-syntax#task-output), [Export](/reference/workflows/specification-syntax#export), and [Variable expressions](/reference/workflows/specification-syntax#variable-expressions) in the specification reference.
</Info>

### Exporting task output to workflow context

Every task produces a JSON output object after execution. Use `export.as` with a jq expression to capture values into the workflow context (`$context`):

```yaml theme={null}
- createView:
    call: CreateMaterializedViewIfNotExists
    with:
      nql: "CREATE MATERIALIZED VIEW active_users AS SELECT user_id, email FROM company_data.users WHERE is_active = true"
    export:
      as: '$context + { datasetId: .datasetId }'
```

After this task runs, `$context.datasetId` contains the ID of the created dataset.

### Using variable expressions in parameters

Use `${…}` in parameter values to inject data from the previous task's output (`.`) or the workflow context (`$context`). Use jq string interpolation `\(expr)` to embed values inside strings:

```yaml theme={null}
- logResult:
    call: ExecuteDml
    with:
      nql: |
        ${"INSERT INTO company_data.pipeline_log (dataset_id, status) VALUES (\($context.datasetId), 'created')"}
```

A pure `${…}` expression (as the entire parameter value) preserves the JSON type. When used inside a string or with `${"..."}`, the result is always a string.

### Complete example

This workflow creates a materialized view, captures its dataset ID in the workflow context, then logs the operation using both direct output access and context:

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: etl
  name: create-and-log
  version: '1.0.0'
do:
  - createView:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: "CREATE MATERIALIZED VIEW active_users AS SELECT user_id, email FROM company_data.users WHERE is_active = true"
      export:
        as: '$context + { datasetId: .datasetId }'
  - logCreatedDataset:
      call: ExecuteDml
      with:
        nql: |
          ${"INSERT INTO company_data.pipeline_log (dataset_id, status) VALUES (\(.datasetId), 'created')"}
  - logFromContext:
      call: ExecuteDml
      with:
        nql: |
          ${"INSERT INTO company_data.pipeline_log (dataset_id, status) VALUES (\($context.datasetId), 'created')"}
```

Here's what happens at each step:

1. **`createView`** creates the materialized view. Its output includes `datasetId`. The `export.as` expression merges this into `$context` as `datasetId`.
2. **`logCreatedDataset`** uses `.datasetId` — the direct output of the previous task — in a variable expression to insert a log record.
3. **`logFromContext`** uses `$context.datasetId` to access the same value via the accumulated workflow context. This is useful when the value was exported several tasks ago and is no longer in the immediately preceding output.

***

## Scheduling workflows

Instead of triggering runs manually, you can set workflows to run on a schedule using cron expressions.

Add a `schedule` block to your workflow specification:

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: etl
  name: hourly-refresh
  version: '1.0.0'
schedule:
  cron: '0 * * * *'
do:
  - refreshData:
      call: RefreshMaterializedView
      with:
        datasetName: hourly_metrics
```

### Common schedules

| Expression  | Schedule           |
| ----------- | ------------------ |
| `0 * * * *` | Every hour         |
| `0 0 * * *` | Daily at midnight  |
| `0 0 * * 0` | Weekly on Sunday   |
| `0 0 1 * *` | Monthly on the 1st |

<Tip>
  Run your workflow manually at least once before adding a schedule. This lets you verify it works correctly without waiting for the next scheduled run.
</Tip>

***

## Handling errors

Workflows use fail-fast behavior: if any task fails, execution stops immediately and remaining tasks are skipped. The workflow run status becomes `failed`.

To investigate a failure:

1. Check the workflow run status via the [runs endpoint](/api-reference/workflows/list-workflow-runs)
2. Identify which task failed from the run details
3. Query the individual job using its `job_id` for detailed error information

### Common errors

| Error                            | Cause                                  | Solution                                                                           |
| -------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------- |
| `Unsupported task`               | Using a task not in the supported list | Use only the [supported tasks](/reference/workflows/tasks)                         |
| `Missing required field`         | Incomplete YAML specification          | Ensure `document`, `dsl`, `namespace`, `name`, `version`, and `do` are all present |
| `Invalid workflow specification` | Malformed YAML                         | Validate your YAML syntax before submitting                                        |

***

## Best practices

* **Name workflows clearly.** Use descriptive names that explain what the workflow does (e.g., `daily-audience-refresh` rather than `workflow-1`).
* **Version your workflows.** Increment the version when you change the specification to maintain a clear history.
* **Start with a single task.** Build and test one task at a time before assembling multi-step pipelines.
* **Test before scheduling.** Run workflows manually to confirm they work before setting up automatic execution.

***

## Limitations

* **Sequential only** — tasks execute one at a time in order; parallel execution is not supported
* **No conditional logic** — all tasks run unconditionally; there is no if/else branching
* **No loops** — iterative operations are not supported
* **No automatic retries** — failed tasks are not retried automatically

***

## Related content

<CardGroup>
  <Card title="Workflow Orchestration" icon="gears" href="/concepts/workflows/workflow-orchestration">
    How workflows work and why they're designed this way
  </Card>

  <Card title="Task Reference" icon="rectangle-terminal" href="/reference/workflows/tasks">
    Complete reference for all supported workflow tasks and parameters
  </Card>

  <Card title="Workflows API Reference" icon="code" href="/api-reference/workflows/list-workflows">
    Complete API documentation for workflow endpoints
  </Card>

  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    How to create and manage materialized views used in workflow tasks
  </Card>
</CardGroup>


# Materialized View Syntax
Source: https://docs.narrative.io/nql/commands/create-materialized-view

Complete reference for CREATE MATERIALIZED VIEW statement

The `CREATE MATERIALIZED VIEW` statement creates a stored dataset from a query result. Unlike regular queries that execute each time, materialized views persist their results for faster access.

## Syntax

```sql theme={null}
CREATE MATERIALIZED VIEW "view_name"
[REFRESH_SCHEDULE = <schedule>]
[DISPLAY_NAME = '<display_name>']
[DESCRIPTION = '<description>']
[EXPIRE = <retention_policy>]
[TAGS = ( '<tag1>', '<tag2>', ... )]
[WRITE_MODE = '<mode>']
[EXTENDED_STATS = '<level>']
[PARTITIONED_BY <field> <transform>[, <field2> <transform2>]]
AS
<select_query>
[MERGE ON <condition>
WHEN MATCHED THEN UPDATE SET <column> = source.<column>, ...
WHEN NOT MATCHED THEN INSERT (<columns>) VALUES (<values>)]
```

## Options summary

| Option             | Type                    | Required | Description                              |
| ------------------ | ----------------------- | -------- | ---------------------------------------- |
| `REFRESH_SCHEDULE` | enum or CRON            | No       | When to automatically refresh the view   |
| `DISPLAY_NAME`     | string                  | No       | Human-readable name for the UI           |
| `DESCRIPTION`      | string                  | No       | Documentation text                       |
| `EXPIRE`           | ISO 8601 period or enum | No       | Data retention policy                    |
| `TAGS`             | array                   | No       | Labels for organization and filtering    |
| `WRITE_MODE`       | enum                    | No       | How refresh operations write data        |
| `EXTENDED_STATS`   | enum                    | No       | Whether to compute additional statistics |
| `PARTITIONED_BY`   | field + transform       | No       | How to partition stored data             |
| `MERGE ON`         | condition + clauses     | No       | Upsert logic for incremental updates     |

***

## REFRESH\_SCHEDULE

Defines when the materialized view automatically refreshes with new data.

**Syntax:**

```sql theme={null}
REFRESH_SCHEDULE = '@daily'
REFRESH_SCHEDULE = '0 2 * * *'
```

**Values:**

| Value           | Description                                |
| --------------- | ------------------------------------------ |
| `@hourly`       | Refresh every hour                         |
| `@daily`        | Refresh once per day                       |
| `@weekly`       | Refresh once per week                      |
| `@monthly`      | Refresh once per month                     |
| `@once`         | Refresh only once (no recurring schedule)  |
| CRON expression | Custom schedule using standard CRON syntax |

**CRON format:**

```
┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-6, Sunday=0)
│ │ │ │ │
* * * * *
```

**Example:** Refresh every Monday at 3:15 PM UTC:

```sql theme={null}
REFRESH_SCHEDULE = '15 15 * * 1'
```

<Note>
  For large datasets, consider using [Incremental View Maintenance](/concepts/nql/incremental-view-maintenance) to refresh only changed data.
</Note>

***

## DISPLAY\_NAME

A human-readable name shown in the Narrative I/O interface.

**Syntax:**

```sql theme={null}
DISPLAY_NAME = 'Monthly Sales Summary'
```

If not specified, the view name is used as the display name.

***

## DESCRIPTION

Documentation text describing the view's purpose and contents.

**Syntax:**

```sql theme={null}
DESCRIPTION = 'Aggregated sales metrics by month for the executive dashboard'
```

***

## EXPIRE

Sets a data retention policy that determines how long data is kept.

**Syntax:**

```sql theme={null}
EXPIRE = 'P30D'
EXPIRE = 'retain_everything'
EXPIRE = 'expire_everything'
```

**Values:**

| Value               | Description                                                                          |
| ------------------- | ------------------------------------------------------------------------------------ |
| ISO 8601 duration   | Data expires after the specified period (e.g., `P30D` for 30 days, `P1Y` for 1 year) |
| `retain_everything` | Keep all data indefinitely                                                           |
| `expire_everything` | Expire data immediately (effectively clears on each refresh)                         |

**ISO 8601 duration format:**

* `P30D` - 30 days
* `P6M` - 6 months
* `P1Y` - 1 year
* `P1Y6M` - 1 year and 6 months

***

## TAGS

Labels for organizing and filtering materialized views.

**Syntax:**

```sql theme={null}
TAGS = ( '_nio_materialized_view', 'sales', 'monthly' )
```

Tags help categorize views and can be used to filter views in the UI or via API.

***

## WRITE\_MODE

Determines how data is written when the view refreshes.

**Syntax:**

```sql theme={null}
WRITE_MODE = 'overwrite'
WRITE_MODE = 'append'
```

**Values:**

| Value       | Description                                         |
| ----------- | --------------------------------------------------- |
| `overwrite` | Replace all existing data with the new query result |
| `append`    | Add new data to the existing dataset                |

**When to use each:**

* **overwrite**: Use when you want a complete snapshot of the current state. Good for dashboards and reports where you always want the latest data.
* **append**: Use when building a historical record. Good for event logs or time-series data where you want to accumulate data over time.

<Warning>
  When using `append` mode, ensure your query includes appropriate filters to avoid inserting duplicate records.
</Warning>

***

## EXTENDED\_STATS

Controls whether additional statistics are computed for the dataset.

**Syntax:**

```sql theme={null}
EXTENDED_STATS = 'all'
EXTENDED_STATS = 'none'
```

**Values:**

| Value  | Description                                                                                       |
| ------ | ------------------------------------------------------------------------------------------------- |
| `all`  | Compute extended statistics including cardinality estimates, value distributions, and null counts |
| `none` | Skip extended statistics (faster refresh, less metadata)                                          |

Extended statistics help the query optimizer but increase refresh time. Use `all` for frequently queried views and `none` for large views where refresh speed is critical.

***

## PARTITIONED\_BY

Defines how the stored data is partitioned for improved query performance.

**Syntax:**

```sql theme={null}
PARTITIONED_BY <field> <transform>
PARTITIONED_BY <field1> <transform1>, <field2> <transform2>
```

**Transforms:**

| Transform | Description        | Example                           |
| --------- | ------------------ | --------------------------------- |
| `DAY`     | Partition by day   | `PARTITIONED_BY event_date DAY`   |
| `MONTH`   | Partition by month | `PARTITIONED_BY created_at MONTH` |
| `YEAR`    | Partition by year  | `PARTITIONED_BY order_date YEAR`  |

Partitioning improves query performance when filtering on the partitioned field. Choose a partition scheme that matches your most common query patterns.

***

## MERGE ON

Defines upsert logic that matches incoming rows against existing data during refresh. Use MERGE ON to update existing records and insert new ones without creating duplicates.

**Syntax:**

```sql theme={null}
MERGE ON <condition>
WHEN MATCHED THEN UPDATE SET
  <column> = source.<column>,
  ...
WHEN NOT MATCHED THEN INSERT (<columns>) VALUES (<values>)
```

**Auto-generated aliases:**

| Alias    | Description                                            |
| -------- | ------------------------------------------------------ |
| `source` | References columns from the incoming query result      |
| `target` | References columns from the existing materialized view |

**Example:**

```sql theme={null}
MERGE ON target.user_id IS NOT DISTINCT FROM source.user_id
WHEN MATCHED THEN UPDATE SET
  email = source.email,
  last_seen = source.last_seen
WHEN NOT MATCHED THEN INSERT (user_id, email, last_seen)
VALUES (source.user_id, source.email, source.last_seen)
```

<Warning>
  MERGE ON is only valid within `CREATE MATERIALIZED VIEW` statements. It cannot be used as a standalone command.
</Warning>

<Tip>
  For step-by-step guidance and practical examples, see [Incremental Upserts with MERGE ON](/guides/nql/incremental-upserts).
</Tip>

***

## Complete example

```sql theme={null}
CREATE MATERIALIZED VIEW "female_audience"
REFRESH_SCHEDULE = '@daily'
DISPLAY_NAME = 'Female Audience'
DESCRIPTION = 'Audience containing demographic of female users.'
EXPIRE = 'retain_everything'
TAGS = ( '_nio_materialized_view', 'user_engagement', 'metrics' )
WRITE_MODE = 'overwrite'
EXTENDED_STATS = 'all'
PARTITIONED_BY event_date DAY
AS
SELECT
    "unique_identifier",
    "hl7_gender",
    "age"
FROM narrative.rosetta_stone
WHERE
    "hl7_gender"."gender" = 'female'
    AND "event_timestamp" > CURRENT_TIMESTAMP - INTERVAL 160 DAYS
    AND rosetta_stone._price_cpm_usd <= 2.00
```

This example creates a materialized view that:

* Refreshes daily
* Stores female user demographics from the last 160 days
* Partitions data by day for efficient time-based queries
* Retains all historical data indefinitely
* Computes full statistics for query optimization

***

<Note>
  If you need a lightweight query abstraction that re-evaluates at query time instead of storing results, consider using a [view dataset](/concepts/nql/materialized-views#view-datasets) instead. View datasets are created through the SDK or API with the `create_as_view` option rather than through NQL syntax.
</Note>

## Related content

<CardGroup>
  <Card title="Creating Materialized Views" icon="layer-plus" href="/guides/nql/creating-materialized-views">
    Step-by-step guide to creating your first materialized view
  </Card>

  <Card title="Incremental Upserts" icon="code-merge" href="/guides/nql/incremental-upserts">
    Use MERGE ON for deduplication and incremental updates
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    Understand when and why to use materialized views
  </Card>

  <Card title="Incremental View Maintenance" icon="arrows-rotate" href="/concepts/nql/incremental-view-maintenance">
    Learn how NQL optimizes view refreshes
  </Card>
</CardGroup>


# DELETE
Source: https://docs.narrative.io/nql/commands/delete

Reference for the DELETE statement

The `DELETE` statement removes rows from a dataset. A `WHERE` clause restricts which rows are removed.

<Note>
  DELETE runs against datasets you own. To drop an entire dataset rather than delete rows, delete the dataset through the platform UI or API.
</Note>

## Syntax

```sql theme={null}
DELETE FROM table_reference
[WHERE condition]
```

* The `WHERE` clause is optional. If omitted, **every row in the dataset is removed**. The dataset itself remains but is empty.
* The `WHERE` condition can reference any column on the target dataset and use any NQL [operators](/nql/operators) or [functions](/nql/functions/index).
* The `WHERE` condition can reference only the target dataset. Subqueries that read from another dataset (for example, `WHERE name IN (SELECT name FROM company_data."456")`) are not supported.

## Examples

**Delete a single row by key:**

```sql theme={null}
DELETE FROM company_data."123"
WHERE user_id = 42
```

**Delete rows matching a condition:**

```sql theme={null}
DELETE FROM company_data.pipeline_log
WHERE
  status = 'stale'
  AND created_at < CURRENT_TIMESTAMP - INTERVAL '90' DAY
```

**Delete rows with null required fields:**

```sql theme={null}
DELETE FROM company_data."123"
WHERE email IS NULL
```

## Execution context

DELETE statements can run in two places:

| Execution surface                                                                      | When to use                                         |
| -------------------------------------------------------------------------------------- | --------------------------------------------------- |
| NQL query endpoint                                                                     | Ad-hoc deletes from an API client or Data Studio    |
| [`ExecuteDml` workflow task](/guides/workflows/workflow-orchestration#supported-tasks) | Scheduled or event-driven deletes inside a workflow |

<Warning>
  DELETE is irreversible. Running `DELETE FROM table_reference` without a `WHERE` clause empties the dataset. Run the equivalent `SELECT` with the same predicate first to confirm exactly which rows will be removed.
</Warning>

***

## Related content

<CardGroup>
  <Card title="INSERT" icon="plus" href="/nql/commands/insert">
    Add rows to a dataset
  </Card>

  <Card title="UPDATE" icon="pen" href="/nql/commands/update">
    Modify existing rows in a dataset
  </Card>

  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    Preview rows before deleting
  </Card>

  <Card title="ExecuteDml workflow task" icon="diagram-project" href="/guides/workflows/workflow-orchestration#supported-tasks">
    Run DML statements from a workflow
  </Card>
</CardGroup>


# SQL Commands
Source: https://docs.narrative.io/nql/commands/index

Reference for NQL statement types and command syntax

NQL supports statement types for querying, creating stored datasets, and modifying data in existing datasets.

| Statement                  | Purpose                              | Reference                                                          |
| -------------------------- | ------------------------------------ | ------------------------------------------------------------------ |
| `SELECT`                   | Execute a query and return results   | [SELECT](/nql/commands/select)                                     |
| `INSERT`                   | Add rows to an existing dataset      | [INSERT](/nql/commands/insert)                                     |
| `UPDATE`                   | Modify existing rows in a dataset    | [UPDATE](/nql/commands/update)                                     |
| `DELETE`                   | Remove rows from a dataset           | [DELETE](/nql/commands/delete)                                     |
| `CREATE MATERIALIZED VIEW` | Create a stored dataset from a query | [CREATE MATERIALIZED VIEW](/nql/commands/create-materialized-view) |

## Query Templates

NQL also supports parameterized queries through [query templates](/nql/commands/query-templates), which allow you to define reusable queries with placeholder variables.

## Related content

<CardGroup>
  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Primitive and complex types supported in NQL
  </Card>

  <Card title="Functions" icon="function" href="/nql/functions/index">
    Built-in and Narrative-specific functions
  </Card>
</CardGroup>


# INSERT
Source: https://docs.narrative.io/nql/commands/insert

Reference for the INSERT statement

The `INSERT` statement adds rows to an existing dataset. NQL supports two forms: `INSERT ... VALUES (...)` for literal rows, and `INSERT ... SELECT ...` for inserting the result of a query.

<Note>
  INSERT runs against datasets you own. To create a new dataset from a query, use [CREATE MATERIALIZED VIEW](/nql/commands/create-materialized-view). To upsert into an existing materialized view during refresh, use the [MERGE ON clause](/guides/nql/incremental-upserts).
</Note>

## Syntax

### VALUES form

```sql theme={null}
INSERT INTO table_reference (column1, column2, ...)
VALUES
  (value1, value2, ...)
  [, (value1, value2, ...), ...]
```

### SELECT form

```sql theme={null}
INSERT INTO table_reference (column1, column2, ...)
SELECT
  column1,
  column2,
  ...
FROM source_reference
[WHERE condition]
```

## Explicit column list is required

The column list after `INSERT INTO table_reference` is required. NQL does not infer columns from the dataset schema, and wildcards (`SELECT *`) are not supported in the SELECT form either.

```sql theme={null}
-- Correct
INSERT INTO company_data."123" (user_id, email)
VALUES (42, 'alice@example.com')

-- Not supported — missing column list
INSERT INTO company_data."123"
VALUES (42, 'alice@example.com')
```

See [Explicit column selection](/nql/general/explicit-columns) for the rationale and how it applies across NQL.

## Source restrictions for the SELECT form

When you use `INSERT ... SELECT`, the source query can read only from datasets and views your company owns. The following sources are rejected:

* Datasets owned by another company (including those shared with you through access rules)
* Access rules (e.g., `provider_slug."access_rule_name"`)
* `narrative.rosetta_stone`

If you need to ingest data from a shared access rule, materialize it first with [`CREATE MATERIALIZED VIEW`](/nql/commands/create-materialized-view), then `INSERT ... SELECT` from your materialized view.

Subqueries inside the SELECT — including joins, CTEs, and `IN (SELECT ...)` predicates — are subject to the same restriction.

## Column matching and types

* The number of expressions in each `VALUES` row (or columns in the `SELECT` clause) must match the column list.
* Expressions are matched to columns by position, not by name.
* Each expression must be assignable to the target column's [data type](/nql/data-types). Implicit coercions follow the same rules as comparisons elsewhere in NQL.

## Examples

**Single-row VALUES:**

```sql theme={null}
INSERT INTO company_data."123" (dataset_id, status, created_at)
VALUES (456, 'created', CURRENT_TIMESTAMP)
```

**Multi-row VALUES:**

```sql theme={null}
INSERT INTO company_data.pipeline_log (dataset_id, status, created_at)
VALUES
  (101, 'created', CURRENT_TIMESTAMP),
  (102, 'created', CURRENT_TIMESTAMP),
  (103, 'skipped', CURRENT_TIMESTAMP)
```

**INSERT from a SELECT:**

```sql theme={null}
INSERT INTO company_data."456" (user_id, email, created_at)
SELECT
  company_data."123".user_id,
  company_data."123".email,
  company_data."123".created_at
FROM company_data."123"
WHERE company_data."123".created_at >= CURRENT_DATE - INTERVAL '7' DAY
```

**INSERT with transformations and a join:**

```sql theme={null}
INSERT INTO company_data."789" (user_id, email, region)
SELECT
  u.user_id,
  LOWER(u.email) AS email,
  g.region
FROM company_data."123" AS u
JOIN company_data."456" AS g
  ON u.user_id = g.user_id
WHERE u.is_active = TRUE
```

## Execution context

INSERT statements can run in two places:

| Execution surface                                                                      | When to use                                         |
| -------------------------------------------------------------------------------------- | --------------------------------------------------- |
| NQL query endpoint                                                                     | Ad-hoc inserts from an API client or Data Studio    |
| [`ExecuteDml` workflow task](/guides/workflows/workflow-orchestration#supported-tasks) | Scheduled or event-driven inserts inside a workflow |

INSERT does not create datasets. The target dataset must already exist. Use `CREATE MATERIALIZED VIEW` to create datasets from a query, then `INSERT` to add rows afterward.

<Tip>
  For bulk loads from one dataset to another, `INSERT ... SELECT` is almost always faster and cheaper than issuing many single-row `INSERT ... VALUES` statements.
</Tip>

***

## Related content

<CardGroup>
  <Card title="UPDATE" icon="pen" href="/nql/commands/update">
    Modify existing rows in a dataset
  </Card>

  <Card title="DELETE" icon="trash" href="/nql/commands/delete">
    Remove rows from a dataset
  </Card>

  <Card title="Incremental Upserts with MERGE ON" icon="code-merge" href="/guides/nql/incremental-upserts">
    Update or insert rows during materialized view refresh
  </Card>

  <Card title="CREATE MATERIALIZED VIEW" icon="list-check" href="/nql/commands/create-materialized-view">
    Create a new dataset from a query
  </Card>

  <Card title="ExecuteDml workflow task" icon="diagram-project" href="/guides/workflows/workflow-orchestration#supported-tasks">
    Run DML statements from a workflow
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Type rules for INSERT expressions
  </Card>
</CardGroup>


# Query Template Syntax
Source: https://docs.narrative.io/nql/commands/query-templates

Complete reference for query template placeholders and structure

Query templates use placeholders to mark variable parts of a query. When a user runs a template, they provide values for each placeholder, and the system substitutes them into the query before execution.

## Placeholder syntax

Placeholders use double curly braces with a name:

```
{{placeholder_name}}
```

Placeholder names should be:

* Descriptive and self-documenting
* Lowercase with underscores for word separation
* Unique within the template

**Examples:**

* `{{start_date}}` - a date parameter
* `{{min_revenue_threshold}}` - a numeric threshold
* `{{target_region}}` - a string filter value

***

## Placeholder types

Each placeholder has a type that determines what values it accepts and how the value is substituted into the query.

### Literal value placeholders

Replace constants like strings, numbers, booleans, or dates.

**Supported types:**

| Type        | Description          | Example values                |
| ----------- | -------------------- | ----------------------------- |
| `string`    | Text values          | `'North America'`, `'active'` |
| `number`    | Numeric values       | `100`, `3.14`, `-50`          |
| `boolean`   | True/false values    | `true`, `false`               |
| `date`      | Date values          | `'2024-01-15'`                |
| `timestamp` | Date and time values | `'2024-01-15 14:30:00'`       |

**Example:**

```sql theme={null}
SELECT user_id, email, created_at
FROM users
WHERE
    status = {{user_status}}
    AND created_at >= {{start_date}}
    AND account_balance > {{min_balance}}
```

| Placeholder   | Type   | Description                       |
| ------------- | ------ | --------------------------------- |
| `user_status` | string | Account status to filter by       |
| `start_date`  | date   | Beginning of date range           |
| `min_balance` | number | Minimum account balance threshold |

***

### Column placeholders

Replace column names in SELECT, GROUP BY, or ORDER BY clauses. Use these when users need to choose which fields appear in results.

**Example:**

```sql theme={null}
SELECT
    {{grouping_column}},
    COUNT(1) as record_count,
    SUM(amount) as total_amount
FROM transactions
GROUP BY {{grouping_column}}
ORDER BY {{sort_column}} DESC
```

| Placeholder       | Type   | Description               |
| ----------------- | ------ | ------------------------- |
| `grouping_column` | column | Field to group results by |
| `sort_column`     | column | Field to sort results by  |

<Note>
  Column placeholders should specify which columns are valid options in the placeholder metadata. This prevents users from selecting columns that don't exist or aren't appropriate for the query context.
</Note>

***

### Filter placeholders

Replace entire WHERE conditions. Use these when the filtering logic itself varies, not just the filter values.

**Example:**

```sql theme={null}
SELECT order_id, customer_id, total, order_date
FROM orders
WHERE
    order_date >= {{start_date}}
    AND order_date < {{end_date}}
    AND {{custom_filter}}
```

| Placeholder     | Type   | Description                    |
| --------------- | ------ | ------------------------------ |
| `start_date`    | date   | Beginning of date range        |
| `end_date`      | date   | End of date range              |
| `custom_filter` | filter | Additional filtering condition |

A user might provide `region = 'EMEA' AND total > 1000` for the `custom_filter` placeholder.

<Warning>
  Filter placeholders give users significant flexibility. Use them only when necessary, and document expected filter formats clearly.
</Warning>

***

### Output field placeholders

Replace parts of the SELECT list. Use these when the shape of the output depends on user choice.

**Example:**

```sql theme={null}
SELECT
    customer_id,
    customer_name,
    {{output_fields}}
FROM customers
WHERE region = {{region}}
```

| Placeholder     | Type   | Description                              |
| --------------- | ------ | ---------------------------------------- |
| `output_fields` | output | Additional columns to include in results |
| `region`        | string | Region to filter by                      |

A user might provide `email, phone, account_type` for the `output_fields` placeholder.

***

## Placeholder metadata

Each placeholder includes metadata that describes how users should interact with it.

| Property      | Type    | Required | Description                                                                                  |
| ------------- | ------- | -------- | -------------------------------------------------------------------------------------------- |
| `name`        | string  | Yes      | Unique identifier for the placeholder                                                        |
| `type`        | enum    | Yes      | Data type (`string`, `number`, `boolean`, `date`, `timestamp`, `column`, `filter`, `output`) |
| `description` | string  | Yes      | Explanation of what this placeholder controls                                                |
| `required`    | boolean | No       | Whether a value must be provided (default: `true`)                                           |
| `default`     | varies  | No       | Default value used when not provided                                                         |

### Required vs. optional placeholders

* **Required placeholders** must have values before the query can execute
* **Optional placeholders** use their default value if the user doesn't provide one

**Example with optional placeholder:**

```sql theme={null}
SELECT region, COUNT(1) as user_count
FROM users
WHERE
    created_at >= {{start_date}}
    AND status = {{status}}
GROUP BY region
```

| Placeholder  | Type   | Required | Default    | Description              |
| ------------ | ------ | -------- | ---------- | ------------------------ |
| `start_date` | date   | Yes      | —          | Beginning of date range  |
| `status`     | string | No       | `'active'` | User status to filter by |

If the user doesn't provide a `status` value, the query uses `'active'`.

***

## Complete example

Here's a complete template with multiple placeholder types:

```sql theme={null}
SELECT
    {{grouping_column}},
    COUNT(1) as record_count,
    SUM(revenue) as total_revenue,
    AVG(revenue) as avg_revenue
FROM sales_data
WHERE
    sale_date >= {{start_date}}
    AND sale_date < {{end_date}}
    AND region = {{region}}
    AND revenue >= {{min_revenue}}
GROUP BY {{grouping_column}}
ORDER BY total_revenue DESC
LIMIT {{result_limit}}
```

**Placeholder definitions:**

| Placeholder       | Type   | Required | Default | Description                                                   |
| ----------------- | ------ | -------- | ------- | ------------------------------------------------------------- |
| `grouping_column` | column | Yes      | —       | Field to aggregate by (e.g., `product_category`, `sales_rep`) |
| `start_date`      | date   | Yes      | —       | Beginning of reporting period (inclusive)                     |
| `end_date`        | date   | Yes      | —       | End of reporting period (exclusive)                           |
| `region`          | string | Yes      | —       | Geographic region to analyze                                  |
| `min_revenue`     | number | No       | `0`     | Minimum transaction amount to include                         |
| `result_limit`    | number | No       | `100`   | Maximum rows to return                                        |

***

## Materialized view templates

Templates can create materialized views, allowing users to generate scheduled data products with customized parameters.

```sql theme={null}
CREATE MATERIALIZED VIEW {{view_name}}
REFRESH_SCHEDULE = {{refresh_schedule}}
DISPLAY_NAME = {{display_name}}
AS
SELECT
    user_id,
    COUNT(1) as event_count,
    MAX(event_timestamp) as last_activity
FROM events
WHERE
    event_type = {{event_type}}
    AND event_timestamp > CURRENT_TIMESTAMP - INTERVAL {{lookback_days}} DAY
GROUP BY user_id
```

| Placeholder        | Type   | Required | Description                                     |
| ------------------ | ------ | -------- | ----------------------------------------------- |
| `view_name`        | string | Yes      | Internal identifier for the materialized view   |
| `refresh_schedule` | string | Yes      | When to refresh (e.g., `'@daily'`, `'@hourly'`) |
| `display_name`     | string | Yes      | Human-readable name shown in UI                 |
| `event_type`       | string | Yes      | Type of event to aggregate                      |
| `lookback_days`    | number | Yes      | Days of history to include                      |

For materialized view options, see [Materialized View Syntax](/nql/commands/create-materialized-view).

***

## Related content

<CardGroup>
  <Card title="Using Query Templates" icon="play" href="/guides/nql/using-query-templates">
    Step-by-step guide to creating and using templates
  </Card>

  <Card title="Query Templates" icon="lightbulb" href="/concepts/nql/query-templates">
    Understand when and why to use templates
  </Card>

  <Card title="NQL Data Types" icon="shapes" href="/nql/data-types">
    Complete reference for NQL data types
  </Card>

  <Card title="NQL Syntax" icon="terminal" href="/nql/general/syntax">
    Core NQL syntax reference
  </Card>
</CardGroup>


# SELECT
Source: https://docs.narrative.io/nql/commands/select

Reference for the SELECT statement and all its clauses

The `SELECT` statement retrieves data from one or more datasets.

## Syntax

```sql theme={null}
SELECT [DISTINCT]
  column1 [AS alias1],
  column2 [AS alias2],
  ...
FROM table_reference [AS alias]
[JOIN table_reference ON condition]
[WHERE condition]
[GROUP BY column1, column2, ...]
[HAVING condition]
[QUALIFY condition]
[ORDER BY column1 [ASC|DESC], ...]
[LIMIT n [OFFSET m] ROWS]
```

<Note>
  NQL requires explicit column lists. Wildcards (`SELECT *` and `COUNT(*)`) are not supported. See [Explicit Column Selection](/nql/general/explicit-columns) for details.
</Note>

## Clause order

Clauses must appear in this order:

1. `SELECT` — columns to return
2. `FROM` — data source(s)
3. `JOIN` — additional data sources with join conditions
4. `WHERE` — row-level filters applied before aggregation
5. `GROUP BY` — grouping for aggregations
6. `HAVING` — filters applied after aggregation
7. `QUALIFY` — filters based on window function results
8. `ORDER BY` — result ordering
9. `LIMIT` — result count restriction

## Examples

**Basic query:**

```sql theme={null}
SELECT
  user_id,
  email,
  created_at
FROM company_data."123"
WHERE created_at > '2024-01-01'
```

**Query with aggregation:**

```sql theme={null}
SELECT
  region,
  COUNT(1) AS user_count,
  AVG(lifetime_value) AS avg_ltv
FROM company_data."456"
GROUP BY region
HAVING COUNT(1) > 100
ORDER BY avg_ltv DESC
```

**Query with window function and QUALIFY:**

```sql theme={null}
SELECT
  user_id,
  email,
  last_seen
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

***

## Table references

### Dataset references

Reference your datasets through the `company_data` schema, using either the dataset's numeric ID or its name:

```sql theme={null}
FROM company_data."123"
FROM company_data.my_dataset
```

Numeric IDs must be quoted because they are numeric. Dataset names don't need quoting unless the name is a reserved keyword or contains special characters, in which case wrap it in double quotes.

### Table aliases

Assign aliases to simplify column references:

```sql theme={null}
FROM company_data."123" AS users
SELECT users.email, users.created_at
```

### Rosetta Stone table references

Rosetta Stone provides normalized access to data through three scope levels: global, company-scoped, and dataset-scoped. Each scope determines which data sources your query accesses.

<Note>
  To access another company's data through Rosetta Stone, that company must have
  shared their data with you via an access rule. Your own company's data
  (`company_data`) is always accessible. See [Access Rules](/concepts/primitives/access-rules)
  for details on data sharing.
</Note>

#### Global access

Query all normalized data from all datasets shared with you:

```sql theme={null}
SELECT
  narrative.rosetta_stone.unique_id,
  narrative.rosetta_stone.event_timestamp
FROM narrative.rosetta_stone
WHERE narrative.rosetta_stone.event_timestamp >= CURRENT_DATE - INTERVAL '30' DAY
```

#### Company-scoped access

Query normalized data from a specific company's datasets:

```sql theme={null}
-- All datasets from your company
SELECT
  company_data._rosetta_stone.unique_id,
  company_data._rosetta_stone.event_timestamp
FROM company_data._rosetta_stone

-- All datasets from a specific company (by slug)
SELECT
  partner_company._rosetta_stone.unique_id,
  partner_company._rosetta_stone.event_timestamp
FROM partner_company._rosetta_stone
```

#### Dataset-scoped access

Query normalized data from a specific dataset:

```sql theme={null}
-- By dataset ID
SELECT
  company_data."123"._rosetta_stone.unique_id,
  company_data."123"._rosetta_stone.event_timestamp
FROM company_data."123"._rosetta_stone

-- By dataset slug
SELECT
  company_data.my_dataset._rosetta_stone.unique_id,
  company_data.my_dataset._rosetta_stone.event_timestamp
FROM company_data.my_dataset._rosetta_stone

-- From another company's dataset
SELECT
  partner_company.shared_dataset._rosetta_stone.unique_id,
  partner_company.shared_dataset._rosetta_stone.event_timestamp
FROM partner_company.shared_dataset._rosetta_stone
```

#### Choosing a scope level

| Scope   | Syntax                              | Use when                                                                        |
| ------- | ----------------------------------- | ------------------------------------------------------------------------------- |
| Global  | `narrative.rosetta_stone`           | You want all available normalized data from all sources                         |
| Company | `company_data._rosetta_stone`       | You want data from a specific company only                                      |
| Dataset | `company_data."123"._rosetta_stone` | You need to combine normalized and non-normalized columns from the same dataset |

See [Scoping Your Queries](/concepts/rosetta-stone/normalization-model#scoping-your-queries) for guidance on choosing the right scope.

### Access rules

Query through pre-configured access rules using the provider's company slug:

```sql theme={null}
FROM provider_company."access_rule_name"

SELECT
  provider_company."ar_fitness".Total_Output
FROM provider_company."ar_fitness"
```

***

## Column references

### Fully qualified names

Use fully qualified column names to avoid ambiguity, especially in joins:

```sql theme={null}
SELECT
  company_data."123".user_id,
  company_data."123".email,
  company_data."456".order_id
FROM company_data."123"
JOIN company_data."456" ON company_data."123".user_id = company_data."456".customer_id
```

### Column aliases

Assign aliases to rename columns in results:

```sql theme={null}
SELECT
  user_id AS id,
  email AS contact_email,
  UPPER(name) AS display_name
FROM company_data."123"
```

### Accessing nested fields

Access struct fields using dot notation:

```sql theme={null}
SELECT
  company_data."1".nested_field.deep_field.value
FROM company_data."1"
```

Access array elements using bracket notation:

```sql theme={null}
SELECT
  identifiers[0] AS first_id,
  identifiers[0].type AS first_id_type
FROM company_data."1"
```

<Tip>
  Bracket notation also works for struct fields: `data['field_name']`. This avoids needing to quote reserved keywords. See [Reserved Keywords best practices](/nql/general/reserved-keywords#prefer-bracket-notation-for-property-access).
</Tip>

***

## Special columns

### Price column

Every dataset includes `_price_cpm_usd` (price per 1,000 rows). Use it to filter by cost:

```sql theme={null}
WHERE company_data."123"._price_cpm_usd <= 1.00
```

### Rosetta Stone field access

Access normalized identity data through the `._rosetta_stone` special field on any dataset:

```sql theme={null}
SELECT
  company_data."123"._rosetta_stone.unique_id,
  company_data."123".original_column
FROM company_data."123"
```

This allows you to access both normalized Rosetta Stone attributes and non-normalized columns from the same dataset row.

#### Access specific identifier types

Query specific identifier types using the `unique_id` struct:

```sql theme={null}
SELECT
  narrative.rosetta_stone.unique_id.b."type",
  narrative.rosetta_stone.unique_id.b."value"
FROM narrative.rosetta_stone
WHERE narrative.rosetta_stone.unique_id.b."type" NOT IN ('tdid', 'cookie')
```

For complete details on scope levels and table references, see [Rosetta Stone table references](#rosetta-stone-table-references).

***

## DELTA tables

The `DELTA` function returns only records that have changed since the last query execution. Use it for incremental data processing.

### Syntax

```sql theme={null}
FROM DELTA(TABLE table_reference)
FROM DELTA((subquery))
```

### Examples

**Delta on a dataset:**

```sql theme={null}
SELECT column1
FROM DELTA(TABLE company_data."1") AS t
```

**Delta on Rosetta Stone:**

```sql theme={null}
SELECT long_value
FROM DELTA(TABLE narrative.rosetta_stone)
```

**Delta with filtering:**

```sql theme={null}
SELECT long_value
FROM DELTA((
  SELECT long_value
  FROM company_data.rosetta_stone
  WHERE long_value > 42
))
```

***

## Subqueries

### Subquery in WHERE

```sql theme={null}
SELECT column1
FROM company_data."1"
WHERE value > (
  SELECT AVG(value)
  FROM company_data."1"
)
```

### Subquery in FROM

```sql theme={null}
SELECT sub.total
FROM (
  SELECT SUM(amount) AS total
  FROM company_data."1"
  GROUP BY category
) AS sub
WHERE sub.total > 1000
```

### EXISTS and NOT EXISTS

```sql theme={null}
SELECT t1.column1
FROM company_data."1" t1
WHERE NOT EXISTS (
  SELECT 1
  FROM company_data."2" t2
  WHERE t1.id = t2.id
)
```

***

## Common Table Expressions (WITH)

CTEs let you define named subqueries for reuse within a statement.

### Syntax

```sql theme={null}
WITH cte_name AS (
  SELECT ...
),
another_cte AS (
  SELECT ...
)
SELECT ...
FROM cte_name
```

### Example

```sql theme={null}
CREATE MATERIALIZED VIEW "high_value_users"
AS (
  WITH daily_totals AS (
    SELECT
      user_id,
      DATE_TRUNC('day', event_timestamp) AS event_date,
      SUM(amount) AS daily_spend
    FROM company_data."1"
    GROUP BY user_id, DATE_TRUNC('day', event_timestamp)
  ),
  user_averages AS (
    SELECT
      user_id,
      AVG(daily_spend) AS avg_daily_spend
    FROM daily_totals
    GROUP BY user_id
  )
  SELECT
    user_id,
    avg_daily_spend
  FROM user_averages
  WHERE avg_daily_spend > 100
)
BUDGET 25 USD
```

***

## QUALIFY clause

The `QUALIFY` clause filters results based on window function values. It's evaluated after window functions, making it useful for deduplication and ranking.

### Syntax

```sql theme={null}
SELECT columns
FROM table
QUALIFY window_function_condition
```

### Deduplication example

Keep only the most recent record per user:

```sql theme={null}
SELECT
  user_id,
  email,
  last_seen
FROM company_data."123"
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY user_id
  ORDER BY last_seen DESC
) = 1
```

### Multiple window conditions

```sql theme={null}
SELECT column1
FROM company_data."1"
QUALIFY
  ROW_NUMBER() OVER (PARTITION BY group_col ORDER BY sort_col) = 1
  AND SUM(1) OVER (PARTITION BY other_col) < 10
```

***

## Budget clauses

Budget clauses control spending on data queries. They are required for `CREATE MATERIALIZED VIEW` statements.

### BUDGET

Set a maximum total spend:

```sql theme={null}
BUDGET 50 USD
```

### LIMIT with USD

Alternative syntax for budget limits:

```sql theme={null}
LIMIT 100 USD
```

### Recurring limits

Set budget limits per time period:

```sql theme={null}
LIMIT 500 USD PER CALENDAR_MONTH
LIMIT 50 USD PER CALENDAR_DAY
```

### Row limits

Limit by row count instead of cost:

```sql theme={null}
LIMIT 1000 ROWS
LIMIT 100 OFFSET 10 ROWS
```

***

## SELECT DISTINCT

Remove duplicate rows from results:

```sql theme={null}
SELECT DISTINCT category
FROM company_data."123"
```

***

## CASE expressions

Conditional logic within queries:

```sql theme={null}
SELECT
  user_id,
  CASE
    WHEN lifetime_value > 1000 THEN 'high'
    WHEN lifetime_value > 100 THEN 'medium'
    ELSE 'low'
  END AS value_tier
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Primitive and complex types supported in NQL
  </Card>

  <Card title="Operators" icon="calculator" href="/nql/operators">
    Comparison, logical, and arithmetic operators
  </Card>

  <Card title="Functions" icon="function" href="/nql/functions/index">
    Built-in and Narrative-specific functions
  </Card>

  <Card title="CREATE MATERIALIZED VIEW" icon="list-check" href="/nql/commands/create-materialized-view">
    Complete CREATE MATERIALIZED VIEW reference
  </Card>
</CardGroup>


# UPDATE
Source: https://docs.narrative.io/nql/commands/update

Reference for the UPDATE statement

The `UPDATE` statement modifies existing rows in a dataset. Each `UPDATE` assigns new values to one or more columns, optionally filtered by a `WHERE` clause.

<Note>
  UPDATE runs against datasets you own. To update rows during a materialized view refresh, use the [MERGE ON clause](/guides/nql/incremental-upserts) inside `CREATE MATERIALIZED VIEW` instead.
</Note>

## Syntax

```sql theme={null}
UPDATE table_reference
SET
  column1 = expression1
  [, column2 = expression2, ...]
[WHERE condition]
```

* The `SET` clause lists column assignments. The right-hand side of each assignment can be any NQL expression valid for the target column's [data type](/nql/data-types), including references to other columns in the same row.
* The `WHERE` clause is optional. If omitted, **every row in the dataset is updated** — include a `WHERE` clause unless that is the intended behavior.
* The `SET` expressions and the `WHERE` condition can reference only the target dataset. Subqueries that read from another dataset (for example, `WHERE id IN (SELECT id FROM company_data."456")`) are not supported.

## Examples

**Update a single column with a literal:**

```sql theme={null}
UPDATE company_data."123"
SET status = 'archived'
WHERE user_id = 42
```

**Update multiple columns using an expression:**

```sql theme={null}
UPDATE company_data."123"
SET
  email = LOWER(email),
  last_updated_at = CURRENT_TIMESTAMP
WHERE email IS NOT NULL
```

**Conditional update across many rows:**

```sql theme={null}
UPDATE company_data.pipeline_log
SET status = 'stale'
WHERE
  status = 'created'
  AND created_at < CURRENT_TIMESTAMP - INTERVAL '30' DAY
```

## Execution context

UPDATE statements can run in two places:

| Execution surface                                                                      | When to use                                         |
| -------------------------------------------------------------------------------------- | --------------------------------------------------- |
| NQL query endpoint                                                                     | Ad-hoc updates from an API client or Data Studio    |
| [`ExecuteDml` workflow task](/guides/workflows/workflow-orchestration#supported-tasks) | Scheduled or event-driven updates inside a workflow |

<Warning>
  Running UPDATE without a `WHERE` clause modifies every row in the target dataset. Review the statement — or run the corresponding `SELECT` with the same predicate first — before executing.
</Warning>

***

## Related content

<CardGroup>
  <Card title="INSERT" icon="plus" href="/nql/commands/insert">
    Add rows to a dataset
  </Card>

  <Card title="DELETE" icon="trash" href="/nql/commands/delete">
    Remove rows from a dataset
  </Card>

  <Card title="Incremental Upserts with MERGE ON" icon="code-merge" href="/guides/nql/incremental-upserts">
    Update or insert rows during materialized view refresh
  </Card>

  <Card title="ExecuteDml workflow task" icon="diagram-project" href="/guides/workflows/workflow-orchestration#supported-tasks">
    Run DML statements from a workflow
  </Card>
</CardGroup>


# Data Types
Source: https://docs.narrative.io/nql/data-types

Primitive and complex types supported in NQL

NQL supports primitive types for scalar values and complex types for structured data. This reference covers all supported types and type conversion.

## Primitive types

| Type        | Aliases   | Description                | Example literals            |
| ----------- | --------- | -------------------------- | --------------------------- |
| `STRING`    | `VARCHAR` | Variable-length text       | `'hello world'`             |
| `BOOLEAN`   | —         | True or false              | `true`, `false`             |
| `LONG`      | `BIGINT`  | 64-bit signed integer      | `12345678901234`            |
| `DOUBLE`    | —         | 64-bit floating point      | `3.14159`                   |
| `DECIMAL`   | —         | Arbitrary precision number | `CAST(10.12345 AS DECIMAL)` |
| `DATE`      | —         | Date without time          | `'2024-01-15'`              |
| `TIMESTAMP` | —         | Date with time             | `'2024-01-15 10:30:00'`     |

***

## STRING / VARCHAR

Variable-length character data. String literals use single quotes.

```sql theme={null}
SELECT
  'Hello, World!' AS greeting,
  CONCAT(first_name, ' ', last_name) AS full_name
FROM company_data."123"
```

### Escape sequences

| Sequence | Meaning      |
| -------- | ------------ |
| `''`     | Single quote |
| `\\`     | Backslash    |

```sql theme={null}
SELECT 'It''s a test' AS escaped_quote
```

***

## BOOLEAN

Logical true/false values.

```sql theme={null}
SELECT
  is_active,
  CASE WHEN score > 90 THEN true ELSE false END AS is_high_performer
FROM company_data."123"
WHERE is_verified = true
```

***

## LONG / BIGINT

64-bit signed integers. Range: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.

```sql theme={null}
SELECT
  user_id,
  CAST(timestamp_value AS LONG) AS epoch_millis
FROM company_data."123"
```

***

## DOUBLE

64-bit IEEE 754 floating-point numbers. Use for approximate numeric calculations.

```sql theme={null}
SELECT
  amount * 1.0825 AS amount_with_tax,
  SQRT(value) AS square_root
FROM company_data."123"
```

<Warning>
  Floating-point arithmetic can produce rounding errors. Use DECIMAL for financial calculations requiring exact precision.
</Warning>

***

## DECIMAL

Arbitrary-precision decimal numbers. Use for exact calculations where precision matters.

```sql theme={null}
SELECT
  CAST(price AS DECIMAL) AS exact_price,
  CAST(quantity AS DECIMAL) * CAST(unit_price AS DECIMAL) AS line_total
FROM company_data."123"
```

***

## DATE

Calendar date without time component. Format: `YYYY-MM-DD`.

```sql theme={null}
SELECT
  event_date,
  DATE_TRUNC('month', event_date) AS month_start
FROM company_data."123"
WHERE event_date >= '2024-01-01'
```

### Date arithmetic

```sql theme={null}
SELECT
  event_date,
  DATE_ADD(event_date, 30) AS plus_30_days,
  event_date - INTERVAL '7' DAY AS minus_7_days
FROM company_data."123"
```

***

## TIMESTAMP

Date and time with precision. Format: `YYYY-MM-DD HH:MM:SS`.

```sql theme={null}
SELECT
  event_timestamp,
  DATE_TRUNC('hour', event_timestamp) AS hour_start
FROM company_data."123"
WHERE event_timestamp > CURRENT_TIMESTAMP - INTERVAL '24' HOUR
```

### Current timestamp functions

| Function            | Returns           |
| ------------------- | ----------------- |
| `CURRENT_DATE`      | Current date      |
| `CURRENT_TIMESTAMP` | Current timestamp |

***

## Complex types

NQL supports three complex types for structured data: arrays, structs, and maps.

***

## ARRAY

Ordered collection of elements of the same type.

### Array literal

```sql theme={null}
SELECT ARRAY('value1', 'value2', 'value3') AS my_array
```

### Array type declaration

```sql theme={null}
CAST(column AS ARRAY<STRING>)
CAST(column AS ARRAY<LONG>)
CAST(column AS ARRAY<STRUCT<name:STRING, age:LONG>>)
```

### Accessing array elements

Use zero-based indexing with bracket notation:

```sql theme={null}
SELECT
  identifiers[0] AS first_id,
  identifiers[1] AS second_id,
  identifiers[0].type AS first_id_type
FROM company_data."123"
```

### Array functions

| Function                   | Description                |
| -------------------------- | -------------------------- |
| `SIZE(array)`              | Returns number of elements |
| `ELEMENT_AT(array, index)` | Returns element at index   |
| `ARRAY_FLATTEN(array)`     | Flattens nested arrays     |

See [Functions Reference](/nql/functions/index) for complete array function documentation.

***

## STRUCT

Named collection of fields with potentially different types. Similar to a row or record.

### Struct literal

```sql theme={null}
SELECT NAMED_STRUCT(
  'name', 'John Doe',
  'age', 30,
  'active', true
) AS person
```

### Struct type declaration

```sql theme={null}
CAST(column AS STRUCT<name:STRING, age:LONG>)
CAST(column AS STRUCT<
  user:STRUCT<id:LONG, email:STRING>,
  created_at:TIMESTAMP
>)
```

### Accessing struct fields

Use dot notation to access fields:

```sql theme={null}
SELECT
  person.name,
  person.age,
  address.city
FROM company_data."123"
```

### Nested struct access

```sql theme={null}
SELECT
  company_data."1".nested_field.deep_field.value
FROM company_data."1"
```

***

## MAP

Collection of key-value pairs. Keys must be of a single type, as must values.

### Map type declaration

```sql theme={null}
CAST(column AS MAP<STRING, STRING>)
CAST(column AS MAP<STRING, LONG>)
CAST(column AS MAP<STRING, STRUCT<a:STRING, b:LONG>>)
```

### Accessing map values

Use `ELEMENT_AT` to retrieve values by key:

```sql theme={null}
SELECT
  ELEMENT_AT(properties, 'color') AS color,
  ELEMENT_AT(metadata, 'source') AS source
FROM company_data."123"
```

***

## Nested types

Complex types can be nested to represent hierarchical data structures.

### Array of structs

```sql theme={null}
CAST(column AS ARRAY<STRUCT<
  type:STRING,
  value:STRING,
  timestamp:LONG
>>)
```

Access nested struct fields within arrays:

```sql theme={null}
SELECT
  identifiers[0].type AS first_type,
  identifiers[0].value AS first_value
FROM company_data."123"
```

### Map with struct values

```sql theme={null}
CAST(column AS MAP<STRING, STRUCT<
  count:LONG,
  last_seen:TIMESTAMP
>>)
```

### Deeply nested structures

```sql theme={null}
CAST(column AS STRUCT<
  field1:STRING,
  field2:MAP<STRING, STRUCT<a:LONG>>,
  field3:ARRAY<MAP<STRING, STRING>>
>)
```

***

## Type casting

Use `CAST` to convert between types.

### Syntax

```sql theme={null}
CAST(expression AS target_type)
```

### Common conversions

```sql theme={null}
SELECT
  CAST(numeric_string AS LONG) AS number,
  CAST(amount AS VARCHAR) AS amount_text,
  CAST(timestamp_value AS DATE) AS date_only,
  CAST(epoch_millis AS TIMESTAMP) AS datetime,
  CAST(value AS DECIMAL) AS precise_value
FROM company_data."123"
```

### Casting complex types

```sql theme={null}
SELECT
  CAST(raw_array AS ARRAY<STRING>) AS string_array,
  CAST(raw_struct AS STRUCT<id:LONG, name:STRING>) AS typed_struct,
  CAST(NULL AS MAP<STRING, LONG>) AS empty_map
FROM company_data."123"
```

<Note>
  When casting to complex types, the source data must have a compatible structure. Invalid casts return NULL or raise an error.
</Note>

***

## Type coercion

NQL automatically converts types in some contexts:

| Context       | Behavior                                             |
| ------------- | ---------------------------------------------------- |
| Comparison    | Numeric types promoted to common type                |
| Arithmetic    | Integer operations with DOUBLE operand return DOUBLE |
| Concatenation | Non-strings converted to STRING with `\|\|` operator |
| Aggregation   | Results match input type or use promoted type        |

### Explicit vs implicit conversion

Prefer explicit `CAST` for clarity:

```sql theme={null}
-- Explicit (preferred)
WHERE CAST(string_column AS LONG) > 100

-- Implicit (may work but less clear)
WHERE string_column > '100'
```

***

## NULL values

NULL represents missing or unknown data. NULL behaves specially in comparisons and operations.

### NULL checks

```sql theme={null}
WHERE email IS NOT NULL
WHERE phone IS NULL
```

### NULL in expressions

Any operation involving NULL typically returns NULL:

```sql theme={null}
SELECT
  NULL + 1,           -- Returns NULL
  CONCAT('Hi ', NULL) -- Returns NULL
```

### COALESCE

Return the first non-NULL value:

```sql theme={null}
SELECT COALESCE(preferred_email, backup_email, 'unknown') AS email
FROM company_data."123"
```

### NULLIF

Return NULL if two values are equal:

```sql theme={null}
SELECT NULLIF(status, 'unknown') AS clean_status
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="NQL Syntax" icon="code" href="/nql/general/syntax">
    Query structure and grammar reference
  </Card>

  <Card title="Operators" icon="calculator" href="/nql/operators">
    Comparison, logical, and arithmetic operators
  </Card>

  <Card title="Functions" icon="function" href="/nql/functions/index">
    Type conversion and manipulation functions
  </Card>

  <Card title="Type System Concepts" icon="diagram-project" href="/concepts/nql/type-system">
    Understanding NQL's type design philosophy
  </Card>
</CardGroup>


# Aggregate Functions
Source: https://docs.narrative.io/nql/functions/aggregate-functions

GROUP BY computations: COUNT, SUM, AVG, MIN, MAX, and more

## Aggregate functions

Aggregate functions compute a single result from a set of rows.

### COUNT

Counts rows or non-null values. Use `COUNT(1)` for total row counts and `COUNT(column)` for counting non-null values.

```sql theme={null}
SELECT COUNT(1) AS total_rows FROM company_data."123"
SELECT COUNT(email) AS emails_present FROM company_data."123"

-- For distinct counts, prefer APPROX_COUNT_DISTINCT (see below)
SELECT COUNT(DISTINCT user_id) AS unique_users FROM company_data."123"
```

### APPROX\_COUNT\_DISTINCT

Returns the approximate number of distinct values in a column. Faster and cheaper than `COUNT(DISTINCT ...)`, and returns exact results for low-cardinality columns. Use this instead of `COUNT(DISTINCT ...)` unless your query requires a guaranteed exact count.

```sql theme={null}
SELECT APPROX_COUNT_DISTINCT(user_id) AS unique_users FROM company_data."123"
```

<Tip>
  `APPROX_COUNT_DISTINCT` is the recommended way to count unique values in most queries. It produces exact results when the number of distinct values is small and near-exact results at scale—while using significantly fewer resources than `COUNT(DISTINCT ...)`.
</Tip>

### SUM

Returns the sum of values.

```sql theme={null}
SELECT SUM(amount) AS total_amount FROM company_data."123"
```

### AVG

Returns the average of values.

```sql theme={null}
SELECT AVG(score) AS average_score FROM company_data."123"
```

### MIN / MAX

Returns the minimum or maximum value.

```sql theme={null}
SELECT MIN(created_at) AS earliest FROM company_data."123"
SELECT MAX(score) AS highest_score FROM company_data."123"
```

### STDDEV\_POP / STDDEV\_SAMP

Returns population or sample standard deviation.

```sql theme={null}
SELECT STDDEV_POP(value) AS std_dev FROM company_data."123"
SELECT STDDEV_SAMP(value) AS sample_std_dev FROM company_data."123"
```

### VAR\_POP / VAR\_SAMP

Returns population or sample variance.

```sql theme={null}
SELECT VAR_POP(value) AS variance FROM company_data."123"
SELECT VAR_SAMP(value) AS sample_variance FROM company_data."123"
```

### ARRAY\_AGG

Aggregates values into an array.

```sql theme={null}
SELECT user_id, ARRAY_AGG(tag) AS all_tags
FROM company_data."123"
GROUP BY user_id
```

### STRING\_AGG

Concatenates values into a delimited string.

```sql theme={null}
SELECT user_id, STRING_AGG(category, ', ') AS categories
FROM company_data."123"
GROUP BY user_id
```

### PERCENTILE\_CONT

Returns a percentile value using continuous distribution.

```sql theme={null}
SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY value) AS median
FROM company_data."123"

SELECT PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY response_time) AS p95
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Window Functions" icon="window" href="/nql/functions/window-functions">
    Ranking and analytics
  </Card>

  <Card title="Numeric Functions" icon="hashtag" href="/nql/functions/numeric-functions">
    Math and rounding
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    GROUP BY and HAVING clauses
  </Card>
</CardGroup>


# AI & ML Functions
Source: https://docs.narrative.io/nql/functions/ai-functions

LLM inference and custom model predictions: AI_COMPLETE, CALL_MODEL_FUNCTION

## AI and ML functions

These functions run AI and machine learning operations directly within your query, enabling LLM inference, custom model predictions, and AI-powered data enrichment at scale.

<Note>
  AI and ML functions are currently available on **Snowflake data planes only**. They are not supported on AWS-hosted data planes.
</Note>

### AI\_COMPLETE

Sends a prompt to a large language model and returns the response as a JSON string. Use `AI_COMPLETE` to enrich, classify, or extract structured data from text columns without leaving NQL.

**Syntax:**

```sql theme={null}
AI_COMPLETE(model, prompt, model_parameters, response_format, show_details)
```

**Parameters:**

| Parameter          | Type             | Description                                                                                                                                  |
| ------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`            | STRING           | The model identifier to use (e.g., `'openai-gpt-5'`). Available models depend on your Snowflake Cortex configuration.                        |
| `prompt`           | column reference | A column containing the prompt text. Must be a column reference—string literals are not supported. Build prompts in a CTE or subquery first. |
| `model_parameters` | STRING           | A JSON string of model configuration options. Use `'{}'` for defaults. Supports keys like `temperature` and `max_tokens`.                    |
| `response_format`  | STRING           | A JSON string defining the expected output structure. Contains a `type` field and a `schema` field with a standard JSON Schema definition.   |
| `show_details`     | BOOLEAN          | When `TRUE`, the response includes a `structured_output` array with the full typed response. When `FALSE`, returns a plain string response.  |

**Returns:** STRING — A JSON string containing the model's response.

When `show_details` is `TRUE`, the returned JSON has this structure:

```json theme={null}
{
  "structured_output": [
    {
      "raw_message": {
        // Fields matching your response_format schema
      },
      "type": "json"
    }
  ]
}
```

**Examples:**

Classify a text column using structured output:

```sql theme={null}
SELECT
  AI_COMPLETE(
    'openai-gpt-5',
    company_data."my_dataset".prompt_text,
    '{}',
    '{"type": "json", "schema": {"type": "object", "properties": {"category": {"type": "string"}, "confidence": {"type": "number"}}, "required": ["category", "confidence"], "additionalProperties": false}}',
    TRUE
  ) AS result
FROM company_data."my_dataset"
```

Parse the structured output into individual columns:

```sql theme={null}
SELECT
  CAST(
    PARSE_JSON(result)['structured_output'][0]['raw_message']['category'] AS STRING
  ) AS category,
  CAST(
    PARSE_JSON(result)['structured_output'][0]['raw_message']['confidence'] AS DOUBLE
  ) AS confidence
FROM company_data."enriched_results"
```

Use model parameters to control response behavior:

```sql theme={null}
SELECT
  AI_COMPLETE(
    'openai-gpt-5',
    company_data."my_dataset".prompt_text,
    '{"temperature": 0, "max_tokens": 4096}',
    '{"type": "json", "schema": {"type": "object", "properties": {"summary": {"type": "string"}}, "required": ["summary"], "additionalProperties": false}}',
    TRUE
  ) AS result
FROM company_data."my_dataset"
```

<Tip>
  The `prompt` parameter must be a column reference. To build dynamic prompts from multiple columns, use a CTE to concatenate values into a single prompt column, then pass that column to `AI_COMPLETE`. See the [data enrichment cookbook](/cookbooks/nql/ai-enrichment) for a complete example.
</Tip>

<Note>
  Because `AI_COMPLETE` runs within your Snowflake data plane, your data never leaves your infrastructure. No external API calls are made to model providers. See [Data Privacy in Model Inference](/concepts/model-inference/data-privacy) for details.
</Note>

**Setup requirements:**

Before `AI_COMPLETE` can run, the customer's Snowflake account must grant the Cortex role to the Narrative application and enable cross-region model access. See the [data enrichment cookbook](/cookbooks/nql/ai-enrichment#prerequisites) for the exact `GRANT` statements.

**Common errors:**

| Error                                                               | Cause                                                                   | Resolution                                                                                                                          |
| ------------------------------------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `Insufficient privileges to operate on database role 'CORTEX_USER'` | The Narrative application has not been granted `snowflake.cortex_user`. | Run the grant from the [cookbook setup](/cookbooks/nql/ai-enrichment#prerequisites).                                                |
| `Function AI_COMPLETE does not exist`                               | Query was run against a non-Snowflake data plane.                       | `AI_COMPLETE` is Snowflake-only. Use a [Model Inference job](/guides/sdk/running-model-inference) on other data planes.             |
| `Model '<id>' not available in this region`                         | Cross-region Cortex access is not enabled.                              | Set `CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION'` on the account (see [cookbook setup](/cookbooks/nql/ai-enrichment#prerequisites)). |

**When to use `AI_COMPLETE` vs a Model Inference job:**

| Use `AI_COMPLETE` when...                                   | Use a [Model Inference job](/concepts/model-inference/overview) when... |
| ----------------------------------------------------------- | ----------------------------------------------------------------------- |
| You need row-level inference inline in NQL                  | You are calling inference from the SDK or API, not a query              |
| Your data plane is Snowflake                                | You need to run on a non-Snowflake data plane                           |
| You want to materialize enriched results as a view or table | You need a single on-demand completion (not a batch)                    |

**Related:** [Model Inference Overview](/concepts/model-inference/overview), [Structured Output](/concepts/model-inference/structured-output), [Data Enrichment Cookbook](/cookbooks/nql/ai-enrichment)

### CALL\_MODEL\_FUNCTION

Invokes a function on a custom model registered in the Snowflake ML Model Registry. Use `CALL_MODEL_FUNCTION` to run predictions, embeddings, or other operations from your own trained models directly within NQL.

**Syntax:**

```sql theme={null}
CALL_MODEL_FUNCTION(model_name, model_version, model_function [, arg0, arg1, ...])
```

**Parameters:**

| Parameter         | Type           | Description                                                                                                                                                                               |
| ----------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name`      | STRING         | The model identifier, optionally schema-qualified. Use `'schema.model_name'` or `'"schema"."model_name"'` to specify a schema. If no schema is provided, defaults to the `MODELS` schema. |
| `model_version`   | STRING or NULL | The model version to invoke. Pass `NULL` to use the default version.                                                                                                                      |
| `model_function`  | STRING         | The name of the function to call on the model (e.g., `'PREDICT'`).                                                                                                                        |
| `arg0, arg1, ...` | any            | Zero or more arguments passed to the model function. Typically column references containing the input data.                                                                               |

**Returns:** STRING — A JSON string containing the model function's output.

The return format depends on the model's function signature. Custom models typically return JSON objects:

```json theme={null}
{"OUTPUT": "predicted_value"}
```

**Examples:**

Call a custom model's predict function with a single input column:

```sql theme={null}
SELECT
  CALL_MODEL_FUNCTION(
    'my_schema.product_classifier',
    'v2',
    'PREDICT',
    company_data."products".product_name
  ) AS prediction
FROM company_data."products"
```

Use the default model version by passing `NULL`:

```sql theme={null}
SELECT
  CALL_MODEL_FUNCTION(
    'sentiment_model',
    NULL,
    'PREDICT',
    company_data."reviews".review_text
  ) AS sentiment
FROM company_data."reviews"
```

Parse the model output into typed columns:

```sql theme={null}
WITH predictions AS (
  SELECT
    company_data."products".product_id,
    CALL_MODEL_FUNCTION(
      'my_schema.product_classifier',
      'v2',
      'PREDICT',
      company_data."products".product_name
    ) AS raw_prediction
  FROM company_data."products"
)
SELECT
  predictions.product_id,
  CAST(PARSE_JSON(predictions.raw_prediction)['OUTPUT'] AS STRING) AS predicted_category
FROM predictions
```

Compare outputs across model versions:

```sql theme={null}
SELECT
  company_data."products".product_id,
  CALL_MODEL_FUNCTION(
    'my_schema.classifier',
    'v1',
    'PREDICT',
    company_data."products".product_name
  ) AS v1_prediction,
  CALL_MODEL_FUNCTION(
    'my_schema.classifier',
    'v2',
    'PREDICT',
    company_data."products".product_name
  ) AS v2_prediction
FROM company_data."products"
```

<Tip>
  When no schema is specified in the model name, the function defaults to the `MODELS` schema. To reference a model in a different schema, use dot notation: `'my_schema.model_name'` or quoted identifiers: `'"MY_SCHEMA"."MODEL_NAME"'`.
</Tip>

**Related:** [AI\_COMPLETE](#ai_complete), [Model Inference Overview](/concepts/model-inference/overview)

***

## Related content

<CardGroup>
  <Card title="Data Enrichment with AI" icon="wand-magic-sparkles" href="/cookbooks/nql/ai-enrichment">
    Complete cookbook for enriching data using AI\_COMPLETE
  </Card>

  <Card title="Model Inference" icon="microchip" href="/concepts/model-inference/overview">
    How AI inference works within your data plane
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/concepts/model-inference/structured-output">
    JSON Schema for predictable AI responses
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>
</CardGroup>


# Array & Map Functions
Source: https://docs.narrative.io/nql/functions/array-map-functions

Collection operations: SIZE, ELEMENT_AT, ARRAY_POSITION, FILTER, TRANSFORM, and more

## Array and map functions

### ARRAY

Creates an array from values.

```sql theme={null}
SELECT ARRAY('a', 'b', 'c') -- ['a', 'b', 'c']
SELECT ARRAY(1, 2, 3) AS numbers
```

### SIZE

Returns the number of elements in an array.

```sql theme={null}
SELECT SIZE(tags) AS tag_count FROM company_data."123"
```

### ELEMENT\_AT

Returns an element at a specific index (1-based) or map key.

```sql theme={null}
-- Array (1-based index)
SELECT ELEMENT_AT(ARRAY('a', 'b', 'c'), 2) -- 'b'

-- Map
SELECT ELEMENT_AT(properties, 'color') AS color FROM company_data."123"
```

### ARRAY\_FLATTEN

Flattens nested arrays into a single array.

```sql theme={null}
SELECT ARRAY_FLATTEN(ARRAY(ARRAY(1, 2), ARRAY(3, 4))) -- [1, 2, 3, 4]
```

### ARRAY\_SORT

Sorts array elements.

```sql theme={null}
SELECT ARRAY_SORT(ARRAY(3, 1, 2)) -- [1, 2, 3]
```

### ARRAY\_POSITION

Returns the 0-based index of the first occurrence of an element in an array. Returns `NULL` if the element is not found or if the search value is `NULL`.

**Syntax:** `ARRAY_POSITION(array, value)`

```sql theme={null}
-- Find the position of an element
SELECT ARRAY_POSITION(ARRAY('john', 'jack', 'jill'), 'jack') -- 1

-- Use with a dataset column
SELECT ARRAY_POSITION(tags, 'premium') AS premium_index
FROM company_data."123"

-- Returns NULL when not found
SELECT ARRAY_POSITION(ARRAY('a', 'b', 'c'), 'z') -- NULL
```

<Note>
  `ARRAY_POSITION` uses 0-based indexing, so the first element is at position `0`.
</Note>

### FILTER

Filters elements of an array using a lambda predicate. Returns a new array containing only elements for which the lambda returns `true`.

**Syntax:** `FILTER(array, element -> condition)`

```sql theme={null}
-- Keep only elements matching a condition
SELECT FILTER(tags, t -> t = 'active') AS active_tags
FROM company_data."123"

-- Filter with IN operator
SELECT FILTER(names, n -> n IN ('john', 'jack')) AS matched_names
FROM company_data."123"

-- Combine FILTER with other array functions
SELECT SIZE(FILTER(tags, t -> t = 'premium')) AS premium_count
FROM company_data."123"
```

### TRANSFORM

Applies a function to each element of an array, returning a new array of transformed values.

**Syntax:** `TRANSFORM(array, element -> expression)`

```sql theme={null}
-- Uppercase every element
SELECT TRANSFORM(names, n -> UPPER(n)) AS upper_names
FROM company_data."123"

-- Extract a field from an array of structs
SELECT TRANSFORM(items, x -> x."name") AS item_names
FROM company_data."123"

-- Chain FILTER and TRANSFORM together
SELECT TRANSFORM(FILTER(names, x -> x IN ('john', 'jack')), n -> UPPER(n)) AS result
FROM company_data."123"
```

### EXPLODE

Converts an array into multiple rows (use with LATERAL JOIN).

```sql theme={null}
SELECT t.user_id, tag
FROM company_data."123" t
LATERAL JOIN EXPLODE(t.tags) AS tag
```

### UNNEST

Expands an array into rows (alternative to EXPLODE).

```sql theme={null}
SELECT t.user_id, element
FROM company_data."123" t, UNNEST(t.tags) AS element
```

***

## Related content

<CardGroup>
  <Card title="String Functions" icon="font" href="/nql/functions/string-functions">
    Text manipulation
  </Card>

  <Card title="Conditional Functions" icon="code-branch" href="/nql/functions/conditional-functions">
    IF, COALESCE, NULLIF
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    ARRAY, MAP, and STRUCT type reference
  </Card>
</CardGroup>


# Conditional Functions
Source: https://docs.narrative.io/nql/functions/conditional-functions

Conditional logic: IF, COALESCE, NULLIF, CASE

## Conditional functions

### COALESCE

Returns the first non-null value.

```sql theme={null}
SELECT COALESCE(preferred_email, backup_email, 'unknown') AS email
FROM company_data."123"
```

### NULLIF

Returns NULL if two values are equal.

```sql theme={null}
SELECT NULLIF(status, 'unknown') AS clean_status
FROM company_data."123"
```

### IF

Conditional expression (ternary).

```sql theme={null}
SELECT IF(score > 90, 'A', 'B') AS grade
FROM company_data."123"
```

### CASE

Multi-branch conditional expression.

```sql theme={null}
SELECT
  CASE
    WHEN score >= 90 THEN 'A'
    WHEN score >= 80 THEN 'B'
    WHEN score >= 70 THEN 'C'
    ELSE 'F'
  END AS grade
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="String Functions" icon="font" href="/nql/functions/string-functions">
    Text manipulation
  </Card>

  <Card title="Operators" icon="calculator" href="/nql/operators">
    Comparison and logical operators
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    NULL handling and type coercion
  </Card>
</CardGroup>


# Date & Time Functions
Source: https://docs.narrative.io/nql/functions/date-time-functions

Temporal operations: DATE_TRUNC, DATE_ADD, FORMAT_TIMESTAMP, and more

## Date and time functions

### CURRENT\_DATE

Returns the current date.

```sql theme={null}
SELECT CURRENT_DATE -- '2024-01-15'
SELECT user_id, email, event_date FROM company_data."123" WHERE event_date = CURRENT_DATE
```

### CURRENT\_TIMESTAMP

Returns the current timestamp.

```sql theme={null}
SELECT CURRENT_TIMESTAMP -- '2024-01-15 14:30:00'
```

### DATE\_ADD

Adds days to a date.

```sql theme={null}
SELECT DATE_ADD('2024-01-01', 7) -- '2024-01-08'
SELECT DATE_ADD(start_date, 30) AS end_date FROM company_data."123"
```

### DATE\_SUB

Subtracts days from a date.

```sql theme={null}
SELECT DATE_SUB(CURRENT_DATE, 7) -- 7 days ago
```

### DATEDIFF

Returns the number of days between two dates.

```sql theme={null}
SELECT DATEDIFF('2024-01-10', '2024-01-01') -- 9
SELECT DATEDIFF(end_date, start_date) AS duration FROM company_data."123"
```

### DATE\_TRUNC

Truncates a timestamp to a specified precision.

```sql theme={null}
SELECT DATE_TRUNC('day', '2024-01-15 14:30:00')   -- '2024-01-15 00:00:00'
SELECT DATE_TRUNC('month', '2024-01-15 14:30:00') -- '2024-01-01 00:00:00'
SELECT DATE_TRUNC('year', '2024-01-15 14:30:00')  -- '2024-01-01 00:00:00'
```

**Supported units:** `year`, `month`, `day`, `hour`, `minute`, `second`

### EXTRACT

Extracts a component from a date or timestamp.

```sql theme={null}
SELECT EXTRACT(YEAR FROM '2024-01-15')  -- 2024
SELECT EXTRACT(MONTH FROM '2024-01-15') -- 1
SELECT EXTRACT(DAY FROM '2024-01-15')   -- 15
SELECT EXTRACT(HOUR FROM '2024-01-15 14:30:00') -- 14
```

**Supported parts:** `YEAR`, `MONTH`, `DAY`, `HOUR`, `MINUTE`, `SECOND`, `DAYOFWEEK`, `DAYOFYEAR`

### TO\_DATE

Converts a string to a date.

```sql theme={null}
SELECT TO_DATE('2024-01-15') -- DATE '2024-01-15'
SELECT TO_DATE(date_string) FROM company_data."123"
```

### TO\_TIMESTAMP

Converts a string to a timestamp.

```sql theme={null}
SELECT TO_TIMESTAMP('2024-01-15 10:30:00')
```

### FROM\_UNIXTIME

Converts Unix epoch seconds to timestamp.

```sql theme={null}
SELECT FROM_UNIXTIME(1705312200) -- '2024-01-15 10:30:00'
SELECT FROM_UNIXTIME(epoch_seconds) AS event_time FROM company_data."123"
```

### UNIX\_TIMESTAMP

Converts a timestamp to Unix epoch seconds.

```sql theme={null}
SELECT UNIX_TIMESTAMP('2024-01-15 10:30:00') -- 1705312200
SELECT UNIX_TIMESTAMP(event_timestamp) AS epoch FROM company_data."123"
```

### FORMAT\_TIMESTAMP

Formats a timestamp column using a specified pattern string. Returns the formatted timestamp as a string.

```sql theme={null}
SELECT
  FORMAT_TIMESTAMP('YYYY-MM-DD', timestamp_col) AS date_only,
  FORMAT_TIMESTAMP('HH24:MI:SS', timestamp_col) AS time_only,
  FORMAT_TIMESTAMP('YYYY-MM-DDTHH24:MI:SS', timestamp_col) AS iso_no_tz,
  FORMAT_TIMESTAMP('YYYY-MM-DDTHH24:MI:SS TZH', timestamp_col) AS iso_with_tz,
  FORMAT_TIMESTAMP('YYYY-MM-DDTHH24:MI:SS TZH:TZM', timestamp_col) AS iso_with_full_tz
FROM company_data."123"
```

**Common patterns:**

* `YYYY-MM-DD` — Date only (2024-01-15)
* `HH24:MI:SS` — 24-hour time (14:30:00)
* `YYYY-MM-DDTHH24:MI:SS` — ISO 8601 without timezone
* `YYYY-MM-DDTHH24:MI:SS TZH:TZM` — ISO 8601 with full timezone offset

See [Snowflake date/time format elements](https://docs.snowflake.com/en/sql-reference/date-time-input-output#about-the-elements-used-in-input-and-output-formats) for all supported pattern elements.

***

## Related content

<CardGroup>
  <Card title="Numeric Functions" icon="hashtag" href="/nql/functions/numeric-functions">
    Math and rounding
  </Card>

  <Card title="String Functions" icon="font" href="/nql/functions/string-functions">
    Text manipulation
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    DATE and TIMESTAMP type reference
  </Card>
</CardGroup>


# Geospatial Functions
Source: https://docs.narrative.io/nql/functions/geospatial-functions

Geographic calculations: ST_GeoHash, ST_Distance, and more

## Geospatial functions

NQL includes Sedona-based geospatial functions for location data.

### ST\_SafePoint

Creates a point geometry from longitude and latitude.

```sql theme={null}
SELECT ST_SafePoint(longitude, latitude) AS location
FROM company_data."123"
```

### ST\_Circle

Creates a circular geometry around a point.

```sql theme={null}
SELECT ST_Circle(ST_SafePoint(longitude, latitude), radius_meters) AS area
FROM company_data."123"
```

### ST\_DistanceSphere

Calculates the distance between two points on a sphere (in meters).

```sql theme={null}
SELECT ST_DistanceSphere(point1, point2) AS distance_meters
FROM company_data."123"
```

### ST\_GeoHash

Converts a point to a geohash string.

```sql theme={null}
SELECT ST_GeoHash(ST_SafePoint(longitude, latitude), 6) AS geohash
FROM company_data."123"
```

### ST\_H3CellIDs

Returns H3 hexagon cell IDs covering a geometry.

```sql theme={null}
SELECT ST_H3CellIDs(geometry, resolution) AS h3_cells
FROM company_data."123"
```

### ST\_H3KRing

Returns neighboring H3 cells within k rings.

```sql theme={null}
SELECT ST_H3KRing(h3_cell, k) AS neighbors
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Numeric Functions" icon="hashtag" href="/nql/functions/numeric-functions">
    Math and rounding
  </Card>

  <Card title="String Functions" icon="font" href="/nql/functions/string-functions">
    Text manipulation
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Location Data" icon="location-dot" href="/concepts/data-formats/location-data">
    Working with geographic data
  </Card>
</CardGroup>


# Functions
Source: https://docs.narrative.io/nql/functions/index

Overview of built-in and Narrative-specific functions in NQL

NQL includes standard SQL functions plus Narrative-specific functions for data collaboration. Functions are organized by category.

<CardGroup>
  <Card title="String Functions" icon="font" href="/nql/functions/string-functions">
    Text manipulation: UPPER, LOWER, CONCAT, SUBSTRING, TRIM, and more
  </Card>

  <Card title="Date & Time Functions" icon="clock" href="/nql/functions/date-time-functions">
    Temporal operations: DATE\_TRUNC, DATE\_ADD, FORMAT\_TIMESTAMP, and more
  </Card>

  <Card title="Numeric Functions" icon="hashtag" href="/nql/functions/numeric-functions">
    Math operations: ROUND, ABS, CEIL, FLOOR, SQRT, and more
  </Card>

  <Card title="Aggregate Functions" icon="chart-bar" href="/nql/functions/aggregate-functions">
    GROUP BY computations: COUNT, SUM, AVG, MIN, MAX, and more
  </Card>

  <Card title="Array & Map Functions" icon="brackets-curly" href="/nql/functions/array-map-functions">
    Collection operations: FILTER, TRANSFORM, SIZE, ELEMENT\_AT, and more
  </Card>

  <Card title="Window Functions" icon="window" href="/nql/functions/window-functions">
    Ranking and analytics: ROW\_NUMBER, RANK, LAG, LEAD, and more
  </Card>

  <Card title="Geospatial Functions" icon="map-location-dot" href="/nql/functions/geospatial-functions">
    Geographic calculations: ST\_GeoHash, ST\_Distance, and more
  </Card>

  <Card title="Narrative Functions" icon="gear" href="/nql/functions/narrative-functions">
    Platform-specific: DELTA, budget functions, and more
  </Card>

  <Card title="AI & ML Functions" icon="brain" href="/nql/functions/ai-functions">
    LLM inference and custom models: AI\_COMPLETE, CALL\_MODEL\_FUNCTION
  </Card>

  <Card title="Conditional Functions" icon="code-branch" href="/nql/functions/conditional-functions">
    Conditional logic: IF, COALESCE, NULLIF, CASE
  </Card>
</CardGroup>

## Related content

<CardGroup>
  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Types accepted and returned by functions
  </Card>

  <Card title="Operators" icon="calculator" href="/nql/operators">
    Comparison, logical, and arithmetic operators
  </Card>

  <Card title="Common Query Patterns" icon="book" href="/cookbooks/nql/common-queries">
    Real-world examples using these functions
  </Card>
</CardGroup>


# Narrative Functions
Source: https://docs.narrative.io/nql/functions/narrative-functions

Platform-specific functions: DELTA, budget functions, and more

## Narrative-specific functions

These functions are specific to the Narrative platform.

### NORMALIZE\_EMAIL

Normalizes an email address for consistent matching.

```sql theme={null}
SELECT NORMALIZE_EMAIL('John.Doe+test@Gmail.COM') -- 'johndoe@gmail.com'
SELECT NORMALIZE_EMAIL(email) AS normalized_email FROM company_data."123"
```

Normalization includes:

* Lowercase conversion
* Removal of plus-addressing
* Provider-specific handling (Gmail dot removal, etc.)

### NORMALIZE\_PHONE

Normalizes a phone number to a standard format.

```sql theme={null}
SELECT NORMALIZE_PHONE('NATIONAL', '+15145555555', null) AS normalized
SELECT NORMALIZE_PHONE('E164', phone, 'US') AS e164_phone FROM company_data."123"
```

### HASH

Generates a hash from one or more values.

```sql theme={null}
SELECT HASH('value1', 'value2') AS combined_hash
SELECT HASH(user_id, email) AS user_hash FROM company_data."123"
```

### MD5 / SHA1 / SHA2

Standard cryptographic hash functions.

```sql theme={null}
SELECT MD5('hello')           -- MD5 hash
SELECT SHA1('hello')          -- SHA-1 hash
SELECT SHA2('hello', 256)     -- SHA-256 hash
```

### HMAC\_SHA256

HMAC using SHA-256.

```sql theme={null}
SELECT HMAC_SHA256('message', 'secret_key') AS hmac
```

### ADDRESS\_HASHES

Generates hash variants for address matching.

```sql theme={null}
SELECT ADDRESS_HASHES('123 Main St', 'New York', 'NY', '10001') AS hashes
```

### NARRATIVE\_ID\_ENCODE

Encodes a clear text identifier into a Narrative ID within a specific encoding space. Narrative IDs are per-partner pseudonymous identifiers that enable privacy-safe data collaboration.

**Syntax:**

```sql theme={null}
NARRATIVE_ID_ENCODE(raw_identifier, key_id)
```

**Parameters:**

| Parameter        | Type   | Description                                                            |
| ---------------- | ------ | ---------------------------------------------------------------------- |
| `raw_identifier` | string | The clear text identifier to encode (email, hashed email, phone, etc.) |
| `key_id`         | string | The encoding key ID specifying the target encoding space               |

**Returns:** string - The encoded Narrative ID

**Examples:**

```sql theme={null}
-- Encode an email address
SELECT NARRATIVE_ID_ENCODE(LOWER(TRIM(email)), 'your_org_key_id') AS narrative_id
FROM company_data."123"

-- Encode an already-hashed email
SELECT NARRATIVE_ID_ENCODE(email_sha256, 'your_org_key_id') AS narrative_id
FROM company_data."123"

-- Create a dataset with encoded identifiers
SELECT
  NARRATIVE_ID_ENCODE(email_sha256, 'your_org_key_id') AS narrative_id,
  event_timestamp,
  event_properties
FROM company_data."123"
WHERE event_timestamp >= '2024-01-01'
```

<Note>
  The same identifier encoded with different `key_id` values produces different Narrative IDs. Always normalize identifiers (lowercase, trim whitespace) before encoding for consistent matching.
</Note>

**Related:** [Narrative ID Concepts](/concepts/security/narrative-id), [Using Narrative ID](/guides/collaboration/using-narrative-id)

### NARRATIVE\_ID\_TRANSLATE

Translates a Narrative ID from one partner's encoding space to another. This enables secure data collaboration by converting identifiers without exposing the underlying clear text values.

**Syntax:**

```sql theme={null}
NARRATIVE_ID_TRANSLATE(narrative_id, target_key_id)
```

**Parameters:**

| Parameter       | Type   | Description                                                 |
| --------------- | ------ | ----------------------------------------------------------- |
| `narrative_id`  | string | An existing Narrative ID to translate                       |
| `target_key_id` | string | The encoding key ID for the target partner's encoding space |

**Returns:** string - The translated Narrative ID in the target encoding space

**Examples:**

```sql theme={null}
-- Translate Narrative IDs for a partner collaboration
SELECT
  NARRATIVE_ID_TRANSLATE(narrative_id, 'partner_b_key_id') AS partner_b_narrative_id,
  customer_segment,
  activity_score
FROM company_data."123"

-- Join across partner datasets using translation
SELECT
  a.campaign_name,
  a.impressions,
  b.conversions
FROM partner_a."ad_data" a
JOIN partner_b."conversion_data" b
  ON NARRATIVE_ID_TRANSLATE(a.narrative_id, 'partner_b_key_id') = b.narrative_id

-- Prepare data for multiple partners
SELECT
  narrative_id AS original_id,
  NARRATIVE_ID_TRANSLATE(narrative_id, 'partner_a_key_id') AS partner_a_id,
  NARRATIVE_ID_TRANSLATE(narrative_id, 'partner_b_key_id') AS partner_b_id,
  audience_segment
FROM company_data."123"
```

**Use case:**

When sharing data with a partner, translate your Narrative IDs to their encoding space so they can match against their own Narrative IDs. This enables cross-organization matching without exposing underlying identifiers.

**Related:** [Narrative ID Concepts](/concepts/security/narrative-id), [Using Narrative ID](/guides/collaboration/using-narrative-id)

### UNIVERSE\_SAMPLE

Deterministic sampling based on an identifier.

```sql theme={null}
SELECT user_id, email, created_at
FROM company_data."123"
WHERE UNIVERSE_SAMPLE(user_id, 0.1)  -- 10% sample
```

### DETERMINISTIC\_RAND

Generates a deterministic random number from a seed.

```sql theme={null}
SELECT DETERMINISTIC_RAND(user_id) AS random_value
FROM company_data."123"
```

### PARSE\_JSON / TRY\_PARSE\_JSON

Parses a JSON string.

```sql theme={null}
SELECT PARSE_JSON('{"key": "value"}') AS parsed
SELECT TRY_PARSE_JSON(maybe_json) AS safe_parsed  -- Returns NULL on error
FROM company_data."123"
```

### OBJECT\_REMOVE\_NULLS

Returns a struct with null-valued fields removed. On AWS hosted data planes, the function is accepted for cross-platform query compatibility but operates as a no-op.

```sql theme={null}
SELECT OBJECT_REMOVE_NULLS(NAMED_STRUCT('a', 1, 'b', NULL, 'c', 3))
-- Input schema:  STRUCT<a: INT, b: INT, c: INT>
-- Output schema: STRUCT<a: INT, c: INT>
-- Result: {a: 1, c: 3}
```

### TOKENIZE

Tokenizes text using a specified model.

```sql theme={null}
SELECT TOKENIZE(text_column) AS tokens
FROM company_data."123"
```

### TOKENIZE\_SIZE

Returns the token count for text.

```sql theme={null}
SELECT TOKENIZE_SIZE(text_column) AS token_count
FROM company_data."123"
```

### EMBED\_TEXT\_768

Generates a 768-dimensional text embedding.

```sql theme={null}
SELECT EMBED_TEXT_768(description) AS embedding
FROM company_data."123"
```

### COSINE\_SIMILARITY

Calculates cosine similarity between two vectors.

```sql theme={null}
SELECT COSINE_SIMILARITY(embedding1, embedding2) AS similarity
FROM company_data."123"
```

### TEXT\_SEARCH

Scores text relevance against a search query.

```sql theme={null}
SELECT TEXT_SEARCH(description, 'machine learning') AS relevance_score
FROM company_data."123"
```

### UUID\_GENERATE

Generates a random UUID.

```sql theme={null}
SELECT UUID_GENERATE() AS new_id
```

***

## Related content

<CardGroup>
  <Card title="Aggregate Functions" icon="chart-bar" href="/nql/functions/aggregate-functions">
    GROUP BY computations
  </Card>

  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    DELTA tables and budget clauses
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="NQL Design Philosophy" icon="lightbulb" href="/concepts/nql/design-philosophy">
    Why NQL extends standard SQL
  </Card>
</CardGroup>


# Numeric Functions
Source: https://docs.narrative.io/nql/functions/numeric-functions

Math operations: ROUND, ABS, CEIL, FLOOR, SQRT, and more

## Numeric functions

### ABS

Returns the absolute value.

```sql theme={null}
SELECT ABS(-5) -- 5
SELECT ABS(balance) AS absolute_balance FROM company_data."123"
```

### CEIL / CEILING

Rounds up to the nearest integer.

```sql theme={null}
SELECT CEIL(4.2)  -- 5
SELECT CEIL(-4.2) -- -4
```

### FLOOR

Rounds down to the nearest integer.

```sql theme={null}
SELECT FLOOR(4.8)  -- 4
SELECT FLOOR(-4.2) -- -5
```

### ROUND

Rounds to a specified number of decimal places.

```sql theme={null}
SELECT ROUND(4.567)     -- 5
SELECT ROUND(4.567, 2)  -- 4.57
SELECT ROUND(4.567, 0)  -- 5
```

### POWER / POW

Raises a number to a power.

```sql theme={null}
SELECT POWER(2, 3) -- 8
SELECT POW(10, 2)  -- 100
```

### SQRT

Returns the square root.

```sql theme={null}
SELECT SQRT(16) -- 4
SELECT SQRT(2)  -- 1.414...
```

### LN

Returns the natural logarithm.

```sql theme={null}
SELECT LN(2.718281828) -- ~1
```

### LOG10

Returns the base-10 logarithm.

```sql theme={null}
SELECT LOG10(100) -- 2
```

### EXP

Returns e raised to a power.

```sql theme={null}
SELECT EXP(1) -- 2.718...
```

### MOD

Returns the remainder after division.

```sql theme={null}
SELECT MOD(10, 3) -- 1
SELECT MOD(user_id, 100) AS bucket FROM company_data."123"
```

### GREATEST

Returns the largest value from a list.

```sql theme={null}
SELECT GREATEST(1, 5, 3) -- 5
SELECT GREATEST(score1, score2, score3) AS max_score FROM company_data."123"
```

### LEAST

Returns the smallest value from a list.

```sql theme={null}
SELECT LEAST(1, 5, 3) -- 1
SELECT LEAST(price1, price2) AS min_price FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Aggregate Functions" icon="chart-bar" href="/nql/functions/aggregate-functions">
    GROUP BY computations
  </Card>

  <Card title="Date & Time Functions" icon="clock" href="/nql/functions/date-time-functions">
    Temporal operations
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    LONG, DOUBLE, and DECIMAL type reference
  </Card>
</CardGroup>


# String Functions
Source: https://docs.narrative.io/nql/functions/string-functions

Text manipulation functions: UPPER, LOWER, CONCAT, SUBSTRING, TRIM, and more

## String functions

### UPPER

Converts a string to uppercase.

```sql theme={null}
SELECT UPPER('hello world') -- 'HELLO WORLD'
SELECT UPPER(name) AS uppercase_name FROM company_data."123"
```

### LOWER

Converts a string to lowercase.

```sql theme={null}
SELECT LOWER('HELLO WORLD') -- 'hello world'
SELECT LOWER(email) AS lowercase_email FROM company_data."123"
```

### TRIM

Removes leading and trailing whitespace.

```sql theme={null}
SELECT TRIM('  hello  ') -- 'hello'
SELECT TRIM(user_input) AS cleaned_input FROM company_data."123"
```

### LTRIM / RTRIM

Removes leading (LTRIM) or trailing (RTRIM) whitespace.

```sql theme={null}
SELECT LTRIM('  hello') -- 'hello'
SELECT RTRIM('hello  ') -- 'hello'
```

### LENGTH

Returns the number of characters in a string.

```sql theme={null}
SELECT LENGTH('hello') -- 5
SELECT LENGTH(description) AS desc_length FROM company_data."123"
```

### CONCAT

Concatenates multiple strings.

```sql theme={null}
SELECT CONCAT('Hello', ' ', 'World') -- 'Hello World'
SELECT CONCAT(first_name, ' ', last_name) AS full_name FROM company_data."123"
```

### CONCAT\_WS

Concatenates strings with a separator.

```sql theme={null}
SELECT CONCAT_WS('-', 'a', 'b', 'c') -- 'a-b-c'
SELECT CONCAT_WS(', ', city, state, country) AS location FROM company_data."123"
```

### SUBSTRING

Extracts a portion of a string.

```sql theme={null}
-- SUBSTRING(string, start_position, length)
SELECT SUBSTRING('hello world', 1, 5) -- 'hello'
SELECT SUBSTRING('hello world', 7, 5) -- 'world'
```

<Note>
  String positions are 1-based, not 0-based.
</Note>

### LEFT / RIGHT

Returns leftmost or rightmost characters.

```sql theme={null}
SELECT LEFT('hello', 2)  -- 'he'
SELECT RIGHT('hello', 2) -- 'lo'
```

### REPLACE

Replaces occurrences of a substring.

```sql theme={null}
SELECT REPLACE('hello', 'l', 'x') -- 'hexxo'
SELECT REPLACE(phone, '-', '') AS clean_phone FROM company_data."123"
```

### SPLIT

Splits a string into an array.

```sql theme={null}
SELECT SPLIT('a,b,c', ',') -- ['a', 'b', 'c']
SELECT SPLIT(tags, '|') AS tag_array FROM company_data."123"
```

### LPAD / RPAD

Pads a string to a specified length.

```sql theme={null}
SELECT LPAD('5', 3, '0')    -- '005'
SELECT RPAD('hello', 10, '.') -- 'hello.....'
```

### REGEXP\_EXTRACT

Extracts a substring matching a regular expression.

```sql theme={null}
SELECT REGEXP_EXTRACT('abc123def', '[0-9]+') -- '123'
SELECT REGEXP_EXTRACT(email, '@(.+)$') AS domain FROM company_data."123"
```

### LEVENSHTEIN

Calculates the edit distance between two strings.

```sql theme={null}
SELECT LEVENSHTEIN('hello', 'hallo') -- 1
SELECT LEVENSHTEIN(input_name, canonical_name) AS distance FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Date & Time Functions" icon="clock" href="/nql/functions/date-time-functions">
    Temporal operations
  </Card>

  <Card title="Conditional Functions" icon="code-branch" href="/nql/functions/conditional-functions">
    IF, COALESCE, NULLIF
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    STRING type reference
  </Card>
</CardGroup>


# Window Functions
Source: https://docs.narrative.io/nql/functions/window-functions

Ranking and analytics: ROW_NUMBER, RANK, LAG, LEAD, and more

## Window functions

Window functions perform calculations across a set of rows related to the current row.

### ROW\_NUMBER

Assigns a unique sequential number to each row within a partition.

```sql theme={null}
SELECT
  user_id,
  event_timestamp,
  ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY event_timestamp) AS event_seq
FROM company_data."123"
```

### RANK

Assigns a rank with gaps for ties.

```sql theme={null}
SELECT
  user_id,
  score,
  RANK() OVER (ORDER BY score DESC) AS rank
FROM company_data."123"
```

### DENSE\_RANK

Assigns a rank without gaps for ties.

```sql theme={null}
SELECT
  user_id,
  score,
  DENSE_RANK() OVER (ORDER BY score DESC) AS dense_rank
FROM company_data."123"
```

### PERCENT\_RANK

Returns the relative rank as a percentage (0 to 1).

```sql theme={null}
SELECT
  user_id,
  score,
  PERCENT_RANK() OVER (ORDER BY score) AS percentile
FROM company_data."123"
```

### LAG

Returns the value from a previous row.

```sql theme={null}
SELECT
  event_date,
  value,
  LAG(value, 1) OVER (ORDER BY event_date) AS previous_value,
  value - LAG(value, 1) OVER (ORDER BY event_date) AS change
FROM company_data."123"
```

### LEAD

Returns the value from a following row.

```sql theme={null}
SELECT
  event_date,
  value,
  LEAD(value, 1) OVER (ORDER BY event_date) AS next_value
FROM company_data."123"
```

### SUM / AVG / COUNT (windowed)

Aggregate functions can be used as window functions.

```sql theme={null}
SELECT
  event_date,
  amount,
  SUM(amount) OVER (ORDER BY event_date) AS running_total,
  AVG(amount) OVER (PARTITION BY category) AS category_avg
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="Aggregate Functions" icon="chart-bar" href="/nql/functions/aggregate-functions">
    GROUP BY computations
  </Card>

  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    QUALIFY clause for window function filtering
  </Card>

  <Card title="All Functions" icon="function" href="/nql/functions/index">
    Browse all function categories
  </Card>

  <Card title="Common Query Patterns" icon="book" href="/cookbooks/nql/common-queries">
    Real-world deduplication examples
  </Card>
</CardGroup>


# Explicit Column Selection
Source: https://docs.narrative.io/nql/general/explicit-columns

Why NQL requires explicit column lists instead of wildcards

NQL requires you to explicitly list the columns you want to select. Wildcard syntax (`SELECT *` and `COUNT(*)`) is not supported.

## Why explicit columns are required

### Cost transparency

NQL queries can have costs associated with them based on the data accessed. By requiring explicit column lists, you know exactly what data you're requesting and can better predict query costs. Implicit column selection through `*` would obscure this.

### Governance compliance

Narrative's governance engine evaluates every query to determine what data can be accessed based on permissions and access rules. Explicit column lists allow the governance engine to make precise decisions about query eligibility. With wildcard syntax, the engine couldn't determine ahead of time which columns would be accessed.

### Query plan optimization

This is especially important for [Rosetta Stone](/concepts/rosetta-stone/overview) queries. When you query Rosetta Stone, NQL compiles your query into SQL that pulls from multiple underlying tables.

Consider this query:

```sql theme={null}
SELECT
  rs.email,
  rs.phone
FROM narrative.rosetta_stone rs
```

NQL compiles this into something like:

```sql theme={null}
SELECT email, NULL AS phone FROM email_table
UNION ALL
SELECT NULL AS email, phone FROM phone_table
```

With explicit columns, the query planner knows exactly which tables to include in the union. Tables that don't contain any of the requested columns can be pruned entirely.

If wildcards were allowed, a query like `SELECT * FROM narrative.rosetta_stone` would need to include **every column from every table** in the Rosetta Stone catalog. This would create massive query plans with potentially hundreds of tables in the union, even when you only need a few fields.

***

## Migration from SQL

If you're coming from standard SQL, here's how to adapt your queries:

### Replace SELECT \*

Instead of selecting all columns, list the specific columns you need:

```sql theme={null}
-- Instead of this (not supported):
SELECT * FROM company_data."123"

-- Write this:
SELECT
  user_id,
  email,
  created_at
FROM company_data."123"
```

### Replace COUNT(\*)

For counting rows, use `COUNT(1)`:

```sql theme={null}
-- Instead of this (not supported):
SELECT COUNT(*) FROM company_data."123"

-- Write this:
SELECT COUNT(1) AS row_count FROM company_data."123"
```

When you specifically want to count non-null values in a column, use `COUNT(column_name)`:

```sql theme={null}
-- Count rows where email is not null
SELECT COUNT(email) AS emails_present FROM company_data."123"
```

### Replace SELECT t1.*, t2.*

In joins, enumerate the columns from each table:

```sql theme={null}
-- Instead of this (not supported):
SELECT t1.*, t2.*
FROM company_data."123" t1
JOIN company_data."456" t2 ON t1.id = t2.id

-- Write this:
SELECT
  t1.user_id,
  t1.email,
  t2.order_id,
  t2.amount
FROM company_data."123" t1
JOIN company_data."456" t2 ON t1.id = t2.id
```

***

## Benefits of explicit selection

While requiring explicit columns may seem like extra work, it provides several benefits:

* **Clearer intent**: Your query documents exactly what data you need
* **Faster queries**: Only requested columns are processed and transferred
* **Easier maintenance**: Changes to the underlying schema won't unexpectedly affect your query results
* **Better cost control**: You pay only for the data you actually use

***

## Related content

<CardGroup>
  <Card title="NQL vs SQL" icon="code-compare" href="/concepts/nql/sql-comparison">
    Other differences between NQL and standard SQL
  </Card>

  <Card title="NQL Syntax" icon="code" href="/nql/general/syntax">
    Complete query structure and grammar
  </Card>

  <Card title="Rosetta Stone" icon="language" href="/concepts/rosetta-stone/overview">
    Understanding identity resolution across datasets
  </Card>

  <Card title="Query Optimization" icon="bolt" href="/guides/nql/query-optimization">
    Tips for writing efficient queries
  </Card>
</CardGroup>


# Reserved Keywords
Source: https://docs.narrative.io/nql/general/reserved-keywords

Keywords that require quoting when used as identifiers

Reserved keywords have special meaning in NQL and cannot be used as unquoted identifiers (table names, column names, aliases). To use a reserved keyword as an identifier, enclose it in double quotes.

## Using reserved keywords as identifiers

```sql theme={null}
-- This fails: "type" is a reserved keyword
SELECT type FROM company_data."123"

-- This works: quoted identifier
SELECT "type" FROM company_data."123"

-- Column alias using reserved keyword
SELECT category AS "group" FROM company_data."123"
```

***

## Reserved keywords list

The following keywords are reserved in NQL:

### A-C

| Keyword             | Category   |
| ------------------- | ---------- |
| `ALL`               | Clause     |
| `AND`               | Operator   |
| `ANY`               | Clause     |
| `ARRAY`             | Type       |
| `AS`                | Clause     |
| `ASC`               | Clause     |
| `BETWEEN`           | Operator   |
| `BIGINT`            | Type       |
| `BOOLEAN`           | Type       |
| `BUDGET`            | Narrative  |
| `BY`                | Clause     |
| `CALENDAR_DAY`      | Narrative  |
| `CALENDAR_MONTH`    | Narrative  |
| `CASE`              | Expression |
| `CAST`              | Function   |
| `CREATE`            | Statement  |
| `CROSS`             | Join       |
| `CURRENT_DATE`      | Function   |
| `CURRENT_TIMESTAMP` | Function   |

### D-G

| Keyword          | Category   |
| ---------------- | ---------- |
| `DATE`           | Type       |
| `DAY`            | Interval   |
| `DECIMAL`        | Type       |
| `DELETE`         | Statement  |
| `DELTA`          | Narrative  |
| `DESC`           | Clause     |
| `DESCRIPTION`    | Narrative  |
| `DISPLAY_NAME`   | Narrative  |
| `DISTINCT`       | Clause     |
| `DOUBLE`         | Type       |
| `ELSE`           | Expression |
| `END`            | Expression |
| `EXISTS`         | Operator   |
| `EXPLAIN`        | Statement  |
| `EXPIRE`         | Narrative  |
| `EXTENDED_STATS` | Narrative  |
| `FALSE`          | Literal    |
| `FROM`           | Clause     |
| `FULL`           | Join       |
| `GROUP`          | Clause     |

### H-L

| Keyword    | Category  |
| ---------- | --------- |
| `HAVING`   | Clause    |
| `HOUR`     | Interval  |
| `IF`       | Function  |
| `IN`       | Operator  |
| `INNER`    | Join      |
| `INSERT`   | Statement |
| `INT`      | Type      |
| `INTEGER`  | Type      |
| `INTERVAL` | Type      |
| `INTO`     | Clause    |
| `IS`       | Operator  |
| `JOIN`     | Join      |
| `LATERAL`  | Join      |
| `LEFT`     | Join      |
| `LIKE`     | Operator  |
| `LIMIT`    | Clause    |
| `LONG`     | Type      |

### M-O

| Keyword        | Category  |
| -------------- | --------- |
| `MAP`          | Type      |
| `MATCHED`      | Merge     |
| `MATERIALIZED` | Statement |
| `MERGE`        | Statement |
| `MINUTE`       | Interval  |
| `MONTH`        | Interval  |
| `NOT`          | Operator  |
| `NULL`         | Literal   |
| `OFFSET`       | Clause    |
| `ON`           | Clause    |
| `OR`           | Operator  |
| `ORDER`        | Clause    |
| `OUTER`        | Join      |
| `OVER`         | Window    |

### P-R

| Keyword            | Category  |
| ------------------ | --------- |
| `PARTITION`        | Window    |
| `PARTITIONED_BY`   | Narrative |
| `PER`              | Narrative |
| `QUALIFY`          | Clause    |
| `REFRESH_SCHEDULE` | Narrative |
| `RIGHT`            | Join      |
| `ROWS`             | Clause    |

### S-T

| Keyword     | Category   |
| ----------- | ---------- |
| `SECOND`    | Interval   |
| `SELECT`    | Statement  |
| `SET`       | Statement  |
| `SOURCE`    | Merge      |
| `STATUS`    | Narrative  |
| `STRING`    | Type       |
| `STRUCT`    | Type       |
| `TABLE`     | Statement  |
| `TAGS`      | Narrative  |
| `TARGET`    | Merge      |
| `THEN`      | Expression |
| `TIMESTAMP` | Type       |
| `TRUE`      | Literal    |

### U-Z

| Keyword      | Category   |
| ------------ | ---------- |
| `UNION`      | Clause     |
| `UNNEST`     | Function   |
| `UPDATE`     | Statement  |
| `USD`        | Narrative  |
| `VALUES`     | Clause     |
| `VARCHAR`    | Type       |
| `VIEW`       | Statement  |
| `WHEN`       | Expression |
| `WHERE`      | Clause     |
| `WITH`       | Clause     |
| `WITHIN`     | Clause     |
| `WRITE_MODE` | Narrative  |
| `YEAR`       | Interval   |

***

## Categories

| Category   | Description                          |
| ---------- | ------------------------------------ |
| Clause     | SQL clause keywords                  |
| Expression | CASE/conditional expression keywords |
| Function   | Built-in function names              |
| Interval   | Date/time interval units             |
| Join       | JOIN clause keywords                 |
| Literal    | Literal value keywords               |
| Merge      | MERGE statement keywords             |
| Narrative  | Narrative-specific extensions        |
| Operator   | Comparison and logical operators     |
| Statement  | Statement type keywords              |
| Type       | Data type keywords                   |
| Window     | Window function keywords             |

***

## Best practices

### Quote numeric dataset IDs

Numeric dataset IDs must be quoted because they are numeric. Dataset names are identifiers and don't need quoting — unless the name is a reserved keyword or contains special characters, in which case wrap it in double quotes:

```sql theme={null}
-- Correct
FROM company_data."123"
FROM company_data.my_dataset
FROM company_data."select"   -- name that is a reserved keyword

-- Incorrect
FROM company_data.123
```

### Quote ambiguous column names

When column names might conflict with keywords:

```sql theme={null}
SELECT
  "type",
  "group",
  "order",
  "value"
FROM company_data."123"
```

### Use aliases for clarity

Assign clear aliases to avoid quoting in subsequent references:

```sql theme={null}
SELECT
  "type" AS identifier_type,
  "group" AS category_group
FROM company_data."123"
WHERE identifier_type = 'email'
```

### Prefer bracket notation for property access

When accessing struct fields, bracket notation avoids reserved keyword conflicts:

```sql theme={null}
-- Bracket notation: no quoting needed for reserved keywords
SELECT
  data['type'],
  data['value'],
  data['group']
FROM company_data."123"

-- Dot notation: reserved keywords must be double-quoted
SELECT
  data."type",
  data."value",
  data."group"
FROM company_data."123"
```

Both syntaxes work, but bracket notation is simpler when field names may include reserved keywords.

***

## Related content

<CardGroup>
  <Card title="NQL Syntax" icon="code" href="/nql/general/syntax">
    Query structure and grammar reference
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Type keywords and declarations
  </Card>
</CardGroup>


# NQL Syntax
Source: https://docs.narrative.io/nql/general/syntax

Overview of NQL statement types and general grammar

NQL (Narrative Query Language) uses SQL-like syntax with extensions for data collaboration. This page provides a high-level overview of statement types and links to detailed references.

## Statement types

NQL supports two main statement types:

| Statement                  | Purpose                              | Reference                                                          |
| -------------------------- | ------------------------------------ | ------------------------------------------------------------------ |
| `SELECT`                   | Execute a query and return results   | [SELECT](/nql/commands/select)                                     |
| `CREATE MATERIALIZED VIEW` | Create a stored dataset from a query | [CREATE MATERIALIZED VIEW](/nql/commands/create-materialized-view) |

## Query Templates

NQL also supports parameterized queries through [query templates](/nql/commands/query-templates), which use placeholder syntax to create reusable, customizable queries.

## Key syntax rules

* **Explicit columns required**: NQL does not support `SELECT *` or `COUNT(*)`. See [Explicit Column Selection](/nql/general/explicit-columns).
* **Reserved keywords**: Certain keywords must be quoted when used as identifiers. See [Reserved Keywords](/nql/general/reserved-keywords).
* **Fully qualified names**: Use fully qualified column names (e.g., `company_data."123".column_name` or `company_data.my_dataset.column_name`) to avoid ambiguity, especially in joins.
* **Dataset references**: Reference a dataset in the `company_data` schema by its numeric ID (`company_data."123"`) or its name (`company_data.my_dataset`). Numeric IDs must be quoted because they're numeric; names need quoting only when they're a reserved keyword or contain special characters. See [NQL vs. SQL](/concepts/nql/sql-comparison) for when each form is used and why.

## Related content

<CardGroup>
  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    Full SELECT statement syntax and clauses
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Primitive and complex types supported in NQL
  </Card>

  <Card title="Operators" icon="calculator" href="/nql/operators">
    Comparison, logical, and arithmetic operators
  </Card>

  <Card title="Functions" icon="function" href="/nql/functions/index">
    Built-in and Narrative-specific functions
  </Card>

  <Card title="NQL vs. SQL" icon="right-left" href="/concepts/nql/sql-comparison">
    How NQL differs from standard SQL, including dataset references
  </Card>
</CardGroup>


# NQL Reference
Source: https://docs.narrative.io/nql/index

Complete reference documentation for Narrative Query Language

NQL (Narrative Query Language) is a SQL-like language with extensions for data collaboration. This reference covers all syntax, types, functions, and operators supported by NQL.

<Note>
  Looking to learn NQL? Start with [NQL Design Philosophy](/concepts/nql/design-philosophy) for concepts or [Write Your First NQL Query](/getting-started/first-nql-query) for a hands-on tutorial.
</Note>

## SQL Commands

Write and execute queries and create stored datasets.

<CardGroup>
  <Card title="SELECT" icon="magnifying-glass" href="/nql/commands/select">
    Retrieve data from datasets
  </Card>

  <Card title="CREATE MATERIALIZED VIEW" icon="layer-plus" href="/nql/commands/create-materialized-view">
    Create stored datasets from queries
  </Card>
</CardGroup>

<Card title="Query Templates" icon="code" href="/nql/commands/query-templates">
  Parameterized queries with placeholder syntax
</Card>

## Data Types

<Card title="Data Types" icon="shapes" href="/nql/data-types">
  Primitive types, complex types, type casting, and NULL handling
</Card>

## Operators

<Card title="Operators" icon="calculator" href="/nql/operators">
  Comparison, logical, arithmetic, string, and pattern matching operators
</Card>

## Functions

Built-in and Narrative-specific functions organized by category.

<CardGroup>
  <Card title="String" icon="font" href="/nql/functions/string-functions">
    Text manipulation
  </Card>

  <Card title="Date & Time" icon="clock" href="/nql/functions/date-time-functions">
    Temporal operations
  </Card>

  <Card title="Numeric" icon="hashtag" href="/nql/functions/numeric-functions">
    Math and rounding
  </Card>

  <Card title="Aggregate" icon="chart-bar" href="/nql/functions/aggregate-functions">
    GROUP BY computations
  </Card>

  <Card title="Array & Map" icon="brackets-curly" href="/nql/functions/array-map-functions">
    Collection operations
  </Card>

  <Card title="Window" icon="window" href="/nql/functions/window-functions">
    Ranking and analytics
  </Card>

  <Card title="Geospatial" icon="map-location-dot" href="/nql/functions/geospatial-functions">
    Geographic calculations
  </Card>

  <Card title="Narrative" icon="gear" href="/nql/functions/narrative-functions">
    Platform-specific functions
  </Card>

  <Card title="Conditional" icon="code-branch" href="/nql/functions/conditional-functions">
    IF, COALESCE, NULLIF
  </Card>
</CardGroup>

## General Reference

<CardGroup>
  <Card title="Syntax Overview" icon="code" href="/nql/general/syntax">
    Statement types and grammar
  </Card>

  <Card title="Explicit Columns" icon="list-check" href="/nql/general/explicit-columns">
    Why wildcards are not supported
  </Card>

  <Card title="Reserved Keywords" icon="lock" href="/nql/general/reserved-keywords">
    Keywords requiring quoting
  </Card>
</CardGroup>


# Operators
Source: https://docs.narrative.io/nql/operators

Comparison, logical, arithmetic, and special operators in NQL

NQL supports standard SQL operators for comparisons, logic, arithmetic, and pattern matching.

## Comparison operators

Compare two values and return a boolean result.

| Operator | Description           | Example                      |
| -------- | --------------------- | ---------------------------- |
| `=`      | Equal to              | `status = 'active'`          |
| `<>`     | Not equal to          | `status <> 'deleted'`        |
| `<`      | Less than             | `age < 30`                   |
| `>`      | Greater than          | `score > 90`                 |
| `<=`     | Less than or equal    | `price <= 100`               |
| `>=`     | Greater than or equal | `created_at >= '2024-01-01'` |

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  status = 'active'
  AND age >= 18
  AND age < 65
  AND score > 75
```

<Note>
  Comparisons involving NULL return NULL, not true or false. Use `IS NULL` or `IS NOT NULL` for NULL checks.
</Note>

***

## Logical operators

Combine boolean expressions.

| Operator | Description           | Example                        |
| -------- | --------------------- | ------------------------------ |
| `AND`    | Both conditions true  | `a > 0 AND b > 0`              |
| `OR`     | Either condition true | `status = 'a' OR status = 'b'` |
| `NOT`    | Negates condition     | `NOT is_deleted`               |

### Operator precedence

`NOT` is evaluated before `AND`, which is evaluated before `OR`. Use parentheses for clarity:

```sql theme={null}
-- Without parentheses: AND binds tighter than OR
WHERE a = 1 OR b = 2 AND c = 3
-- Equivalent to: a = 1 OR (b = 2 AND c = 3)

-- With parentheses for clarity
WHERE (a = 1 OR b = 2) AND c = 3
```

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  (status = 'active' OR status = 'pending')
  AND NOT is_deleted
  AND (score > 90 OR is_premium = true)
```

***

## Arithmetic operators

Perform mathematical calculations on numeric values.

| Operator | Description    | Example                 |
| -------- | -------------- | ----------------------- |
| `+`      | Addition       | `price + tax`           |
| `-`      | Subtraction    | `total - discount`      |
| `*`      | Multiplication | `quantity * unit_price` |
| `/`      | Division       | `total / count`         |

### Examples

```sql theme={null}
SELECT
  quantity * unit_price AS line_total,
  (quantity * unit_price) * 1.0825 AS total_with_tax,
  total_amount / num_items AS average_item_price,
  score - baseline AS improvement
FROM company_data."123"
```

<Warning>
  Division by zero returns NULL in NQL. Consider using NULLIF to handle potential zero divisors:
  `total / NULLIF(count, 0)`
</Warning>

***

## String operators

### Concatenation

The `||` operator concatenates strings:

```sql theme={null}
SELECT
  first_name || ' ' || last_name AS full_name,
  city || ', ' || state AS location
FROM company_data."123"
```

Non-string values are implicitly converted to strings:

```sql theme={null}
SELECT
  'User #' || user_id AS user_label,
  'Score: ' || score || '%' AS score_display
FROM company_data."123"
```

***

## IN and NOT IN

Test membership in a set of values.

### Syntax

```sql theme={null}
column IN (value1, value2, value3)
column NOT IN (value1, value2)
```

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  status IN ('active', 'pending', 'review')
  AND category NOT IN ('test', 'demo', 'internal')
```

### IN with subquery

```sql theme={null}
SELECT user_id, email, status
FROM company_data."1"
WHERE user_id IN (
  SELECT user_id
  FROM company_data."2"
  WHERE is_premium = true
)
```

<Note>
  For large value lists, consider using a JOIN instead of IN for better performance.
</Note>

***

## BETWEEN

Test if a value falls within a range (inclusive).

### Syntax

```sql theme={null}
column BETWEEN lower_bound AND upper_bound
```

Equivalent to:

```sql theme={null}
column >= lower_bound AND column <= upper_bound
```

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  age BETWEEN 25 AND 45
  AND event_date BETWEEN '2024-01-01' AND '2024-12-31'
  AND score BETWEEN 80 AND 100
```

***

## NULL operators

### IS NULL / IS NOT NULL

Test for NULL values. Regular comparison operators don't work with NULL.

```sql theme={null}
-- Correct
WHERE email IS NOT NULL
WHERE phone IS NULL

-- Incorrect (always returns no rows)
WHERE email = NULL
WHERE email <> NULL
```

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  email IS NOT NULL
  AND phone IS NOT NULL
  AND secondary_email IS NULL
```

### IS DISTINCT FROM / IS NOT DISTINCT FROM

NULL-safe equality comparison. Treats NULL as a comparable value.

| Expression    | Standard `=` | `IS NOT DISTINCT FROM` |
| ------------- | ------------ | ---------------------- |
| `1 = 1`       | true         | true                   |
| `1 = 2`       | false        | false                  |
| `1 = NULL`    | NULL         | false                  |
| `NULL = NULL` | NULL         | true                   |

```sql theme={null}
-- Useful for MERGE ON conditions
MERGE ON target.id IS NOT DISTINCT FROM source.id

-- In WHERE clauses
WHERE status IS NOT DISTINCT FROM 'active'
```

***

## Pattern matching

### LIKE

Match string patterns using wildcards.

| Wildcard | Matches                 |
| -------- | ----------------------- |
| `%`      | Zero or more characters |
| `_`      | Exactly one character   |

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  name LIKE 'John%'           -- Starts with "John"
  AND email LIKE '%@gmail.com' -- Ends with "@gmail.com"
  AND code LIKE 'A__'          -- "A" followed by exactly 2 characters
```

### Case sensitivity

LIKE is case-sensitive. Use `UPPER` or `LOWER` for case-insensitive matching:

```sql theme={null}
WHERE UPPER(name) LIKE 'JOHN%'
WHERE LOWER(email) LIKE '%@gmail.com'
```

### NOT LIKE

```sql theme={null}
WHERE email NOT LIKE '%@test.com'
```

***

## Interval operators

Perform date and time arithmetic using intervals.

### Syntax

```sql theme={null}
timestamp_column + INTERVAL 'n' unit
timestamp_column - INTERVAL 'n' unit
```

### Supported units

| Unit     | Description     |
| -------- | --------------- |
| `DAY`    | Calendar days   |
| `MONTH`  | Calendar months |
| `YEAR`   | Calendar years  |
| `HOUR`   | Hours           |
| `MINUTE` | Minutes         |
| `SECOND` | Seconds         |

### Examples

```sql theme={null}
SELECT user_id, email, status, age
FROM company_data."123"
WHERE
  -- Last 30 days
  event_timestamp > CURRENT_DATE - INTERVAL '30' DAY

  -- Last 3 months
  AND created_at >= CURRENT_DATE - INTERVAL '3' MONTH

  -- Events in the next 7 days
  AND scheduled_date <= CURRENT_DATE + INTERVAL '7' DAY
```

### Combining with date functions

```sql theme={null}
SELECT
  event_timestamp,
  event_timestamp + INTERVAL '1' HOUR AS one_hour_later,
  DATE_TRUNC('month', event_timestamp) AS month_start,
  DATE_TRUNC('month', event_timestamp) + INTERVAL '1' MONTH - INTERVAL '1' DAY AS month_end
FROM company_data."123"
```

***

## Operator precedence

From highest to lowest precedence:

| Precedence  | Operators                                                      |
| ----------- | -------------------------------------------------------------- |
| 1 (highest) | Unary `-`, `+`                                                 |
| 2           | `*`, `/`                                                       |
| 3           | `+`, `-`                                                       |
| 4           | `\|\|` (concatenation)                                         |
| 5           | `=`, `<>`, `<`, `>`, `<=`, `>=`, `IS`, `LIKE`, `IN`, `BETWEEN` |
| 6           | `NOT`                                                          |
| 7           | `AND`                                                          |
| 8 (lowest)  | `OR`                                                           |

### Using parentheses

Use parentheses to override default precedence or improve readability:

```sql theme={null}
SELECT
  -- Arithmetic: multiplication before addition
  price + quantity * discount,       -- = price + (quantity * discount)
  (price + quantity) * discount,     -- Override precedence

  -- Logical: AND before OR
  a OR b AND c,                      -- = a OR (b AND c)
  (a OR b) AND c                     -- Override precedence
FROM company_data."123"
```

***

## Related content

<CardGroup>
  <Card title="NQL Syntax" icon="code" href="/nql/general/syntax">
    Query structure and grammar reference
  </Card>

  <Card title="Data Types" icon="shapes" href="/nql/data-types">
    Type system and conversions
  </Card>

  <Card title="Functions" icon="function" href="/nql/functions/index">
    Built-in functions for data manipulation
  </Card>

  <Card title="Filtering Data" icon="filter" href="/guides/nql/filtering-transforming">
    Guide to writing effective WHERE clauses
  </Card>
</CardGroup>


# Agent Conversations Reference
Source: https://docs.narrative.io/reference/architecture/agent-conversations

Build LLM-driven workflows that can call MCP tools, ask the caller for input, and return structured answers — all through five HTTP endpoints.

The Agent Conversations API lets you ask a language model a question, have it call tools to
gather more information, and get back a structured answer — all through a few HTTP calls. The
model can use **MCP-resolved tools** (anything exposed by a Model Context Protocol server, like
the Narrative docs search) or **caller-declared tools** (where it asks *you* a question and
waits for your reply).

This page is a complete reference: the moving pieces, the parameters, three end-to-end
examples, and the most common things that go wrong.

## What it does

A typical request goes like this:

1. You create a **conversation** with a system prompt, a model choice, and a list of tools the
   model is allowed to use.
2. You start a **run** by sending the model a user message.
3. The model decides whether to answer directly or call one of the tools.
   * If it picks an **MCP-resolved tool**, the platform calls it for you and feeds the result
     back into the next round of reasoning. This can repeat several times.
   * If it picks a **caller-declared tool**, the run pauses and asks you to provide the
     answer. You start a new run with the answer; the model continues from there.
4. Eventually the model produces a final answer matching the structured schema you provided.
   The run reaches `completed` and you fetch the result.

The platform keeps the full conversation in a database, so you can resume a paused run later,
inspect every message the model saw, and start follow-up runs on the same conversation.

<Info>
  The shape mirrors the [OpenAI Assistants API](https://platform.openai.com/docs/assistants): a
  **thread** is a conversation, a **run** is one user-initiated turn, a **step** is one inference
  iteration inside a run. If you already know that mental model, the only new ideas here are
  *tool aliases* and *optimistic-concurrency versioning* (covered below).
</Info>

## API endpoints

All endpoints live under `/agents` and require a Bearer token with
`agent_conversations` read/write permission.

<Info>
  **Scope: per-user, not per-company.** Conversations and runs are keyed on the bearer
  token's `(company_id, user_id)` pair. Peers in the same company cannot see each other's
  conversations, runs, or messages — every endpoint returns 404 for cross-user access,
  identical to cross-company access. The API does not distinguish "doesn't exist" from
  "owned by another user".
</Info>

| Method | Path                                          | Purpose                                                                              |
| ------ | --------------------------------------------- | ------------------------------------------------------------------------------------ |
| `POST` | `/agents/conversations`                       | Create an empty conversation, pinning its model + tool catalog                       |
| `GET`  | `/agents/conversations?page=N&per_page=K`     | List the calling user's conversations (paginated, newest first)                      |
| `GET`  | `/agents/conversations/{id}`                  | Read conversation metadata (current version, defaults)                               |
| `GET`  | `/agents/conversations/{id}/messages?since=N` | Page through messages with `sequence_no > N`                                         |
| `POST` | `/agents/conversations/{id}/runs`             | Start a new run — either a fresh user message, or tool outputs resuming a paused run |
| `GET`  | `/agents/runs/{id}`                           | Read a run's current status, results, and pending tool calls                         |

The first request (`POST /agents/conversations`) returns the conversation `id` you need for
all the others. The fourth (`POST .../runs`) returns the run `id` you can poll with the fifth.

### Listing your conversations

`GET /agents/conversations` returns the calling user's conversations in `created_at`
descending order, wrapped in the standard pagination envelope:

```bash theme={null}
curl "$API/agents/conversations?page=1&per_page=20" \
  -H "Authorization: Bearer $TOKEN"
```

```json theme={null}
{
  "prev_page": null,
  "current_page": 1,
  "next_page": 2,
  "total_records": 47,
  "total_pages": 3,
  "records": [
    { "id": "...", "name": "rate-limiting q&a", "version": 4, "...": "..." }
  ]
}
```

Default `per_page` is 50. Results are filtered by both `company_id` and `user_id` —
peers in the same company do **not** see each other's conversations (see the scope
callout above).

## Core concepts in one paragraph each

### Conversation

The long-lived container. The `system_prompt` is **pinned for life** — once set at creation
time, it applies to every run that follows and cannot be changed. Everything else in `defaults`
(model, tools, max iterations, temperature, output schema, etc.) is exactly that — a
**default**: it applies unless a particular run explicitly replaces it. The conversation also
carries a monotonic `version` counter that every successful run bumps. The counter doubles as
both a delta cursor (for `GET .../messages?since=N`) and a compare-and-swap token (more on
that under [Troubleshooting](#troubleshooting)).

### Run

One user-initiated turn of conversation. Each run kicks off a workflow on the platform that
performs zero or more inference rounds plus tool calls. A run is **asynchronous**: the `POST`
returns immediately with `status: "pending"`, and you poll `GET /agents/runs/{id}` until it
reaches a terminal state (`completed`, `requires_action`, or `failed`).

A run may **override the conversation's defaults** for that single turn via `config_override` —
pick a different model, raise `max_iterations` for a hard question, swap in a different tool
catalog, tighten `temperature` for a deterministic answer, etc. The only thing a run cannot
change is the `system_prompt` (that's permanently set on the conversation). Two run-only
settings have no conversation-level counterpart: `tool_choice` (which biases the model toward
a particular tool on the first iteration) and the run's `payload` itself.

### Tools

A run's tool catalog comes from two sources, listed side by side in the conversation
`defaults`:

* **`mcp_servers[]`** — each entry has an `alias` (1–8 letters/digits, must start with a
  letter), a URL, and an optional description. **You do not declare individual tools** — the
  platform discovers each server's catalog at run start via the JSON-RPC `tools/list` method
  and stitches the wire names as `{alias}-{tool_name}` before handing them to the model.
  When the model emits a call to `{alias}-{tool_name}`, the workflow makes the `tools/call`
  request, gets the result, and feeds it back into the next inference round inside the same
  run.
* **`tools[]`** — caller-declared tools you answer yourself. **No alias** — the model sees
  the bare `name`. When the model calls one of these, the run terminates with `status:
  "requires_action"`; you read the question from `pending_tool_calls`, decide on an answer,
  and start a *new* run with `payload.kind: "tool_outputs"` carrying your reply.

<Info>
  **Discovery is per-run, not cached.** Every run re-fetches `tools/list` from each registered
  MCP server. There is no inter-run cache — if the server adds, removes, or renames a tool,
  the next run picks it up automatically. The trade-off: each new run pays one HTTP roundtrip
  per registered server before the first inference. For 1–3 typical servers this is in the
  noise (10–100ms total); if you ever register a slow-discovering server, expect it to delay
  every run's first iteration. Discovery failures terminate the run with
  [MCP Discovery Failed](./agent-conversations/errors/mcp-discovery-failed).
</Info>

<Info>
  **Narrative-owned MCP servers are authenticated automatically.** When an
  `mcp_servers[].url` matches the [Data Collaboration MCP Server](/reference/integrations/mcp-server)
  (`https://mcp.narrative.io/mcp` in prod), the platform mints a Default-scoped API token
  for the conversation's user and company and attaches it as
  `Authorization: Bearer ...` on every `tools/list` and `tools/call` request to that
  server. The token is created server-side, lives only for the duration of the request
  chain, and is never persisted to `agent_runs.effective_config`, Temporal event history,
  or the `GET /agents/runs/{id}` echo. Public or third-party MCP servers (any URL outside
  the allowlist) continue to be called without auth — the existing behavior is unchanged.
</Info>

<Info>
  **Tool input schemas are normalized at discovery time.** MCP servers can declare
  `inputSchema` shapes that Bedrock Converse and Anthropic Messages don't accept verbatim
  — `oneOf`, `$ref`, `format` validators, `additionalProperties: true`, and similar JSON
  Schema features. The platform runs every discovered tool's schema through a converter
  that translates `oneOf` to `anyOf`, inlines non-recursive `$ref`, replaces recursive
  `$ref` with `{}` to break cycles, forces `additionalProperties: false`, and lifts
  unsupported keywords (`format`, `pattern`, range validators) into the property's
  `description`. Tools that previously had to be dropped wholesale because they carried
  `$ref` now load successfully; nothing is rejected at the discovery layer. The converter
  emits `strict: false` because Bedrock caps strict-tool counts per request — the model
  still gets the schema, and each MCP server validates arguments on its own side at call
  time.
</Info>

**The dash is the routing discriminator.** MCP wire names always carry the `{alias}-`
prefix; caller-declared tool names must not contain a dash. This is what lets the workflow
classify a hallucinated or unknown call without ambiguity: dash + unknown alias →
`AgentLoopUnknownToolAlias`; dash-free + unknown name → same error. The validation rule "no
dash in `tools[].name`" is the cornerstone.

### Tool choice

Per-run policy that nudges the model toward (or away from) using tools on the *first*
iteration. Three options:

* `{"kind": "auto"}` — model decides. The default.
* `{"kind": "any"}` — model must call some tool (no plain-text answer allowed).
* `{"kind": "specific_tool", "name": "confirm_booking"}` — model must call exactly this
  caller-declared tool. To pin an MCP-resolved tool instead, include the explicit
  `mcp_alias`: `{"kind": "specific_tool", "mcp_alias": "docs", "name": "search_kb"}`. The
  platform validates the target against the catalog synchronously and returns 400 with
  [Unknown Tool Choice Name](/reference/architecture/agent-conversations/errors/unknown-tool-choice-name)
  or
  [Unknown Tool Choice MCP Alias](/reference/architecture/agent-conversations/errors/unknown-tool-choice-mcp-alias)
  on a miss.

The choice only applies to iteration 1 **of the current run**. From iteration 2 onward inside
that same run the model is back on `{"kind": "auto"}`. There is no carry-over to subsequent
runs — every new run picks its own `tool_choice` afresh. That means you can absolutely
re-impose a forced tool call on a follow-up run (for example, push the model toward a specific
caller-declared tool every time you need to ask the user something), so long as you set
`tool_choice` on the run's request body.

## Configuration parameters

When you create a conversation, you pass a `defaults` object that pins how every run on that
conversation behaves. Each field can be overridden per-run via `config_override`, except where
noted.

| Field                  | Required           | What it means                                                                                                                                                                                                                                                                                                                                                                                                            | Typical value                                             |
| ---------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
| `model`                | yes                | Which language model to use. The platform exposes a fixed set of identifiers.                                                                                                                                                                                                                                                                                                                                            | `"anthropic.claude-opus-4.6"`                             |
| `data_plane_id`        | yes                | Which compute environment runs the inference. Each company has at least one.                                                                                                                                                                                                                                                                                                                                             | UUID from your platform admin                             |
| `execution_cluster`    | yes                | Which job-executor pool routes the inference job to AWS Bedrock or Snowflake Cortex. Inference itself runs in the external model service, not on a platform cluster — the executor only dispatches the HTTP call. **Use `"shared"` for almost every case;** the value only matters if your company runs a dedicated executor pool with isolation requirements.                                                           | `"shared"`                                                |
| `max_iterations`       | no (default 3)     | How many inference rounds the model is allowed before the platform forces a `failed` run. Each iteration costs one model call.                                                                                                                                                                                                                                                                                           | `3` for trivial questions, `8–15` for multi-step research |
| `max_tokens`           | no (default 2048)  | Cap on the **model's reply** per iteration. Doesn't include the prompt.                                                                                                                                                                                                                                                                                                                                                  | `1024`–`4096`                                             |
| `temperature`          | no (default `0.0`) | How creative the model is allowed to be. `0.0` is deterministic; `1.0` is creative.                                                                                                                                                                                                                                                                                                                                      | `0.0` for factual answers, `0.7` for brainstorming        |
| `output_format_schema` | no                 | A JSON Schema describing the **structure** of the final answer. When omitted, the run is in **text mode** and the answer comes back as `final_text`. When provided, the run is in **structured mode** and the answer comes back as `final_structured_output`, conforming verbatim to your schema. The two response fields are mutually exclusive. Supports a **subset** of Draft 2020-12 — see the note below the table. | See examples below                                        |
| `mcp_servers`          | no (default `[]`)  | List of MCP servers the model may call. Each entry is `{alias, url, description?}`; tools are discovered per-run via `tools/list`.                                                                                                                                                                                                                                                                                       | See Example 2                                             |
| `tools`                | no (default `[]`)  | List of caller-declared tools that pause the run with `requires_action`. **No alias** — each entry's `name` must be dash-free.                                                                                                                                                                                                                                                                                           | See Example 3                                             |
| `system_prompt`        | no                 | A "stage direction" prepended to every run. **Pinned at creation time**, cannot be overridden per-run.                                                                                                                                                                                                                                                                                                                   | `"You are a helpful assistant. Answer concisely."`        |

<Note>
  `mcp_servers` and `tools` are **wholesale-replaced**, not merged, when overridden in a run.
  If you set them in `config_override`, you replace the whole list. This keeps tool namespacing
  predictable.
</Note>

<Warning>
  **`output_format_schema` is a JSON Schema subset.** The platform accepts the same subset of
  Draft 2020-12 that Bedrock structured-output accepts. Common features that **do not work**:
  union types (`oneOf`, `anyOf`, `allOf`, `not`), schema composition (`$ref`, `$defs`),
  conditional shape (`if`/`then`/`else`), `pattern` regexes on strings, and `format` validators
  (`date-time`, `email`, etc.). Supported: `type`, `properties`, `required`, `enum`,
  `additionalProperties: false`, `minimum`/`maximum` on numbers, `minLength`/`maxLength` on
  strings, `minItems`/`maxItems` on arrays. See [JSON Schema
  Reference](/reference/model-inference/json-schema-reference) for the full list with examples.
</Warning>

## Run payload shapes

`POST /agents/conversations/{id}/runs` accepts one of two payload kinds:

```json theme={null}
// A fresh user message
{
  "payload": { "kind": "user_message", "text": "What is 2 + 2?" }
}

// The user/system answering a tool call from a previously paused run
{
  "payload": {
    "kind": "tool_outputs",
    "outputs": [
      {
        "tool_use_id": "tooluse_xyz",
        "content": "User confirmed the proposed slot.",
        "is_error": false
      }
    ]
  }
}
```

The full request body also includes:

* `client_op_id` — a UUID *you* generate, unique per `(conversation, request)`. Used as an
  idempotency key — re-sending the same `client_op_id` returns the original run unchanged.
  This lets you retry safely across network blips.
* `expected_version` — the conversation's current version, which you got from the most recent
  `GET /agents/conversations/{id}`. The platform rejects the run if the version has moved on
  since you read it (see [Version conflicts](#version-conflicts-409)).
* `tool_choice` — optional, per-run only.
* `config_override` — optional, sparse — only the fields you want to change.

## Run status lifecycle

Every run starts at `pending` and eventually reaches one of three terminal states:

```
pending  →  running  →  completed         ← model produced a final answer
                    ↘   requires_action   ← model called a caller-declared tool
                    ↘   failed            ← platform error, non-recoverable model error,
                                            or a deliberate cancellation
```

Poll `GET /agents/runs/{id}` every few seconds (start with 2–4 seconds; back off if you don't
care about latency). When you see one of the terminal states you can stop polling.

`completed` runs populate exactly one of two fields, depending on whether you supplied an
`output_format_schema`:

* **No schema (text mode)** → `final_text` carries the model's reply as a plain string;
  `final_structured_output` is `null`.
* **Schema supplied (structured mode)** → `final_structured_output` carries the parsed JSON
  object conforming verbatim to your schema; `final_text` is `null`. Your schema does **not**
  need a top-level `text` field — whatever shape you declare is what you get back.

`requires_action` runs populate `pending_tool_calls` — an array of caller-declared tool
calls waiting for you to answer. See Example 3 for the resume flow.

`failed` runs populate `error.type` (an opaque incident code) and `error.message` (a
human-readable detail). The response also includes `error.title` and `error.docs_url` pointing
at the relevant [error catalog](./agent-conversations/errors/conversation-not-found) page.

A run that was deliberately **cancelled** also lands in `failed`, but carries the reserved
`error.type: "AgentLoopCancelled"` so you can tell an intentional stop from a genuine error.
The run's in-flight inference job is cancelled too, so the run and its data-plane job end up
consistent. See [`/errors/cancelled`](./agent-conversations/errors/cancelled).

<Info>
  **One error vocabulary for both surfaces.** The agent API exposes errors in two physical shapes
  — **HTTP 4xx/5xx with an [RFC 7807](https://www.rfc-editor.org/rfc/rfc7807) body** for
  synchronous failures (bad request body, conversation not found, version conflict, etc.), and
  **HTTP 200 with `status: "failed"` and an `error` object** for failures that happen inside the
  workflow after the run has been accepted (max iterations exceeded, MCP server unreachable,
  invalid effective config, etc.).

  The two shapes carry the **same caller-facing fields**:

  | RFC 7807 (synchronous) | `RunErrorDto` on a failed run (asynchronous) | Meaning                                            |
  | ---------------------- | -------------------------------------------- | -------------------------------------------------- |
  | `type` (URL)           | `error.docs_url`                             | Stable URL to the docs page for this failure class |
  | `title`                | `error.title`                                | Short, caller-facing summary                       |
  | `status` (HTTP)        | n/a (the call itself returned 200)           | —                                                  |
  | `detail`               | `error.message`                              | Request-specific detail string                     |
  | `instance` (path)      | n/a                                          | —                                                  |
  | `log_id`               | n/a; correlate via the run id                | —                                                  |
  | —                      | `error.type` (incident code)                 | Internal stable tag for log dashboards             |

  Both shapes link into the same [error catalog](./agent-conversations/errors/conversation-not-found),
  which means a single playbook covers both. Whether your client got a 409 on `POST .../runs` or a
  200 with `error.type: "AgentLoopMaxIterationsExceeded"` on `GET /agents/runs/{id}`, the docs URL
  in the response is the canonical "how do I recover" entry point.

  The workflow side itself never knows about caller-facing presentation — it only emits opaque
  incident codes (`"AgentLoopMaxIterationsExceeded"`, `"UnknownTool"`, etc.). The translation to
  `title` + `docs_url` happens at the API boundary against a single catalog kept in sync with the
  docs pages.
</Info>

## The `live` view

Every run response (`POST .../runs` and `GET /agents/runs/{id}`) carries a `live` object. Unlike
the `status`-dependent fields, `live` is **orthogonal to `status`** and is populated on every
read — it reflects the conversation's *current, still-mutating* state, so a run you polled
minutes ago can surface newer values on the next read without a separate
`GET /agents/conversations/{id}` call.

```json theme={null}
{
  "id": "<run-uuid>",
  "status": "running",
  "live": {
    "current_name": "Tomorrow Afternoon Meeting",
    "messages": [
      { "turn_index": 0, "role": "assistant", "content_blocks": [{ "type": "tool_use", "tool_use_id": "tooluse_abc", "name": "docs-search_kb", "arguments": { "query": "rate limiting" } }] },
      { "turn_index": 1, "role": "tool",      "content_blocks": [{ "type": "tool_result", "tool_use_id": "tooluse_abc", "content": [{ "type": "text", "text": "..." }], "is_error": false }] }
    ]
  }
}
```

`live` holds two fields:

* **`current_name`** — the conversation's display name, or `null` if it has none yet. This is the
  same value as `name` on `GET /agents/conversations/{id}`; it's mirrored onto the run so a
  client that's already polling a run sees title changes for free.
* **`messages`** — the run's produced turns *so far*, streamed while the run is still in flight so
  you can show tool execution live. See [Streaming tool execution](#streaming-tool-execution) below.

### Streaming tool execution

While a run is **non-terminal** (`pending` / `running`), `live.messages` carries the turns the
loop has produced so far — each tool-use assistant turn appears *before* the platform executes the
call (so you can render "calling `docs-search_kb`…"), and the matching `tool_result` turn appears
once it returns. This lets a chat UI animate a multi-iteration tool loop from the **same run poll**
you're already doing, with no extra endpoint.

Each entry has a `turn_index` (per-run ordering, independent of `sequence_no`), a `role`, and the
same `content_blocks` shape as committed messages.

The handoff to the committed log is lossless:

* Render `committed messages (GET .../messages) ++ live.messages` while the run is non-terminal.
* Once the run reaches a **terminal** status, `live.messages` is **empty** — the same turns are now
  committed and authoritative via `GET .../messages?since=…`. Drop your live tail and keep the
  committed rows.
* De-dupe is exact: the live `tool_use` / `tool_result` blocks carry the same `tool_use_id`s that
  land in the committed `content_blocks`.

Live streaming is **best-effort**: it never blocks or fails the run, and it never touches the
conversation `version` or the committed message stream (live turns aren't messages — they carry no
`sequence_no` and don't advance your `expected_version`). A run with no server-side tool calls (a
direct answer) simply shows an empty `live.messages` and you get the answer via `final_text` /
`final_structured_output` at completion.

### Automatic titling

When you start the **first** run on a conversation that has **no name**, the platform kicks off
a small asynchronous job that generates a short title from your first user message and writes it
to the conversation's `name`. Because it's asynchronous and independent of your run, the timing
is best-effort:

* `current_name` is usually `null` on the first read right after the run is created, then
  becomes the generated title a beat later — keep reading the run (or the conversation) and it
  appears.
* A `name` you set **explicitly** at conversation-creation time is **never overwritten** — auto
  titling only fills an empty name.
* Titling never touches the conversation `version` or the message stream: it's not a message,
  doesn't advance your `expected_version`, and never shows up in `GET .../messages`.

<Info>
  **`live` is a growth point.** It accumulates state that changes after a run was written, so a
  single run poll can stand in for several reads (`current_name`, `messages`, more over time). Treat
  it as "render whatever keys are present" rather than assuming a fixed set.
</Info>

## Example 1 — hello world (no tools)

The simplest possible run: one user message, one assistant reply, no tools.

### Create the conversation

```bash theme={null}
curl -X POST "$API/agents/conversations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "hello world",
    "system_prompt": "You are a helpful assistant. Answer concisely.",
    "defaults": {
      "model": "anthropic.claude-opus-4.6",
      "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
      "execution_cluster": "shared",
      "max_iterations": 3,
      "max_tokens": 1024,
      "temperature": 0.0
    }
  }'
```

No `output_format_schema` here — this is **text mode**. The final answer comes back as
`final_text`. See [Example 4](#example-4--structured-output) for the structured-mode flow.

Response (abbreviated):

```json theme={null}
{ "id": "<conv-uuid>", "version": 0, "name": "hello world", ... }
```

### Start the run

```bash theme={null}
curl -X POST "$API/agents/conversations/<conv-uuid>/runs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "client_op_id": "'"$(uuidgen | tr A-Z a-z)"'",
    "expected_version": 0,
    "payload": { "kind": "user_message", "text": "What is 2 + 2?" }
  }'
```

Response:

```json theme={null}
{ "id": "<run-uuid>", "status": "pending", "started_at": "...", ... }
```

### Poll until terminal

```bash theme={null}
curl "$API/agents/runs/<run-uuid>" -H "Authorization: Bearer $TOKEN"
```

After a few seconds:

```json theme={null}
{
  "id": "<run-uuid>",
  "status": "completed",
  "iterations_used": 1,
  "usage": { "completion_tokens": 8, "prompt_tokens": 169, "total_tokens": 177 },
  "final_text": "4",
  "final_structured_output": null,
  "error": null,
  ...
}
```

### Read the message stream

```bash theme={null}
curl "$API/agents/conversations/<conv-uuid>/messages?since=0" \
  -H "Authorization: Bearer $TOKEN"
```

```json theme={null}
{
  "current_version": 2,
  "messages": [
    { "sequence_no": 1, "role": "user",      "content_blocks": [{"type": "text", "text": "What is 2 + 2?"}], ... },
    { "sequence_no": 2, "role": "assistant", "content_blocks": [{"type": "text", "text": "4"}],              ... }
  ]
}
```

That's the whole loop: one user turn (sequence 1) → one assistant turn (sequence 2) →
`current_version: 2`. Future runs on this conversation start with `expected_version: 2`.

## Example 2 — searching docs via an MCP server

Now the model has tools. It searches the Narrative docs for "rate limiting", makes a few
attempts to locate the right page, and produces a grounded summary.

### Create the conversation with `mcp_servers`

```bash theme={null}
curl -X POST "$API/agents/conversations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "docs search",
    "system_prompt": "You are an AI assistant that answers questions about the Narrative Data Marketplace. Use the available tools to look things up before answering. Once you have enough information, produce a concise final answer that matches the provided output schema.",
    "defaults": {
      "model": "anthropic.claude-opus-4.6",
      "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
      "execution_cluster": "shared",
      "max_iterations": 8,
      "max_tokens": 2048,
      "temperature": 0.0,
      "mcp_servers": [{
        "alias": "docs",
        "url": "https://docs.narrative.io/mcp",
        "description": "Narrative.io documentation MCP server"
      }]
    }
  }'
```

Key points:

* `alias: "docs"` — the platform discovers the server's tools at run start, then prefixes
  each with the alias before showing them to the model. An MCP-side tool named
  `search_narrative_i_o_knowledge_base` becomes `docs-search_narrative_i_o_knowledge_base`
  on the wire.
* **No `tools[]` here** — discovery is automatic. To inspect what the server exposes,
  call its `tools/list` directly (e.g. `curl https://docs.narrative.io/mcp ...`).
* `max_iterations: 8` — gives the model room to: search, read, refine, then answer.

### Start the run

Same shape as Example 1 — just a user message. The model decides what tools to call.

```json theme={null}
{
  "client_op_id": "...",
  "expected_version": 0,
  "payload": { "kind": "user_message", "text": "Search the Narrative docs for information about rate limiting and summarize what you find in 2-3 sentences." },
  "tool_choice": { "kind": "auto" }
}
```

### What you see while polling

`status` cycles `pending` → `running` → `running` → ... — each `running` you see corresponds
roughly to one inference iteration. Be patient: MCP roundtrips can take 30–60 seconds total
for multi-step research like this.

Final state:

```json theme={null}
{
  "status": "completed",
  "iterations_used": 6,
  "usage": { "completion_tokens": 510, "prompt_tokens": 14842, "total_tokens": 15352 },
  "submitted_inference_job_ids": [
    "<job-1>", "<job-2>", "<job-3>", "<job-4>", "<job-5>", "<job-6>"
  ],
  "final_text": "According to the Narrative.io documentation, the API returns a 429 ...",
  ...
}
```

The message stream now has 12 entries — the user message, plus alternating `assistant` (with
`tool_use` blocks) and `tool` (with `tool_result` blocks) turns, and a final `assistant` turn
carrying the answer.

<Tip>
  **Prompt tokens grow with each iteration** because each round re-sends the full conversation
  history to the model. Six iterations on a conversation with several search hits can easily
  reach 15,000 prompt tokens. Tighter system prompts and shorter tool descriptions help.
</Tip>

## Example 3 — asking the user a question (caller-declared tool)

Sometimes the model needs information only the caller has. Configure a `tools[]` entry,
force the model to use it with `tool_choice`, and the run will pause at `requires_action`
until you answer.

### Create the conversation with `tools[]`

```bash theme={null}
curl -X POST "$API/agents/conversations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "booking flow",
    "system_prompt": "You are a booking assistant. Use the available tool to propose a meeting slot to the user.",
    "defaults": {
      "model": "anthropic.claude-opus-4.6",
      "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
      "execution_cluster": "shared",
      "max_iterations": 5,
      "tools": [{
        "name": "confirm_booking",
        "description": "Ask the user to confirm a proposed booking slot.",
        "input_schema": {
          "type": "object",
          "additionalProperties": false,
          "required": ["proposed_slot"],
          "properties": { "proposed_slot": { "type": "string" } }
        }
      }]
    }
  }'
```

### Start the run and force the tool call

```json theme={null}
{
  "client_op_id": "...",
  "expected_version": 0,
  "payload": { "kind": "user_message", "text": "Find me a meeting time for tomorrow afternoon." },
  "tool_choice": { "kind": "specific_tool", "name": "confirm_booking" }
}
```

### Poll until `requires_action`

```json theme={null}
{
  "status": "requires_action",
  "iterations_used": 1,
  "pending_tool_calls": [
    {
      "tool_use_id": "tooluse_DWXPKZ50JDGib5GmShyUgJ",
      "name": "confirm_booking",
      "arguments": { "proposed_slot": "tomorrow afternoon" }
    }
  ],
  "final_text": null,
  "final_structured_output": null
}
```

The run is paused. `pending_tool_calls[].tool_use_id` is the handle you'll need to resume.

### Resume with `tool_outputs`

Re-read the conversation to get the new version (it has advanced because the assistant turn is
now persisted):

```bash theme={null}
curl "$API/agents/conversations/<conv-uuid>" -H "Authorization: Bearer $TOKEN"
# → { "version": 2, ... }
```

Now start a follow-up run with the answer:

```json theme={null}
{
  "client_op_id": "...",
  "expected_version": 2,
  "payload": {
    "kind": "tool_outputs",
    "outputs": [{
      "tool_use_id": "tooluse_DWXPKZ50JDGib5GmShyUgJ",
      "content": "User confirmed the proposed slot.",
      "is_error": false
    }]
  }
}
```

The model picks up where it left off, sees the tool result, and produces a final answer:

```json theme={null}
{
  "status": "completed",
  "iterations_used": 1,
  "final_text": "Great news! Your meeting has been confirmed for tomorrow afternoon. ...",
  "final_structured_output": null
}
```

<Note>
  You can answer with `is_error: true` to tell the model the tool failed (e.g. the user
  declined). The model decides whether to try another approach, ask a different question, or
  fall through to a final answer.
</Note>

## Example 4 — structured output

When you need the model's answer as a typed object rather than free text, supply an
`output_format_schema` describing the shape you want. The model is grammar-constrained to
produce JSON matching it, and the parsed value comes back on `final_structured_output`.

This example records a login event using a discriminated union (`anyOf`) of `login` /
`logout` variants. Note that the schema has **no top-level `text` field** — you declare
whatever shape your application needs.

### Create the conversation with a structured schema

```bash theme={null}
curl -X POST "$API/agents/conversations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "event logger",
    "system_prompt": "You record events as structured data. Respond only via the structured output schema.",
    "defaults": {
      "model": "anthropic.claude-opus-4.6",
      "data_plane_id": "f79cbdae-4848-47ca-95e8-69588364d185",
      "execution_cluster": "shared",
      "max_iterations": 3,
      "max_tokens": 1024,
      "temperature": 0.0,
      "output_format_schema": {
        "type": "object",
        "additionalProperties": false,
        "required": ["event"],
        "properties": {
          "event": {
            "anyOf": [
              {
                "type": "object",
                "title": "Login",
                "additionalProperties": false,
                "required": ["type", "user_id", "session_duration_seconds"],
                "properties": {
                  "type": { "const": "login" },
                  "user_id": { "type": "string" },
                  "session_duration_seconds": { "type": "integer" }
                }
              },
              {
                "type": "object",
                "title": "Logout",
                "additionalProperties": false,
                "required": ["type", "user_id"],
                "properties": {
                  "type": { "const": "logout" },
                  "user_id": { "type": "string" }
                }
              }
            ]
          }
        }
      }
    }
  }'
```

### Start the run

```json theme={null}
{
  "client_op_id": "...",
  "expected_version": 0,
  "payload": {
    "kind": "user_message",
    "text": "User u-123 logged in and stayed for 1800 seconds. Record the structured event."
  }
}
```

### Final state

```json theme={null}
{
  "status": "completed",
  "iterations_used": 1,
  "final_text": null,
  "final_structured_output": {
    "event": {
      "type": "login",
      "user_id": "u-123",
      "session_duration_seconds": 1800
    }
  }
}
```

`final_text` is `null` because the run is in structured mode; the answer is on
`final_structured_output`, conforming verbatim to your schema. The two fields are mutually
exclusive — text-mode runs (Examples 1–3) populate `final_text`; structured-mode runs
populate `final_structured_output`.

<Tip>
  Your schema can be anything Bedrock's structured-output sampler accepts — see the supported
  subset in the warning at the top of [Configuration fields](#configuration-fields). Common
  shapes that work well: a single typed object, a discriminated union via `anyOf` with `const`
  discriminators (as above), an enum-tagged sum type, an array of typed records.
</Tip>

## Troubleshooting

### Version conflicts (409)

You get a 409 from `POST .../runs` with `error.type` pointing at
[`/errors/version-conflict`](./agent-conversations/errors/version-conflict).

**What happened:** between the moment you read `version: N` and the moment you posted a run
with `expected_version: N`, something else added messages to the conversation (a previous
run's finalize, or a concurrent caller). The platform refuses to start a run that would
conflict at finalize time.

**How to recover:**

```bash theme={null}
# 1. Refetch the conversation to get the new version
curl "$API/agents/conversations/<conv-uuid>" -H "Authorization: Bearer $TOKEN"

# 2. Refetch the messages since your last known version, so you know what changed
curl "$API/agents/conversations/<conv-uuid>/messages?since=<your-old-version>" \
  -H "Authorization: Bearer $TOKEN"

# 3. Re-post the same run body, but with the fresh expected_version
```

If you keep hitting this without obvious cause, check whether a previous run that you thought
was `completed` is actually `requires_action` — the latest assistant turn might be a tool-call
prompt that needs your reply, not a finished answer.

### "Bad request" with a tool-alias message (400)

The `defaults.mcp_servers[].alias` you sent isn't in the allowed shape. MCP aliases must:

* be 1–8 characters total,
* start with a letter,
* contain only ASCII letters and digits (no underscores, no dashes, no other punctuation),
* be unique across `mcp_servers` for the conversation.

Caller-declared tools (under `tools[]`) have no alias — their `name` must be non-empty and
dash-free. See
[`/errors/invalid-caller-tool-name`](./agent-conversations/errors/invalid-caller-tool-name).

Full alias rules and examples: [`/errors/invalid-tool-alias`](./agent-conversations/errors/invalid-tool-alias).

### "Tool wire name too long" (400)

The combined `{alias}-{tool_name}` exceeds 64 characters (an underlying Bedrock limit).
Shorten the alias or the underlying tool name. See
[`/errors/tool-name-too-long`](./agent-conversations/errors/tool-name-too-long).

### `requires_action` and you don't know what to answer

`pending_tool_calls[]` lists every tool call awaiting your reply, with the tool name and the
arguments the model produced. When you post `tool_outputs`, **every** entry in
`pending_tool_calls` must have a matching `outputs[]` entry — no missing, no extras. If you
have nothing useful to say for a particular call (e.g. the user dismissed the prompt), still
post an entry with `is_error: true` and a short reason.

Common mistakes around tool outputs:

* [Unknown `tool_use_id`](./agent-conversations/errors/unknown-tool-use-id) — you sent an id
  the latest assistant turn never produced. Almost always a typo or a stale resume payload.
* [Not a client tool call](./agent-conversations/errors/not-a-client-tool-call) — you sent a
  `tool_use_id` whose `name` has the `{alias}-{tool}` shape (an MCP-resolved call). The
  platform already answered those; only dash-free names belong in `tool_outputs`.
* [Incomplete tool outputs](./agent-conversations/errors/incomplete-tool-outputs) — you missed
  one of the pending ids, or sent extras the model didn't ask for.

### Run failed with `AgentLoopMaxIterationsExceeded`

The model used up its `max_iterations` budget without producing a final answer. Either:

* raise `max_iterations` in `defaults` (or `config_override` per run),
* tighten the system prompt so the model is steered toward an answer sooner,
* inspect the message stream — repeated identical tool calls suggest the model is stuck in a
  loop because the tool isn't giving it useful new information.

Details: [`/errors/max-iterations-exceeded`](./agent-conversations/errors/max-iterations-exceeded).

### Run failed with `AgentLoopSchemaDecodeFailed`

In **text mode** (no `output_format_schema`) this is almost always a truncation issue: the
model's reply got cut off before the platform could read it. Raise `max_tokens`.

In **structured mode** this means the model produced JSON that didn't match the shape your
`output_format_schema` declared, or returned prose instead of JSON. Fixes:

* be explicit in the system prompt: "respond with a JSON object matching this schema; no other
  text."
* consider widening `max_tokens` — sometimes the model is truncated mid-JSON.
* simplify the schema. Bedrock's structured-output sampler enforces a subset of Draft 2020-12;
  features outside that subset can silently fall through to free-text generation. See the
  warning under [Configuration fields](#configuration-fields).
* inspect the last assistant turn to see what the model actually said.

Details: [`/errors/schema-decode-failed`](./agent-conversations/errors/schema-decode-failed).

### Run failed with `AgentLoopCancelled`

Not an error — the run was deliberately cancelled before it finished. The run lands in `failed`
with this reserved code (so it's distinguishable from a genuine failure), and its in-flight
inference job is cancelled too. Any partial work is preserved on the run row and the message
stream. If you didn't expect the cancellation, find out who issued it (operations tooling, a
worker drain on deploy, a client abort); the conversation is intact, so start a fresh run to
continue.

Details: [`/errors/cancelled`](./agent-conversations/errors/cancelled).

### Anything else

Every other failure class has its own page. Start at
[the error catalog](./agent-conversations/errors/conversation-not-found) and use the page name
that matches the `error.docs_url` or `type` in the response.

## Where to go next

* [Full error catalog](./agent-conversations/errors/conversation-not-found) — every
  failure mode the API can return, with cause and fix.
* [Concepts: Model Context Protocol](https://modelcontextprotocol.io/) — the open standard
  behind `mcp_servers`. Helpful if you're wiring up a tool the platform doesn't ship by
  default.


# Agent Run Not Found
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/agent-run-not-found

The referenced agent run does not exist or is not accessible to your company.

**HTTP status:** 404

## When this error occurs

This page covers two distinct sources of the same failure class:

1. **Synchronous** — `GET /agents/runs/{id}` returned 404 because the run id is unknown to the
   system, or it belongs to a different company. The response detail is generic on purpose; the
   API doesn't leak existence across companies.
2. **Asynchronous** — `error.type = "AgentLoopAgentRunNotFound"` on a failed run row means the
   workflow's load activity tried to read the `agent_runs` row by id and got nothing back. This
   only happens if the row was deleted between API insert and workflow start (unusual; investigate
   via `log_id` and the workflow worker logs).

## How to fix

1. Verify the run id from the original `POST /agents/conversations/{id}/runs` response.
2. Verify your bearer token's company id matches the conversation that owns the run.
3. For the workflow-side variant, check the worker logs filtering on the `agent_run_<uuid>`
   Temporal workflow id; the load activity logs include the database query result.


# Run Cancelled
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/cancelled

The run was deliberately cancelled before it reached a final answer.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopCancelled"`

## When this error occurs

This is not a failure — it marks a run whose underlying workflow execution was **cancelled**
before it converged. Cancellation is a deliberate action (an operator or an automated process
cancelled the run's Temporal workflow), as opposed to the run hitting a genuine error.

A cancelled run shares the `failed` status with real errors, but carries this reserved
`error.type` so you can tell the two apart: `AgentLoopCancelled` means "stopped on purpose,"
not "broke."

When a run is cancelled, the platform also cancels its in-flight inference job in the data
plane, so the run row and the data-plane job end up in a consistent cancelled state — no orphan
job keeps running after the run is gone.

## How to handle it

Nothing is broken, so there is nothing to fix:

* **You expected the cancellation** — treat the run as stopped. Any partial work
  (`iterations_used`, `usage`, the message turns produced before the cancel) is preserved on
  the run row and in the conversation's message stream.
* **You did not expect it** — someone or something cancelled this run's workflow. Check who
  issued the cancel (operations tooling, a deploy that drained workers, a client-initiated
  abort). The conversation itself is intact; start a fresh run on it to continue from where the
  message stream left off.


# Concurrent Conversation Append
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/concurrent-conversation-append

Another writer advanced the conversation version while this run was in flight.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopConcurrentConversationAppend"`

## When this error occurs

`FinalizeAgentRunActivity` writes the run's produced messages + bumps `agent_conversations.version`
in a single CAS-protected transaction. This error means the CAS failed: between the API's
pre-flight version check at `POST .../runs` time and the workflow's finalize, another writer
(a concurrent run, or a manual database edit) bumped the version.

Only one run at a time can finalize cleanly on a given conversation. The losing run terminates
with this error and its messages are not persisted.

## How to fix

This is **expected behavior under concurrent runs on the same conversation**. Treat it as a
retryable signal:

1. `GET /agents/conversations/{id}/messages?since={prior_version}` to see what landed.
2. Start a fresh run with the new version as `expected_version`. Re-post the original prompt
   or tool outputs.

If you don't expect concurrency on this conversation, look at the messages stream — the
"other" writer's identity will be in the latest message's `run_id`.


# Conversation Not Found
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/conversation-not-found

The referenced agent conversation does not exist or is not accessible to your company.

**HTTP status:** 404

## Error message

```
Resource not found or not accessible.
```

## When this error occurs

`GET /agents/conversations/{id}`, `GET /agents/conversations/{id}/messages`, or
`POST /agents/conversations/{id}/runs` was called with an `{id}` that either:

* does not exist in the system, or
* belongs to a different company than the bearer token's `company_id`.

The response detail is deliberately generic and does not distinguish between the two cases —
the API does not leak existence across company boundaries.

## How to fix

1. Confirm the conversation id is correct (no typos, no truncation).
2. Confirm the token you're using belongs to the company that owns the conversation —
   `GET /whoami` returns your `company_id`.
3. If you recently created the conversation, confirm the `POST /agents/conversations` response
   returned the same `id` you're now passing.

## Variant: a failed run carrying this code

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopConversationNotFound"`

A run can also fail with this code (rather than a 404) when the agent workflow loads the run's
context and finds the run's conversation row gone. A run always references a conversation, so this
is a data-integrity violation that should not happen in practice — not a caller mistake. It is
non-retryable.

If you see this, report the `log_id` and run id to support; it points at a missing or deleted
`agent_conversations` row for an existing run.


# Duplicate Caller-Declared Tool Name
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/duplicate-caller-tool-name

Two entries in tools[] share the same name.

**HTTP status:** 400

## Error message

```
duplicate caller-declared tool names (must be unique within tools[]): {names}
```

## When this error occurs

Caller-declared tools (top-level `tools[]`) are looked up by exact name match when a model
emits a dash-free tool call. Two entries with the same `name` would make resolution
ambiguous (which `input_schema` does the model see? which `tools[]` entry does the call
resolve to?), so the API rejects the config up front.

## How to fix

Give each caller-declared tool a unique `name`. If two tools really need similar names,
differentiate them with a suffix (`confirm_booking_user`, `confirm_booking_admin`).


# Duplicate Tool Alias
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/duplicate-tool-alias

The same alias is used twice in mcp_servers.

**HTTP status:** 400

## Error message

```
duplicate mcp_server aliases (must be unique): {dupes}
```

## When this error occurs

Aliases namespace each MCP server's tools — when the model emits a call like
`{alias}-{tool_name}`, the workflow routes by `alias`. Two MCP servers with the same alias
would make routing ambiguous, so the API rejects them up front.

Caller-declared tools (in top-level `tools[]`) have **no** alias — the dash in the wire
name is the routing discriminator. Uniqueness only matters within `mcp_servers[]`.

## How to fix

Pick distinct aliases for each MCP server:

```json theme={null}
{
  "mcp_servers": [
    { "alias": "docs", ... },
    { "alias": "wiki", ... }
  ]
}
```

If you genuinely need two MCP servers for related capabilities, distinguish them
semantically (`docsv1`, `docsv2`) rather than reusing one alias.


# Incomplete Tool Outputs
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/incomplete-tool-outputs

The tool_outputs payload is missing some pending caller-declared tool calls or contains extras.

**HTTP status:** 400

## Error message

```
tool_outputs do not match the pending client tool calls — missing: {ids}; unexpected: {ids}
```

(One or both of the `missing`/`unexpected` clauses may be omitted if that side is empty.)

## When this error occurs

When resuming a `requires_action` run, **every** entry in
`pending_tool_calls` from the prior run must have a matching entry in the new run's
`payload.outputs`, and there must be no extra entries beyond those. The API enforces a strict
1:1 match. (Only caller-declared tool calls — dash-free `name`s — appear in `pending_tool_calls`;
MCP-resolved calls were settled inline by the platform.)

The two cases:

* **Missing** — a pending caller-declared tool call has no corresponding output. The model can't
  continue without all results.
* **Unexpected** — an output references a `tool_use_id` that *is* a caller-declared call but
  isn't in the current pending set. Usually means resuming with a payload that targets an older
  `requires_action` state.

## How to fix

1. `GET /agents/runs/{prior_run_id}` and copy the entire `pending_tool_calls` array verbatim.
2. Produce exactly one `outputs[]` entry per `pending_tool_calls[]` entry — same `tool_use_id`,
   same count.
3. Refetch `expected_version` before posting to avoid `version_conflict`.

If a tool call's result truly is unavailable (e.g. user dismissed the prompt), still post an
output with `is_error: true` and a short message — the model can then choose how to proceed.


# Inference Job Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/inference-job-failed

The model-inference job the agent run submitted reached a Failed terminal state.

**Surface:** `GET /agents/runs/{id}`, `error.type = "JobFailed"`

## When this error occurs

Each agent-loop iteration submits a model-inference job to the data plane and polls it to
completion. This error means that job reached a `Failed` terminal state rather than completing —
the inference itself never produced a usable result.

Common causes:

* Data Plane issues. No rights to invoke AI, for example
* The model provider rejected the request (throttling, content too long, an unsupported parameter).
* A transient data-plane / execution-cluster failure that exhausted the job's own retries.
* An inference configuration that only fails at execution time.

## How to fix

1. If the failure looks transient (provider throttling, a one-off cluster blip), start a new run on
   the conversation.
2. If it reproduces, report the `log_id` and run id to support — the `jobs.<job_id>` row holds the
   detailed failure reason.
3. Check the run's `effective_config` (model, max tokens, data plane); a persistently-failing job
   often points at a configuration the data plane can't satisfy.


# Invalid Caller-Declared Tool Name
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invalid-caller-tool-name

A tools[] entry has a name that is empty, contains a dash, or exceeds 64 characters.

**HTTP status:** 400

## Error message

```
invalid caller-declared tool names (must be non-empty, dash-free, and ≤ 64 chars): {names}
```

## When this error occurs

Caller-declared tools (under top-level `tools[]`) have no alias — the platform tells them
apart from MCP tools by the absence of a dash in the wire name. If a caller-declared
`tools[].name` contains a dash, it would collide with the MCP `{alias}-{tool}` shape and
the workflow can't safely route the call.

Validation rules for `tools[].name`:

* Non-empty.
* No dashes (`-`).
* ≤ 64 characters total (Bedrock's wire-name cap; MCP tools split this budget between
  alias + name, but caller-declared names use the full 64).

## How to fix

Rename the offending tool to a dash-free identifier:

```json theme={null}
{
  "tools": [
    { "name": "confirm_booking", "description": "...", "input_schema": {...} },
    { "name": "lookup_account", "description": "...", "input_schema": {...} }
  ]
}
```

Use underscores or camelCase instead of dashes for multi-word names.


# Invalid Effective Config
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invalid-effective-config

The workflow could not decode the run's effective_config JSONB into the expected shape.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopInvalidEffectiveConfig"`

## When this error occurs

When a workflow starts, its load activity reads `agent_runs.effective_config` (the merged
defaults + override snapshot produced by the API at run-creation time) and decodes it into the
workflow's typed config. This error means the JSONB doesn't match the workflow's current shape.

Causes:

* **Schema drift** — the API persisted a shape the workflow doesn't recognise yet (deploy
  ordering issue: workers older than the API).
* **Discriminator mismatch** — a discriminated union (`tool_choice.kind`, `payload.kind`, etc.)
  carries an unexpected discriminator value. The `error.message` field includes the failing
  JSON path.

## How to fix

If a redeploy is in progress, wait for the workers to catch up; the run row is already in
`failed` status, but you can re-post the same `client_op_id` after the deploy completes —
the API will return the existing terminal row (no retry) and you'll need to start a fresh run
with a new `client_op_id`. Production-wise, deploy workers before the API to avoid this race.

For caller-driven causes (unexpected `tool_choice` shape, etc.), inspect `error.message` for
the path and compare against the API docs for the relevant DTO.


# Invalid Request Payload
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invalid-request-payload

The workflow could not decode the run's request_payload JSONB into a known shape.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopInvalidRequestPayload"`

## When this error occurs

The workflow's load activity reads `agent_runs.request_payload` and expects one of:

* `{"kind": "user_message", "text": "..."}`
* `{"kind": "tool_outputs", "outputs": [...]}`

This error means the JSONB doesn't carry a recognised `kind` value or fails to decode the body
for the recognised kind. Most commonly: a payload that bypassed the API and was hand-inserted
into the database with an unexpected shape, or a future API version emitting a `kind` value
the workers don't recognise yet.

## How to fix

If the row was created through the API, this is a server-side schema-drift bug — file an
issue with the `log_id` and the run id. If the row was hand-seeded (e.g. via the
`AgentLoopStarter` test harness), check the payload JSON against the
`RunRequestPayload` shape.


# Invalid Stored Field
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invalid-stored-field

A persisted JSONB field on agent_conversations or agent_runs failed to decode.

**HTTP status:** 500

## Error message

The detail is generic — the actual field name and decode failure are in the server log,
correlated via `log_id`.

## When this error occurs

A `GET` on a conversation, run, or message list found a row whose JSONB column doesn't decode
into the expected case class. Concretely:

* `agent_conversations.defaults` doesn't decode as `ConversationDefaults`.
* `agent_runs.effective_config` doesn't decode as `ConversationDefaults`.
* `agent_runs.tool_choice` doesn't decode as a workflow `ToolChoice`.
* `agent_runs.pending_tool_calls` / `submitted_inference_job_ids` / `usage` shape drift.
* `agent_conversation_messages.content_blocks` doesn't decode as `List[ContentBlock]`.

This is almost always a **schema-drift bug** introduced by a deploy that changed the case-class
shape without a migration, or by a hand-edited row.

## How to fix

This is a server-side error — there's nothing the caller can do besides report the `log_id` to
support. Repeat-readers can `GET /agents/conversations/{id}/messages?since=N` to skip past the
broken row if only a single message is affected.


# Invalid Tool Alias
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invalid-tool-alias

One or more tool aliases do not match the required regex.

**HTTP status:** 400

## Error message

```
invalid tool aliases (must match ^[a-zA-Z][a-zA-Z0-9]{0,7}$): {bad-aliases}
```

## When this error occurs

Every entry under `defaults.mcp_servers[].alias` must match `^[a-zA-Z][a-zA-Z0-9]{0,7}$`:

* 1–8 characters total,
* starts with an ASCII letter,
* contains only ASCII letters and digits — **no underscores, dashes, or other punctuation**.

The dash is reserved as the separator in the wire form `{alias}-{tool_name}` that the model sees,
so aliases can't contain dashes themselves.

Caller-declared tools (under top-level `tools[]`) have no alias — their `name` is what the model
sees on the wire. See
[Invalid Caller-Declared Tool Name](./invalid-caller-tool-name) for the analogous rules on those
names.

## How to fix

| Bad                            | Fix                  |
| ------------------------------ | -------------------- |
| `my_docs` (underscore)         | `mydocs`             |
| `docs-v2` (dash)               | `docsv2`             |
| `1docs` (digit-leading)        | `docs1`              |
| `verylongdocsalias` (>8 chars) | `docs` or `verylong` |

Re-post the request with the corrected aliases.


# Internal Invariant Violated
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/invariant-violated

A workflow finalize step found a state that should be unreachable — a server-side bug.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopFinalizeInvariantViolated"`

## When this error occurs

`FinalizeAgentRunActivity` enforces several invariants that should be impossible to violate
given correct upstream code (e.g. attempting to finalize a run with a non-terminal status,
mismatch between the requested terminal status and the populated `TerminalFields`). This error
means one of those invariants tripped.

It is **always a server-side bug**, not a caller-recoverable condition.

## How to fix

Report the `log_id` and the run id to support. The workflow logs include the activity input
that triggered the assertion. Re-running the same conversation with a fresh run id will
typically work — the bug is deterministic on the inputs that triggered it.


# Inference Job Result Decode Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/job-result-decode-failed

The workflow could not parse the inference job's result blob into a known model output.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopJobResultDecodeFailed"`

## When this error occurs

After each inference iteration, the workflow polls the corresponding `jobs` row and parses
`jobs.result` into `ModelInferenceRunResult`. This error means the parse failed — the result
blob doesn't match the shape the workflow expects.

Almost always a **server-side schema-drift bug** caused by a data-plane deploy that changed
the result shape without the workflow being updated.

## How to fix

Caller has nothing to do. Report the `log_id` and the run id to support. The data plane's
`jobs.<job_id>` row contains the raw result for inspection.


# Max Iterations Exceeded
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/max-iterations-exceeded

The agent loop ran the configured maximum number of iterations without converging on a final answer.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopMaxIterationsExceeded"`

## When this error occurs

Each run's `effective_config.max_iterations` caps how many inference rounds the workflow will
attempt before giving up. If the model keeps emitting tool calls without producing a final
answer, the workflow terminates with this error rather than running indefinitely.

The cap acts as both a safety net (against runaway tool-loops) and a cost ceiling — each
iteration costs one inference job.

## How to fix

Depending on the cause:

* **Cap is too low** — increase `max_iterations` in `defaults` (for the conversation) or in
  `config_override.max_iterations` (per-run). The cap of 3 is a sensible default for trivial
  prompts; multi-step research flows often need 8–15.
* **Model is looping** — examine the message stream. Repeated calls to the same tool with the
  same arguments suggest the prompt isn't giving the model enough information to terminate.
  Improve the system prompt, narrow the tool catalog, or pre-fill more context in the user
  message.
* **One tool is unreliable** — repeated failures from the same MCP tool can also cause looping.
  Check `tool_result.is_error` patterns in the message stream.


# MCP Discovery Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/mcp-discovery-failed

A registered MCP server failed to respond to tools/list — either at the start of a run, or during synchronous tool_choice validation.

**HTTP / status:** Surfaces on two paths.

* **Synchronous (`POST /agents/conversations/{id}/runs`)**: HTTP 400 RFC 7807 with
  `type` URL pointing to this page. Triggered by `tool_choice: { kind: "specific_tool",
  mcp_alias: "...", name: "..." }` — the platform does a targeted `tools/list` against
  the named server to confirm the tool exists before accepting the run. The 400 (not 5xx)
  reflects that the failure is caller-actionable: the MCP server you registered isn't
  responding, and fixing it is on you.
* **Asynchronous (run-time)**: the run lands in `status: failed` with
  `error.type = "AgentLoopMcpDiscoveryFailed"`. Triggered at workflow start when the
  load activity fans out one `tools/list` per registered MCP server to build the
  inference catalog.

## When this error occurs

The Agent Conversations API doesn't keep a cached MCP catalog. Every run rediscovers each
registered server's `tools/list`, and `tool_choice` with an `mcp_alias` re-discovers that
single server synchronously. Common failure modes:

* The MCP server URL is unreachable (DNS, firewall, TLS handshake failure).
* The MCP server returns 5xx (transient outage or unhandled exception server-side).
* The MCP server responds, but the body isn't a well-formed JSON-RPC envelope.
* The MCP server doesn't implement `tools/list` (responds with `-32601 method not found`).

## How to fix

* Verify the MCP server URL is reachable from the platform's outbound workers. For
  public servers, try `curl -X POST $URL` with a minimal initialize JSON-RPC body. For
  private/VPC-only servers, confirm the network path from the data plane.
* Check the server's logs for crashes or rate-limit errors around the failure timestamp.
* Confirm the server exposes both `initialize` and `tools/list` per the
  [Model Context Protocol](https://modelcontextprotocol.io/) wire spec.
* If the MCP server is permanently down, drop the entry from `mcp_servers` (or override
  the conversation defaults via `config_override.mcp_servers` on the affected run) — the
  agent loop can't proceed without a working catalog for every registered server.

## See also

* [`tool_choice` reference](/reference/architecture/agent-conversations#tool-choice) —
  the API field that drives the synchronous discovery hop.
* [Unknown Tool Choice Name](/reference/architecture/agent-conversations/errors/unknown-tool-choice-name)
  — discovery succeeded but the named tool wasn't in the returned catalog.


# MCP Protocol Error
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/mcp-protocol-error

The MCP server returned a response that is not valid JSON-RPC or violates the MCP protocol.

**Surface:** `GET /agents/runs/{id}`, `error.type = "McpProtocol"`

## When this error occurs

The workflow's `McpToolCallActivity` sent a JSON-RPC `tools/call` request and got back a
response that:

* Wasn't valid JSON, or
* Had the wrong `Content-Type` (expected `application/json` or `text/event-stream`), or
* Was JSON but didn't match JSON-RPC 2.0 (missing `jsonrpc`, wrong `id`, structural mismatch),
* Was a JSON-RPC error with a code outside the documented range that the workflow knows how to
  classify (`-32601` → UnknownTool, `-32602` → ToolValidation, anything else → here).

This error is **not retried** by the workflow — the response is structurally broken, so a
retry would hit the same response.

## How to fix

This is almost always a problem on the MCP server side. Owners of the server should check
their logs around the failed call timestamp. The workflow logs include the raw response body
for support to inspect.


# MCP Server Error
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/mcp-server-error

The MCP server returned a 5xx HTTP status or the workflow hit a network error reaching it.

**Surface:** `GET /agents/runs/{id}`, `error.type = "McpServerError"`

## When this error occurs

The HTTP call to the MCP server failed at the transport or 5xx-status layer:

* Connection refused / DNS resolution failed.
* TLS handshake error.
* 500 / 502 / 503 / 504 HTTP status.
* Connection reset mid-stream (for streaming `text/event-stream` responses).

Unlike `McpProtocol` (which means "we got a response but couldn't make sense of it"), this
class means "we never got a usable response at all".

This error **is retried** by the workflow with exponential backoff (1s → 2s → 4s → 8s →
giving up after 5 attempts). If you're seeing this on the run row, all retries exhausted.

## How to fix

* Verify the MCP server is reachable from the workers' VPC — try a manual JSON-RPC `tools/list`
  curl from inside the cluster.
* For self-hosted MCP servers, check the server's own logs around the failed timestamp.
* For third-party MCP servers (e.g. `docs.narrative.io/mcp`), check their status page or wait
  and retry — the failure is on their side.


# MCP Tool Argument Validation Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/mcp-tool-validation

The MCP server reported invalid-params (JSON-RPC -32602) for the call arguments, or local arg-shape check failed.

**Surface:** `GET /agents/runs/{id}`, `error.type = "ToolValidation"`

## When this error occurs

The model produced arguments that didn't satisfy the tool's input schema:

* Required field missing.
* Field type didn't match (`string` expected, `integer` given).
* `additionalProperties: false` rejected an extra key.

The MCP server rejected the call with JSON-RPC `-32602`. (The platform doesn't carry a copy of
the input schema for MCP tools — it discovers `tools/list` per run and forwards the schema to
the model verbatim, then forwards model output to the server. Validation happens server-side.)

## How to fix

* **Improve the input\_schema** in the run's config to be tighter — required fields, strict
  enums, `additionalProperties: false`. Tighter schemas → fewer model mistakes.
* **Strengthen the tool description** with a concrete example so the model copies the shape.
* This error is **not retried** by the workflow; the activity tag is in
  `RetryOptions.setDoNotRetry`. The run fails fast and the caller can re-prompt.


# Unknown MCP Tool
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/mcp-unknown-tool

The MCP server reported method-not-found (JSON-RPC -32601) for the called tool.

**Surface:** `GET /agents/runs/{id}`, `error.type = "UnknownTool"`

## When this error occurs

The workflow called an MCP server's `tools/call` for a tool name that the server doesn't
recognise. JSON-RPC reports this as `-32601 method not found`.

Discovery normally prevents this — the workflow calls `tools/list` at run start and uses only
the names the server actually exposes. This error fires when something changes between
discovery and the `tools/call`:

* The server is behind a load balancer where some instances have the tool and some don't
  (deploy in progress on the server side).
* The server's catalog was edited between the run's `tools/list` and `tools/call` (rare race;
  same-run discovery + call usually happen within a few hundred ms).
* The model hallucinated a tool name that wasn't in the discovered catalog — but in that case
  the workflow would normally reject earlier with
  [Unknown Tool Alias](./unknown-tool-alias). Seeing `UnknownTool` instead suggests a
  classifier mismatch worth investigating.

## How to fix

1. Verify the tool name against the MCP server's current `tools/list` (curl it directly with a
   JSON-RPC client) — has the catalog drifted?
2. If the discrepancy is intermittent, fix the server-side deploy first.
3. If the catalog now permanently lacks the tool the agent loop was calling, either restore the
   tool or update your system prompt to stop referencing it.


# No Assistant Turn Yet
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/no-assistant-turn

Cannot submit tool_outputs on a conversation that has not yet produced an assistant turn.

**HTTP status:** 400

## Error message

```
cannot submit tool_outputs: conversation {id} has no assistant turn yet
```

## When this error occurs

`POST /agents/conversations/{id}/runs` with `payload.kind = tool_outputs` requires a prior
assistant turn carrying the `tool_use` blocks that the outputs reference. This error means the
conversation has none yet:

* It's a brand-new conversation with no runs at all.
* All prior runs terminated with `final_text` (no tool calls), so there's nothing pending.
* All prior runs failed before producing an assistant turn.

## How to fix

If you intended to start a new conversational turn from the user side, change the payload to
`{"kind": "user_message", "text": "..."}` — `tool_outputs` is exclusively for resuming a
`requires_action` run.

If you intended to resume a paused run, double-check the conversation id — `tool_outputs` is
posted onto the same conversation that contained the `requires_action` run, not a fresh one.


# Not A Client Tool Call
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/not-a-client-tool-call

A tool_outputs entry references a tool_use_id whose name has the MCP {alias}-{tool} shape.

**HTTP status:** 400

## Error message

```
tool_use_id(s) refer to server-resolved calls (no caller output expected): {ids}
```

## When this error occurs

Each tool call the model emits is routed by the shape of its wire name:

* Dash-prefixed (`{alias}-{tool}`) → MCP-resolved. The platform calls the MCP server inline
  and persists the result. These calls never appear in `pending_tool_calls`.
* Dash-free (`{tool}`) → caller-declared. The run pauses at `requires_action` and the
  caller posts the output as `tool_outputs` on the next run.

This error means a `tool_outputs[].tool_use_id` references an assistant turn block whose
`name` contains a dash — an MCP call the platform already settled. There's nothing for the
caller to contribute.

## How to fix

Filter your `tool_outputs` to only the ids whose corresponding `name` in the prior
`requires_action` response is dash-free. The `pending_tool_calls` list from
`GET /agents/runs/{id}` already excludes MCP-resolved ids — copy from there rather than
re-deriving from the message stream.


# Output Schema Decode Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/schema-decode-failed

The model's final answer did not validate against the configured output_format_schema.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopSchemaDecodeFailed"`

## When this error occurs

When the model produces a final answer (no tool calls, just a content message), the workflow
parses the content against the run's output schema. This error means the parse failed:

* **Text mode** (no `output_format_schema` supplied): the model's response could not be
  decoded as the platform's default text-bearing shape (a top-level `text` string).
* **Structured mode** (`output_format_schema` supplied): the model returned JSON that did not
  conform to the caller's schema — for example plain prose instead of a structured object,
  or an object missing a required field.
* The model returned multiple text blocks; only the first is considered.

## How to fix

* **Tighten the system prompt** — explicitly tell the model "respond with a JSON object
  matching this schema; do not include any other text." Models often add prose when the schema
  isn't reinforced in instructions.
* **Simplify the schema** — Bedrock's structured-output sampler enforces a subset of Draft
  2020-12; deeply nested or exotic constructs (e.g. unconstrained `oneOf`) are more likely to
  trip the decoder.
* **Increase `max_tokens`** — sometimes the model is truncated mid-JSON.
* **Inspect the last assistant turn** in the message stream to see exactly what the model
  produced.

<Note>
  Before V2026.x, every caller schema had to declare a top-level `text: string` because the
  workflow unconditionally extracted `structuredOutput.text`. That requirement is gone:
  structured-mode runs now return the caller's schema verbatim on `final_structured_output`,
  and text-mode runs return a plain string on `final_text`. See
  [Agent Conversations](/reference/architecture/agent-conversations) for the full response
  contract.
</Note>


# Tool Wire Name Too Long
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/tool-name-too-long

A tool wire name exceeds Bedrock's 64-character limit.

**HTTP status:** 400

## When this error occurs

AWS Bedrock's Converse API rejects any tool name longer than 64 characters. The agent loop's
wire name for an MCP tool is the stitched form `{alias}-{tool_name}`; for a caller-declared
tool it's the bare `name`.

In the current implementation:

* **Caller-declared tool names** are validated synchronously at conversation create against
  the 64-char cap (along with dash-free / non-empty rules). A violation surfaces as
  [Invalid Caller-Declared Tool Name](./invalid-caller-tool-name), not this one.
* **MCP tools** are discovered at run start, so the API can't pre-check their wire-name length.
  If a discovered `{alias}-{tool_name}` exceeds 64 chars, Bedrock rejects the inference call
  and the run lands `failed` with a downstream
  [Inference Job Result Decode Failed](./job-result-decode-failed) or
  [MCP Protocol Error](./mcp-protocol-error).

This error class remains in the API for stable docs URLs and possible future use (e.g. a
post-discovery validation step). Current callers shouldn't normally see it.

## How to fix

If you do see it:

* **MCP alias too long** — shorten the alias (`docs` over `verylonglyalias`).
* **MCP tool name too long** — fix it on the MCP server side; you can't rename discovered tools
  client-side.
* **Caller-declared name too long** — shorten the entry in `tools[]`.

Aim for `len(alias) + 1 + len(name) ≤ 64` on the stitched wire form.


# Unexpected Error
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/unexpected

Catch-all entry for workflow incident codes that the current API version does not recognise.

**Surface:** `GET /agents/runs/{id}`

## When this error occurs

`RunErrorDto.docs_url` points here when the run's `error.type` is not in the API's catalog —
either because:

* The workflow workers have been upgraded ahead of the API and emit a new incident code the
  API doesn't recognise yet.
* An uncaught JVM exception escaped into the workflow body's `classifyForCleanup`, and the
  type tag is the exception's class name rather than a known `AgentLoop*` tag.

The original incident code is still present on `error.type` for log correlation.

## How to fix

Caller has nothing actionable. Report the `log_id` and the unrecognised `error.type` value
to support. If a redeploy of the API is pending, the next deploy will likely add the missing
catalog entry.


# Unknown Tool Alias
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/unknown-tool-alias

The model emitted a tool_use whose name is neither a registered MCP wire form nor a caller-declared tool.

**Surface:** `GET /agents/runs/{id}`, `error.type = "AgentLoopUnknownToolAlias"`

## When this error occurs

The workflow routes every emitted tool call by the shape of its wire name:

* **Dash-prefixed (`{alias}-{tool}`)**: alias must match an `mcp_servers[].alias` for this
  run's effective config.
* **Dash-free (`{tool}`)**: name must match an entry in caller-declared `tools[]`.

This error means neither lookup hit — the model emitted a name that doesn't resolve to a
registered tool. There's nothing for the platform to call and nothing meaningful to surface
in `pending_tool_calls`, so the run fails.

Common causes:

* The system prompt or the tool catalog references a tool that wasn't actually registered
  for this run (typo in alias / name).
* The model hallucinated a tool name (rare with `tool_choice` set; more common with `auto`
  and a loose prompt).

## How to fix

* Make sure every tool the system prompt mentions also appears in `mcp_servers` or
  `tools[]` for the run's effective config.
* Tighten the prompt: "Only use the tools listed below; do not invent new ones."
* Use `tool_choice: {kind: any}` to force a real tool, or `specific_tool` (with optional
  `mcp_alias`) if you know which one is expected next.


# Unknown Tool Choice MCP Alias
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/unknown-tool-choice-mcp-alias

tool_choice.mcp_alias does not match any registered MCP server.

**HTTP status:** 400

## Error message

```
tool_choice references unknown mcp_alias '{alias}' — not a registered MCP server
```

## When this error occurs

When `tool_choice: { kind: "specific_tool", mcp_alias: "...", name: "..." }` carries an
`mcp_alias`, the platform looks the alias up in the effective config's `mcp_servers[]`. If
no MCP server has that alias for the run (typo? missing entry in a `config_override` that
replaced the conversation's default list?), the call is rejected synchronously.

## How to fix

Use one of the registered aliases — or drop `mcp_alias` entirely if you meant a
caller-declared tool. Sample shapes:

```json theme={null}
// MCP-pinned (alias "docs" must be in mcp_servers)
{ "kind": "specific_tool", "mcp_alias": "docs", "name": "search_kb" }

// Caller-declared (no mcp_alias)
{ "kind": "specific_tool", "name": "confirm_booking" }
```

Remember that `config_override.mcp_servers` replaces the conversation default wholesale —
the run sees only what's listed in the override (or, if absent, the defaults). If you're
overriding the list, every alias your `tool_choice` references has to be in that override.


# Unknown Tool Choice Name
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/unknown-tool-choice-name

tool_choice references a specific tool name that is not in the conversation catalog.

**HTTP status:** 400

## Error message

```
tool_choice references unknown tool name '{name}' — not found in the catalog
```

## When this error occurs

`tool_choice: { kind: "specific_tool", ... }` pins the model to a single tool on iteration
1 of the run. The platform validates that the named tool exists in the effective catalog
**before** kicking off the workflow — better to fail synchronously here than later as a
confusing Bedrock-side error inside the agent loop.

This error surfaces when either:

* `mcp_alias` is absent and `name` doesn't appear in caller-declared `tools[]`.
* `mcp_alias` is present (and the alias is registered), but `name` doesn't appear in that
  MCP server's declared `tools[]`.

## How to fix

Make sure `tool_choice.name` matches the catalog exactly:

```json theme={null}
// Caller-declared target
{ "kind": "specific_tool", "name": "confirm_booking" }

// MCP-pinned target
{ "kind": "specific_tool", "mcp_alias": "docs", "name": "search_kb" }
```

If you only know the wire form (`docs-search_kb`), split it: prefix before the first dash
becomes `mcp_alias`, suffix becomes `name`.


# Unknown tool_use_id
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/unknown-tool-use-id

A tool_outputs payload references a tool_use_id that is not on the latest assistant turn.

**HTTP status:** 400

## Error message

```
tool_use_id(s) not found on latest assistant turn: {ids}
```

## When this error occurs

`POST /agents/conversations/{id}/runs` with `payload.kind = tool_outputs` was called, and at
least one of the `outputs[].tool_use_id` values does not match any `tool_use` block on the
conversation's latest assistant turn.

Common causes:

* Hand-edited or copy-pasted `tool_use_id` with a typo.
* Resuming an old `requires_action` run after the conversation has already advanced past it
  (the latest assistant turn is now different).
* Posting `tool_outputs` to a conversation whose previous run terminated as `completed` rather
  than `requires_action` (in which case there's nothing pending).

## How to fix

1. `GET /agents/runs/{run_id}` for the run that returned `requires_action`. Use the
   `pending_tool_calls[].tool_use_id` values from there verbatim.
2. If you've lost the run id, walk back via
   `GET /agents/conversations/{id}/messages?since=0` and look for the latest assistant turn —
   the `tool_use` blocks within carry the ids you need.
3. Refetch the conversation's `version` and pass it as `expected_version` to avoid posting onto
   a stale head.


# Conversation Version Conflict
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/version-conflict

The expected_version in the request does not match the conversation's current head version.

**HTTP status:** 409

## Error message

```
conversation version mismatch: expected {N}, current {M}
```

Example: `conversation version mismatch: expected 2, current 5`.

## When this error occurs

`POST /agents/conversations/{id}/runs` was called with an `expected_version` that no longer
matches the conversation's current head. This is an [optimistic-concurrency](https://en.wikipedia.org/wiki/Optimistic_concurrency_control)
guard: the API rejects the run before starting a workflow that would conflict at finalize time.

It happens when another writer (a previous run, or a concurrent caller) advanced the conversation
between the time you read `version` and the time you posted the new run.

## How to fix

1. Re-read the conversation via `GET /agents/conversations/{id}` to get the current `version`.
2. If you were resuming from a `requires_action` run, also re-read the latest assistant turn via
   `GET /agents/conversations/{id}/messages?since={your_known_version}` to learn whether the
   pending tool calls have changed.
3. Retry the run with the fresh version.

## Race-free continuation pattern

For tool-output resumes, always refetch the conversation's `version` immediately before posting —
don't rely on a value cached from earlier in the conversation's lifecycle.


# Workflow Start Failed
Source: https://docs.narrative.io/reference/architecture/agent-conversations/errors/workflow-start-failed

The API persisted the agent_runs row but failed to start the Temporal workflow.

**HTTP status:** 500

## When this error occurs

`POST /agents/conversations/{id}/runs` succeeded in writing the `agent_runs` row, but
`WorkflowClient.start` raised before the Temporal frontend acknowledged the workflow execution.
Common causes (visible only in the server log):

* Temporal frontend is unreachable (network partition, deploy-in-flight).
* The workflow id `agent_run_<uuid>` collides with an existing execution (very rare;
  effectively impossible given the UUID).
* The Temporal namespace doesn't exist or the worker isn't connected to it.

The row remains in `pending` status. The cleanup activity will eventually flip it to `failed`
via a heartbeat / supervisor mechanism, or operators can re-attempt the workflow start manually.

## How to fix

This is a transient platform error. Retry the same request with the **same `client_op_id`** —
the idempotency guarantee returns the existing `agent_runs` row (still `pending` or possibly
already `running` if the workflow eventually did start). If the underlying Temporal outage has
resolved, the workflow start will succeed on retry.


# Dataset Statistics Reference
Source: https://docs.narrative.io/reference/architecture/dataset-statistics

Complete reference for dataset statistics: available metrics, type compatibility, configuration schema, and validation errors

This page is the definitive reference for dataset statistics — the column-level metrics the platform computes, their type compatibility, and the configuration API for controlling what gets computed and when.

For background on what statistics are and why they matter, see [Dataset Statistics](/concepts/primitives/dataset-statistics). To configure statistics through the platform UI, see the [configuring dataset statistics](/guides/platform/configuring-dataset-statistics) guide.

## API endpoints

| Method   | Path                                              | Description                                    |
| -------- | ------------------------------------------------- | ---------------------------------------------- |
| `PUT`    | `/datasets/{dataset_id}/statistics-configuration` | Create or replace the statistics configuration |
| `GET`    | `/datasets/{dataset_id}/statistics-configuration` | Retrieve the current configuration             |
| `DELETE` | `/datasets/{dataset_id}/statistics-configuration` | Disable statistics (stop computing)            |

All endpoints require a Bearer token with dataset write permissions.

## Configuration schema overview

A statistics configuration has four top-level fields:

```json theme={null}
{
  "defaults": { ... },       // required — global stat selection
  "refresh": { ... },        // required — when to recompute
  "dataset": { ... },        // optional — overrides for dataset columns
  "rosetta_stone": { ... }   // optional — overrides for Rosetta Stone attributes
}
```

### `defaults` (required)

The global stat selection applied to every eligible field. Must include `enabled_stats` — there is no parent to inherit from at this level.

```json theme={null}
{
  "enabled_stats": ["value_count", "null_value_count", "histogram"],
  "stat_options": {
    "histogram": { "max_bins": 100, "overflow": "truncate" }
  }
}
```

### `refresh` (required)

Controls when statistics are recomputed. Exactly one trigger type:

| Trigger     | Payload                                                 | Behavior                                |
| ----------- | ------------------------------------------------------- | --------------------------------------- |
| `cron`      | `{ "trigger": "cron", "cron_expression": "0 0 * * *" }` | Recompute on a UTC cron schedule        |
| `on_update` | `{ "trigger": "on_update" }`                            | Recompute whenever dataset data changes |
| `manual`    | `{ "trigger": "manual" }`                               | Only compute on demand                  |

### `dataset` and `rosetta_stone` (optional)

Namespace-level overrides. Each namespace has:

* **`scope`** — overrides defaults for all fields in this namespace
* **`fields`** — per-field overrides for individual columns/attributes

At least one of `scope` or `fields` must be provided if the namespace is present.

## Supported statistics

Each statistic is computed per column. The computation column shows the underlying SQL-level operation.

| Stat name               | Description                                                                               | Computation                        |
| ----------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------- |
| `value_count`           | Total number of non-null values in the column                                             | `COUNT(column)`                    |
| `null_value_count`      | Number of null values in the column                                                       | `COUNT(*) - COUNT(column)`         |
| `nan_value_count`       | Number of NaN (not-a-number) values — floating-point columns only                         | `COUNT_IF(IS_NAN(column))`         |
| `lower_bound`           | Minimum value in the column                                                               | `MIN(column)`                      |
| `upper_bound`           | Maximum value in the column                                                               | `MAX(column)`                      |
| `approx_count_distinct` | Approximate number of distinct values, using HyperLogLog for efficiency on large datasets | `APPROX_COUNT_DISTINCT(column)`    |
| `count_distinct`        | Exact number of distinct values                                                           | `COUNT(DISTINCT column)`           |
| `histogram`             | Frequency distribution of values across distinct buckets                                  | Aggregation over value frequencies |
| `mean`                  | Arithmetic mean — numeric columns only                                                    | `AVG(column)`                      |
| `standard_deviation`    | Population standard deviation — numeric columns only                                      | `STDDEV(column)`                   |
| `completeness`          | Ratio of non-null values to total rows (0 to 1)                                           | `COUNT(column) / COUNT(*)`         |

<Warning>
  `approx_count_distinct` and `count_distinct` cannot both appear in the same `enabled_stats` list. Choose one. See [Conflicting Distinct Count Methods](/reference/architecture/dataset-statistics/errors/distinct-conflict).
</Warning>

## Type compatibility matrix

Not all statistics apply to all data types. The table below shows which statistics are computed for each type.

| Type              | Supported statistics                                                                                                                                                                          |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DoubleType`      | All 11: value\_count, null\_value\_count, nan\_value\_count, approx\_count\_distinct, count\_distinct, lower\_bound, upper\_bound, histogram, mean, standard\_deviation, completeness         |
| `LongType`        | 10 (all except nan\_value\_count): value\_count, null\_value\_count, approx\_count\_distinct, count\_distinct, lower\_bound, upper\_bound, histogram, mean, standard\_deviation, completeness |
| `StringType`      | 8: value\_count, null\_value\_count, approx\_count\_distinct, count\_distinct, lower\_bound, upper\_bound, histogram, completeness                                                            |
| `TimestampTzType` | 8: value\_count, null\_value\_count, approx\_count\_distinct, count\_distinct, lower\_bound, upper\_bound, histogram, completeness                                                            |
| `BooleanType`     | 6: value\_count, null\_value\_count, approx\_count\_distinct, count\_distinct, histogram, completeness                                                                                        |
| `ArrayType`       | 2: value\_count, null\_value\_count                                                                                                                                                           |
| `ObjectType`      | 2: value\_count, null\_value\_count                                                                                                                                                           |

<Note>
  `DoubleType` is the only type that supports `nan_value_count`, since NaN is a floating-point concept. `mean` and `standard_deviation` are limited to numeric types (`DoubleType` and `LongType`). `lower_bound` and `upper_bound` require ordered types (numeric, timestamp, or string).
</Note>

## Inheritance model

The configuration uses a layered inheritance chain:

```
defaults → namespace scope → field override → nested node overrides
```

Two distinct rules apply:

### 1. Stat set (`enabled_stats`) — all-or-nothing replacement

When a level provides `enabled_stats`, it **completely replaces** the parent's stat set. There is no merging.

### 2. Stat options (`stat_options`) — per-property inheritance

When a level provides only `stat_options` (without `enabled_stats`), it inherits the stat set from its parent and overrides only the specified option properties. Unspecified properties keep the value from the **resolved parent** — not from the original defaults.

This means overrides accumulate through the chain:

```
defaults:  histogram(max_bins=50, overflow=none)
    ↓ scope overrides max_bins
scope:     histogram(max_bins=200, overflow=none)
    ↓ field overrides overflow
field:     histogram(max_bins=200, overflow=truncate)
```

The field gets `max_bins=200` from scope, not `max_bins=50` from defaults.

### Primitive vs. non-primitive fields

Fields map to one of two node kinds based on their schema type:

| Node kind         | Schema types                               | Available config fields         |
| ----------------- | ------------------------------------------ | ------------------------------- |
| **Primitive**     | string, integer, float, boolean, timestamp | `enabled_stats`, `stat_options` |
| **Non-primitive** | object, array                              | `self`, `properties`, `items`   |

You cannot mix primitive and non-primitive fields on the same node. See [MixedNodeConfig](/reference/architecture/dataset-statistics/errors/mixed-node-config).

For non-primitive nodes:

* **`self`** — stats on the container column itself (only `null_value_count` is supported)
* **`properties`** — explicit configs for named children of an object (opt-in, not inherited)
* **`items`** — config for array element nodes

## Example 1: Simple defaults-only configuration

The simplest useful configuration: define defaults and a refresh trigger, and every eligible field gets the same stats.

**Goal:** Compute `value_count`, `null_value_count`, and `completeness` for all fields, refreshing daily at midnight UTC.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "completeness"]
  },
  "refresh": {
    "trigger": "cron",
    "cron_expression": "0 0 * * *"
  }
}
```

That's it. The server applies these three stats to every compatible column in both the dataset schema and Rosetta Stone attributes. Stats incompatible with a field's type (e.g., `mean` on a string column) are automatically filtered.

### Adding histogram with custom options

Extend the defaults to include a histogram with 200 bins and truncation overflow:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "completeness", "histogram"],
    "stat_options": {
      "histogram": {
        "max_bins": 200,
        "overflow": "truncate"
      }
    }
  },
  "refresh": {
    "trigger": "on_update"
  }
}
```

### Disabling stats for a specific namespace

If you want stats on dataset columns but not Rosetta Stone attributes:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count"]
  },
  "refresh": { "trigger": "manual" },
  "rosetta_stone": {
    "scope": {
      "enabled_stats": []
    }
  }
}
```

The empty `enabled_stats: []` explicitly disables all stats for Rosetta Stone fields.

## Example 2: Complex configuration with nested structures

**Scenario:** A dataset with these columns:

* `user_id` (string) — needs exact distinct count
* `price` (float) — needs histogram with custom bins
* `address` (object with children `city`, `zip`, `state`) — need stats on `city` and `zip` only
* `tags` (array of strings) — need stats on array items, and null count on the array column itself

### Step 1: Set defaults

Start with a broad set of defaults:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram", "completeness"],
    "stat_options": {
      "histogram": { "max_bins": 50 }
    }
  },
  "refresh": { "trigger": "cron", "cron_expression": "0 3 * * 1" }
}
```

### Step 2: Add namespace scope to tune histogram bins

Override histogram bins for all dataset fields to 200, while keeping the stat set inherited from defaults:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram", "completeness"],
    "stat_options": {
      "histogram": { "max_bins": 50 }
    }
  },
  "refresh": { "trigger": "cron", "cron_expression": "0 3 * * 1" },
  "dataset": {
    "scope": {
      "stat_options": {
        "histogram": { "max_bins": 200 }
      }
    }
  }
}
```

Because scope provides only `stat_options` (no `enabled_stats`), it inherits the full stat set from defaults. Only `max_bins` is overridden — `overflow` retains the server default of `none`.

### Step 3: Add per-field overrides

Now add the field-specific configurations:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram", "completeness"],
    "stat_options": {
      "histogram": { "max_bins": 50 }
    }
  },
  "refresh": { "trigger": "cron", "cron_expression": "0 3 * * 1" },
  "dataset": {
    "scope": {
      "stat_options": {
        "histogram": { "max_bins": 200 }
      }
    },
    "fields": [
      {
        "field_name": "user_id",
        "enabled_stats": ["value_count", "count_distinct"]
      },
      {
        "field_name": "price",
        "stat_options": {
          "histogram": { "overflow": "truncate" }
        }
      },
      {
        "field_name": "address",
        "self": {
          "enabled_stats": ["null_value_count"]
        },
        "properties": [
          {
            "path": "city",
            "enabled_stats": ["value_count", "approx_count_distinct"]
          },
          {
            "path": "zip",
            "enabled_stats": ["value_count", "null_value_count", "histogram"],
            "stat_options": {
              "histogram": { "max_bins": 1000, "overflow": "truncate" }
            }
          }
        ]
      },
      {
        "field_name": "tags",
        "self": {
          "enabled_stats": ["null_value_count"]
        },
        "items": {
          "enabled_stats": ["value_count", "approx_count_distinct", "histogram"]
        }
      }
    ]
  }
}
```

### What each field resolves to

| Field            | Resolution                                                                         | Effective stats                                                                                 |
| ---------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `user_id`        | `enabled_stats` replaces inherited set                                             | `value_count`, `count_distinct`                                                                 |
| `price`          | Inherits stat set from scope (which inherited from defaults); overrides `overflow` | `value_count`, `null_value_count`, `histogram(max_bins=200, overflow=truncate)`, `completeness` |
| `address` (self) | Only `null_value_count` (the only stat supported for non-primitive self)           | `null_value_count`                                                                              |
| `address.city`   | `enabled_stats` replaces inherited set                                             | `value_count`, `approx_count_distinct`                                                          |
| `address.zip`    | `enabled_stats` with custom histogram options                                      | `value_count`, `null_value_count`, `histogram(max_bins=1000, overflow=truncate)`                |
| `address.state`  | Not listed in `properties` — receives no stats                                     | (none)                                                                                          |
| `tags` (self)    | Only `null_value_count` for the array column                                       | `null_value_count`                                                                              |
| `tags` (items)   | `enabled_stats` replaces inherited set                                             | `value_count`, `approx_count_distinct`, `histogram(max_bins=200, overflow=none)`                |

<Note>
  The `address.state` field gets no stats because it is not listed in the `properties` array. Nested property configuration is opt-in — unlisted children are skipped entirely.
</Note>

## Validation errors

When a configuration violates a constraint, the API returns a `400 Bad Request` with a descriptive error message. Multiple validation errors may be reported in a single response, separated by semicolons.

Each error is documented with its exact trigger condition, an example request that produces it, and how to fix it:

<CardGroup>
  <Card title="Missing Required Default Statistics" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/defaults-missing-enabled-stats">
    defaults must include enabled\_stats
  </Card>

  <Card title="Duplicate Statistics" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/duplicate-stats">
    Same stat listed more than once
  </Card>

  <Card title="Conflicting Distinct Count Methods" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/distinct-conflict">
    approx\_count\_distinct + count\_distinct together
  </Card>

  <Card title="Invalid Histogram Bin Count" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/invalid-max-bins">
    max\_bins outside \[2, 100000]
  </Card>

  <Card title="Histogram Options Without Histogram Enabled" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/histogram-options-without-stat">
    Histogram options without histogram stat
  </Card>

  <Card title="Too Many Histograms" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/too-many-histograms">
    More than 64 histogram-enabled primitive leaves
  </Card>

  <Card title="Mixed Primitive and Non-Primitive Configuration" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/mixed-node-config">
    Primitive + non-primitive fields on same node
  </Card>

  <Card title="Empty Node Configuration" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/empty-node-config">
    Node with no configuration fields
  </Card>

  <Card title="Empty Namespace Configuration" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/empty-namespace">
    Namespace without scope or fields
  </Card>

  <Card title="Empty Scope Configuration" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/empty-scope-config">
    Scope without enabled\_stats or stat\_options
  </Card>

  <Card title="Empty Properties List" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/empty-properties">
    Empty properties array
  </Card>

  <Card title="Duplicate Field Names" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/duplicate-field-names">
    Same field\_name appears more than once
  </Card>

  <Card title="Duplicate Property Paths" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/duplicate-property-paths">
    Same property path appears more than once
  </Card>

  <Card title="Conflicting Object and Array Configuration" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/properties-and-items-conflict">
    Both properties and items on same node
  </Card>

  <Card title="Unsupported Statistics for Container Node" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/unsupported-self-stats">
    Stats other than null\_value\_count in self
  </Card>

  <Card title="Invalid Cron Expression" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/invalid-cron-expression">
    Malformed cron expression
  </Card>

  <Card title="Type Validation Error" icon="circle-exclamation" href="/reference/architecture/dataset-statistics/errors/type-validation-error">
    Stats incompatible with the dataset schema types
  </Card>
</CardGroup>

## Related content

<CardGroup>
  <Card title="Dataset Statistics" icon="lightbulb" href="/concepts/primitives/dataset-statistics">
    Why statistics matter and how they work
  </Card>

  <Card title="Configuring Dataset Statistics" icon="gear" href="/guides/platform/configuring-dataset-statistics">
    Set up statistics through the platform UI
  </Card>

  <Card title="Job Types" icon="list-check" href="/reference/architecture/job-types">
    Job types including statistics computation jobs
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor job status and handle completion
  </Card>
</CardGroup>


# Missing Required Default Statistics
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/defaults-missing-enabled-stats

The defaults object must include enabled_stats because there is no parent scope to inherit from

## Error message

```
defaults: enabled_stats is required (no parent scope to inherit from)
```

## When this error occurs

The `defaults` object is the root of the inheritance chain. Every other level (namespace scope, field overrides, nested nodes) can optionally omit `enabled_stats` and inherit from its parent. But `defaults` has no parent — so `enabled_stats` must be explicitly provided.

This error is raised when:

* `defaults` contains only `stat_options` without `enabled_stats`
* `defaults` is an empty object

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

The server cannot determine which stats to compute because `enabled_stats` is missing from `defaults`.

## How to fix

Add `enabled_stats` to the `defaults` object. Even if you only want to configure histogram options, you must declare the stat set:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"],
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

To disable all stats globally, use an empty array:

```json theme={null}
{
  "defaults": {
    "enabled_stats": []
  },
  "refresh": { "trigger": "manual" }
}
```

## Related errors

* [EmptyScopeConfig](/reference/architecture/dataset-statistics/errors/empty-scope-config) — similar constraint at the namespace scope level


# Conflicting Distinct Count Methods
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/distinct-conflict

approx_count_distinct and count_distinct cannot both be enabled in the same stat set

## Error message

```
{context}: approx_count_distinct and count_distinct cannot both be enabled
```

Example: `defaults: approx_count_distinct and count_distinct cannot both be enabled`

## When this error occurs

`approx_count_distinct` uses HyperLogLog for fast approximate cardinality. `count_distinct` computes the exact value. Enabling both on the same scope is redundant and indicates a configuration mistake, so the server rejects it.

This validation applies at every level: defaults, namespace scope, field overrides, and nested nodes.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": [
      "value_count",
      "approx_count_distinct",
      "count_distinct"
    ]
  },
  "refresh": { "trigger": "manual" }
}
```

## How to fix

Choose one of the two:

* **`approx_count_distinct`** — faster, suitable for large datasets where an estimate is acceptable
* **`count_distinct`** — exact, more expensive to compute

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "approx_count_distinct"]
  },
  "refresh": { "trigger": "manual" }
}
```


# Duplicate Field Names
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/duplicate-field-names

The same field name appears more than once in a namespace fields array

## Error message

```
{context}: duplicate field names: {names}
```

Example: `dataset.fields: duplicate field names: user_id`

## When this error occurs

Each entry in the `fields` array of a `dataset` or `rosetta_stone` namespace must have a unique identifier (`field_name` or `attribute_name`). This error is raised when two or more entries share the same name.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "user_id",
        "enabled_stats": ["value_count", "count_distinct"]
      },
      {
        "field_name": "user_id",
        "enabled_stats": ["value_count"]
      }
    ]
  }
}
```

`user_id` appears twice in `dataset.fields`.

## How to fix

Remove the duplicate entry. If you intended different configurations, merge them into a single entry:

```json theme={null}
"dataset": {
  "fields": [
    {
      "field_name": "user_id",
      "enabled_stats": ["value_count", "count_distinct"]
    }
  ]
}
```

## Related errors

* [DuplicatePropertyPaths](/reference/architecture/dataset-statistics/errors/duplicate-property-paths) — same issue for nested property paths


# Duplicate Property Paths
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/duplicate-property-paths

The same property path appears more than once in a properties array

## Error message

```
{context}: duplicate property paths: {paths}
```

Example: `dataset.fields[address].properties: duplicate property paths: city`

## When this error occurs

Each entry in a `properties` array must have a unique `path`. This error is raised when two or more entries share the same path within the same parent object node.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "properties": [
          {
            "path": "city",
            "enabled_stats": ["value_count"]
          },
          {
            "path": "city",
            "enabled_stats": ["approx_count_distinct"]
          }
        ]
      }
    ]
  }
}
```

`city` appears twice in the `address` properties array.

## How to fix

Merge the duplicate entries into one:

```json theme={null}
{
  "field_name": "address",
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count", "approx_count_distinct"]
    }
  ]
}
```

## Related errors

* [DuplicateFieldNames](/reference/architecture/dataset-statistics/errors/duplicate-field-names) — same issue for top-level field names


# Duplicate Statistics
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/duplicate-stats

The same stat name appears more than once in an enabled_stats list

## Error message

```
{context}: duplicate stats: {stat_names}
```

Example: `defaults: duplicate stats: value_count`

## When this error occurs

Each `enabled_stats` array must contain unique entries. This error is raised when the same stat name appears two or more times at any level — defaults, namespace scope, field override, or nested node.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "value_count"]
  },
  "refresh": { "trigger": "manual" }
}
```

`value_count` appears twice in the defaults stat list.

## How to fix

Remove the duplicate entries:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count"]
  },
  "refresh": { "trigger": "manual" }
}
```

The `{context}` prefix in the error message tells you exactly where the duplicate was found (e.g., `defaults`, `dataset.scope`, `dataset.fields[price]`).


# Empty Namespace Configuration
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/empty-namespace

A namespace must have at least one of scope or fields

## Error message

```
{context}: at least one of scope or fields must be provided
```

Example: `dataset: at least one of scope or fields must be provided`

## When this error occurs

The `dataset` and `rosetta_stone` namespace objects are optional at the top level. But if you include one, it must contain at least `scope`, `fields`, or both. An empty namespace object has no effect and indicates a configuration mistake.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {}
}
```

The `dataset` namespace is present but has neither `scope` nor `fields`.

## How to fix

Either add content to the namespace or remove it entirely:

**Option A: Add a scope**

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "scope": {
      "enabled_stats": ["value_count", "null_value_count"]
    }
  }
}
```

**Option B: Add fields**

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "user_id",
        "enabled_stats": ["value_count", "count_distinct"]
      }
    ]
  }
}
```

**Option C: Remove the namespace**

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" }
}
```

If you want to disable stats for a namespace, use an empty `enabled_stats` array in the scope:

```json theme={null}
"dataset": {
  "scope": {
    "enabled_stats": []
  }
}
```


# Empty Node Configuration
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/empty-node-config

A nested node or property must have at least one configuration field

## Error message

```
{context}: at least one configuration field must be provided
```

## When this error occurs

This error applies to **nested nodes** (`items`) and **nested properties** (entries in a `properties` array). Unlike top-level fields, which can be empty to indicate inheritance, nested nodes must explicitly declare their configuration.

A nested node or property with none of `enabled_stats`, `stat_options`, `self`, `properties`, or `items` is invalid.

<Note>
  This error does **not** apply to top-level field entries (e.g., `dataset.fields[user_id]`). A top-level field with only a `field_name` and no other fields is valid — it means "inherit from the namespace scope or defaults."
</Note>

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "properties": [
          {
            "path": "city"
          }
        ]
      }
    ]
  }
}
```

The property `city` has only a `path` — no stat configuration.

## How to fix

Add at least one configuration field to the nested node or property:

```json theme={null}
{
  "field_name": "address",
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count", "null_value_count"]
    }
  ]
}
```

If you don't want to configure stats for a specific property, remove it from the `properties` array entirely. Unlisted properties receive no stats.

## Related errors

* [MixedNodeConfig](/reference/architecture/dataset-statistics/errors/mixed-node-config) — has fields from both kinds
* [EmptyProperties](/reference/architecture/dataset-statistics/errors/empty-properties) — the entire properties array is empty


# Empty Properties List
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/empty-properties

The properties array must not be empty — if present, it must contain at least one entry

## Error message

```
{context}: properties list must not be empty
```

## When this error occurs

The `properties` array configures stats for named children of an object node. If you include `properties`, it must have at least one entry. An empty array is ambiguous — it could mean "no stats for any children" (in which case you should remove `properties` entirely) or be an accidental omission.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "properties": []
      }
    ]
  }
}
```

## How to fix

Either add property entries or remove the `properties` field:

**Option A: Add property configurations**

```json theme={null}
{
  "field_name": "address",
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count", "approx_count_distinct"]
    }
  ]
}
```

**Option B: Use `self` only (no child stats)**

```json theme={null}
{
  "field_name": "address",
  "self": {
    "enabled_stats": ["null_value_count"]
  }
}
```

**Option C: Inherit from defaults (no-op field entry)**

```json theme={null}
{
  "field_name": "address"
}
```

## Related errors

* [EmptyNodeConfig](/reference/architecture/dataset-statistics/errors/empty-node-config) — a node with no configuration fields at all


# Empty Scope Configuration
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/empty-scope-config

A scope must have at least one of enabled_stats or meaningful stat_options

## Error message

```
{context}: scope must have at least one of enabled_stats or stat_options
```

## When this error occurs

The `scope` field within a namespace (`dataset.scope` or `rosetta_stone.scope`) must contain at least one of:

* `enabled_stats` — an explicit stat set
* `stat_options` with at least one actual option (e.g., `histogram` with tuning parameters)

This error is raised when scope is present but effectively empty — for example, `stat_options: {}` with no `enabled_stats`, or `stat_options: { "histogram": null }`.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "scope": {
      "stat_options": {}
    }
  }
}
```

The scope has `stat_options` but no actual options inside it.

## How to fix

Add meaningful content to the scope:

```json theme={null}
"dataset": {
  "scope": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"]
  }
}
```

Or provide actual stat options:

```json theme={null}
"dataset": {
  "scope": {
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  }
}
```

If you don't need a scope override, remove the `scope` field and use `fields` only, or remove the namespace entirely.

## Related errors

* [DefaultsMissingEnabledStats](/reference/architecture/dataset-statistics/errors/defaults-missing-enabled-stats) — similar constraint at the defaults level
* [EmptyNamespace](/reference/architecture/dataset-statistics/errors/empty-namespace) — the namespace itself is empty


# Histogram Options Without Histogram Enabled
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/histogram-options-without-stat

Histogram options were provided but the histogram stat is not in the resolved stat set

## Error message

```
{context}: histogram options provided but histogram stat is not enabled; enable the stat or remove the options
```

## When this error occurs

`stat_options.histogram` tunes the behavior of the `histogram` stat. If `histogram` is not present in the **resolved** stat set for that level, the options have nothing to apply to and the server rejects the configuration.

This can happen in three scenarios:

### Scenario 1: `enabled_stats` is present but does not include `histogram`

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count"],
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

The stat set is `[value_count, null_value_count]` — no `histogram`, so the options are invalid.

### Scenario 2: `stat_options`-only at a level where the parent has no `histogram`

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "mean"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "scope": {
      "stat_options": {
        "histogram": { "max_bins": 200 }
      }
    }
  }
}
```

The scope inherits the stat set from defaults `[value_count, mean]` — no `histogram` to tune.

### Scenario 3: `stat_options`-only at scope when defaults has an empty stat set

```json theme={null}
{
  "defaults": {
    "enabled_stats": []
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "scope": {
      "stat_options": {
        "histogram": { "max_bins": 200 }
      }
    }
  }
}
```

Defaults disable all stats (empty array). The scope inherits an empty set — no `histogram` to tune.

## How to fix

Either add `histogram` to `enabled_stats`, or remove the histogram options:

**Option A: Enable the stat**

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"],
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

**Option B: Remove the options**

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count"]
  },
  "refresh": { "trigger": "manual" }
}
```

## Related errors

* [InvalidMaxBins](/reference/architecture/dataset-statistics/errors/invalid-max-bins) — max\_bins value out of range


# Invalid Cron Expression
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/invalid-cron-expression

The cron_expression value is not a valid cron expression

## Error message

```
refresh: Invalid cron expression '{expression}': {detail}
```

## When this error occurs

When `refresh.trigger` is `"cron"`, the `cron_expression` field must contain a valid 5-field cron expression. This error is raised when the expression cannot be parsed.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": {
    "trigger": "cron",
    "cron_expression": "not a cron"
  }
}
```

## Cron expression format

```
┌───────────── minute (0–59)
│ ┌───────────── hour (0–23)
│ │ ┌───────────── day of month (1–31)
│ │ │ ┌───────────── month (1–12)
│ │ │ │ ┌───────────── day of week (0–7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *
```

Supported syntax: `*`, `,`, `-`, `/`, `?`, `L`, `W`, `LW`, `#`, and nicknames (`@daily`, `@hourly`, etc.). All times are UTC.

## How to fix

Provide a valid cron expression:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": {
    "trigger": "cron",
    "cron_expression": "0 0 * * *"
  }
}
```

Common examples:

| Expression     | Meaning                              |
| -------------- | ------------------------------------ |
| `0 0 * * *`    | Every day at midnight UTC            |
| `0 3 * * 1`    | Every Monday at 03:00 UTC            |
| `*/15 * * * *` | Every 15 minutes                     |
| `0 0 1 * *`    | First day of every month at midnight |
| `0 6 * * 1-5`  | Weekdays at 06:00 UTC                |

If you don't need scheduled refresh, consider `"trigger": "on_update"` or `"trigger": "manual"` instead.


# Invalid Histogram Bin Count
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/invalid-max-bins

The max_bins value for histogram options is outside the allowed range [2, 100000]

## Error message

```
{context}: max_bins must be between 2 and 100000, got {value}
```

Example: `defaults: max_bins must be between 2 and 100000, got 1`

## When this error occurs

The `max_bins` parameter in histogram options controls how many bins the frequency distribution uses. It must be an integer between 2 and 100,000 (inclusive). This error is raised when the value falls outside that range.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["histogram"],
    "stat_options": {
      "histogram": { "max_bins": 1 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

A histogram with fewer than 2 bins is meaningless.

## How to fix

Set `max_bins` to a value between 2 and 100,000. If omitted, the server defaults to 50:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["histogram"],
    "stat_options": {
      "histogram": { "max_bins": 200 }
    }
  },
  "refresh": { "trigger": "manual" }
}
```

Common values:

* **50** (default) — good for most columns
* **200–1000** — useful for high-cardinality categorical columns
* **100,000** — maximum, for very granular distributions


# Mixed Primitive and Non-Primitive Configuration
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/mixed-node-config

A node cannot mix primitive fields (enabled_stats/stat_options) with non-primitive fields (self/properties/items)

## Error message

```
{context}: cannot mix primitive fields (enabled_stats/stat_options) with non-primitive fields (self/properties/items)
```

## When this error occurs

Each field or nested node is either **primitive** or **non-primitive**, and the configuration fields must match:

| Node kind                                | Allowed fields                  |
| ---------------------------------------- | ------------------------------- |
| Primitive (string, integer, float, etc.) | `enabled_stats`, `stat_options` |
| Non-primitive (object, array)            | `self`, `properties`, `items`   |

This error is raised when both sets of fields appear on the same node. The design is intentional: primitive and non-primitive nodes have fundamentally different stat semantics, and mixing them creates ambiguous configurations.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "enabled_stats": ["value_count"],
        "properties": [
          {
            "path": "city",
            "enabled_stats": ["value_count"]
          }
        ]
      }
    ]
  }
}
```

The `address` field has both `enabled_stats` (primitive) and `properties` (non-primitive).

## How to fix

Choose the appropriate config shape for the field's actual schema type:

**If the field is an object (non-primitive):**

```json theme={null}
{
  "field_name": "address",
  "self": {
    "enabled_stats": ["null_value_count"]
  },
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

Use `self` for stats on the container column and `properties` for named children.

**If the field is a primitive (string, integer, etc.):**

```json theme={null}
{
  "field_name": "address",
  "enabled_stats": ["value_count"]
}
```

Remove `properties`, `items`, and `self`.

## Related errors

* [EmptyNodeConfig](/reference/architecture/dataset-statistics/errors/empty-node-config) — node with no configuration at all
* [UnsupportedSelfStats](/reference/architecture/dataset-statistics/errors/unsupported-self-stats) — only `null_value_count` allowed in `self`


# Conflicting Object and Array Configuration
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/properties-and-items-conflict

A non-primitive node cannot have both properties and items — it is either an object or an array

## Error message

```
{context}: properties and items cannot both be specified
```

## When this error occurs

Non-primitive nodes model either **objects** (with named children via `properties`) or **arrays** (with element config via `items`). A node cannot be both. This error is raised when both `properties` and `items` appear on the same node.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "metadata",
        "properties": [
          {
            "path": "key",
            "enabled_stats": ["value_count"]
          }
        ],
        "items": {
          "enabled_stats": ["value_count"]
        }
      }
    ]
  }
}
```

The `metadata` field has both `properties` (object) and `items` (array).

## How to fix

Use only the one that matches the field's actual schema type:

**If the field is an object:**

```json theme={null}
{
  "field_name": "metadata",
  "properties": [
    {
      "path": "key",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

**If the field is an array:**

```json theme={null}
{
  "field_name": "metadata",
  "items": {
    "enabled_stats": ["value_count"]
  }
}
```

## Related errors

* [MixedNodeConfig](/reference/architecture/dataset-statistics/errors/mixed-node-config) — mixing primitive and non-primitive fields


# Too Many Histograms
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/too-many-histograms

The configuration enables histogram on more primitive leaves than the platform allows

## Error message

```
Configuration contains {actual} histogram leaves, which exceeds the maximum of {max}
```

Example: `Configuration contains 78 histogram leaves, which exceeds the maximum of 64`

## When this error occurs

The platform caps the total number of histogram-enabled primitive columns at **64** per configuration. This limit applies across **both** the dataset and Rosetta Stone namespaces combined.

Every primitive leaf (string, integer, float, boolean, timestamp) that resolves to having the `histogram` stat enabled counts toward this limit. Object and array columns themselves do not count — only their primitive children do.

This error is raised before saving the configuration, so no resources are wasted on an overly broad histogram setup.

### Common causes

1. **Broad defaults with `histogram` enabled** — if defaults include `histogram` and the dataset has many columns, every compatible column gets a histogram automatically.
2. **Large nested objects** — an object with many primitive properties, each inheriting `histogram` from defaults or scope, can quickly exhaust the limit.
3. **Both namespaces enabled** — histograms on dataset columns plus Rosetta Stone attributes add up.

## Example: request that triggers this error

Given a dataset with 50 primitive columns and 20 Rosetta Stone attributes (all primitive):

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"]
  },
  "refresh": { "trigger": "manual" }
}
```

This enables histogram on all 70 primitive leaves (50 dataset + 20 Rosetta Stone), exceeding the limit of 64.

## How to fix

Reduce the number of histogram-enabled leaves below 64. Several strategies:

### Option A: Remove histogram from defaults, enable per-field

Move histogram out of defaults and enable it only on the fields that matter:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "price",
        "enabled_stats": ["value_count", "null_value_count", "histogram"]
      },
      {
        "field_name": "age",
        "enabled_stats": ["value_count", "null_value_count", "histogram"]
      }
    ]
  }
}
```

### Option B: Keep histogram in defaults, disable for one namespace

If you want histograms on dataset columns but not Rosetta Stone attributes:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"]
  },
  "refresh": { "trigger": "manual" },
  "rosetta_stone": {
    "scope": {
      "enabled_stats": ["value_count", "null_value_count"]
    }
  }
}
```

The Rosetta Stone scope replaces the inherited stat set with one that excludes `histogram`.

### Option C: Disable histogram on specific high-cardinality fields

Override individual fields to drop histogram while keeping defaults broad:

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count", "null_value_count", "histogram"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "raw_payload",
        "enabled_stats": ["value_count", "null_value_count"]
      },
      {
        "field_name": "metadata",
        "self": {
          "enabled_stats": ["null_value_count"]
        }
      }
    ]
  }
}
```

Fields listed with `enabled_stats` that exclude `histogram` no longer count toward the limit. Unlisted fields still inherit from defaults.

## Counting rules

* Only **primitive** types count: `StringType`, `LongType`, `DoubleType`, `BooleanType`, `TimestampTzType`
* **Array items** are excluded from the count (they are filtered at execution time)
* **Object** and **array** columns themselves do not count — only their primitive children
* The count spans both the `dataset` and `rosetta_stone` namespaces

## Related errors

* [Histogram Options Without Histogram Enabled](/reference/architecture/dataset-statistics/errors/histogram-options-without-stat) — histogram options present but stat not enabled
* [Invalid Histogram Bin Count](/reference/architecture/dataset-statistics/errors/invalid-max-bins) — max\_bins value out of range


# Type Validation Error
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/type-validation-error

The statistics configuration is incompatible with the dataset schema or Rosetta Stone attribute types

## Error message

```
{path}: {detail}; {path}: {detail}; ...
```

Examples:

* `price: mean is not compatible with type string`
* `address: primitive config does not match schema type object`
* `events.payload: property 'user_agent' not found in schema`

## When this error occurs

After the configuration passes structural validation (no duplicate stats, no mixed node types, etc.), the server validates it against the **actual schema** of the dataset or Rosetta Stone attributes. This error is raised when the configured statistics are incompatible with the field types in the schema.

There are four categories of type validation errors:

### 1. Stat incompatible with field type

A statistic is requested on a column whose type does not support it.

**Type compatibility rules:**

| Stat                                              | Compatible types                                           |
| ------------------------------------------------- | ---------------------------------------------------------- |
| `value_count`, `null_value_count`, `completeness` | All types                                                  |
| `approx_count_distinct`, `count_distinct`         | All types                                                  |
| `histogram`                                       | All types                                                  |
| `mean`, `standard_deviation`                      | Numeric only (`long`, `double`)                            |
| `nan_value_count`                                 | Numeric only (`long`, `double`)                            |
| `lower_bound`, `upper_bound`                      | Ordered types (`long`, `double`, `timestamp_tz`, `string`) |

**Example:** requesting `mean` on a string column.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "country",
        "enabled_stats": ["value_count", "mean"]
      }
    ]
  }
}
```

If `country` is a `string` column in the dataset schema, the server returns:

```
country: mean is not compatible with type string
```

**Fix:** Remove `mean` and use stats compatible with strings:

```json theme={null}
{
  "field_name": "country",
  "enabled_stats": ["value_count", "approx_count_distinct"]
}
```

### 2. Shape mismatch between config and schema

The configuration structure does not match the field's schema type. For example, using primitive config fields (`enabled_stats`) on an object column, or using non-primitive config fields (`properties`) on a string column.

**Example:** treating an object column as a primitive.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "enabled_stats": ["value_count", "histogram"]
      }
    ]
  }
}
```

If `address` is an `object` column (with children `city`, `zip`, `state`), the server returns:

```
address: primitive config does not match schema type object
```

**Fix:** Use non-primitive config for object columns:

```json theme={null}
{
  "field_name": "address",
  "self": {
    "enabled_stats": ["null_value_count"]
  },
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count", "histogram"]
    }
  ]
}
```

The reverse also applies — using object config on a primitive column:

```json theme={null}
{
  "field_name": "user_id",
  "properties": [
    {
      "path": "name",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

If `user_id` is a `string`, the server returns:

```
user_id: object config does not match schema type string
```

### 3. Unknown property in schema

A property listed in the `properties` array does not exist in the object's schema.

**Example:** configuring a non-existent child property.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "properties": [
          {
            "path": "city",
            "enabled_stats": ["value_count"]
          },
          {
            "path": "country",
            "enabled_stats": ["value_count"]
          }
        ]
      }
    ]
  }
}
```

If `address` has children `city`, `zip`, and `state` but no `country`, the server returns:

```
address.country: property 'country' not found in schema
```

**Fix:** Remove the property or correct the path to match the schema:

```json theme={null}
{
  "field_name": "address",
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count"]
    },
    {
      "path": "state",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

### 4. Nested type incompatibility

For deeply nested structures (arrays of objects, objects containing arrays), the validation recurses into children. Errors report the full path.

**Example:** requesting `mean` on a string property inside an array of objects.

Consider a dataset with column `events` of type `array<object<name: string, score: double>>`.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "events",
        "self": {
          "enabled_stats": ["null_value_count"]
        },
        "items": {
          "properties": [
            {
              "path": "name",
              "enabled_stats": ["value_count", "mean"]
            },
            {
              "path": "score",
              "enabled_stats": ["value_count", "mean"]
            }
          ]
        }
      }
    ]
  }
}
```

The `score` field (double) supports `mean`, but `name` (string) does not. The server returns:

```
events.[].name: mean is not compatible with type string
```

**Fix:** Remove `mean` from the string property:

```json theme={null}
{
  "path": "name",
  "enabled_stats": ["value_count", "approx_count_distinct"]
}
```

### 5. Field not found in dataset schema

A `field_name` in the `dataset.fields` array does not match any column in the dataset.

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "nonexistent_column",
        "enabled_stats": ["value_count"]
      }
    ]
  }
}
```

```
nonexistent_column: field 'nonexistent_column' not found in dataset schema
```

**Fix:** Verify the column name matches the dataset schema exactly. Column names are case-sensitive.

## Multiple errors in one response

The validator reports all errors found, not just the first. Multiple errors are separated by semicolons:

```
price: mean is not compatible with type string; address: primitive config does not match schema type object; events.[].name: nan_value_count is not compatible with type string
```

## Quick reference: common mistakes

| Mistake                                   | Error                                                 | Fix                                                  |
| ----------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------- |
| `mean` on a string column                 | `mean is not compatible with type string`             | Use `approx_count_distinct` or `histogram` instead   |
| `nan_value_count` on a non-numeric column | `nan_value_count is not compatible with type boolean` | Remove — NaN only applies to floats                  |
| `lower_bound` on a boolean column         | `lower_bound is not compatible with type boolean`     | Remove — booleans have no meaningful ordering        |
| Flat stats on an object column            | `primitive config does not match schema type object`  | Use `self` + `properties` instead of `enabled_stats` |
| Object config on a scalar column          | `object config does not match schema type string`     | Use `enabled_stats` directly                         |
| Non-existent property in `properties`     | `property 'x' not found in schema`                    | Check the schema for correct property names          |
| Non-existent field in `fields`            | `field 'x' not found in dataset schema`               | Check the dataset schema for correct column names    |

## Related errors

* [Mixed Primitive and Non-Primitive Configuration](/reference/architecture/dataset-statistics/errors/mixed-node-config) — structural mismatch caught before type validation
* [Unsupported Statistics for Container Node](/reference/architecture/dataset-statistics/errors/unsupported-self-stats) — `self` stats limited to `null_value_count`


# Unsupported Statistics for Container Node
Source: https://docs.narrative.io/reference/architecture/dataset-statistics/errors/unsupported-self-stats

Only null_value_count is supported for non-primitive self configs

## Error message

```
{context}: only null_value_count is supported for non-primitive self configs; unsupported: {stat_names}
```

Example: `dataset.fields[address].self: only null_value_count is supported for non-primitive self configs; unsupported: value_count, histogram`

## When this error occurs

The `self` field on a non-primitive node (object or array) configures stats for the **container column itself** — not its children. For a struct or array column, the only meaningful stat at the container level is `null_value_count` (counting rows where the entire container is NULL).

Stats like `value_count`, `histogram`, `mean`, etc. don't have well-defined semantics on a container value and are rejected.

## Example: request that triggers this error

```json theme={null}
{
  "defaults": {
    "enabled_stats": ["value_count"]
  },
  "refresh": { "trigger": "manual" },
  "dataset": {
    "fields": [
      {
        "field_name": "address",
        "self": {
          "enabled_stats": ["value_count", "null_value_count", "histogram"]
        },
        "properties": [
          {
            "path": "city",
            "enabled_stats": ["value_count"]
          }
        ]
      }
    ]
  }
}
```

`self` includes `value_count` and `histogram`, which are not supported for non-primitive self configs.

## How to fix

Restrict `self.enabled_stats` to only `null_value_count`:

```json theme={null}
{
  "field_name": "address",
  "self": {
    "enabled_stats": ["null_value_count"]
  },
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

If you don't need null counting on the container, remove `self` entirely:

```json theme={null}
{
  "field_name": "address",
  "properties": [
    {
      "path": "city",
      "enabled_stats": ["value_count"]
    }
  ]
}
```

## Related errors

* [MixedNodeConfig](/reference/architecture/dataset-statistics/errors/mixed-node-config) — mixing primitive and non-primitive config fields


# Job Types
Source: https://docs.narrative.io/reference/architecture/job-types

Reference for the different types of jobs coordinated between control plane and data planes

The [job queue](/reference/glossary#job-queue) coordinates work between the [control plane](/concepts/primitives/control-plane) and [data planes](/concepts/primitives/data-planes). This reference describes the different types of jobs that flow through the queue.

## Overview

Jobs are the unit of work exchanged between the control plane and data plane operators. When the control plane needs something executed in a data plane, it creates a job and enqueues it. Data plane [operators](/reference/glossary#operator-data-plane) poll the queue, claim jobs, execute them, and report results.

## Job states

Every job progresses through a lifecycle tracked by its state.

| State                  | Description                                      |
| ---------------------- | ------------------------------------------------ |
| `pending`              | Job is queued and waiting for execution          |
| `running`              | Job is currently being executed                  |
| `completed`            | Job finished successfully                        |
| `pending_cancellation` | Job is marked for cancellation but still running |
| `cancelled`            | Job was cancelled before completion              |
| `failed`               | Job execution failed                             |

## Core job structure

All jobs share common fields that identify and track them through the system.

| Field           | Description                                           |
| --------------- | ----------------------------------------------------- |
| `id`            | Unique identifier for the job                         |
| `type`          | Job type string that determines routing and execution |
| `state`         | Current lifecycle state                               |
| `company_id`    | Company that owns the job                             |
| `data_plane_id` | Target data plane for execution (if specified)        |
| `created_at`    | When the job was created                              |
| `updated_at`    | When the job was last modified                        |
| `completed_at`  | When the job finished (if applicable)                 |
| `failures`      | List of failure records if the job encountered errors |

***

## Forecasting jobs

Jobs that estimate data volumes and costs before committing to operations.

### forecast

Estimates data volumes and costs for a marketplace subscription before purchase.

This job runs when you preview a subscription in the marketplace or use the forecasting API. It samples data to quickly estimate costs without scanning the full dataset.

**Executor:** Spark Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "as_of": "2024-01-01T00:00:00Z",
    "details": {
      "type": "marketplace",
      "company_constraint": {
        "type": "inclusion",
        "company_ids": [56]
      },
      "data_rules": {
        "attributes": [
          {
            "attribute_id": 1,
            "fields": [
              { "field": "unique_id.type", "filter": null, "exported": true },
              { "field": "unique_id.value", "filter": "include_only_if_not_null_filter", "exported": true }
            ],
            "optional": false
          }
        ],
        "ingestion_timestamp_filter": {
          "recency": "P30D",
          "from": { "type": "inclusive", "value": "2024-01-01T00:00:00Z" }
        }
      },
      "pricing": { "micro_cents_usd": 1000000000 }
    },
    "dimensions": { "distinct_counts": [], "group_by": null },
    "sample_rate_denominator": 128,
    "schedule_unsampled_forecast": false
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "success",
    "cost": 5000000000,
    "rows": 1500000,
    "datasets": [
      { "dataset_id": 424, "rows": 1500000, "cost": 5000000000 }
    ],
    "estimator": { "type": "sample", "rate": 0.0078125 },
    "scan_cost": {
      "rows": 190001186647,
      "size": 6286551251334,
      "sources": [
        { "rows": 190001186647, "size": 6286551251334, "source": { "type": "dataset", "dataset_id": 424 } }
      ]
    }
  }
  ```
</Accordion>

### forecast-internal

<Note>
  This is an internal job type used by the system. You may see it in job listings but cannot create it directly.
</Note>

Internal variant of the forecast job used for unsampled follow-up forecasts and accuracy comparisons.

This job runs automatically after a sampled `forecast` job completes when the system needs precise cost estimates. It uses the same input and output format as the `forecast` job.

**Executor:** Spark Data Plane

### costs

Calculates scan costs for a subscription query without performing a full forecast.

This job provides a quick cost estimate by analyzing the query plan without sampling data. It's faster than a full forecast but provides less detail.

**Executor:** Spark Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "as_of": null,
    "details": {
      "type": "marketplace",
      "company_constraint": {
        "type": "inclusion",
        "company_ids": [56]
      },
      "data_rules": {
        "attributes": [
          {
            "attribute_id": 1,
            "fields": [
              { "field": "unique_id.type", "filter": null, "exported": true },
              { "field": "unique_id.value", "filter": "include_only_if_not_null_filter", "exported": true }
            ]
          }
        ]
      },
      "pricing": { "micro_cents_usd": 1000000000 }
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "success",
    "cost": {
      "rows": 190001186647,
      "size": 6286551251334,
      "sources": [
        { "rows": 0, "size": 0, "source": { "type": "dataset", "dataset_id": 340 } },
        { "rows": 190001186647, "size": 6286551251334, "source": { "type": "dataset", "dataset_id": 424 } }
      ]
    }
  }
  ```
</Accordion>

### nql-forecast

Executes an NQL forecast query to estimate results without materializing data.

This job runs when you preview an NQL query to see estimated row counts and structure before creating a materialized view.

**Executor:** Spark Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "nql": "SELECT COUNT(*) FROM narrative.rosetta WHERE country = 'US'",
    "compiled_sql": "SELECT COUNT(*) FROM rosetta_table WHERE country = 'US'"
  }
  ```
</Accordion>

<Accordion title="Output example (success)">
  ```json theme={null}
  {
    "type": "Success",
    "result": {
      "columns": ["count"],
      "rows": [[1500000]]
    }
  }
  ```
</Accordion>

<Accordion title="Output example (failure)">
  ```json theme={null}
  {
    "type": "Failure",
    "msg": "Query execution timed out after 300 seconds"
  }
  ```
</Accordion>

***

## Materialization jobs

Jobs that execute queries and persist results to datasets.

### materialize-view

Executes NQL to refresh a materialized view dataset.

This is the core job type for dataset refreshes. It runs the compiled query against source data and writes results to the target dataset as an Iceberg table snapshot.

**Executor:** Spark Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "nql": "SELECT unique_id FROM narrative.rosetta WHERE timestamp > _nio_start_ts",
    "compiled_select": "SELECT unique_id FROM rosetta_table WHERE timestamp > '2024-01-01'",
    "dataset_id": 1234,
    "stats_enabled": true,
    "contains_delta_syntax": true,
    "first_run": false,
    "merge": false,
    "partitions": [
      { "field": "timestamp", "fn": "day", "args": [] }
    ],
    "chunk_metadata": {
      "batch_id": "550e8400-e29b-41d4-a716-446655440000",
      "chunk_sequence": { "number": 1, "of": 5 }
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "dataset_id": 1234,
    "snapshot_id": 9876543210,
    "recalculation_id": "550e8400-e29b-41d4-a716-446655440001"
  }
  ```
</Accordion>

**Related:** [Materialized Views](/concepts/nql/materialized-views), [Query Processing](/concepts/architecture/query-processing)

***

## Dataset operations

Jobs that manage dataset tables and data within the data plane.

### datasets\_create\_table

Creates a new table in the data plane for a dataset.

This job runs when provisioning a new dataset. It creates the underlying Iceberg table in the company's schema.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234,
    "compiled_sql": "CREATE TABLE company_56.dataset_1234 (id STRING, name STRING, created_at TIMESTAMP_TZ) CLUSTER BY (created_at)"
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_create_table"
  }
  ```
</Accordion>

### datasets\_delete\_table

Deletes a dataset's table from the data plane.

This job runs when deprovisioning a dataset. It drops the table and removes associated Iceberg metadata.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "external_id": "company_56.dataset_1234",
    "is_narrative_managed": true
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_delete_table"
  }
  ```
</Accordion>

### datasets\_truncate\_table

Truncates all data from a dataset's table while preserving the table structure.

This job removes all rows from the dataset but keeps the schema intact for future use.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_truncate_table"
  }
  ```
</Accordion>

### datasets\_sample

Generates a sample of data from a dataset for preview purposes.

This job retrieves a small subset of rows—including any [Rosetta Stone](/concepts/rosetta-stone/how-it-works) normalized attributes—for display in the UI. The sample data is stored separately from the main dataset.

**Executor:** Snowflake Data Plane, AWS Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_sample"
  }
  ```
</Accordion>

**Related:** [Data Flow](/concepts/architecture/data-flow#how-you-view-query-results), [Managing Datasets](/guides/sdk/managing-datasets#sampling-data)

### datasets\_calculate\_column\_stats

Calculates column-level statistics for a dataset.

This job computes statistics like null counts, value ranges, distinct counts, and histograms. Stats are persisted to the dataset metrics store for use in query optimization and data profiling.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234,
    "enabled_column_stats": {
      "user_id": {
        "data_type": "string",
        "enabled_column_stats": {
          "null_value_count": true,
          "value_count": true,
          "lower_bound": true,
          "upper_bound": true,
          "approx_count_distinct": true,
          "completeness": true
        }
      },
      "amount": {
        "data_type": "double",
        "enabled_column_stats": {
          "nan_value_count": true,
          "null_value_count": true,
          "value_count": true,
          "lower_bound": true,
          "upper_bound": true,
          "histogram": true,
          "mean": true,
          "standard_deviation": true,
          "completeness": true
        }
      }
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_calculate_column_stats"
  }
  ```
</Accordion>

### datasets\_execute\_dml

Executes a DML statement against a dataset.

This job handles [INSERT](/nql/commands/insert), [UPDATE](/nql/commands/update), [DELETE](/nql/commands/delete), and [MERGE](/guides/nql/incremental-upserts) operations on dataset tables.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234,
    "nql": "INSERT INTO my_dataset SELECT * FROM source_table WHERE status = 'active'",
    "compiled_sql": "INSERT INTO company_56.dataset_1234 SELECT * FROM company_56.source_table WHERE status = 'active'",
    "nio_last_modified_at": "2024-01-15T10:30:00Z"
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "datasets_execute_dml",
    "affected_rows": 15000
  }
  ```
</Accordion>

***

## Model operations

Jobs that run inference and train machine learning models.

### model\_inference\_run

Runs inference using a large language model (LLM).

This job sends prompts to models from Anthropic (Claude) or OpenAI and returns structured output. All inference runs within the data plane—no data is sent to external providers.

**Executor:** AWS Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "model": "anthropic.claude-sonnet-4.5",
    "messages": [
      {
        "role": "system",
        "content": [
          { "type": "text", "text": "You are a helpful assistant that extracts structured data." }
        ]
      },
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Extract the company name and revenue from: Acme Corp reported $1.5M in Q4 revenue." }
        ]
      }
    ],
    "inference_config": {
      "output_format_schema": {
        "type": "object",
        "properties": {
          "company_name": { "type": "string" },
          "revenue": { "type": "number" }
        },
        "required": ["company_name", "revenue"]
      },
      "max_tokens": 1000,
      "temperature": 0.1
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "model_inference_run",
    "structured_output": {
      "company_name": "Acme Corp",
      "revenue": 1500000
    },
    "usage": {
      "completion_tokens": 25,
      "prompt_tokens": 150,
      "total_tokens": 175
    }
  }
  ```
</Accordion>

**Related:** [Model Inference Overview](/concepts/model-inference/overview), [Running Model Inference](/guides/sdk/running-model-inference), [Supported Models](/reference/model-inference/supported-models)

### model\_training\_run

Executes a model fine-tuning job.

This job trains a custom model using a base model and your training dataset. The fine-tuned model is stored in the Narrative model repository.

**Executor:** AWS Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "company_id": 56,
    "dataset_id": 1234,
    "instance_type": "ml.g5.2xlarge",
    "base_model_id": "meta-llama/Llama-2-7b-hf",
    "base_model_repository": "huggingface",
    "trained_model_version": "1.0.0",
    "trained_model": {
      "name": "Custom Llama for Classification",
      "collaborators": {
        "owner_company_id": 56,
        "viewer_company_ids": []
      },
      "description": "Fine-tuned Llama model for product classification",
      "tags": ["classification", "products"]
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "model_training_run",
    "result": {
      "model_id": "company-56-custom-llama-v1",
      "status": "completed",
      "metrics": {
        "training_loss": 0.15,
        "validation_loss": 0.18
      }
    }
  }
  ```
</Accordion>

### models\_train\_classifier

Trains a classifier model using Snowflake's built-in ML capabilities.

This job uses Snowflake-native training to build classification models directly in the data plane.

**Executor:** Snowflake Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "config": {
      "model_type": "classification",
      "target_column": "category",
      "feature_columns": ["description", "price", "brand"],
      "training_table": "company_56.training_data",
      "output_model_name": "product_classifier_v1"
    },
    "data_plane_id": "dp-snowflake-123",
    "tags": ["classification"]
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "models_train_classifier",
    "result": {
      "model_name": "product_classifier_v1",
      "accuracy": 0.92,
      "f1_score": 0.89,
      "feature_importance": {
        "description": 0.45,
        "price": 0.35,
        "brand": 0.20
      }
    }
  }
  ```
</Accordion>

### models\_deliver\_model

Delivers a trained model to an external destination.

This job exports a model from the Narrative repository to destinations like HuggingFace Hub.

**Executor:** AWS Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "company_id": 56,
    "app_id": 789,
    "connection_id": "conn-123",
    "profile_id": "550e8400-e29b-41d4-a716-446655440000",
    "model_id": "company-56-custom-llama",
    "model_name": "Custom Llama Classifier",
    "model_version": "1.0.0",
    "quick_settings": {
      "repository_visibility": "private"
    }
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "models_deliver_model",
    "result": {
      "destination_url": "https://huggingface.co/company-56/custom-llama-classifier",
      "status": "completed",
      "model_files_uploaded": 12,
      "total_size_bytes": 14000000000
    }
  }
  ```
</Accordion>

***

## System jobs

Administrative jobs that support platform operations.

### health\_check

Validates the operational health of a [data plane](/concepts/primitives/data-planes) and its [compute pools](/concepts/primitives/compute-pools). You can trigger a health check through the **Run health check** action on the **My Data Planes** list or **Jobs** page in the Dashboard, or through the `POST /data-planes/{data_plane_id}/health-check` API endpoint. You can optionally target a specific compute pool via the API.

**What it checks depends on the data plane type:**

* **Snowflake data planes:** Verifies that each registered warehouse (compute pool) still exists and is accessible. If a warehouse reference is no longer valid, the corresponding compute pool is automatically archived. When an archived pool was the default, the system elects another active pool as the new default.
* **AWS data planes:** Validates cross-account IAM trust relationships using STS AssumeRole and verifies effective permissions by simulating attached IAM policies for each operator subsystem.

**Executor:** Data plane operator (Snowflake Native App or AWS operator)

<Accordion title="Request body (optional)">
  ```json theme={null}
  {
    "compute_pool_id": "550e8400-e29b-41d4-a716-446655440000"
  }
  ```

  If `compute_pool_id` is omitted, the data plane's default compute pool is used.
</Accordion>

<Accordion title="Result example (Snowflake)">
  ```json theme={null}
  {
    "compute_pools": [
      {
        "status": "ok",
        "narrative_compute_pool_id": "550e8400-e29b-41d4-a716-446655440000",
        "warehouse_info": {
          "warehouse_name": "NARRATIVE_WH",
          "size": "X-Small",
          "warehouse_type": "STANDARD",
          "auto_suspend": 300,
          "auto_resume": true
        }
      }
    ]
  }
  ```
</Accordion>

<Accordion title="Result example (AWS)">
  ```json theme={null}
  {
    "status": "healthy",
    "checked_at": "2026-04-06T12:00:00Z",
    "operator_version": "22.25.0",
    "iam_roles": [
      {
        "module": "model-inference",
        "role_arn": "arn:aws:iam::123456789012:role/narrative-model-inference",
        "status": {
          "type": "ok"
        }
      }
    ]
  }
  ```
</Accordion>

### materialized\_views\_collect\_access\_rules\_billing\_data

<Note>
  This is an internal system job. You may see it in job listings but it runs automatically as part of materialized view refreshes.
</Note>

Collects billing data for access rules associated with a materialized view refresh.

This job runs after a `materialize-view` job completes to track data consumption from different access rules and generate billing records.

**Executor:** Spark Data Plane

<Accordion title="Input example">
  ```json theme={null}
  {
    "dataset_id": 1234,
    "nio_last_modified_at": "2024-01-15T10:30:00Z",
    "compiled_query": "SELECT _nio_access_rule_id, COUNT(*) as rows, _nio_price_per_row FROM company_56.dataset_1234 WHERE _nio_last_modified_at = '2024-01-15T10:30:00Z' GROUP BY _nio_access_rule_id, _nio_price_per_row",
    "refresh_job_id": "550e8400-e29b-41d4-a716-446655440000"
  }
  ```
</Accordion>

<Accordion title="Output example">
  ```json theme={null}
  {
    "type": "materialized_views_collect_access_rules_billing_data",
    "sources": [
      {
        "access_rule_id": 456,
        "rows": 50000,
        "price_per_row": { "micro_cents_usd": 100 }
      },
      {
        "access_rule_id": 789,
        "rows": 25000,
        "price_per_row": { "micro_cents_usd": 150 }
      }
    ]
  }
  ```
</Accordion>

## Related content

<CardGroup>
  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    How queries flow through the job queue to execution
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Where jobs are executed by operators
  </Card>

  <Card title="Tracking Job Status" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor jobs and handle completion events
  </Card>

  <Card title="Data Flow" icon="diagram-project" href="/concepts/architecture/data-flow">
    How data moves through the system
  </Card>
</CardGroup>


# Pricing
Source: https://docs.narrative.io/reference/billing/pricing

Service fees, pricing models, and usage calculation for Narrative

Narrative offers two pricing models to fit different usage patterns and commitment levels.

## Pricing models

| Model               | Description                                                                                            |
| ------------------- | ------------------------------------------------------------------------------------------------------ |
| **Pay-as-you-go**   | Use Narrative's Data Collaboration Platform with no up-front costs or minimum commitments              |
| **Committed-Usage** | Lock in discounted usage rates when you commit to a set amount of platform usage over a period of time |

<Note>
  Contact our sales team to discuss usage rate discounts for Committed-Usage plans.
</Note>

***

## Service fees

| Service                      | Pay-as-you-go | Committed-Usage |
| ---------------------------- | ------------- | --------------- |
| **Processing**               | \$0.65/GB     | varies          |
| **Transfer**                 | \$1.50/GB     | varies          |
| **Transaction Services Fee** | 25%           | varies          |
| **Professional Services**    | \$150/hr      | varies          |

***

## Fee details

### Processing

Processing fees are based on the bytes processed per month during your usage of Narrative's products and services, including:

* Ingestion
* Evaluation
* Transactions on the Narrative Marketplace or private transactions
* Forecasting
* Order management

<Note>
  **PAYG waiver for data transactions:** Pay-as-you-go Processing Fees are waived for activity directly supporting data transactions where you are offering data to be licensed or evaluated by a third-party, provided that the transaction (i) occurs or is expected to occur within 90 days of such activity, and (ii) is subject to Transaction Services Fees.
</Note>

### Transfer (Egress)

Transfer fees are calculated based on the volume of bytes transferred out of your Narrative account per month.

<Tip>
  Egress Transfer to client-owned locations within AWS Region "us-east-1" (US East, N. Virginia) does not incur Transfer Fees.
</Tip>

### Transaction Services Fee

The Transaction Services Fee is a percentage of the [Licensed Data Fees](#licensed-data-fees) you transact, source, or facilitate on the platform or via Narrative Services, including via any Connector App.

<Note>
  Narrative neither buys nor sells data. We provide software that powers commercial data transactions between companies. Fees for the licensing of data (Licensed Data Fees) are set by and paid to Data Licensors through Narrative's software.
</Note>

### Professional Services

Optional professional support services are available at an hourly rate.

***

## Usage calculation

Processing and Transfer are calculated in binary gigabytes (GB):

| Unit | Definition                       |
| ---- | -------------------------------- |
| 1 GB | 2^30 bytes (1,073,741,824 bytes) |
| 1 TB | 2^40 bytes (1,024 GB)            |

***

## Glossary

### Licensed Data Fees

Fees paid to Data Licensors for the licensing of data, set by and paid to the licensor through Narrative's software.

***

## Related content

<CardGroup>
  <Card title="Data Flow" icon="arrows-split-up-and-left" href="/concepts/architecture/data-flow">
    Understand how data moves through Narrative's platform
  </Card>

  <Card title="Query Processing" icon="microchip" href="/concepts/architecture/query-processing">
    Learn how queries are processed and executed
  </Card>
</CardGroup>


# Amazon S3 Connector
Source: https://docs.narrative.io/reference/connectors/amazon-s3

Deliver data files to your Amazon S3 bucket

The Amazon S3 Connector delivers data files directly to your S3 bucket, enabling flexible data export for custom integrations and analytics workflows.

## Overview

Unlike advertising platform connectors that format data for specific destinations, the S3 Connector delivers your data as files in your chosen format. This makes it ideal for:

* Custom downstream processing
* Integration with data warehouses
* Backup and archival
* Feeding other systems that consume files from S3

## Requirements

### AWS configuration

| Requirement     | Description                                                |
| --------------- | ---------------------------------------------------------- |
| S3 bucket       | The destination bucket for your data                       |
| IAM credentials | Access key and secret with write permissions to the bucket |
| Bucket policy   | Permissions allowing Narrative to write to the bucket      |

### Supported formats

The connector can deliver data in multiple file formats:

* CSV
* JSON
* Parquet

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **S3 Connector**.
  </Step>

  <Step title="Configure bucket access">
    Enter your S3 bucket name and AWS credentials (access key ID and secret access key).
  </Step>

  <Step title="Set delivery options">
    Configure the file format, compression, and path prefix for delivered files.
  </Step>
</Steps>

## Delivery options

### Sort columns alphabetically

For CSV deliveries, the S3 Connector can either sort columns alphabetically or preserve the delivering dataset's schema column order. Toggle **Sort columns alphabetically** on the dataset connection form to switch between the two:

* **Enabled (default)** — Columns in delivered CSV files are sorted alphabetically by name.
* **Disabled** — Columns follow the order defined by the dataset schema.

This setting applies to CSV deliveries only. Existing connections without the setting behave as if it were enabled, preserving prior behavior.

## Managed S3 buckets

If you don't have an existing S3 bucket, Narrative can provision a managed bucket for you. Contact your Narrative representative to set up a managed S3 destination.

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>
</CardGroup>


# Google Ads Connector
Source: https://docs.narrative.io/reference/connectors/google-ads

Deliver audience segments to Google Ads for Customer Match campaigns

The Google Ads Connector delivers audience data to your Google Ads account using Google's Audience Partner API, enabling you to activate segments for [Customer Match](https://support.google.com/google-ads/answer/6379332) advertising campaigns.

## Overview

The connector uploads user lists to Google Ads, where they become available as audience segments for targeting across Search, Shopping, Gmail, YouTube, and Display campaigns. It supports creating new user lists or appending to existing ones.

## Requirements

### Account setup

| Requirement        | Description                                                                                                                       |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| Google Ads account | A Google Ads Customer ID                                                                                                          |
| Account linkage    | Your Google Ads account must be linked to Narrative's Data Partner account. Contact your Narrative representative to set this up. |

<Note>
  Account linkage is a one-time setup step handled by Narrative support. Once linked, you can create profiles and deliver audiences without further assistance.
</Note>

### Supported identifiers

| Identifier Type                     | Rosetta Stone Attribute                               |
| ----------------------------------- | ----------------------------------------------------- |
| SHA256 hashed email                 | `sha256_hashed_email`                                 |
| Hashed email (SHA256)               | `hashed_email` (type `sha256_email`)                  |
| Raw email                           | `raw_email` (automatically normalized and hashed)     |
| Hashed name + postal code + country | `person_name`, `iso_3166_1_country`, `postal_address` |

Your data must include at least one email-based identifier or the complete set of name, postal code, and country attributes. Both can be present on the same row.

For name-based matching, all three attributes are required together:

* SHA256 hashed first name and last name
* Postal code
* Country code (ISO 3166-1)

<Warning>
  Unlike the [Google DV360 connector](/reference/connectors/google-dv360), the Google Ads connector does not support phone number identifiers.
</Warning>

## Installation

<Steps>
  <Step title="Link your Google Ads account">
    Contact your Narrative representative to link your Google Ads account to Narrative's Data Partner account. You'll need to provide your Google Ads Customer ID.
  </Step>

  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Google Ads Connector**.
  </Step>

  <Step title="Create a profile">
    Enter your **Google Ads Customer ID**, a **Profile name**, and an optional description.
  </Step>
</Steps>

## Delivering audiences

When connecting a dataset to the Google Ads connector, you choose how the audience is created:

### New user list

Creates a new Customer Match user list in your Google Ads account.

| Setting             | Description                                                                                    |
| ------------------- | ---------------------------------------------------------------------------------------------- |
| Name                | Display name for the audience in Google Ads (max 64 characters, defaults to dataset name)      |
| Description         | Optional description shown in Google Ads                                                       |
| Membership duration | How long users remain in the list (1–540 days, default 540). Set to maximum for no expiration. |
| Historical data     | Whether to include previously ingested data (default: enabled)                                 |

### Existing user list

Appends data to a user list that already exists in your Google Ads account.

| Setting         | Description                                                    |
| --------------- | -------------------------------------------------------------- |
| User list ID    | The numeric ID of the existing user list in Google Ads         |
| Historical data | Whether to include previously ingested data (default: enabled) |

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Google DV360 Connector" icon="google" href="/reference/connectors/google-dv360">
    Activate audiences in Display & Video 360
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for Google Ads
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>
</CardGroup>


# Google DV360 Connector
Source: https://docs.narrative.io/reference/connectors/google-dv360

Deliver audience data to Google Display & Video 360

The Google DV360 Connector delivers audience data directly to your Google Display & Video 360 Partner Account for programmatic display and video advertising.

## Overview

The connector enables seamless audience data delivery to DV360, allowing you to activate your segments for targeting across Google's programmatic ecosystem.

## Requirements

### Account access

* A Google Display & Video 360 Partner Account
* Appropriate permissions to create audiences

### Supported identifiers

Google DV360 accepts multiple identifier types. Your data must include at least one of the following:

| Identifier Type             | Rosetta Stone Attribute         |
| --------------------------- | ------------------------------- |
| SHA256 hashed email         | `unique_id.hashed_email.sha256` |
| SHA256 hashed phone         | `unique_id.hashed_phone.sha256` |
| Hashed name + zip + country | Multiple attributes required    |

For the name-based matching option, you need:

* SHA256 hashed first name
* SHA256 hashed last name
* Zip code
* Country code

<Note>
  All identifiers must be hashed with SHA256. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Google DV360 Connector**.
  </Step>

  <Step title="Authenticate with Google">
    Click **Connect** and authorize Narrative to access your DV360 account.
  </Step>

  <Step title="Select your Partner Account">
    Choose the Partner Account where audiences will be created.
  </Step>
</Steps>

## Creating your profile

After installation, configure your delivery profile to specify how audiences should be created in DV360.

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for DV360
  </Card>
</CardGroup>


# Hugging Face Connector
Source: https://docs.narrative.io/reference/connectors/hugging-face

Deliver trained AI models from Narrative to Hugging Face Hub

The Hugging Face Connector exports trained AI models from Narrative's model repository to [Hugging Face Hub](https://huggingface.co), making them available for inference and sharing.

## Overview

Unlike audience delivery connectors that push data to advertising platforms, the Hugging Face Connector handles **model delivery**. After you train a model in Narrative's Model Studio, the connector uploads it to your Hugging Face account as a private model repository.

Delivered models include:

* All model weights and configuration files
* Inference files (`handler.py` and `requirements.txt`) for Hugging Face's inference infrastructure
* A version tag (e.g., `v1`) for tracking model iterations

## Requirements

### Hugging Face access token

You need a Hugging Face **User Access Token** with write permissions. The connector supports two token types:

| Token Type           | Required Permissions                                                     |
| -------------------- | ------------------------------------------------------------------------ |
| Fine-grained token   | `repo.content.read` and `repo.write`, scoped to at least one entity      |
| Write token (legacy) | Automatically grants access to your namespace and all your organizations |

<Warning>
  Organization tokens and other non-user token types are not supported. The token must be a **User Access Token**.
</Warning>

The token must grant write access to at least one model repository, organization, or user namespace.

### Delivery destinations

When delivering a model, you choose where it goes on Hugging Face:

| Option                    | Description                                                                      |
| ------------------------- | -------------------------------------------------------------------------------- |
| New model repository      | Creates a new private repository under the selected owner (organization or user) |
| Existing model repository | Delivers to a repository that already exists                                     |

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** in the left navigation and select **Hugging Face Connector**.
  </Step>

  <Step title="Add a profile">
    Click **Add profile** to begin connecting your Hugging Face account.
  </Step>

  <Step title="Enter your access token">
    Paste your Hugging Face User Access Token into the token field and click **Validate**.
  </Step>

  <Step title="Review accessible entities">
    After validation, expand **Hugging Face Access Token Details** to see the organizations, users, and model repositories your token grants write access to. These are the destinations available for model delivery.
  </Step>

  <Step title="Name your profile">
    Enter a **Profile name** and optional **Description**, then click **Create**.
  </Step>
</Steps>

Your profile is automatically enabled after creation and appears in the **Connector profiles** list on the settings page.

## Delivering a model

Model delivery happens automatically after training completes in Model Studio:

1. Train a model in **Model Studio** by selecting a base model, training dataset, and compute instance
2. Once training finishes, a delivery job is created that uses your connector profile
3. The connector downloads the model from Narrative's model store, prepares inference files, and uploads everything to Hugging Face Hub
4. The model appears as a **private repository** in your Hugging Face account, tagged with the model version

<Note>
  Delivery can take up to 30 minutes depending on model size. The process is idempotent—if a delivery is interrupted, it can safely retry without creating duplicates.
</Note>

## Troubleshooting

### Token validation errors

| Error                       | Cause                                               | Resolution                                                                                       |
| --------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Token is invalid            | Token doesn't authenticate with Hugging Face        | Verify the token value in your [Hugging Face settings](https://huggingface.co/settings/tokens)   |
| Must be a user access token | Token is an organization token or unrecognized type | Create a new User Access Token on Hugging Face                                                   |
| Must grant write access     | Token has no writable entities                      | Update token permissions to include `repo.content.read` and `repo.write` for at least one entity |

### Expired or revoked token

If your token expires or is revoked after profile creation, the profile detail page displays a warning. Contact Narrative Support to resolve the issue—you'll need to create a new profile with a valid token.

***

## Related content

<CardGroup>
  <Card title="Connectors Overview" icon="plug" href="/reference/connectors/index">
    Reference documentation for all available connectors
  </Card>

  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>
</CardGroup>


# Connectors
Source: https://docs.narrative.io/reference/connectors/index

Reference documentation for Narrative connectors and destination platforms

Connectors enable you to deliver data from Narrative to external platforms. Each connector is a pre-built integration that handles authentication, data formatting, and delivery to a specific destination.

## Available connectors

### Advertising platforms

<CardGroup>
  <Card title="The Trade Desk" icon="bullhorn" href="/reference/connectors/the-trade-desk">
    Deliver audiences to TTD's first-party or third-party marketplace with automatic UID2 generation
  </Card>

  <Card title="Meta Custom Audiences" icon="meta" href="/reference/connectors/meta">
    Push custom audiences to Facebook and Instagram advertising
  </Card>

  <Card title="Meta Conversions API" icon="meta" href="/reference/connectors/meta-conversions-api">
    Send conversion events for ad attribution and campaign optimization
  </Card>

  <Card title="Google Ads" icon="google" href="/reference/connectors/google-ads">
    Deliver Customer Match audiences to Google Ads
  </Card>

  <Card title="Google DV360" icon="google" href="/reference/connectors/google-dv360">
    Activate audiences in Google Display & Video 360
  </Card>

  <Card title="TikTok" icon="tiktok" href="/reference/connectors/tiktok">
    Deliver custom audiences to TikTok Ads Manager
  </Card>

  <Card title="Yahoo DSP" icon="y" href="/reference/connectors/yahoo-dsp">
    Activate audiences in Yahoo's demand-side platform
  </Card>

  <Card title="Pinterest" icon="pinterest" href="/reference/connectors/pinterest">
    Deliver custom audiences to Pinterest for ad targeting
  </Card>

  <Card title="Magnite" icon="chart-line" href="/reference/connectors/magnite">
    Deliver audience segments to Magnite's SSP
  </Card>

  <Card title="PubMatic" icon="rectangle-ad" href="/reference/connectors/pubmatic">
    Deliver audience segments to PubMatic for programmatic advertising
  </Card>
</CardGroup>

### AI/ML

<CardGroup>
  <Card title="Hugging Face" icon="brain-circuit" href="/reference/connectors/hugging-face">
    Export trained AI models to Hugging Face Hub
  </Card>
</CardGroup>

### Cloud storage

<CardGroup>
  <Card title="Amazon S3" icon="aws" href="/reference/connectors/amazon-s3">
    Deliver data files to your Amazon S3 bucket
  </Card>
</CardGroup>

***

## Connector requirements

Each connector requires specific identifier types to match users on the destination platform. Ensure your data includes at least one supported identifier before delivery.

| Connector             | Required Identifiers                                                                      |
| --------------------- | ----------------------------------------------------------------------------------------- |
| The Trade Desk        | UID2, SHA256 hashed email, SHA256 hashed phone, Apple IDFA, or Android Advertising ID     |
| Meta Custom Audiences | SHA256 hashed email, SHA256 hashed phone                                                  |
| Meta Conversions API  | Mapped to `meta_conversion_event` attribute                                               |
| Google Ads            | SHA256 hashed email, or hashed name + postal code + country                               |
| Google DV360          | SHA256 hashed email, SHA256 hashed phone, or hashed name + zip + country                  |
| TikTok                | SHA256 hashed email, SHA256 hashed phone, MAID                                            |
| Pinterest             | SHA256 hashed email, SHA256 hashed phone, GAID                                            |
| Yahoo DSP             | SHA256 hashed email, MAID                                                                 |
| PubMatic              | MAID, Android Advertising ID, Apple IDFA, Mobile ID, SHA256 hashed email, or hashed email |
| Hugging Face          | N/A (model delivery)                                                                      |
| Amazon S3             | Any (file delivery)                                                                       |

<Note>
  For detailed identifier formatting requirements, see [Hashing PII for Upload](/guides/ingestion/hashing-pii).
</Note>

<Warning>
  Connectors cannot be attached to [view datasets](/concepts/nql/materialized-views#view-datasets). If you need to deliver query results to an external platform, use a materialized view instead.
</Warning>

***

## Common workflows

### Delivering first-party data

Push your own customer data to platforms for targeting:

1. [Prepare your data](/guides/ingestion/hashing-pii) with hashed identifiers
2. [Upload to a dataset](/guides/sdk/uploading-data)
3. [Map to Rosetta Stone](/guides/rosetta-stone/mapping-schemas) attributes
4. Configure and deliver through the connector

### Delivering acquired data

Activate third-party data purchased through Narrative:

1. Query data from data collaborators using NQL
2. Materialize results into a dataset
3. Deliver through the appropriate connector

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors fit into the activation workflow
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize your data for efficient delivery
  </Card>

  <Card title="Rosetta Stone Mapping" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Map your schema for connector compatibility
  </Card>

  <Card title="Schema Incompatibility Errors" icon="plug-circle-exclamation" href="/reference/errors/dataset-schema-connector-incompatible">
    Resolve HTTP 400 errors when a dataset schema doesn't match a connector
  </Card>
</CardGroup>


# Magnite Connector
Source: https://docs.narrative.io/reference/connectors/magnite

Deliver audience segments to Magnite SSP

The Magnite Connector delivers audience data to Magnite's supply-side platform (SSP), enabling publishers and data providers to make their segments available for programmatic advertising.

## Overview

The connector pushes your audience data to Magnite, where it becomes available for targeting by advertisers buying inventory through Magnite's exchange.

## Requirements

### Account access

* A Magnite account with appropriate permissions
* API credentials from Magnite

### Supported identifiers

Refer to Magnite's documentation for the specific identifier types supported for your integration.

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Magnite Connector**.
  </Step>

  <Step title="Configure credentials">
    Enter your Magnite API credentials and account information.
  </Step>

  <Step title="Complete integration">
    Follow the Magnite-specific setup steps to complete the integration.
  </Step>
</Steps>

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>
</CardGroup>


# Meta Custom Audiences Connector
Source: https://docs.narrative.io/reference/connectors/meta

Deliver custom audiences to Facebook and Instagram advertising

The Meta Custom Audiences Connector delivers audience data directly to Meta's Custom Audiences feature, enabling you to target users on Facebook and Instagram.

<Note>
  Looking to send conversion events instead? See the [Meta Conversions API Connector](/reference/connectors/meta-conversions-api).
</Note>

<iframe title="Meta Connector Demo" />

## Overview

The Meta Connector pushes your audience data to Meta's advertising platform, where it becomes available as a Custom Audience for targeting in your ad campaigns.

## Requirements

### Account access

* A Meta Business Manager account
* Admin access to the Ad Account where audiences will be created
* Appropriate permissions to create Custom Audiences

### Supported identifiers

| Identifier Type     | Rosetta Stone Attribute         |
| ------------------- | ------------------------------- |
| SHA256 hashed email | `unique_id.hashed_email.sha256` |
| SHA256 hashed phone | `unique_id.hashed_phone.sha256` |

<Note>
  Meta requires identifiers to be hashed with SHA256 before upload. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Meta Connector**.
  </Step>

  <Step title="Authenticate with Meta">
    Click **Connect** and authorize Narrative to access your Meta Business Manager account.
  </Step>

  <Step title="Select your Ad Account">
    Choose the Ad Account where you want audiences to be created.
  </Step>
</Steps>

## Delivering audiences

Once configured, you can deliver datasets containing hashed identifiers to create Custom Audiences in Meta.

## Troubleshooting

### Ad account eligibility warnings

When you select an ad account during profile creation, Narrative flags accounts that may not be eligible for Custom Audience delivery. You can still select a flagged account and complete profile creation, but deliveries will fail until the underlying issue is resolved on Meta's side. See [Ad Account Eligibility for Custom Audiences](/guides/connectors/meta/ad-account-access) for detailed steps covering:

* Accepting the Custom Audiences Terms of Service
* Associating the ad account with a Business Manager
* Verifying role permissions
* Completing business verification

### Ads datasets not shown

Ads datasets are only available for profiles whose ad account is associated with a Business Manager. If a profile's ad account has no business association, the ads datasets table is hidden on the profile details page.

***

## Related content

<CardGroup>
  <Card title="Meta Conversions API Connector" icon="meta" href="/reference/connectors/meta-conversions-api">
    Send conversion events for ad attribution
  </Card>

  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for Meta
  </Card>
</CardGroup>


# Meta Conversions API Connector
Source: https://docs.narrative.io/reference/connectors/meta-conversions-api

Send conversion events to Meta for ad attribution and campaign optimization

The Meta Conversions API (CAPI) Connector sends conversion events from Narrative directly to Meta's Conversions API, improving Facebook and Instagram ad attribution and campaign performance.

<iframe title="Meta Conversions API Connector Setup" />

## Overview

The connector routes conversion data — purchases, leads, checkouts, and more — from any dataset in Narrative to a Meta Ads dataset. Events are matched against Meta user accounts using hashed customer information, enabling Meta to attribute conversions back to ad campaigns and optimize delivery.

This connector supports:

* **Offline conversions** (e.g., in-store purchases, phone orders)
* **Online conversions** (e.g., website purchases, lead submissions)
* **App events**

Events are batched and delivered on a schedule you configure. PII fields are SHA-256 hashed before transmission in accordance with Meta's requirements.

## Prerequisites

* A **Meta Ads account** with an active dataset (formerly "pixel"). Your dataset ID is the numeric ID found in Meta Events Manager.
* A Narrative dataset containing the conversion data you want to send.

## Sending conversion events

Delivering conversion events to Meta involves three steps: creating a connector profile, preparing your dataset, and adding a connection.

### Step 1: Create a connector profile

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Connectors** and select **Meta Connector**.
  </Step>

  <Step title="Authenticate with Meta">
    Click **Connect** and sign in with your Meta account. During the OAuth flow, you will be prompted to:

    1. **Select your business portfolio** — Choose the Meta business portfolio that owns the ad accounts you want to deliver events to.
    2. **Grant ad account access** — Select the ad accounts that Narrative should be able to access. You must grant access to at least one ad account.

    Make sure the ad accounts you select have an active dataset (pixel) configured in Meta Events Manager.
  </Step>

  <Step title="Select your ad account">
    Choose the ad account where conversion events should be delivered.
  </Step>
</Steps>

### Step 2: Create your dataset

Your dataset must be mapped to the [`meta_conversion_event`](https://app.narrative.io/platform/rosetta-stone/attributes/meta_conversion_event) Rosetta Stone attribute. The recommended approach is to map your source data to this attribute and then materialize it using Data Studio or NQL.

<Note>
  The connector profile requires your dataset to include `meta_conversion_event` **and** at least one [supported user identifier](#supported-user-identifiers) for user matching. Meta uses these identifiers to match conversion events to user accounts. Any dataset that contains the required attributes in the correct schema is eligible for delivery.
</Note>

See the [attribute schema](#the-meta_conversion_event-attribute) below for the full list of event fields, and [Supported user identifiers](#supported-user-identifiers) for the identity attributes your dataset must include.

### Step 3: Add a connection

<Steps>
  <Step title="Open your connector profile">
    Navigate to the Meta connector profile you created in Step 1.
  </Step>

  <Step title="Add a new connection">
    Click **Add Connection** and select **Deliver Conversion Events** as the connection type.
  </Step>

  <Step title="Configure delivery options">
    * **Deliver all data** — Toggle on to deliver all data currently in the dataset. If off, only new data written after the connection is created will be delivered.
    * **Test Event Code** *(optional)* — Enter a test event code from Meta Events Manager. When provided, events are sent as test events and appear in Meta's **Test Events** tab instead of being processed as real conversions. Use this to validate your setup before going live.
  </Step>

  <Step title="Connect">
    Click **Connect** to activate. Events will begin delivering on the next scheduled run.
  </Step>
</Steps>

## The `meta_conversion_event` attribute

The [`meta_conversion_event`](https://app.narrative.io/platform/rosetta-stone/attributes/meta_conversion_event) attribute is Narrative's standardized Rosetta Stone schema for conversion events sent to Meta. When you map your dataset columns to this attribute, Narrative handles transformation, hashing, batching, and delivery to Meta's API.

### Top-level fields

| Narrative Field                   | Meta CAPI Field                   | Type            | Required | Description                                                                             |
| --------------------------------- | --------------------------------- | --------------- | -------- | --------------------------------------------------------------------------------------- |
| `meta_dataset_id`                 | *(routing)*                       | `long`          | **Yes**  | The Meta Ads dataset/pixel ID to deliver events to.                                     |
| `event_id`                        | `event_id`                        | `string`        | **Yes**  | Unique string for deduplication. Use an order number, transaction ID, or random string. |
| `event_name`                      | `event_name`                      | `string` (enum) | **Yes**  | The type of conversion event. See [Event names](#event-names).                          |
| `event_timestamp`                 | `event_time`                      | `timestamptz`   | **Yes**  | Timestamp when the event occurred.                                                      |
| `source`                          | `action_source`                   | `string` (enum) | **Yes**  | Where the conversion occurred. See [Action sources](#action-sources).                   |
| `source_url`                      | `event_source_url`                | `string`        | No       | The browser URL where the event occurred. Required for `website` events.                |
| `opt_out`                         | `opt_out`                         | `boolean`       | No       | If `true`, the event is used for attribution only, not delivery optimization.           |
| `data_processing_options`         | `data_processing_options`         | `string[]`      | No       | Use `["LDU"]` to enable Limited Data Use for non-consented users.                       |
| `data_processing_options_country` | `data_processing_options_country` | `long`          | No       | Required if LDU is set. `1` = USA, `0` = geolocate.                                     |
| `data_processing_options_state`   | `data_processing_options_state`   | `long`          | No       | Required if LDU is set without an IP address. `1000` = California, `0` = geolocate.     |

### Custom data fields

Additional event data used by Meta for attribution and optimization. Some fields are required depending on event type.

| Narrative Field                 | Meta CAPI Field                 | Type       | Required For   | Description                                                                                                                                                                 |
| ------------------------------- | ------------------------------- | ---------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `custom_data.value`             | `custom_data.value`             | `double`   | `purchase`     | Monetary value of the conversion                                                                                                                                            |
| `custom_data.currency`          | `custom_data.currency`          | `string`   | `purchase`     | ISO 4217 currency code (e.g., `USD`)                                                                                                                                        |
| `custom_data.order_id`          | `custom_data.order_id`          | `string`   | Offline events | Transaction or order ID for offline event deduplication                                                                                                                     |
| `custom_data.predicted_ltv`     | `custom_data.predicted_ltv`     | `double`   | —              | Predicted lifetime value of the conversion                                                                                                                                  |
| `custom_data.content_ids`       | `custom_data.content_ids`       | `string[]` | —              | Product SKUs or content IDs                                                                                                                                                 |
| `custom_data.content_type`      | `custom_data.content_type`      | `string`   | —              | `product` or `product_group`                                                                                                                                                |
| `custom_data.contents`          | `custom_data.contents`          | `object[]` | —              | Array of product objects. Each requires `id` (string) and `quantity` (long). Optional: `item_price` (double), `delivery_category` (`curbside`, `home_delivery`, `in_store`) |
| `custom_data.num_items`         | `custom_data.num_items`         | `long`     | —              | Number of items. Use only with `initiate_checkout`                                                                                                                          |
| `custom_data.search_string`     | `custom_data.search_string`     | `string`   | —              | Search query. Use only with `search` events                                                                                                                                 |
| `custom_data.delivery_category` | `custom_data.delivery_category` | `string`   | —              | Delivery type for purchase events: `curbside`, `home_delivery`, or `in_store`                                                                                               |
| `custom_data.item_number`       | `custom_data.item_number`       | `string`   | —              | Identifier to distinguish multiple events within the same order                                                                                                             |

### Store data fields

For physical store conversions only.

| Narrative Field            | Meta CAPI Field            | Type     | Description                                              |
| -------------------------- | -------------------------- | -------- | -------------------------------------------------------- |
| `store_data.brand_page_id` | `store_data.brand_page_id` | `long`   | Facebook Page ID of the brand                            |
| `store_data.store_code`    | `store_data.store_code`    | `string` | Unique alphanumeric store identifier (max 64 characters) |
| `store_data.store_page_id` | `store_data.store_page_id` | `long`   | Facebook Page ID of the store                            |

***

## Supported user identifiers

In addition to the `meta_conversion_event` attribute, your dataset must include at least one of the following Rosetta Stone identifier attributes. Meta uses these to match conversion events to user accounts. All PII values are SHA-256 hashed by Narrative before transmission.

Providing more identifier types improves Meta's match quality.

| Identifier Attribute                                    | Description                                                          |
| ------------------------------------------------------- | -------------------------------------------------------------------- |
| `sha256_hashed_email`                                   | SHA-256 hashed email address                                         |
| `hashed_email`                                          | Hashed email address (any supported hash)                            |
| `e164_phone_number`                                     | Phone number in E.164 format                                         |
| `telephone_number`                                      | Phone number                                                         |
| `person_name` + `iso_3166_1_country` + `postal_address` | Full name, country, and postal address (all three required together) |
| `android_advertising_id`                                | Android advertising ID (GAID)                                        |
| `apple_idfa`                                            | Apple Identifier for Advertisers (IDFA)                              |
| `mobile_id_unique_identifier`                           | Mobile advertising ID                                                |
| `unique_id`                                             | Any unique user identifier                                           |
| `narrative_id`                                          | Narrative platform identity                                          |

***

## Reference

### Event names

The `event_name` field accepts the following values. Custom event names are not supported.

| Value                   | Meta Standard Event  |
| ----------------------- | -------------------- |
| `add_payment_info`      | AddPaymentInfo       |
| `add_to_cart`           | AddToCart            |
| `add_to_wishlist`       | AddToWishlist        |
| `complete_registration` | CompleteRegistration |
| `contact`               | Contact              |
| `customize_product`     | CustomizeProduct     |
| `donate`                | Donate               |
| `find_location`         | FindLocation         |
| `initiate_checkout`     | InitiateCheckout     |
| `lead`                  | Lead                 |
| `other`                 | Other                |
| `purchase`              | Purchase             |
| `schedule`              | Schedule             |
| `search`                | Search               |
| `start_trial`           | StartTrial           |
| `submit_application`    | SubmitApplication    |
| `subscribe`             | Subscribe            |
| `view_content`          | ViewContent          |

### Action sources

| Value                | Use When                                                                             |
| -------------------- | ------------------------------------------------------------------------------------ |
| `physical_store`     | Conversion happened in a brick-and-mortar location. Required for all offline events. |
| `website`            | Conversion happened on a website. Requires `source_url`.                             |
| `app`                | Conversion happened in a mobile app.                                                 |
| `email`              | Conversion originated from an email campaign.                                        |
| `phone_call`         | Conversion happened over the phone.                                                  |
| `chat`               | Conversion happened via chat.                                                        |
| `business_messaging` | Conversion via business messaging (e.g., WhatsApp).                                  |
| `system_generated`   | Automated or system-generated event.                                                 |
| `other`              | Any other source.                                                                    |

### Event age limits

Meta enforces limits on how old events can be when received:

* **Offline events** (`physical_store`): up to **90 days** old
* **All other event types**: must be **less than 7 days** old

Events outside these windows will be rejected by Meta.

### Deduplication

Meta uses `custom_data.order_id` to deduplicate offline conversion events. For web and app events, `event_id` is used. Ensure these values are stable and consistent if you are sending the same events from multiple sources (e.g., both a pixel and CAPI).

***

## Data privacy

Narrative hashes all customer information fields (email, phone, name, etc.) using SHA-256 before transmission. Raw PII is never sent to Meta.

For users who have not consented to data sharing, set `data_processing_options` to `["LDU"]` and provide the appropriate `data_processing_options_country` and `data_processing_options_state` values to enable Meta's [Limited Data Use](https://developers.facebook.com/docs/marketing-api/overview/data-processing-options) mode.

***

## Related content

<CardGroup>
  <Card title="Meta Custom Audiences Connector" icon="meta" href="/reference/connectors/meta">
    Deliver audience data to Facebook and Instagram advertising
  </Card>

  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for delivery
  </Card>
</CardGroup>


# Pinterest Connector
Source: https://docs.narrative.io/reference/connectors/pinterest

Deliver custom audiences to Pinterest for ad targeting

The Pinterest Connector delivers audience data to Pinterest, enabling you to create custom audiences for advertising campaigns on the Pinterest platform.

## Overview

The connector pushes your audience data to Pinterest, where it becomes available as a Custom Audience for targeting in your Pinterest ad campaigns. You can connect multiple Pinterest profiles, manage ad accounts, and invite third-party partners to connect their accounts using [app invites](/concepts/apps/app-invites).

## Requirements

### Account access

* A Pinterest Business account
* Access to Pinterest Ads Manager
* Appropriate permissions to create Custom Audiences

### Supported identifiers

| Identifier Type               | Rosetta Stone Attribute         |
| ----------------------------- | ------------------------------- |
| SHA256 hashed email           | `unique_id.hashed_email.sha256` |
| SHA256 hashed phone           | `unique_id.hashed_phone.sha256` |
| Android advertising ID (GAID) | `unique_id.maid`                |

<Note>
  All PII identifiers must be hashed with SHA256. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Pinterest Connector**.
  </Step>

  <Step title="Connect your Pinterest account">
    In the **Profiles** tab, click **Connect** and authorize Narrative to access your Pinterest Business account. You can connect multiple profiles.
  </Step>

  <Step title="Select your ad account">
    In the **Ad Accounts** tab, choose the ad account where audiences will be created.
  </Step>
</Steps>

## Delivering audiences

When configuring a connection to deliver data through the Pinterest Connector, you provide:

| Field                | Description                                                                                                                    |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Audience name        | Display name for the audience in Pinterest (auto-populated from your dataset)                                                  |
| Audience description | Optional description shown in Pinterest                                                                                        |
| Profile              | The Pinterest profile to use for delivery                                                                                      |
| Ad Account           | The ad account where the audience will be created                                                                              |
| Deliver all data     | Whether to include historical data already in the dataset, or only deliver new data written after the connection is configured |

## Reconnecting a profile

Pinterest OAuth tokens expire periodically. When a token expires, the connector can no longer deliver audiences through that profile. Instead of deleting and recreating the profile, you can refresh the token in place.

<Steps>
  <Step title="Open the profile details">
    In the **Profiles** tab, click **Manage Profile** on the profile you want to reconnect.
  </Step>

  <Step title="Update the token">
    In the **Token Management** section, click **Update token**. A Pinterest OAuth window opens.
  </Step>

  <Step title="Re-authorize with Pinterest">
    Sign in with the same Pinterest account that was originally used to create the profile and authorize access. The window closes automatically when complete.
  </Step>
</Steps>

<Warning>
  You must use the same Pinterest account that was originally used to create the profile. Using a different account will not update the token.
</Warning>

After reconnection, the profile's token expiration date updates and audience delivery resumes.

## Inviting partners

The Pinterest Connector supports [app invites](/concepts/apps/app-invites), which let you onboard third-party partners without requiring them to have a Narrative account. In the **App Invites** tab, you can generate shareable invitation links that allow partners to connect their Pinterest accounts directly.

Each invite includes a unique URL and code. When a partner opens the link and authorizes their Pinterest account, a profile is automatically created in your Narrative account.

### Reconnect invites

You can also create app invites that allow a partner to refresh the OAuth token on an existing profile. When creating an invite, select **Update existing profile token** and choose the profile to reconnect. The partner receives a link that walks them through the same OAuth flow, updating the token on the existing profile rather than creating a new one.

For more details on how app invites work, see [App Invites](/concepts/apps/app-invites).

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="App Invites" icon="envelope-open-text" href="/concepts/apps/app-invites">
    Onboard partners with shareable invitation links
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for Pinterest
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>
</CardGroup>


# PubMatic Connector
Source: https://docs.narrative.io/reference/connectors/pubmatic

Deliver audience segments to PubMatic for programmatic advertising

The PubMatic Connector delivers audience data to PubMatic's platform via SFTP, enabling data providers and advertisers to make their segments available for programmatic advertising across PubMatic's exchange.

## Overview

The connector pushes your audience data to PubMatic through a secure SFTP connection. Once delivered, your segments become available for targeting by advertisers buying inventory through PubMatic's exchange. Each connector profile stores your SFTP credentials and PubMatic Data Provider ID (DPID), so you can manage multiple delivery configurations.

## Requirements

### Account access

* A PubMatic account with a Data Provider ID (DPID) from your PubMatic account representative
* SFTP credentials (host, port, username, and password) provided by PubMatic
* Delivery and taxonomy destination paths on the PubMatic SFTP server

### Supported identifiers

| Identifier Type              | Rosetta Stone Attribute                 |
| ---------------------------- | --------------------------------------- |
| Mobile advertising ID (MAID) | `unique_id.maid`                        |
| Android Advertising ID       | `unique_id.android_advertising_id`      |
| Apple IDFA                   | `unique_id.apple_idfa`                  |
| Mobile ID                    | `unique_id.mobile_id_unique_identifier` |
| SHA256 hashed email          | `unique_id.hashed_email.sha256`         |
| Hashed email                 | `unique_id.hashed_email`                |

<Note>
  Your dataset must contain at least one of the supported identifier types. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **PubMatic Connector**.
  </Step>

  <Step title="Create a profile">
    Click **New Profile** and provide the following information:

    * **Profile name** — a unique name to identify this configuration
    * **Data Provider ID (DPID)** — the numeric identifier from your PubMatic account representative
    * **SFTP Host** — the SFTP server address (defaults to `sftp-uswest.pubmatic.com`)
    * **SFTP Port** — the port number for the SFTP connection (defaults to `8822`)
    * **SFTP Username** — your PubMatic SFTP username
    * **SFTP Password** — your PubMatic SFTP password
    * **Delivery Destination** — the SFTP path where audience data files are delivered
    * **Taxonomy Destination** — the SFTP path for taxonomy files
  </Step>

  <Step title="Test the connection">
    Click **Test Connection** to verify your SFTP credentials before saving. This confirms that Narrative can reach PubMatic's SFTP server with the provided settings.
  </Step>

  <Step title="Save the profile">
    Once the connection test succeeds, click **Create Profile** to save your configuration.
  </Step>
</Steps>

## Delivering audiences

After creating a profile, you can connect datasets for delivery:

1. Navigate to the dataset you want to deliver.
2. Open the **Connections** tab and select the PubMatic Connector.
3. Choose your PubMatic profile and configure the audience name.
4. Optionally enable **Deliver all data** to include existing data in addition to new records.

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for PubMatic delivery
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>

  <Card title="Schema Incompatibility Errors" icon="plug-circle-exclamation" href="/reference/errors/dataset-schema-connector-incompatible">
    Resolve errors when a dataset schema doesn't match a connector
  </Card>
</CardGroup>


# The Trade Desk Connector
Source: https://docs.narrative.io/reference/connectors/the-trade-desk

Deliver audience data to The Trade Desk for programmatic advertising

The Trade Desk Connector delivers audience data directly to The Trade Desk (TTD) platform, enabling you to activate your segments for programmatic advertising campaigns.

<iframe title="Trade Desk Connector Demo" />

## Overview

The Trade Desk Connector supports two delivery modes:

* **First-party (1P)**: Share your data with your own advertiser account or with specific advertisers who have provided their credentials
* **Third-party (3P)**: Sell data through TTD's marketplace to any advertiser

## Requirements

### Credentials

| Credential    | Description                         | How to obtain                                                                  |
| ------------- | ----------------------------------- | ------------------------------------------------------------------------------ |
| Brand ID      | Identifies your organization in TTD | Request from your TTD account representative or Narrative relationship manager |
| Advertiser ID | Required for 1P delivery            | Provided by the advertiser                                                     |
| Secret Key    | Required for 1P delivery            | Provided by the advertiser                                                     |

### Supported identifiers

| Identifier Type            | Rosetta Stone Attribute              |
| -------------------------- | ------------------------------------ |
| UID2                       | `unified_id_2`                       |
| SHA256 hashed email        | `sha256_hashed_email`                |
| SHA256 hashed phone number | `sha256_hashed_phone_number`         |
| Apple IDFA                 | `apple_idfa`                         |
| Android Advertising ID     | `android_advertising_id`             |
| Hashed email (SHA256)      | `hashed_email` (type `sha256_email`) |

Your data must include at least one of these identifier types. The connector also accepts the generic `unique_id` and `mobile_id_unique_identifier` attributes when their `type` resolves to one of the identifiers above.

<Note>
  The connector automatically generates UID2 tokens from hashed email addresses. You don't need to provide UID2 directly.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** in the left navigation and select **Trade Desk Connector**.
  </Step>

  <Step title="Add your Brand ID">
    In the **Settings** tab, enter your TTD Brand ID.
  </Step>

  <Step title="Configure delivery mode">
    Choose between first-party (1P) or third-party (3P) delivery based on your use case.
  </Step>
</Steps>

## Taxonomy management

The Trade Desk organizes audiences into a hierarchical taxonomy. You create and manage this taxonomy within the connector interface.

### Creating a taxonomy

<Steps>
  <Step title="Open Taxonomy Management">
    Click on the **Taxonomy Management** tab.
  </Step>

  <Step title="Create root folder">
    Click **Create Manually** and enter a name for your root folder (top-level category).
  </Step>

  <Step title="Add structure">
    Use the **+** icon to add folders (for organization) or datasets (for audiences).
  </Step>

  <Step title="Configure pricing">
    Set rate cards on folders and datasets specifying **Percent of media cost rate** and **CPM rate**.
  </Step>
</Steps>

### Taxonomy fields

| Field             | Description                                                       |
| ----------------- | ----------------------------------------------------------------- |
| Name              | Display name of the segment in The Trade Desk                     |
| Description       | Segment description shown in TTD                                  |
| Element ID        | Unique identifier for this segment in TTD                         |
| Parent Element ID | ID of the parent folder                                           |
| Rate Card         | Pricing configuration (Inherited, System, Advertiser, or Partner) |
| Received IDs      | Number of unique IDs sent to TTD                                  |
| Active IDs        | Number of IDs TTD matched successfully                            |
| Sync Status       | Current synchronization state                                     |
| Buyable           | Whether the segment is purchasable (vs. organizational folder)    |
| Dataset ID        | Source dataset for this segment                                   |

## Syncing your taxonomy

After configuring your taxonomy:

1. Click **Actions** in the top right corner
2. Select **Sync taxonomy**
3. Monitor the sync status for each segment

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Structuring Audiences" icon="sitemap" href="/guides/activation/audience-strategies">
    Organize datasets for delivery
  </Card>

  <Card title="UID2" icon="shield-check" href="/concepts/identifiers/uid2">
    Learn about the UID2 identifier
  </Card>
</CardGroup>


# TikTok Connector
Source: https://docs.narrative.io/reference/connectors/tiktok

Deliver custom audiences to TikTok Ads Manager

The TikTok Connector delivers audience data to TikTok Ads Manager, enabling you to create custom audiences for advertising on the TikTok platform.

## Overview

The connector pushes your audience data to TikTok, where it becomes available as a Custom Audience for targeting in your TikTok ad campaigns.

## Requirements

### Account access

* A TikTok for Business account
* Access to TikTok Ads Manager
* Appropriate permissions to create Custom Audiences

### Supported identifiers

| Identifier Type              | Rosetta Stone Attribute         |
| ---------------------------- | ------------------------------- |
| SHA256 hashed email          | `unique_id.hashed_email.sha256` |
| SHA256 hashed phone          | `unique_id.hashed_phone.sha256` |
| Mobile advertising ID (MAID) | `unique_id.maid`                |

<Note>
  All PII identifiers must be hashed with SHA256. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **TikTok Connector**.
  </Step>

  <Step title="Authenticate with TikTok">
    Click **Connect** and authorize Narrative to access your TikTok for Business account.
  </Step>

  <Step title="Select your Ad Account">
    Choose the Ad Account where audiences will be created.
  </Step>
</Steps>

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Hashing PII" icon="hashtag" href="/guides/ingestion/hashing-pii">
    Prepare identifiers for TikTok
  </Card>
</CardGroup>


# Yahoo DSP Connector
Source: https://docs.narrative.io/reference/connectors/yahoo-dsp

Deliver audience data to Yahoo DSP for programmatic advertising

The Yahoo DSP Connector delivers audience data to Yahoo's demand-side platform, enabling you to activate your segments for programmatic advertising campaigns.

## Overview

The connector pushes your audience data to Yahoo DSP, where it becomes available for targeting in your programmatic ad campaigns across Yahoo's network.

## Requirements

### Account access

* A Yahoo DSP account
* API credentials from your Yahoo DSP account

### Supported identifiers

| Identifier Type              | Rosetta Stone Attribute         |
| ---------------------------- | ------------------------------- |
| SHA256 hashed email          | `unique_id.hashed_email.sha256` |
| Mobile advertising ID (MAID) | `unique_id.maid`                |

<Note>
  Email identifiers must be hashed with SHA256. See [Hashing PII for Upload](/guides/ingestion/hashing-pii) for formatting requirements.
</Note>

## Installation

<Steps>
  <Step title="Navigate to the connector">
    In Narrative, go to **Installed Apps** and select **Yahoo DSP Connector**.
  </Step>

  <Step title="Configure credentials">
    Enter your Yahoo DSP API credentials.
  </Step>

  <Step title="Select your account">
    Choose the account where audiences will be created.
  </Step>
</Steps>

## Opt-outs

Yahoo DSP supports opt-out processing. When users opt out through your data subject request workflow, those opt-outs can be propagated to Yahoo DSP.

***

## Related content

<CardGroup>
  <Card title="Data Activation Overview" icon="bullseye-arrow" href="/concepts/data-activation/overview">
    Understand how connectors deliver data
  </Card>

  <Card title="Managing DSRs" icon="user-shield" href="/guides/compliance/managing-dsrs">
    Handle opt-out requests
  </Card>
</CardGroup>


# Access Rule, Table, or View Already Exists
Source: https://docs.narrative.io/reference/errors/access-rule-name-conflict

Resolve HTTP 409 naming conflicts when creating access rules, datasets, or views

This error occurs when you attempt to create or update a dataset, access rule, or view with a `unique_name` that conflicts with an existing resource. Datasets, access rules, and views all share a single namespace — NQL does not distinguish between them — so `unique_name` must be unique across all three resource types.

## Error response

```json theme={null}
{
  "type": "https://docs.narrative.io/reference/errors/access-rule-name-conflict",
  "title": "Access rule, table, or view already exists",
  "status": 409,
  "detail": "An access rule, table, or view with the name '<name>' already exists.",
  "instance": "/api/...",
  "logId": "unique-error-identifier"
}
```

## When this occurs

You receive this error when creating or renaming any of the following and the chosen `unique_name` is already in use:

* **Dataset** — A new or renamed dataset conflicts with an existing dataset, access rule, or view
* **Access rule** — A new or renamed access rule conflicts with an existing dataset, access rule, or view
* **View** — A new or renamed view conflicts with an existing dataset, access rule, or view

The conflict can occur across resource types. For example, creating an access rule named `customer_events` will fail if a dataset or view with that name already exists.

## How to resolve

1. **Check the `detail` field** — It identifies the conflicting name
2. **Choose a unique name** — Pick a `unique_name` that doesn't conflict with any existing dataset, access rule, or view
3. **Check for cross-type conflicts** — If the name looks unfamiliar, the conflict may be with a different resource type (e.g., a dataset blocking an access rule name)
4. **Update instead of create** — If you intended to modify an existing resource, use the update endpoint with the correct resource ID instead of creating a new one

<Tip>
  Because NQL uses `unique_name` to reference datasets, access rules, and views interchangeably, all three share a single namespace. A name used by any one of them cannot be reused by the others.
</Tip>

## Related content

<CardGroup>
  <Card title="Access Rules" icon="key" href="/concepts/primitives/access-rules">
    Understand how access rules control data access
  </Card>

  <Card title="Datasets" icon="database" href="/concepts/primitives/datasets">
    Understand dataset schemas and structure
  </Card>

  <Card title="Error Reference" icon="triangle-exclamation" href="/reference/errors/index">
    Overview of common API errors
  </Card>
</CardGroup>


# Dataset Schema Incompatible with Connector
Source: https://docs.narrative.io/reference/errors/dataset-schema-connector-incompatible

Resolve HTTP 400 schema validation failures when connecting datasets to connectors

This error occurs when you attempt to connect a dataset to a connector, but the dataset's schema is not compatible with the connector's requirements.

## Error response

```json theme={null}
{
  "type": "https://docs.narrative.io/reference/errors/dataset-schema-connector-incompatible",
  "title": "Dataset schema incompatible with connector",
  "status": 400,
  "detail": "The dataset schema is not compatible with the connector. <specific details about the incompatibility>",
  "instance": "/api/connectors",
  "logId": "unique-error-identifier"
}
```

## Common causes

* **Missing required columns** — The dataset is missing columns that the connector requires
* **Incompatible column types** — A column exists but its data type does not match what the connector expects
* **Schema structure mismatch** — The dataset's overall schema structure does not align with the connector's input requirements

## How to resolve

1. **Check the `detail` field** — It describes the specific incompatibility between your dataset schema and the connector
2. **Review connector requirements** — Each connector expects a specific schema structure; verify your dataset matches
3. **Update your dataset schema** — Modify the dataset to include the required columns with compatible data types
4. **Try a different connector** — If the dataset cannot be modified, use a connector that supports the dataset's schema

## Error response fields

| Field      | Description                                                     |
| ---------- | --------------------------------------------------------------- |
| `type`     | Link to this documentation page                                 |
| `title`    | Brief description of the schema incompatibility                 |
| `status`   | `400` — indicates a client request validation error             |
| `detail`   | Specific explanation of which columns or types are incompatible |
| `instance` | The API endpoint that produced the error                        |
| `logId`    | Unique identifier for support requests                          |

## Related content

<CardGroup>
  <Card title="Connectors" icon="plug" href="/reference/connectors/index">
    Reference for available connectors and their requirements
  </Card>

  <Card title="Datasets" icon="database" href="/concepts/primitives/datasets">
    Understand dataset schemas and structure
  </Card>

  <Card title="Error Reference" icon="triangle-exclamation" href="/reference/errors/index">
    Overview of common API errors
  </Card>
</CardGroup>


# Error Reference
Source: https://docs.narrative.io/reference/errors/index

Common API error responses and how to resolve them

When an API request fails, the platform returns a structured error response with details about what went wrong. This section helps you diagnose and resolve common errors.

## Understanding error responses

API errors return a JSON response with the following structure:

```json theme={null}
{
  "type": "https://docs.narrative.io/reference/errors/...",
  "title": "Error Title",
  "status": 409,
  "detail": "Detailed explanation of what went wrong",
  "instance": "/api/endpoint",
  "logId": "unique-error-identifier"
}
```

| Field      | Description                                          |
| ---------- | ---------------------------------------------------- |
| `type`     | URL identifying the error type with resolution steps |
| `title`    | Brief error summary                                  |
| `status`   | HTTP status code                                     |
| `detail`   | Detailed explanation of the problem                  |
| `instance` | The API endpoint that produced the error             |
| `logId`    | Unique identifier for support requests               |

<Tip>
  When contacting support about an error, include the `logId` value to help quickly identify your specific issue.
</Tip>

## Common errors

<CardGroup>
  <Card title="Access Rule, Table, or View Already Exists" icon="clone" href="/reference/errors/access-rule-name-conflict">
    Resolve HTTP 409 naming conflicts when creating access rules, datasets, or views
  </Card>

  <Card title="Dataset Schema Incompatible with Connector" icon="plug-circle-exclamation" href="/reference/errors/dataset-schema-connector-incompatible">
    Resolve HTTP 400 schema validation failures when connecting datasets to connectors
  </Card>
</CardGroup>

## General troubleshooting steps

1. **Read the error detail** — The `detail` field usually explains exactly what's wrong
2. **Check the status code** — Identifies the category of error (400 for validation, 409 for conflicts, etc.)
3. **Review your request** — Look for the specific element mentioned in the error
4. **Check the `type` URL** — Follow the link for detailed resolution steps


# Glossary
Source: https://docs.narrative.io/reference/glossary

Definitions for terms and concepts used throughout Narrative I/O

This glossary defines key terms and concepts used throughout the Narrative I/O documentation and platform.

***

## A

### AI\_COMPLETE

An NQL function that runs large language model inference directly within a query on Snowflake data planes. `AI_COMPLETE` takes a model identifier, a prompt column, model configuration, and a JSON Schema for structured output, then returns the model's response as a JSON string. Because inference executes within the data plane, data never leaves customer-controlled infrastructure. Currently available on Snowflake data planes only.

**Related:** [AI Functions](/nql/functions/ai-functions), [Model Inference Overview](/concepts/model-inference/overview), [Data Enrichment Cookbook](/cookbooks/nql/ai-enrichment)

### Audience

A segment of records from a dataset, defined by a set of filter conditions, that is delivered to one or more external platforms for targeting, measurement, or analysis. Audiences are created using Audience Studio or by writing NQL queries directly. Each audience is materialized as its own dataset and can be refreshed on a schedule to stay current as source data changes.

**Related:** [Audience Studio](/concepts/data-activation/audience-studio), [Building an Audience](/guides/audience-studio/building-an-audience), [Structuring Audiences](/guides/activation/audience-strategies)

### Audience Studio

The platform's visual audience builder. Provides a guided workflow for selecting a data source, applying filters, choosing connector destinations, forecasting audience size, and creating the audience — without writing NQL by hand.

**Related:** [Audience Studio](/concepts/data-activation/audience-studio), [Building an Audience](/guides/audience-studio/building-an-audience)

### Addressability expansion

The practice of appending additional identifiers to existing identity graph records to improve match rates in downstream activation platforms, without changing the graph's structure. Distinct from graph enrichment, which adds edges between nodes to improve resolution.

**Related:** [Addressability Expansion](/concepts/identifiers/addressability), [Identity Graphs](/concepts/identifiers/identity-graph), [Graph Enrichment](/concepts/identifiers/graph-enrichment)

### Access rules

Business rules that define how other organizations can access data within the Narrative platform. Access rules give data owners fine-grained control over who can query their data, what records and fields are accessible, and how much access costs. Every dataset requires at least one access rule before other organizations can query it.

Access rules are enforced at query execution time by the control plane and data planes. They support flexible pricing (including free access for non-commercial sharing), partner-specific permissions, and field-level controls.

**Related:** [Access Rules](/concepts/primitives/access-rules), [Security Model](/concepts/architecture/security-model)

### API key

A long-lived authentication token for programmatic access to the Narrative API. Each API key is scoped with permissions that control which resources it can access and at what level (Read or Write). Keys support expiration dates and should be rotated regularly.

**Related:** [API Keys](/account-settings/api-keys), [API Key Permissions](/concepts/security/api-key-permissions), [Permissions Reference](/reference/security/permissions)

### Attribute lineage

The automatic tracking and preservation of Rosetta Stone attribute mappings when creating materialized views. When source datasets have existing attribute mappings, lineage ensures these mappings persist in derived views without requiring manual re-mapping. Mappings are preserved when selecting unaltered attributes, querying from existing materialized views, or selecting all fields that constitute an attribute mapping.

**Related:** [Materialized Views](/concepts/nql/materialized-views#attribute-lineage), [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works)

### Attributes

Standardized field definitions in Rosetta Stone that form the common schema for data normalization. Each attribute has an immutable numeric ID, a name, description, type, and optional validations. Attributes can be global (platform-wide standards like `hl7_gender`) or organization-specific (scoped to one organization and its collaborators). Also referred to as **normalized attributes** or **Rosetta Stone attributes**.

**Related:** [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works), [Creating Normalized Attributes](/guides/rosetta-stone/creating-normalized-attributes), [Attribute Types](/reference/rosetta-stone/attribute-types)

### App code

A unique, publicly accessible identifier assigned to each app invite. Included in the shareable invite URL, it allows invitees to look up invite details without Narrative authentication.

**Related:** [App Invites](/concepts/apps/app-invites)

### App invite

A mechanism that allows applications to generate shareable invitation links for third-party account onboarding. The invitee follows the link, connects their account, and a profile is created in the inviting company's Narrative account without requiring Narrative authentication. Invites progress through `pending`, `active`, and `archived` states.

**Related:** [App Invites](/concepts/apps/app-invites)

***

## B

### Base model

A pre-trained foundation model that serves as the starting point for fine-tuning. Base models have been trained on large datasets to develop general capabilities, which can then be specialized for specific tasks or domains through fine-tuning with custom training data. In LLM Studio, you select a base model before providing your training dataset.

**Related:** [LLM Studio Interface](/reference/ui/llm-studio), [Fine-tuning](#fine-tuning)

***

## C

### CALL\_MODEL\_FUNCTION

An NQL function that invokes a function on a custom model registered in the Snowflake ML Model Registry. Takes a model name, optional version, function name, and variadic arguments, and returns the model's output as a JSON string. Used for running custom predictions, embeddings, or other ML operations directly within NQL queries. Currently available on Snowflake data planes only.

**Related:** [AI & ML Functions](/nql/functions/ai-functions), [AI\_COMPLETE](/nql/functions/ai-functions#ai_complete)

### CCPA (California Consumer Privacy Act)

A comprehensive privacy law that gives California residents rights over their personal information, including the rights to know what data is collected, to delete it, to opt-out of its sale, and to non-discrimination for exercising these rights. As amended by CPRA, it applies to for-profit businesses meeting certain revenue or data volume thresholds.

**Related:** [CCPA](/concepts/compliance/ccpa), [GDPR](#gdpr-general-data-protection-regulation)

### Classifier Studio

A UI tool for building and training classification models on datasets within Narrative's platform. Classifier Studio provides a guided builder with steps for dataset and label column selection, feature configuration, algorithm selection (Logistic Regression or Random Forest) with hyperparameter tuning and test/train split settings, and a finalize step for naming, versioning, and reviewing the training request. Training runs on the data plane — data never leaves your infrastructure.

**Related:** [Classifier Studio Interface](/reference/ui/classifier-studio), [LLM Studio](#llm-studio), [Model Inference](/concepts/model-inference/overview)

### Chunking

An automatic query execution strategy that splits large dataset scans into smaller, time-bounded segments. Chunking improves stability and cost efficiency for queries over very large datasets (tens of gigabytes to hundreds of terabytes) by limiting the blast radius of failures and enabling targeted retries. The platform determines when chunking applies based on query structure—no user configuration is required.

**Related:** [Chunking](/concepts/nql/chunking), [Query Processing](/concepts/architecture/query-processing), [Materialized Views](/concepts/nql/materialized-views)

### Cardinality

The number of distinct values in a column. Approximate cardinality uses probabilistic algorithms (HyperLogLog) for efficiency on large datasets, while exact cardinality counts every unique value.

**Related:** [Dataset Statistics](/concepts/primitives/dataset-statistics), [Dataset Statistics Reference](/reference/architecture/dataset-statistics)

### Committed-Usage

A pricing model that provides discounted usage rates when you commit to a set amount of platform usage over a period of time. Unlike pay-as-you-go pricing, Committed-Usage requires a contractual agreement and offers rates negotiated with Narrative's sales team.

**Related:** [Pricing](/reference/billing/pricing), [Processing Fee](#processing-fee), [Transfer Fee](#transfer-fee)

### Consent

In privacy law, a data subject's freely given, specific, informed, and unambiguous indication of agreement to the processing of their personal data. Under GDPR, consent is one of six legal bases for processing. Under CCPA, consent is typically required for selling personal information of minors.

**Related:** [Legal basis](#legal-basis), [GDPR](/concepts/compliance/gdpr)

### Control plane

The centralized orchestration layer that manages metadata, enforces permissions, and coordinates queries across data planes. The control plane never touches raw data directly—it only handles the coordination and governance aspects of data collaboration.

**Related:** [Control Plane Architecture](/concepts/primitives/control-plane)

### Context selector

The UI element in the platform's top navigation that lets you view and change your execution context—the combination of data plane, compute pool, database, and schema that controls where operations run and what data is visible.

**Related:** [Using the Context Selector](/guides/platform/using-the-context-selector), [Execution Context](/concepts/primitives/execution-context)

### Cookie syncing

The process of sharing cookie-based identifiers between domains by passing IDs as query parameters. Because web browsers enforce same-origin policies that prevent one domain from reading another domain's cookies, cookie syncing creates a shared understanding of user identity across separate cookie namespaces.

**Related:** [Data Collection Endpoint](/reference/integrations/data-collection-endpoint)

### Connected component

In an identity graph, a cluster of identifiers that the system has determined belong to the same individual or household. Every identifier in a connected component is reachable from every other through some path of linkages.

**Related:** [Identity Graphs](/concepts/identifiers/identity-graph), [ID Mapping](/concepts/identifiers/id-mapping)

### Company ID

A unique identifier assigned to each partner or customer organization on the Narrative data collaboration platform. Company IDs are used internally to identify organizations in API calls, integrations, and data collaboration workflows.

To find the company name associated with a Company ID, or to determine your own organization's Company ID, contact your Narrative partner success representative or [reach out to support](mailto:support@narrative.io).

**Related:** [Data Collection Endpoint](/reference/integrations/data-collection-endpoint)

### Completeness

The ratio of non-null values to total rows for a column, expressed as a value between 0 and 1. A completeness of 1.0 means every row has a value; 0.0 means the column is entirely null.

**Related:** [Dataset Statistics](/concepts/primitives/dataset-statistics), [Dataset Statistics Reference](/reference/architecture/dataset-statistics)

### Confidence score

A numerical rating (0-100%) generated by AI that indicates the likely accuracy of a Rosetta Stone mapping. Higher scores suggest the mapping is more reliable. Confidence scores are calculated based on column name analysis, data sample inspection, and transformation logic evaluation. Scores are grouped into tiers: high (80-100%), medium (50-79%), and low (0-49%).

**Related:** [Confidence Scoring](/concepts/rosetta-stone/confidence-scoring), [Managing Evaluations](/guides/rosetta-stone/managing-evaluations)

### Compute pool

A compute resource allocation that determines the processing power available for query execution within a data plane. On AWS-based data planes, compute pools are either dedicated (isolated resources) or shared (pooled resources). On Snowflake-based data planes, each compute pool maps to a registered Snowflake warehouse. The available options depend on your data plane's underlying provider.

**Related:** [Compute Pools](/concepts/primitives/compute-pools), [Execution Context](/concepts/primitives/execution-context)

### Connector

A pre-built integration that connects Narrative to an external destination platform. Connectors handle the complexities of data delivery including format translation, identity matching, API integration, and audience management. Each connector is specific to a destination (e.g., The Trade Desk Connector, Meta Connector).

**Related:** [Data Activation](/concepts/data-activation/overview), [Destination](#destination), [Connector Reference](/reference/connectors/index)

### CPRA (California Privacy Rights Act)

An amendment to CCPA that took effect January 1, 2023, strengthening California's privacy law by adding new consumer rights (correction, limiting sensitive data use), creating the California Privacy Protection Agency (CPPA) for enforcement, and expanding business obligations.

**Related:** [CCPA](#ccpa-california-consumer-privacy-act), [CCPA Concepts](/concepts/compliance/ccpa)

***

## D

### Data collaboration

The process by which multiple organizations share, query, and analyze each other's data while maintaining control and compliance. Data collaboration in Narrative enables organizations to unlock value from combined datasets without sacrificing data governance.

### Data controller

Under GDPR, an entity that determines the purposes and means of processing personal data. The controller decides what data to collect, why to collect it, and how it will be used. Controllers bear primary responsibility for compliance with data protection principles.

**Related:** [Data processor](#data-processor), [GDPR](/concepts/compliance/gdpr)

### Data crosswalk

A method for linking or matching records across different data sources without exposing the underlying identifiers. In privacy-safe implementations, crosswalks use pseudonymization techniques like hashing to enable record matching while protecting individual privacy. Data crosswalks are essential for data collaboration scenarios where organizations need to find common records (such as overlapping customers) without sharing raw PII.

**Related:** [Data Pseudonymization](/concepts/security/data-pseudonymization), [Hashing](/concepts/security/hashing)

### Data enrichment

The process of enhancing existing customer or audience data with additional attributes from external sources. Common enrichment attributes include demographics (age, gender), location data, purchase behavior, and interest signals. In Narrative, enrichment is performed by joining seed datasets with provider data through Rosetta Stone identity matching. Unlike per-use data licensing models, enriched data can typically be used across multiple platforms and use cases under an omni-use license.

**Related:** [Data collaboration](#data-collaboration), [Demographic Enrichment](/cookbooks/use-cases/demographic-enrichment)

### Data plane

The infrastructure where data physically resides and where queries execute. Data planes can be hosted by Narrative (Narrative-hosted) or deployed within a customer's own infrastructure (customer-hosted) for maximum control over data residency.

**Related:** [Data Planes](/concepts/primitives/data-planes)

### Data portability

A data subject right under GDPR allowing individuals to receive their personal data in a structured, commonly used, machine-readable format and to transmit that data to another controller. This right supports individuals' ability to switch between service providers.

**Related:** [GDPR](/concepts/compliance/gdpr)

### Data processor

Under GDPR, an entity that processes personal data on behalf of a data controller. Processors act only on the controller's documented instructions and must maintain appropriate security measures. Service providers under CCPA serve a similar role.

**Related:** [Data controller](#data-controller), [GDPR](/concepts/compliance/gdpr)

### Data Processing Agreement (DPA)

A legally binding contract between a data controller and data processor that governs the processing of personal data. Required under GDPR, DPAs specify the subject matter, duration, nature, and purpose of processing, as well as the processor's obligations regarding data security, sub-processors, and data subject rights.

**Related:** [Data controller](#data-controller), [Data processor](#data-processor), [GDPR](/concepts/compliance/gdpr)

### Data products

Curated, packaged datasets designed for sharing with partner organizations. Data products include relevant metadata, access controls, and documentation to enable effective data collaboration.

### Data subject

Under GDPR, an identified or identifiable natural person whose personal data is being processed. Data subjects have various rights including access, rectification, erasure, and data portability.

**Related:** [GDPR](/concepts/compliance/gdpr)

### Data Subject Request (DSR)

A formal request from a data subject exercising their rights under privacy laws such as GDPR or CCPA. DSRs include opt-out requests (where the data subject asks to be removed from marketing or advertising) and data erasure requests (where the data subject requests complete deletion of their personal data). In Narrative, DSRs are submitted by mapping identifiers to the Data Privacy Request Identifier attribute, which automatically excludes those identifiers from queries and propagates requests to downstream data collaborators.

**Related:** [Data subject](#data-subject), [Managing Data Subject Requests](/guides/compliance/managing-dsrs), [GDPR](/concepts/compliance/gdpr), [CCPA](/concepts/compliance/ccpa)

### Dataset statistics

Column-level metrics computed over dataset contents, including counts, bounds, distributions, and quality indicators. Statistics are computed per column and vary by data type—numeric columns support the full set of metrics, while complex types like arrays support only basic counts and storage metrics.

**Related:** [Dataset Statistics](/concepts/primitives/dataset-statistics), [Dataset Statistics Reference](/reference/architecture/dataset-statistics)

### Dataset

A structured collection of data registered in Narrative with a defined schema. Datasets function like database tables—they have fields (columns), hold records (rows), and can be queried using NQL. Each dataset is owned by a single company, scoped to a specific data plane, and can be shared with other organizations through access rules.

**Related:** [Datasets](/concepts/primitives/datasets), [Managing Datasets](/guides/sdk/managing-datasets)

### Data sample

A preview of up to 1,000 rows from a dataset, retrieved via a sampling job and stored in the [control plane](#control-plane) for quick access through the API or UI. Samples are used to view [interactive query](#interactive-query) results and inspect dataset contents without downloading the full dataset from the [data plane](#data-plane). Samples can be requested on demand and cleared when no longer needed.

**Related:** [Data Flow](/concepts/architecture/data-flow#how-you-view-query-results), [Managing Datasets](/guides/sdk/managing-datasets#sampling-data)

### Device alias

A data field that provides details about a user's device, broken out into make, model, and operating system. Device alias data helps categorize user devices and is collected through SDKs or real-time bidding environments. Examples include "iPhone X, iOS 13.3.1" or "Pixel 3, Android 10.0".

### Encoding space

The per-partner namespace for Narrative IDs. The same clear text identifier encoded for different partners produces different Narrative IDs, ensuring that identifiers cannot be correlated across partners without going through Narrative's translation functionality.

**Related:** [Narrative ID](/concepts/security/narrative-id), [Using Narrative ID](/guides/collaboration/using-narrative-id)

***

## E

### Execution context

The combination of data plane, compute pool, database, and schema that determines where and how platform operations execute. Your execution context controls which datasets are visible, where queries run, and what compute resources are used. The context selector in the platform's top navigation lets you view and change these settings.

**Related:** [Execution Context](/concepts/primitives/execution-context), [Compute Pools](/concepts/primitives/compute-pools), [Using the Context Selector](/guides/platform/using-the-context-selector)

***

## F

### Fine-tuning

The process of training a pre-trained AI model on a specific dataset to adapt its behavior for particular tasks or domains. Fine-tuning adjusts the model's weights based on your training examples, enabling the model to perform better on your use case while retaining its general capabilities. Narrative's LLM Studio provides an interface for fine-tuning models using datasets prepared in Prompt Studio.

**Related:** [LLM Studio Interface](/reference/ui/llm-studio), [Prompt Studio Interface](/reference/ui/prompt-studio), [Base model](#base-model)

### fine\_tuning\_conversation

A Rosetta Stone attribute format for training data used in AI model fine-tuning. Each row represents a structured conversation with system, user, and assistant roles. Datasets must be mapped to this attribute and materialized before use in LLM Studio.

**Related:** [LLM Studio Interface](/reference/ui/llm-studio), [Prompt Studio Interface](/reference/ui/prompt-studio)

***

## G

### Graph enrichment

The practice of using third-party identity data to create or strengthen connections between nodes in an identity graph, improving person-level and household-level resolution. Distinct from addressability expansion, which appends identifiers without changing graph structure.

**Related:** [Graph Enrichment](/concepts/identifiers/graph-enrichment), [Identity Graphs](/concepts/identifiers/identity-graph), [Addressability Expansion](/concepts/identifiers/addressability)

### GDPR (General Data Protection Regulation)

The European Union's comprehensive data protection law that governs the collection, processing, and storage of personal data for individuals in the EU. Effective since May 2018, GDPR establishes data subject rights, requires legal bases for processing, mandates breach notification, and imposes significant penalties for non-compliance. It has influenced privacy legislation worldwide.

**Related:** [GDPR](/concepts/compliance/gdpr), [CCPA](#ccpa-california-consumer-privacy-act)

***

## H

### Health check

A diagnostic job that validates the operational health of a [data plane](/concepts/primitives/data-planes) and its [compute pools](/concepts/primitives/compute-pools). Health checks verify that infrastructure dependencies—such as Snowflake warehouse references or AWS IAM trust relationships—are correctly configured and accessible. When issues are detected, the system can automatically remediate (for example, archiving compute pools whose backing resources no longer exist).

**Related:** [Job Types](/reference/architecture/job-types), [Data Planes](/concepts/primitives/data-planes), [Compute Pools](/concepts/primitives/compute-pools)

### Hash function

A one-way mathematical algorithm that converts input data (like an email address) into a fixed-length string of characters. Hash functions are deterministic (same input always produces same output), non-reversible (you cannot recover the original input from the hash), and collision-resistant (different inputs produce different outputs). Common hash functions include MD5, SHA-1, and SHA-256.

**Related:** [Data Pseudonymization](/concepts/security/data-pseudonymization), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Histogram

A frequency distribution showing how values in a column are distributed across distinct buckets. In the context of dataset statistics, histograms are computed per column and power UI features like filter dropdowns in Data Studio.

**Related:** [Dataset Statistics](/concepts/primitives/dataset-statistics), [Dataset Statistics Reference](/reference/architecture/dataset-statistics)

### Hash join

An efficient join algorithm where the query planner builds a hash table from one table's join keys and probes it with the other table's keys. Hash joins require single-key equality conditions and are significantly faster than nested loop joins for large datasets.

**Related:** [Understanding JOIN Performance](/concepts/nql/join-performance)

***

## I

### Identity graph

A data structure that connects identifiers (emails, device IDs, phone numbers, cookies) across devices, channels, and platforms into unified profiles. Each cluster of connected identifiers (a connected component) represents an individual or household.

**Related:** [Identity Graphs](/concepts/identifiers/identity-graph), [Graph Enrichment](/concepts/identifiers/graph-enrichment), [Addressability Expansion](/concepts/identifiers/addressability)

### Identity provider (IdP)

An enterprise system (such as Okta, Azure AD, or OneLogin) that manages user authentication and maintains the authoritative record of user identities and credentials. Organizations can integrate their IdP with Narrative for single sign-on access.

**Related:** [SSO Configuration](/account-settings/sso), [SSO Concepts](/concepts/security/sso)

### Interactive query

A query executed through the Query Editor or NQL API that stores results as a temporary dataset. Interactive queries are implemented as [materialized views](#materialized-view) with a 24-hour retention policy and an automatic row limit. Users view results through [data sampling](#data-sample) rather than receiving the full result set directly.

**Related:** [Query Processing](/concepts/architecture/query-processing), [Data Flow](/concepts/architecture/data-flow), [Write Your First Query](/getting-started/first-nql-query)

### Incremental View Maintenance (IVM)

An optimization technique that updates materialized views by processing only changed data rather than recomputing the entire result. IVM dramatically reduces refresh time for views over large datasets with relatively small changes between refreshes.

**Related:** [Incremental View Maintenance](/concepts/nql/incremental-view-maintenance), [Materialized view](#materialized-view)

### Inference configuration

A set of parameters that control how an LLM processes a [Model Inference](#model-inference) request. Configuration options include `output_format_schema` (JSON Schema defining the expected response structure), `max_tokens` (maximum response length), `temperature` (response randomness from 0-1), `top_p` (nucleus sampling parameter), and `stop_sequences` (tokens that terminate generation).

**Related:** [Model Inference Overview](/concepts/model-inference/overview), [Running Model Inference](/guides/sdk/running-model-inference)

### Idempotency key

A unique identifier included in each [webhook subscription](#webhook-subscription) event delivery. If the same event is delivered more than once, the idempotency key remains the same, allowing your endpoint to detect and discard duplicate notifications.

**Related:** [Webhook Event Reference](/reference/webhooks/event-reference), [Subscribing to Notifications](/guides/webhooks/subscribing-to-notifications)

### Inference job

A [job](#job-queue) type that submits a prompt to an LLM hosted within a [data plane](#data-plane) and returns a [structured output](#structured-output-inference). Unlike external AI API calls, inference jobs keep data within customer infrastructure—no data is sent to external model providers. The job returns token usage statistics and a response conforming to the specified JSON Schema.

**Related:** [Model Inference Overview](/concepts/model-inference/overview), [Job Types](/reference/architecture/job-types)

***

## J

### Job queue

A [control plane](#control-plane) component that coordinates work between the control plane and [data planes](#data-plane). The job queue holds various types of jobs—including compiled queries, dataset operations, and system tasks—that data plane [operators](#operator-data-plane) poll and execute. This pull-based architecture ensures the control plane never needs direct access to data plane infrastructure.

**Related:** [Query Processing](/concepts/architecture/query-processing), [Operator](#operator-data-plane), [Job Types](/reference/architecture/job-types)

### Just-in-time provisioning (JIT)

The automatic creation of user accounts on first login, eliminating the need for manual account creation before SSO access. When enabled, Narrative creates user accounts automatically when users authenticate through SSO for the first time.

**Related:** [SSO Configuration](/account-settings/sso)

***

## L

### Legal basis

Under GDPR, a lawful ground that permits the processing of personal data. GDPR recognizes six legal bases: consent, contract performance, legal obligation, vital interests, public task, and legitimate interests. Organizations must identify and document a valid legal basis before processing personal data.

**Related:** [Consent](#consent), [GDPR](/concepts/compliance/gdpr)

### Look-alike audience

An audience built by scoring a population dataset against a seed audience and keeping the users that most resemble it. Used for prospecting beyond known customers. Built in Look-alike Studio.

**Related:** [Look-alike Studio](#look-alike-studio), [Building a look-alike audience](/guides/audience-studio/building-a-lookalike-audience)

### Look-alike Studio

A guided builder for creating look-alike audiences. The workflow takes a seed audience and a population dataset, classifies population fields into identity, feature, and metadata attributes, and produces a new audience of the top-scoring users.

**Related:** [Look-alike audience](#look-alike-audience), [Audience Studio](/concepts/data-activation/audience-studio), [Building a look-alike audience](/guides/audience-studio/building-a-lookalike-audience)

### LLM Studio

A UI tool for training and fine-tuning AI models using datasets within Narrative's platform. LLM Studio integrates dataset selection, base model selection, and compute resource configuration into a unified workflow. Training datasets must be prepared in the `fine_tuning_conversation` format using Prompt Studio before use.

**Related:** [LLM Studio Interface](/reference/ui/llm-studio), [Prompt Studio](/reference/ui/prompt-studio), [Fine-tuning](#fine-tuning)

### Licensed Data Fees

Fees paid to Data Licensors for the licensing of data transacted through Narrative's platform. These fees are set by and paid to the Data Licensor (the party providing the data), not to Narrative. Narrative charges a Transaction Services Fee as a percentage of the Licensed Data Fees.

**Related:** [Pricing](/reference/billing/pricing), [Transaction Services Fee](#transaction-services-fee)

***

## M

### Mapping (Rosetta Stone)

A translation rule that associates a Rosetta Stone attribute with a specific column in a dataset. Mappings can include transformation expressions to convert source data formats to the standard attribute format. Each mapping connects a source column to a target attribute and optionally includes NQL logic to transform values.

**Related:** [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works), [Mapping Schemas](/guides/rosetta-stone/mapping-schemas)

### Mapping evaluation

The AI-powered analysis process that assesses the quality of existing Rosetta Stone mappings. Evaluations generate confidence scores for each mapping and identify potential issues such as missing transformation cases, type mismatches, or edge case handling. Run evaluations to understand the overall health of your normalizations.

**Related:** [Confidence Scoring](/concepts/rosetta-stone/confidence-scoring), [Managing Evaluations](/guides/rosetta-stone/managing-evaluations)

### Mapping suggestion

An AI-generated recommendation for mapping a dataset column to a Rosetta Stone attribute. Suggestions include the proposed transformation expression, confidence score, reasoning, and sample output preview showing before/after values. Users can accept suggestions to create mappings or reject them if they don't fit the use case.

**Related:** [Accepting AI Suggestions](/guides/rosetta-stone/accepting-ai-suggestions)

### Marketplace

The Narrative platform's catalog where organizations discover available datasets from other organizations. The marketplace enables data discovery and facilitates data collaboration relationships.

### Marketplace Match Report

A report that measures how a customer's identity data overlaps with a third-party supplier's data asset. It quantifies three things: the identity match (how many customer persons the supplier can identify), identifier enrichment (how many additional identifiers the supplier contributes), and demographic enrichment (the supplier's demographic and behavioral data for matched persons). All counts use approximate distinct counting (HyperLogLog). Reports run inside the secure enclave without either party seeing the other's raw data.

**Related:** [Building a Marketplace Match Report](/guides/graph-studio/building-a-marketplace-match-report), [Viewing a Marketplace Match Report](/guides/graph-studio/viewing-marketplace-match-reports)

### Mobile Advertising ID (MAID)

A unique pseudo-anonymous identifier tied to a mobile device, provided by the mobile operating system. iOS calls this the Identifier For Advertisers (IDFA), while Android calls it the Advertising ID (GAID or Ad ID). Both consist of 32 hyphen-separated characters. MAIDs are resettable and users can opt out of their collection.

**Related:** [Mobile Ad IDs](/concepts/identifiers/mobile-ad-ids)

### Match table

A pre-generated table that pairs Narrative IDs with clear text identifiers, allowing partners to use Narrative IDs for matching independently of the Narrative platform. Match tables enable offline workflows while maintaining privacy through the encoding.

**Related:** [Narrative ID](/concepts/security/narrative-id), [Using Narrative ID](/guides/collaboration/using-narrative-id)

### MD5

Message-Digest Algorithm 5—a widely-used hash function that produces a 128-bit (32-character hexadecimal) hash value. While cryptographically broken for security purposes, MD5 remains acceptable for non-security applications like identifier matching in data collaboration. Narrative supports MD5 alongside SHA-1 and SHA-256 for hashing PII.

**Related:** [Hash function](#hash-function), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Data Collaboration MCP Server (Model Context Protocol)

A remote server that exposes Narrative platform operations as tools that AI agents can discover and call through the [Model Context Protocol](https://modelcontextprotocol.io/). MCP-compatible clients — such as Claude Code, Claude Desktop, Cursor, and Windsurf — connect to the server and use its tools to search datasets, run NQL queries, manage materialized views, and more on behalf of an authenticated user.

**Related:** [Data Collaboration MCP Server](/concepts/mcp/overview), [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant), [MCP Server Reference](/reference/integrations/mcp-server)

### Materialized view

A pre-computed query result stored as a dataset. Unlike executing a query each time, a materialized view stores results physically for faster access. Views can be refreshed on a schedule or manually to incorporate changes in underlying data.

**Related:** [Materialized Views](/concepts/nql/materialized-views), [Creating Materialized Views](/guides/nql/creating-materialized-views), [Materialized View Syntax](/nql/commands/create-materialized-view)

### Model Inference

A job type that enables AI-powered operations by running LLM inference within a customer's [data plane](#data-plane). Model Inference supports multiple models from Anthropic (Claude Haiku, Sonnet, Opus) and OpenAI (GPT-4.1, o4-mini) and guarantees [structured output](#structured-output-inference) via JSON Schema. Because inference runs within the data plane, data never leaves customer-controlled infrastructure—no external API calls are made to model providers.

**Related:** [Model Inference Overview](/concepts/model-inference/overview), [Running Model Inference](/guides/sdk/running-model-inference), [Supported Models](/reference/model-inference/supported-models)

***

## N

### Native Apps

Pre-built applications within the Narrative platform that perform specific data collaboration tasks. Native Apps provide ready-to-use functionality for common workflows without requiring custom development. Users can launch and configure Native Apps through the Web UI to accomplish tasks like data ingestion, transformation, and activation. The Snowflake Native App is a specific instance that operates as a customer-hosted [data plane](#data-plane) operator, executing queries directly within a customer's Snowflake environment.

**Related:** [Web UI](#web-ui)

### Nested loop join

A join algorithm that iterates through each row of one table and, for each row, scans the other table for matches. Less efficient than hash joins, nested loops are often the fallback when query conditions prevent hash join optimization, such as when OR operators appear in JOIN clauses.

**Related:** [Understanding JOIN Performance](/concepts/nql/join-performance)

### Narrative ID

A secure, pseudonymous identifier derived from a clear text identifier (such as an email or hashed email) through Narrative's secure encoding methodology. Unlike simple hashing, Narrative IDs are encoded per-partner, meaning the same identifier produces different Narrative IDs for different partners. This enables privacy-safe data collaboration without exposing underlying identifiers.

**Related:** [Narrative ID Concepts](/concepts/security/narrative-id), [Using Narrative ID](/guides/collaboration/using-narrative-id), [Pseudonymization](#pseudonymization)

### Normalized dataset

A dataset with active Rosetta Stone mappings, making its data queryable through the `narrative.rosetta_stone` table. Normalized datasets appear in the Normalized Datasets interface where you can evaluate mapping quality using AI-powered confidence scoring and review AI-generated mapping suggestions.

**Related:** [Rosetta Stone Overview](/concepts/rosetta-stone/overview), [Managing Evaluations](/guides/rosetta-stone/managing-evaluations)

### NQL (Narrative Query Language)

Narrative's proprietary query language designed specifically for data collaboration and normalization. NQL is an interpreted language—queries are parsed by the [control plane](#control-plane) and transpiled to native SQL dialects (Snowflake SQL, Spark SQL, etc.) depending on the target [data plane](#data-plane). This abstraction enables a single query to run against data in different database systems while enforcing permissions and integrating [Rosetta Stone](#rosetta-stone) normalization.

**Related:** [NQL Design Philosophy](/concepts/nql/design-philosophy), [Query Processing](/concepts/architecture/query-processing)

***

## O

### Operator (data plane)

A software component running within a customer-hosted [data plane](#data-plane) that bridges the [control plane](#control-plane) and the customer's data infrastructure. The operator polls the control plane's [job queue](#job-queue) for compiled queries, executes them against the local database engine (Snowflake, Spark, etc.), and reports results back. Because the operator runs in the customer's infrastructure, raw data never leaves the data plane.

**Related:** [Data Planes](/concepts/primitives/data-planes), [Query Processing](/concepts/architecture/query-processing)

***

## P

### PII (Personally Identifiable Information)

Data that can be used to identify a specific individual, either directly or in combination with other data. Common examples include email addresses, phone numbers, names, addresses, and government-issued IDs. Narrative requires that PII be pseudonymized (hashed) before upload to protect individual privacy.

**Related:** [Data Pseudonymization](/concepts/security/data-pseudonymization), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Pseudonymization

A data protection technique that replaces directly identifying information with artificial identifiers (pseudonyms). In Narrative's context, pseudonymization is achieved by hashing PII such as email addresses and phone numbers. Unlike anonymization, pseudonymized data can still be matched across datasets (same input produces same hash) while protecting individual privacy.

**Related:** [Data Pseudonymization](/concepts/security/data-pseudonymization), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Permission (API key)

A pair of an access level and a resource that defines what an API key can do. Access levels are Read (view and list) and Write (create, update, and delete). Resources represent functional platform areas such as Datasets, Connections, or Jobs.

**Related:** [API Key Permissions](/concepts/security/api-key-permissions), [Permissions Reference](/reference/security/permissions)

### Placeholder (query template)

A variable element in a [query template](#query-template) marked with `{{name}}` syntax that is replaced with a user-provided value at execution time. Placeholder types include literal values (strings, numbers, dates), columns, filters, and output fields. Each placeholder has metadata including name, type, description, and whether it is required.

**Related:** [Query Template Syntax](/nql/commands/query-templates), [Using Query Templates](/guides/nql/using-query-templates)

### Processing Fee

A service fee based on the volume of bytes processed per month during usage of Narrative's products and services. Processing includes ingestion, evaluation, transactions, forecasting, and order management. For pay-as-you-go customers, the rate is \$0.65/GB. Processing fees may be waived for activity directly supporting data transactions under certain conditions.

**Related:** [Pricing](/reference/billing/pricing), [Transfer Fee](#transfer-fee), [Committed-Usage](#committed-usage)

### Profile (app)

A configuration within a company account that stores credentials and settings for an external platform connection created through the app invite flow. Once created, profiles are usable like any other profile in the account.

**Related:** [App Invites](/concepts/apps/app-invites)

### Prompt Studio

A UI tool for transforming datasets into structured, fine-tuning-ready examples for AI models. Prompt Studio allows you to configure system, user, and assistant prompts with dynamic macros that pull values from dataset fields, NQL expressions, or literal text. Each row in the source dataset becomes a conversation-formatted training example.

**Related:** [Prompt Studio Interface](/reference/ui/prompt-studio), [Model Inference](/concepts/model-inference/overview)

***

## Q

### Query compilation

The process by which the [control plane](#control-plane) transforms an [NQL](#nql-narrative-query-language) query into executable SQL for a target [data plane](#data-plane). Query compilation includes parsing, permission validation, optimization, and [transpilation](#transpilation) to the target database's SQL dialect.

**Related:** [Query Processing](/concepts/architecture/query-processing), [Transpilation](#transpilation)

### Query template

A reusable NQL query pattern with configurable [placeholders](#placeholder-query-template) that users fill in at execution time. Templates enable users to run customized queries without writing NQL directly, making complex queries accessible to non-technical users while ensuring consistent query structure.

**Related:** [Query Templates](/concepts/nql/query-templates), [Using Query Templates](/guides/nql/using-query-templates), [Query Template Syntax](/nql/commands/query-templates)

***

## R

### Relationships

Connections between datasets that enable querying across multiple related data sources. Relationships allow organizations to join and analyze data across different datasets while respecting access controls.

### REST API

The programmatic interface for interacting with the Narrative platform. The REST API follows standard conventions including resource-oriented URLs, standard HTTP verbs (GET, POST, PUT, DELETE), JSON-encoded request and response bodies, and Bearer token authentication. All functionality available in the Web UI is also accessible via the API.

**Related:** [REST API Documentation](/api-reference/introduction), [SDKs](/reference/sdks), [API Keys](/account-settings/api-keys)

### Retention policy

A set of rules that governs how long data within a dataset is kept before automatic deletion. Retention policies help manage storage costs, comply with data governance requirements, and ensure data is only stored for as long as necessary. Narrative supports three policy classes: **Row TTL** (performs row-level hard deletes based on a timestamp column, supported on AWS and Snowflake data planes), **Snapshot TTL** (deletes old ingestion snapshots based on snapshot age), and **Table TTL** (drops the entire dataset when table age exceeds the interval). Policies use ISO 8601 durations (for example, `P90D` for 90 days) to specify retention intervals.

**Related:** [Dataset Retention Policies](/concepts/primitives/dataset-retention-policies), [Datasets](/concepts/primitives/datasets), [EXPIRE clause](/nql/commands/create-materialized-view#expire)

### Right to erasure

A data subject right under GDPR (also known as the "right to be forgotten") allowing individuals to request deletion of their personal data under certain circumstances, such as when the data is no longer necessary, when consent is withdrawn, or when processing was unlawful. Similar deletion rights exist under CCPA.

**Related:** [Data subject](#data-subject), [GDPR](/concepts/compliance/gdpr), [CCPA](/concepts/compliance/ccpa)

### Rosetta AI Assistant

An AI-powered assistant that interprets plain English instructions and executes actions within the Narrative platform. Rosetta (accessible at [rosett.st](https://rosett.st)) can help users accomplish data collaboration tasks including complex data operations like querying data. The assistant makes data collaboration more accessible by eliminating the need to learn query syntax or navigate complex interfaces.

**Related:** [NQL](#nql-narrative-query-language), [Web UI](#web-ui)

### Rosetta Stone

Narrative's schema normalization system that enables data collaboration across organizations with different data structures. Rosetta Stone uses two core primitives—attributes (standardized field definitions) and mappings (translations between source columns and attributes)—to automatically normalize data from multiple sources into a queryable common schema.

The system combines machine learning and human curation to create accurate mappings, handles schema changes transparently, and eliminates the need for custom ETL pipelines between partners.

**Related:** [Rosetta Stone Overview](/concepts/rosetta-stone/overview), [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works), [Mapping Schemas](/guides/rosetta-stone/mapping-schemas)

### Rosetta Stone access patterns

The three ways to scope Rosetta Stone queries in NQL, each controlling which data sources are included:

* **Global access** (`narrative.rosetta_stone`): Queries all normalized data from all datasets shared with you across all companies.
* **Company-scoped access** (`company_data._rosetta_stone` or `<company_slug>._rosetta_stone`): Queries normalized data from a specific company's datasets.
* **Dataset-scoped access** (`company_data.<dataset>._rosetta_stone`): Queries normalized data from a specific dataset, enabling combination with non-normalized columns.

**Related:** [NQL Syntax Reference](/nql/general/syntax#rosetta-stone-table-references), [Scoping Your Queries](/concepts/rosetta-stone/normalization-model#scoping-your-queries)

***

## S

### SAML 2.0

Security Assertion Markup Language 2.0—the industry standard protocol enabling secure authentication between an identity provider and Narrative. SAML 2.0 is used for enterprise single sign-on integrations.

**Related:** [SSO Configuration](/account-settings/sso), [SSO Concepts](/concepts/security/sso)

### Schema

The structural definition of a dataset that specifies its fields, data types, and validation rules. A schema acts as the blueprint that determines what data can be stored in a dataset and how it should be organized. Schemas ensure data consistency and enable Narrative to validate incoming records during ingestion.

**Related:** [Datasets](/concepts/primitives/datasets), [Managing Datasets](/guides/sdk/managing-datasets)

### Schema inference

The automated process by which Narrative analyzes uploaded data to identify column types and suggest appropriate Rosetta Stone attribute mappings. Schema inference uses pattern recognition and machine learning to propose mappings that can be accepted, modified, or rejected.

**Related:** [How Rosetta Stone Works](/concepts/rosetta-stone/how-it-works), [Mapping Schemas](/guides/rosetta-stone/mapping-schemas)

### Schema mapping

The process of defining how fields from one data schema correspond to fields in another schema or to a standard schema. Schema mapping is a key component of Narrative's Rosetta Stone functionality.

**Related:** [Rosetta Stone Overview](/concepts/rosetta-stone/overview), [Mapping Schemas](/guides/rosetta-stone/mapping-schemas)

### Schema normalization

The process of mapping different data schemas to a common standardized format, enabling organizations to work with data in consistent ways regardless of how it was originally structured.

**Related:** [The Normalization Model](/concepts/rosetta-stone/normalization-model), [Rosetta Stone Overview](/concepts/rosetta-stone/overview)

### SDK (Software Development Kit)

A collection of libraries, tools, and documentation that enables developers to integrate with the Narrative platform programmatically. The official TypeScript SDK (`@narrative.io/data-collaboration-sdk-ts`) provides type-safe access to all Narrative APIs, including dataset management, NQL query execution, and job tracking.

**Related:** [TypeScript SDK Reference](/reference/sdks/typescript), [SDK Quickstart](/getting-started/sdk-quickstart), [SDK Guides](/guides/sdk)

### Secret Sharing

A Tools feature that enables one-time-use encrypted sharing of sensitive information like API keys, passwords, and credentials through unique retrieval links.

**Related:** [Secret Sharing Guide](/guides/tools/secret-sharing)

### Schema preset

A pre-defined collection of Rosetta Stone attributes designed for common use cases such as demographic data, marketing events, or identity resolution. Organizations can use public presets to accelerate mapping or create private presets to standardize across teams.

**Related:** [The Normalization Model](/concepts/rosetta-stone/normalization-model)

### SHA-1

Secure Hash Algorithm 1—a hash function that produces a 160-bit (40-character hexadecimal) hash value. More secure than MD5 but now considered cryptographically deprecated for security purposes. Still acceptable for identifier matching in data collaboration. Narrative supports SHA-1 alongside MD5 and SHA-256 for hashing PII.

**Related:** [Hash function](#hash-function), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Snapshot

A point-in-time collection of files that were ingested together into a dataset. When you upload data, the ingestion process creates a new snapshot containing the validated records. [Snapshot TTL retention policies](/concepts/primitives/dataset-retention-policies#snapshot-ttl) evaluate data based on snapshot age—the time since the snapshot was created—to determine when snapshots should be automatically deleted.

**Related:** [Datasets](/concepts/primitives/datasets), [Dataset Retention Policies](/concepts/primitives/dataset-retention-policies)

### SHA-256

Secure Hash Algorithm 256-bit—a member of the SHA-2 family that produces a 256-bit (64-character hexadecimal) hash value. Currently considered cryptographically secure and is the recommended algorithm for new implementations. Narrative supports SHA-256 as the preferred hashing algorithm alongside MD5 and SHA-1.

**Related:** [Hash function](#hash-function), [Hashing PII for Upload](/guides/ingestion/hashing-pii)

### Skill

A pre-built workflow that orchestrates the [Data Collaboration MCP Server](/concepts/mcp/overview) to accomplish a specific task — drafting an NQL query, generating Rosetta Stone mappings, building an identity graph, and more. Skills package opinionated procedures on top of the MCP tool catalog. A skill is harness-agnostic — a portable instruction file rather than logic tied to one AI assistant; Narrative distributes its skills as Claude Code plugins from the public `narrative-io/narrative-skills-marketplace` repository.

**Related:** [Skills](/concepts/mcp/skills), [Installing Skills in Claude Code](/guides/mcp/installing-skills)

### Skill plugin

A versioned bundle of one or more Skills published to the Claude Code marketplace. Narrative ships two plugins: `narrative-common` for general authoring, querying, and Rosetta Stone work, and `narrative-identity` for identity resolution work. Installing a plugin makes every Skill it contains available in the current Claude Code session.

**Related:** [Skills](/concepts/mcp/skills), [Installing Skills in Claude Code](/guides/mcp/installing-skills)

### SSO (Single Sign-On)

An authentication method that allows users to access Narrative using their organization's existing credentials and identity provider. SSO eliminates the need for separate Narrative-specific passwords and enables centralized access management.

**Related:** [SSO Configuration](/account-settings/sso), [SSO Concepts](/concepts/security/sso)

### Structured output (inference)

A [Model Inference](#model-inference) response that conforms to a predefined JSON Schema. Structured output ensures the LLM returns data in a predictable, typed format suitable for programmatic consumption. The schema is specified in the `output_format_schema` field of the [inference configuration](#inference-configuration) and can define required fields, data types, enums, and nested structures.

**Related:** [Structured Output Concepts](/concepts/model-inference/structured-output), [JSON Schema Reference](/reference/model-inference/json-schema-reference)

***

## T

### Transaction Services Fee

A service fee calculated as a percentage of the Licensed Data Fees transacted on Narrative's platform. For pay-as-you-go customers, the rate is 25% of Licensed Data Fees. This fee applies to data you transact, source, or facilitate on the platform or via Narrative Services, including via any Connector App.

**Related:** [Pricing](/reference/billing/pricing), [Licensed Data Fees](#licensed-data-fees)

### Transfer Fee

A service fee based on the volume of bytes transferred out of your Narrative account per month, also known as an egress fee. For pay-as-you-go customers, the rate is \$1.50/GB. Egress transfers to client-owned locations within AWS Region "us-east-1" (US East, N. Virginia) do not incur Transfer Fees.

**Related:** [Pricing](/reference/billing/pricing), [Processing Fee](#processing-fee), [Committed-Usage](#committed-usage)

### Transpilation

The process of converting [NQL](#nql-narrative-query-language) code to equivalent native SQL code for a specific database system. The [control plane](#control-plane) transpiles NQL to Snowflake SQL, Spark SQL, or other dialects depending on where the target [data plane](#data-plane) stores its data. Transpilation is distinct from compilation in that it produces human-readable code in another query language rather than machine code.

**Related:** [Query Processing](/concepts/architecture/query-processing), [Query compilation](#query-compilation)

### Transformation expression

An NQL expression used within a Rosetta Stone mapping to convert source data to the target attribute format. Transformation expressions can include type conversions, string manipulation, conditional logic (CASE statements), and null handling. They enable mappings to normalize diverse source formats into consistent attribute values.

**Related:** [Mapping Schemas](/guides/rosetta-stone/mapping-schemas), [Transformation Functions](/reference/rosetta-stone/transformation-functions), [Edge Cases](/guides/rosetta-stone/edge-cases)

### Task output

The structured JSON result produced by a workflow task after it executes. Task output contains metadata such as dataset IDs, snapshot IDs, and row counts. Output can be captured into the [workflow context](#workflow-context) via `export` and injected into subsequent tasks via [variable expressions](#variable-expression).

**Related:** [Task Reference](/reference/workflows/tasks), [Specification Syntax](/reference/workflows/specification-syntax#task-output)

***

## U

### UID2 (Unified ID 2.0)

A privacy-focused user identifier developed by the Identity Consortium (including The Trade Desk) that enables cross-platform data matching without compromising user privacy. UID2 is an open-source, standalone solution that can be generated from hashed emails or phone numbers. Narrative supports automatic UID2 generation from uploaded data.

**Related:** [UID2](/concepts/identifiers/uid2)

### UNNEST

An NQL function that expands an array column into multiple rows, with one row generated for each element in the array. UNNEST is commonly used to flatten multi-key matching scenarios into single-key joins that the query planner can optimize efficiently.

**Related:** [NQL Functions Reference](/nql/functions/index), [Avoid OR in JOIN Clauses](/guides/nql/query-optimization/avoid-or-in-join)

### Unix time

The number of seconds (or milliseconds) elapsed since January 1st, 1970 at 00:00:00 UTC (the Unix epoch). Narrative's Data Streaming Platform uses Unix time in milliseconds for all timestamp fields, providing millisecond-level precision for data collaboration.

**Related:** [Unix Time](/concepts/data-formats/unix-time)

### Upsert

A database operation that inserts a new record if it does not exist, or updates the existing record if it does. The term combines "update" and "insert." In NQL, upsert behavior is achieved using the `MERGE ON` clause within `CREATE MATERIALIZED VIEW` statements.

**Related:** [Incremental Upserts with MERGE ON](/guides/nql/incremental-upserts), [Materialized View Syntax](/nql/commands/create-materialized-view)

### URI (Uniform Resource Identifier)

A string that represents a particular resource. Common resources include web pages (full URLs like `http://example.org/page`) and mobile applications (iOS app IDs like `553834731` or Android package names like `com.example.app`). In Narrative, URI is a required attribute in the Digital Consumption data type, denoting where a digital behavior took place.

### User agent

A string from a user's device that identifies the operating system, browser, carrier, and hardware information. User agent strings are collected in digital consumption scenarios and help identify device characteristics. Example: `Mozilla/5.0 (iPhone; CPU iPhone OS 13_3 like Mac OS X) AppleWebKit/605.1.15`.

***

## V

### View dataset

A dataset that stores only an NQL query definition rather than physical query results. When another query references a view dataset, the stored NQL is inlined and re-evaluated at query time—similar to a database view. View datasets cannot have access rules, connectors, or features like `MERGE ON` or `PARTITIONED_BY` applied to them. Create a view dataset by passing `create_as_view: true` when executing a query.

**Related:** [Materialized Views — View Datasets](/concepts/nql/materialized-views#view-datasets), [Executing NQL Queries](/guides/sdk/executing-queries#creating-a-view-dataset)

### Variable expression

A `${…}` placeholder in workflow task parameters that is evaluated before the task executes. Variable expressions use jq syntax and can reference the previous task's output (`.`) or the accumulated workflow context (`$context`). A pure expression preserves the JSON type of the result; a string-interpolated expression (e.g., `${"text \(.value)")}`) always produces a string.

**Related:** [Specification Syntax](/reference/workflows/specification-syntax#variable-expressions), [Task output](#task-output), [Workflow context](#workflow-context)

***

## W

### Webhook subscription

A registration that tells the Narrative platform to send an HTTP POST notification to a specified URL when [job](#job-queue) state changes occur. Subscriptions can filter by job IDs, job types, and job states, so you receive only the events you care about. Each subscription includes a shared secret for verifying that incoming requests are authentic.

**Related:** [Webhook Concepts](/concepts/webhooks/overview), [Subscribing to Notifications](/guides/webhooks/subscribing-to-notifications), [Webhook Event Reference](/reference/webhooks/event-reference)

### Web UI

The browser-based interface for the Narrative data collaboration platform, accessible at [app.narrative.io](https://app.narrative.io/platform/). The Web UI provides visual access to all platform functionality including dataset management, access rule configuration, NQL query execution, and Native App deployment. Organized like a file explorer, the interface enables users to accomplish data collaboration tasks without writing code.

**Related:** [Native Apps](#native-apps), [REST API](#rest-api)

### Workflow context

An accumulator object (`$context`) that threads through every task in a workflow run. It starts as an empty object `{}` and grows as tasks use `export.as` to merge their output into it. Subsequent tasks can read values from `$context` using variable expressions (`${…}`), enabling structured data passing across multiple workflow steps.

**Related:** [Specification Syntax](/reference/workflows/specification-syntax#export), [Task output](#task-output), [Variable expression](#variable-expression)


# Reference
Source: https://docs.narrative.io/reference/index

Complete reference documentation for NQL syntax, UI components, and terminology

Reference documentation provides detailed, accurate information for looking things up. Use this section when you need specifics about syntax, available functions, or UI components.

## NQL Reference

Complete documentation for Narrative Query Language is available in the dedicated NQL tab.

<Card title="NQL Reference" icon="terminal" href="/nql/index">
  SQL commands, data types, functions, operators, and general reference
</Card>

## Workflow Reference

<CardGroup>
  <Card title="Specification Syntax" icon="list-check" href="/reference/workflows/specification-syntax">
    YAML specification format and supported tasks
  </Card>

  <Card title="Task Reference" icon="rectangle-terminal" href="/reference/workflows/tasks">
    Supported tasks and their parameters
  </Card>
</CardGroup>

## Webhook Reference

<CardGroup>
  <Card title="Webhook Event Reference" icon="list-check" href="/reference/webhooks/event-reference">
    Subscription fields, event payloads, and supported enum values
  </Card>
</CardGroup>

## Architecture Reference

<CardGroup>
  <Card title="Job Types" icon="list-check" href="/reference/architecture/job-types">
    Job types coordinated between control plane and data planes
  </Card>

  <Card title="Dataset Statistics" icon="chart-column" href="/reference/architecture/dataset-statistics">
    Statistics reference, configuration schema, and validation errors
  </Card>
</CardGroup>

## Security Reference

<Card title="Permissions" icon="lock-keyhole" href="/reference/security/permissions">
  All API key permission resources and their available access levels
</Card>

## Integrations

<Card title="Data Collection Endpoint" icon="cloud-arrow-up" href="/reference/integrations/data-collection-endpoint">
  Reference for the HTTPS-based data collection endpoint for identifiers and custom data
</Card>

## Glossary

<Card title="Glossary" icon="book-bookmark" href="/reference/glossary">
  Definitions for terms and concepts used throughout Narrative I/O
</Card>

## Billing

<Card title="Pricing" icon="credit-card" href="/reference/billing/pricing">
  Service fees, pricing models, and usage calculation details
</Card>

## Related resources

<CardGroup>
  <Card title="API Reference" icon="code" href="/api-reference">
    Full REST API documentation with endpoints and examples
  </Card>

  <Card title="Concepts" icon="lightbulb" href="/concepts">
    Understand why things work the way they do
  </Card>
</CardGroup>


# Data Collection Endpoint
Source: https://docs.narrative.io/reference/integrations/data-collection-endpoint

Reference for the HTTPS-based data collection endpoint used to ingest identifiers and custom data into Narrative

The Narrative data collection endpoint is an HTTPS-based integration point that enables partners to send identifiers and arbitrary data to Narrative by passing key-value pairs as query parameters. Data sent to this endpoint is ingested into a Narrative dataset.

## Overview

The data collection endpoint supports two primary use cases:

* **Cookie syncing:** Share browser-based identifiers between domains to create a shared understanding of user identity across separate cookie namespaces.
* **Custom data collection:** Send arbitrary key-value pairs alongside identifiers, which are ingested into a Narrative dataset for downstream use.

Both use cases work through the same endpoint and share the same base parameters.

## Endpoint

```
//io.narrative.io/?
```

## Query parameters

### Required parameters

| Parameter   | Required | Description                                                                                                                                    |
| ----------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `companyId` | Yes      | Unique ID assigned to the partner by Narrative                                                                                                 |
| `id`        | Yes      | Type and value of the identifier, formatted as `partner_id_type:partner_id_value`. The `partner_id_type` is assigned to partners by Narrative. |

### Optional parameters

| Parameter | Required | Description                                                                                                 |
| --------- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `red`     | No       | URL-encoded redirect URL. If provided, Narrative redirects back to this URL with the Narrative ID appended. |

### Custom parameters

Any additional query parameters beyond `companyId`, `id`, and `red` are ingested as custom key-value pairs into the partner's Narrative dataset. This allows partners to enrich each request with arbitrary metadata without requiring changes to the endpoint schema.

Custom parameter values are ingested as strings. Use descriptive, consistent key names across requests.

## Macros

When using the redirect parameter, you can include macros that Narrative replaces with actual values:

| Macro                   | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `${narrative.id.value}` | Replaced with the Narrative cookie identifier on redirect |

## Examples

### Basic cookie sync

A partner called BrandA wants to sync their customer identifier with Narrative.

**Given:**

* Company ID: `789`
* Customer ID: `123456789`
* Partner ID type (assigned by Narrative): `brandA_id`

**Template:**

```
//io.narrative.io/?companyId=XXX&id=partner_id_type:[[partner_id_value]]
```

**Implemented:**

```
//io.narrative.io/?companyId=789&id=brandA_id:123456789
```

### Cookie sync with redirect

To receive the Narrative identifier back, include a URL-encoded redirect URL with the macro:

```
//io.narrative.io/?companyId=789&id=brandA_id:123456789&red=https%3A%2F%2Fexample.com%2F%3Fnarrative_id%3D%24%7Bnarrative.id.value%7D
```

When decoded, the redirect URL is:

```
https://example.com/?narrative_id=${narrative.id.value}
```

Narrative replaces `${narrative.id.value}` with the actual Narrative cookie identifier before redirecting.

### Custom data collection

Partners can include additional query parameters to send custom data alongside the identifier. These key-value pairs are ingested into the partner's Narrative dataset.

**Given:**

* Company ID: `789`
* Customer ID: `123456789`
* Partner ID type: `brandA_id`
* Custom data: page category, campaign ID

**Implemented:**

```
//io.narrative.io/?companyId=789&id=brandA_id:123456789&page_category=electronics&campaign_id=summer2025
```

In this example, `page_category` and `campaign_id` are ingested as custom attributes in the partner's dataset alongside the standard identifier data.

## Related content

<CardGroup>
  <Card title="Glossary" icon="book-bookmark" href="/reference/glossary">
    Definitions for cookie syncing, data collection, and other terms
  </Card>
</CardGroup>


# Agent Feedback MCP Server
Source: https://docs.narrative.io/reference/integrations/mcp-agent-feedback

Public MCP server agents use to submit structured product feedback and bug reports

The Narrative Agent Feedback MCP server lets an AI agent file structured product feedback and bug reports without leaving the conversation. When a skill or workflow hits a rough edge, the agent can capture what happened and send it straight to the Narrative team. It pairs with the `narrative-common` and `narrative-identity` skill plugins, which call it to close the loop on agent-reported issues.

It is a public server with no authentication, so any connected MCP client can submit feedback.

For background on the Model Context Protocol, see [Data Collaboration MCP Server overview](/concepts/mcp/overview).

## Server URL

```
https://narrative.support/mcp
```

| Property       | Value                |
| -------------- | -------------------- |
| Transport      | Streamable HTTP      |
| Hosting        | Managed by Narrative |
| Authentication | None                 |
| Pricing        | Free                 |

## Tools

### `submit_feedback`

Submit structured product feedback or a bug report from an agent session.

| Parameter  | Type   | Required | Description                                                                                            |
| ---------- | ------ | -------- | ------------------------------------------------------------------------------------------------------ |
| `summary`  | string | Yes      | Short summary of the feedback or bug                                                                   |
| `details`  | string | Yes      | Full description, including reproduction steps, expected vs. actual behavior, and any relevant context |
| `category` | string | No       | Feedback category (e.g., `bug`, `feature_request`, `documentation`)                                    |

The server captures the agent's session context (active skill, model, and any user-provided tags) alongside the submitted message.

## Setup

Add the server to your MCP client the same way you would any remote HTTP MCP server. Use the URL above and skip the OAuth step.

For example, in Claude Code:

```bash theme={null}
claude mcp add --transport http narrative-agent-feedback https://narrative.support/mcp
```

Or in a JSON config:

```json theme={null}
{
  "mcpServers": {
    "narrative-agent-feedback": {
      "type": "http",
      "url": "https://narrative.support/mcp"
    }
  }
}
```

## Related

* [Data Collaboration MCP Server](/reference/integrations/mcp-server) — agent-facing access to the Narrative platform
* [Knowledge Base MCP Server](/reference/integrations/mcp-knowledge-base) — searchable Narrative documentation for agents
* [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant) — client setup walkthrough


# Knowledge Base MCP Server
Source: https://docs.narrative.io/reference/integrations/mcp-knowledge-base

Public MCP server that gives an agent grounded access to the Narrative documentation

The Narrative Knowledge Base MCP server lets an AI agent search the Narrative documentation, guides, API reference, and NQL syntax — and answer questions with citations to the source material. It is a public, read-only server with no authentication, so you can add it to any MCP client alongside the [Data Collaboration MCP server](/reference/integrations/mcp-server).

For background on the Model Context Protocol, see [Data Collaboration MCP Server overview](/concepts/mcp/overview).

## Server URL

```
https://docs.narrative.io/mcp
```

| Property       | Value                |
| -------------- | -------------------- |
| Transport      | Streamable HTTP      |
| Hosting        | Managed by Narrative |
| Authentication | None                 |
| Pricing        | Free                 |

## When to use it

Pair this server with the [Data Collaboration MCP server](/reference/integrations/mcp-server) so your agent can both **act** on your data and **explain** how the platform works. Common patterns:

* The agent hits an unfamiliar NQL function and queries the docs to find correct syntax before retrying.
* A user asks "how do access rules work?" — the agent answers with citations instead of guessing.
* The agent is drafting a workflow and looks up task-specific reference material on the fly.

## Tools

### `search_narrative_knowledge_base`

Full-text search across the Narrative documentation and guides.

| Parameter | Type   | Required | Description |
| --------- | ------ | -------- | ----------- |
| `query`   | string | Yes      | Search term |

Returns matching documentation entries with titles, snippets, and source URLs.

### `query_docs`

Ask a natural-language question and get an answer grounded in the docs.

| Parameter | Type   | Required | Description               |
| --------- | ------ | -------- | ------------------------- |
| `query`   | string | Yes      | Natural-language question |

Returns a synthesized answer with citations back to the docs pages that informed it.

## Setup

Add the server to your MCP client the same way you would the Data Collaboration server — see [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant) for client-specific instructions. Use the URL above and skip the OAuth step (no authentication is required).

For example, in Claude Code:

```bash theme={null}
claude mcp add --transport http narrative-knowledge-base https://docs.narrative.io/mcp
```

Or in a JSON config:

```json theme={null}
{
  "mcpServers": {
    "narrative-knowledge-base": {
      "type": "http",
      "url": "https://docs.narrative.io/mcp"
    }
  }
}
```

## Related

* [Data Collaboration MCP Server](/reference/integrations/mcp-server) — the OAuth-authenticated server that lets agents act on your data
* [Agent Feedback MCP Server](/reference/integrations/mcp-agent-feedback) — the server agents use to file structured product feedback
* [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant) — client setup walkthrough


# Data Collaboration MCP Server
Source: https://docs.narrative.io/reference/integrations/mcp-server

Tool reference for the Narrative Model Context Protocol server

## Overview

The Narrative MCP server exposes platform operations as tools that AI assistants can discover and call. All tool names use the `narrative_` prefix. This page documents every available tool, its parameters, and usage patterns.

For setup instructions, see [Connecting to the Narrative MCP Server](/guides/mcp/connecting-an-ai-assistant). For background on how the server works, see [MCP Server for Agents](/concepts/mcp/overview).

## Server URL

```
https://mcp.narrative.io/mcp
```

## Context tools

Context tools manage your authenticated session and company scope.

### `narrative_context_get`

Returns the current session context: authenticated user, role, and active company (if one has been set).

| Parameter | Type | Required | Description                                                  |
| --------- | ---- | -------- | ------------------------------------------------------------ |
| *(none)*  |      |          | Returns the current user, role, and active-company selection |

Use this to confirm which company is active before running queries or operations. If no company is set, follow up with `narrative_context_set_company`.

### `narrative_context_get_companies`

Lists the companies the authenticated user belongs to.

| Parameter | Type | Required | Description                                |
| --------- | ---- | -------- | ------------------------------------------ |
| *(none)*  |      |          | Returns all companies for the current user |

Returns each company's id and name. Use this to discover which companies you can switch to.

### `narrative_context_set_company`

Switches the active company for the current session. All subsequent tool calls operate against the selected company until changed.

| Parameter    | Type    | Required | Description              |
| ------------ | ------- | -------- | ------------------------ |
| `company_id` | integer | Yes      | The company to switch to |

### `narrative_context_search_companies`

Searches companies by name using case-insensitive substring matching. Available to global administrators only.

| Parameter     | Type    | Required | Description                                |
| ------------- | ------- | -------- | ------------------------------------------ |
| `search_term` | string  | Yes      | Search term to match against company names |
| `per_page`    | integer | No       | Results per page (default: 50, max: 100)   |

Returns `{ id, name }` for each matching company.

## Dataset tools

Dataset tools let you search, describe, and inspect datasets.

### `narrative_datasets_search`

Finds datasets by fuzzy keyword match, tags, or data plane. At least one filter must be supplied — calling with no filters returns an error rather than the full corpus. Filters compose with `AND`.

| Parameter       | Type          | Required | Description                                                                                                                                                                                   |
| --------------- | ------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `search_term`   | string        | No       | Fuzzy-matched against dataset `name`, `display_name`, and `description`. Matches on `name` and `display_name` rank above `description`-only matches of equal strength. Does NOT match `tags`. |
| `tags`          | string\[]     | No       | Exact-match all-of filter (case-sensitive). A dataset is returned only if it carries every supplied tag.                                                                                      |
| `data_plane_id` | string (uuid) | No       | Restrict to a single data plane. Use `narrative_data_planes_list` to discover ids.                                                                                                            |
| `include`       | string\[]     | No       | Sections to include per match                                                                                                                                                                 |

Returns at most 50 ranked matches. If the response indicates more matches exist, narrow the filters and retry. For full metadata on a shortlisted dataset, call `narrative_datasets_describe` with its id.

### `narrative_datasets_describe`

Describes one or more datasets with configurable detail levels. Accepts multiple dataset ids in a single call and fetches them in parallel.

| Parameter     | Type       | Required | Description                                             |
| ------------- | ---------- | -------- | ------------------------------------------------------- |
| `dataset_ids` | integer\[] | Yes      | One or more dataset ids to describe                     |
| `include`     | string\[]  | No       | Sections to include (default: `["metadata", "schema"]`) |
| `format`      | string     | No       | Response format: `"markdown"` (default) or `"json"`     |

**Available `include` values:**

| Value                 | Description                                |
| --------------------- | ------------------------------------------ |
| `metadata`            | Name, id, description, owner               |
| `schema`              | Column types and attribute mappings        |
| `mappings`            | Rosetta Stone mappings on the dataset      |
| `sample`              | Sample rows                                |
| `stats`               | Dataset-level statistics (not per-column)  |
| `column_stats_config` | Which columns have statistics configured   |
| `retention_policy`    | Data retention settings                    |
| `nql`                 | Generated NQL snippet to query the dataset |

### `narrative_dataset_get_column_stats`

Returns per-column statistics for a single dataset.

| Parameter    | Type      | Required | Description                                               |
| ------------ | --------- | -------- | --------------------------------------------------------- |
| `dataset_id` | integer   | Yes      | The dataset to inspect                                    |
| `columns`    | string\[] | Yes      | Column names to retrieve stats for                        |
| `include`    | string\[] | No       | Stat types to include (default: `["basic_column_stats"]`) |
| `format`     | string    | No       | Response format: `"markdown"` (default) or `"json"`       |

**Available `include` values:**

| Value                | Description                                               |
| -------------------- | --------------------------------------------------------- |
| `basic_column_stats` | All statistics except histogram — safe for context budget |
| `histogram`          | Opt-in; can produce large responses on wide columns       |

### `narrative_dataset_set_column_stats_config`

Configures which columns have statistics collected on a dataset.

| Parameter    | Type    | Required | Description                     |
| ------------ | ------- | -------- | ------------------------------- |
| `dataset_id` | integer | Yes      | The dataset to configure        |
| `config`     | object  | Yes      | Column statistics configuration |

## Dataset operation tools

These tools trigger asynchronous operations on datasets. All return a job or operation id that you can poll for completion.

### `narrative_dataset_request_sample`

Enqueues a sampling job for a dataset.

| Parameter         | Type    | Required | Description                                                                  |
| ----------------- | ------- | -------- | ---------------------------------------------------------------------------- |
| `dataset_id`      | integer | Yes      | The dataset to sample                                                        |
| `compute_pool_id` | string  | No       | Target a specific compute pool; otherwise uses the dataset's configured pool |

Returns the job id. Poll completion with `narrative_jobs_describe`.

### `narrative_dataset_refresh_materialized_view`

Triggers a refresh of a materialized-view dataset.

| Parameter         | Type    | Required | Description                                                                              |
| ----------------- | ------- | -------- | ---------------------------------------------------------------------------------------- |
| `dataset_id`      | integer | Yes      | The materialized view dataset to refresh                                                 |
| `type`            | string  | Yes      | `"full"` (reprocess all snapshots) or `"incremental"` (skip already-processed snapshots) |
| `compute_pool_id` | string  | No       | Target a specific compute pool                                                           |

Returns the refresh job id. Poll completion with `narrative_jobs_describe`.

### `narrative_dataset_recalculate_statistics`

Enqueues a column-statistics recalculation across the dataset's snapshot range.

| Parameter       | Type    | Required | Description                                                                                     |
| --------------- | ------- | -------- | ----------------------------------------------------------------------------------------------- |
| `dataset_id`    | integer | Yes      | The dataset to recalculate stats for                                                            |
| `from_snapshot` | string  | No       | Start from a specific snapshot; otherwise recalculates all snapshots in the stats configuration |

Returns the recalculation id, queued snapshot count, and snapshot timestamp range.

## Attribute tools

Attribute tools let you explore and inspect [Rosetta Stone](/concepts/rosetta-stone/overview) attributes.

### `narrative_attributes_search`

Searches Rosetta Stone attributes by name.

| Parameter     | Type      | Required | Description                                            |
| ------------- | --------- | -------- | ------------------------------------------------------ |
| `search_term` | string    | Yes      | Text to search across attribute names and descriptions |
| `include`     | string\[] | No       | Sections to include (default: `["metadata"]`)          |
| `format`      | string    | No       | Response format: `"markdown"` (default) or `"json"`    |

**Available `include` values:**

| Value      | Description                                     |
| ---------- | ----------------------------------------------- |
| `metadata` | Id, name, display name, description, type, tags |
| `schema`   | Properties, required fields, enum values        |

### `narrative_attributes_describe`

Describes one or more attributes by id. Accepts multiple attribute ids in a single call.

| Parameter       | Type       | Required | Description                                             |
| --------------- | ---------- | -------- | ------------------------------------------------------- |
| `attribute_ids` | integer\[] | Yes      | One or more attribute ids to describe                   |
| `include`       | string\[]  | No       | Sections to include (default: `["metadata", "schema"]`) |
| `format`        | string     | No       | Response format: `"markdown"` (default) or `"json"`     |

Uses the same `include` values as `narrative_attributes_search`.

## Access rule tools

Access rule tools let you discover and inspect [access rules](/concepts/primitives/access-rules) — the permissions and pricing controls applied to datasets — that your company owns or has been granted.

### `narrative_access_rules_search`

Searches access rules the active company has access to. At least one filter must be provided; calling with no filters returns an error rather than dumping the full corpus.

| Parameter                 | Type          | Required | Description                                                                                                                                                                        |
| ------------------------- | ------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `search_term`             | string        | No       | Fuzzy-matched against name, display name, description, owning company name, and slug. Matches on name and display name rank above description / company matches of equal strength. |
| `exposed_attribute_names` | string\[]     | No       | Filter to rules that expose any of these [Rosetta Stone](/concepts/rosetta-stone/overview) attributes (discover names via `narrative_attributes_search`)                           |
| `tags`                    | string\[]     | No       | Filter to rules carrying any of these tags                                                                                                                                         |
| `dataset_ids`             | integer\[]    | No       | Filter to rules referencing any of these datasets                                                                                                                                  |
| `owned_only`              | boolean       | No       | Restrict to rules owned by the active company                                                                                                                                      |
| `shared_only`             | boolean       | No       | Restrict to rules shared with (but not owned by) the active company                                                                                                                |
| `can_be_used_in_nql_only` | boolean       | No       | Drops rules that cannot be referenced in NQL (default: `true`). Set to `false` to include those too.                                                                               |
| `data_plane_id`           | string (uuid) | No       | Restrict to rules on a single data plane. Use `narrative_data_planes_list` to discover ids.                                                                                        |
| `include`                 | string\[]     | No       | Sections to include per match (default: `["data_plane_id", "description", "mappings"]`)                                                                                            |

Filters compose with `AND`. Matches are ranked by fuzzy score with name-side fields weighted above description and company fields, and truncated with an explicit note when the cap fires; refine the filters if you see the truncation message.

**Available `include` values:**

| Value           | Description                             |
| --------------- | --------------------------------------- |
| `data_plane_id` | UUID of the data plane the rule runs on |
| `description`   | Rule description                        |
| `mappings`      | Attributes the rule exposes             |
| `tags`          | Tags attached to the rule               |

To reference a rule in NQL, use `<owning_company_slug>.<access_rule_name>`.

### `narrative_access_rules_describe`

Describes one or more access rules by id. Accepts multiple ids in a single call and fetches them in parallel.

| Parameter         | Type       | Required | Description                                               |
| ----------------- | ---------- | -------- | --------------------------------------------------------- |
| `access_rule_ids` | integer\[] | Yes      | One or more access rule ids to describe                   |
| `include`         | string\[]  | No       | Sections to include (default: `["metadata", "mappings"]`) |

**Available `include` values:**

| Value           | Description                                                                                                 |
| --------------- | ----------------------------------------------------------------------------------------------------------- |
| `metadata`      | Name, display name, description, owning company, data plane id, type, status, tags, dataset ids, timestamps |
| `mappings`      | Attributes the rule exposes                                                                                 |
| `nql`           | Generated NQL snippet for the rule                                                                          |
| `schema`        | Output schema as JSON                                                                                       |
| `collaborators` | Sharing and collaborator configuration                                                                      |
| `pricing`       | Price in CPM USD                                                                                            |

## Mapping tools

Mapping tools let you create [Rosetta Stone](/concepts/rosetta-stone/overview) mappings from datasets to attributes in the active company.

### `narrative_mapping_create`

Creates a private mapping from a dataset to an attribute in the currently selected company. The mapping defines how each of the attribute's properties is built from a row of the source dataset, using [NQL](/concepts/nql/design-philosophy) expressions over the dataset's columns.

| Parameter      | Type    | Required | Description                                                                          |
| -------------- | ------- | -------- | ------------------------------------------------------------------------------------ |
| `attribute_id` | integer | Yes      | ID of the target attribute being mapped to                                           |
| `dataset_id`   | integer | Yes      | ID of the source dataset the mapping draws columns from                              |
| `mapping`      | object  | Yes      | The mapping definition — either an `object_mapping` or a `value_mapping` (see below) |

The `mapping` argument accepts one of two shapes:

* **`object_mapping`** — for attributes whose value is an object. Supply `type: "object_mapping"` and `property_mappings`, a list of `{ path, expression }` entries where `path` is a dotted path to an attribute property (e.g. `product.brand`) and `expression` is an NQL expression over dataset columns (e.g. `itemBrand`). Provide a mapping for each of the attribute's required properties. Quote string literals like `'SKU'`.
* **`value_mapping`** — for scalar attributes. Supply `type: "value_mapping"` and a single `expression` that produces the attribute's value.

Only `object_mapping` and `value_mapping` are supported through this tool; [`cached_mapping`](/reference/rosetta-stone/mapping-types#cached_mapping) is not. The dataset must be active and owned by the current company, and you must have map permission on the attribute. The created mapping's id, scope, source, status, and creation timestamp are returned in the response.

<Tip>
  Use `narrative_nql_validate` to test a candidate mapping expression before submitting it. Run `SELECT <expression> FROM company_data."<dataset_id>"` — if it compiles, the expression is valid against the dataset's schema.
</Tip>

See [mapping types](/reference/rosetta-stone/mapping-types) for the full schema and [Mapping schemas](/guides/rosetta-stone/mapping-schemas) for guidance on writing mappings by hand.

## NQL tools

NQL tools let you validate, execute, and track [NQL](/concepts/nql/design-philosophy) queries.

### `narrative_nql_validate`

Compile-checks an NQL query without executing it. Returns the compiled SQL on success, or validation errors pointing to the offending part of the query.

| Parameter | Type   | Required | Description               |
| --------- | ------ | -------- | ------------------------- |
| `query`   | string | Yes      | The NQL query to validate |

<Tip>
  Use `narrative_nql_validate` to test Rosetta Stone mapping expressions cheaply. Validate a candidate expression by running `SELECT expression FROM company_data."dataset_id"` through the tool — if it compiles, the expression is valid against the dataset's schema.
</Tip>

### `narrative_nql_run`

Submits an NQL query for asynchronous execution. Supports `SELECT` queries and `CREATE MATERIALIZED VIEW` statements.

| Parameter         | Type   | Required | Description                                                        |
| ----------------- | ------ | -------- | ------------------------------------------------------------------ |
| `query`           | string | Yes      | The NQL query to execute                                           |
| `data_plane_id`   | string | No       | Target a specific data plane; otherwise uses the company default   |
| `compute_pool_id` | string | No       | Target a specific compute pool; otherwise uses the company default |

Returns a job descriptor in `state: pending`. Poll it with `narrative_jobs_describe`.

## Job tools

Job tools let you discover and follow Narrative jobs — the asynchronous units of work spawned by `narrative_nql_run`, dataset operations, and workflow runs. Jobs are generic across types; the `type` field on each response distinguishes `nql-forecast`, `materialize-view`, `datasets_sample`, and others.

### `narrative_jobs_describe`

Describes one or more jobs by id in a single call. Use this to poll jobs through to completion — re-invoke until each job's `state` is `completed`, `failed`, or `cancelled`.

| Parameter | Type      | Required | Description                                                         |
| --------- | --------- | -------- | ------------------------------------------------------------------- |
| `job_ids` | string\[] | Yes      | One or more job ids (up to 50 per call)                             |
| `include` | string\[] | No       | Sections to include (default: `["failures", "metadata", "result"]`) |

**Available `include` values:**

| Value          | Description                                            |
| -------------- | ------------------------------------------------------ |
| `metadata`     | State, type, timestamps, data plane, compute pool      |
| `result`       | The job's result payload (shape depends on job `type`) |
| `failures`     | Any errors recorded against the job                    |
| `input`        | The original job input                                 |
| `compiled_sql` | Compiled SQL (NQL-derived jobs only)                   |
| `tags`         | Tags attached to the job                               |

### `narrative_jobs_search`

Searches jobs by tags, state, type, dataset, or data plane. At least one filter must be supplied — calling with no filters returns an error rather than dumping the full corpus.

| Parameter       | Type      | Required | Description                                                                                                               |
| --------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------- |
| `tags`          | string\[] | No       | Filter by tag (exact-match, case-sensitive). Useful patterns: `workflow_run_id=<run-uuid>`, `workflow_id=<workflow-uuid>` |
| `state`         | string    | No       | One of `pending`, `running`, `completed`, `pending_cancellation`, `cancelled`, `failed`                                   |
| `type`          | string    | No       | Job-type filter (free-form, e.g. `materialize-view`, `nql-forecast`, `datasets_sample`)                                   |
| `dataset_id`    | integer   | No       | Only return jobs whose input references this dataset                                                                      |
| `data_plane_id` | string    | No       | Data plane UUID                                                                                                           |
| `per_page`      | integer   | No       | Items per page (default: 25, max: 500)                                                                                    |
| `offset`        | integer   | No       | Number of records to skip. Use `offset = page_index * per_page` to walk pages                                             |
| `include`       | string\[] | No       | Sections to include per match (default: `["metadata"]`; also accepts `"tags"`)                                            |

Multiple filters compose with AND. Pass the returned ids to `narrative_jobs_describe` for full per-job detail.

## Data plane tools

Data plane tools let you discover the [data planes](/concepts/primitives/data-planes) your company has access to. The ids returned here are used by workflow tools and the optional `data_plane_id` argument on `narrative_nql_run`.

### `narrative_data_planes_list`

Lists every data plane available to the active company, including Narrative-hosted and any company-owned planes.

| Parameter | Type      | Required | Description                                              |
| --------- | --------- | -------- | -------------------------------------------------------- |
| `include` | string\[] | No       | Sections to include per result (default: `["metadata"]`) |

### `narrative_data_planes_describe`

Describes one or more data planes by id. Accepts multiple ids in a single call and fetches them in parallel.

| Parameter        | Type      | Required | Description                                   |
| ---------------- | --------- | -------- | --------------------------------------------- |
| `data_plane_ids` | string\[] | Yes      | One or more data plane UUIDs to describe      |
| `include`        | string\[] | No       | Sections to include (default: `["metadata"]`) |

**Available `include` values** (shared by both data plane tools):

| Value           | Description                                                                                                                                                                                                                  |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata`      | Display name, status, description, default compute pool, last heartbeat                                                                                                                                                      |
| `compute_pools` | Compute pools attached to the data plane, each with `status`, `size`, `idle_timeout_seconds`, and a derived `always_on` flag (`true`/`false` for AWS EMR pools, `n/a` for Snowflake warehouses that use auto-suspend/resume) |
| `platform`      | Platform-specific configuration as JSON                                                                                                                                                                                      |

Both tool descriptions carry usage guidance for connected AI assistants: compute pools with `always_on: true` skip cold-start cluster spin-up, making small always-on pools ideal for lightweight jobs like `narrative_dataset_request_sample`.

## Workflow tools

Workflow tools let you submit, inspect, and track runs of [workflows](/concepts/workflows/workflow-orchestration). The workflow YAML is passed as a free-form string — see the [workflow task reference](/reference/workflows/tasks) for the YAML shape.

### `narrative_workflows_create`

Submits a new workflow from a YAML specification.

| Parameter              | Type      | Required | Description                                                                                         |
| ---------------------- | --------- | -------- | --------------------------------------------------------------------------------------------------- |
| `specification`        | string    | Yes      | The workflow YAML as a plain string                                                                 |
| `data_plane_id`        | string    | Yes      | UUID of the data plane the workflow should run on. Use `narrative_data_planes_list` to discover ids |
| `trigger_immediately`  | boolean   | No       | Trigger a run as soon as the workflow is created (default: `false`)                                 |
| `schedule_immediately` | boolean   | No       | Activate the workflow's schedule as soon as it is created (default: `false`)                        |
| `tags`                 | string\[] | No       | Optional tags to attach to the workflow                                                             |

Returns the created workflow's id and metadata. When `trigger_immediately=true`, the response also includes a `run_id` you can poll with `narrative_workflow_runs_list`.

### `narrative_workflows_trigger`

Triggers a new run of an existing workflow. Returns immediately with the new run's id; poll the run's status via `narrative_workflow_runs_list`.

| Parameter     | Type   | Required | Description                     |
| ------------- | ------ | -------- | ------------------------------- |
| `workflow_id` | string | Yes      | UUID of the workflow to trigger |

<Note>
  Workflows created through this tool are automatically tagged with `created_by_mcp_server` so you can filter and audit agent-created workflows in the Narrative app. If the agent supplies the tag explicitly, it is not duplicated.
</Note>

### `narrative_workflows_describe`

Describes one or more workflows by id. Accepts multiple ids in a single call and fetches them in parallel.

| Parameter      | Type      | Required | Description                                   |
| -------------- | --------- | -------- | --------------------------------------------- |
| `workflow_ids` | string\[] | Yes      | One or more workflow UUIDs to describe        |
| `include`      | string\[] | No       | Sections to include (default: `["metadata"]`) |

**Available `include` values:**

| Value           | Description                                                |
| --------------- | ---------------------------------------------------------- |
| `metadata`      | Name, status, data plane, timestamps, tags, archived state |
| `specification` | Full workflow YAML — opt in, can be large                  |

<Tip>
  Avoid `include=["specification"]` when scanning many workflows. Request it only after you have narrowed to the ones you actually want to inspect.
</Tip>

### `narrative_workflow_runs_list`

Lists runs of a single workflow, in the order the upstream returns them (typically most recent first). Mirrors the upstream paging surface — opaque cursor pagination via `page_token`.

| Parameter     | Type    | Required | Description                                                                         |
| ------------- | ------- | -------- | ----------------------------------------------------------------------------------- |
| `workflow_id` | string  | Yes      | UUID of the workflow whose runs to list                                             |
| `per_page`    | integer | No       | Items per page (default: 10)                                                        |
| `page_token`  | string  | No       | Opaque cursor from a previous response's `next_page_token`. Omit for the first page |

To find the jobs spawned by a specific run, call `narrative_jobs_search` with `tags=["workflow_run_id=<run-uuid>"]`.

## Usage patterns

### Recommended NQL flow

The most effective pattern for running queries is **validate → run → poll**:

1. **Validate** the query with `narrative_nql_validate` to catch syntax errors before enqueuing a job.
2. **Run** the validated query with `narrative_nql_run` to submit it for execution.
3. **Poll** with `narrative_jobs_describe` until the job reaches a terminal state.

This avoids enqueuing jobs that would fail at compile time and keeps long-running executions out of the synchronous request path.

### Dataset inspection flow

To understand a dataset's structure and content:

1. **Search** with `narrative_datasets_search` to find datasets by name or description.
2. **Describe** with `narrative_datasets_describe` using `include=["metadata", "schema"]` for an overview, or add `"sample"` and `"stats"` for deeper inspection.
3. **Inspect columns** with `narrative_dataset_get_column_stats` for detailed per-column statistics.

### Access rule discovery flow

To find and inspect the access rules available to your company:

1. **Search** with `narrative_access_rules_search`, providing at least one filter (e.g. `search_term`, `exposed_attribute_names`, `tags`, `dataset_ids`).
2. **Describe** shortlisted rules with `narrative_access_rules_describe` using `include=["metadata", "mappings"]` for an overview, or add `"nql"`, `"schema"`, `"collaborators"`, or `"pricing"` for deeper detail.
3. **Reference in NQL** using `<owning_company_slug>.<access_rule_name>`.

### Workflow submission flow

To submit and track a workflow from an AI assistant:

1. **Discover** available data planes with `narrative_data_planes_list` to pick a `data_plane_id`.
2. **Create** the workflow with `narrative_workflows_create`, passing the YAML as `specification`. Set `trigger_immediately=true` to start a run right away, or call `narrative_workflows_trigger` later.
3. **Poll** runs with `narrative_workflow_runs_list` until the most recent run reaches a terminal state.
4. **Inspect job-level progress** by calling `narrative_jobs_search` with `tags=["workflow_run_id=<run-uuid>"]`, then `narrative_jobs_describe` on the returned ids.
5. **Describe** the workflow with `narrative_workflows_describe` to inspect its specification or current status.

### Multi-company sessions

If you belong to multiple companies:

1. **List** your companies with `narrative_context_get_companies`.
2. **Switch** with `narrative_context_set_company` to set the active company.
3. All subsequent tool calls operate against the selected company until you switch again.

## Naming conventions

* **Plural names** (`datasets_describe`, `attributes_describe`) accept multiple ids per call.
* **Singular names** (`dataset_get_column_stats`, `dataset_request_sample`) operate on a single entity.
* All tools use the `narrative_` prefix.

## Related content

<CardGroup>
  <Card title="Connecting to the Narrative MCP Server" icon="plug" href="/guides/mcp/connecting-an-ai-assistant">
    Set up your MCP client
  </Card>

  <Card title="Data Collaboration MCP Server" icon="lightbulb" href="/concepts/mcp/overview">
    How the MCP server works
  </Card>

  <Card title="NQL Design Philosophy" icon="terminal" href="/concepts/nql/design-philosophy">
    Understand the query language behind NQL tools
  </Card>

  <Card title="Rosetta Stone" icon="language" href="/concepts/rosetta-stone/overview">
    The schema normalization system behind attribute tools
  </Card>
</CardGroup>


# Snowflake Native App
Source: https://docs.narrative.io/reference/integrations/snowflake-native-app

Reference documentation for the Snowflake Native App data plane operator

## Overview

The Snowflake Native App installs as a customer-hosted [data plane](/concepts/primitives/data-planes) within your Snowflake environment. It runs the operator component that bridges Narrative's control plane with your Snowflake data—polling for jobs, executing NQL queries, and returning results, all without your data leaving your Snowflake account.

## Capabilities

* Register Snowflake tables and views as Narrative datasets
* Register one or more Snowflake warehouses as [compute pools](/concepts/primitives/compute-pools)
* Execute NQL queries directly in your Snowflake environment
* Participate in cross-data-plane data collaboration
* Keep data within your Snowflake security perimeter

## Requirements

| Requirement         | Details                                                      |
| ------------------- | ------------------------------------------------------------ |
| Snowflake role      | ACCOUNTADMIN or equivalent                                   |
| Narrative account   | Required ([create account](/getting-started/create-account)) |
| API token           | With specific permissions (see below)                        |
| Snowflake warehouse | For query execution                                          |
| Marketplace access  | Access to the Narrative listing on the Snowflake Marketplace |

## Privileges

The app requires the following Snowflake privileges:

| Privilege               | Purpose                                                      |
| ----------------------- | ------------------------------------------------------------ |
| `EXECUTE TASK`          | Schedule and run jobs on behalf of Narrative's control plane |
| `CREATE COMPUTE POOL`   | Provision compute resources for query execution              |
| `BIND SERVICE ENDPOINT` | Communicate with Narrative's control plane API               |

## Registering warehouses as compute pools

To register a Snowflake warehouse as a [compute pool](/concepts/primitives/compute-pools), grant the app usage on the warehouse:

```sql theme={null}
GRANT USAGE ON WAREHOUSE <warehouse_name> TO APPLICATION NARRATIVE_DATA_COLLABORATION;
```

You can register multiple warehouses to separate workloads across different compute pools.

## API token permissions

Your Narrative API token must include the following permissions:

| Permission      | Access level   |
| --------------- | -------------- |
| Compute Pools   | Read and Write |
| Data Planes     | Read and Write |
| Data Plane Logs | Read and Write |
| Datasets        | Read and Write |
| Jobs            | Read and Write |
| Models          | Read           |

See [API Keys](/account-settings/api-keys) for instructions on creating and managing tokens.

## Dataset naming conventions

When registering a Snowflake table or view as a Narrative dataset, the dataset name must follow these rules:

* Uppercase letters only (A–Z)
* Numbers (0–9)
* Underscores (`_`)

For example: `MY_DATASET_NAME`, `CUSTOMER_EVENTS_2024`.

## Installation

For step-by-step installation instructions, see [Install the Snowflake Native App](/guides/data-planes/snowflake-native-app-installation).

***

## Related content

<CardGroup>
  <Card title="Installation Guide" icon="download" href="/guides/data-planes/snowflake-native-app-installation">
    Step-by-step setup instructions
  </Card>

  <Card title="Data Planes" icon="database" href="/concepts/primitives/data-planes">
    Understand how data planes and operators work
  </Card>

  <Card title="Execution Context" icon="circle-play" href="/concepts/primitives/execution-context">
    Learn how queries are executed across data planes
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Manage API tokens and permissions
  </Card>
</CardGroup>


# JSON Schema Reference
Source: https://docs.narrative.io/reference/model-inference/json-schema-reference

Supported JSON Schema features for Model Inference structured output

Model Inference uses [JSON Schema](https://json-schema.org/) to define the structure of responses. This reference documents supported features and patterns.

## Supported draft version

Model Inference supports JSON Schema Draft 2020-12 features commonly used for structured output definition.

## Basic types

### String

```json theme={null}
{
  "type": "string"
}
```

With constraints:

```json theme={null}
{
  "type": "string",
  "minLength": 1,
  "maxLength": 1000
}
```

### Number

```json theme={null}
{
  "type": "number"
}
```

With constraints:

```json theme={null}
{
  "type": "number",
  "minimum": 0,
  "maximum": 1
}
```

### Integer

```json theme={null}
{
  "type": "integer",
  "minimum": 0
}
```

### Boolean

```json theme={null}
{
  "type": "boolean"
}
```

### Null

```json theme={null}
{
  "type": "null"
}
```

## Object type

Define structured objects with named properties:

```json theme={null}
{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" },
    "active": { "type": "boolean" }
  },
  "required": ["name", "age"]
}
```

### Required properties

Use the `required` array to specify mandatory fields:

```json theme={null}
{
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "title": { "type": "string" },
    "description": { "type": "string" }
  },
  "required": ["id", "title"]
}
```

### Additional properties

Control whether extra properties are allowed:

```json theme={null}
{
  "type": "object",
  "properties": {
    "name": { "type": "string" }
  },
  "additionalProperties": false
}
```

## Array type

### Simple arrays

```json theme={null}
{
  "type": "array",
  "items": { "type": "string" }
}
```

### Array constraints

```json theme={null}
{
  "type": "array",
  "items": { "type": "string" },
  "minItems": 1,
  "maxItems": 10
}
```

### Arrays of objects

```json theme={null}
{
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "value": { "type": "number" }
    },
    "required": ["name", "value"]
  }
}
```

## Enum constraint

Restrict values to a predefined set:

```json theme={null}
{
  "type": "string",
  "enum": ["low", "medium", "high"]
}
```

Enum with numbers:

```json theme={null}
{
  "type": "integer",
  "enum": [1, 2, 3, 4, 5]
}
```

## Numeric constraints

| Keyword            | Description                            |
| ------------------ | -------------------------------------- |
| `minimum`          | Value must be >= this number           |
| `maximum`          | Value must be \<= this number          |
| `exclusiveMinimum` | Value must be > this number            |
| `exclusiveMaximum` | Value must be \< this number           |
| `multipleOf`       | Value must be divisible by this number |

Example:

```json theme={null}
{
  "type": "number",
  "minimum": 0,
  "maximum": 100,
  "multipleOf": 0.01
}
```

## String constraints

| Keyword     | Description                         |
| ----------- | ----------------------------------- |
| `minLength` | Minimum string length               |
| `maxLength` | Maximum string length               |
| `pattern`   | Regex pattern the string must match |

Example:

```json theme={null}
{
  "type": "string",
  "minLength": 1,
  "maxLength": 500,
  "pattern": "^[A-Z][a-z]+$"
}
```

## Array constraints

| Keyword       | Description              |
| ------------- | ------------------------ |
| `minItems`    | Minimum array length     |
| `maxItems`    | Maximum array length     |
| `uniqueItems` | All items must be unique |

Example:

```json theme={null}
{
  "type": "array",
  "items": { "type": "string" },
  "minItems": 1,
  "maxItems": 5,
  "uniqueItems": true
}
```

## Descriptions

Add descriptions to help the model understand field purpose:

```json theme={null}
{
  "type": "object",
  "properties": {
    "sentiment": {
      "type": "string",
      "enum": ["positive", "negative", "neutral"],
      "description": "The overall emotional tone of the text"
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1,
      "description": "Confidence score where 0 is uncertain and 1 is certain"
    }
  },
  "required": ["sentiment", "confidence"]
}
```

## Nested structures

### Nested objects

```json theme={null}
{
  "type": "object",
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "created_at": { "type": "string" },
        "version": { "type": "integer" }
      },
      "required": ["created_at"]
    },
    "data": {
      "type": "object",
      "properties": {
        "items": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    }
  },
  "required": ["metadata"]
}
```

## Common patterns

### Classification result

```json theme={null}
{
  "type": "object",
  "properties": {
    "category": {
      "type": "string",
      "enum": ["A", "B", "C", "D"]
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    },
    "reasoning": {
      "type": "string"
    }
  },
  "required": ["category", "confidence"]
}
```

### Entity extraction

```json theme={null}
{
  "type": "object",
  "properties": {
    "entities": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "text": { "type": "string" },
          "type": {
            "type": "string",
            "enum": ["person", "organization", "location", "date"]
          },
          "start_index": { "type": "integer" },
          "end_index": { "type": "integer" }
        },
        "required": ["text", "type"]
      }
    }
  },
  "required": ["entities"]
}
```

### Summary with key points

```json theme={null}
{
  "type": "object",
  "properties": {
    "summary": {
      "type": "string",
      "maxLength": 500
    },
    "key_points": {
      "type": "array",
      "items": { "type": "string" },
      "minItems": 1,
      "maxItems": 5
    },
    "word_count": {
      "type": "integer",
      "minimum": 0
    }
  },
  "required": ["summary", "key_points"]
}
```

### CRON expression result

```json theme={null}
{
  "type": "object",
  "properties": {
    "cron_expression": {
      "type": "string",
      "pattern": "^[\\d\\*\\/\\-\\,]+\\s+[\\d\\*\\/\\-\\,]+\\s+[\\d\\*\\/\\-\\,\\?LW]+\\s+[\\d\\*\\/\\-\\,]+\\s+[\\d\\*\\/\\-\\,\\?L#]+$",
      "description": "Valid CRON expression"
    },
    "human_readable": {
      "type": "string",
      "description": "Human-readable description of the schedule"
    },
    "next_runs": {
      "type": "array",
      "items": { "type": "string" },
      "maxItems": 3,
      "description": "Next 3 scheduled execution times in ISO format"
    }
  },
  "required": ["cron_expression", "human_readable"]
}
```

## TypeScript mapping

| JSON Schema Type | TypeScript Type                  |
| ---------------- | -------------------------------- |
| `string`         | `string`                         |
| `number`         | `number`                         |
| `integer`        | `number`                         |
| `boolean`        | `boolean`                        |
| `null`           | `null`                           |
| `array`          | `T[]`                            |
| `object`         | `{ [key: string]: T }`           |
| `enum`           | Union type (`'a' \| 'b' \| 'c'`) |

## Related content

<CardGroup>
  <Card title="Using Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Practical guide to schema definition
  </Card>

  <Card title="Model Inference API" icon="code" href="/reference/sdks/typescript/model-inference-api">
    Complete API reference
  </Card>

  <Card title="Structured Output Concepts" icon="lightbulb" href="/concepts/model-inference/structured-output">
    Why structured output matters
  </Card>

  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Submit inference requests
  </Card>
</CardGroup>


# Supported Models
Source: https://docs.narrative.io/reference/model-inference/supported-models

Available models for Model Inference and their specifications

Model Inference supports models from Anthropic and OpenAI, all hosted within your [data plane](/concepts/primitives/data-planes). This reference lists available models and links to their official documentation.

## Available models

| Model ID                      | Provider  | Description                                     |
| ----------------------------- | --------- | ----------------------------------------------- |
| `anthropic.claude-haiku-4.5`  | Anthropic | Fast, cost-effective model for simple tasks     |
| `anthropic.claude-sonnet-4.5` | Anthropic | Balanced model for most use cases               |
| `anthropic.claude-sonnet-4.6` | Anthropic | Latest balanced model with improved reasoning   |
| `anthropic.claude-opus-4.5`   | Anthropic | Highly capable model for complex reasoning      |
| `anthropic.claude-opus-4.6`   | Anthropic | Latest most capable model for complex reasoning |
| `openai.gpt-4.1`              | OpenAI    | Advanced reasoning and analysis                 |
| `openai.o4-mini`              | OpenAI    | Fast, efficient model                           |
| `openai.gpt-oss-120b`         | OpenAI    | Open-source large language model                |

## Anthropic Claude models

Claude models excel at analysis, summarization, and following detailed instructions.

### Claude Haiku 4.5

**Model ID:** `anthropic.claude-haiku-4.5`

The fastest Claude model, optimized for high-throughput tasks where speed matters more than deep analysis.

**Best for:**

* Simple classification tasks
* Data extraction
* Quick summaries
* High-volume processing

**Official documentation:** [Claude Haiku](https://docs.anthropic.com/en/docs/about-claude/models#claude-3-5-haiku)

### Claude Sonnet 4.5

**Model ID:** `anthropic.claude-sonnet-4.5`

A balanced model offering strong performance across most tasks with good efficiency.

**Best for:**

* Content analysis
* Code generation
* Multi-step reasoning
* General-purpose tasks

**Official documentation:** [Claude Sonnet](https://docs.anthropic.com/en/docs/about-claude/models#claude-3-5-sonnet)

### Claude Sonnet 4.6

**Model ID:** `anthropic.claude-sonnet-4.6`

The latest Sonnet generation, building on 4.5 with improved reasoning and instruction following.

**Best for:**

* Content analysis
* Code generation
* Multi-step reasoning
* General-purpose tasks where the latest capabilities are preferred

**Official documentation:** [Claude Sonnet](https://docs.anthropic.com/en/docs/about-claude/models#claude-3-5-sonnet)

### Claude Opus 4.5

**Model ID:** `anthropic.claude-opus-4.5`

A highly capable Claude model for complex tasks requiring deep analysis and nuanced understanding.

**Best for:**

* Complex reasoning chains
* Nuanced analysis
* Research tasks
* High-stakes decisions

**Official documentation:** [Claude Opus](https://docs.anthropic.com/en/docs/about-claude/models#claude-3-opus)

### Claude Opus 4.6

**Model ID:** `anthropic.claude-opus-4.6`

The latest Opus generation, the most capable Claude model available for the most demanding inference workloads.

**Best for:**

* Complex reasoning chains
* Nuanced analysis
* Research tasks
* High-stakes decisions where the latest capabilities are preferred

**Official documentation:** [Claude Opus](https://docs.anthropic.com/en/docs/about-claude/models#claude-3-opus)

## OpenAI models

OpenAI models offer strong reasoning capabilities and broad knowledge.

### GPT-4.1

**Model ID:** `openai.gpt-4.1`

Advanced reasoning model with strong analytical capabilities.

**Best for:**

* Complex analysis
* Technical reasoning
* Multi-domain tasks

**Official documentation:** [GPT-4 Models](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo)

### o4-mini

**Model ID:** `openai.o4-mini`

Smaller, faster model optimized for efficiency.

**Best for:**

* Quick responses
* Simple tasks
* Cost-sensitive applications

**Official documentation:** [OpenAI Models](https://platform.openai.com/docs/models)

### GPT-oss-120b

**Model ID:** `openai.gpt-oss-120b`

Open-source large language model with broad capabilities.

**Best for:**

* General-purpose tasks
* Open-source requirements

## Choosing a model

| Consideration                | Recommendation          |
| ---------------------------- | ----------------------- |
| **Speed priority**           | Claude Haiku or o4-mini |
| **Balance of speed/quality** | Claude Sonnet           |
| **Complex reasoning**        | Claude Opus or GPT-4.1  |
| **Cost sensitivity**         | Claude Haiku or o4-mini |
| **High accuracy required**   | Claude Opus             |

For detailed guidance on model selection, see [Choosing the Right Model](/guides/sdk/choosing-inference-models).

## Usage example

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

// Use Claude Sonnet for balanced performance
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'user',
      content: [{ type: 'text', text: 'Analyze this dataset...' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        analysis: { type: 'string' }
      },
      required: ['analysis']
    }
  }
});
```

## Related content

<CardGroup>
  <Card title="Choosing Models" icon="scale-balanced" href="/guides/sdk/choosing-inference-models">
    Detailed model selection guide
  </Card>

  <Card title="Running Inference" icon="play" href="/guides/sdk/running-model-inference">
    Submit inference requests
  </Card>

  <Card title="Model Inference API" icon="code" href="/reference/sdks/typescript/model-inference-api">
    Complete API reference
  </Card>

  <Card title="Model Inference Overview" icon="brain" href="/concepts/model-inference/overview">
    How inference works
  </Card>
</CardGroup>


# Attribute Types
Source: https://docs.narrative.io/reference/rosetta-stone/attribute-types

Complete reference for Rosetta Stone attribute type definitions

This reference documents all attribute types available in Rosetta Stone, including their properties, validations, and usage examples.

## Attribute structure

All attributes share a common structure based on the platform's type system:

```json theme={null}
{
  "id": 123,
  "name": "attribute_name",
  "display_name": "Human Readable Name",
  "description": "Description of the attribute",
  "type": "string",
  "validations": ["LENGTH($this) >= 1", "LENGTH($this) <= 255"]
}
```

**Required properties:**

* `id` (number): Unique identifier for the attribute
* `name` (string): Machine-readable name
* `type` (string): One of the supported types

**Optional properties:**

* `display_name` (string): Human-readable name for UI display
* `description` (string): Explanation of the attribute's purpose
* `validations` (string\[]): Array of validation rules
* `metadata` (object): Additional metadata including co-occurrence data

## Primitive types

Primitive types represent basic data values.

| Type          | Description                   | NQL equivalent | Example values                  |
| ------------- | ----------------------------- | -------------- | ------------------------------- |
| `string`      | Variable-length text          | `VARCHAR`      | `"hello"`, `"user@example.com"` |
| `long`        | Whole numbers (64-bit signed) | `BIGINT`       | `42`, `-17`, `0`                |
| `double`      | Double-precision decimal      | `DOUBLE`       | `3.14159`, `-0.001`             |
| `boolean`     | True or false                 | `BOOLEAN`      | `true`, `false`                 |
| `timestamptz` | Date and time with timezone   | `TIMESTAMP`    | `2024-01-15T14:30:00Z`          |

### String

Variable-length Unicode text. Can optionally include an `enum` property to restrict allowed values.

**Basic definition:**

```json theme={null}
{
  "id": 1001,
  "name": "customer_name",
  "type": "string",
  "description": "Full name of the customer"
}
```

**With validations:**

```json theme={null}
{
  "id": 1002,
  "name": "email_sha256",
  "type": "string",
  "description": "SHA-256 hash of email address",
  "validations": ["LENGTH($this) = 64"]
}
```

**With enum (restricted values):**

```json theme={null}
{
  "id": 1003,
  "name": "hl7_gender",
  "type": "string",
  "enum": ["male", "female", "other", "unknown"],
  "description": "Gender using HL7 administrative gender codes"
}
```

When a string attribute has an `enum` property, only those values are allowed. Values are case-sensitive.

### Long

64-bit signed whole numbers. Range: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.

**Definition:**

```json theme={null}
{
  "id": 1004,
  "name": "age",
  "type": "long",
  "description": "Age in years",
  "validations": ["$this >= 0", "$this <= 150"]
}
```

### Double

Double-precision (64-bit) floating-point numbers.

**Definition:**

```json theme={null}
{
  "id": 1005,
  "name": "latitude",
  "type": "double",
  "description": "Geographic latitude",
  "validations": ["$this >= -90", "$this <= 90"]
}
```

### Boolean

True or false values.

**Definition:**

```json theme={null}
{
  "id": 1006,
  "name": "opted_in",
  "type": "boolean",
  "description": "Whether the user has opted in to marketing"
}
```

**Mapping from source data:**

```sql theme={null}
-- From string values
CASE UPPER(opt_in_col)
  WHEN 'Y' THEN true
  WHEN 'YES' THEN true
  WHEN 'TRUE' THEN true
  WHEN '1' THEN true
  ELSE false
END
```

### Timestamptz

Date and time with timezone, stored in UTC. Follows ISO 8601 format.

**Definition:**

```json theme={null}
{
  "id": 1007,
  "name": "event_timestamp",
  "type": "timestamptz",
  "description": "When the event occurred"
}
```

**Mapping from source data:**

```sql theme={null}
-- From various formats
TO_TIMESTAMP(date_col, 'YYYY-MM-DD HH24:MI:SS')
TO_TIMESTAMP(date_col, 'MM/DD/YYYY')
```

***

## Object type

Objects group related fields into a single composite value. Use `properties` to define the fields and `required` to specify which fields are mandatory.

**Definition:**

```json theme={null}
{
  "id": 2001,
  "name": "geo_coordinates",
  "type": "object",
  "description": "Geographic coordinates",
  "properties": {
    "latitude": {
      "type": "double"
    },
    "longitude": {
      "type": "double"
    },
    "accuracy_meters": {
      "type": "double"
    }
  },
  "required": ["latitude", "longitude"]
}
```

**Properties:**

* `properties`: Object mapping field names to field definitions
* `required`: Array of field names that must be present

**Nested objects:**

Objects can contain other objects:

```json theme={null}
{
  "id": 2002,
  "name": "address",
  "type": "object",
  "properties": {
    "street": { "type": "string" },
    "city": { "type": "string" },
    "state": { "type": "string" },
    "postal_code": { "type": "string" },
    "coordinates": {
      "type": "object",
      "properties": {
        "latitude": { "type": "double" },
        "longitude": { "type": "double" }
      }
    }
  },
  "required": ["street", "city"]
}
```

**Mapping example:**

```sql theme={null}
STRUCT(
  CAST(lat AS DOUBLE) AS latitude,
  CAST(lon AS DOUBLE) AS longitude,
  CAST(acc AS DOUBLE) AS accuracy_meters
)
```

**Accessing object fields in queries:**

```sql theme={null}
SELECT
  geo_coordinates.latitude,
  geo_coordinates.longitude
FROM narrative.rosetta_stone
WHERE geo_coordinates.accuracy_meters < 100
```

***

## Array type

Arrays contain multiple values of the same type.

**Definition:**

```json theme={null}
{
  "id": 3001,
  "name": "interest_categories",
  "type": "array",
  "items": {
    "type": "string"
  },
  "description": "List of interest categories"
}
```

**Properties:**

* `items`: Definition of the element type

**Array of objects:**

```json theme={null}
{
  "id": 3002,
  "name": "identifiers",
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "type": { "type": "string" },
      "value": { "type": "string" }
    }
  }
}
```

**Mapping examples:**

```sql theme={null}
-- From comma-separated string
SPLIT(interests, ',')

-- From JSON array
JSON_PARSE(json_array_col)

-- Building an array from multiple columns
ARRAY(email_sha256, phone_sha256)
```

**Querying arrays:**

```sql theme={null}
-- Check if array contains a value
SELECT unique_id, interest_categories
FROM narrative.rosetta_stone
WHERE ARRAY_CONTAINS(interest_categories, 'sports')

-- Unnest array for row-level operations
SELECT
  id,
  interest
FROM narrative.rosetta_stone
CROSS JOIN UNNEST(interest_categories) AS t(interest)
```

***

## Reference type

References link to other attribute definitions using their numeric ID. This enables reuse of standardized definitions.

**Definition:**

```json theme={null}
{
  "id": 4001,
  "name": "purchase_event",
  "type": "object",
  "properties": {
    "timestamp": {
      "$ref": 1007
    },
    "user": {
      "$ref": 5001
    },
    "amount": {
      "type": "double"
    }
  },
  "required": ["timestamp", "user", "amount"]
}
```

**Properties:**

* `$ref`: The numeric ID of the referenced attribute

**Behavior:**

* The field inherits the full type definition of the referenced attribute
* Validations from the referenced attribute apply
* Changes to the referenced attribute automatically propagate

**Use cases:**

* Reusing standardized definitions (like `event_timestamp`)
* Ensuring consistency across attributes
* Building complex schemas from well-defined components

***

## Validation expressions

Validations are NQL expressions that filter out invalid values. Use `$this` to reference the value being validated. These expressions are injected into compiled NQL queries to enforce data quality.

### Numeric validations

Comparison operators check value ranges:

```json theme={null}
{
  "id": 10,
  "name": "birth_year",
  "type": "long",
  "description": "The year that an entity was born",
  "validations": ["LENGTH($this) = 4", "$this >= 1900"]
}
```

| Expression                   | Description               |
| ---------------------------- | ------------------------- |
| `$this >= 0`                 | Value must be at least 0  |
| `$this <= 100`               | Value must be at most 100 |
| `$this > 0 AND $this < 1000` | Combined range check      |

### String length validations

Use `LENGTH()` to validate string length:

| Expression             | Description            |
| ---------------------- | ---------------------- |
| `LENGTH($this) >= 1`   | At least 1 character   |
| `LENGTH($this) <= 255` | At most 255 characters |
| `LENGTH($this) = 2`    | Exactly 2 characters   |

### Pattern matching

Use `LIKE` for pattern validation:

| Expression           | Description                   |
| -------------------- | ----------------------------- |
| `$this LIKE 'A%'`    | Starts with "A"               |
| `$this LIKE '%@%.%'` | Contains @ and . (email-like) |

### Examples

```json theme={null}
{
  "id": 5001,
  "name": "age",
  "type": "long",
  "validations": ["$this >= 0", "$this <= 150"]
}
```

```json theme={null}
{
  "id": 5002,
  "name": "country_code",
  "type": "string",
  "validations": ["LENGTH($this) = 2"]
}
```

***

## Join key attributes

Primitive attributes can be marked as join keys using the `is_join_key` property:

```json theme={null}
{
  "id": 6001,
  "name": "email_sha256",
  "type": "string",
  "is_join_key": true,
  "description": "SHA-256 hash of email for identity matching"
}
```

Join key attributes are optimized for use in JOIN operations across datasets.

***

## Related content

<CardGroup>
  <Card title="Transformation Functions" icon="function" href="/reference/rosetta-stone/transformation-functions">
    Functions for mapping transformations
  </Card>

  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand attributes and mappings
  </Card>

  <Card title="The Normalization Model" icon="layer-group" href="/concepts/rosetta-stone/normalization-model">
    Type system and validation concepts
  </Card>

  <Card title="Creating Normalized Attributes" icon="pen-ruler" href="/guides/rosetta-stone/creating-normalized-attributes">
    Best practices for designing attributes
  </Card>
</CardGroup>


# Attributes Interface
Source: https://docs.narrative.io/reference/rosetta-stone/attributes-ui

Reference for the Rosetta Stone Attributes listing and Create Attribute pages

This reference documents the UI elements and actions available in the Rosetta Stone Attributes interface, where you can browse all attributes and create custom attributes for your organization.

## Listing page

**Path:** Rosetta Stone → Attributes

The listing page displays all Rosetta Stone attributes available to your organization, including global platform attributes and custom attributes created by your company or shared with you.

### Search and sort

* **Search** — Filter attributes by name, display name, or description using the search field
* **Sort** — Click column headers or use the **Sort** control to reorder the list

### Columns

| Column           | Description                                                                   | Sortable |
| ---------------- | ----------------------------------------------------------------------------- | -------- |
| **ID**           | Unique numeric identifier for the attribute                                   | Yes      |
| **Name**         | Machine-readable `snake_case` name (e.g., `iso_3166_1_country`)               | Yes      |
| **Display Name** | Human-readable label shown in UIs (e.g., "Country Code (ISO 3166-1 alpha-2)") | Yes      |
| **Description**  | Brief explanation of what the attribute represents                            | No       |

Each attribute row also displays a type icon indicating its data type.

### Row actions

Click the three-dot menu on any attribute row to access additional actions.

***

## Create Attribute panel

**Access:** Click the **Create Attribute +** button in the top right of the listing page

The Create Attribute panel opens as a side drawer where you can define a new custom attribute. Fields marked with a sparkle icon support AI-assisted generation — Rosetta Stone can suggest a display name, name, and description based on what you enter.

### Basic Information

| Field            | Description                                                                           | Required |
| ---------------- | ------------------------------------------------------------------------------------- | -------- |
| **Display Name** | Human-readable label shown in UIs. AI-assisted.                                       | Yes      |
| **Name**         | Machine-readable identifier. Must be `snake_case` (e.g., `user_email`). AI-assisted.  | Yes      |
| **Description**  | Explains what the attribute represents and how it should be interpreted. AI-assisted. | Yes      |
| **Tags**         | Freeform labels for organizing attributes. Type a tag and press Enter to add it.      | No       |

<Tip>
  Start by entering a display name — Rosetta Stone can generate the `snake_case` name and a description for you automatically.
</Tip>

### Type

| Field    | Description                                                                                                        | Required |
| -------- | ------------------------------------------------------------------------------------------------------------------ | -------- |
| **Type** | The data type for the attribute. Options: `String`, `Long`, `Double`, `Boolean`, `Timestamptz`, `Object`, `Array`. | Yes      |

See [Attribute Types](/reference/rosetta-stone/attribute-types) for details on each type and when to use them.

### Type Configuration

| Field              | Description                                                                                                   |
| ------------------ | ------------------------------------------------------------------------------------------------------------- |
| **Allowed Values** | Optionally restrict the attribute to a specific set of values (enum). Type a value and press Enter to add it. |
| **Join Key**       | Toggle on to mark this attribute as usable for joining datasets. Only applicable to primitive types.          |

### Collaborators

Control who can see and use your custom attribute. Both fields default to **Only my company**, keeping the attribute private to your organization.

| Field           | Description                                      | Default         |
| --------------- | ------------------------------------------------ | --------------- |
| **View Access** | Who can see this attribute in the attribute list | Only my company |
| **Map Access**  | Who can create mappings to this attribute        | Only my company |

To share your attribute more broadly, change these to include other companies or make the attribute visible to everyone on the platform.

### Creating the attribute

Click **Create** at the bottom of the panel to save the new attribute. The attribute is immediately available for mapping in the [Normalized Datasets](/reference/rosetta-stone/normalized-datasets-ui) interface.

***

## Related content

<CardGroup>
  <Card title="Creating Normalized Attributes" icon="pen-ruler" href="/guides/rosetta-stone/creating-normalized-attributes">
    Best practices for designing attributes
  </Card>

  <Card title="Attribute Types Reference" icon="shapes" href="/reference/rosetta-stone/attribute-types">
    Complete type system documentation
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Map dataset columns to your attributes
  </Card>

  <Card title="How Rosetta Stone Works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Understand the normalization pipeline
  </Card>
</CardGroup>


# Mapping types
Source: https://docs.narrative.io/reference/rosetta-stone/mapping-types

API reference for the mapping types accepted by the Mappings endpoints

A mapping connects a source dataset to a Rosetta Stone attribute. The Mappings API accepts three mapping types, distinguished by the `type` field on the request body:

| Type             | Use when the target attribute is...                                                          |
| ---------------- | -------------------------------------------------------------------------------------------- |
| `value_mapping`  | A primitive (`array`, `boolean`, `double`, `long`, `string`, `timestamptz`)                  |
| `object_mapping` | An `object`, with one expression per property                                                |
| `cached_mapping` | Resolved by joining the source dataset against a precomputed **cache dataset** at query time |

Dependencies are computed server-side from each expression; clients do not provide them on create or update.

For background on how mappings fit into the normalization layer, see [How Rosetta Stone works](/concepts/rosetta-stone/how-it-works). For the UI and end-to-end workflow, see [Mapping schemas](/guides/rosetta-stone/mapping-schemas).

## value\_mapping

A single SQL expression that produces the target value from each row of the source dataset.

<ParamField type="string">
  Must be `"value_mapping"`.
</ParamField>

<ParamField type="string">
  A SQL expression evaluated against the source dataset.
</ParamField>

```json theme={null}
{
  "type": "value_mapping",
  "expression": "LOWER(email)"
}
```

## object\_mapping

A set of `property_mappings`, one per property of the target object attribute. Required properties of the attribute must be covered.

<ParamField type="string">
  Must be `"object_mapping"`.
</ParamField>

<ParamField type="array">
  One entry per target property. Each entry has:

  * `path` — dotted path into the target object (e.g. `product.id.value`)
  * `expression` — SQL expression evaluated against the source dataset
</ParamField>

```json theme={null}
{
  "type": "object_mapping",
  "property_mappings": [
    { "path": "product.brand",    "expression": "itemBrand" },
    { "path": "product.category", "expression": "itemCategory" },
    { "path": "product.id.type",  "expression": "'SKU'" },
    { "path": "product.id.value", "expression": "itemID" }
  ]
}
```

## cached\_mapping

A `cached_mapping` does not compute the attribute value from each source row. Instead, NQL joins the source dataset to a separate **cache dataset** at query time and reads the precomputed value out of the cache's `output` column.

Use a cached mapping when the transformation is expensive, depends on data outside the source dataset, or is produced by a workflow that writes its results into a dedicated cache dataset.

<ParamField type="string">
  Must be `"cached_mapping"`.
</ParamField>

<ParamField type="integer">
  ID of the cache dataset to look up against. The dataset must already exist and be owned by the same company.
</ParamField>

<ParamField type="array">
  One or more SQL expressions evaluated against the source dataset to build the join key. Multi-column keys are concatenated with a separator at query time, so the cache dataset's `input` column must use the same encoding.
</ParamField>

```json theme={null}
{
  "type": "cached_mapping",
  "cache_dataset_id": 789,
  "input_expressions": ["LOWER(email)"]
}
```

### Cache dataset requirements

The dataset referenced by `cache_dataset_id` must:

* Be owned by the same company creating the mapping.
* Expose an `input` column that holds the join key (matching the encoding produced by `input_expressions`).
* Expose an `output` column whose type matches the target attribute. NQL reads the attribute value directly from this column.

### Constraints

<Warning>
  Cached mappings are subject to additional validation. Requests that violate these rules are rejected at create time.
</Warning>

* **Company scope only.** `cached_mapping` is only accepted on the company-scoped create endpoint (`POST /mappings/companies/{company_id}`). Global mapping endpoints do not accept it.
* **Forbidden attributes.** Cached mappings cannot target attributes used by the opt-out pipeline: `data_privacy_request_identifier`, `unique_id`, or `identifier_relation`.
* **No materialized fields.** You cannot create a materialized field (`Attr` or `AttrSample`) on top of a cached mapping, and you cannot convert an existing mapping to a cached mapping while an active materialized field references it.
* **No DELETE filters.** `DELETE` statements whose `WHERE` clause references an attribute backed by a cached mapping return `UnsupportedQuery: DELETE on cached mappings not yet supported`.
* **Preview is empty.** `POST /mappings/test` returns `{ "records": [] }` for cached mappings — there is no per-source-row preview because the lookup is resolved at query time.

## Testing a mapping

`POST /mappings/test` accepts the same request shape as create and returns sampled, mapped output. For `value_mapping` and `object_mapping` the response includes one entry per sampled source row with both the original `input` and the transformed `output`. For `cached_mapping` the response is always `{ "records": [] }`.

## Related content

<Columns>
  <Card title="Mapping schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Walk through the UI and API for creating mappings.
  </Card>

  <Card title="Transformation functions" icon="function" href="/reference/rosetta-stone/transformation-functions">
    Functions available in mapping expressions.
  </Card>

  <Card title="How Rosetta Stone works" icon="gears" href="/concepts/rosetta-stone/how-it-works">
    Background on attributes, mappings, and normalization.
  </Card>

  <Card title="Attribute types" icon="shapes" href="/reference/rosetta-stone/attribute-types">
    The target shapes a mapping can produce.
  </Card>
</Columns>


# Normalized Datasets Interface
Source: https://docs.narrative.io/reference/rosetta-stone/normalized-datasets-ui

Reference for the Normalized Datasets listing and detail pages

This reference documents the UI elements, metrics, and actions available in the Normalized Datasets interface.

## Listing page

**Path:** Rosetta Stone → Normalized Datasets

The listing page displays all datasets that have Rosetta Stone mappings applied.

### Columns

| Column              | Description                                                       | Sortable |
| ------------------- | ----------------------------------------------------------------- | -------- |
| **Dataset name**    | Display name of the dataset; click to navigate to detail page     | Yes      |
| **Mappings**        | Count of non-system mappings on the dataset                       | Yes      |
| **Rows**            | Total record count in the dataset                                 | Yes      |
| **Avg. Confidence** | Average confidence score across all evaluated mappings            | Yes      |
| **Updated**         | Timestamp of the last modification to the dataset or its mappings | Yes      |

### Average confidence display

The average confidence column uses color coding:

| Color  | Score range   | Interpretation                      |
| ------ | ------------- | ----------------------------------- |
| Green  | 80-100%       | Mappings are generally high quality |
| Yellow | 50-79%        | Some mappings may need attention    |
| Red    | 0-49%         | Significant mapping issues likely   |
| Gray   | Not evaluated | No evaluation has been run          |

### Empty state

When no datasets have mappings, the listing displays a message with a link to the mapping documentation.

***

## Detail page

**Path:** Rosetta Stone → Normalized Datasets →&#x20;

The detail page shows evaluation results and mapping suggestions for a specific dataset. It has two tabs: **Evaluate** and **Normalize**.

***

## Evaluate tab

The Evaluate tab displays AI-powered quality scores for existing mappings.

### Confidence gradient bar

A horizontal bar at the top showing the distribution of mappings across confidence tiers:

| Section           | Color  | Represents               |
| ----------------- | ------ | ------------------------ |
| High confidence   | Green  | Mappings scoring 80-100% |
| Medium confidence | Yellow | Mappings scoring 50-79%  |
| Low confidence    | Red    | Mappings scoring 0-49%   |
| Not scored        | Gray   | Mappings without scores  |

The bar width of each section is proportional to the count of mappings in that tier.

### Confidence level cards

Four clickable cards showing mapping counts by confidence tier:

| Card           | Score range | Click action                                   |
| -------------- | ----------- | ---------------------------------------------- |
| **High**       | 80-100%     | Filter to show only high-confidence mappings   |
| **Medium**     | 50-79%      | Filter to show only medium-confidence mappings |
| **Low**        | 0-49%       | Filter to show only low-confidence mappings    |
| **Not Scored** | N/A         | Filter to show only unscored mappings          |

Click the active filter again or click **All** to clear the filter.

### Overview cards

Five cards displaying dataset statistics:

| Card                  | Description                                              |
| --------------------- | -------------------------------------------------------- |
| **Total Columns**     | Number of columns in the source dataset                  |
| **Attributes Mapped** | Count of distinct Rosetta Stone attributes with mappings |
| **System Mappings**   | Mappings auto-generated for internal columns             |
| **AI Mappings**       | Mappings created from accepted AI suggestions            |
| **User Mappings**     | Mappings created manually by users                       |

Click a mapping type card to filter the mapping list to that type.

### LLM summary section

When an evaluation completes, this section displays an AI-generated analysis including:

* Overall quality assessment
* Key issues identified
* Recommended actions
* Patterns observed across mappings

### Transformation cards

Each mapping displays as a card showing:

| Element          | Description                             |
| ---------------- | --------------------------------------- |
| Source column    | The column being mapped                 |
| Arrow indicator  | Visual showing the mapping direction    |
| Target attribute | The Rosetta Stone attribute             |
| Confidence badge | Score with color indicator              |
| Mapping type     | Icon indicating system/AI/user/lineage  |
| Expand button    | Shows detailed AI feedback when clicked |

**Expanded card view:**

| Element         | Description                                  |
| --------------- | -------------------------------------------- |
| Score breakdown | Factors contributing to the confidence score |
| Issues          | Specific problems identified by the AI       |
| Recommendations | Suggested improvements                       |
| Sample analysis | Before/after values from the dataset         |

### Evaluate mappings button

**Location:** Top right of the Evaluate tab

**Action:** Triggers a new AI evaluation of all mappings on the dataset

**States:**

* **Ready:** Button displays "Evaluate Mappings"
* **Running:** Button shows loading animation; icon animates
* **Complete:** Results display in the interface

***

## Normalize tab

The Normalize tab generates and displays AI mapping suggestions for unmapped columns.

### Start analysis button

**Location:** Top of the Normalize tab (when no suggestions exist)

**Action:** Triggers AI analysis to generate mapping suggestions for unmapped columns

**States:**

* **Ready:** Button displays "Start Analysis"
* **Running:** Button shows loading animation
* **Complete:** Suggestions appear grouped by confidence

### Global warnings section

Displays dataset-level warnings from the AI analysis:

| Warning type         | Description                                      |
| -------------------- | ------------------------------------------------ |
| High null rate       | Many columns have significant null values        |
| Inconsistent formats | Same column contains multiple data formats       |
| Low coverage         | Few columns could be mapped to known attributes  |
| Duplicate columns    | Multiple columns appear to contain the same data |

### Confidence group sections

Suggestions are organized into collapsible sections by confidence tier:

| Section               | Initial state | Contains                    |
| --------------------- | ------------- | --------------------------- |
| **High Confidence**   | Expanded      | Suggestions scoring 80-100% |
| **Medium Confidence** | Collapsed     | Suggestions scoring 50-79%  |
| **Low Confidence**    | Collapsed     | Suggestions scoring 0-49%   |

Click section headers to expand or collapse.

### Accept all high confidence button

**Location:** Header of the High Confidence section

**Action:** Bulk accepts all suggestions in the high-confidence tier

**Confirmation:** Displays count of mappings to be created before proceeding

### Suggestion cards

Each suggestion displays as a card showing:

| Element              | Description                                           |
| -------------------- | ----------------------------------------------------- |
| Source column        | The unmapped column in your dataset                   |
| Target attribute     | The AI-recommended Rosetta Stone attribute            |
| Confidence score     | 0-100% with color indicator                           |
| Reasoning            | Brief explanation of why the AI proposed this mapping |
| Before/after samples | Example values showing transformation results         |
| Accept button        | Creates the mapping                                   |
| Reject button        | Dismisses the suggestion                              |

**Sample display format:**

| Before                     | After                           |
| -------------------------- | ------------------------------- |
| Original value from source | Transformed value for attribute |

Up to 5 sample pairs are shown per suggestion.

***

## Actions reference

### Dataset-level actions

| Action                | Location      | Description                                  | Result                                   |
| --------------------- | ------------- | -------------------------------------------- | ---------------------------------------- |
| **Evaluate Mappings** | Evaluate tab  | Run AI quality analysis on all mappings      | Confidence scores and feedback displayed |
| **Start Analysis**    | Normalize tab | Generate AI suggestions for unmapped columns | Suggestions displayed by confidence tier |

### Mapping-level actions

| Action             | Location                 | Description               | Result                             |
| ------------------ | ------------------------ | ------------------------- | ---------------------------------- |
| **View Details**   | Transformation card      | Expand to see AI feedback | Detailed analysis displayed        |
| **Edit Mapping**   | Transformation card menu | Modify the mapping        | Opens mapping editor               |
| **Delete Mapping** | Transformation card menu | Remove the mapping        | Mapping deleted after confirmation |

### Suggestion-level actions

| Action              | Location                | Description                                 | Result                              |
| ------------------- | ----------------------- | ------------------------------------------- | ----------------------------------- |
| **Accept**          | Suggestion card         | Create mapping from suggestion              | Mapping created; suggestion removed |
| **Reject**          | Suggestion card         | Dismiss the suggestion                      | Suggestion removed; won't reappear  |
| **Accept All High** | High Confidence section | Bulk accept all high-confidence suggestions | Multiple mappings created           |

***

## Status indicators

### Evaluation status

| Indicator                 | Meaning                                |
| ------------------------- | -------------------------------------- |
| **No results**            | Evaluation has never been run          |
| **Evaluating...**         | Evaluation in progress                 |
| **Last evaluated:&#x20;** | Results are from the indicated date    |
| **Stale** badge           | Mappings changed since last evaluation |

### Suggestion status

| Indicator             | Meaning                              |
| --------------------- | ------------------------------------ |
| **No suggestions**    | Analysis has never been run          |
| **Generating...**     | Analysis in progress                 |
| **&#x20;suggestions** | Suggestions available for review     |
| **All processed**     | All suggestions accepted or rejected |

***

## Filtering and sorting

### Listing page

* Click column headers to sort by that column
* Default sort: Updated (descending)

### Detail page mappings

* Click confidence cards to filter by confidence tier
* Click overview cards to filter by mapping type
* Filters can be combined (e.g., low-confidence user mappings)
* Click **All** to clear filters

***

## Related content

<CardGroup>
  <Card title="Managing Evaluations" icon="gauge" href="/guides/rosetta-stone/managing-evaluations">
    How to run and interpret evaluations
  </Card>

  <Card title="Accepting AI Suggestions" icon="wand-magic-sparkles" href="/guides/rosetta-stone/accepting-ai-suggestions">
    How to review and accept suggestions
  </Card>

  <Card title="Confidence Scoring" icon="gauge-high" href="/concepts/rosetta-stone/confidence-scoring">
    How confidence scores are calculated
  </Card>

  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Creating mappings manually
  </Card>
</CardGroup>


# Transformation Functions
Source: https://docs.narrative.io/reference/rosetta-stone/transformation-functions

Functions available for mapping transformation expressions

This reference documents the functions available in Rosetta Stone mapping transformation expressions. Transformation expressions use NQL syntax.

## String functions

Functions for manipulating text values.

### LOWER

Converts a string to lowercase.

**Syntax:** `LOWER(string)`

**Example:**

```sql theme={null}
LOWER(gender)  -- 'MALE' → 'male'
```

### UPPER

Converts a string to uppercase.

**Syntax:** `UPPER(string)`

**Example:**

```sql theme={null}
UPPER(country_code)  -- 'us' → 'US'
```

### TRIM

Removes leading and trailing whitespace.

**Syntax:** `TRIM(string)`

**Example:**

```sql theme={null}
TRIM(email)  -- '  user@example.com  ' → 'user@example.com'
```

### LTRIM / RTRIM

Removes leading (LTRIM) or trailing (RTRIM) whitespace.

**Syntax:** `LTRIM(string)`, `RTRIM(string)`

### SUBSTRING

Extracts a portion of a string.

**Syntax:** `SUBSTRING(string, start, length)`

* `start`: 1-based starting position
* `length`: number of characters to extract

**Example:**

```sql theme={null}
SUBSTRING(zip_code, 1, 5)  -- '12345-6789' → '12345'
```

### CONCAT

Concatenates multiple strings.

**Syntax:** `CONCAT(string1, string2, ...)`

**Example:**

```sql theme={null}
CONCAT(first_name, ' ', last_name)  -- 'John', 'Doe' → 'John Doe'
```

### LENGTH

Returns the length of a string.

**Syntax:** `LENGTH(string)`

**Example:**

```sql theme={null}
LENGTH(country_code)  -- 'US' → 2
```

### REPLACE

Replaces occurrences of a substring.

**Syntax:** `REPLACE(string, search, replacement)`

**Example:**

```sql theme={null}
REPLACE(phone, '-', '')  -- '555-123-4567' → '5551234567'
```

### SPLIT

Splits a string into an array by delimiter.

**Syntax:** `SPLIT(string, delimiter)`

**Example:**

```sql theme={null}
SPLIT(interests, ',')  -- 'sports,music,travel' → ['sports', 'music', 'travel']
```

### SPLIT\_PART

Extracts a specific part from a delimited string.

**Syntax:** `SPLIT_PART(string, delimiter, index)`

* `index`: 1-based position of the part to extract

**Example:**

```sql theme={null}
SPLIT_PART(full_name, ' ', 1)  -- 'John Doe' → 'John'
```

### POSITION

Returns the position of a substring (1-based), or 0 if not found.

**Syntax:** `POSITION(substring IN string)`

**Example:**

```sql theme={null}
POSITION('@' IN email)  -- 'user@example.com' → 5
```

### REGEXP\_REPLACE

Replaces text matching a regular expression.

**Syntax:** `REGEXP_REPLACE(string, pattern, replacement)`

**Example:**

```sql theme={null}
REGEXP_REPLACE(phone, '[^0-9]', '')  -- '(555) 123-4567' → '5551234567'
```

### REGEXP\_EXTRACT

Extracts text matching a regular expression.

**Syntax:** `REGEXP_EXTRACT(string, pattern)`

**Example:**

```sql theme={null}
REGEXP_EXTRACT(email, '@(.+)$')  -- 'user@example.com' → 'example.com'
```

***

## Date and time functions

Functions for working with dates and timestamps.

### TO\_TIMESTAMP

Parses a string into a timestamp.

**Syntax:** `TO_TIMESTAMP(string, format)`

**Common format patterns:**

| Pattern   | Description     | Example    |
| --------- | --------------- | ---------- |
| `YYYY`    | 4-digit year    | `2024`     |
| `MM`      | 2-digit month   | `01`-`12`  |
| `DD`      | 2-digit day     | `01`-`31`  |
| `HH24`    | 24-hour hour    | `00`-`23`  |
| `HH12`    | 12-hour hour    | `01`-`12`  |
| `MI`      | Minutes         | `00`-`59`  |
| `SS`      | Seconds         | `00`-`59`  |
| `AM`/`PM` | AM/PM indicator | `AM`, `PM` |

**Examples:**

```sql theme={null}
TO_TIMESTAMP(date_col, 'YYYY-MM-DD')
TO_TIMESTAMP(datetime_col, 'MM/DD/YYYY HH24:MI:SS')
TO_TIMESTAMP(datetime_col, 'YYYY-MM-DD"T"HH24:MI:SS')
```

### TRY\_TO\_TIMESTAMP

Like TO\_TIMESTAMP, but returns NULL instead of error on parse failure.

**Syntax:** `TRY_TO_TIMESTAMP(string, format)`

**Example:**

```sql theme={null}
COALESCE(
  TRY_TO_TIMESTAMP(date_col, 'YYYY-MM-DD'),
  TRY_TO_TIMESTAMP(date_col, 'MM/DD/YYYY')
)
```

### TO\_DATE

Parses a string into a date (without time component).

**Syntax:** `TO_DATE(string, format)`

**Example:**

```sql theme={null}
TO_DATE(birth_date, 'YYYY-MM-DD')
```

### CURRENT\_TIMESTAMP

Returns the current timestamp in UTC.

**Syntax:** `CURRENT_TIMESTAMP`

### CURRENT\_DATE

Returns the current date.

**Syntax:** `CURRENT_DATE`

### DATE\_DIFF

Calculates the difference between two dates.

**Syntax:** `DATE_DIFF(unit, start, end)`

**Units:** `'year'`, `'month'`, `'day'`, `'hour'`, `'minute'`, `'second'`

**Example:**

```sql theme={null}
DATE_DIFF('year', birth_date, CURRENT_DATE)  -- Calculate age
```

### DATE\_ADD / DATE\_SUB

Adds or subtracts an interval from a date.

**Syntax:** `DATE_ADD(date, interval)`, `DATE_SUB(date, interval)`

**Example:**

```sql theme={null}
DATE_ADD(timestamp_col, INTERVAL '30' DAY)
DATE_SUB(timestamp_col, INTERVAL '1' HOUR)
```

### YEAR / MONTH / DAY

Extracts components from a date or timestamp.

**Syntax:** `YEAR(date)`, `MONTH(date)`, `DAY(date)`

**Example:**

```sql theme={null}
YEAR(birth_date)  -- 1990
MONTH(event_timestamp)  -- 6
```

### HOUR / MINUTE / SECOND

Extracts time components from a timestamp.

**Syntax:** `HOUR(timestamp)`, `MINUTE(timestamp)`, `SECOND(timestamp)`

***

## Type conversion functions

Functions for converting between data types.

### CAST

Converts a value to a specified type.

**Syntax:** `CAST(expression AS type)`

**Supported types:** `INTEGER`, `BIGINT`, `FLOAT`, `DOUBLE`, `VARCHAR`, `BOOLEAN`, `TIMESTAMP`, `DATE`

**Examples:**

```sql theme={null}
CAST(string_number AS INTEGER)  -- '42' → 42
CAST(timestamp_col AS DATE)     -- 2024-01-15T14:30:00Z → 2024-01-15
CAST(integer_col AS VARCHAR)    -- 42 → '42'
```

### TRY\_CAST

Like CAST, but returns NULL instead of error on conversion failure.

**Syntax:** `TRY_CAST(expression AS type)`

**Example:**

```sql theme={null}
TRY_CAST(maybe_number AS INTEGER)  -- 'abc' → NULL instead of error
```

***

## Conditional functions

Functions for conditional logic.

### CASE

Evaluates conditions and returns a result.

**Simple CASE syntax:**

```sql theme={null}
CASE expression
  WHEN value1 THEN result1
  WHEN value2 THEN result2
  ELSE default_result
END
```

**Searched CASE syntax:**

```sql theme={null}
CASE
  WHEN condition1 THEN result1
  WHEN condition2 THEN result2
  ELSE default_result
END
```

**Examples:**

```sql theme={null}
-- Simple CASE
CASE gender
  WHEN 'M' THEN 'male'
  WHEN 'F' THEN 'female'
  ELSE 'unknown'
END

-- Searched CASE
CASE
  WHEN age < 18 THEN 'minor'
  WHEN age < 65 THEN 'adult'
  ELSE 'senior'
END
```

### COALESCE

Returns the first non-null argument.

**Syntax:** `COALESCE(expr1, expr2, ...)`

**Example:**

```sql theme={null}
COALESCE(primary_email, secondary_email, 'unknown')
```

### NULLIF

Returns NULL if two expressions are equal, otherwise returns the first expression.

**Syntax:** `NULLIF(expr1, expr2)`

**Example:**

```sql theme={null}
NULLIF(value, '')  -- Treat empty string as NULL
NULLIF(value, 'N/A')  -- Treat 'N/A' as NULL
```

### IF

Simple conditional expression (shorthand for CASE).

**Syntax:** `IF(condition, true_result, false_result)`

**Example:**

```sql theme={null}
IF(age >= 18, 'adult', 'minor')
```

***

## Null handling functions

Functions for working with NULL values.

### COALESCE

(See Conditional functions above)

### NULLIF

(See Conditional functions above)

### IFNULL

Returns the second argument if the first is NULL.

**Syntax:** `IFNULL(expr, default_value)`

**Example:**

```sql theme={null}
IFNULL(middle_name, '')
```

### IS NULL / IS NOT NULL

Tests for NULL values.

**Example:**

```sql theme={null}
CASE WHEN value IS NULL THEN 'missing' ELSE value END
```

***

## Array functions

Functions for working with arrays.

### ARRAY

Creates an array from values.

**Syntax:** `ARRAY(value1, value2, ...)`

**Example:**

```sql theme={null}
ARRAY(email_sha256, phone_sha256)
```

### ARRAY\_CONTAINS

Checks if an array contains a value.

**Syntax:** `ARRAY_CONTAINS(array, value)`

**Example:**

```sql theme={null}
ARRAY_CONTAINS(tags, 'premium')
```

### ARRAY\_POSITION

Returns the 0-based index of the first occurrence of an element in an array. Returns `NULL` if not found.

**Syntax:** `ARRAY_POSITION(array, value)`

**Example:**

```sql theme={null}
ARRAY_POSITION(tags, 'premium')
```

### TRANSFORM

Applies a function to each array element.

**Syntax:** `TRANSFORM(array, element -> expression)`

**Example:**

```sql theme={null}
TRANSFORM(categories, x -> LOWER(TRIM(x)))
```

### FILTER

Filters array elements based on a condition.

**Syntax:** `FILTER(array, element -> condition)`

**Example:**

```sql theme={null}
FILTER(values, x -> x IS NOT NULL)
FILTER(numbers, x -> x > 0)
```

### ARRAY\_JOIN

Joins array elements into a string.

**Syntax:** `ARRAY_JOIN(array, delimiter)`

**Example:**

```sql theme={null}
ARRAY_JOIN(tags, ', ')  -- ['a', 'b', 'c'] → 'a, b, c'
```

***

## Struct functions

Functions for working with structs.

### STRUCT

Creates a struct from named values.

**Syntax:** `STRUCT(expr1 AS name1, expr2 AS name2, ...)`

**Example:**

```sql theme={null}
STRUCT(
  'email_sha256' AS type,
  hash_value AS value,
  'hashed_email' AS context
)
```

### Accessing struct fields

Use dot notation to access struct fields.

**Example:**

```sql theme={null}
unique_identifier.type
geo_coordinates.latitude
```

***

## JSON functions

Functions for parsing and working with JSON data.

### JSON\_PARSE

Parses a JSON string into a structured value.

**Syntax:** `JSON_PARSE(json_string)`

**Example:**

```sql theme={null}
JSON_PARSE(properties_json)
```

### JSON\_EXTRACT

Extracts a value from JSON using a path.

**Syntax:** `JSON_EXTRACT(json, path)`

**Example:**

```sql theme={null}
JSON_EXTRACT(data, '$.user.email')
```

### JSON\_EXTRACT\_SCALAR

Extracts a scalar value from JSON.

**Syntax:** `JSON_EXTRACT_SCALAR(json, path)`

**Example:**

```sql theme={null}
JSON_EXTRACT_SCALAR(data, '$.count')
```

***

## Hash functions

Functions for hashing values.

### SHA256

Computes SHA-256 hash.

**Syntax:** `SHA256(string)`

**Returns:** 64-character hexadecimal string

**Example:**

```sql theme={null}
SHA256(LOWER(TRIM(email)))
```

### MD5

Computes MD5 hash.

**Syntax:** `MD5(string)`

**Returns:** 32-character hexadecimal string

<Warning>
  Hash functions are available but it's recommended to hash PII before uploading to Narrative. See [Hashing PII for Upload](/guides/ingestion/hashing-pii).
</Warning>

***

## Mathematical functions

Functions for numeric operations.

### ABS

Returns absolute value.

**Syntax:** `ABS(number)`

### ROUND

Rounds to specified decimal places.

**Syntax:** `ROUND(number, decimals)`

**Example:**

```sql theme={null}
ROUND(price, 2)  -- 19.999 → 20.00
```

### FLOOR / CEIL

Rounds down (FLOOR) or up (CEIL) to nearest integer.

**Syntax:** `FLOOR(number)`, `CEIL(number)`

### MOD

Returns remainder of division.

**Syntax:** `MOD(dividend, divisor)`

***

## Related content

<CardGroup>
  <Card title="Mapping Schemas" icon="diagram-project" href="/guides/rosetta-stone/mapping-schemas">
    Use these functions in your mappings
  </Card>

  <Card title="Edge Cases" icon="triangle-exclamation" href="/guides/rosetta-stone/edge-cases">
    Complex transformation patterns
  </Card>

  <Card title="Attribute Types" icon="shapes" href="/reference/rosetta-stone/attribute-types">
    Target types for transformations
  </Card>

  <Card title="NQL Functions Reference" icon="code" href="/nql/functions/index">
    Complete NQL function documentation
  </Card>
</CardGroup>


# SDKs
Source: https://docs.narrative.io/reference/sdks/index

Official SDKs for programmatic access to Narrative I/O

Narrative provides official SDKs for integrating with the platform programmatically. SDKs simplify authentication, provide type safety, and offer convenient methods for common operations.

## Available SDKs

<CardGroup>
  <Card title="TypeScript SDK" icon="js" href="/reference/sdks/typescript">
    Full-featured SDK for Node.js and TypeScript applications. Includes complete type definitions and supports all API operations.
  </Card>
</CardGroup>

## When to use an SDK

| Approach     | Best for                                                   |
| ------------ | ---------------------------------------------------------- |
| **Web UI**   | Manual operations, data exploration, one-off tasks         |
| **REST API** | Custom integrations, languages without SDK support         |
| **SDK**      | TypeScript/JavaScript apps, type safety, rapid development |

### Benefits of using an SDK

* **Type safety** - Full TypeScript support with autocomplete
* **Convenience** - Pre-built methods for common operations
* **Authentication** - Simplified API key management
* **Error handling** - Consistent error types and messages
* **Documentation** - Inline documentation and examples

## Quick comparison

```typescript theme={null}
// With SDK
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({ apiKey: process.env.NARRATIVE_API_KEY });
const datasets = await api.getDatasets();
```

```typescript theme={null}
// With REST API directly
const response = await fetch('https://api.narrative.io/datasets', {
  headers: {
    'Authorization': `Bearer ${process.env.NARRATIVE_API_KEY}`,
    'Content-Type': 'application/json',
  },
});
const datasets = await response.json();
```

## Getting started

<Steps>
  <Step title="Create an API key">
    Navigate to [Settings > API Keys](/account-settings/api-keys) and create a new key with appropriate permissions.
  </Step>

  <Step title="Install the SDK">
    ```bash theme={null}
    npm install @narrative.io/data-collaboration-sdk-ts
    ```
  </Step>

  <Step title="Initialize and use">
    ```typescript theme={null}
    import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

    const api = new NarrativeApi({
      apiKey: process.env.NARRATIVE_API_KEY,
    });

    const datasets = await api.getDatasets();
    ```
  </Step>
</Steps>

## Related resources

<CardGroup>
  <Card title="SDK Quickstart" icon="rocket" href="/getting-started/sdk-quickstart">
    Tutorial for first-time SDK users
  </Card>

  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys
  </Card>

  <Card title="REST API Reference" icon="book" href="/api-reference">
    Full REST API documentation
  </Card>

  <Card title="SDK Guides" icon="book-open" href="/guides/sdk">
    How-to guides for common tasks
  </Card>
</CardGroup>


# API Modules
Source: https://docs.narrative.io/reference/sdks/typescript/api-modules

Reference for all TypeScript SDK API modules

The `NarrativeApi` class exposes multiple API modules through a mixin pattern. This reference documents all available methods.

## NQL API

Execute, validate, and compile NQL queries.

### executeNql

Execute an NQL query and return results.

```typescript theme={null}
const result = await api.executeNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
  execution_cluster: { type: 'shared' },  // optional
  create_as_view: false,                   // optional
});
```

| Parameter           | Type                                | Required | Description                                                                                                      |
| ------------------- | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------- |
| `nql`               | `string`                            | Yes      | The NQL query to execute                                                                                         |
| `data_plane_id`     | `string \| null`                    | Yes      | Target data plane ID                                                                                             |
| `execution_cluster` | `{ type: 'shared' \| 'dedicated' }` | No       | Execution cluster type                                                                                           |
| `create_as_view`    | `boolean`                           | No       | Create result as a [view dataset](/concepts/nql/materialized-views#view-datasets) instead of a materialized view |

**Returns:** `Promise<NqlResult>`

### compileNql

Compile an NQL query and return the transpiled SQL.

```typescript theme={null}
const compiled = await api.compileNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
});

console.log(compiled.sql);
```

**Returns:** `Promise<NqlCompileResult>`

### validateNql

Validate an NQL query's syntax.

```typescript theme={null}
const ast = await api.validateNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
});
```

**Returns:** `Promise<NqlAst>`

### parseNql

Parse an NQL query into an abstract syntax tree.

```typescript theme={null}
const ast = await api.parseNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
});
```

**Returns:** `Promise<NqlAst>`

### getNqlByJobId

Retrieve results from a previous query by job ID.

```typescript theme={null}
const result = await api.getNqlByJobId('job-id-here');
```

**Returns:** `Promise<NqlResult>`

***

## Datasets API

Manage datasets and their metadata.

### getDatasets

List all accessible datasets.

```typescript theme={null}
const response = await api.getDatasets();
console.log(response.records);
```

**Returns:** `Promise<ApiRecords<Dataset>>`

### getDataset

Get a specific dataset by ID.

```typescript theme={null}
const dataset = await api.getDataset(12345);
```

| Parameter   | Type     | Required | Description |
| ----------- | -------- | -------- | ----------- |
| `datasetId` | `number` | Yes      | Dataset ID  |

**Returns:** `Promise<Dataset>`

### createDataset

Create a new dataset.

```typescript theme={null}
const dataset = await api.createDataset({
  name: 'my_dataset',
  display_name: 'My Dataset',
  description: 'Description here',
  schema: { properties: { ... } },
  tags: ['tag1', 'tag2'],
});
```

**Returns:** `Promise<Dataset>`

### updateDataset

Update dataset metadata.

```typescript theme={null}
const dataset = await api.updateDataset({
  dataset_id: 12345,
  display_name: 'Updated Name',
  description: 'Updated description',
});
```

**Returns:** `Promise<Dataset>`

### deleteDataset

Delete a dataset permanently.

```typescript theme={null}
await api.deleteDataset(12345);
```

**Returns:** `Promise<void>`

### addTags

Add tags to a dataset.

```typescript theme={null}
const dataset = await api.addTags(12345, ['production', 'verified']);
```

**Returns:** `Promise<Dataset>`

### removeTags

Remove tags from a dataset.

```typescript theme={null}
const dataset = await api.removeTags(12345, ['deprecated']);
```

**Returns:** `Promise<Dataset>`

### getStatistics

Get dataset statistics.

```typescript theme={null}
const stats = await api.getStatistics(12345, 0, 1000);
```

| Parameter   | Type     | Required | Description                    |
| ----------- | -------- | -------- | ------------------------------ |
| `datasetId` | `number` | Yes      | Dataset ID                     |
| `offset`    | `number` | No       | Pagination offset (default: 0) |
| `size`      | `number` | No       | Page size (default: 1000)      |

**Returns:** `Promise<ApiRecords<DatasetTableSummary>>`

### getColumnStats

Get column-level statistics.

```typescript theme={null}
const columnStats = await api.getColumnStats(12345);
```

**Returns:** `Promise<ColumnStatistics>`

### getDatasetSample

Get a sample of dataset records.

```typescript theme={null}
const sample = await api.getDatasetSample(12345, 1000);
```

**Returns:** `Promise<ApiRecords<Record<string, string>>>`

### requestDatasetSample

Request a new sample (async operation).

```typescript theme={null}
const { job_id } = await api.requestDatasetSample(12345);
```

**Returns:** `Promise<{ job_id: string }>`

### getDatasetFiles

List files in a dataset.

```typescript theme={null}
const files = await api.getDatasetFiles(
  12345,           // datasetId
  100,             // size
  0,               // offset
  '2024-01-01',    // afterTimestamp (optional)
  67890            // fromSnapshotId (optional)
);
```

**Returns:** `Promise<FilePerSnapshotResponse>`

### getDatasetFileDownloadUrl

Get a download URL for a specific file.

```typescript theme={null}
const { download_url } = await api.getDatasetFileDownloadUrl(
  12345,           // datasetId
  67890,           // snapshotId
  'file-id-here'   // fileId
);
```

**Returns:** `Promise<{ download_url: string }>`

### refreshMaterializedView

Refresh a materialized view.

```typescript theme={null}
const job = await api.refreshMaterializedView(12345);
console.log('Job ID:', job.id);
```

**Returns:** `Promise<UnknownJob>`

### updateRetentionPolicy

Update the retention policy.

```typescript theme={null}
const dataset = await api.updateRetentionPolicy(12345, {
  type: 'time_based',
  retention_days: 365,
});
```

**Returns:** `Promise<Dataset>`

### activateDataset

Activate a pending dataset.

```typescript theme={null}
await api.activateDataset(12345);
```

**Returns:** `Promise<void>`

***

## Jobs API

Track asynchronous job status.

### getJobs

List all jobs.

```typescript theme={null}
const response = await api.getJobs({
  page: 1,
  per_page: 50,
});
```

**Returns:** `Promise<ApiRecordsV2<Job>>`

### getJob

Get a specific job by ID.

```typescript theme={null}
const job = await api.getJob('job-id-here');
```

**Returns:** `Promise<Job>`

***

## Model Inference API

Run LLM inference within your data plane.

### runModelInference

Submit a model inference request.

```typescript theme={null}
const job = await api.runModelInference({
  data_plane_id: 'dp_your_data_plane_id',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'system',
      content: [{ type: 'text', text: 'You are a helpful assistant.' }]
    },
    {
      role: 'user',
      content: [{ type: 'text', text: 'Analyze this data...' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        analysis: { type: 'string' }
      },
      required: ['analysis']
    },
    max_tokens: 1000,
    temperature: 0.7
  },
  tags: ['analysis']  // optional
});

console.log('Job ID:', job.id);
```

| Parameter          | Type                 | Required | Description                              |
| ------------------ | -------------------- | -------- | ---------------------------------------- |
| `data_plane_id`    | `string`             | Yes      | Target data plane ID                     |
| `model`            | `InferenceModel`     | Yes      | Model to use for inference               |
| `messages`         | `InferenceMessage[]` | Yes      | Conversation messages                    |
| `inference_config` | `InferenceConfig`    | Yes      | Inference configuration with JSON Schema |
| `tags`             | `string[]`           | No       | Optional tags for the job                |

**Returns:** `Promise<ModelInferenceRunJob>`

For complete type definitions and detailed examples, see [Model Inference API Reference](/reference/sdks/typescript/model-inference-api).

***

## Access Rules API

Manage access permissions.

### getAccessRules

List access rules.

```typescript theme={null}
const rules = await api.getAccessRules();
```

**Returns:** `Promise<ApiRecords<AccessRule>>`

### createAccessRule

Create a new access rule.

```typescript theme={null}
const rule = await api.createAccessRule({
  // rule configuration
});
```

**Returns:** `Promise<AccessRule>`

### updateAccessRule

Update an existing access rule.

```typescript theme={null}
const rule = await api.updateAccessRule({
  // updated configuration
});
```

**Returns:** `Promise<AccessRule>`

### deleteAccessRule

Delete an access rule.

```typescript theme={null}
await api.deleteAccessRule(ruleId);
```

**Returns:** `Promise<void>`

***

## Access Tokens API

Manage API tokens.

### getAccessTokens

List all access tokens.

```typescript theme={null}
const tokens = await api.getAccessTokens();
```

**Returns:** `Promise<ApiRecords<AccessToken>>`

### getAccessToken

Get a specific token.

```typescript theme={null}
const token = await api.getAccessToken(tokenId);
```

**Returns:** `Promise<AccessToken>`

### createAccessToken

Create a new access token.

```typescript theme={null}
const token = await api.createAccessToken({
  name: 'My Token',
  permissions: ['datasets:read'],
  expires_at: '2025-12-31T23:59:59Z',
});
```

**Returns:** `Promise<AccessToken>`

### updateAccessToken

Update an existing token.

```typescript theme={null}
const token = await api.updateAccessToken({
  id: tokenId,
  name: 'Updated Name',
});
```

**Returns:** `Promise<AccessToken>`

### deleteAccessToken

Delete an access token.

```typescript theme={null}
await api.deleteAccessToken(tokenId);
```

**Returns:** `Promise<void>`

***

## Mappings API

Work with Rosetta Stone mappings.

### getMappings

List mappings for a company.

```typescript theme={null}
const mappings = await api.getMappings(companyId);
```

**Returns:** `Promise<ApiRecords<Mapping>>`

### createPrivateMapping

Create a mapping for a dataset.

<Note>
  Despite its name, this method now creates **global** mappings. All new mappings are global by default — the private-then-promote workflow has been deprecated for new mappings.
</Note>

```typescript theme={null}
const mapping = await api.createPrivateMapping(companyId, {
  // mapping configuration
});
```

**Returns:** `Promise<Mapping>`

### testMapping

Test a mapping configuration.

```typescript theme={null}
const result = await api.testMapping({
  // test configuration
});
```

**Returns:** `Promise<MappingTestResult>`

### previewMapping

Preview mapping results.

```typescript theme={null}
const preview = await api.previewMapping({
  // preview configuration
});
```

**Returns:** `Promise<MappingPreview>`

***

## Subscriptions API

Manage data subscriptions.

### getSubscriptions

List all subscriptions.

```typescript theme={null}
const subscriptions = await api.getSubscriptions();
```

**Returns:** `Promise<ApiRecords<Subscription>>`

***

## Health API

Check API health status.

### getHealthCheck

Check if the API is healthy.

```typescript theme={null}
const health = await api.getHealthCheck();
console.log('Status:', health.status);
```

**Returns:** `Promise<HealthCheckResponse>`

***

## TypeScript types

The SDK exports all type definitions:

```typescript theme={null}
import type {
  // Core types
  NarrativeApi,
  Config,
  Environment,

  // Dataset types
  Dataset,
  DatasetStatus,
  DatasetWriteMode,
  Schema,
  SchemaProperty,

  // NQL types
  NqlQueryInput,
  NqlResult,
  NqlCompileResult,

  // Job types
  Job,

  // Access types
  AccessRule,
  AccessToken,

  // Pagination
  ApiRecords,
  ApiRecordsV2,
  PaginationOptions,
} from '@narrative.io/data-collaboration-sdk-ts';
```

***

## Related content

<CardGroup>
  <Card title="SDK Overview" icon="js" href="/reference/sdks/typescript">
    TypeScript SDK overview
  </Card>

  <Card title="Configuration" icon="gear" href="/reference/sdks/typescript/configuration">
    Configuration options
  </Card>

  <Card title="SDK Guides" icon="book-open" href="/guides/sdk">
    How-to guides
  </Card>

  <Card title="REST API" icon="code" href="/api-reference">
    Full REST API reference
  </Card>
</CardGroup>


# Configuration
Source: https://docs.narrative.io/reference/sdks/typescript/configuration

Configuration options for the Narrative TypeScript SDK

The `NarrativeApi` class accepts a configuration object that controls authentication, environment targeting, and request behavior.

## Constructor options

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: 'your-api-key',
  environment: 'prod',
  headers: {},
  verbose: false,
});
```

| Option        | Type                        | Default     | Description                                           |
| ------------- | --------------------------- | ----------- | ----------------------------------------------------- |
| `apiKey`      | `string`                    | `undefined` | API key for authentication. Passed as a Bearer token. |
| `environment` | `'prod' \| 'dev' \| string` | `'prod'`    | Target environment or custom base URL                 |
| `headers`     | `Record<string, string>`    | `{}`        | Additional HTTP headers to include in requests        |
| `verbose`     | `boolean`                   | `false`     | Enable verbose logging (automatically enabled in dev) |

## Environment URLs

The SDK supports two built-in environments, or you can specify a custom URL:

| Environment | Base URL                        |
| ----------- | ------------------------------- |
| `prod`      | `https://api.narrative.io/`     |
| `dev`       | `https://api-dev.narrative.io/` |
| Custom      | Any valid URL you provide       |

## Configuration examples

### Production (default)

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});
```

### Development environment

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: 'dev',
});
```

### Custom base URL

For self-hosted or custom deployments:

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: 'https://custom.narrative.example.com/',
});
```

### Enable verbose logging

Useful for debugging API requests:

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  verbose: true,
});
```

When verbose mode is enabled, the SDK logs:

* Request URLs and methods
* Request payloads
* Response data
* Configuration details

<Note>
  Verbose mode is automatically enabled when using the `dev` environment.
</Note>

### Custom headers

Add custom headers to all requests:

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  headers: {
    'X-Custom-Header': 'custom-value',
  },
});
```

## Runtime configuration

You can update the API key after initialization:

```typescript theme={null}
const api = new NarrativeApi({});

// Set API key later
api.setApiKey('new-api-key');

// Get current API key
const currentKey = api.getApiKey();

// Get current environment
const environment = api.getEnvironment();

// Get base URL
const baseUrl = api.getBaseUrl();
```

## Environment variables

While the SDK doesn't automatically read environment variables, the recommended pattern is:

```typescript theme={null}
const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: process.env.NARRATIVE_ENVIRONMENT || 'prod',
});
```

| Variable                | Description                                       |
| ----------------------- | ------------------------------------------------- |
| `NARRATIVE_API_KEY`     | Your API key for authentication                   |
| `NARRATIVE_ENVIRONMENT` | Target environment (`prod`, `dev`, or custom URL) |

## TypeScript types

The SDK exports configuration types for TypeScript users:

```typescript theme={null}
import type { Config, Environment } from '@narrative.io/data-collaboration-sdk-ts';

const config: Config = {
  apiKey: process.env.NARRATIVE_API_KEY,
  environment: 'prod' as Environment,
};

const api = new NarrativeApi(config);
```

## Related content

<CardGroup>
  <Card title="API Keys" icon="key" href="/account-settings/api-keys">
    Create and manage API keys
  </Card>

  <Card title="Authentication Guide" icon="lock" href="/guides/sdk/authentication">
    Authentication best practices
  </Card>

  <Card title="SDK Overview" icon="js" href="/reference/sdks/typescript">
    TypeScript SDK overview
  </Card>

  <Card title="API Modules" icon="puzzle-piece" href="/reference/sdks/typescript/api-modules">
    Available API methods
  </Card>
</CardGroup>


# TypeScript SDK
Source: https://docs.narrative.io/reference/sdks/typescript/index

Reference documentation for the Narrative TypeScript SDK

The Narrative TypeScript SDK (`@narrative.io/data-collaboration-sdk-ts`) provides type-safe access to the Narrative API for Node.js and TypeScript applications.

## Installation

<Tabs>
  <Tab title="npm">
    ```bash theme={null}
    npm install @narrative.io/data-collaboration-sdk-ts
    ```
  </Tab>

  <Tab title="yarn">
    ```bash theme={null}
    yarn add @narrative.io/data-collaboration-sdk-ts
    ```
  </Tab>

  <Tab title="bun">
    ```bash theme={null}
    bun add @narrative.io/data-collaboration-sdk-ts
    ```
  </Tab>
</Tabs>

## Quick start

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

// Execute an NQL query
const result = await api.executeNql({
  nql: 'SELECT user_id, email, created_at FROM company_data."my_dataset" LIMIT 10',
  data_plane_id: null,
});

// List datasets
const datasets = await api.getDatasets();
```

## Architecture

The SDK uses a mixin pattern to combine multiple API modules into a single `NarrativeApi` class. This provides a unified interface while maintaining type safety for each module's methods.

```typescript theme={null}
// All API methods are available on the NarrativeApi instance
const api = new NarrativeApi({ apiKey: 'your-key' });

// NQL operations
await api.executeNql({ ... });
await api.compileNql({ ... });

// Dataset operations
await api.getDatasets();
await api.getDataset(datasetId);

// Job tracking
await api.getJobs();
await api.getJob(jobId);
```

## Available API modules

The SDK provides access to the following API modules:

| Module          | Methods                                                                                  | Description                          |
| --------------- | ---------------------------------------------------------------------------------------- | ------------------------------------ |
| NQL             | `executeNql()`, `compileNql()`, `validateNql()`, `parseNql()`                            | Execute and analyze NQL queries      |
| Datasets        | `getDatasets()`, `getDataset()`, `createDataset()`, `updateDataset()`, `deleteDataset()` | Manage datasets                      |
| Jobs            | `getJobs()`, `getJob()`                                                                  | Track asynchronous job status        |
| Model Inference | `runModelInference()`                                                                    | Run LLM inference within data planes |
| Uploads         | Upload file operations                                                                   | Upload data to datasets              |
| Access Rules    | `getAccessRules()`, `createAccessRule()`, `updateAccessRule()`, `deleteAccessRule()`     | Manage permissions                   |
| Mappings        | Rosetta Stone mapping operations                                                         | Work with schema mappings            |
| Data Planes     | `getDataPlanes()`                                                                        | Data plane operations                |
| Subscriptions   | `getSubscriptions()`                                                                     | Manage data subscriptions            |
| Access Tokens   | Token management operations                                                              | Manage API tokens                    |

For complete method signatures and parameters, see [API Modules Reference](/reference/sdks/typescript/api-modules).

## Requirements

* **Node.js**: 18 or later
* **TypeScript**: 4.7 or later (for TypeScript users)

## Source and issues

* **npm**: [@narrative.io/data-collaboration-sdk-ts](https://www.npmjs.com/package/@narrative.io/data-collaboration-sdk-ts)
* **GitHub**: [narrative-io/data-collaboration-sdk-ts](https://github.com/narrative-io/data-collaboration-sdk-ts)

## Related content

<CardGroup>
  <Card title="Configuration" icon="gear" href="/reference/sdks/typescript/configuration">
    Configuration options and environments
  </Card>

  <Card title="API Modules" icon="puzzle-piece" href="/reference/sdks/typescript/api-modules">
    Complete method reference
  </Card>

  <Card title="SDK Quickstart" icon="rocket" href="/getting-started/sdk-quickstart">
    Get started with the SDK
  </Card>

  <Card title="SDK Guides" icon="book-open" href="/guides/sdk">
    How-to guides for common tasks
  </Card>
</CardGroup>


# Model Inference API
Source: https://docs.narrative.io/reference/sdks/typescript/model-inference-api

Complete reference for Model Inference SDK methods and types

The Model Inference API enables you to run LLM inference within your [data plane](/concepts/primitives/data-planes). This reference documents all methods, types, and interfaces available in the TypeScript SDK.

## Methods

### runModelInference

Submits a model inference request and returns a job that can be polled for results.

```typescript theme={null}
async runModelInference(request: ModelInferenceRunRequest): Promise<ModelInferenceRunJob>
```

**Parameters:**

| Name      | Type                       | Required | Description                         |
| --------- | -------------------------- | -------- | ----------------------------------- |
| `request` | `ModelInferenceRunRequest` | Yes      | The inference request configuration |

**Returns:** `Promise<ModelInferenceRunJob>` - A job object that can be polled for completion.

**Example:**

```typescript theme={null}
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

const job = await api.runModelInference({
  data_plane_id: 'dp_abc123',
  model: 'anthropic.claude-sonnet-4.5',
  messages: [
    {
      role: 'system',
      content: [{ type: 'text', text: 'You are a helpful assistant.' }]
    },
    {
      role: 'user',
      content: [{ type: 'text', text: 'Summarize this data in one sentence.' }]
    }
  ],
  inference_config: {
    output_format_schema: {
      type: 'object',
      properties: {
        summary: { type: 'string' }
      },
      required: ['summary']
    },
    max_tokens: 500,
    temperature: 0.7
  }
});

console.log('Job ID:', job.id);
```

***

## Types

### InferenceModel

Model identifiers supported by the Narrative model inference API.

```typescript theme={null}
type InferenceModel =
  | 'anthropic.claude-haiku-4.5'
  | 'anthropic.claude-sonnet-4.5'
  | 'anthropic.claude-sonnet-4.6'
  | 'anthropic.claude-opus-4.5'
  | 'anthropic.claude-opus-4.6'
  | 'openai.gpt-oss-120b'
  | 'openai.gpt-4.1'
  | 'openai.o4-mini';
```

| Model                         | Provider  | Use Case                                        |
| ----------------------------- | --------- | ----------------------------------------------- |
| `anthropic.claude-haiku-4.5`  | Anthropic | Fast, cost-effective tasks                      |
| `anthropic.claude-sonnet-4.5` | Anthropic | Balanced performance and capability             |
| `anthropic.claude-sonnet-4.6` | Anthropic | Latest balanced model with improved reasoning   |
| `anthropic.claude-opus-4.5`   | Anthropic | Complex reasoning and analysis                  |
| `anthropic.claude-opus-4.6`   | Anthropic | Latest most capable model for complex reasoning |
| `openai.gpt-oss-120b`         | OpenAI    | Open-source large model                         |
| `openai.gpt-4.1`              | OpenAI    | Advanced reasoning                              |
| `openai.o4-mini`              | OpenAI    | Fast, efficient responses                       |

***

### MessageRole

The role of a message in the conversation.

```typescript theme={null}
type MessageRole = 'user' | 'assistant' | 'system';
```

| Role        | Description                                             |
| ----------- | ------------------------------------------------------- |
| `system`    | Sets the model's behavior and context                   |
| `user`      | Input from the user or application                      |
| `assistant` | Previous model responses (for multi-turn conversations) |

***

### InferenceMessage

A message in the inference conversation. The `content` field is an ordered array of
content blocks; the dominant block type is `text`. Agent-loop flows additionally emit
`tool_use` (model requesting a tool call) and `tool_result` (response) blocks — see
`ContentBlock` below.

```typescript theme={null}
interface InferenceMessage {
  role: MessageRole;
  content: ContentBlock[];
}

type ContentBlock =
  | { type: 'text';        text: string }
  | { type: 'tool_use';    tool_use_id: string; name: string; arguments: Record<string, unknown> }
  | { type: 'tool_result'; tool_use_id: string; content: ContentBlock[]; is_error: boolean };
```

| Property  | Type             | Required | Description                                  |
| --------- | ---------------- | -------- | -------------------------------------------- |
| `role`    | `MessageRole`    | Yes      | The role of the message sender               |
| `content` | `ContentBlock[]` | Yes      | Ordered content blocks making up the message |

**Example:**

```typescript theme={null}
const messages: InferenceMessage[] = [
  {
    role: 'system',
    content: [{ type: 'text', text: 'You are a data classification expert.' }]
  },
  {
    role: 'user',
    content: [{ type: 'text', text: 'Classify the following record: {...}' }]
  }
];
```

<Note>
  The legacy `{ role, text: string }` shape is still accepted on requests for backwards
  compatibility — the API auto-canonicalizes it into a single `text` content block.
  Responses always emit the content-block shape. Migrate to the new shape on next edit.
</Note>

***

### InferenceConfig

Configuration parameters for the inference request.

```typescript theme={null}
interface InferenceConfig {
  output_format_schema: Record<string, unknown>;
  max_tokens?: number;
  temperature?: number;
  top_p?: number;
  stop_sequences?: string[];
}
```

| Property               | Type                      | Required | Description                                            |
| ---------------------- | ------------------------- | -------- | ------------------------------------------------------ |
| `output_format_schema` | `Record<string, unknown>` | Yes      | JSON Schema defining the expected output format        |
| `max_tokens`           | `number`                  | No       | Maximum number of tokens to generate                   |
| `temperature`          | `number`                  | No       | Sampling temperature (0-1). Lower = more deterministic |
| `top_p`                | `number`                  | No       | Nucleus sampling parameter (0-1)                       |
| `stop_sequences`       | `string[]`                | No       | Sequences that will stop generation                    |

**Example:**

```typescript theme={null}
const config: InferenceConfig = {
  output_format_schema: {
    type: 'object',
    properties: {
      category: {
        type: 'string',
        enum: ['retail', 'finance', 'healthcare', 'technology']
      },
      confidence: {
        type: 'number',
        minimum: 0,
        maximum: 1
      },
      reasoning: {
        type: 'string'
      }
    },
    required: ['category', 'confidence']
  },
  max_tokens: 1000,
  temperature: 0.3
};
```

***

### ModelInferenceRunRequest

The complete request body for running a model inference job.

```typescript theme={null}
interface ModelInferenceRunRequest {
  data_plane_id: string;
  model: InferenceModel;
  messages: InferenceMessage[];
  inference_config: InferenceConfig;
  tags?: string[];
}
```

| Property           | Type                 | Required | Description                                     |
| ------------------ | -------------------- | -------- | ----------------------------------------------- |
| `data_plane_id`    | `string`             | Yes      | The data plane ID where inference will execute  |
| `model`            | `InferenceModel`     | Yes      | The model to use for inference                  |
| `messages`         | `InferenceMessage[]` | Yes      | The conversation messages                       |
| `inference_config` | `InferenceConfig`    | Yes      | Configuration for the inference                 |
| `tags`             | `string[]`           | No       | Optional tags for organizing and filtering jobs |

***

### InferenceUsage

Token usage metrics from the inference response.

```typescript theme={null}
interface InferenceUsage {
  total_tokens: number;
  prompt_tokens: number;
  completion_tokens: number;
}
```

| Property            | Type     | Description                             |
| ------------------- | -------- | --------------------------------------- |
| `total_tokens`      | `number` | Total tokens used (prompt + completion) |
| `prompt_tokens`     | `number` | Tokens in the input messages            |
| `completion_tokens` | `number` | Tokens in the generated response        |

***

### ModelInferenceRunResult

The result from a completed model inference job.

```typescript theme={null}
interface ModelInferenceRunResult<T = unknown> {
  usage: InferenceUsage;
  structured_output: T;
}
```

| Property            | Type             | Description                                          |
| ------------------- | ---------------- | ---------------------------------------------------- |
| `usage`             | `InferenceUsage` | Token usage metrics                                  |
| `structured_output` | `T`              | The model's response, typed according to your schema |

**Example with typed output:**

```typescript theme={null}
interface ClassificationResult {
  category: string;
  confidence: number;
  reasoning?: string;
}

// After job completion
const result = job.result as ModelInferenceRunResult<ClassificationResult>;

console.log('Category:', result.structured_output.category);
console.log('Confidence:', result.structured_output.confidence);
console.log('Tokens used:', result.usage.total_tokens);
```

***

### ModelInferenceRunJob

The job object returned when submitting an inference request. Extends the base job type with inference-specific result typing.

```typescript theme={null}
interface ModelInferenceRunJob extends Job {
  type: 'model_inference';
  result?: ModelInferenceRunResult;
}
```

| Property     | Type                                                | Description                             |
| ------------ | --------------------------------------------------- | --------------------------------------- |
| `id`         | `string`                                            | Unique job identifier                   |
| `type`       | `'model_inference'`                                 | The job type                            |
| `state`      | `'pending' \| 'running' \| 'completed' \| 'failed'` | Current job state                       |
| `result`     | `ModelInferenceRunResult`                           | Present when job completes successfully |
| `failures`   | `object[]`                                          | Present when job fails                  |
| `created_at` | `string`                                            | ISO timestamp of job creation           |
| `updated_at` | `string`                                            | ISO timestamp of last update            |

***

## Error handling

Model Inference jobs can fail for several reasons:

| Error                | Cause                                       | Solution                                 |
| -------------------- | ------------------------------------------- | ---------------------------------------- |
| Invalid schema       | JSON Schema is malformed                    | Validate schema before submission        |
| Model unavailable    | Requested model not available in data plane | Check supported models                   |
| Token limit exceeded | Response would exceed max\_tokens           | Increase max\_tokens or simplify request |
| Invalid data plane   | Data plane ID not found or no access        | Verify data plane ID and permissions     |

**Example error handling:**

```typescript theme={null}
const job = await api.runModelInference(request);

// Poll for completion
const completedJob = await waitForJob(job.id);

if (completedJob.state === 'failed') {
  console.error('Inference failed:', completedJob.failures);
  // Handle specific failure types
} else {
  console.log('Result:', completedJob.result.structured_output);
}
```

***

## Related content

<CardGroup>
  <Card title="Running Model Inference" icon="play" href="/guides/sdk/running-model-inference">
    Step-by-step guide to submitting inference requests
  </Card>

  <Card title="Structured Output" icon="brackets-curly" href="/guides/sdk/structured-inference-output">
    Working with JSON Schema for typed responses
  </Card>

  <Card title="Tracking Jobs" icon="spinner" href="/guides/sdk/tracking-jobs">
    Monitor inference job status
  </Card>

  <Card title="Supported Models" icon="microchip" href="/reference/model-inference/supported-models">
    Available models and specifications
  </Card>
</CardGroup>


# Permissions
Source: https://docs.narrative.io/reference/security/permissions

Complete reference for all API key permission resources and their available access levels

This page lists every permission resource available when configuring API keys. For an explanation of how permissions work, see [API Key Permissions](/concepts/security/api-key-permissions).

## Access levels

Each permission pairs a resource with one of two access levels:

| Access Level | Capabilities                         |
| ------------ | ------------------------------------ |
| **Read**     | View, list, and retrieve resources   |
| **Write**    | Create, update, and delete resources |

Not all resources support both access levels. The tables below indicate which access levels are available for each resource.

## Core Data

| Resource      | Read | Write | Description                                             |
| ------------- | ---- | ----- | ------------------------------------------------------- |
| Datasets      | Yes  | Yes   | Dataset metadata, samples, statistics, and access rules |
| DataStreams   | Yes  | Yes   | Data stream ingestion and publishing                    |
| Views         | Yes  | Yes   | Saved view definitions                                  |
| Queries       | Yes  | Yes   | NQL query management and execution                      |
| Subscriptions | Yes  | Yes   | Data subscriptions and NQL operations                   |

## Infrastructure

| Resource            | Read | Write | Description                                       |
| ------------------- | ---- | ----- | ------------------------------------------------- |
| ComputePools        | Yes  | Yes   | Compute pool registration and configuration       |
| DataPlanes          | Yes  | Yes   | Data plane configuration and encryption materials |
| Connections         | Yes  | Yes   | External system connections                       |
| Destinations        | Yes  | Yes   | Data delivery destinations                        |
| EncryptionMaterials | —    | Yes   | Encryption key management                         |
| Resources           | Yes  | Yes   | Platform resource buckets                         |

## Applications

| Resource           | Read | Write | Description                                  |
| ------------------ | ---- | ----- | -------------------------------------------- |
| AppInvites         | Yes  | Yes   | App invite creation and lifecycle management |
| Apps               | Yes  | Yes   | Application and interface management         |
| AppProfiles        | Yes  | Yes   | Application profiles                         |
| AppEvents          | —    | Yes   | Application event tracking                   |
| Installations      | Yes  | Yes   | Application installations                    |
| ProfileStoreSchema | Yes  | Yes   | Profile store schema definitions             |
| ProfileStoreData   | —    | Yes   | Profile store data records                   |

## Identity & Access

| Resource     | Read | Write | Description                        |
| ------------ | ---- | ----- | ---------------------------------- |
| AccessTokens | Yes  | Yes   | API key management                 |
| Users        | —    | Yes   | User profile updates               |
| CompanyInfo  | Yes  | Yes   | Organization settings and metadata |

## Automation

| Resource      | Read | Write | Description                       |
| ------------- | ---- | ----- | --------------------------------- |
| Jobs          | Yes  | Yes   | Job execution and status          |
| Workflows     | Yes  | Yes   | Automated pipeline definitions    |
| Webhooks      | Yes  | Yes   | Webhook endpoint configuration    |
| Models        | Yes  | Yes   | AI/ML model definitions           |
| ModelTraining | —    | Yes   | Model training and inference jobs |

## Schema & Mapping

| Resource     | Read | Write | Description                         |
| ------------ | ---- | ----- | ----------------------------------- |
| Mappings     | Yes  | Yes   | Rosetta Stone schema mappings       |
| SchemaPreset | Yes  | Yes   | Predefined schema templates         |
| Attributes   | Yes  | —     | Rosetta Stone normalized attributes |

## Billing

| Resource       | Read | Write | Description                  |
| -------------- | ---- | ----- | ---------------------------- |
| Billings       | Yes  | —     | Billing and storage usage    |
| PaymentMethods | Yes  | Yes   | Payment method configuration |
| Usage          | —    | Yes   | Platform usage reporting     |

## Other

| Resource                        | Read | Write | Description                                       |
| ------------------------------- | ---- | ----- | ------------------------------------------------- |
| Contracts                       | Yes  | —     | Data contract viewing                             |
| Uploads                         | —    | Yes   | File upload operations                            |
| DataPlaneLogs                   | —    | Yes   | Data plane log submission                         |
| Commands.Subscriptions.Delivery | Yes  | —     | Subscription delivery commands and file downloads |


# Classifier Studio interface
Source: https://docs.narrative.io/reference/ui/classifier-studio

Reference for the Classifier Studio UI for training classification models on your data

This reference documents the UI elements, configuration options, and actions available in the Classifier Studio interface.

## Overview

Classifier Studio enables training classification models that categorize and label data within Narrative's platform. It integrates data selection, label configuration, and compute resource allocation into a streamlined workflow.

**Path:** My Models → Classifier Studio

## Entry points

You can open Classifier Studio two ways:

* **Empty builder** — Navigate to **My Models → Classifier Studio** and configure every step from scratch.
* **Pre-seeded from Label Studio** — Clicking **Train Model** in a Label Studio project workspace opens Classifier Studio with the **Dataset**, **Label column**, and a single **Feature** already filled in: the project's label dataset, `label` as the target, and `input_string` as a text feature. The seed reuses Label Studio's existing source NQL expression, so no source-column materialization is needed before training. The handoff is gated on the project having at least 10 labeled rows across at least 2 distinct classes; below those thresholds Label Studio's Train Model action stays disabled.

After a pre-seeded handoff you still pick the algorithm, hyperparameters, test/train split, and finalize the model name and tags — only the first three steps come pre-filled.

<Note>
  Classifier Studio and Label Studio are only available on Snowflake data planes — their seed, write, and training workflows rely on Snowflake-only DML — and the side navigation hides both entries on non-Snowflake planes (for example AWS).
</Note>

## Builder interactions

The builder is a step-by-step flow. Each configured step exposes inline actions so you can revise selections without navigating back through earlier steps:

| Action     | Description                                                                     |
| ---------- | ------------------------------------------------------------------------------- |
| **Edit**   | Re-opens the step's configuration view with your existing selections pre-filled |
| **Remove** | Clears the step's configuration (and any dependent downstream steps)            |

When training is submitted, a compact **toast notification** confirms success and links directly to the **Jobs** page to track progress.

***

## Dataset Selection module

The Dataset Selection module lets you choose the dataset that contains your labeled training data.

| Element                   | Description                                      |
| ------------------------- | ------------------------------------------------ |
| **Select Dataset button** | Opens the dataset selection view                 |
| **Dataset name**          | Displays the currently selected training dataset |

### Dataset selection view

When you click **Select Dataset**, a full selection view appears with:

| Element          | Description                                                                  |
| ---------------- | ---------------------------------------------------------------------------- |
| **Dataset list** | Searchable, virtualized list of available datasets in the current data plane |
| **Next button**  | Proceeds to label column selection (enabled after selecting a dataset)       |
| **Cancel**       | Returns to the builder start view                                            |

<Note>
  Only datasets available in your currently selected data plane are shown.
</Note>

***

## Label Column module

The Label Column module lets you select the column that contains the classification target — the categories your model will learn to predict.

| Element                 | Description                                                                     |
| ----------------------- | ------------------------------------------------------------------------------- |
| **Select Label button** | Opens the label column selection view (requires a dataset to be selected first) |
| **Column name**         | Displays the currently selected label column                                    |

### Label column selection view

| Element             | Description                                                          |
| ------------------- | -------------------------------------------------------------------- |
| **Column dropdown** | Filterable list of primitive-type columns from the selected dataset  |
| **Column type**     | Displays the data type next to each column name                      |
| **Back button**     | Returns to dataset selection                                         |
| **Next button**     | Proceeds to feature configuration (enabled after selecting a column) |

### Supported column types

Only columns with primitive data types are available as label columns:

| Type          | Description            |
| ------------- | ---------------------- |
| `string`      | Text-based categories  |
| `boolean`     | Binary classification  |
| `double`      | Numeric labels         |
| `long`        | Integer labels         |
| `timestamptz` | Timestamp-based labels |

<Tip>
  For best results, choose a column with a balanced distribution of category values in your training data.
</Tip>

***

## Feature Configuration module

The Feature Configuration module lets you define which columns from your dataset serve as input features for the classifier and how they should be processed.

| Element              | Description                                  |
| -------------------- | -------------------------------------------- |
| **Configure button** | Opens the feature configuration view         |
| **Feature list**     | Displays configured features and their types |

### Feature types

| Type               | Description                                              |
| ------------------ | -------------------------------------------------------- |
| `text`             | Free-form text processed via natural language techniques |
| `categorical`      | Discrete categories encoded for model input              |
| `numeric`          | Continuous numeric values                                |
| `count_vectorizer` | Text converted to token frequency vectors                |
| `embedding`        | Pre-computed vector embeddings                           |

***

## Algorithm Configuration module

The Algorithm Configuration module lets you choose the classification algorithm for training.

| Element              | Description                           |
| -------------------- | ------------------------------------- |
| **Configure button** | Opens the algorithm selection view    |
| **Algorithm name**   | Displays the selected classifier type |

### Available algorithms

| Algorithm               | Description                                                                              |
| ----------------------- | ---------------------------------------------------------------------------------------- |
| **Logistic Regression** | Linear model suited for binary and multi-class classification with interpretable results |
| **Random Forest**       | Ensemble method that builds multiple decision trees for robust predictions               |

### Hyperparameters

Each algorithm exposes its own set of tunable hyperparameters with sensible defaults. The configuration view surfaces the parameters relevant to your selected algorithm — for example, regularization strength for Logistic Regression, or tree count and maximum depth for Random Forest — so you can adjust only what matters for your use case.

### Test/train split

Configure how the dataset is partitioned into training and evaluation sets:

| Setting            | Description                                                                                                      |
| ------------------ | ---------------------------------------------------------------------------------------------------------------- |
| **Test size**      | Fraction of rows reserved for evaluation                                                                         |
| **Random state**   | Integer seed that makes the split deterministic across retrains                                                  |
| **Stratification** | When enabled, preserves the label distribution in both the train and test splits — useful for imbalanced classes |

***

## Finalize module

The Finalize module is the last step before training. It lets you name and version the model, attach metadata, confirm the execution environment, and review everything you've configured in a single summary view.

| Element                   | Description                                                                                               |
| ------------------------- | --------------------------------------------------------------------------------------------------------- |
| **Model name**            | Human-readable name for the trained classifier                                                            |
| **Model version**         | Version identifier for this training run, enabling side-by-side comparison of retrains                    |
| **Tags**                  | Keywords for organizing and identifying trained classifiers                                               |
| **Data plane selector**   | Confirm the data plane where training executes                                                            |
| **Configuration summary** | Read-only review of your dataset, label column, features, algorithm, and split settings before submission |

<Info>
  Classifier training runs on Snowflake's built-in ML capabilities within your data plane. Data never leaves your infrastructure.
</Info>

***

## Actions reference

### Configuration actions

| Action                  | Location                       | Description                                                          | Result                                     |
| ----------------------- | ------------------------------ | -------------------------------------------------------------------- | ------------------------------------------ |
| **Select Dataset**      | Dataset Selection module       | Choose training dataset                                              | Dataset selected, columns become available |
| **Select Label**        | Label Column module            | Choose target column                                                 | Classification target defined              |
| **Configure Features**  | Feature Configuration module   | Define input features                                                | Feature columns and types configured       |
| **Configure Algorithm** | Algorithm Configuration module | Choose classifier type, hyperparameters, and split                   | Algorithm selected for training            |
| **Finalize**            | Finalize module                | Name/version the model, add tags, confirm data plane, review summary | Training request fully configured          |

### Training actions

| Action               | Location     | Description                                            | Result                                        |
| -------------------- | ------------ | ------------------------------------------------------ | --------------------------------------------- |
| **Train Classifier** | Page toolbar | Start training (enabled when all steps are configured) | Training job submitted and progress displayed |

***

## Training output

After you click **Train Classifier**:

* A training job is submitted to the selected data plane
* A success confirmation appears when the job is accepted
* Monitor training progress on the Jobs page
* The trained classifier becomes available for use in your data workflows

***

## Workflow summary

1. **Select dataset** → Choose the training dataset in the Dataset Selection module
2. **Select label column** → Pick the column containing classification labels
3. **Configure features** → Define input features and their processing types
4. **Configure algorithm** → Choose between logistic regression and random forest, tune hyperparameters, and set the test/train split
5. **Finalize** → Name and version the model, add tags, confirm the data plane, and review the configuration summary
6. **Train classifier** → Click **Train Classifier** and monitor progress from the Jobs page

***

## Related content

<CardGroup>
  <Card title="LLM Studio" icon="brain-circuit" href="/reference/ui/llm-studio">
    Train and fine-tune LLM models using prepared datasets
  </Card>

  <Card title="Label Studio" icon="tag" href="/reference/ui/classifier-studio#entry-points">
    Build labeling projects whose **Train Model** action seeds Classifier Studio
  </Card>

  <Card title="AI enrichment with NQL" icon="wand-magic-sparkles" href="/cookbooks/nql/ai-enrichment">
    Use AI functions in NQL queries for data classification and enrichment
  </Card>

  <Card title="Model Inference" icon="brain" href="/concepts/model-inference/overview">
    Run inference using AI models within your data plane
  </Card>

  <Card title="Datasets" icon="database" href="/concepts/primitives/datasets">
    Understanding datasets in Narrative
  </Card>
</CardGroup>


# LLM Studio Interface
Source: https://docs.narrative.io/reference/ui/llm-studio

Reference for the LLM Studio UI for training and fine-tuning AI models

This reference documents the UI elements, configuration options, and actions available in the LLM Studio interface.

## Overview

LLM Studio enables training and fine-tuning of AI models using datasets within Narrative's platform. It integrates datasets, base models, and compute resources into a streamlined workflow.

**Path:** My Models → LLM Studio

***

## Base Model module

The Base Model module lets you select the foundation model for fine-tuning.

| Element           | Description                                |
| ----------------- | ------------------------------------------ |
| **Select button** | Opens the model selection dialog           |
| **Model name**    | Displays the currently selected base model |
| **Model details** | Shows model size and capabilities          |

### Available base models

| Model               | Description                                  |
| ------------------- | -------------------------------------------- |
| **Llama-3.2-1B**    | Meta's lightweight 1 billion parameter model |
| **Mistral-7b-v0.1** | Mistral AI's 7 billion parameter model       |

<Note>
  Additional base models may be available. Check the model selection dialog for the current list.
</Note>

***

## Training Data module

The Training Data module lets you select the dataset to use for fine-tuning.

| Element           | Description                                          |
| ----------------- | ---------------------------------------------------- |
| **Select button** | Opens the dataset selection dialog                   |
| **Dataset name**  | Displays the currently selected training dataset     |
| **Row count**     | Shows the number of training examples in the dataset |

### Dataset requirements

Datasets must be mapped to a supported attribute and materialized in the corresponding format before use in LLM Studio.

| Attribute                      | Format                 | Description                                                                           |
| ------------------------------ | ---------------------- | ------------------------------------------------------------------------------------- |
| **fine\_tuning\_conversation** | Conversation structure | Each row contains a structured conversation with system, user, and assistant messages |

<Info>
  Use [Prompt Studio](/reference/ui/prompt-studio) to transform datasets into the `fine_tuning_conversation` format.
</Info>

### Accessing training data with NQL

To query conversation data from a prepared dataset:

```sql theme={null}
SELECT
    d._rosetta_stone.fine_tuning_conversation.conversation
FROM company_data.my_dataset_name d
```

<Note>
  Additional fine-tuning attributes will be supported in future updates.
</Note>

***

## Compute module

The Compute module lets you configure the compute resources for training.

| Element               | Description                                 |
| --------------------- | ------------------------------------------- |
| **Select button**     | Opens the compute instance selection dialog |
| **Instance type**     | Displays the selected compute configuration |
| **GPU configuration** | Shows GPU count and type                    |

### Compute instance selection

Choose an instance based on your training requirements:

| Factor            | Consideration                                      |
| ----------------- | -------------------------------------------------- |
| **Model size**    | Larger models require more GPU memory              |
| **Dataset size**  | Larger datasets benefit from more compute capacity |
| **Training time** | Higher-tier instances reduce training duration     |

Available instances include AWS G5 instances with various GPU configurations.

***

## Trained Model Details module

The Trained Model Details module captures metadata for the fine-tuned model.

| Element         | Description                                            |
| --------------- | ------------------------------------------------------ |
| **Add button**  | Opens the metadata configuration dialog                |
| **Edit button** | Modify existing metadata (after initial configuration) |

### Metadata fields

| Field           | Required | Description                                          |
| --------------- | -------- | ---------------------------------------------------- |
| **Unique Name** | Yes      | Identifier for the trained model                     |
| **Description** | No       | Purpose or use case for the model                    |
| **Tags**        | No       | Keywords for identification and categorization       |
| **License**     | No       | License under which the model will be shared or used |

***

## Actions reference

### Configuration actions

| Action                   | Location                     | Description               | Result                           |
| ------------------------ | ---------------------------- | ------------------------- | -------------------------------- |
| **Select base model**    | Base Model module            | Choose foundation model   | Model selected for fine-tuning   |
| **Select training data** | Training Data module         | Choose prepared dataset   | Dataset linked to training job   |
| **Select compute**       | Compute module               | Choose compute resources  | Instance allocated for training  |
| **Add model details**    | Trained Model Details module | Configure output metadata | Metadata saved for trained model |

### Training actions

| Action          | Location     | Description       | Result                                        |
| --------------- | ------------ | ----------------- | --------------------------------------------- |
| **Train Model** | Page toolbar | Initiate training | Training job starts and progress is displayed |

***

## Training output

Once training completes, the fine-tuned model is available with:

* The configured metadata (name, description, tags, license)
* Full compatibility with the training dataset format
* Readiness for deployment or inference

***

## Workflow summary

1. **Select base model** → Choose the foundation model in the Base Model module
2. **Select training data** → Choose a prepared dataset in the Training Data module
3. **Configure compute** → Select appropriate compute resources
4. **Add metadata** → Provide model name, description, tags, and license
5. **Train model** → Click Train Model and monitor progress

***

## Related content

<CardGroup>
  <Card title="Prompt Studio" icon="message-bot" href="/reference/ui/prompt-studio">
    Prepare datasets for fine-tuning with conversation formatting
  </Card>

  <Card title="Model Inference" icon="brain" href="/concepts/model-inference/overview">
    Run inference using trained models within your data plane
  </Card>

  <Card title="Supported Models" icon="list" href="/reference/model-inference/supported-models">
    Reference for available AI models
  </Card>

  <Card title="Datasets" icon="database" href="/concepts/primitives/datasets">
    Understanding datasets in Narrative
  </Card>
</CardGroup>


# Prompt Studio Interface
Source: https://docs.narrative.io/reference/ui/prompt-studio

Reference for the Prompt Studio UI for creating fine-tuning-ready datasets

This reference documents the UI elements, configuration options, and actions available in the Prompt Studio interface.

## Overview

Prompt Studio transforms datasets into structured, fine-tuning-ready examples for AI models. Each row in the output dataset represents a valid training example formatted as a conversation with system, user, and assistant messages.

**Path:** My Models → Prompt Studio

***

## Prompt Builder module

The Prompt Builder is the main workspace for configuring how dataset rows transform into conversation-formatted training examples.

### Dataset selector

| Element              | Description                                      |
| -------------------- | ------------------------------------------------ |
| **Dataset dropdown** | Select the source dataset to transform           |
| **Row count**        | Displays total rows available for transformation |

The selected dataset provides the fields available for macro substitution in prompts.

### Role configuration panels

Three panels allow you to configure prompts for each conversation role:

| Role          | Purpose                            | Typical content                               |
| ------------- | ---------------------------------- | --------------------------------------------- |
| **System**    | Sets the AI's behavior and context | Instructions, persona definition, constraints |
| **User**      | Represents the human input         | Questions, requests, data to analyze          |
| **Assistant** | Represents the AI response         | Expected output format, example responses     |

Each panel contains:

| Element              | Description                                            |
| -------------------- | ------------------------------------------------------ |
| **Select button**    | Opens the prompt editor for that role                  |
| **Prompt text area** | Displays the configured prompt with macro placeholders |
| **Macro indicators** | Visual badges showing embedded macros                  |

***

## Prompt editor

The prompt editor opens when you click **Select** on any role panel.

### Text input area

| Element                      | Description                                   |
| ---------------------------- | --------------------------------------------- |
| **Prompt text field**        | Multi-line text area for writing the prompt   |
| **Macro placeholder syntax** | Use `{{macro_name}}` to insert dynamic values |
| **Character count**          | Displays current prompt length                |

### Macro configuration

After adding macro placeholders to your prompt text, configure each macro's data source:

| Macro type  | Description                                                       | Use case                                               |
| ----------- | ----------------------------------------------------------------- | ------------------------------------------------------ |
| **Field**   | Maps to a column in the source dataset                            | Pull values directly from dataset columns              |
| **NQL**     | Uses a [Narrative Query Language](/nql/general/syntax) expression | Compute derived values, transform data, combine fields |
| **Literal** | Fixed text value                                                  | Static content that doesn't change per row             |

### Macro configuration panel

| Element                 | Description                                           |
| ----------------------- | ----------------------------------------------------- |
| **Macro name**          | The identifier used in `{{macro_name}}` syntax        |
| **Type selector**       | Choose between Field, NQL, or Literal                 |
| **Value configuration** | Field picker, NQL editor, or text input based on type |
| **Preview value**       | Sample resolved value from the first dataset row      |

### Editor actions

| Action     | Description                                                  |
| ---------- | ------------------------------------------------------------ |
| **Save**   | Saves the prompt configuration and returns to Prompt Builder |
| **Cancel** | Discards changes and returns to Prompt Builder               |

***

## Preview screen

The Preview screen validates your prompt configuration by showing resolved output for each dataset row.

**Path:** Click **Preview** in the Prompt Builder

### Conversation display

| Element               | Description                                           |
| --------------------- | ----------------------------------------------------- |
| **System message**    | Resolved system prompt with all macros substituted    |
| **User message**      | Resolved user prompt with all macros substituted      |
| **Assistant message** | Resolved assistant prompt with all macros substituted |

Messages display in a conversational format, visually distinguishing each role.

### Navigation controls

| Control           | Description                                                 |
| ----------------- | ----------------------------------------------------------- |
| **Prev**          | Navigate to the previous dataset row                        |
| **Next**          | Navigate to the next dataset row                            |
| **Row indicator** | Shows current row number and total (e.g., "Row 5 of 1,234") |

### Validation indicators

| Indicator           | Meaning                                                |
| ------------------- | ------------------------------------------------------ |
| **Green checkmark** | All macros resolved successfully                       |
| **Yellow warning**  | Some macros returned empty or null values              |
| **Red error**       | Macro resolution failed (e.g., invalid NQL expression) |

***

## Output format

The transformed dataset contains rows formatted as structured conversations:

```json theme={null}
{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant..." },
    { "role": "user", "content": "Analyze this customer data..." },
    { "role": "assistant", "content": "Based on the data provided..." }
  ]
}
```

This format is compatible with fine-tuning APIs from major AI model providers.

***

## Actions reference

### Builder actions

| Action             | Location               | Description                   | Result                                     |
| ------------------ | ---------------------- | ----------------------------- | ------------------------------------------ |
| **Select dataset** | Dataset module         | Choose source dataset         | Dataset fields become available for macros |
| **Configure role** | Role panel             | Open prompt editor for a role | Prompt editor opens                        |
| **Preview**        | Prompt Builder toolbar | Validate prompt resolution    | Preview screen opens                       |

### Editor actions

| Action              | Location       | Description               | Result                      |
| ------------------- | -------------- | ------------------------- | --------------------------- |
| **Add macro**       | Prompt text    | Type `{{name}}` syntax    | Macro placeholder created   |
| **Configure macro** | Macro panel    | Set type and value        | Macro resolution configured |
| **Save**            | Editor toolbar | Save prompt configuration | Returns to Prompt Builder   |

### Preview actions

| Action                | Location            | Description                 | Result                                 |
| --------------------- | ------------------- | --------------------------- | -------------------------------------- |
| **Navigate rows**     | Navigation controls | View different dataset rows | Conversation updates with new row data |
| **Return to builder** | Preview toolbar     | Exit preview mode           | Returns to Prompt Builder              |

### Output actions

| Action               | Location               | Description                 | Result                                      |
| -------------------- | ---------------------- | --------------------------- | ------------------------------------------- |
| **Generate dataset** | Prompt Builder toolbar | Create materialized dataset | Opens Data Studio with prompt configuration |

***

## Workflow summary

1. **Select dataset** → Choose source data in the Dataset module
2. **Configure prompts** → Define system, user, and assistant prompts with macros
3. **Configure macros** → Map each macro to a field, NQL expression, or literal value
4. **Preview output** → Validate resolved prompts row by row
5. **Generate dataset** → Use Data Studio to materialize the transformed dataset

***

## Related content

<CardGroup>
  <Card title="LLM Studio" icon="brain-circuit" href="/reference/ui/llm-studio">
    Train and fine-tune models using prepared datasets
  </Card>

  <Card title="NQL Syntax" icon="code" href="/nql/general/syntax">
    Reference for NQL expressions used in macros
  </Card>

  <Card title="Model Inference" icon="brain" href="/concepts/model-inference/overview">
    Using AI models within your data plane
  </Card>

  <Card title="Creating Materialized Views" icon="layer-group" href="/guides/nql/creating-materialized-views">
    Generate datasets from your prompt configurations
  </Card>
</CardGroup>


# Webhook Event Reference
Source: https://docs.narrative.io/reference/webhooks/event-reference

Complete field reference for webhook subscriptions, event payloads, and supported enum values

This reference documents all fields, types, and enum values for the webhook subscription system. For a conceptual overview, see [Webhooks](/concepts/webhooks/overview). For step-by-step usage, see [Subscribing to Notifications](/guides/webhooks/subscribing-to-notifications).

## Subscription response fields

When you create or retrieve a webhook subscription, the response contains these fields.

| Field       | Type                    | Required | Description                                                               |
| ----------- | ----------------------- | -------- | ------------------------------------------------------------------------- |
| `id`        | string (UUID)           | Yes      | Unique identifier for the subscription                                    |
| `secret`    | string (UUID)           | Yes      | Shared secret for verifying webhook deliveries. Only returned on creation |
| `status`    | string                  | Yes      | Subscription status: `active` or `archived`                               |
| `companyId` | integer                 | Yes      | ID of the company that owns the subscription                              |
| `jobIds`    | array of strings (UUID) | No       | Job IDs this subscription is filtered to                                  |
| `jobStates` | array of strings        | No       | Job states this subscription is filtered to                               |
| `jobTypes`  | array of strings        | No       | Job types this subscription is filtered to                                |
| `url`       | string (URI)            | Yes      | HTTPS endpoint that receives event payloads                               |
| `createdAt` | string (date-time)      | Yes      | When the subscription was created                                         |
| `updatedAt` | string (date-time)      | Yes      | When the subscription was last updated                                    |

**Example response:**

```json theme={null}
{
  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "secret": "8b6dba8c-9a8b-4a3a-8469-9dc7a33c3e17",
  "status": "active",
  "companyId": 42,
  "jobIds": ["d35a12c1-7a2f-46b9-8d60-9e0a2a84c205"],
  "jobStates": ["completed", "failed"],
  "jobTypes": ["materialize-view"],
  "url": "https://your-app.example.com/webhooks/narrative",
  "createdAt": "2025-04-29T16:12:45.123456",
  "updatedAt": "2025-04-29T16:12:45.123456"
}
```

***

## Job webhook request fields

To create a job webhook subscription, send a `POST /webhooks` request with these fields.

| Field       | Type                    | Required | Description                                           |
| ----------- | ----------------------- | -------- | ----------------------------------------------------- |
| `type`      | string                  | Yes      | Must be `webhook_subscription_jobs`                   |
| `url`       | string (URI)            | Yes      | HTTPS endpoint to receive events                      |
| `name`      | string                  | No       | Human-readable name for the subscription              |
| `job_ids`   | array of strings (UUID) | No       | Specific job IDs to monitor                           |
| `job_types` | array of strings        | No       | Job types to monitor (see [Job types](#job-types))    |
| `states`    | array of strings        | No       | Job states to monitor (see [Job states](#job-states)) |

**Example request:**

```json theme={null}
{
  "type": "webhook_subscription_jobs",
  "url": "https://your-app.example.com/webhooks/narrative",
  "name": "failed_job_alerts",
  "job_types": ["materialize-view", "forecast"],
  "states": ["failed"]
}
```

<Note>
  If you omit all filter fields (`job_ids`, `job_types`, `states`), the subscription fires for every job state change in your company.
</Note>

***

## Job types

The `job_types` filter accepts these values. For detailed descriptions of each type, see [Job Types](/reference/architecture/job-types).

| Value                             | Description                            |
| --------------------------------- | -------------------------------------- |
| `costs`                           | Cost calculation job                   |
| `datasets_calculate_column_stats` | Column statistics computation          |
| `datasets_delete_table`           | Dataset table deletion                 |
| `datasets_deliver_data`           | Data delivery to external destinations |
| `datasets_sample`                 | Dataset sampling                       |
| `datasets_suggest_mappings`       | AI-powered mapping suggestions         |
| `delete-snowflake-table`          | Snowflake table deletion               |
| `forecast`                        | Query cost forecasting                 |
| `materialize-view`                | Materialized view creation or refresh  |
| `models_deliver_model`            | Model delivery                         |
| `model_training_run`              | Model training execution               |
| `nql-forecast`                    | NQL query forecasting                  |
| `upload-stats`                    | Upload statistics computation          |

***

## Job states

The `states` filter accepts these values. Jobs progress through these states during their lifecycle.

| Value                  | Description                                      |
| ---------------------- | ------------------------------------------------ |
| `pending`              | Job is queued and waiting for execution          |
| `running`              | Job is currently being executed                  |
| `completed`            | Job finished successfully                        |
| `pending_cancellation` | Job is marked for cancellation but still running |
| `cancelled`            | Job was cancelled before completion              |
| `failed`               | Job execution failed                             |

For more details on job state transitions, see [Tracking Job Status](/guides/sdk/tracking-jobs).

***

## Subscription status values

| Value      | Description                                            |
| ---------- | ------------------------------------------------------ |
| `active`   | Subscription is live and receiving events              |
| `archived` | Subscription is disabled and no longer receives events |

***

## API endpoints

| Method   | Path                                  | Description                                                                            |
| -------- | ------------------------------------- | -------------------------------------------------------------------------------------- |
| `GET`    | `/webhooks`                           | [List all webhook subscriptions](/api-reference/webhooks/list-webhook-subscriptions)   |
| `POST`   | `/webhooks`                           | [Create a webhook subscription](/api-reference/webhooks/create-a-webhook-subscription) |
| `GET`    | `/webhooks/{webhook_subscription_id}` | [Get a subscription by ID](/api-reference/webhooks/get-a-webhook-subscription-by-id)   |
| `DELETE` | `/webhooks/{webhook_subscription_id}` | [Archive a subscription](/api-reference/webhooks/archive-a-webhook-subscription)       |

***

## Related content

<CardGroup>
  <Card title="Webhooks" icon="bell" href="/concepts/webhooks/overview">
    How webhooks work and when to use them
  </Card>

  <Card title="Subscribing to Notifications" icon="bell" href="/guides/webhooks/subscribing-to-notifications">
    Step-by-step guide to creating and managing subscriptions
  </Card>

  <Card title="Job Types" icon="list" href="/reference/architecture/job-types">
    Detailed descriptions of all job types
  </Card>

  <Card title="Webhooks API" icon="code" href="/api-reference/webhooks/list-webhook-subscriptions">
    Full REST API documentation for webhook endpoints
  </Card>
</CardGroup>


# Workflow Specification Syntax
Source: https://docs.narrative.io/reference/workflows/specification-syntax

Complete reference for the workflow YAML specification format and supported tasks

Workflows are defined using the [Serverless Workflow DSL](https://github.com/serverlessworkflow/specification/blob/main/dsl-reference.md) in YAML format. This reference covers the specification structure. For the full list of available tasks, see the [Task Reference](/reference/workflows/tasks).

## Specification structure

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: <namespace>
  name: <workflow-name>
  version: '<version>'
schedule:                          # optional
  cron: '<cron-expression>'
do:
  - <taskName>:
      call: <TaskName>
      with:
        <param>: <value>
      export:                      # optional
        as: '<jq-expression>'
```

## Document fields

The `document` block defines workflow metadata.

| Field       | Type   | Required | Description                                                  |
| ----------- | ------ | -------- | ------------------------------------------------------------ |
| `dsl`       | string | Yes      | DSL version. Use `'1.0.0'`                                   |
| `namespace` | string | Yes      | Logical grouping for the workflow (e.g., `analytics`, `etl`) |
| `name`      | string | Yes      | Unique identifier for the workflow within its namespace      |
| `version`   | string | Yes      | Version string for tracking changes (e.g., `'1.0.0'`)        |

**Example:**

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: etl
  name: daily-refresh-pipeline
  version: '2.1.0'
```

***

## Schedule

The optional `schedule` block configures automatic execution.

| Field  | Type   | Required                | Description                                     |
| ------ | ------ | ----------------------- | ----------------------------------------------- |
| `cron` | string | Yes (within `schedule`) | CRON expression defining when the workflow runs |

**CRON format:**

```
┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-6, Sunday=0)
│ │ │ │ │
* * * * *
```

**Common expressions:**

| Expression    | Schedule                           |
| ------------- | ---------------------------------- |
| `0 * * * *`   | Every hour                         |
| `0 0 * * *`   | Daily at midnight UTC              |
| `0 0 * * 0`   | Weekly on Sunday at midnight UTC   |
| `0 0 1 * *`   | Monthly on the 1st at midnight UTC |
| `15 15 * * 1` | Every Monday at 3:15 PM UTC        |
| `0 2 1 * *`   | 1st of each month at 2:00 AM UTC   |

**Example:**

```yaml theme={null}
schedule:
  cron: '0 0 * * *'
```

<Note>
  Schedules are activated separately from workflow creation. Use the [Schedule a workflow](/api-reference/workflows/schedule-a-workflow) endpoint or set `schedule_immediately: true` when creating the workflow.
</Note>

***

## Task structure

The `do` block contains an ordered list of tasks. Each task is a YAML mapping with a unique name as the key.

| Field    | Type   | Required | Description                                                                           |
| -------- | ------ | -------- | ------------------------------------------------------------------------------------- |
| `call`   | string | Yes      | The task to execute. Must be one of the [supported tasks](/reference/workflows/tasks) |
| `with`   | object | Yes      | Parameters for the task. Fields vary by task                                          |
| `export` | object | No       | Controls how task output is merged into the workflow context. See [Export](#export)   |

Tasks execute sequentially in the order they appear. Each task waits for the previous one to complete before starting.

**Example:**

```yaml theme={null}
do:
  - firstTask:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: 'CREATE MATERIALIZED VIEW user_list AS SELECT id, name FROM users'
  - secondTask:
      call: RefreshMaterializedView
      with:
        datasetName: user_list
```

***

## Task output

Every task produces a JSON object after execution. The output fields use camelCase names and vary by task — see the [Task Reference](/reference/workflows/tasks) for per-task output schemas.

Within an `export.as` expression, `.` (dot) refers to the current task's output. In `${…}` variable expressions in the *next* task's parameters, `.` also refers to the immediately preceding task's output.

***

## Export

The optional `export` block controls how a task's output is merged into the **workflow context** — an object (`$context`) that accumulates data as the workflow runs. The context starts as an empty object `{}` and is threaded through every task.

| Field       | Type                   | Description                                               |
| ----------- | ---------------------- | --------------------------------------------------------- |
| `export.as` | string (jq expression) | A jq expression that produces the new value of `$context` |

**Available variables in `export.as`:**

| Variable   | Description                                       |
| ---------- | ------------------------------------------------- |
| `.`        | The current task's output                         |
| `$context` | The accumulated workflow context before this task |

When `export` is omitted, `$context` is unchanged.

**Examples:**

Store the entire task output as context:

```yaml theme={null}
export:
  as: '.'
```

Merge a single field into the existing context:

```yaml theme={null}
export:
  as: '$context + { datasetId: .datasetId }'
```

Build a running list of dataset IDs:

```yaml theme={null}
export:
  as: '$context + { datasetIds: (($context.datasetIds // []) + [.datasetId]) }'
```

***

## Variable expressions

Task parameters support `${…}` variable expressions that inject values from previous task output or the workflow context. Variable expressions are evaluated before the task executes.

**Syntax rules:**

| Pattern                         | Behavior                                         | Example                                                       |
| ------------------------------- | ------------------------------------------------ | ------------------------------------------------------------- |
| `${<expr>}` as the entire value | Preserves the JSON type of the expression result | `${.datasetId}` → integer `42`                                |
| `${" ... "}` or mixed with text | Result is always a string                        | `${"Dataset \(.datasetId) created"}` → `"Dataset 42 created"` |

**Available variables in `${…}`:**

| Variable   | Description                             |
| ---------- | --------------------------------------- |
| `.`        | The immediately preceding task's output |
| `$context` | The accumulated workflow context        |

Use jq string interpolation `\(expr)` to embed values inside a string expression:

```yaml theme={null}
nql: |
  ${"INSERT INTO company_data.log (id) VALUES (\(.datasetId))"}
```

***

## Supported tasks

The following tasks are available in workflows. See the [Task Reference](/reference/workflows/tasks) for full parameter details, constraints, and examples.

| Task                                   | Description                                                                                                                                                                                                        |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CreateMaterializedViewIfNotExists`    | Task that creates a materialized view if it does not already exist.                                                                                                                                                |
| `RefreshMaterializedView`              | Task that triggers a refresh of an existing materialized view.                                                                                                                                                     |
| `ExecuteDml`                           | Task that executes a DML statement on a dataset.                                                                                                                                                                   |
| `RunModelInference`                    | Task that runs a model inference job.                                                                                                                                                                              |
| `LabelConnectedComponents`             | Task that runs bipartite label propagation for cross-system customer identity resolution. Finds connected components in a customer identity graph by linking customer IDs across platforms via shared identifiers. |
| `CreateRosettaStoneMappingsIfNotExist` | Task that creates Rosetta Stone attribute mappings for a dataset.                                                                                                                                                  |
| `CreateDatasetSample`                  | Task that generates a sample for a dataset.                                                                                                                                                                        |
| `RecalculateStatistics`                | Task that triggers a recalculation of a dataset's column statistics and waits for it to complete.                                                                                                                  |

***

## Complete example

A multi-step workflow that creates two materialized views and refreshes one, running daily:

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: analytics
  name: daily-user-pipeline
  version: '1.0.0'
schedule:
  cron: '0 0 * * *'
do:
  - createActiveUsers:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: |
          CREATE MATERIALIZED VIEW active_users AS
          SELECT
            id,
            name,
            email
          FROM users
          WHERE active = true
  - createUserMetrics:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: |
          CREATE MATERIALIZED VIEW active_users_with_names AS
          SELECT
            id,
            name
          FROM company_data.active_users
          WHERE name IS NOT NULL
  - refreshMetrics:
      call: RefreshMaterializedView
      with:
        datasetName: daily_kpis
```

### Example with data passing

A workflow that creates a view, captures its dataset ID, and logs the result:

```yaml theme={null}
document:
  dsl: '1.0.0'
  namespace: etl
  name: create-and-log
  version: '1.0.0'
do:
  - createView:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: "CREATE MATERIALIZED VIEW active_users AS SELECT user_id, email FROM company_data.users WHERE is_active = true"
      export:
        as: '$context + { datasetId: .datasetId }'
  - logCreatedDataset:
      call: ExecuteDml
      with:
        nql: |
          ${"INSERT INTO company_data.pipeline_log (dataset_id, status) VALUES (\(.datasetId), 'created')"}
  - logFromContext:
      call: ExecuteDml
      with:
        nql: |
          ${"INSERT INTO company_data.pipeline_log (dataset_id, status) VALUES (\($context.datasetId), 'created')"}
```

In this example:

1. `createView` creates a materialized view and exports `datasetId` into `$context.datasetId`
2. `logCreatedDataset` uses `.datasetId` (the previous task's direct output) in a variable expression
3. `logFromContext` uses `$context.datasetId` to access the same value via the workflow context

***

## Related content

<CardGroup>
  <Card title="Task Reference" icon="rectangle-terminal" href="/reference/workflows/tasks">
    Full reference for all supported tasks
  </Card>

  <Card title="Automating Multi-Step Pipelines" icon="diagram-project" href="/guides/workflows/workflow-orchestration">
    Step-by-step guide to creating and running workflows
  </Card>

  <Card title="Workflow Orchestration" icon="gears" href="/concepts/workflows/workflow-orchestration">
    How workflows work and why they're designed this way
  </Card>

  <Card title="Workflows API" icon="code" href="/api-reference/workflows/list-workflows">
    REST API endpoints for managing workflows
  </Card>
</CardGroup>


# Task Reference
Source: https://docs.narrative.io/reference/workflows/tasks

Complete reference for supported workflow tasks and their parameters

Each task in a workflow's `do` block calls a supported task. This reference documents every available task, its parameters, output schema, and usage examples.

For how tasks fit into the overall workflow specification, see [Workflow Specification Syntax](/reference/workflows/specification-syntax#task-structure).

## Supported tasks

### CreateMaterializedViewIfNotExists

Task that creates a materialized view if it does not already exist.

**Parameters:**

| Parameter       | Type   | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nql`           | string | Yes      | An NQL `CREATE MATERIALIZED VIEW` statement.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `computePoolId` | string | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Output:**

| Field             | Type            | Always present | Description                                                                                                                                                                                                                                                   |
| ----------------- | --------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`       | integer         | No             | The ID of the created or existing dataset.                                                                                                                                                                                                                    |
| `created`         | boolean         | No             | Whether the materialized view was newly created by this task.                                                                                                                                                                                                 |
| `snapshotId`      | integer or null | No             | The Iceberg snapshot ID of the initial refresh. Non-null only when `created` is `true`.                                                                                                                                                                       |
| `recalculationId` | string or null  | No             | The recalculation ID, if applicable. Non-null only when `created` is `true`.                                                                                                                                                                                  |
| `rowStats`        | object or null  | No             | Row-level statistics produced by a refresh.  ## Platform behavior  - **Snowflake** dataplanes populate this object with real counts. - **AWS** dataplanes return `null` — row-level statistics are not yet   produced for materialized-view refreshes on AWS. |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: analytics
  name: create-active-users-view
  version: 1.0.0
do:
  - createView:
      call: CreateMaterializedViewIfNotExists
      with:
        nql: CREATE MATERIALIZED VIEW active_users AS SELECT user_id, email, last_login FROM company_data.users WHERE is_active = true
```

***

### RefreshMaterializedView

Task that triggers a refresh of an existing materialized view.

**Parameters:**

| Parameter              | Type    | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ---------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`            | integer | No       | The numeric id of an existing dataset.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `datasetName`          | string  | No       | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `computePoolId`        | string  | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |
| `deltaBoundsOverrides` | string  | No       | Overrides the lower bound used to read each delta source during this refresh. When omitted, delta bounds are computed automatically from the dataset's tracked state.  - `{ type: for_all, value: <ISO-8601 timestamp> }` applies that one lower   bound to every currently-tracked delta source. - `{ type: for_all, value: null }` clears the tracked bounds, forcing a   full reprocess of every source. - The whole field may instead be a `${…}` jq expression that resolves to   either of the above forms (or to `null`).                                                                  |

**Output:**

| Field             | Type           | Always present | Description                                                                                                                                                                                                                                                   |
| ----------------- | -------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`       | integer        | No             | The ID of the refreshed dataset.                                                                                                                                                                                                                              |
| `snapshotId`      | integer        | No             | The new Iceberg snapshot ID after the refresh.                                                                                                                                                                                                                |
| `recalculationId` | string or null | No             | The recalculation ID, if applicable.                                                                                                                                                                                                                          |
| `rowStats`        | object or null | No             | Row-level statistics produced by a refresh.  ## Platform behavior  - **Snowflake** dataplanes populate this object with real counts. - **AWS** dataplanes return `null` — row-level statistics are not yet   produced for materialized-view refreshes on AWS. |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: analytics
  name: refresh-active-users
  version: 1.0.0
do:
  - refreshView:
      call: RefreshMaterializedView
      with:
        datasetName: active_users
```

***

### ExecuteDml

Task that executes a DML statement on a dataset.

**Parameters:**

| Parameter       | Type   | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `nql`           | string | Yes      | An NQL DML statement. Supports `INSERT`, `UPDATE`, and `DELETE`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `computePoolId` | string | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Output:**

| Field          | Type    | Always present | Description                                                          |
| -------------- | ------- | -------------- | -------------------------------------------------------------------- |
| `affectedRows` | integer | Yes            | Total rows affected by the DML statement (insert + update + delete). |
| `insertedRows` | integer | Yes            | Rows inserted by the DML statement.                                  |
| `updatedRows`  | integer | Yes            | Rows updated by the DML statement.                                   |
| `deletedRows`  | integer | Yes            | Rows deleted by the DML statement.                                   |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: etl
  name: insert-audit-record
  version: 1.0.0
do:
  - insertAudit:
      call: ExecuteDml
      with:
        nql: INSERT INTO company_data.audit_log (action, timestamp) VALUES ('manual_run', CURRENT_TIMESTAMP)
```

***

### RunModelInference

Task that runs a model inference job.

**Parameters:**

| Parameter         | Type                                                                                                                                                       | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`           | enum (`anthropic.claude-haiku-4.5`, `anthropic.claude-sonnet-4.5`, `anthropic.claude-opus-4.5`, `openai.gpt-oss-120b`, `openai.gpt-4.1`, `openai.o4-mini`) | Yes      | The narrative model ID to use for inference.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `messages`        | array                                                                                                                                                      | Yes      | A list of messages to send to the model.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `inferenceConfig` | object                                                                                                                                                     | Yes      | Configuration for the model inference.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `computePoolId`   | string                                                                                                                                                     | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Output:**

| Field              | Type   | Always present | Description                                                                          |
| ------------------ | ------ | -------------- | ------------------------------------------------------------------------------------ |
| `structuredOutput` | object | No             | The structured output from the model, conforming to the provided outputFormatSchema. |
| `usage`            | object | No             | Token usage information.                                                             |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: ml
  name: classify-records
  version: 1.0.0
do:
  - classify:
      call: RunModelInference
      with:
        model: anthropic.claude-sonnet-4.5
        messages:
          - role: user
            text: 'Classify the following record as spam or not spam: ...'
        inferenceConfig:
          outputFormatSchema:
            type: object
            properties:
              classification:
                type: string
                enum:
                  - spam
                  - not_spam
            required:
              - classification
```

***

### LabelConnectedComponents

Task that runs bipartite label propagation for cross-system customer
identity resolution. Finds connected components in a customer identity
graph by linking customer IDs across platforms via shared identifiers.

**Parameters:**

| Parameter                | Type    | Required | Default    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------------ | ------- | -------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `edgeDataset`            | string  | Yes      | —          | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `outputDataset`          | string  | Yes      | —          | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `maxDegreeThreshold`     | integer | No       | `100`      | Maximum number of connections a single vertex can have before it is excluded as a "supernode." Prevents a single overly-connected identifier from incorrectly merging thousands of unrelated customers.                                                                                                                                                                                                                                                                                                                                                                                           |
| `maxComponentSize`       | integer | No       | `100`      | Maximum number of members allowed in a single resolved component. Prevents runaway merges that would create implausibly large identity groups.                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `maxIterations`          | integer | No       | `10`       | Upper bound on how many times the label propagation loop can run before stopping, even if not fully converged. Safety valve against infinite loops.                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `convergenceThreshold`   | number  | No       | `0.000001` | Stop label propagation when the fraction of vertices that changed label in an iteration drops below this value. Must be in the range `[0, 1]`.                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `sourceIdCol`            | string  | Yes      | —          | Column name in the edge table containing the customer ID.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `sourceSystemCol`        | string  | Yes      | —          | Column name identifying which platform the customer ID came from.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `bridgeKeyCol`           | string  | Yes      | —          | Column name for the shared identifier value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `bridgeKeyTypeCol`       | string  | Yes      | —          | Column name for the type/category of the shared identifier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `firstPartySources`      | array   | No       | `[]`       | Ordered list of first-party platform identifiers. Order determines priority when selecting the representative component ID.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `thirdPartySources`      | array   | No       | `[]`       | List of third-party platform identifiers.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `exclusiveAttributeCols` | array   | No       | `[]`       | Edge-table columns whose values are exclusive to one identity (e.g. an SSN token column). After label propagation converges, any component containing two or more distinct non-null values of any listed column is split so that each resulting component agrees on every listed column. Records without a value follow their edges into whichever fragment they are still connected to (ties resolved deterministically by minimum label). Empty by default (no splitting).                                                                                                                      |
| `computePoolId`          | string  | No       | —          | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Output:**

| Field       | Type    | Always present | Description            |
| ----------- | ------- | -------------- | ---------------------- |
| `datasetId` | integer | Yes            | The ID of the dataset. |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: identity
  name: resolve-connected-components
  version: 1.0.0
do:
  - resolveIdentities:
      call: LabelConnectedComponents
      with:
        edgeDataset: edge_table
        outputDataset: connected_components_result
        maxDegreeThreshold: 500
        maxComponentSize: 10000
        maxIterations: 50
        sourceIdCol: customer_id
        sourceSystemCol: source_system
        bridgeKeyCol: bridge_key
        bridgeKeyTypeCol: bridge_key_type
        firstPartySources:
          - AFTERPAY
          - CASHAPP
          - SQUARE
        thirdPartySources:
          - EXPERIAN
          - ACXIOM
        computePoolId: 11111111-1111-1111-1111-111111111111
```

***

### CreateRosettaStoneMappingsIfNotExist

Task that creates Rosetta Stone attribute mappings for a dataset.

**Parameters:**

| Parameter      | Type    | Required | Default | Description                                                                                                                                                           |
| -------------- | ------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`    | integer | No       | —       | The numeric id of an existing dataset.                                                                                                                                |
| `datasetName`  | string  | No       | —       | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                            |
| `mappings`     | array   | Yes      | —       | A list of mapping definitions to create.                                                                                                                              |
| `allowPartial` | boolean | No       | `true`  | When `true`, individual mapping failures don't prevent other valid mappings from being created. When `false`, any single failure causes the entire operation to fail. |

**Output:**

| Field              | Type  | Always present | Description                                                   |
| ------------------ | ----- | -------------- | ------------------------------------------------------------- |
| `createdMappings`  | array | Yes            | Mappings that were successfully created.                      |
| `failedMappings`   | array | Yes            | Mappings that failed to create.                               |
| `conflictMappings` | array | Yes            | Mappings skipped because an identical mapping already exists. |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: etl
  name: map-identity-seed
  version: 1.0.0
do:
  - mapToRosettaStone:
      call: CreateRosettaStoneMappingsIfNotExist
      with:
        datasetName: identity_seed
        mappings:
          - attributeId: 92
            mapping:
              type: object_mapping
              propertyMappings:
                - path: value
                  expression: SHA2(NORMALIZE_EMAIL(email), 256)
                - path: type
                  expression: '''sha256_email'''
          - attributeId: 50
            mapping:
              type: value_mapping
              expression: country_code
          - attributeId: 73
            mapping:
              type: cached_mapping
              inputExpressions:
                - device_id
```

***

### CreateDatasetSample

Task that generates a sample for a dataset.

**Parameters:**

| Parameter       | Type    | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`     | integer | No       | The numeric id of an existing dataset.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `datasetName`   | string  | No       | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `computePoolId` | string  | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Output:**

| Field       | Type    | Always present | Description                                       |
| ----------- | ------- | -------------- | ------------------------------------------------- |
| `datasetId` | integer | Yes            | The id of the dataset whose sample was generated. |
| `rowCount`  | integer | Yes            | The number of rows captured in the sample.        |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: analytics
  name: create-dataset-sample-after-refresh
  version: 1.0.0
do:
  - refreshView:
      call: RefreshMaterializedView
      with:
        datasetName: active_users
  - createDatasetSample:
      call: CreateDatasetSample
      with:
        datasetName: active_users
```

***

### RecalculateStatistics

Task that triggers a recalculation of a dataset's column statistics and waits for it to complete.

**Parameters:**

| Parameter       | Type    | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `datasetId`     | integer | No       | The numeric id of an existing dataset.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `datasetName`   | string  | No       | The name of a dataset. Must contain only alphanumeric characters and underscores, with a maximum length of 256 characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `computePoolId` | string  | No       | The compute pool ID to use for running the task.  When omitted, the resolution depends on whether the task operates on an existing dataset:    - If it does (e.g. `RefreshMaterializedView`, `ExecuteDml`,     `CreateDatasetSample`), the dataset's default compute pool is used;     if the dataset has no default, the dataplane's default compute pool     is used.   - If it does not (e.g. `CreateMaterializedViewIfNotExists`, where the     dataset is being created, or `RunModelInference`, which is not tied     to a dataset), the dataplane's default compute pool is used directly. |

**Example:**

```yaml theme={null}
document:
  dsl: 1.0.0
  namespace: analytics
  name: recalculate-statistics-after-load
  version: 1.0.0
do:
  - loadData:
      call: ExecuteDml
      with:
        nql: INSERT INTO company_data.active_users (user_id, email) SELECT user_id, email FROM company_data.users WHERE is_active = true
  - recalculateStats:
      call: RecalculateStatistics
      with:
        datasetName: active_users
```

***

## Related content

<CardGroup>
  <Card title="Workflow Specification Syntax" icon="list-check" href="/reference/workflows/specification-syntax">
    Full specification format for document, schedule, and task blocks
  </Card>

  <Card title="Automating Multi-Step Pipelines" icon="diagram-project" href="/guides/workflows/workflow-orchestration">
    Step-by-step guide to creating and running workflows
  </Card>

  <Card title="Materialized Views" icon="layer-group" href="/concepts/nql/materialized-views">
    How materialized views work
  </Card>

  <Card title="Workflows API" icon="code" href="/api-reference/workflows/list-workflows">
    REST API endpoints for managing workflows
  </Card>
</CardGroup>