Derivations API - Narrative I/O Knowledge Base

A derivation rule describes how the value of one Rosetta Stone attribute is computed from another — for example, a rule from raw_email to sha256_hashed_email might map the target’s value property to SHA2(NORMALIZE_EMAIL($source.value), 256). Once a rule exists, queries can resolve the target attribute from the source without a stored mapping: an NQL query that reads sha256_hashed_email against a dataset whose owner mapped only raw_email compiles and returns the hashed values, and the same resolution runs in every executor (API, Spark on EMR, transaction jobs, workflow worker, table maintenance, ingestion). The /derivations endpoints let you list, fetch, create, update, and delete rules. Read access follows collaborator sharing; create, update, and delete are restricted to admins on the company that owns the rule. To explore how rules connect the attributes around a given one, use the derivation graph endpoint described below.

Concepts

Source, target, and transformation

Every rule has one source attribute (read through $source inside the transformation) and one target attribute (the value the rule produces). The mapping field on the request holds the transformation and mirrors mapping types with two differences:

Only value_mapping and object_mapping (as RawValueMapping and RawObjectMapping) are accepted. cached_mapping is rejected because there is no source dataset to look values up from.
Expressions reference the source attribute’s shape through $source instead of source-column names. For an object source, $source.<property> reads a property; for a primitive source, $source is the value.

Enum constraints and validation expressions on the target attribute do not restrict the transformation, matching the behavior of a dataset mapping.

Validation

Every transformation is validated before the rule is written. The API parses each expression, compiles it against a synthetic single-column table built from the source attribute, and type-checks the result against the target attribute using the same rules as a dataset mapping. Every required target property must be covered; unknown properties are rejected. If several problems are present in one mapping, they come back together as a single 400. Non-deterministic and dynamic functions like CURRENT_DATE, CURRENT_TIMESTAMP, and RAND are permitted. A derivation expression runs when a query resolves the rule, so a rule from birthdate to age in years is correct as of each query that uses it.

Cycle warnings

A rule from normalized_email to raw_email alongside one going the other way is a pair of correct rules, so a create or update that closes a cycle in the derivation graph is written anyway. The response returns a cyclic_derivation warning listing the attribute ids in the cycle. Cycle detection is bounded: only cycles of up to six attributes are reported, and at most ten come back per response. List and get responses do not compute warnings.

Permissions

Action	Grant on `attribute_derivations`	Ownership requirement
List, get	`read`	The rule is owned by your company or shared with it through collaborators.
Create	`admin`, or a user token whose user is a Narrative global admin	The request’s `company_id` must equal your company’s. The target attribute must grant your company `map`.
Update, delete	`admin`	The rule must be owned by your company.

A rule your company cannot see returns 404, not 403. Only Narrative can share a rule broadly, with all or with an exclusion set on collaborators.use. The read grant is included in every user token by default, so platform login sessions and MCP-issued tokens can list and describe rules without extra configuration. POST /derivations accepts a global admin’s user token without the explicit admin grant; an app-installation token always needs the grant.

Endpoints

`GET /derivations`

List the derivation rules your company can use. Filter with repeated source_attribute_id and target_attribute_id query parameters; both accept multiple ids and combine as AND.

curl -H "Authorization: Bearer $TOKEN" \
  "https://api.narrative.io/derivations?source_attribute_id=101&target_attribute_id=202"

The response is { "records": [ ... ] }. Warnings are empty on reads.

`GET /derivations/{derivation_id}`

Fetch a single rule by id. Returns 404 if the rule is not usable by your company.

`POST /derivations`

Create a rule. Requires an admin grant on attribute_derivations, or a user token whose user is a Narrative global admin (an app-installation token always needs the grant). The transformation is validated before the rule is written.

{
  "name": "raw_email_to_sha256",
  "description": "Hashes the raw email address with SHA-256.",
  "source_attribute_id": 101,
  "target_attribute_id": 202,
  "company_id": 42,
  "mapping": {
    "type": "value_mapping",
    "expression": "SHA2(NORMALIZE_EMAIL($source.value), 256)"
  },
  "lossy": true,
  "imprecise": false,
  "fidelity_note": "Hashing is one-way, so the original address cannot be recovered.",
  "collaborators": { "use": { "type": "none" } },
  "cost": 1,
  "active": true
}

Returns 201 with the created rule and any warnings. Error responses:

Status	Meaning
`400`	The transformation fails to parse, compile, or type-check against the target; a required target property is uncovered; an unknown property is present; a `cached_mapping` was submitted; the source and target ids are equal; or another field is invalid.
`403`	The caller is not an admin, the rule’s `company_id` differs from the caller’s, the collaborators are not allowed, or the target attribute does not grant the caller’s company `map`.
`409`	Another of the company’s rules already uses the same `name`, or already covers the same `(source_attribute_id, target_attribute_id)` pair.

`PATCH /derivations/{derivation_id}`

Update fields on an existing rule. Requires an admin grant, and only the owning company can update. Only fields present in the request change; a changed mapping runs the same validation as create.

source_attribute_id and target_attribute_id are immutable — there is no key for them, and one sent anyway is ignored. Create a new rule instead.
Setting fidelity_note to null clears the note. Leaving the key out keeps the current value.
Unknown keys are rejected.

Returns 200 with the updated rule and any warnings.

`DELETE /derivations/{derivation_id}`

Permanently remove a rule. Requires an admin grant, and only the owning company can delete. To take a rule out of use while keeping it, PATCH active to false instead.

`GET /attributes/{attribute_name_or_id}/derivation-graph`

Return the part of the derivation graph reachable from an attribute: the attributes it can be derived from, the attributes that can be derived from it, the rules connecting them, and the routes out of the attribute with their total cost and fidelity. Only active rules your company may use are followed. An attribute you cannot view is left out of the response along with the rules touching it. The walk goes as far as query resolution can chain rules, which is three rules, so every path listed is one a query can use. At most 50 paths are returned, descendants before ancestors. Authentication is optional. Anonymous callers see the rules and attributes shared with every company. A token that is present but invalid is rejected rather than falling back to the anonymous view.

curl "https://api.narrative.io/attributes/normalized_email/derivation-graph?direction=both"

string

default:"both"

Which way to walk from the attribute. descendants follows rules from source to target and reaches what can be derived from the attribute. ancestors follows them from target to source and reaches what the attribute can be derived from. both walks both ways.

The graph can contain cycles (a rule from birthdate to age alongside one going the other way is a pair of correct rules), so nodes and edges are flat lists rather than a nested tree. A program walks the adjacency through edges; a person reads paths.

integer

Id of the requested attribute.

string

Name of the requested attribute.

string

The direction the walk followed, as requested.

array

The attributes the walk reached, the requested attribute first, then the rest by distance and then by id. Each node carries id, name, display_name, distance (rules between it and the requested attribute along the shortest route; 0 for the requested attribute), and relationship: self, descendant (it can be derived from the requested attribute), ancestor (the requested attribute can be derived from it), or both (reachable either way, which only a cycle makes possible).

array

Every rule connecting two of the nodes, in rule id order. Each edge carries source and target (each { id, name }) and derivation: the rule’s id, name, description, cost, lossy, and imprecise, plus fidelity_note when the rule has one. The transformation is not included; fetch the full rule from GET /derivations/{derivation_id}.

array

The routes out of the requested attribute, one rule per step, visiting no attribute twice. Routes are reported once they cannot be extended, so every shorter route is a prefix of one listed. Each path carries relationship (descendant or ancestor), total_cost (the sum of the cost of every rule on the route), lossy and imprecise (set when any rule on the route has the flag), and steps, each step naming the rule taken (derivation_id) and the attribute it leads to (attribute_id, attribute_name).

{
  "attribute_id": 291,
  "attribute_name": "normalized_email",
  "direction": "both",
  "nodes": [
    { "id": 291, "name": "normalized_email", "display_name": "Normalized Email", "distance": 0, "relationship": "self" },
    { "id": 76, "name": "sha256_hashed_email", "display_name": "SHA-256 Hashed Email", "distance": 1, "relationship": "descendant" },
    { "id": 118, "name": "raw_email", "display_name": "Raw Email", "distance": 1, "relationship": "ancestor" }
  ],
  "edges": [
    {
      "source": { "id": 118, "name": "raw_email" },
      "target": { "id": 291, "name": "normalized_email" },
      "derivation": {
        "id": 1,
        "name": "raw_email_to_normalized_email",
        "description": "Lower-cases the address and strips dots and plus-tags where the provider ignores them.",
        "cost": 1,
        "lossy": true,
        "imprecise": false,
        "fidelity_note": "Normalization cannot be undone."
      }
    },
    {
      "source": { "id": 291, "name": "normalized_email" },
      "target": { "id": 76, "name": "sha256_hashed_email" },
      "derivation": {
        "id": 2,
        "name": "normalized_email_to_sha256_hashed_email",
        "description": "Hashes the normalized address with SHA-256.",
        "cost": 1,
        "lossy": true,
        "imprecise": false
      }
    }
  ],
  "paths": [
    {
      "relationship": "descendant",
      "lossy": true,
      "imprecise": false,
      "total_cost": 1,
      "steps": [{ "attribute_id": 76, "attribute_name": "sha256_hashed_email", "derivation_id": 2 }]
    },
    {
      "relationship": "ancestor",
      "lossy": true,
      "imprecise": false,
      "total_cost": 1,
      "steps": [{ "attribute_id": 118, "attribute_name": "raw_email", "derivation_id": 1 }]
    }
  ]
}

Error responses:

Status	Meaning
`400`	`direction` is not one of `descendants`, `ancestors`, `both`.
`401`	The bearer token is expired or does not grant read access to attributes.
`403`	The bearer token could not be parsed or resolved.
`404`	No attribute with the given name or id is visible to the caller.

Listing derived mappings

The mapping and dataset list endpoints can return the derived mappings your stored mappings imply through the rules your company may use, so you can see that a mapping to raw_email unlocks normalized_email and sha256_hashed_email without composing them from /derivations by hand. Pass a repeatable source filter (mapping_source on the dataset reads) and include derived alongside any stored sources you want:

curl -H "Authorization: Bearer $TOKEN" \
  "https://api.narrative.io/mappings?source=company&source=derived"

Endpoints that accept the filter:

GET /mappings?source=...
GET /mappings/companies/{company_id}?source=...
GET /datasets?mapping_source=...
GET /datasets/{dataset_id}?mapping_source=...

Omitted, every stored mapping is returned as today. Naming a source without derived narrows to stored mappings of that source only. Unknown values return 400. On the dataset reads the filter narrows the embedded mappings array; it does not filter which datasets come back. With derived in the filter, the attribute_id filter is applied after resolution rather than in the query, since a derived row’s attribute is not the attribute of the mapping it came from.

Filter mappings by the dataset field they read

The /mappings reads also take a repeatable depends_on_dataset_field query parameter that narrows the results to mappings whose expression reads a given column. Name a top-level column to match every mapping that reads it or anything nested beneath it, or a dotted path to match only mappings that read that nested field:

curl -H "Authorization: Bearer $TOKEN" \
  "https://api.narrative.io/mappings?dataset_id=3&depends_on_dataset_field=email&depends_on_dataset_field=address.postal_code"

Matching uses the dependencies computed when each mapping was saved, so with source=derived a derived mapping matches through the stored expression it was built from. The filter requires dataset_id; a request that supplies depends_on_dataset_field without dataset_id, or with an empty path or an empty segment such as . or idValue., returns 400.

Shape of a derived row

A derived row differs from a stored one in three ways:

source is derived.
id is absent. A derived mapping is computed per request from a stored mapping and a derivation rule, so GET /mappings/{id}, PUT /mappings/{id}, and DELETE /mappings/{id} cannot reach it and are not extended to.
provenance is present and names how the row was produced.

provenance fields:

string

The stored mapping the derivation was applied to.

integer

The attribute of the source mapping.

array

The derivation rule ids applied, in order, from the source attribute to this mapping’s attribute. Up to three elements, one per rule in the chain. Each id resolves through GET /derivations/{id}.

boolean

True when any rule on the path is lossy.

boolean

True when any rule on the path is imprecise.

{
  "attribute_id": 76,
  "dataset_id": 3,
  "mapping": {
    "type": "value_mapping",
    "expression": "SHA2(NORMALIZE_EMAIL(email), 256)"
  },
  "status": "active",
  "created_at": "2021-12-10T00:00:00Z",
  "updated_at": "2021-12-10T00:00:01Z",
  "created_by": 1,
  "updated_by": 1,
  "company_id": 100,
  "scope": "global",
  "source": "derived",
  "provenance": {
    "source_mapping_id": "ea9dddd2-e3ee-40b5-b03e-c3cd37c8a6f0",
    "source_attribute_id": 118,
    "derivation_path": [1, 2],
    "lossy": true,
    "imprecise": false
  }
}

provenance and derived_from are unrelated. derived_from is a stored column recording the mapping this one was copied from at authoring time; provenance records the derivation rules applied at request time.

Omitting the filter leaves the responses unchanged apart from id no longer being listed as required on the MappingResponse schema, since it is absent on derived rows.

Request and response fields

Rule fields

string

required

Name of the rule, unique within the owning company. Up to 256 characters.

string

required

What the rule does, for whoever is deciding whether to accept derived data. Up to 2048 characters.

integer

required

The attribute the rule reads from through $source. Immutable after create.

integer

required

The attribute whose value the rule produces. Must differ from source_attribute_id. Immutable after create.

integer

required

The company that owns the rule. Must equal the caller’s company on create.

object

required

The transformation. Same shape as value_mapping or object_mapping in Mapping types, except expressions reference $source instead of source-column names. cached_mapping is not allowed.

boolean

required

The transformation loses information — for example, hashing.

boolean

required

The transformation is approximate — for example, bucketing an age into a range.

object

Which other companies may have the rule applied to their data. A rule grants one thing, so there is a single use dimension. all and exclusion are reserved for Narrative-owned rules.

{ "use": { "type": "inclusion", "company_ids": [17, 92] } }

string | null

Explains the lossy and imprecise flags to whoever is deciding whether to accept derived data. Up to 2048 characters. PATCH with null clears the note.

integer

default:"1"

Breaks ties when more than one rule reaches the same target attribute — cheapest first. Multi-step resolution sums the cost of every rule on a path and prefers the cheapest total. Minimum 1.

boolean

default:"true"

Inactive rules are never applied.

Response-only fields

integer

Unique identifier of the rule.

string

ISO 8601 timestamp of creation.

integer

Id of the user who created the rule.

string

ISO 8601 timestamp of the last update.

integer

Id of the user who last updated the rule.

array

Populated on create and update responses; empty on reads. Each warning is an object; today the only type is cyclic_derivation, whose cycle field lists the attribute ids visited in order (the closing step back to the first id is implied).

Attribute derivations

How rules become derived mappings and which mapping wins.

Creating derivation rules

Create, check, and retire a rule step by step.

Mapping types

The mapping shapes derivation transformations reuse.

Transformation functions

Functions available inside derivation expressions.

Attribute types

The source and target shapes a rule connects.

How Rosetta Stone works

Background on attributes, mappings, and normalization.

​Concepts

​Source, target, and transformation

​Validation

​Cycle warnings

​Permissions

​Endpoints

​GET /derivations

​GET /derivations/{derivation_id}

​POST /derivations

​PATCH /derivations/{derivation_id}

​DELETE /derivations/{derivation_id}

​GET /attributes/{attribute_name_or_id}/derivation-graph

​Listing derived mappings

​Filter mappings by the dataset field they read

​Shape of a derived row

​Request and response fields

​Rule fields

​Response-only fields

​Related content

Attribute derivations

Creating derivation rules

Mapping types

Transformation functions

Attribute types

How Rosetta Stone works

Concepts

Source, target, and transformation

Validation

Cycle warnings

Permissions

Endpoints

`GET /derivations`

`GET /derivations/{derivation_id}`

`POST /derivations`

`PATCH /derivations/{derivation_id}`

`DELETE /derivations/{derivation_id}`

`GET /attributes/{attribute_name_or_id}/derivation-graph`

Listing derived mappings

Filter mappings by the dataset field they read

Shape of a derived row

Request and response fields

Rule fields

Response-only fields

Related content