Derivations API

A derivation rule describes how the value of one Rosetta Stone attribute is computed from another — for example, a rule from raw_email to sha256_hashed_email might map the target’s value property to SHA2(NORMALIZE_EMAIL($source.value), 256). Once a rule exists, queries can resolve the target attribute from the source without a stored mapping. The /derivations endpoints let you list, fetch, create, update, and delete rules. Read access follows collaborator sharing; create, update, and delete are restricted to admins on the company that owns the rule. To explore how rules connect the attributes around a given one, use the derivation graph endpoint described below.

Concepts

Source, target, and transformation

Every rule has one source attribute (read through $source inside the transformation) and one target attribute (the value the rule produces). The mapping field on the request holds the transformation and mirrors mapping types with two differences:

Only value_mapping and object_mapping (as RawValueMapping and RawObjectMapping) are accepted. cached_mapping is rejected because there is no source dataset to look values up from.
Expressions reference the source attribute’s shape through $source instead of source-column names. For an object source, $source.<property> reads a property; for a primitive source, $source is the value.

Enum constraints and validation expressions on the target attribute do not restrict the transformation, matching the behavior of a dataset mapping.

Validation

Every transformation is validated before the rule is written. The API parses each expression, compiles it against a synthetic single-column table built from the source attribute, and type-checks the result against the target attribute using the same rules as a dataset mapping. Every required target property must be covered; unknown properties are rejected. If several problems are present in one mapping, they come back together as a single 400. Non-deterministic and dynamic functions like CURRENT_DATE, CURRENT_TIMESTAMP, and RAND are permitted. A derivation expression runs when a query resolves the rule, so a rule from birthdate to age in years is correct as of each query that uses it.

Cycle warnings

A rule from normalized_email to raw_email alongside one going the other way is a pair of correct rules, so a create or update that closes a cycle in the derivation graph is written anyway. The response returns a cyclic_derivation warning listing the attribute ids in the cycle. Cycle detection is bounded: only cycles of up to six attributes are reported, and at most ten come back per response. List and get responses do not compute warnings.

Permissions

Action	Grant on `attribute_derivations`	Ownership requirement
List, get	`read`	The rule is owned by your company or shared with it through collaborators.
Create	`admin`	The request’s `company_id` must equal your company’s. The target attribute must grant your company `map`.
Update, delete	`admin`	The rule must be owned by your company.

A rule your company cannot see returns 404, not 403. Only Narrative can share a rule broadly, with all or with an exclusion set on collaborators.use.

Endpoints

`GET /derivations`

List the derivation rules your company can use. Filter with repeated source_attribute_id and target_attribute_id query parameters; both accept multiple ids and combine as AND.

curl -H "Authorization: Bearer $TOKEN" \
  "https://api.narrative.io/derivations?source_attribute_id=101&target_attribute_id=202"

The response is { "records": [ ... ] }. Warnings are empty on reads.

`GET /derivations/{derivation_id}`

Fetch a single rule by id. Returns 404 if the rule is not usable by your company.

`POST /derivations`

Create a rule. Requires an admin grant on attribute_derivations. The transformation is validated before the rule is written.

{
  "name": "raw_email_to_sha256",
  "description": "Hashes the raw email address with SHA-256.",
  "source_attribute_id": 101,
  "target_attribute_id": 202,
  "company_id": 42,
  "mapping": {
    "type": "value_mapping",
    "expression": "SHA2(NORMALIZE_EMAIL($source.value), 256)"
  },
  "lossy": true,
  "imprecise": false,
  "fidelity_note": "Hashing is one-way, so the original address cannot be recovered.",
  "collaborators": { "use": { "type": "none" } },
  "cost": 1,
  "active": true
}

Returns 201 with the created rule and any warnings. Error responses:

Status	Meaning
`400`	The transformation fails to parse, compile, or type-check against the target; a required target property is uncovered; an unknown property is present; a `cached_mapping` was submitted; the source and target ids are equal; or another field is invalid.
`403`	The caller is not an admin, the rule’s `company_id` differs from the caller’s, the collaborators are not allowed, or the target attribute does not grant the caller’s company `map`.
`409`	Another of the company’s rules already uses the same `name`, or already covers the same `(source_attribute_id, target_attribute_id)` pair.

`PATCH /derivations/{derivation_id}`

Update fields on an existing rule. Requires an admin grant, and only the owning company can update. Only fields present in the request change; a changed mapping runs the same validation as create.

source_attribute_id and target_attribute_id are immutable — there is no key for them, and one sent anyway is ignored. Create a new rule instead.
Setting fidelity_note to null clears the note. Leaving the key out keeps the current value.
Unknown keys are rejected.

Returns 200 with the updated rule and any warnings.

`DELETE /derivations/{derivation_id}`

Permanently remove a rule. Requires an admin grant, and only the owning company can delete. To take a rule out of use while keeping it, PATCH active to false instead.

`GET /attributes/{attribute_name_or_id}/derivation-graph`

Return the part of the derivation graph reachable from an attribute: the attributes it can be derived from, the attributes that can be derived from it, the rules connecting them, and the routes out of the attribute with their total cost and fidelity. Only active rules your company may use are followed. An attribute you cannot view is left out of the response along with the rules touching it. The walk goes as far as query resolution can chain rules, which is one rule today, so every path listed is one a query can use. At most 50 paths are returned, descendants before ancestors. Authentication is optional. Anonymous callers see the rules and attributes shared with every company. A token that is present but invalid is rejected rather than falling back to the anonymous view.

curl "https://api.narrative.io/attributes/normalized_email/derivation-graph?direction=both"

string

default:"both"

Which way to walk from the attribute. descendants follows rules from source to target and reaches what can be derived from the attribute. ancestors follows them from target to source and reaches what the attribute can be derived from. both walks both ways.

The graph can contain cycles (a rule from birthdate to age alongside one going the other way is a pair of correct rules), so nodes and edges are flat lists rather than a nested tree. A program walks the adjacency through edges; a person reads paths.

integer

Id of the requested attribute.

string

Name of the requested attribute.

string

The direction the walk followed, as requested.

array

The attributes the walk reached, the requested attribute first, then the rest by distance and then by id. Each node carries id, name, display_name, distance (rules between it and the requested attribute along the shortest route; 0 for the requested attribute), and relationship: self, descendant (it can be derived from the requested attribute), ancestor (the requested attribute can be derived from it), or both (reachable either way, which only a cycle makes possible).

array

Every rule connecting two of the nodes, in rule id order. Each edge carries source and target (each { id, name }) and derivation: the rule’s id, name, description, cost, lossy, and imprecise, plus fidelity_note when the rule has one. The transformation is not included; fetch the full rule from GET /derivations/{derivation_id}.

array

The routes out of the requested attribute, one rule per step, visiting no attribute twice. Routes are reported once they cannot be extended, so every shorter route is a prefix of one listed. Each path carries relationship (descendant or ancestor), total_cost (the sum of the cost of every rule on the route), lossy and imprecise (set when any rule on the route has the flag), and steps, each step naming the rule taken (derivation_id) and the attribute it leads to (attribute_id, attribute_name).

{
  "attribute_id": 291,
  "attribute_name": "normalized_email",
  "direction": "both",
  "nodes": [
    { "id": 291, "name": "normalized_email", "display_name": "Normalized Email", "distance": 0, "relationship": "self" },
    { "id": 76, "name": "sha256_hashed_email", "display_name": "SHA-256 Hashed Email", "distance": 1, "relationship": "descendant" },
    { "id": 118, "name": "raw_email", "display_name": "Raw Email", "distance": 1, "relationship": "ancestor" }
  ],
  "edges": [
    {
      "source": { "id": 118, "name": "raw_email" },
      "target": { "id": 291, "name": "normalized_email" },
      "derivation": {
        "id": 1,
        "name": "raw_email_to_normalized_email",
        "description": "Lower-cases the address and strips dots and plus-tags where the provider ignores them.",
        "cost": 1,
        "lossy": true,
        "imprecise": false,
        "fidelity_note": "Normalization cannot be undone."
      }
    },
    {
      "source": { "id": 291, "name": "normalized_email" },
      "target": { "id": 76, "name": "sha256_hashed_email" },
      "derivation": {
        "id": 2,
        "name": "normalized_email_to_sha256_hashed_email",
        "description": "Hashes the normalized address with SHA-256.",
        "cost": 1,
        "lossy": true,
        "imprecise": false
      }
    }
  ],
  "paths": [
    {
      "relationship": "descendant",
      "lossy": true,
      "imprecise": false,
      "total_cost": 1,
      "steps": [{ "attribute_id": 76, "attribute_name": "sha256_hashed_email", "derivation_id": 2 }]
    },
    {
      "relationship": "ancestor",
      "lossy": true,
      "imprecise": false,
      "total_cost": 1,
      "steps": [{ "attribute_id": 118, "attribute_name": "raw_email", "derivation_id": 1 }]
    }
  ]
}

Error responses:

Status	Meaning
`400`	`direction` is not one of `descendants`, `ancestors`, `both`.
`401`	The bearer token is expired or does not grant read access to attributes.
`403`	The bearer token could not be parsed or resolved.
`404`	No attribute with the given name or id is visible to the caller.

Request and response fields

Rule fields

string

required

Name of the rule, unique within the owning company. Up to 256 characters.

string

required

What the rule does, for whoever is deciding whether to accept derived data. Up to 2048 characters.

integer

required

The attribute the rule reads from through $source. Immutable after create.

integer

required

The attribute whose value the rule produces. Must differ from source_attribute_id. Immutable after create.

integer

required

The company that owns the rule. Must equal the caller’s company on create.

object

required

The transformation. Same shape as value_mapping or object_mapping in Mapping types, except expressions reference $source instead of source-column names. cached_mapping is not allowed.

boolean

required

The transformation loses information — for example, hashing.

boolean

required

The transformation is approximate — for example, bucketing an age into a range.

object

Which other companies may have the rule applied to their data. A rule grants one thing, so there is a single use dimension. all and exclusion are reserved for Narrative-owned rules.

{ "use": { "type": "inclusion", "company_ids": [17, 92] } }

string | null

Explains the lossy and imprecise flags to whoever is deciding whether to accept derived data. Up to 2048 characters. PATCH with null clears the note.

integer

default:"1"

Breaks ties when more than one rule reaches the same target attribute — cheapest first. Multi-step resolution sums the cost of every rule on a path and prefers the cheapest total. Minimum 1.

boolean

default:"true"

Inactive rules are never applied.

Response-only fields

integer

Unique identifier of the rule.

string

ISO 8601 timestamp of creation.

integer

Id of the user who created the rule.

string

ISO 8601 timestamp of the last update.

integer

Id of the user who last updated the rule.

array

Populated on create and update responses; empty on reads. Each warning is an object; today the only type is cyclic_derivation, whose cycle field lists the attribute ids visited in order (the closing step back to the first id is implied).

Mapping types

The mapping shapes derivation transformations reuse.

Transformation functions

Functions available inside derivation expressions.

Attribute types

The source and target shapes a rule connects.

How Rosetta Stone works

Background on attributes, mappings, and normalization.

Overview

Connectors

Integrations

Workflow Reference

Webhook Reference

Security Reference

Architecture Reference

Error Reference

Deprecations

Model Inference Reference

Rosetta Stone Reference

UI Reference

SDKs

Glossary

Billing

Concepts

Source, target, and transformation

Validation

Cycle warnings

Permissions

Endpoints

`GET /derivations`

`GET /derivations/{derivation_id}`

`POST /derivations`

`PATCH /derivations/{derivation_id}`

`DELETE /derivations/{derivation_id}`

`GET /attributes/{attribute_name_or_id}/derivation-graph`

Request and response fields

Rule fields

Response-only fields

Mapping types

Transformation functions

Attribute types

How Rosetta Stone works

​Concepts

​Source, target, and transformation

​Validation

​Cycle warnings

​Permissions

​Endpoints

​GET /derivations

​GET /derivations/{derivation_id}

​POST /derivations

​PATCH /derivations/{derivation_id}

​DELETE /derivations/{derivation_id}

​GET /attributes/{attribute_name_or_id}/derivation-graph

​Request and response fields

​Rule fields

​Response-only fields

​Related content

Mapping types

Transformation functions

Attribute types

How Rosetta Stone works

Concepts

Source, target, and transformation

Validation

Cycle warnings

Permissions

Endpoints

`GET /derivations`

`GET /derivations/{derivation_id}`

`POST /derivations`

`PATCH /derivations/{derivation_id}`

`DELETE /derivations/{derivation_id}`

`GET /attributes/{attribute_name_or_id}/derivation-graph`

Request and response fields

Rule fields

Response-only fields

Related content