Amazon S3 Connector API

The Amazon S3 Connector API lets an unattended integration deliver the rows of a Narrative dataset as files into an S3 bucket you own. Narrative delivers each new batch as it lands, and can also deliver the batches from up to 30 days before you connect the dataset. A person signs in once to create an API key and once to set up the bucket, and the integration works from the API key after that. Complete the prerequisites and set up API access on this page before you follow the guide.

Choose a guide

File delivery

Create a dataset, upload rows, connect it to your bucket, and keep delivering as new rows arrive.

Prerequisites

Prerequisite	Needed for
API key	Everything
S3 Connector installed	Everything
A profile for your bucket	File delivery

API key

Create the key under Settings → API Keys with read and write on datasets and connections, write on uploads, and read on installations, apps, and app_profiles. See API Keys for the procedure and the Permissions Reference for what each resource covers. Create it from a shared team account rather than a personal login, so the integration keeps working when someone changes roles.

S3 Connector installed

The S3 Connector is app 7. Find your installation of it:

curl -s "https://api.narrative.io/installations?app_categories=destination_connector" \
  -H "Authorization: Bearer $NIO_API_TOKEN"
# find the record with "app_id": 7 and note its "id"

Record the installation ID. If no record has "app_id": 7, install the S3 Connector from the Narrative Marketplace.

A profile for your bucket

A connector profile names one S3 bucket and the file format Narrative writes into it. You set up a profile once per bucket, in the Narrative UI, with access to the bucket’s settings in AWS:

Open the connector

In Narrative, go to Installed Apps, select S3 Connector, and choose Add profile.

Name the profile and bucket

Enter a profile name, a description, and the name of the bucket Narrative delivers to.

Apply the bucket policy in AWS

Copy the bucket policy the wizard shows, and apply it in the AWS console under the bucket’s Permissions → Bucket policy, or with aws s3api put-bucket-policy. The policy lets Narrative’s delivery role write, list, and delete objects in the bucket. You can also fetch the policy over the API, as shown in Get the bucket policy.

Tag the bucket in AWS

Add the tag the wizard shows to the bucket, under Properties → Tags. The tag is named NARRATIVE_S3_CONNECTOR_ID, and its value ties the bucket to your company.

Test access

Run the wizard’s access test. Narrative writes, lists, and deletes a test object in the bucket and reads the tag.

Choose the file format and save

Choose CSV, TSV, JSON, or Parquet. For CSV or TSV, also set the delimiter, quote, and escape characters, and whether to include a header row. Choose Save and finish, and the profile is enabled and ready for connections.

Every request in the guide then uses the profile ID. List your profiles shows how to look it up.

Set up API access

The guide talks to two hosts, and each takes its own credential:

Host	Base URL	Credential	Handles
Narrative API	`https://api.narrative.io`	Your API key	Datasets, uploads, connections, profiles
S3 connector	`https://aws-s3.narrativeconnectors.com`	An installation token	Bucket and file format of each profile, bucket policy

Get an installation token

The S3 connector authenticates requests with an installation token, which you get by exchanging your API key. Use the installation ID from S3 Connector installed:

curl -s -X POST https://api.narrative.io/installations/{installation_id}/token \
  -H "Authorization: Bearer $NIO_API_TOKEN"
# → {"access_token": "<INSTALLATION_TOKEN>", "access_token_expires_in": 10799,
#    "refresh_token": "<REFRESH_TOKEN>", "refresh_token_expires_in": 86399}

Store access_token as $INSTALLATION_TOKEN. It lasts about three hours, so request a new one at the start of each run of your integration. Confirm that both hosts answer:

curl -s https://api.narrative.io/company-info/whoami \
  -H "Authorization: Bearer $NIO_API_TOKEN"
# → {"id": 1234}

curl -s https://aws-s3.narrativeconnectors.com/profiles \
  -H "Authorization: Bearer $INSTALLATION_TOKEN"
# → [{"id": "<PROFILE_ID>", "status": "enabled", "bucket_name": "<BUCKET_NAME>", ...}]

Send the API key to api.narrative.io and the installation token to aws-s3.narrativeconnectors.com.

API keys contain characters that shells mangle. Read the key from a file or an environment variable rather than pasting it inline.

List your profiles

The Narrative API lists the profiles under your installation, with the status of each:

curl -s https://api.narrative.io/installations/{installation_id}/profiles \
  -H "Authorization: Bearer $NIO_API_TOKEN"

{
  "records": [
    {
      "id": "<PROFILE_ID>",
      "name": "Example delivery bucket",
      "description": "Example delivery bucket",
      "status": "enabled",
      "app_id": null,
      "app_name": null,
      "tags": []
    }
  ]
}

The S3 connector returns the same profiles along with the bucket and file format of each:

curl -s https://aws-s3.narrativeconnectors.com/profiles \
  -H "Authorization: Bearer $INSTALLATION_TOKEN"

[
  {
    "id": "<PROFILE_ID>",
    "name": "Example delivery bucket",
    "description": "Example delivery bucket",
    "status": "enabled",
    "bucket_name": "<BUCKET_NAME>",
    "config": {
      "file_type": "csv",
      "header": true,
      "delimiter": ",",
      "quote": "\"",
      "escape": "\"",
      "sort_columns": true,
      "compression": null
    }
  }
]

To read one profile, call GET /profiles/{profile_id} on the connector, which returns the same shape. Use a profile whose status is enabled, and record its id.

Get the bucket policy

The connector generates the bucket policy for a bucket name:

curl -s https://aws-s3.narrativeconnectors.com/bucket/{bucket_name}/bucket-policy \
  -H "Authorization: Bearer $INSTALLATION_TOKEN"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<NARRATIVE_DELIVERY_ROLE_ARN>" },
      "Action": ["s3:DeleteObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    },
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<NARRATIVE_DELIVERY_ROLE_ARN>" },
      "Action": ["s3:List*", "s3:GetBucketTagging"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    }
  ]
}

Apply the policy exactly as the connector returns it, with Narrative’s delivery role in Principal.

Read the delivery settings schema

Each connection carries delivery settings for the connector’s delivery interface. The schema lists every setting with its type and description:

curl -s https://api.narrative.io/apps/7/interfaces/delivery \
  -H "Authorization: Bearer $NIO_API_TOKEN"
# → {"id": "delivery", "app_id": 7, "metadata": {"tags": ["s3"], ...},
#    "schema": {"properties": {"bucket_prefix": {...}, "historical_data_enabled": {...}, ...}, ...}}

The connector serves the same schema at https://aws-s3.narrativeconnectors.com/interfaces/delivery, with no credential. What the connection does explains each field.

What the connection does

A connection’s quick_settings set where and how the connection writes files. The S3 Connector has one interface, delivery. The file format, compression, and CSV options come from the profile.

Field	What it does	Allowed values	Default
`bucket_prefix`	The folder under the bucket that files go into	A path such as `exports/narrative`	The bucket root
`historical_data_enabled`	Whether the connection also delivers data already in the dataset	`true`, `false`	`false`
`historical_period_seconds`	How far back `historical_data_enabled` reaches, in seconds	Any positive integer. Values over 2,592,000 (30 days) count as 30 days.	2,592,000
`remove_metadata`	Drops Narrative’s `_nio` metadata columns from delivered files	`true`, `false`	`false`
`use_delivery_date`	Names each folder after the delivery date (`yyyy-MM-dd`, UTC) instead of the snapshot ID	`true`, `false`	`false`
`include_success_file`	Writes an empty `_SUCCESS` file once every file in a delivery has landed	`true`, `false`	`false`
`write_narrative_commit_file`	Writes an empty `_NIO_COMMIT` file once every file in a delivery has landed	`true`, `false`	`false`

Existing and new rows

A connection delivers every snapshot written to the dataset after the connection is created. The historical_data_enabled quick setting decides whether the connection also delivers data that was already in the dataset when you created it:

With historical_data_enabled set to true, the connection first delivers data already in the dataset, then keeps delivering new snapshots as they land.
With historical_data_enabled set to false, the connection delivers only snapshots written after it was created.

How far back the first delivery reaches depends on the connector, and each connector’s guide states it. For the S3 Connector, historical_data_enabled delivers the dataset’s snapshots from the last historical_period_seconds before the connection was created. After that, each new snapshot is delivered as it lands. Each source file in a snapshot becomes one object in the bucket, under <bucket_prefix>/snapshotId=<snapshot ID>/. A delivery only adds objects. When you overwrite the dataset, the new snapshot lands in a new folder and the earlier folders stay in the bucket.