Uploading Data

The Narrative SDK provides methods to upload data files to your datasets programmatically. This guide covers the upload workflow and best practices.

Prerequisites

SDK installed and configured (see SDK Quickstart)
A target dataset (see Managing Datasets)
An API key with write permissions
Data file in a supported format (CSV, JSON, Parquet)

Supported formats

Format	Extension	Notes
CSV	`.csv`	Comma-separated values with headers
JSON	`.json`	JSON Lines (one object per line) or JSON array
Parquet	`.parquet`	Apache Parquet columnar format

Upload workflow

The upload process involves these steps:

Get an upload URL - Request a pre-signed URL for uploading
Upload the file - Send the file to the pre-signed URL
Confirm the upload - Notify Narrative that the upload is complete
Monitor ingestion - Track the ingestion job status

Basic upload

Here’s a complete example using Node.js:

import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadFile(datasetId: number, filePath: string) {
  const fileName = path.basename(filePath);
  const fileContent = fs.readFileSync(filePath);

  // Step 1: Get upload URL
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  console.log('Got upload URL');

  // Step 2: Upload file to S3
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  console.log('File uploaded to S3');

  // Step 3: Confirm upload
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
  console.log('Upload confirmed');

  return uploadUrl.upload_id;
}

// Usage
const uploadId = await uploadFile(12345, './data/customer_events.csv');
console.log('Upload ID:', uploadId);

Ingesting from a URL

If your data is already accessible via URL, you can ingest directly:

await api.ingestDatasetFile(
  { source_file: 'https://example.com/data/events.csv' },
  12345  // dataset ID
);

console.log('Ingestion started');

Monitoring upload status

After confirming an upload, monitor the ingestion job:

import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function waitForIngestion(jobId: string) {
  while (true) {
    const job = await api.getJob(jobId);

    console.log(`Status: ${job.state}`);

    if (job.state === 'completed') {
      console.log('Ingestion completed successfully');
      return job;
    }

    if (job.state === 'failed') {
      throw new Error(`Ingestion failed: ${JSON.stringify(job.failures)}`);
    }

    // Wait before checking again
    await new Promise(resolve => setTimeout(resolve, 5000));
  }
}

Uploading multiple files

Upload multiple files to the same dataset:

import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadMultipleFiles(datasetId: number, filePaths: string[]) {
  const results = [];

  for (const filePath of filePaths) {
    const fileName = path.basename(filePath);
    const fileContent = fs.readFileSync(filePath);

    console.log(`Uploading ${fileName}...`);

    const uploadUrl = await api.getUploadUrl(datasetId, fileName);
    await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
    await api.confirmUpload(datasetId, uploadUrl.upload_id);

    results.push({
      file: fileName,
      uploadId: uploadUrl.upload_id,
    });

    console.log(`Completed ${fileName}`);
  }

  return results;
}

// Usage
const files = [
  './data/events_2024_01.csv',
  './data/events_2024_02.csv',
  './data/events_2024_03.csv',
];

const results = await uploadMultipleFiles(12345, files);
console.log('Uploaded:', results);

Data preparation

Before uploading, ensure your data meets the dataset schema requirements.

CSV files

Include a header row matching schema field names
Use consistent formatting for dates and timestamps
Handle special characters properly (escape or quote)

customer_id,event_type,event_timestamp,event_value
cust_123,purchase,2024-01-15T10:30:00Z,99.99
cust_456,view,2024-01-15T11:00:00Z,0

JSON files

Use JSON Lines format (one object per line):

{"customer_id": "cust_123", "event_type": "purchase", "event_timestamp": "2024-01-15T10:30:00Z", "event_value": 99.99}
{"customer_id": "cust_456", "event_type": "view", "event_timestamp": "2024-01-15T11:00:00Z", "event_value": 0}

Hashing PII

If your data contains PII, hash it before uploading:

import crypto from 'crypto';

function hashEmail(email: string): string {
  const normalized = email.toLowerCase().trim();
  return crypto.createHash('sha256').update(normalized).digest('hex');
}

// Before uploading, transform your data
const hashedData = rawData.map(row => ({
  ...row,
  sha256_hashed_email: hashEmail(row.email),
}));

For detailed hashing instructions, see Hashing PII for Upload.

Error handling

try {
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
} catch (error) {
  if (error.status === 400) {
    console.error('Invalid request:', error.message);
  } else if (error.status === 403) {
    console.error('Permission denied - check API key permissions');
  } else if (error.status === 404) {
    console.error('Dataset not found');
  } else {
    console.error('Upload failed:', error);
  }
}

Best practices

Practice	Description
Validate before upload	Check file format and schema match
Hash PII	Never upload unhashed personal data
Use appropriate formats	Parquet for large datasets, CSV/JSON for smaller
Monitor jobs	Track ingestion status to catch failures
Handle errors	Implement retry logic for transient failures

Troubleshooting

Issue	Cause	Solution
Schema mismatch	File columns don’t match dataset schema	Verify column names and types
Upload timeout	File too large or slow connection	Try smaller files or increase timeout
Permission denied	API key lacks write access	Check API key permissions
Invalid format	File format not recognized	Verify file extension and content

Hashing PII

Prepare PII for upload

Tracking Jobs

Monitor upload progress

Managing Datasets

Dataset operations

Overview

Platform

Data Ingestion

Data Planes

Querying with NQL

Data Collaboration

Identity Resolution

Data Activation

Audience Studio

Connectors

Compliance

Rosetta Stone

Account Settings

Workflows

Webhooks

Data Collaboration MCP Server

Tools

SDKs

Prerequisites

Supported formats

Upload workflow

Basic upload

Ingesting from a URL

Monitoring upload status

Uploading multiple files

Data preparation

CSV files

JSON files

Hashing PII

Error handling

Best practices

Troubleshooting

Hashing PII

Tracking Jobs

Managing Datasets

​Prerequisites

​Supported formats

​Upload workflow

​Basic upload

​Ingesting from a URL

​Monitoring upload status

​Uploading multiple files

​Data preparation

​CSV files

​JSON files

​Hashing PII

​Error handling

​Best practices

​Troubleshooting

​Related content

Hashing PII

Tracking Jobs

Managing Datasets

Prerequisites

Supported formats

Upload workflow

Basic upload

Ingesting from a URL

Monitoring upload status

Uploading multiple files

Data preparation

CSV files

JSON files

Hashing PII

Error handling

Best practices

Troubleshooting

Related content