Skip to main content
The Narrative SDK provides methods to upload data files to your datasets programmatically. This guide covers the upload workflow and best practices.
For recurring uploads, consider using Automated Syncs instead of manual SDK uploads.

​
Prerequisites

  • SDK installed and configured (see SDK Quickstart)
  • A target dataset (see Managing Datasets)
  • An API key with write permissions
  • Data file in a supported format (CSV, JSON, Parquet)

​
Supported formats

FormatExtensionNotes
CSV.csvComma-separated values with headers
JSON.jsonJSON Lines (one object per line) or JSON array
Parquet.parquetApache Parquet columnar format

​
Upload workflow

The upload process involves these steps:
  1. Get an upload URL - Request a pre-signed URL for uploading
  2. Upload the file - Send the file to the pre-signed URL
  3. Confirm the upload - Notify Narrative that the upload is complete
  4. Monitor ingestion - Track the ingestion job status

​
Basic upload

Here’s a complete example using Node.js: 
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadFile(datasetId: number, filePath: string) {
  const fileName = path.basename(filePath);
  const fileContent = fs.readFileSync(filePath);

  // Step 1: Get upload URL
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  console.log('Got upload URL');

  // Step 2: Upload file to S3
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  console.log('File uploaded to S3');

  // Step 3: Confirm upload
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
  console.log('Upload confirmed');

  return uploadUrl.upload_id;
}

// Usage
const uploadId = await uploadFile(12345, './data/customer_events.csv');
console.log('Upload ID:', uploadId);

​
Ingesting from a URL

If your data is already accessible via URL, you can ingest directly: 
await api.ingestDatasetFile(
  { source_file: 'https://example.com/data/events.csv' },
  12345  // dataset ID
);

console.log('Ingestion started');

​
Monitoring upload status

After confirming an upload, monitor the ingestion job: 
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function waitForIngestion(jobId: string) {
  while (true) {
    const job = await api.getJob(jobId);

    console.log(`Status: ${job.state}`);

    if (job.state === 'completed') {
      console.log('Ingestion completed successfully');
      return job;
    }

    if (job.state === 'failed') {
      throw new Error(`Ingestion failed: ${JSON.stringify(job.failures)}`);
    }

    // Wait before checking again
    await new Promise(resolve => setTimeout(resolve, 5000));
  }
}

​
Uploading multiple files

Upload multiple files to the same dataset: 
import { NarrativeApi } from '@narrative.io/data-collaboration-sdk-ts';
import fs from 'fs';
import path from 'path';

const api = new NarrativeApi({
  apiKey: process.env.NARRATIVE_API_KEY,
});

async function uploadMultipleFiles(datasetId: number, filePaths: string[]) {
  const results = [];

  for (const filePath of filePaths) {
    const fileName = path.basename(filePath);
    const fileContent = fs.readFileSync(filePath);

    console.log(`Uploading ${fileName}...`);

    const uploadUrl = await api.getUploadUrl(datasetId, fileName);
    await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
    await api.confirmUpload(datasetId, uploadUrl.upload_id);

    results.push({
      file: fileName,
      uploadId: uploadUrl.upload_id,
    });

    console.log(`Completed ${fileName}`);
  }

  return results;
}

// Usage
const files = [
  './data/events_2024_01.csv',
  './data/events_2024_02.csv',
  './data/events_2024_03.csv',
];

const results = await uploadMultipleFiles(12345, files);
console.log('Uploaded:', results);

​
Data preparation

Before uploading, ensure your data meets the dataset schema requirements.

​
CSV files

  • Include a header row matching schema field names
  • Use consistent formatting for dates and timestamps
  • Handle special characters properly (escape or quote)
customer_id,event_type,event_timestamp,event_value
cust_123,purchase,2024-01-15T10:30:00Z,99.99
cust_456,view,2024-01-15T11:00:00Z,0

​
JSON files

Use JSON Lines format (one object per line): 
{"customer_id": "cust_123", "event_type": "purchase", "event_timestamp": "2024-01-15T10:30:00Z", "event_value": 99.99}
{"customer_id": "cust_456", "event_type": "view", "event_timestamp": "2024-01-15T11:00:00Z", "event_value": 0}

​
Hashing PII

If your data contains PII, hash it before uploading: 
import crypto from 'crypto';

function hashEmail(email: string): string {
  const normalized = email.toLowerCase().trim();
  return crypto.createHash('sha256').update(normalized).digest('hex');
}

// Before uploading, transform your data
const hashedData = rawData.map(row => ({
  ...row,
  sha256_hashed_email: hashEmail(row.email),
}));
For detailed hashing instructions, see Hashing PII for Upload.

​
Error handling

try {
  const uploadUrl = await api.getUploadUrl(datasetId, fileName);
  await api.uploadFileToS3(uploadUrl.url, fileContent, uploadUrl.headers);
  await api.confirmUpload(datasetId, uploadUrl.upload_id);
} catch (error) {
  if (error.status === 400) {
    console.error('Invalid request:', error.message);
  } else if (error.status === 403) {
    console.error('Permission denied - check API key permissions');
  } else if (error.status === 404) {
    console.error('Dataset not found');
  } else {
    console.error('Upload failed:', error);
  }
}

​
Best practices

PracticeDescription
Validate before uploadCheck file format and schema match
Hash PIINever upload unhashed personal data
Use appropriate formatsParquet for large datasets, CSV/JSON for smaller
Monitor jobsTrack ingestion status to catch failures
Handle errorsImplement retry logic for transient failures

​
Troubleshooting

IssueCauseSolution
Schema mismatchFile columns don’t match dataset schemaVerify column names and types
Upload timeoutFile too large or slow connectionTry smaller files or increase timeout
Permission deniedAPI key lacks write accessCheck API key permissions
Invalid formatFile format not recognizedVerify file extension and content

​
Related content

Hashing PII

Prepare PII for upload

Connecting Data Sources

Alternative ingestion methods

Tracking Jobs

Monitor upload progress

Managing Datasets

Dataset operations