Uploads via createUpload is crazy slow

[roger]

Apologies for the delayed response here @tim1 and @nroth… I’m trying to see if we can rearrange the forum a bit to prevent posts like this from falling through the cracks Sorry about that.

---

I dug more deeply into this, and TLDR, the CREATING_UPLOAD_OBJECT step creates an async job on our server that you shouldn’t have to wait for, but our client makes you wait I’ll flag it for the devs and see if it’s an easy fix for them…

Good news, though:

In the meantime, here is a workaround that should let you exit early. It reimplements much of the same logic as our real client does (using some of its undocumented exports), but then exits early as soon as CREATING_UPLOAD_OBJECT begins, without waiting for it to finish (which is what was taking so long, but that’s just a serverside op that your client shouldn’t need to wait for).

That’s all your script or serverless func should need to do. Then, a few seconds later, the image will magically appear in your DatoCMS media area… but your function won’t have to keep waiting for that to happen (unlike our official client, which keeps polling until it succeeds, which as you saw can take like 30+ seconds for big enough images).

Here’s a sample implementation that should exit as soon as the “create” step begins and the job starts, without waiting for it to finish. The image itself won’t show up in the media area until about 30-40 secs later, but your client won’t have to wait:

import {buildClient, downloadFile, LogLevel, uploadLocalFileAndReturnPath} from "@datocms/cma-client-node";
import {makeCancelablePromise} from '@datocms/rest-client-utils';

let jobUrl: string | undefined;

const logParser = (message: string) => {
    // Exit early if the job URL is already set
    if (jobUrl) return;

    // Otherwise, parse the log messages and get back the job ID
    const jobUrlMatch = message.match(/GET (https:\/\/site-api\.datocms\.com\/job-results\/.+)$/);
    if (jobUrlMatch && jobUrlMatch[1]) {
        jobUrl = jobUrlMatch[1];
    }
}

const client = buildClient({
    apiToken: "YOUR_API_TOKEN",
    logLevel: LogLevel.BODY, // Important, keep this! We have to parse it manually to get your job ID :(
    logFn: logParser, // This is the function that parses the log to get the job ID
});

async function run() {
    const smallFile = "https://upload.wikimedia.org/wikipedia/commons/c/ca/Crater_Lake_from_Watchman_Lookout.jpg";
    const bigFile = 'https://upload.wikimedia.org/wikipedia/commons/7/7d/%22_The_Calutron_Girls%22_Y-12_Oak_Ridge_1944_Large_Format_%2832093954911%29_%282%29.jpg'

    // First download the file locally
    console.log('\nStarting download...')
    const downloadPromise = downloadFile(bigFile, {onProgress: handleProgress})
    const {filePath} = await downloadPromise;
    console.log(`File downloaded to ${filePath}`)

    // Then upload it to S3 and get back a path
    console.log('\nStarting upload...')
    const remotePath = await uploadLocalFileAndReturnPath(client, filePath, {onProgress: handleProgress})
    console.log(`File uploaded to ${remotePath}`)

    // Tell DatoCMS to link the S3 file to your media area
    // Note that we do NOT await it. We will forcibly cancel it later.
    console.log(`\nStarting async job to create file in Dato from your S3 upload...`);
    const asyncCreatePromise = makeCancelablePromise(
        client.uploads.rawCreate(
            {
                data: {
                    type: "upload",
                    attributes: {
                        path: remotePath
                    }
                }
            }
        ));

    console.log('Created the promise, but still waiting for a job URL...');

    (function waitForJobURL() {
        if (jobUrl) {
            console.log(`Found Job URL. Canceling the promise now...`);
            asyncCreatePromise.cancel();
            console.log(`It's safe to exit now. You can ignore the canceled promise error and manually check job status at ${jobUrl}`);
        } else {
            setTimeout(waitForJobURL, 50); // Check again in 100ms
        }
    })();
}

function handleProgress(info) {
    console.log("Phase:", info.type);
    console.log("Details:", info.payload);
}

run();

It’s really a pretty ugly hack, but it gets the job done. I’ll try to get the devs to update the official client too.