Uploads via createUpload is crazy slow

We had an architectural discussion about this and found out a little more information:

  • The reason it takes so long is that during that time, our backend is making several other API calls to generate blurhashes, thumbhashes, smart tags, dominant colors, and other EXIF and metadata.
  • A developer is investigating whether it’s possible to postpone some of these tasks until later on, but that may not be an easy thing to do (because it would probably need additional “image created but not all metadata ready yet” states in the API and UI). I wouldn’t count on this being added anytime soon.
  • However, we did find a cleaner workaround for the client to exit early (EDIT: fixed some bugs):
const client = buildClient({apiToken: "XXX"});

// Configure the client to fast return with a fake response instead of polling for the real job status
client.jobResultsFetcher = async (jobId) => {
    return {
        id: jobId,
        type: "job_result",
        status: 200,
        payload: {
            "data": {
                id: jobId,
            },
        },
    }
};

// This call will be much faster (no waiting for serverside operations), but the actual
// upload entity will not yet be created. It will actually be created when the job ends

// Optionally, you can generate your own Dato-style UUID client-side to have it immediately available:
const imageId = generateId(); // Must be a valid Dato-style ID
console.log(`Image ID will be ${imageId}`)

const result = await client.uploads.createFromUrl({
    id: imageId, // If you leave this out, the server will generate one for you, but you won't know what it is until the job is done
    url: "https://www.example.com/image.jpg",
});

// the result will be simply the job id:
console.log(result) // => { id: 'f62d2b3fb428b1ace205b5ad' }

// You can poll for it separately using 
// `await client.jobResults.find('f62d2b3fb428b1ace205b5ad')`

The end effect of this will be similar to my workaround above (you get a job status back immediately, at the risk of your client not waiting for 100% success of the image creation and all metadata being generated, i.e. you’ll need some separate, out-of-band check to ensure correctness). But it’s a much cleaner/shorter implementation using client.jobResultsFetcher() (which I didn’t know about before).