Load video by lambda function and NodeJS client

Hi!
We use simple forms in Gatsby with lambda functions on Netlify to add a new record in DatoCMS.
In one form we need to send and write video, so we use NodeJS client and base64 format for file .

Is any limit for a size of files written by this way in DatoCMS?

Best regards,
Helen

1 Like

Hello Helen!

We have currently tested files up to 100MB. We might consider lifting this limit even further, but I’m not sure about that.

Is 100MB good enough for you?

Thank you for answer!
100 MB is good enough for our short videos.
But we have got the error “Your lambda function took longer than 10 seconds to finish.”
May be using Base64 for transport file to lambda function and writing files from this format is not a good way for video files.

Could you give the cue for better way to send video file by the form?

Best regards,
Helen

1 Like

Aren’t you able to upload directly from the browser maybe?

Have a look here:

https://www.datocms.com/docs/content-management-api/resources/upload#create

you can do step 2 from the browser and step 1 and 3 from your lambda so that you don’t have to share the write token to the browser, what do you think?

Inspect what the browser is doing in our media area, it’s the exact same idea :slight_smile:

Thank you for your piece of advice!

We try to reproduce step 2 and get first response with status “200 OK” :

But after that we have got another response with Error status “403 Forbidden”:

Could you help us to resolve this problem?

Best regards

The request URL for the PUT should also have some credentials, some more GET params.

And also you should append as payload the binary of your file.

Are you sure you are using the full URL you got from the POST to the /upload-request?

We use axios to send data

try {
        const res = await axios.put(
          `https://dato-images.s3.eu-west-1.amazonaws.com/9865/${shortVideo.name}`,
          {
            file: shortVideo,
          },
          {
            headers: {
              'Content-Disposition': `attachment; filename=${shortVideo.name}`,
              'Access-Control-Allow-Origin': '*',
              'Access-Control-Allow-Methods': 'PUT',
              'Content-Type': shortVideo.type,
            },
            params: {
              'x-amz-acl': 'public-read-write',
            },
          },
        )

        if (res.status === 200) {
          this.setState({
            isSended: true,
            form: initForm,
          })
        }
      } catch (error) {
        this.setState({
          isSended: true,
          isFailed: true,
        })
        // TODO: Add error handling
        console.log(error)
      }

where shortVideo = acceptedFiles[0] from <Dropzone /> from ‘react-dropzone’

We are not sure about full URL, but we couldn’t see it from example
PUT https://dato-images.s3.eu-west-1.amazonaws.com/205/1565776891-image.png?x-amz-acl=public-read&... HTTP/1.1<YOUR_FILE_BINARY_CONTENT>

The URL comes from the call done in the previous step, you cannot do it as you are.

Are you doing the call as in step 1 before: https://www.datocms.com/docs/content-management-api/resources/upload#create ?

As I mentioned here: Load video by lambda function and NodeJS client you should do step 1 in the lambda and get the URL to the browser where you can do the upload and then go back to the lambda to create the upload object in Dato. Is that any helpful?

Hello Matheo. We try to get URL with AWS Lambda function by Axios. There is our code:

const KEY = '***SECURE KEY***'
const options = {
    headers: {
      Authorization: `Bearer ${KEY}`,
      'Content-Type': 'application/json;charset=utf-8',
      Accept: 'application/json',
      'Access-Control-Allow-Origin': '*',
      'X-Api-Version': '2',
    },
    data: {
      type: 'upload_request',
      attributes: {
        filename: 'video.mp4',
      },
    },
  }

  try {
    const res = await axios.post('https://site-api.datocms.com/upload-requests', options)

    if (res.status === 200) {
      console.log('getUrl', res)
      return res
    }
  } catch (error) {
    console.log('error lambda', error)
    return error
  }

But, we have got error with status code 406. Please, tell us, about requered headers. Maybe we missed some required attributes?
We took required attributes from documentation ( Step 1: Request an upload permission)

Best regards.

Hello @dev

try this:

const options = {
  headers: {
    'Authorization': `Bearer ${KEY}`,
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'X-Api-Version': '2',
  }
};
const data = {
  data: {
    type: 'upload_request',
    attributes: {
      filename: 'video.mp4',
    },
  }
};

try {
  const res = await axios.post('https://site-api.datocms.com/upload-requests', data, options);
  console.log(res.data);
} catch (error) {
  console.log('error lambda', error);
}

I think it was just a misconfiguration in axios. Works fine here, but let me know otherwise!

Thank you very much!
We have got url with your help.

1 Like

Hello

Unfortunaly, we’re stuck on loading big files to S3.

Here is our code using node for loading files to S3.
It works for small files, but we can’t load even a file with size 14.5 Mb there.

const axios = require("axios");
var fs = require("fs");

const KEY = "00000000000";

async function getUrl() {
  const filename = "test-video.mp4";

  const options = {
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
      Accept: "application/json",
      "X-Api-Version": "2"
    }
  };

  const data = {
    data: {
      type: "upload_request",
      attributes: {
        filename: `${filename}`
      }
    }
  };

  try {
    const res = await axios.post(
      "https://site-api.datocms.com/upload-requests",
      data,
      options
    );
    const url = res.data.data.attributes.url;
    const file = await fs.readFileSync("src/assets/t_video546.mp4");
    return { url, file };
  } catch (error) {
    console.log("error", error);
  }
}

async function loadFile(url, data) {
  const config = {
    method: "put",
    url: url,
    data: data,
    headers: { "Content-Type": "video/mp4" }
  };

  try {
    let res = await axios(config);
    console.log(res.request);
  } catch (error) {
    console.log("error loadFile", error);
  }
}

getUrl()
  .then(res => {
    console.log("res getUrl", { res });

    loadFile(res.url, res.file);
  })
  .catch(err => console.log("Error getUrl", err));

For big files we got Error: Request body larger than maxBodyLength limit

But according doc (look at Uploading Objects)

Depending on the size of the data you are uploading, Amazon S3 offers the following options:

Could you help us to resolve this ussue

Best regards

can you try this loadFile function please?

I think the problem is with setting the maxContentLength config parameter in axios.

But let me know if I’m wrong.

Thank you a lot!
It has helped :slight_smile:

1 Like