Load video by lambda function and NodeJS client

dev · September 2, 2019, 9:11am

Hi!
We use simple forms in Gatsby with lambda functions on Netlify to add a new record in DatoCMS.
In one form we need to send and write video, so we use NodeJS client and base64 format for file .

Is any limit for a size of files written by this way in DatoCMS?

Best regards,
Helen

mat_jack1 · September 2, 2019, 9:57am

Hello Helen!

We have currently tested files up to 100MB. We might consider lifting this limit even further, but I’m not sure about that.

Is 100MB good enough for you?

dev · September 2, 2019, 4:03pm

Thank you for answer!
100 MB is good enough for our short videos.
But we have got the error “Your lambda function took longer than 10 seconds to finish.”
May be using Base64 for transport file to lambda function and writing files from this format is not a good way for video files.

Could you give the cue for better way to send video file by the form?

Best regards,
Helen

mat_jack1 · September 3, 2019, 8:09am

Aren’t you able to upload directly from the browser maybe?

Have a look here:

https://www.datocms.com/docs/content-management-api/resources/upload#create

you can do step 2 from the browser and step 1 and 3 from your lambda so that you don’t have to share the write token to the browser, what do you think?

Inspect what the browser is doing in our media area, it’s the exact same idea

dev · September 3, 2019, 1:27pm

Thank you for your piece of advice!

We try to reproduce step 2 and get first response with status “200 OK” :

But after that we have got another response with Error status “403 Forbidden”:

Could you help us to resolve this problem?

Best regards

mat_jack1 · September 3, 2019, 2:14pm

The request URL for the PUT should also have some credentials, some more GET params.

And also you should append as payload the binary of your file.

Are you sure you are using the full URL you got from the POST to the /upload-request?

dev · September 3, 2019, 2:44pm

We use axios to send data

try {
        const res = await axios.put(
          `https://dato-images.s3.eu-west-1.amazonaws.com/9865/${shortVideo.name}`,
          {
            file: shortVideo,
          },
          {
            headers: {
              'Content-Disposition': `attachment; filename=${shortVideo.name}`,
              'Access-Control-Allow-Origin': '*',
              'Access-Control-Allow-Methods': 'PUT',
              'Content-Type': shortVideo.type,
            },
            params: {
              'x-amz-acl': 'public-read-write',
            },
          },
        )

        if (res.status === 200) {
          this.setState({
            isSended: true,
            form: initForm,
          })
        }
      } catch (error) {
        this.setState({
          isSended: true,
          isFailed: true,
        })
        // TODO: Add error handling
        console.log(error)
      }

where shortVideo = acceptedFiles[0] from <Dropzone /> from ‘react-dropzone’

We are not sure about full URL, but we couldn’t see it from example
PUT https://dato-images.s3.eu-west-1.amazonaws.com/205/1565776891-image.png?x-amz-acl=public-read&... HTTP/1.1<YOUR_FILE_BINARY_CONTENT>

mat_jack1 · September 3, 2019, 2:59pm

The URL comes from the call done in the previous step, you cannot do it as you are.

Are you doing the call as in step 1 before: https://www.datocms.com/docs/content-management-api/resources/upload#create ?

As I mentioned here: Load video by lambda function and NodeJS client you should do step 1 in the lambda and get the URL to the browser where you can do the upload and then go back to the lambda to create the upload object in Dato. Is that any helpful?

dev · September 4, 2019, 11:31am

Hello Matheo. We try to get URL with AWS Lambda function by Axios. There is our code:

const KEY = '***SECURE KEY***'
const options = {
    headers: {
      Authorization: `Bearer ${KEY}`,
      'Content-Type': 'application/json;charset=utf-8',
      Accept: 'application/json',
      'Access-Control-Allow-Origin': '*',
      'X-Api-Version': '2',
    },
    data: {
      type: 'upload_request',
      attributes: {
        filename: 'video.mp4',
      },
    },
  }

  try {
    const res = await axios.post('https://site-api.datocms.com/upload-requests', options)

    if (res.status === 200) {
      console.log('getUrl', res)
      return res
    }
  } catch (error) {
    console.log('error lambda', error)
    return error
  }

But, we have got error with status code 406. Please, tell us, about requered headers. Maybe we missed some required attributes?
We took required attributes from documentation ( Step 1: Request an upload permission)

Best regards.

mat_jack1 · September 5, 2019, 8:25am

Hello @dev

try this:

const options = {
  headers: {
    'Authorization': `Bearer ${KEY}`,
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'X-Api-Version': '2',
  }
};
const data = {
  data: {
    type: 'upload_request',
    attributes: {
      filename: 'video.mp4',
    },
  }
};

try {
  const res = await axios.post('https://site-api.datocms.com/upload-requests', data, options);
  console.log(res.data);
} catch (error) {
  console.log('error lambda', error);
}

I think it was just a misconfiguration in axios. Works fine here, but let me know otherwise!

dev · September 6, 2019, 7:57am

Thank you very much!
We have got url with your help.

dev · September 12, 2019, 11:16am

Hello

Unfortunaly, we’re stuck on loading big files to S3.

Here is our code using node for loading files to S3.
It works for small files, but we can’t load even a file with size 14.5 Mb there.

const axios = require("axios");
var fs = require("fs");

const KEY = "00000000000";

async function getUrl() {
  const filename = "test-video.mp4";

  const options = {
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
      Accept: "application/json",
      "X-Api-Version": "2"
    }
  };

  const data = {
    data: {
      type: "upload_request",
      attributes: {
        filename: `${filename}`
      }
    }
  };

  try {
    const res = await axios.post(
      "https://site-api.datocms.com/upload-requests",
      data,
      options
    );
    const url = res.data.data.attributes.url;
    const file = await fs.readFileSync("src/assets/t_video546.mp4");
    return { url, file };
  } catch (error) {
    console.log("error", error);
  }
}

async function loadFile(url, data) {
  const config = {
    method: "put",
    url: url,
    data: data,
    headers: { "Content-Type": "video/mp4" }
  };

  try {
    let res = await axios(config);
    console.log(res.request);
  } catch (error) {
    console.log("error loadFile", error);
  }
}

getUrl()
  .then(res => {
    console.log("res getUrl", { res });

    loadFile(res.url, res.file);
  })
  .catch(err => console.log("Error getUrl", err));

For big files we got Error: Request body larger than maxBodyLength limit

But according doc (look at Uploading Objects)

Depending on the size of the data you are uploading, Amazon S3 offers the following options:

Upload objects in a single operation—With a single PUT operation, you can upload objects up to 5 GB in size.For more information, see Uploading Objects in a Single Operation.

Could you help us to resolve this ussue

Best regards

mat_jack1 · September 12, 2019, 12:29pm

can you try this loadFile function please?

dev:

async function loadFile(url, data) {
  const config = {
    method: "put",
    url: url,
    data: data,
    maxContentLength: 10485760, // 10MB, you want more probably
    headers: { "Content-Type": "video/mp4" }
  };

  try {
    let res = await axios(config);
    console.log(res.request);
  } catch (error) {
    console.log("error loadFile", error);
  }
}

I think the problem is with setting the maxContentLength config parameter in axios.

But let me know if I’m wrong.

dev · September 12, 2019, 1:02pm

Thank you a lot!
It has helped