Error RATE_LIMIT_EXCEEDED when building Nextjs app

Hi, I receive this error when trying to build my nextjs app. I’m using p-limit inside getStaticPaths, otherwise I receive a timeout in vercel. How can I solve this limit error?

Hello @julian1

This error means that you are making to many parallel requests at the same time: Overview - DatoCMS

Our JS client manages this automatically by retrying the calls after some time, but perhaps by wrapping the client calls with p-limit you have blocked the auto-retry from the JS client.

I’m fetching the data with a normal fetch, because I’m using the GraphQL requests. Is there a way to increase the limit?

@julian1

In that case this limits are the ones that may apply: Content Delivery API - Rate limiting - DatoCMS Docs

Just as a confirmation, are you creating any records inside your project? Because it seems like your account is just over the record number limit, and that could be triggering an error

No, I’m not, I’m just reading data. And I have this problem in an account with Professional plan.

In that case then it must be the case that too many requests are being made in parallel and are going over the rate limit of 40 requests per second or 1,000 requests per minute per API token
So way to go here would be spacing out the requests so they meet the rate limits

@m.finamor We are trying to integrate DatoCMS with our current content, but the amount of requests we need to load all required data for all the content is massive and compromises our current Professional plan.

Our modelling is the following

  • Trip
  • Review
  • Landing Pages
  • Guide

We have 13k trips and 2k guides

To load trips to next.js we need to

  • Request all trips (to know their slugs). This is 130 requests, due to the 100 result limit in the GraphQL API
  • Request each trip for individual trip information (13k requests)
  • Request reviews for each trip (13k requests). Because DatoCMS is not providing inverse relationships.
  • Same for related Landing Pages.

This is over 40k requests to DatoCMS to just load 13k Trip pages.

If we wish to deploy once per day this leads to 30*40k=1.2M requests per month. Over the 1M limit for our current plan.

We also chose DatoCMS for multilingual content, so when we translate the 13k programs to other language, this is 26k published programs, and hence 80k requests per deploy so 2.4M requests per month. This is over the scale limit.

What can we do to tackle our use case? I don’t think our numbers are extreme or outrageous at all, and our startup has plenty of room to grow.

Also, this example here (https://github.com/datocms/nextjs-demo) would be flawed due to the expressed limitations above…

Please, can you give us a hint on how to solve our problem?

Hello @development1 and @julian1

The problem here is the separation of requests to “filter by trip” every time. However the optimal approach here would be to fetch everything in a single request, with their respective IDs (and reference IDs) and when the need to relate A review with a Trip, (or a landing page with a trip) comes up, you can simply compare the already fetched trip IDs with the trip reference ID:

query MyQuery {
  allTrips {
    id #Will be used to compare to the other models references
    slug
    title
    tripStatus
    tripType
    #...All other information you may need from a trip
  }
  allReviews {
    content
    trip {
      id #Used to relate to the corresponding trip Server side
    }
    #...All other information you may need from a review
  }
  allLandingPages {
    id
    title
    topTrips {
      id #Used to relate to the corresponding trip Server side
    }
    #...All other information you may need from a landing page
  }
}

This would reduce the 40k requests to 130 requests, furthermore, after adding locales you wont have to do additional requests, as when you fetch localised content without specifying the locale it fetches all locales by default.

Would that be a solution for your use case?

Another possible solution would be to re-model the schema, as an example, making all reviews being nested inside their respective trips, as blocks or record links, this way you wouldn’t have to neither filter on graphQL nor compare IDs, as all of the trips would have their respective reviews nested inside them. (same goes for landing pages, or any other related concept)
This would be a cleaner solution, but would require a restructuring of your models.

@m.finamor Thanks a lot for shedding some light on this! We’ll give it a shot combining this tactic with some next.js hacks.

1 Like