Pagination during content insertion

🆘 Support
,
1

I have a question regarding which strategy DatoCMS applies.

Let’s say I have 2 different flows.
1 uses the CDA to users in real-time, the records in question are being fetched using DatoCMS pagination support. There are around 4000.

Meanwhile, another process is updating these records using CMA and could be adding/removing them.

What will happen to the users doing pagination? Is the pagination “aware” of this and the “cursor” remains intact or are we subject to possible duplicate results or results not being shown that should be?

2

Hi @siggi,

There is no guarantee that the CDA will return the records you expect. Every time you run a query with a first and skip parameter, it runs the query and returns the results in real time. There is no “snapshot” of results from before the updates.

For example, if you have a:

  • A background process on a loop that adds 1 record a second
  • A CDA query that’s fetching the first X records, skip Y (doesn’t matter)

Then by default it will return the most recently updated/inserted X-Y records. Changing your orderBy will change the records returned, but still not in a “static” way. Essentially, the CDA query doesn’t have the concept of a frozen state or previous context; it’s just querying the latest information, whatever the latest in.

I am not sure what your use case here is, but if you need to guarantee some “atomicity” of insertions/updates, such that the CDA will not return new data until ALL the updates are done, would one of these workarounds do the job?

  • You can insert/update the records as drafts, and then bulk publish them at once (up to 200 at a time): Publish items in bulk - Record - Content Management API. As long as your CDA query doesn’t include drafts, it will keep showing the old versions until you publish the updates.

  • If you need to update/publish more than 200 records at once, you could also consider using our environments for this, which allows you to make any edits you want in a separate sandbox, and then promote that sandbox to primary once it’s ready.

Does that help at all?

3

Thanks for the detailed reply Roger.

We will take a look at your suggestions and come back if we have any issues.