Fields updated by CMA leads to republishing content

This is more of a question on what is the best approach for doing this in DatoCMS.

We have multiple fields that are set by an automated process using the content management API. The model these fields belong to is using the draft/publish system because of the nature of the rest of the content in that model requires writers to verify the content before publishing.

Our current approach with the automated process is to update those fields and re-publish the content if the content was previously published by someone. That results in a lot of publishing activity as these fields are updated, regularly (think of fields tracking popularity of a piece of content).

We have thought of breaking those fields into a separate model that is linked to the original model, but I was curious if there was a better way to do this. While, the new model sounds like a decent solution, it will also result in doubling our record counts for a model that is already quite large and growing.

In summary, we’re trying to regularly update a set of fields (every 10-20 minutes) for a model that is using the draft/publish system and find that it creates too much publishing activity. These fields are used in the display of the content on our website, so they are important to what we do.

Hey @jearle, welcome to the forum!

Is it necessary for each record to keep its own metadata? What about a separate, single-instance model with a simple JSON field that can keep track of this for you?

Something like this:

(Edit: If you do this, be sure to use optimistic locking to avoid edit conflicts in case two records are liked at once and both try to update the record at the same time.)

How many records do you have (or expect to grow into)? Depending on if we’re talking about a few dozen, a few hundred, or much more… you may eventually run into scaling constraints here. Each record has a size limit of 300KB, for example, so that JSON field can’t be infinitely big. But you can also use other protocols, like MessagePack in a plaintext field, to fit more more data in the same record.

But that should significantly cut down the number of publishes you need to do.

Alternatively, depending on how your frontend is currently set up, that’s the sort of data that could maybe live somewhere else entirely, like cloud key-value stores on Vercel or Cloudflare? Since it’s just metadata, it won’t benefit from our editor UI anyway, and doesn’t really have to live in DatoCMS.

I think our CMS makes sense for write-rarely, read-often content that needs a pretty UI for editors, like your content text and media. But for transactional, write-often-read-often interactions or metadata (like “Visits” or “Likes”), our system and API aren’t necessarily the most suitable choice. Sure, you could squeeze it in there (as you have), but it may not be the best developer or end-user experience…?

As another alternative… this is a really ugly hack… but you might be able to abuse the Locales system for this. I don’t recommend the following because of its confusing impacts on developer and editor experience, so it’s just a last resort if nothing else would work…

Only consider me if none of the above options work for you

Let’s say your content is usually in English (or several languages). You can add a new locale that you know you’ll never use, which should be possible because there are dozens to choose from, like the locale ff (Fulah) just as an example.

You can make the “Likes” field localizable, but then hide that field from editors and don’t give them permission to edit it. Your API can update that field only in the Fulah locale and query it directly in draft mode without having to republish it.

But the downside is that it would be really messy and confusing to your editors, because they would see the record as changed:

You might be able to work around that with the roles & permissions system, but I still think it’s an ugly and brittle hack…

Hi roger,

Thanks for the input. We have talked about using a separate data store for this issue and that may be the route we pursue.

We’re moving from a self-maintained system to DatoCMS and other technologies/services and it requires some re-thinking of how we would normally do things. I wanted to check with you guys and get your input before checking with other options.

Thank you

1 Like

Well, at least that’s how I would do it if I were building such a site :slight_smile:

I guess that’s the part of the pros and cons of a headless setup as opposed to be a big ol’ monolith… your editors get a nice, easy place to manage content, but other website features usually need to be added in a la carte from elsewhere. It’d be a similar situation if you needed to process e-commerce orders, forms, and other user-generated content.

I think it’s a really valid and interesting use case, but not one that we’ve specifically solved for yet. Sorry about that!