This is more of a question on what is the best approach for doing this in DatoCMS.
We have multiple fields that are set by an automated process using the content management API. The model these fields belong to is using the draft/publish system because of the nature of the rest of the content in that model requires writers to verify the content before publishing.
Our current approach with the automated process is to update those fields and re-publish the content if the content was previously published by someone. That results in a lot of publishing activity as these fields are updated, regularly (think of fields tracking popularity of a piece of content).
We have thought of breaking those fields into a separate model that is linked to the original model, but I was curious if there was a better way to do this. While, the new model sounds like a decent solution, it will also result in doubling our record counts for a model that is already quite large and growing.
In summary, weāre trying to regularly update a set of fields (every 10-20 minutes) for a model that is using the draft/publish system and find that it creates too much publishing activity. These fields are used in the display of the content on our website, so they are important to what we do.
Is it necessary for each record to keep its own metadata? What about a separate, single-instance model with a simple JSON field that can keep track of this for you?
(Edit: If you do this, be sure to use optimistic locking to avoid edit conflicts in case two records are liked at once and both try to update the record at the same time.)
How many records do you have (or expect to grow into)? Depending on if weāre talking about a few dozen, a few hundred, or much moreā¦ you may eventually run into scaling constraints here. Each record has a size limit of 300KB, for example, so that JSON field canāt be infinitely big. But you can also use other protocols, like MessagePack in a plaintext field, to fit more more data in the same record.
But that should significantly cut down the number of publishes you need to do.
Alternatively, depending on how your frontend is currently set up, thatās the sort of data that could maybe live somewhere else entirely, like cloud key-value stores on Vercel or Cloudflare? Since itās just metadata, it wonāt benefit from our editor UI anyway, and doesnāt really have to live in DatoCMS.
I think our CMS makes sense for write-rarely, read-often content that needs a pretty UI for editors, like your content text and media. But for transactional, write-often-read-often interactions or metadata (like āVisitsā or āLikesā), our system and API arenāt necessarily the most suitable choice. Sure, you could squeeze it in there (as you have), but it may not be the best developer or end-user experienceā¦?
As another alternativeā¦ this is a really ugly hackā¦ but you might be able to abuse the Locales system for this. I donāt recommend the following because of its confusing impacts on developer and editor experience, so itās just a last resort if nothing else would workā¦
Only consider me if none of the above options work for you
Letās say your content is usually in English (or several languages). You can add a new locale that you know youāll never use, which should be possible because there are dozens to choose from, like the locale ff (Fulah) just as an example.
You can make the āLikesā field localizable, but then hide that field from editors and donāt give them permission to edit it. Your API can update that field only in the Fulah locale and query it directly in draft mode without having to republish it.
But the downside is that it would be really messy and confusing to your editors, because they would see the record as changed:
Thanks for the input. We have talked about using a separate data store for this issue and that may be the route we pursue.
Weāre moving from a self-maintained system to DatoCMS and other technologies/services and it requires some re-thinking of how we would normally do things. I wanted to check with you guys and get your input before checking with other options.
Well, at least thatās how I would do it if I were building such a site
I guess thatās the part of the pros and cons of a headless setup as opposed to be a big olā monolithā¦ your editors get a nice, easy place to manage content, but other website features usually need to be added in a la carte from elsewhere. Itād be a similar situation if you needed to process e-commerce orders, forms, and other user-generated content.
I think itās a really valid and interesting use case, but not one that weāve specifically solved for yet. Sorry about that!
