Now that Nextjs 16 is live with support for cacheTags which go up to 128 tags, is this model compatible with DatoCMS CacheTags?
The previous solution with storing the DatoCMS cacheTags in a DB is cumbersome and since the Nextjs cacheTag lets you tag a cache after a fetch, this would be a perfect fit.
So I guess the question is, how many CacheTags can DatoCMS return (please be <=128)
Hey @k.sprengers,
Thanks for the heads-up!
Hmm, I wonder when they increased the cache tags limit. It used to be 64 at some point, but their docs now say 128, and that applies back to v14 at least.
Iâll bring this up with the devs and see if that new limit is high enough. I know we currently return more than 64 sometimes, but not sure if it goes up to 128. Iâll report back as soon as I find out!
Sorry, we can sometimes exceed more than 128 cache tags, still 
Our defined limit is the lesser of either 500 cache tags OR 14 KB in header size. This is due to some upstream CDN limits. But either of those are still more than Nextâs 128 limit, so Iâm afraid having an external tag cache is still the way to go for now (or using another approach, like ISR, or time-based fetch revalidation, or âon publishâ webhooks, etc.)
Itâs also possible that simpler use cases with few records and a simple enough schema may never reach 128 cache tags in a response, but we cannot guarantee it. And even if it works in the beginning, setting up a project that way means that eventually it might fail if it keeps growing⌠not a great idea since that can often happen long after the original devs and editors are gone, and the new people are left figuring out why some pages refresh correctly and others donât⌠probably better to go with something more deterministic and safer.
Hey @roger , just out of curiosity. What if we compress the DatoCMS tag set (on the Next.js site) into a small, deterministic set of âbucket tagsâ that always stays within Nextâs limits?
I donât quite understand, sorry, could you please explain and maybe provide a hypothetical example?
It would make the whole revalidating caches in Nextjs way more easy if we either could limit the total cache tags in a way. Or if Nextjs or in my case Vercel can increase the limits (I asked Vercel this very question, but still waiting for a response).
But for now it might be interesting information on the Dato docs â 500 cache tags OR 14 KB in header sizeâ
Yes, that would be great Next.js/Vercel is unfortunately one of the frameworks that has this issue, and itâs not something we are able to change on our end (itâs up to Vercel and their CDNs).
In our Next starter kit, we have an example of how to map Dato cache tags to specific Next queries and then store that in an external KV (in this case, Turso): https://github.com/datocms/nextjs-with-cache-tags-starter/blob/main/lib/database.ts
This other discussion also has an example of how to modify that implementation to work with a redis-compatible store: NextJS + Vercel + Dato Cache Tags not always working - #5 by roger. At the time, that example targeted Vercel KV, a now-defunct, whitelabeled version of Upstash. Vercel deprecated that and now just links to third-party redis implementations, but the basic logic should still be the same and it shouldnât be too hard to modify it e.g. for Valkey or similar.
This sort of mapping is not ideal, but is necessary as long as Next/Vercel has these small cache tag limits
I know this is frustrating, but as far as we can tell, our hands are kind of tied here due to Nextâs limits. This is something weâve given a significant amount of thought and experimentation to in the past, and could not find a good solution to, hence our workarounds and all the mapping needed. If anyone has a better idea about how we could or should do this (like @martin.palma âcompressed bucketsâ ideaâŚ? which I still donât quite understand, sorry)⌠please let us know and we can definitely investigate!
Hey @roger, what I meant with âdeterministic bucketingâ:
bk:...:0 â bk:...:119So instead of attaching up to 500 DatoCMS cache tags, you attach at most B buckets.
The attentive reader will clearly spot the main problem with this approach:
Multiple DatoCMS cache tags will land in the same bucket, therefore revalidating a bucket can cause revalidation of content which wasnât changed.
Iâm thinking Iâm going to test this approach for the project, since:
Hope this make sense.
I see, @martin.palma, thank you for explaining.
Below Iâll provide my 2c, but Iâll also ping our cache tags developer to see if they have any thoughts theyâd like to share.
My opinion only (Iâm not the person who developed them):
I suppose that could work, but itâd have to be something you implement on your own, for your particular frontend(s). e.g. for a response we return like:
$o )>l "s3ze_ "l-hv6 ##<-k~ @i(yu\p^k#*-&qmv&t@)6x# #cy4~z "{[.@4 "{[.@j )._6s)m )._6rk/ )._6r{j )._6s"d )._6s$4 )._6s%! )._6s&n )._6s&y )._6s@* )._6s@+ )._6s@0 )._6s@: )._6s@h )._6s@{ )._6s@~ )._6s'* )._6s4$ )._6s5( )._6s9= )._6s9_ )._6s>< )._6sc\ )._6sc] )._6sdd )._6sz+ @gti4%[s0@&%h_6>\>%rwks "{[.@z )._6sck )._6rw$ )._6rk[ )._6rl[ )._6s)d (plc>?'
Youâd have to figure out a way to hash the different parts into several shared buckets. But that wonât be easy because the tags are opaque, so the hash function doesnât necessarily know what )._6s references even though it appears as a prefix several times. You canât necessary guarantee an orderly or sensible mapping of schema/record/metadata : cache tag : related CDA queries : your hash : frontend components : one or more URLs.
So it would be deterministic in the sense of âthese certain characters will always be hashed to the same bucketâ, but not in the sense of âthis model will always go to this bucket, this relationship will always go to that other bucket, etc.â
The hash tags are already âcompressedâ in the sense that they are encodings of larger chunks of metadata, e.g. âpublished record #627049417â becomes )._6sxz[ (just an example).
The reason thereâs so many of them is because of relationships in the CMS (uploads, locales, linked records, etc.); our backend analyzes these relationships and generates the proper cache tags for all the related items too. Itâs like a diff of a relationship graph.
In simpler setups with only a few isolated records, you wonât have that many cache tags to begin with, but in more complex projects, each change can trigger a cascade of relationship updates which in turn trigger their own cascade. Itâs exactly as you said:
Except the problem will scale as the complexity of the content model(s) and queries scale.
If you erase that sort of relationship encoding by hashing/truncating cache tags into fewer buckets, then yes, you can squeeze them into smaller headers, but then youâre essentially saying âwhenever this undeciphered blob updates, also invalidate all of these other random things, even if we have no idea what any of the things areâ. I suppose that would work, but it would be quite difficult to reason about or maintain/troubleshoot.
If the specificity of the invalidations is important, bucketization would lose that. If specificity isnât important, I think âon publishâ webhooks that invalidate too much would have the same basic effect while being easier to reason about. For in-between needs, probably time-based ISR or fetch caching would still be much simplerâŚ? It is also possible to cache Next.jsâs outputs behind a CDN and have the CDN apply page-level cache tag invalidations instead.
Regardless, you are certainly free to try this if youâd like, but it seems maybe a tad over-engineered, and even more opaque than the cache tags already are?
Iâll ping the dev who wrote this functionality and let you know if they have any further thoughts!