The script migration tool in the CLI is generally advised for schema migrations between environments, not for content migrations. Content migrations between environments are generally ill advised as blocks and linked records have different IDs between environments (which is also what is causing the problem on your end of running the script multiple times)
So the safe and recommended ways to “transfer” content between environments is to fork and promote them when necessary.
The best way to go here would possibly be to: Fork the main environment when you are starting to work on it, and enable maintenance mode on the main environment for the duration of your work (not the two weeks, just a partial day-stretch) once done, promote that environment as a main environment, and if necessary, block the access to unfinished models or “work-in-progress” sections through roles, this way only developers can access unfinished parts of the environment.
Another solution here would be to add some logic to the script to detect an update in a block through a UUID (which can become quite complex very fast, specially if you have nested blocks) and is probably not the way to go
I understand that this is your recommendation when the workflow is one way, I fork primary env make the changes needed promote the env and everything is fine.
I have a different case, which for you said migrations exists in the first place, I have primary env that was forked twice, one for each development branch as we are working in parallel.
after first changes were approved and sandbox1 was promoted to primary I use the migration script to add the changes that are now on primary to sandbox2 before promoting it to be main.
But in this case, and as for your recommendation (and technical limit because of different ID’s across env’s) I need to migrate the schema, and then go and add manually all the content that is now missing? this is a huge change and such a cumbersome solution.
could you have a more streamlined solution for my workflow?
Sorry, this isn’t something we currently support I’ve converted this into a Feature Request for you, however.
In the meantime, if the sandboxes only ever have new records from primary added to them (and there’s not a need to incrementally update them after creation), you can use the CMA to look for new records on primary and create them on the sandboxes: Create a new record — Record — Content Management API
But yes, as you noticed, this workflow will not easily work with our current environments system. Sorry about that!
so in case of adding new models with content to primary env you would even suggest against you default migration script but to write my own that copies new entries with their content from primary to sandbox? — this is somewhat of a solution but the truth of it is that we need to be on our toes all the time to see exactly what changes were made and what kind of solution we need.
Thank you for making it into a feature request — indeed we feel that development environment can vary in the way companies work, this is a big hiccup, only administrator really know the changes made to the schema, so now we put them in the role of content people to go around DatoCMS and update content, and to hope that they got everything correct, because also fields not marked as required can just stay empty and be missed.
Of course that testing is part of our workflow and staging env and all the precautions but its just adding another level of complexity that using a CMS will usually here to solve
Yes, unfortunately, the migration scripts ONLY copy schema and never content. If you need to copy content across environments, typically it’s just easier to fork an env (which clones everything, schema and content and plugins etc.). Otherwise, it will indeed be very difficult to keep content synced across multiple environments (requiring cumbersome & fragile manual use of the CMA).
In your situation, within the constraints of our system, could you maybe do something like this:
(As an example only):
You have a Primary env (the default), where editors work. (This assumes this is the ONLY environment where real content changes are made.)
You fork Primary into a new Dev 1 environment and do some work on that. A developer makes schema changes there, but all the test content isn’t real and can be discarded.
At the same time, you also fork Primary into a separate Dev 2 environment, where another dev does the same.
Dev 1 finishes their work and is ready to merge…
(Optional but recommended) You can briefly turn on Maintenance Mode for the primary env (just for a few min), so editors don’t make any changes while you’re merging.
You fork Primary into a new env, let’s call it Staging. This will have the latest content from Primary. Then you autogenerate and run a migration script from Dev 1 against Staging.
Now Staging will have the latest content from Primary and the latest schema from Dev 1, combined. It will NOT have any content from Dev 1 (but there should be no real data there anyway). You can now promote Staging to the new Primary and turn off maintenance mode.
Dev 2 finishes their work and is ready to merge.
You repeat the same process, forking the new Primary (which contains Dev 1’s schema changes and any additional content added between then and Dev 2’s merge), etc.
Basically, you’d just have fork the Primary into a new Staging env every time, and run the migrations against that. That way your editors only ever have one env (Primary) to edit content in, and you never have to manually sync content across envs.
It can be cumbersome, yes, but probably less so than trying to copy data back and forth manually
I know that workflow won’t work for every team / org. I do agree that it would be helpful to have the ability to easily copy content between envs (assuming compatible schema), it’s just not something we offer right now without manual CMA commands. Sorry about that
Let’s leave this feature request though and see if it becomes more popular (please remember to vote on it at the top).
I’ve converted this into a Feature Request for you, however.