Multiple schema+data environments (production, staging, etc.)

Would be helpful to be able to sync existing models from one project to another.

Might be related to this https://github.com/datocms/feature-requests/issues/75 across languages.

Related: I would like see a solution for safely modifying models/content in a “development” context. In other words, I could safely prototype new fields AND content without breaking something in production.

I’m using DatoCMS to prototype new features and models in an online magazine that has a minimalist proprietary CMS. We could almost even use DatoCMS in production if there was a way to migrate changes more safely.

I realize there is a draft/publish feature, but it doesn’t really solve the issue of breaking a high traffic production app by modifying a field.

Even a way to sync or import/export between projects would be acceptable.

This would be a huge help with any moderate size project :+1:

Has anyone tried this via their Content Management API? Wondering if this is possible aswell.

@hellojebus you can already do that using our CM API, you can find more info here: https://www.datocms.com/docs/content-management-api/resources/item-type

I think we were discussing here about a built-in option to make the process easier.

I see this is tagged as “planned”. Does that mean there is a timeline for when this will be rolled out?

Would you currently have to set up two separate accounts if you wanted to experiment with different content on development vs. production environments?

To provide a little more background, we use DatoCMS basically as a page builder where people can select from different components and add content. Our ideal workflow would be having 3 environments with separate content models, content, deployments:

  • dev: Where developers can modify models and content to roll out new functionality (without fear of affecting prod data). Occasionally we’d restore dev with prod data to keep them in sync.
  • staging: Where all features (content models / content) are rolled out. This would be the environment used by writers.
  • prod: The data that is used to build the live website. We’d never want anyone to actually touch this directly. Whenever we want to deploy changes to prod, we’d just overwrite prod with the staging data.
1 Like

hello @hula and welcome to Community!

Yes, planned means that it’s on our backlog and we are about to start development of this! So it should come relatively soon.

Regarding your use case instead, you can duplicate the project for now (maybe get in touch at support@datocms.com as we do discounts for this case right now because there’s not the feature yet).

Also regarding staging/prod, isn’t the draft/published system (https://www.datocms.com/docs/general-concepts/draft-published) enough for you?

Hi @mat_jack1,

Thanks for the response and that information on the duplicate project. If we were to use a duplicate project, would the Dato team be able to migrate changes from the dev account to the prod account? Or would we need to manually migrate individual changes? If bulk migrations are possible, we would be able to make it work.

In terms of staging and prod, the draft-published system is very close to the functionality we’re looking for (and can be made to work), but we experience a few headaches while using it.

Currently, there is not way to bulk publish all drafted changes in the system. So if we were to stage a new design using the draft system, we would have to individually publish all those changes (which would cause our CI to be triggered several times).

In addition, we are using gatsby+netlify for production and would have to set up an actual server for the staging branch to constantly listen for drafted changes from DatoCMS. If there were a completely separate environment for staging, we would be able to use static deploys and feel more comfortable knowing that staging and prod were being served via the same method (both static).

Lastly, it would just be nice to know that minor mistakes on staging (such as accidentally publishing a record) would not update the production website.

So while this method would work short term, I believe the solution could be more robust with mass migration functionality.

hey @hula looks like you could use the “deployment environments” feature that we have already, since you are using a static site generation tool anyway.

I’m talking about this

https://www.datocms.com/docs/general-concepts/deployment

have you investigated that already? You can trigger a publication manually, and also restrict that functionality only on specific roles.

Hi @mat_jack1,

I believe we can use the deployment to make our staging / production process more seamless.

We could set up a server to host a writers version of the website that uses the staging code and automatically updates with Dato draft changes so that content writers can preview changes in real time.

We could then set up webhooks for whenever content is published to rebuild the static staging version of the website.

Then, once staging has been tested, we could use the deployment feature to publish all staging changes to the production version of the website.

It seems like the only thing this workflow is really missing is the ability to reference the actual current production content at a given time (an untouched version of Dato that tracks what is currently being used for production). However, we can probably just use the content history feature to look back if we needed to.

I suppose the only thing we would require this feature request for is to maintain a dev version of the website where developers can implement new content models before migrating them to staging. We would need to push content model changes up from “dev” to “staging” in order to provide new components / features AND also be able to restore “dev” from “staging” in order to keep it up to date whenever a lot of content is added by writers to the staging version (otherwise all of this work has to be done manually). For that reason, separate schema and data environments could still be very useful.

4 Likes

Curious if there’s an ETA for this since the topic is a few months old now

hi @operations,

right now we are writing documentation and we are adding a way to create migration scripts. You can find more info here. Right now it’s a closed beta. If you are interested reach out to us using our support form!

2 Likes

Thanks for the clarfiication.

Are there plans to add the ability to automatically create migration scripts by comparing the schema of two environments using the CLI? Right now it seems like it’s going to be a rather manual process and require insight into all available fields in the migration script.

I could be mistaken, though.

Currently it’s a manual process, you are right. The idea is that you do all the changes with the migration scripts, so that you don’t really need to compare the two.

So you don’t change the schema directly via the CMS, but you always use the migration scripts. Does that make sense?

Yes, that makes sense.

Are there plans to eventually automate the migration process by comparing the two environments? I imagine that maintaning migrations scripts could add some friction to workflows (especially when experimenting with multiple changes to the same content model).

2 Likes

I’m wondering the same thing as @hula. I think it would save a lot of time when we could automate this and generate the migration script by comparing a sandbox environment to the primary environment. I know this is probably a lot to ask for now, but it could be a nice addition in the future imo :slight_smile:

We’re thinking of creating an interactive CLI tool to create migration files faster but if generating them is already on your roadmap it would be a waste of time for us. Happy to hear your thoughts!

8 Likes

A CLI tool would certainly make the process less painful @floris.tenhove. However, this does seem like a feature that would pair very nicely with the new multi-environment feature if it is already part of the DatoCMS roadmap.

Without something that integrates a little better with the current Content Model console, it basically means that the recommended workflow for working with multiple environments is to completely ignore that feature of DatoCMS’ UI.

3 Likes

thank you @hula and @floris.tenhove your remarks are very useful. We’ll consider those very soon, if we cannot do anything quickly we’ll open a feature request to be considered for the future

3 Likes

oh, we forgot to update this feature request! This is now live :slight_smile:

We’re very happy with the sandbox environments and migrations! @ramon.gebben worked on a utils package to make our lives a bit easier when working with migrations, you can find it here: https://github.com/greenberrynl/datocms-migration-utils. Hopefully it can benefit others as well. It doesn’t cover all fields at this point and please beware it’s in active development.

We’re also planning on creating utils for easy data seeding as we want to separate data from structure.

@matjack1 do you perhaps have an update on generating migrations scripts from the current environment? Is this something that will be on your roadmap? Would love to get an update.