Allow migrations to be applied directly to the primary environment

✋ Feature Requests
1

This document explains how to apply migrations to the primary/main environment:

The requirement to create a forked environment, apply the migrations, and promote the environment to primary/main is a real impediment to how we work.

By the time we’re ready to apply one or more migrations to the primary environment, we’ve already applied them to our “development”, “test”, and “uat” environments. So we know they work, and we know our application works with them.

We can’t promote any of those environments to primary because they will contain non-production content.

So we’d like to be able to apply our migrations directly to the primary environment. The CLI can issue a warning or require a special argument to indicate we know what we’re doing. But this would make our lives a lot easier.

Thank you.

2

We just discovered a more serious issue with the “fork - apply migrations - promote” approach to applying migrations to the primary environment.

The name of the new environment will be different than the previous/original primary environment. And that means the environment name we pass in the X-Environment header to the API has to be updated every time we need to get migrations into the primary environment.

This isn’t a scalable process. It really needs an alternative approach.

Thanks.

1 Like
3

We worked around the X-Environment issue to not pass this header when we’re deploying to production. If the header is not set it always uses to primary environment.

1 Like
4

We have the same issue! Even if it’s possible to work around it, I think it should be possible to have a way to target the primary environment without having to take into account that it can change name :thinking:

5

Have there been any changes regarding how to apply migrations to the primary environment?

If not, for reasons explained in this thread, we won’t be able to use the migrations feature. That would be really disappointing - migrations is one of the features that led us in the direction of DatoCMS (which, overall, we’re extremely happy with!). We just simply cannot be required to fork the primary environment before applying a migration and then promote the forked environment to be primary.

That’s not the way database migrations work, and we do those all the time.

Please let me know. Many thanks.

6

Sorry @donnie.hale, it’s still the same as before :frowning: However, in early April of this year we did release the ability to rename environments: Environments renaming — DatoCMS Does that help at all?

Unfortunately, as far as I know there is still no planned work to allow migrations to be done directly to the primary environment.

In your original post, you said:

We can’t promote any of those environments to primary because they will contain non-production content.

Per the guide, our recommendation is to put the primary env in maintenance mode, create a fresh new fork (at the time of migration), apply the migration there, and then promote that to primary (renaming it if you need to). That shouldn’t have any of your content from the other non-production sandboxes.

If that workflow still wouldn’t work for you, could you please explain why? It would help our developers better understand the shortcomings of the current system, and hopefully rethink the situation here…?

Thank you, and sorry for the bad news!

7

Thank you very much for your reply - that’s super kind of you.

I tried to explain why the “fork - apply migrations - promote” workflow doesn’t work for us in the original post of this issue, but I’ll take another stab at it. In short, I think it’s a difference in presuppositions about how a CMS fits within an overall application platform. We see a CMS as just another “external service” that makes up the entirety of our platform whereas it seems (I could be wrong) that you (DatoCMS) view the CMS as having its own way of being used within a platform.

We have 4 environments that are part of our development process and lifecycle: Development; Test, UAT; and Production. Each environment has its own: static web app and/or mobile app (front end); API (app service); database; blob storage; etc. Apart from some kind of catastrophe or massive requirements change, those environments are fixed. We apply migrations to our databases on pretty much every release. When we do that, we don’t fork a database and then apply migrations and then promote the result to become the new database.

Our expectations for our 4 corresponding CMS environments are the same. Because of how we manage any of our environments, including CMS environments, we don’t have to worry about anything we don’t want being in our primary environment at the point that we’re ready to apply a migration. Anything like that would have been done in one of the other environments. Our primary/production environment is always up-to-date and correct with production content.

I’d argue that it’s also inconsistent to allow us to directly apply migrations to non-primary environments but not to the primary environment. If it’s bad for the primary environment based on your premises about why you should first fork, it’s bad for lower environments for the same reason.

In short, having to fork and promote sort of throws a wrench into how we do everything related to releases for the entire rest of our platform. We’d also have to do the “rename” of the promoted environment. And at some point we have to delete the previous primary environment. When we do a release to Development, or to Test, or to UAT, we don’t have to fork-apply-promote-rename. All of a sudden, to release to production, something quite different has to take place.

All of that is overhead we don’t have to do for any other part of our platform. And it all seems to be based on a premise of how a CMS fits into a larger platform or or how a CMS should be used - a premise which isn’t true for us.

Sorry to be so long-winded. Let me know if you have follow-up questions. And, sincerely, thanks again for the follow-up.

8

Thank you, @donnie.hale! Your example helps a lot in illustrating how our systems are used by real-world teams in their own workflows. And it’s clear that this limitation has an impact for your team. Thank you again for the detailed explanation :slight_smile:

For what it’s worth, I agree that simply allowing an override would make this process a whole lot simpler. For example, this is currently what happens if you try to force a migration on the primary env:

image

It’s forbidden. Does it have to be? I’m not sure if this the result of some deep philosophical view they have (@mat_jack1 and @m.finamor, any thoughts?) or maybe they were just afraid it’d be a risky footgun…? But maybe allowing that to happen with an override flag/confirmation like --allow-primary-in-place --confirm-dangerous-operation would be enough…? (I believe that’s what you’re hoping for, right?)

In any case, just so you know how these feature requests work, generally the developers plan new work about every 6 weeks, some of which come from the top-voted feature requests. Not all feature requests get implemented, and of those that do, some of them can take a long time… (as you already see, sorry about that!).

As a support agent, the extent of my powers is to give these a preliminary review, requesting more information where needed, and trying to provide workarounds if possible. Unfortunately, I can’t directly implement this feature for you :frowning: That’s a long-winded way of saying, essentially, that there is not yet an ETA for implementation of this feature, as it is not planned work right now. It might take a while, even if it’s chosen for eventual implementation.

However, what I CAN do is try to help provide some workarounds if you need to move ahead on a quicker timeline…? For example, some things we could explore to see if they’d work in your team’s workflow:

  • Just running the migration scripts directly, outside the CLI’s purview. The scripts the CLI generate in the ./migrations folder are really just simple CMA API commands (see example script) that you can manually run against any environment you like (including the primary). In your build pipelines, you can import our scripts and run them with any params you want by supplying your own client instance.

  • We could make a simple CI/CD pipeline script (in JS, Bash, probably YAML-based runners too, anything that can execute NPM commands programmatically) that automates the tedious release process of maintenance-fork-migrate-promote-rename-delete by programatically executing the appropriate CLI commands.

  • We could fork the CLI app itself (which is open source) to remove the limitation on primary in-place migrations (and possibly contribute it as a PR back upstream, which would get it merged into the official one if accepted).

None of those are perfect solutions, but they would likely be quicker than waiting for an official implementation of this…? Again, not trying to convince you of any one approach to this – in regards to your “difference in presuppositions about how a CMS fits within an overall application platform” concern – just trying to find a pragmatic way to get this working for you one way or another.

We should definitely leave this feature request in place so the devs can see and further evaluate it (and hopefully implement it), but in the meantime, please let me know if you’d like any help exploring any of the workarounds?

9

Once again, thanks for the reply.

A parameter to the migrations command line to allow the migration to happen against “primary” would be a great solution.

We will probably just wait for the permanent fix vs. trying to adopt one of your workarounds. But thanks for providing them.

1 Like