Bug: Autogenerating and running migrations tries to generate schema_migration model twice

[ldc]

Our intended workflow to apply changes to multiple environments (staging, uat, primary).

1. Fork the staging environment → staging-copy

2. Apply changes on staging-copy by using the web interface

3. Generate a migration script.

   npx datocms migrations:new 'test' --autogenerate=staging-copy:staging

   • This generates code to add a schema_migration model.

     console.log('Create model "Schema migration" (`schema_migration`)');
     await client.itemTypes.create(
       {
         id: 'Xe4JOR40Q_mKYcM__-mjNQ',
         name: 'Schema migration',
         api_key: 'schema_migration',
         inverse_relationships_enabled: false,
       },
       { skip_menu_item_creation: true },
     );
   
     console.log('Creating new fields/fieldsets');
   
     console.log(
       'Create Single-line string field "Migration file name" (`name`) in model "Schema migration" (`schema_migration`)',
     );
     await client.fields.create('Xe4JOR40Q_mKYcM__-mjNQ', {
       id: 'eCHXonAyQD2yxELfGb-2ZA',
       label: 'Migration file name',
       field_type: 'string',
       api_key: 'name',
       validators: { required: {} },
       appearance: { addons: [], editor: 'single_line', parameters: { heading: false } },
       default_value: '',
     });
   

4. Apply the changes to staging

   npx datocms migrations:run --source=staging --in-place

   This command implicitly generates the schema_migration model on staging.

   Now, the migration fails because it tried to generate the schema_migration model twice.

   Error: POST https://site-api.datocms.com/item-types?skip_menu_item_creation=true: 422 Unprocessable Entity
   
   [
     {
       "id": "c97148",
       "type": "api_error",
       "attributes": {
         "code": "INVALID_FIELD",
         "details": {
           "field": "name",
           "code": "VALIDATION_UNIQUENESS",
           "record_type": "ItemType"
         }
       }
     },
     {
       "id": "ce9dcb",
       "type": "api_error",
       "attributes": {
         "code": "INVALID_FIELD",
         "details": {
           "field": "api_key",
           "code": "VALIDATION_UNIQUENESS",
           "record_type": "ItemType"
         }
       }
     }
   ]
   

[roger]

Hey @ldc,

I’ve been trying to replicate this, but haven’t been able to locally. Can I ask, please:

• Which version of @datocms-cli you’re using? (I tried on 1.3.3 and it seems to work)
• Was this the first time you generated a staging-copy and ran the migration generator? Or have you done this previously (and it worked)?
• Is there more than one migration script in your repo? (./migrations/*.js)

Also, to help us troubleshoot, would you mind posting (or emailing support@datocms.com) with your project ID/URL so we can take a look and see if it’s something specific to your schema? With your permission, I’d like to fork your staging environment into a copy and a fork of that (like datocms-support-staging-copy and datocms-support-staging-copy-forked-for-migration) and try to do a migration on that, to see if I can locate the issue more specifically.

[jay1]

i’m experiencing the same issue as the OP, i’ve got way more changes in my generated migration script but it fails on the same error:

Error: POST https://site-api.datocms.com/item-types?skip_menu_item_creation=true: 422 Unprocessable Entity

[
  {
    "id": "c97148",
    "type": "api_error",
    "attributes": {
      "code": "INVALID_FIELD",
      "details": {
        "field": "name",
        "code": "VALIDATION_UNIQUENESS",
        "record_type": "ItemType"
      }
    }
  },
  {
    "id": "ce9dcb",
    "type": "api_error",
    "attributes": {
      "code": "INVALID_FIELD",
      "details": {
        "field": "api_key",
        "code": "VALIDATION_UNIQUENESS",
        "record_type": "ItemType"
      }
    }
  }
]

[roger]

EDIT: Sorry, Jay, just saw your email! Will look into it now and report back as soon as I can

Please disregard the original post below:

Hi @jay1, and welcome! Same questions to you, please, as the ones in my previous post: Bug: Autogenerating and running migrations tries to generate schema_migration model twice - #2 by roger

Any specific information you can provide would help track this down. Sorry, I haven’t been able to replicate the issue locally with my tests yet.

At the least, could you please send us your project ID/URL (either here or via email) so we can take a look and see if there’s something in your particular schema that’s causing this issue?

[ldc]

@roger

Hi roger, thanks for your reply!

• I’m using the cli through npx:

❯ npx datocms --version
@datocms/cli/1.2.3 darwin-arm64 node-v18.18.2

• Yes this was the first migration for staging-copy (we didn’t have a schema_migration model yet).

Unfortunately, we’ve now applied the migration on our staging environment already, by removing the generated code that creates the schema_migration model.

So now, we can no longer reproduce this issue and we expect follow-up migrations to work.

• This was the only migration script at the time in the ./migrations folder.

Do you think it’s still useful if we share the project ID/URL?
After editing the generated code and applying the migration, our staging environment looks as expected.
I did want to report this as it looks like a bug.

[roger]

@ldc

Do you think it’s still useful if we share the project ID/URL?

Do you mind doing so, please? If it’s OK with you, I’d like to clone your environment to new copies (so we don’t mess up your existing production/staging envs) and try to replicate this bug.

I’ve also requested the same of jay1. Hopefully, between the two, we can track down the actual bug here?

I did want to report this as it looks like a bug.

It sure sounds like one! But I’m having trouble replicating it with our example projects, so seeing the exact situation (i.e. your affected environments) could help us isolate the bug. Sorry for the hassle here!

If it’s too much trouble, I understand… since you already fixed the issue on your end, please don’t feel like you have to jump through hoops to help us track it down. But if you do have time, it might help us catch and fix the bug once and for all, both for other users and for the next time you want to do a new migration

[roger]

@jay1 very helpfully provided us with some detailed specifics over email (thank you so much!), and so far it seems like one way this can fail is if:

• You fork an environment, like from primary to staging
• In staging, you delete an existing model, like post, and then re-create the model again with different fields, but keep the same name post
• The migration script then (erroneously) tries to create a model of post when run, causing failures both in the model and menu item creation

I’ve reported this issue to the devs and will update here as soon as we get more info!

@ldc Could that also be what you experienced, or do you think it was a different issue?

[ldc]

Not sure if you’ll find anything on our environment, since we have the schema_migration model now generated on our staging environment.

If I find time, I’ll see if we can spin up a new sandbox project to try and reproduce.

Thanks for the follow-up!

[ldc]

We’ve further stumbled upon some issues, and while I can’t fully articulate them, I can hopefully give some pointers.

Somehow, we’ve ended up with a different recordID for the schema_migration models on staging and uat.

This is probably what’s causing the migration:new script to want to re-create the schema_migration model, even when it already exists.

I think we mixed updating our records by clicking in the web interface with creating them with migrations, and this is the root of what’s causing us major issues in using the migration tooling.

[roger]

@ldc

This is probably what’s causing the migration:new script to want to re-create the schema_migration model, even when it already exists.

Thank you for sharing this! It sounds similar to jay1’s situation, where conflicting (human/UI) edits of the same model names causes the scripting to break.

We’ve filed a bug report about it internally, and will report back as soon as we have more information!

[m.finamor]

Hello @ldc

We just shipped a fix for this in the CLI v1.3.4

Let us know if it works after the update