Bug in autogeneration of datocms migrations

I found a bug in the autogeneration process of migrations.

Reproduction path

This reproduction path assumes youā€™ve just created a new blank datocms instance without having made any changes to the primary env. It also assumes you have the datocms cli installed and configured. This is what my datocms.donfig.json looks like:

{
  "profiles": {
    "default": {
      "logLevel": "BODY",
      "migrations": {
        "directory": "./datocms/migrations",
        "modelApiKey": "schema_migration"
      }
    }
  }
}

Autogenerating migrations fails when the primary env is empty
(due to not being able to diff, might also be a bug?).
So create a ā€˜Schema migrationā€™ (schema_migration) model in the primary env.
Add the following single-line string field to the model ā€œMigration file nameā€ (name) and make the field required under the ā€˜validationsā€™ tab.

1. Create a fork of the primary env called ā€˜featureā€™.

$ npx datocms environments:fork main feature --json --api-token=YOUR_DATO_API_TOKEN

2. Make changes to the ā€˜featureā€™ env in datocms using the website UI.
a. Go to datocms and switch to the ā€˜featureā€™ env .
b. Create a new model called modelA (settings > models > add new model > enter modelA as the name > save model).
c. Create another new model called modelB (settings > models > add new model > enter modelB as the name > save model).
d. You should now have a content tab in the top nav bar and when clicked you should see the following 4 menu items: Schema migration, modelA, modelB, Settings (in that particular order).

3. Generate a migration file for the changes you just made in the ā€˜featureā€™ env.

$ npx datocms migrations:new myMigration --autogenerate=feature:main --json --api-token=YOUR_DATO_API_TOKEN

4. Create a second fork of the primary env called ā€˜feature-testā€™.

$ npx datocms environments:fork main feature-test --json --api-token=YOUR_DATO_API_TOKEN

5. Run the generated migration file against on the ā€˜feature-testā€™ env.

$ npx datocms migrations:run --source=feature-test --in-place --json --api-token=YOUR_DATO_API_TOKEN

This works fine and does exactly what it needs to

Moving on to the bugā€¦

6. Delete the ā€˜featureā€™ env.

$ npx datocms environments:destroy feature --json --api-token=YOUR_DATO_API_TOKEN

7. Delete the ā€˜feature-testā€™ env.

$ npx datocms environments:destroy feature-test --json --api-token=YOUR_DATO_API_TOKEN

8. Repeat the process, but this time with an extra change.
Repeat steps 1-5, but add steps 2e and 2f before step 3.
2e. Change the order of the menu items in the ā€˜contentā€™ tab. Change the order from Schema migration, modelA, modelB, Settings to Schema migration, modelB, modelA, Settings (essentially switching modelA and modelB in the order).
2f. Delete the generated migration file (manually in the IDE) THIS IS VERY IMPORTANT

When you have completed step 5 againā€¦
You should see be able to verify in your cli that the migration failed:

{
  "error": {
    "message": "Migration \"1673011341_myMigration.js\" failed",
    "oclif": {
      "exit": 2
    }
  }
}

Other notes

After some digging I found out that the reason the migration fails is because of this line (note: the ID ā€œ569389ā€ will be different for you):

await client.menuItems.update("569389", { position: 3 });

There are no records/menuItems with ID ā€œ569389ā€ making the migration fail in the process. (It seems like menuItem IDs are outdated if you ever generate a new migration file after already having created one once in the same dato env).

This is the simplest reproduction path I could come up with, but you can also get this bug without intentionally changing the menu item order; when you have multiple models, deleting one of the models can cause a ā€˜shiftā€™ in menu items also causing this bug to occur.

1 Like

Hello @wrsmit00_datocms and welcome to the community,

It is a bug indeed, thank you so much for the detailed report!

Iā€™ve contacted the development team and iā€™ll get back to you as soon as possible with a fix.

Thanks @wrsmit00_datocms, please upgrade to v >= 1.0.22, the new migration generated by the CLI should become:

await client.menuItems.update(newMenuItems['569389'], { position: 3 });

fixing the issue. Again, thanks for the report!

I have not been able to verify this as of yet because everytime I try to apply my autogenerated migration I get the following error (which I have never encountered).

The error message to me isnā€™t clear and the migration is very simple and I cannot image that is where it goes wrong.

Any ideas whether this is something on my end or your end @s.verna @m.finamor ?

Below some extra info:

@datocms/cli version = ā€œ^1.0.22ā€

This is the CLI output

wesselsmit@Wessels-MBP-2 project-setup % npx datocms migrations:run --source=wessel-test --in-place --api-token=DATOCMS_API_TOKEN

Create new models/block models
Create model "model a" (`model_a`)
Manage menu items
Create menu item "model a"
{
  "error": {
    "message": "POST https://site-api.datocms.com/items: 422 Unprocessable Entity\n\n[\n  {\n    \"id\": \"8d145c\",\n    \"type\": \"api_error\",\n    \"attributes\": {\n      \"code\": \"INVALID_ATTRIBUTES\",\n      \"details\": {\n        \"extraneous_attributes\": [\n          \"name\"\n        ]\n      }\n    }\n  }\n]",
    "request": {
      "url": "https://site-api.datocms.com/items",
      "method": "POST",
      "headers": {
        "content-type": "application/json",
        "accept": "application/json",
        "authorization": "Bearer DATOCMS_API_TOKEN",
        "user-agent": "@datocms/cma-client v1.2.3",
        "x-environment": "wessel-test",
        "x-api-version": "3"
      },
      "body": {
        "data": {
          "type": "item",
          "attributes": {
            "name": "1675956455_addmodela.js"
          },
          "relationships": {
            "item_type": {
              "data": {
                "id": "1322933",
                "type": "item_type"
              }
            }
          }
        }
      }
    },
    "response": {
      "status": 422,
      "statusText": "Unprocessable Entity",
      "headers": {
        "access-control-allow-credentials": "true",
        "access-control-allow-headers": "authorization, content-type, x-environment, x-organization, x-site-domain, x-api-version, user-agent, x-session-id, x-include-drafts, x-exclude-invalid",
        "access-control-allow-methods": "GET, POST, PUT, OPTIONS, DELETE",
        "access-control-allow-origin": "*",
        "access-control-expose-headers": "x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset",
        "access-control-max-age": "1728000",
        "cache-control": "no-cache",
        "cf-cache-status": "DYNAMIC",
        "cf-ray": "796da16bbd690a5c-AMS",
        "connection": "keep-alive",
        "content-encoding": "gzip",
        "content-type": "application/json; charset=utf-8",
        "date": "Thu, 09 Feb 2023 15:27:41 GMT",
        "referrer-policy": "strict-origin-when-cross-origin",
        "server": "cloudflare",
        "strict-transport-security": "max-age=15552000; includeSubDomains; preload",
        "transfer-encoding": "chunked",
        "vary": "Accept,Accept-Encoding",
        "via": "1.1 vegur",
        "x-api-version": "3",
        "x-content-type-options": "nosniff",
        "x-download-options": "noopen",
        "x-environment": "wessel-test",
        "x-frame-options": "SAMEORIGIN",
        "x-permitted-cross-domain-policies": "none",
        "x-queue-time": "6ms",
        "x-ratelimit-limit": "60",
        "x-ratelimit-remaining": "52",
        "x-request-id": "7d8d92d8-1bb8-4077-b57c-70c038415ecd",
        "x-runtime": "0.032423",
        "x-xss-protection": "1; mode=block"
      },
      "body": {
        "data": [
          {
            "id": "8d145c",
            "type": "api_error",
            "attributes": {
              "code": "INVALID_ATTRIBUTES",
              "details": {
                "extraneous_attributes": [
                  "name"
                ]
              }
            }
          }
        ]
      }
    },
    "preCallStack": "Error\n    at Client.request (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@datocms/cma-client/dist/cjs/generated/Client.js:130:225)\n    at Item.rawCreate (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@datocms/cma-client/dist/cjs/generated/resources/Item.js:298:28)\n    at Item.create (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@datocms/cma-client/dist/cjs/generated/resources/Item.js:281:21)\n    at Command.runMigrationScript (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@datocms/cli/lib/commands/migrations/run.js:133:35)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Command.run (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@datocms/cli/lib/commands/migrations/run.js:84:13)\n    at async Command._run (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@oclif/core/lib/command.js:80:22)\n    at async Config.runCommand (/Users/wesselsmit/Sideprojects/project-setup/node_modules/@oclif/core/lib/config/config.js:298:25)"
  }
}

[ERROR] Error: Command failed: npx datocms migrations:run --source=wessel-test --in-place --api-token=DATOCMS_API_TOKEN --json
(node:17850) ExperimentalWarning: stream/web is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

wesselsmit@Wessels-MBP-2 project-setup % 

This is the migration (itā€™s the first and only one in this codebase):

'use strict'

/** @param client { import("@datocms/cli/lib/cma-client-node").Client } */
module.exports = async function (client) {
  const newItemTypes = {}
  const newMenuItems = {}

  console.log('Create new models/block models')

  console.log('Create model "model a" (`model_a`)')
  newItemTypes['1413994'] = await client.itemTypes.create(
    {
      name: 'model a',
      api_key: 'model_a',
      all_locales_required: true,
      collection_appearance: 'table',
      inverse_relationships_enabled: false,
    },
    { skip_menu_item_creation: 'true' }
  )

  console.log('Manage menu items')

  console.log('Create menu item "model a"')
  newMenuItems['726529'] = await client.menuItems.create({
    label: 'model a',
    item_type: newItemTypes['1413994'],
  })
}

The issue is that your environment has a schema_migration model created by hand by you, with a field called migration_filename instead of name. Usually itā€™s the CLI itself that will create the schema_migration model with the correct fields, so I suggest you to simply remove the model, and re-run the command.

1 Like