Migration - Unexpected end of file undici

Hi,

I have a migration which converts markdown to structured text but the migration fails because undici throws an error because of an unexpected end of file?

I can see in the record the structured text field is populated with data as expected but when it attempts the next record in the pagination iterator it fails.

I tried doing some digging but I am unsure what’s causing this?

When I comment out these lines directly in the node_modules folder it works fine?

VERSIONS:

Node: v16.14.2
 "@datocms/cli": "^1.0.21",
    "@datocms/cma-client": "^1.2.1",
    "@datocms/cma-client-node": "^1.2.1",

MIGRATION:

export default async function (client: Client): Promise<void> {
  // Rename article to Post
  await client.itemTypes.update('article', {
    name: 'Post',
    api_key: 'post',
  });

  // Create fieldset
  const metaFieldset = await client.fieldsets.create('post', {
    title: '📖 General Information',
    hint: 'Information regarding the blog post',
    position: 1,
    collapsible: false,
    start_collapsed: false,
  });

  // Update fields
  await client.fields.update('post::title', {
    validators: {
      required: {},
    },
    position: 1,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
    appearance: {
      editor: 'single_line',
      parameters: { heading: true },
      addons: [],
    },
  });

  await client.fields.update('post::slug', {
    position: 2,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  await client.fields.update('post::meta_seo', {
    label: 'Search Engine Optimisation',
    api_key: 'seo',
    position: 3,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  await client.fields.update('post::author', {
    position: 4,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  await client.fields.update('post::categories', {
    position: 5,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  await client.fields.update('post::tags', {
    position: 6,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  await client.fields.update('post::feature', {
    label: 'Featured Post?',
    api_key: 'featured',
    position: 7,
    fieldset: { type: 'fieldset', id: metaFieldset.id },
  });

  // Create Fieldset
  const contentFieldset = await client.fieldsets.create('post', {
    title: '📖 Content',
    hint: 'Content for the award',
    position: 2,
    collapsible: false,
    start_collapsed: false,
  });

  await client.fields.update('post::featured_image', {
    label: 'Featured Image?',
    position: 1,
    fieldset: { type: 'fieldset', id: contentFieldset.id },
  });

  await client.fields.update('post::excerpt', {
    position: 2,
    fieldset: { type: 'fieldset', id: contentFieldset.id },
  });

  await client.fields.update('post::body', {
    position: 3,
    fieldset: { type: 'fieldset', id: contentFieldset.id },
  });

  // Update structured text
  const modelIds = await getModelIdsByApiKey(client);

  await createStructuredTextFieldFrom(client, 'post', 'body', [modelIds.image_block.id]);

  for await (const record of client.items.listPagedIterator({ filter: { type: 'post' }, nested: 'true' })) {

    console.log(record);
    await client.items.update(record.id, {
      structured_text_body: await markdownToStructuredText(record.body as string,
        // convertImgsToBlocks(client, modelIds)
      ),
    });

    if (record.meta.status !== 'draft') {
      console.log('PUBLISHING')
      await client.items.publish(record.id);
    }
  }

  await swapStructuredTextFields(client, 'post', 'body');
}

ERROR:

Buffer false [Function: Buffer] {
  poolSize: 8192,
  from: [Function: from],
  of: [Function: of],
  alloc: [Function: alloc],
  allocUnsafe: [Function: allocUnsafe],
  allocUnsafeSlow: [Function: allocUnsafeSlow],
  isBuffer: [Function: isBuffer],
  compare: [Function: compare],
  isEncoding: [Function: isEncoding],
  concat: [Function: concat],
  byteLength: [Function: byteLength],
  [Symbol(kIsEncodingSymbol)]: [Function: isEncoding]
}
Error true [Function: Error] {
  stackTraceLimit: 10,
  prepareStackTrace: [Function: prepareStackTrace]
}
Error: unexpected end of file
TERMINATE BECAUSE OF BYTES
Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (node:zlib:190:17) {
  errno: -5,
  code: 'Z_BUF_ERROR'
}
Running migration "0000000006_updateArticle.ts"... failed!

----
TypeError: terminated
    at Fetch.onAborted (/Users/david.hewitt/Code/galleria/node_modules/.pnpm/undici@5.16.0/node_modules/undici/lib/fetch/index.js:1935:49)
    at Fetch.emit (node:events:526:28)
    at Fetch.emit (node:domain:475:12)
    at Fetch.terminate (/Users/david.hewitt/Code/galleria/node_modules/.pnpm/undici@5.16.0/node_modules/undici/lib/fetch/index.js:96:10)
    at Fetch.fetchParams.controller.resume (/Users/david.hewitt/Code/galleria/node_modules/.pnpm/undici@5.16.0/node_modules/undici/lib/fetch/index.js:1894:32)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
----

 ›   Error: Migration "0000000006_updateArticle.ts" failed

I have updated to the latest versions:

"@datocms/cli": "^1.0.22",
    "@datocms/cma-client": "^1.2.3",
    "@datocms/cma-client-node": "^1.2.3",

However I am still getting the same error

I hacked the unichi module and modifed the request to add the following header and it works:

Accept-Encoding="*"

I am guessing you will need to update the package that makes the api request to include this header

Hi @david.hewitt! Is this always reproducible, or it happens randomly?

It would be great if you could send us the complete migration file and the project ID, so that we can try the script ourselves… right now some functions (getModelIdsByApiKey, markdownToStructuredText, etc) are not defined. A repo with everything would be ideal. Feel free to send us everything via support@datocms.com you prefer it that way.

Also, can you try to force in your package.json the latest version of undici, and see if it solves the issue?

"devDependencies": {
    "undici": "5.18.0"
  }

Thanks!

1 Like

Hey @s.verna ,

Yep I can confirm that it is always reproducible and fails at the same hurdle.

I can send you the full migration file but I am certain that the migration code is fine and has worked in the past it’s only once I updated the packages to the latest it started failing.

the getModelIdsByApiKey & markdownToStructuredText functions are just the datocms examples defined here:

My fix seems to work is there anything you object to when it comes to adding the accept-encoding header to the cli package?

Nothing to object @david.hewitt, but before making changes to our CLI/API Client we need to be able to reproduce the issue and understand the reason, to make sure that it won’t have any other implications for other users.

I’m also sure that the problem is not the migration itself… we just need something that reproduces the issue! :smile: Launching random migrations to random environments unfortunately works on my end, and with your exact configuration (Node 16, undici@5.16.0, @datocms/cli@1.0.22, @datocms/cma-client-node@1.2.3).

Btw, undici should already add an ‘accept-encoding’ header if it’s missing:

@s.verna
Sorry I thought it might have been our VPN but turns out I forgot I still had my janky solution in place…

Makes sense! I’ll put together a reproduction on Monday and send it over.

Seems weird that ‘*’ works but ‘br, gzip, deflate’ doesn’t is there some other form of encoding on the server that’s not in this list?