Integration with Algolia

✋ Feature Requests
,
1

It would be cool if we could provide some sort of automatic sync between DatoCMS and Algolia, or at least provide some documentation on how to do that with custom code/webhooks.

2

Big :+1: for this one!

3

Agreed! Documentation would be a great start.

4

:+1:

5

up :+1:

6

yes, Please

7

:eyes:

1 Like
8

Not sure if this can help, below there’s a simple postbuild script I use when developing with NextJS using this Algolia API. It should work in any NodeJS environment so not necessarily only with React projects.

Whenever the build webhook is triggered by Dato, after the website is built, the script runs and updates the index.

It updates only Algolia’s objects which have actually been updated and published on Dato by comparing _updatedAt field value. If the object doesn’t exist, it will create a new one in the index.

It also deletes any object from Algolia if the record has been unpublished or deleted on Dato.

Unfortunately, the API used is not able to delete any field from existent objects, whenever they are deleted from the content model.
In such case this API should do the job but it replaces the entire index everytime it runs.

Alternatively, you can clear the index using Algolia’s dashboard and run again the build/script.

package.json

  ...
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "postbuild": "node ./scripts/indexing.js", // <-- Add the script
    "start": "next start",
    "lint": "next lint"
  },
  ...

scripts/indexing.js

require('dotenv').config({
  path: './.env.local',
});

const { GraphQLClient } = require('graphql-request');
const algoliasearch = require('algoliasearch/lite');
const intersection = require('lodash.intersection');
const without = require('lodash.without');

const getMs = (dateString) => new Date(dateString).getTime();

(async () => {
  let haltExecution;

  // Init Algolia client
  const algoliaClient = algoliasearch(
    process.env.NEXT_PUBLIC_ALGOLIA_APP_ID,
    process.env.ALGOLIA_SEARCH_ADMIN_KEY
  );
  const INDEX_NAME = 'Tours';
  const algoliaIndex = algoliaClient.initIndex(INDEX_NAME);

  // Init GraphQL client
  const client = new GraphQLClient('https://graphql.datocms.com/', {
    headers: {
      authorization: `Bearer ${process.env.DATOCMS_API_TOKEN}`,
    },
  });

  const QUERY = `
      {
        allTours(filter: { _status: { eq: published } }) {
          _updatedAt
          objectID: id
          title
          subtitle
        }
      }
    `;

  // Fetch records
  const data = await client.request(QUERY).catch((err) => {
    haltExecution = true;
    console.error('Failed to fetch CMS data.', '\n', err.stack);
  });
  if (haltExecution || !data) return;

  // Transform records to Algolia objects
  const recordsToAlgoliaObjects = data.allTours.map(
    ({ _updatedAt, ...rest }) => ({
      updatedAt: {
        _operation: 'IncrementSet',
        value: getMs(_updatedAt),
      },
      ...rest,
    })
  );

  // Save/update objects to Algolia index
  const saveAlgoliaObjects = await algoliaIndex
    .partialUpdateObjects(recordsToAlgoliaObjects, {
      createIfNotExists: true,
    })
    .catch(() => {
      haltExecution = true;
      console.error('Failed to add/update objects to Algolia index.');
    });
  if (haltExecution || !saveAlgoliaObjects) return;
  console.log(
    `Sucessfully added/updated ${saveAlgoliaObjects.objectIDs.length} objects to Algolia index.`
  );

  // Delete draft/unpublished/deleted records from Algolia index
  // 1. Create a new array of all objectIDs from Algolia index
  let algoliaObjectList = [];
  await algoliaIndex
    .browseObjects({
      query: '',
      attributesToRetrieve: ['objectID'],
      batch: (objectID) =>
        (algoliaObjectList = algoliaObjectList.concat(objectID)),
    })
    .catch(() => {
      haltExecution = true;
      console.error('Failed to retrieve Algolia objects.');
    });
  if (haltExecution) return;
  const algoliaObjectsArray = algoliaObjectList.map(({ objectID }) => objectID);

  // 2. Create a new array of all published records IDs
  const recordsArray = data.allTours.map(({ objectID }) => objectID);

  // 3. Create a new array of all published records already indexed and updated on Algolia
  const publishedRecords = intersection(algoliaObjectsArray, recordsArray);

  // 4. Exclude all updated/published records from Algolia index objects
  const objectsToDelete = without(algoliaObjectsArray, ...publishedRecords);

  // 5. If there are any unpublished/deleted records, delete them from Algolia index
  if (objectsToDelete.length > 0) {
    const deleteObsoleteRecords = await algoliaIndex
      .deleteObjects(objectsToDelete)
      .catch(() => {
        haltExecution = true;
        console.error('Failed to delete Algolia objects.');
      });
    if (haltExecution || !deleteObsoleteRecords) return;

    console.log(
      `Deleted ${deleteObsoleteRecords.objectIDs.length} obsolete records from Algolia index.`
    );
  } else console.log('No obsolete records found.');
})();

You can test the indexing in the terminal with:

node ./scripts/indexing.js.

Hope it helps!

1 Like
9

We have written a blog post on this topic: Algolia Next.js - How to add Algolia to your Next.js project

which hopefully will give a good example on this. I’m closing the feature request, feel free to open a new question if there are any doubts or missing parts.

1 Like