It would be cool if we could provide some sort of automatic sync between DatoCMS and Algolia, or at least provide some documentation on how to do that with custom code/webhooks.
Big for this one!
Agreed! Documentation would be a great start.
up
yes, Please
Not sure if this can help, below there’s a simple postbuild script I use when developing with NextJS using this Algolia API. It should work in any NodeJS environment so not necessarily only with React projects.
Whenever the build webhook is triggered by Dato, after the website is built, the script runs and updates the index.
It updates only Algolia’s objects which have actually been updated and published on Dato by comparing _updatedAt
field value. If the object doesn’t exist, it will create a new one in the index.
It also deletes any object from Algolia if the record has been unpublished or deleted on Dato.
Unfortunately, the API used is not able to delete any field from existent objects, whenever they are deleted from the content model.
In such case this API should do the job but it replaces the entire index everytime it runs.
Alternatively, you can clear the index using Algolia’s dashboard and run again the build/script.
package.json
...
"scripts": {
"dev": "next dev",
"build": "next build",
"postbuild": "node ./scripts/indexing.js", // <-- Add the script
"start": "next start",
"lint": "next lint"
},
...
scripts/indexing.js
require('dotenv').config({
path: './.env.local',
});
const { GraphQLClient } = require('graphql-request');
const algoliasearch = require('algoliasearch/lite');
const intersection = require('lodash.intersection');
const without = require('lodash.without');
const getMs = (dateString) => new Date(dateString).getTime();
(async () => {
let haltExecution;
// Init Algolia client
const algoliaClient = algoliasearch(
process.env.NEXT_PUBLIC_ALGOLIA_APP_ID,
process.env.ALGOLIA_SEARCH_ADMIN_KEY
);
const INDEX_NAME = 'Tours';
const algoliaIndex = algoliaClient.initIndex(INDEX_NAME);
// Init GraphQL client
const client = new GraphQLClient('https://graphql.datocms.com/', {
headers: {
authorization: `Bearer ${process.env.DATOCMS_API_TOKEN}`,
},
});
const QUERY = `
{
allTours(filter: { _status: { eq: published } }) {
_updatedAt
objectID: id
title
subtitle
}
}
`;
// Fetch records
const data = await client.request(QUERY).catch((err) => {
haltExecution = true;
console.error('Failed to fetch CMS data.', '\n', err.stack);
});
if (haltExecution || !data) return;
// Transform records to Algolia objects
const recordsToAlgoliaObjects = data.allTours.map(
({ _updatedAt, ...rest }) => ({
updatedAt: {
_operation: 'IncrementSet',
value: getMs(_updatedAt),
},
...rest,
})
);
// Save/update objects to Algolia index
const saveAlgoliaObjects = await algoliaIndex
.partialUpdateObjects(recordsToAlgoliaObjects, {
createIfNotExists: true,
})
.catch(() => {
haltExecution = true;
console.error('Failed to add/update objects to Algolia index.');
});
if (haltExecution || !saveAlgoliaObjects) return;
console.log(
`Sucessfully added/updated ${saveAlgoliaObjects.objectIDs.length} objects to Algolia index.`
);
// Delete draft/unpublished/deleted records from Algolia index
// 1. Create a new array of all objectIDs from Algolia index
let algoliaObjectList = [];
await algoliaIndex
.browseObjects({
query: '',
attributesToRetrieve: ['objectID'],
batch: (objectID) =>
(algoliaObjectList = algoliaObjectList.concat(objectID)),
})
.catch(() => {
haltExecution = true;
console.error('Failed to retrieve Algolia objects.');
});
if (haltExecution) return;
const algoliaObjectsArray = algoliaObjectList.map(({ objectID }) => objectID);
// 2. Create a new array of all published records IDs
const recordsArray = data.allTours.map(({ objectID }) => objectID);
// 3. Create a new array of all published records already indexed and updated on Algolia
const publishedRecords = intersection(algoliaObjectsArray, recordsArray);
// 4. Exclude all updated/published records from Algolia index objects
const objectsToDelete = without(algoliaObjectsArray, ...publishedRecords);
// 5. If there are any unpublished/deleted records, delete them from Algolia index
if (objectsToDelete.length > 0) {
const deleteObsoleteRecords = await algoliaIndex
.deleteObjects(objectsToDelete)
.catch(() => {
haltExecution = true;
console.error('Failed to delete Algolia objects.');
});
if (haltExecution || !deleteObsoleteRecords) return;
console.log(
`Deleted ${deleteObsoleteRecords.objectIDs.length} obsolete records from Algolia index.`
);
} else console.log('No obsolete records found.');
})();
You can test the indexing in the terminal with:
node ./scripts/indexing.js
.
Hope it helps!
We have written a blog post on this topic: Algolia Next.js - How to add Algolia to your Next.js project
which hopefully will give a good example on this. I’m closing the feature request, feel free to open a new question if there are any doubts or missing parts.