Search spider ignoring noindex data flag

I was experiencing some odd content in the highlights so I added the data-datocms-noindex flag to the header and footer and re-spidered but it appears to be ignoring it?

Alex
I had similar issue recently. I strongly recommend you validate your html, specifically look for incorrectly nested tags (divs, lis etc). Any incorrectly nested tags will prevent site being spidered correctly
Dave

1 Like

Thanks I’ll take a look @cleanwell-ni - I’m using HAML so it’s quite difficult to not close elements but it’s not beyond the realm of possibility!

So the validation didn’t show any issues with incorrectly nested elements. However it did raise an interesting issue. It raised an error saying that data attributes are not allowed on header or footer elements.

But given ignoring headers and footers are two scenarios specifically given in the Dato documentation for site search I’m assuming the spider is not ignoring these @matjack1 ?

we’ll double check this as soon as possible @alex1 thank you for raising it!

hello @alex1, sorry but I cannot reproduce. I’ve tried myself and the spider is correctly ignoring pieces of content with the data attribute. Are you sure that the information is not also present outside of the ignored tag?

If so, can you please share more information about your project? Maybe also via private messages here or on the support form: https://www.datocms.com/support/

I have the same issue. I have placed the proper data attribute tag on the HTML and the data I dont want to be indexed is being searched.

Hey @nirodriguez and welcome to the community!

Can you email us the page that is being indexed as well as your project URL to support@datocms.com so we can take a look at what is happening? Thank you