SEO Spider
How to Crawl with ChatGPT
Introduction To Crawling with ChatGPT
Using the Custom JavaScript feature in the SEO Spider, it’s possible to communicate with APIs such as OpenAI’s ChatGPT during a crawl.
Having AI on hand while crawling opens up a world of possibilities. It enables you to use prompts against elements of a page while crawling.
You could use OpenAI’s various models for all kinds of purposes, such as:
- Generating alt text of images.
- Language, sentiment or intent analysis of page content.
- Scraping specific data.
- Extracting embeddings from page content.
And much more!
You don’t need to know how to write JavaScript to use this feature, it’s possible to use our ChatGPT templates and adjust following the guidance supplied in the comments in each snippet.
This tutorial walks you through how to use our various ChatGPT snippets.
Update: Since launching custom JavaScript, direct AI integration has been introduced to connect to OpenAI, Gemini and Ollama APIs and set up custom prompts against crawl data for greater ease. While custom JavaScript offers more flexibility for advanced users, for many use cases the direct integration is more suitable.
How To Use Preset ChatGPT Snippets
There are various preset ChatGPT JavaScript snippets available within the SEO Spider, that only require an OpenAI API key. Here’s how to use them.
1) Open the Custom JavaScript Config
Navigate to ‘Config > Custom > Custom JavaScript’ and click ‘Add from Library’ in the bottom right.
2) Select a ‘(ChatGPT)…’ Snippet
The ‘System’ tab is a library of preset JavaScript snippets that can be used. The various ChatGPT snippets are appended with ‘(ChatGPT)’.
Any of these snippets can be selected and an OpenAI API key added to use. The snippets highlighted below are the preset snippets that only require an API key and no further adjustment.
The ‘(ChatGPT) Template‘ snippet is a little different and we cover this later in the tutorial. For this example, the ‘(ChatGPT) Generate alt text for images’ snippet has been selected.
Double click the snippet to select it and it will display in the Custom JavaScript config.
3) Use the JavaScript Snippet Editor
Click the the ‘JS’ box to the right-hand side of the snippet row to edit the snippet.
This will launch the JS snippet editor config.
4) Add Your OpenAI API Key
To use any ChatGPT snippet, you’ll need your own OpenAI API key. This is different to a ChatGPT subscription, and is easy to set up and cheap to use.
The easiest way to use it is to sign up to an OpenAI account, select your use as ‘API’, and then head over to the billing section. Here you can select to add a fixed amount of credit to the account, such as $100 as well as adjust spending limits etc. Please carefully review their API pricing to better understand the cost of tokens.
When you have an API key, replace the ‘your_api_key_here’ text on line 25 of the snippet with your API key.
Remember to keep the apostrophes at either end, it should look something like this.
Then click ‘OK’ to close the JS snippet editor, and ‘OK’ again to close the custom JavaScript config.
5) Enable JavaScript Rendering
To run custom JavaScript snippets, JavaScript rendering mode must be enabled via ‘Config > Spider > Rendering’.
Ensure the rendering is set to ‘JavaScript’, so pages will be rendered in the background in headless Chrome.
6) Crawl the Site
Input the website you want to crawl with the ChatGPT snippet and hit ‘Start’.
Alternatively upload a list of URLs using list mode.
7) View the Custom JavaScript Tab
View the results of the ChatGPT JavaScript snippet in real-time in the Custom JavaScript tab and relevant filter.
In this example, images are returned with the generated alt text for each using the gpt-4-vision-preview model in OpenAI.
For other preset ChatGPT snippets, other content types will be returned and use different models where relevant.
How To Create A Custom ChatGPT JS Snippet
It’s possible to create your own custom ChatGPT JavaScript snippet by using the ‘(ChatGPT) Template’ snippet and adjusting it. Lets take a look.
1) Select the ‘(ChatGPT) Template’ Snippet
Navigate to ‘Config > Custom > Custom JavaScript’, click ‘Add from Library’ and select the ‘(ChatGPT) Template’ Snippet’.
Follow the instructions already outlined above to open the JS editor and add your OpenAI API key following instructions in the comments of the snippet.
2) Adjust Your Prompt
This is where the magic happens. On line 26, you can replace ‘What do you think of the following text?:’ with your own custom prompt.
For example, ‘What language is this text?’, or ‘Generate a meta description for the following text that is under 155 characters in length, includes any USPs and a call to action at the end’.
On line 27, you can adjust the content that is used for the prompt. The default is the body text, but this could be updated to the page title, meta description, or heading as examples.
The comments from line 29 to 36 provide examples for these and we’ve included more below.
HTML:
[document.documentElement.outerHTML];
Body Text:
[document.body.innerText];
Page Title:
[document.title];
Meta Description
[document.querySelector('meta[name="description"]')?.getAttribute('content')];
h1 Heading (replace with h2 etc as required)
[...document.querySelectorAll('h1')].map(h => h.textContent);
Insert a URL on the right-hand side of the JS tester and click ‘Test’ to ensure it’s working as expected.
At the bottom of the extractor you can adjust the content type the snippet is run against. The default for the ChatGPT template is HTML.
3) Add Snippet to Your Library
If you’re happy with your snippet and want to save it for use in the future, click ‘Add Snippet to User Library’.
Supply an appropriate name and description and click ‘OK’ to accept, then ‘OK and ‘OK’ again to exit the config dialogs.
4) View the Custom JS Tab
Don’t forget to enable JavaScript rendering, then sit back and watch the results appear in the Custom JavaScript tab and relevant filter.
If you’re familiar with using ChatGPT already, it might take a number of refinements to the prompt to get the data you want.
This example is used for inspiration, and isn’t something we’d recommend using for meta descriptions without review, and editing.
Additional Tips!
There are various things to think about when using custom JavaScript snippets and ChatGPT that you’re likely to need to configure further. These include:
- Models – In each of the ChatGPT snippets, there is a model:. In general ‘gpt-4o’ is used across the HTML specific snippets, but it can be updated in the snippet to use another model, such as ‘gpt-4o-mini’. Check out the full list of OpenAI models.
- Speed – There are various rate limits for different OpenAI models and usage tiers. The speed of requests can be controlled alongside crawl speed in the SEO Spider via ‘Config > Speed’. However, snippets could also be expanded by users to include throttling as an alternative.
- Content Types – For each snippet created, you’re able to define the content types the snippet is run against. For example, text/html for snippets related to text content on the page, or image/ when using the gpt-4-vision-preview model.
- Documentation – Check out our comprehensive documentation on Custom JavaScript to find out more details on how they work.
If you write any unique and insightful JavaScript snippets, let us know via support and we could include it in our snippet library.
Common Errors
Please read our tutorial on How to Debug Custom JavaScript Snippets.
Common errors you may see with ChatGPT snippets are:
TypeError: Failed to fetch
The error will appear in the JS snippet tester and Custom JavaScript tab and column as follows:
TypeError: Failed to fetch
If you test the snippet in Chrome Console as described in the tutorial above, you will often see the following error:
Refused to connect to 'https://api.openai.com/v1/chat/completions' because it violates the following Content Security Policy directive
This means that that the site has a ‘Content-Security-Policy’ header that disallows the OpenAI domain.
Hence, you cannot run any OpenAI ChatGPT snippet for that URL.
The model does not exist
The error will appear in the JS snippet tester and Custom JavaScript tab and column as follows:
The model `gpt-4o` does not exist or you do not have access to it.
If you’re using one of the default snippets, it will use the `gpt-4o` model – which most definitely exists.
Hence, please ensure that you have a funded OpenAI account with the correct access permissions.
You can view which models you have access to over in the OpenAI playground, under the ‘model’ selection tab in the settings.
