> For the complete documentation index, see [llms.txt](https://help.pixfizz.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.pixfizz.com/ecommerce/pixfizz-ecommerce/shopper-managed-storefront/using-the-website-crawler.md).

# Using the Website Crawler

The Website Crawler is a built-in tool that scans your site for broken links and other issues. It follows every internal link it can find, checks whether each page loads correctly, and reports the results.

This is one of the most useful tools in the admin for keeping your site healthy — especially after removing pages, updating navigation, or making content changes.

***

### Where to Find It

**Main Admin > Website > Crawler**

The direct URL is `yourdomain.com/admin/website_crawls` (replace `yourdomain.com` with your actual domain).

***

### What the Crawler Does

When you start a crawl, the tool visits your site's homepage and follows every link it finds. For each link, it records:

* **The URL** that was visited
* **The response status** — whether the page loaded successfully (200), was not found (404), redirected (301/302), or returned another status
* **The linking page** — the page that contained the link (shown in the rightmost column of the results)

The crawler continues following links from each page it visits until it has checked every reachable internal URL on your site.

***

### How to Run a Crawl

1. Go to **Main Admin > Website > Crawler**
2. Start a new crawl
3. Wait for it to complete — this may take a few minutes depending on the size of your site

Each crawl run is saved, so you can compare results over time.

**Automatic crawling:** You can request to have the crawler run automatically every 24 hours. Contact your Pixfizz support team to enable this. Once active, you will always have a recent crawl to review without needing to remember to run one manually.

***

### Reading the Results

The most important thing to look for is **404 errors**. A 404 means the crawler found a link on your site that points to a page that does not exist.

For each 404 result, the rightmost column tells you **which page contains the broken link**. This is the page you need to edit to fix the problem — either by removing the link, updating it to point to the correct URL, or setting up a redirect.

**Common status codes you may see:**

* **200** — the page loaded successfully. No action needed.
* **301** — the page redirects permanently to another URL. This is normal if you have redirects set up. The destination page should load correctly.
* **302** — the page redirects temporarily. Same as 301 in practice, but check whether this is intentional.
* **404** — the page was not found. This is what you want to fix.

***

### When to Run a Crawl

Run a crawl whenever you make changes that could affect links on your site:

* **After removing a page** — to confirm no remaining links point to the deleted page
* **After updating navigation or footer links** — to verify all menu items still resolve correctly
* **After adding redirects** — to confirm they are working
* **Before launching a new site** — as a final check that everything is wired up correctly
* **After a domain change or migration** — to catch any broken links from the move
* **Periodically** — a monthly crawl is a good habit to catch issues early, or enable automatic daily crawls so you always have fresh results

***

### Fixing 404 Errors

When you find a 404 in the crawl results:

1. **Note the broken URL** — this is the link that does not work
2. **Check the linking page** (rightmost column) — this tells you where the broken link lives
3. **Decide how to fix it:**
   * If the page was intentionally removed, update or remove the link from the source page
   * If the page moved to a new URL, update the link to point to the new location
   * If the page should still exist, investigate why it is returning a 404
4. If the broken URL may be linked from external sites or search engines, ask your Pixfizz support team to set up a redirect

**Tip:** Use the **Search** tool in the **Snippets** tab (Main Admin > Website > Snippets) to search for the broken URL across all your snippets. This helps you find every place the link appears, not just the one the crawler reported.

***

### The Crawler, Your Sitemap, and Product Feeds

Your site's XML sitemap is generated automatically at `yourdomain.com/sitemap.xml`. The crawler and the sitemap are separate tools, but they complement each other:

* The **sitemap** tells search engines which pages exist on your site
* The **crawler** tells you which links on your site are broken

If the crawler finds 404 errors, those broken pages should not appear in your sitemap. Deleted pages will drop out of the sitemap automatically once they no longer resolve.

If you also maintain an **HTML sitemap page** (a page on your site listing all your key pages for visitors), that page is managed manually. The crawler can help you verify that all the links on your HTML sitemap page are still valid.

***

### Product Feeds

In addition to the sitemap, your site also generates product feeds that can be submitted to Google Merchant Center, Facebook, and other shopping platforms. These are generated automatically based on your published products.

Your product feeds are available at:

* **XML feed:** `yourdomain.com/feed/products.xml`
* **JSON feed:** `yourdomain.com/product-feed.json`

These feeds include product names, descriptions, prices, images, availability, and other attributes. If you have set custom fields like `google_category`, `gtin`, or `mpn` on your products, those values will be included in the feed as well.

**Things to keep in mind:**

* Feeds update automatically as you add, update, or remove products — there is nothing to regenerate manually
* If you are submitting feeds to Google Merchant Center or Facebook, use the XML feed URL
* If products appear to be missing from a feed, check that they are published and assigned to a collection
* If you notice any issues with the feed URLs themselves, contact your Pixfizz support team

***

### Things the Crawler Does Not Do

* It does not check external links (links to other websites)
* It does not fix broken links — it only reports them
* It does not affect your live site or what visitors see
* It does not check whether your pages appear in Google — use Google Search Console for that

***

### Quick Reference

| Task                                | Where                                       |
| ----------------------------------- | ------------------------------------------- |
| Run a crawl                         | Main Admin > Website > Crawler              |
| Enable automatic daily crawls       | Contact your Pixfizz support team           |
| Find broken links                   | Look for 404 status in crawl results        |
| Find which page has the broken link | Check the rightmost column in crawl results |
| Search snippets for a URL           | Main Admin > Website > Snippets > Search    |
| View your XML sitemap               | `yourdomain.com/sitemap.xml`                |
| View your product feed (XML)        | `yourdomain.com/feed/products.xml`          |
| Submit sitemap to Google            | Google Search Console                       |
