Create a Dynamic Sitemap in Next.js

photo by Stephen Phillips on Unsplash

It might be that your site doesn't show on google search, or just some of its pages. It doesn't mean it is not indexed by google, but some pages might be missing on it. In that case, consider adding a sitemap to help Google discover all your website pages. Let's see how we can do it!

What is a sitemap

A sitemap is a file where you specify your site pages, media, files, their relations and additional information about it. You can specify their URLs, last modified date, change frequency and priority for example. Search engines search and retrieve information from this file to crawl your site and better index it. You can check more on sitemaps.org

Sample XML sitemap file

The sitemap protocol format consists of XML tags. All data values in a Sitemap must be entity-escaped. The file itself must be UTF-8 encoded.

It must:

• begin with an opening <urlset> tag
• end with a closing </urlset> tag.
• Specify the namespace (protocol standard) within the <urlset> tag.
• include a <url> entry for each URL, as a parent XML tag.
• include a <loc> child entry for each <url> parent tag.

All other tags are optional.

1
<?xml version="1.0" encoding="UTF-8"?>
2
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
3
   <url>
4
      <loc>http://www.nunorralves.com/</loc>
5
      <lastmod>2020-10-24</lastmod>
6
      <changefreq>weekly</changefreq>
7
      <priority>0.8</priority>
8
   </url>
9
</urlset> 
10

All URLs in a Sitemap must be from a single host. You can check protocol here

What is robot.txt file

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests.

Sample robot.txt file

Create a static file at public/robots.txt. It will define which files can be crawled and where the sitemap is located.

1
User-agent: *
2
Sitemap: https://www.nunorralves.pt/sitemap.xml
3

Simple as that!

Static vs Dynamic sitemap file

If your site is not frequently updated, you might consider defining and updating this file statically. On other hand, if you update it frequently or you have dozens or even hundreds of pages, you might / should consider doing it dynamically.

Let's implement a dynamic sitemap.xml file generator

First, let's install a very useful package called globby. It will allow us to do user-friendly glob matching.

yarn add globby
# or
npm install globby

Now, let's create our node script for generating sitemap.xml file. Create on root of your project, for example, a folder scripts and a file under it called generate-sitemap.js. Add the following:

1
/* eslint-disable @typescript-eslint/no-var-requires */
2
const fs = require('fs');
3
4
const globby = require('globby');
5
const prettier = require('prettier');
6
7
(async () => {
8
  const prettierConfig = await prettier.resolveConfig(
9
    '../../.prettier.config.js'
10
  );
11
  const pages = await globby([
12
    'src/pages/**/*{.js,.jsx,.ts,.tsx,.mdx}',
13
    '!src/pages/_*{.js,.jsx,.ts,.tsx,.mdx}',
14
    '!src/pages/api'
15
  ]);
16
  const sitemap = `
17
        <?xml version="1.0" encoding="UTF-8"?>
18
        <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
19
            ${pages
20
              .map(page => {
21
                const path = page
22
                  .replace('src/pages', '')
23
                  .replace('.jsx', '')
24
                  .replace('.tsx', '')
25
                  .replace('.mdx', '')
26
                  .replace('.js', '')
27
                  .replace('.ts', '');
28
                const route = path === '/index' ? '' : path;
29
                const changeFreq = '<changefreq>weekly</changefreq>';
30
                const priority = '<priority>0.8</priority>';
31
                return `
32
                        <url>
33
                            <loc>${`https://www.nunorralves.pt${route}`}</loc>
34
                            ${
35
                              path === '/index' || path === '/blog'
36
                                ? changeFreq
37
                                : ''
38
                            }
39
                            ${path === '/index' ? priority : ''}
40
                        </url>
41
                    `;
42
              })
43
              .join('')}
44
        </urlset>
45
    `;
46
47
  const formatted = prettier.format(sitemap, {
48
    ...prettierConfig,
49
    parser: 'html'
50
  });
51
52
  fs.writeFileSync('public/sitemap.xml', formatted);
53
})();
54

As you can read from above, we use globby to define the array of pages path. Then we start generating sitemap.xml according to the template described and on it, iterate over pages, creating respective URL tags. In the end, just format it according to your prettier configurations file and write the generated file to the filesystem under public folder.

To complete the implementation, go to your next.config.js file and extend webpack definition to run the created script at start:

1
module.exports = {
2
  (...)
3
  webpack: (config, { isServer }) => {
4
    if (isServer) {
5
      require('./scripts/generate-sitemap');
6
    }
7
8
    return config;
9
  }
10
  (...)
11
};
12

Nothing else todo!

Test it!

Just start you server. If you follow this tutorial you should see a new file under public/sitemap.xml with something like below:

1
<?xml version="1.0" encoding="UTF-8"?>
2
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
3
  <url>
4
    <loc>https://www.nunorralves.pt/about</loc>
5
  </url>
6
7
  <url>
8
    <loc>https://www.nunorralves.pt/blog</loc>
9
    <changefreq>weekly</changefreq>
10
  </url>
11
12
  <url>
13
    <loc>https://www.nunorralves.pt</loc>
14
    <changefreq>weekly</changefreq>
15
    <priority>0.8</priority>
16
  </url>
17
18
  <url>
19
    <loc>https://www.nunorralves.pt/blog/create-dynamic-sitemap-nextjs</loc>
20
  </url>
21
22
  <url>
23
    <loc
24
      >https://www.nunorralves.pt/blog/create-simple-blog-nextjs-markdown</loc
25
    >
26
  </url>
27
</urlset>
28

And that's it! If you like this post, click Like button below. You can also share your comments or suggestions with me.