How to get your Astro site indexed in 2024

Indexing your Astro site is super important. If it isn’t indexed, it won’t show up in search results. This means that you won’t get any visitors from Google, Bing, Yandex, Seznam.cz and Naver, and that obviously sucks. Especially since you’re probably using Astro for its SEO benefits. In this article, we’ll show you how to get your Astro site indexed on all these search engines so you can actually get some organic traffic to your site for once.

What is indexing?

Indexing is the process through which search engines crawl, analyse, and store information about your site. A well-indexed site is more likely to appear in relevant search results, increasing its visibility to potential visitors and thus your organic traffic. Sorry for the repetition, I just wanted to make sure you understand how important getting your site indexed is.

How do I get my Astro site indexed?

Now that you have a basic understanding of indexing and why you should care, let’s take a look at how you can get your Astro site indexed as soon as possible, so you can grow your site organically (aka for free).

1. Create an XML sitemap

The first step is to create an XML sitemap. An XML sitemap is a file that contains a list of all the pages on your site in a specific format that search engines understand. It’s used by search engines to find and index your pages automatically. Luckily for us, Astro has an integration for this called @astrojs/sitemap.

First of all, you should install it by running one of the following commands in your terminal:

# Using NPM
npx astro add sitemap
# Using Yarn
yarn astro add sitemap
# Using PNPM
pnpm astro add sitemap

Then, configure your astro.config.* file to include the integration as well as your site’s URL (otherwise the integration won’t work):

// astro.config.mjs
import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";

export default defineConfig({
  // ...
  site: "https://indexplease.com",
  integrations: [sitemap()],
  // ...
});

Now, once you build your site for production (via the astro build command), you will find both a sitemap-index.xml and sitemap-0.xml in the output folder. The sitemap-index.xml file contains a list of all your other sitemaps, so if you submit this to a search engine, it will automatically find all the other sitemaps and crawl them.

Finally, you need to add the following lines to your robots.txt file, as well as your site’s <head>, so search engines know where to find your sitemap:

// src/layouts/Layout.astro
<head>
  <link rel="sitemap" href="/sitemap-index.xml" />
</head>

// public/robots.txt
User-agent: *
Allow: /

Sitemap: https://indexplease.com/sitemap-index.xml

2. Submit your sitemap

Once you’ve created your XML sitemap, you need to submit it to each search engine. Depending on the search engine, this is done in different ways. Luckily for you, we’ve written a guide on how to do this for each relevant search engine, so check them out:

As mentioned above, you only need to submit your sitemap-index.xml file to each search engine, as they will automatically find all the other sitemaps, such as sitemap-0.xml, and crawl them.

3. Submit your pages

Once you’ve submitted your sitemap, you can either wait for search engines to crawl your pages automatically (this can take weeks or not happen at all) or manually submit them. This is done by going to each of the aforementioned webmaster tools and going through the process of submitting them one by one. As you can imagine, this is a very time-consuming process, especially if you have a large number of pages. And it doesn’t always guarantee that your pages will actually get indexed. If this doesn’t put you off, we’ve also written guides on how to manually submit your pages for indexing for each search engine:

Automatically index your pages

Most of us have better things to do than sitting around all day and going through the tedious process of manually submitting pages for indexing, or waiting ages for search engines to index them. This is one of the reasons why we built IndexPlease, which will help you do this automatically. It will only cost you 9$/month and we will automatically submit up to 400 pages per day (yes, really), for up to 5 of your sites, so they get indexed on Google, Bing, Yandex, Seznam.cz and Naver within 48 hours.

Getting your pages automatically indexed is only one of the many things you can do with IndexPlease, so if you’re interested in what else we have to offer, check out our features page.