🗺️ Dynamic Sitemap Generator

Chrome extension — Generates sitemap.xml and robots.txt for any website, including static and dynamic (React, Vue, Svelte, etc.) sites.

📋 Table of Contents

• Features
• Installation
• Usage
• How It Works
• Permissions
• Limitations
• Contributing
• License

✨ Features

📦 Installation

Requirements

• Google Chrome (or Chromium-based browser: Edge, Brave, etc.)
• Extension is loaded unpacked in Developer mode

Steps

1. Clone the repo

   git clone https://github.com/eros1sh/sitemap-generator.git
   cd sitemap-generator
   

2. Load the extension in Chrome

   • Open chrome://extensions/ in the address bar
   • Enable Developer mode (top right)
   • Click Load unpacked
   • Select the sitemap-generator folder

3. The Dynamic Sitemap Generator icon appears in the toolbar.

🚀 Usage

1. Open the site you want to generate a sitemap for in Chrome (e.g. https://example.com).
2. Click the Dynamic Sitemap Generator icon in the toolbar.
3. Click Start Crawl.
4. Wait until the crawl finishes; the progress bar and status message update as it runs.
5. sitemap.xml and robots.txt content appears in the text areas.
6. Use Download Sitemap and Download Robots.txt to save the files.
7. Place the downloaded sitemap.xml in your site root and adjust robots.txt if needed.

🔧 How It Works

┌─────────────────────────────────────────────────────────────────┐
│  Popup (popup.js)                                                │
│  • "Start Crawl" → gets active tab URL                           │
│  • Sends START_CRAWLING to background                            │
│  • Listens for CRAWL_STATUS / ERROR_FOUND / CRAWL_COMPLETE      │
│  • Shows progress, error list, sitemap/robots                    │
└───────────────────────────┬─────────────────────────────────────┘
                            │ chrome.runtime.sendMessage
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Service Worker (background.js)                                  │
│  • Queue + visitedUrls + errors                                  │
│  • Per URL: open tab → wait 3s → executeScript                   │
│  • getLinksOnPage(siteOrigin) collects same-origin <a href>      │
│  • BFS crawl of all pages                                        │
│  • generateSitemap() → XML urlset                                │
│  • generateRobotsTxt() → User-agent, Allow, Sitemap              │
└─────────────────────────────────────────────────────────────────┘

• Origin scope: Only links with the same origin as the start URL are included (other subdomains/domains are not).
• Wait: After each page loads, the extension waits 3 seconds so dynamic content and client-side routing can render.
• Timeout: If a page doesn’t load within 20 seconds, it’s skipped and added to the error list.
• XML safety: Characters <, >, &, ', " in sitemap URLs are escaped as XML entities.

🔐 Permissions

The extension only crawls when you click Start Crawl; it does not collect data in the background continuously.

⚠️ Limitations

• Large sites: Crawling hundreds or thousands of pages can take a long time and may hit Chrome tab limits.
• Single origin: Only the start URL’s origin is crawled; other subdomains or domains are not included.
• Hash (#) URLs: Fragment URLs may be treated as one page; manual editing may be needed.
• JS-generated links: On heavy SPAs, 3 seconds may not be enough; you can increase the wait in background.js (e.g. to 5000 ms).

🤝 Contributing

1. Fork this repo.
2. Create a branch: git checkout -b feature/your-feature
3. Commit your changes: git commit -m 'Add some feature'
4. Push the branch: git push origin feature/your-feature
5. Open a Pull Request against this repo.

📄 License

This project is licensed under the MIT License. See LICENSE for details.

👤 Author

eros1sh

• GitHub: @eros1sh
• Project: https://github.com/eros1sh/sitemap-generator

This extension was built to simplify generating sitemaps and robots files for SEO and indexing.