GitHub Store
janreges

siteone-crawler

janreges

A cross-platform website crawler and analyzer written in Rust, ideal for SEO, security, accessibility, and performance optimization, with a single binary and zero runtime dependencies.

v2.5.1
Published at
Jun 27, 2026
938
Stars
19,096
Download
Synced at
Oct 10, 2026

Latest release

v2.5.1Jun 27, 2026, 05:52 AM
Downloads slow? Switch the source anytime above

README

Project Overview

SiteOne Crawler is a cross-platform website crawler and analyzer written in Rust, designed for developers seeking security and performance insights, SEO specialists, and website owners. It comes as a single native binary with zero runtime dependencies.

Key Features

  • Three ways to run: Interactive wizard, command-line interface, and desktop GUI app
  • Rich output formats: Interactive HTML audit report, JSON, and text, with built-in SMTP mailer
  • CI/CD integration: Built-in quality gate (--ci) with configurable thresholds — exit code 10 on failure
  • Offline & Markdown: Complete offline clones or convert entire websites to clean Markdown
  • Deep crawling: Crawl every page and asset, identify errors (404s, redirects), generate sitemaps
  • Performance: 25% faster execution and 30% lower memory consumption than PHP+Swoole version

Target Audience

  • SEO specialists and website optimizers
  • Developers and DevOps engineers
  • QA engineers and consultants

Quick Start

  1. Download the binary for your platform from Releases
  2. Run without arguments for the interactive wizard, or use CLI arguments
  3. See the documentation

Core Advantages

  • High performance: 25% faster, 30% less memory
  • Cross-platform: Windows, macOS, Linux (x64 & arm64)
  • Single binary: No runtime dependencies
  • Open-source & free

Release history

Sort