Index a site
Turn a whole site into a searchable local library
Enter a site address. Novus Learn reads its robots.txt and sitemap, then builds a source-grounded breakdown of each allowed page — one at a time, in your browser. Every page becomes its own local project you can open, study, and search from the Hub. Only the page URLs pass through our server, to fetch them; nothing is stored there.
How this works and stays responsible
- Only same-site pages your robots.txt allows are read; Crawl-delay is respected.
- Pages are fetched one at a time, capped at your chosen limit (up to 50).
- Each breakdown is built and stored only in this browser (IndexedDB); the server keeps nothing.
- Everything stays extractive and source-linked — nothing is generated or invented.
What “index a site” does
Some things you want to learn are spread across a whole site — a documentation set, a handbook, a multi-part guide. Index a site points Novus Learn at one domain, discovers the pages it is allowed to read, and builds each into a source-grounded project you can search from a single command bar.
It is deliberately bounded and polite. This is not a web-scale crawler: it stays on one origin, respects the site's own rules, and keeps the whole library on your device.
How it works
Give it a starting site
Enter a domain or a page on it. Novus Learn reads the site's robots.txt and, when present, its sitemap to find candidate pages.
It respects the site's rules
Disallowed paths are skipped and any crawl-delay is honoured. The crawl stays on the same origin and is capped (25 pages by default, 50 at most) so it never becomes a burden.
You crawl one page at a time
Discovery lists the allowed pages; you build them into projects one at a time, and you can stop at any point. Each page shows its own status so nothing happens silently.
Everything becomes locally searchable
Each indexed page is a source-grounded project saved on your device and added to a local content index, so you can search across the whole set — by title and by the text inside the pages — from the ⌘K hub.
Built to be responsible
The crawl is same-origin only, robots-aware, rate-limited on both your side and the target host, and hard-capped. It exists to help you study a site you already have access to — not to mirror the web.
As with everything in Novus Learn, each page's breakdown is extractive and source-grounded: claims are the page's own sentences, and each links back to where it came from.
When it is the right tool
- Working through a documentation set or knowledge base.
- Studying a multi-page guide or online handbook.
- Building a small, searchable local library from one trusted site.
Frequently asked questions
Does this crawl the entire internet?
No. It stays on the single origin you give it, follows that site's robots.txt and crawl-delay, and is capped at a small number of pages (25 by default, 50 maximum).
Does it obey robots.txt?
Yes — strictly. Disallowed paths are skipped and any crawl-delay is respected. Discovery uses the site's own sitemap when one is available.
Where is the indexed library stored?
On your device. Each page becomes a local project in IndexedDB and is added to a local search index; nothing is uploaded to an account or cloud database.
Can I stop a crawl partway through?
Yes. You build pages one at a time and can cancel at any point. Pages already built stay in your library.
How do I search what I've indexed?
Open the ⌘K hub. It searches your library by page title and by the actual text inside the pages, and deep-links to the matching evidence.