Turn a whole website into a searchable study library

When the thing you want to learn is a whole site

A single article is easy to bring into a study tool. Documentation, handbooks, course notes, and multi-part guides are harder because the explanation is distributed across many pages. The introduction defines the vocabulary, a later page describes the process, and the useful edge cases live somewhere else. Saving one page at a time leaves you with fragments and no reliable way to search across them.

Index a site is Novus Learn's bounded answer to that problem. Give it a public site address, choose a cap of ten, twenty-five, or fifty pages, and it discovers the pages that site permits it to read. Each page then passes through the same source-grounded web-page pipeline as the single-page reader and becomes its own local project, complete with notes, flashcards, quiz, canvas, visuals, Ask, and evidence links.

The result is not a copy of the site and it is not a shared search engine. It is a small personal study library on your current device, built from a source you chose.

Discovery begins with the site's own rules

Novus Learn checks robots.txt before it builds a page list. If the file names one or more sitemaps, those are the starting point; otherwise it tries the conventional sitemap location. A sitemap index can lead to a bounded set of child sitemaps. When no sitemap yields pages, discovery falls back to links found on the site's home page and tells you that it did so.

Only HTTP or HTTPS pages on the same origin are considered. Fragments are removed, duplicates are collapsed, private network addresses are rejected, and paths disallowed by robots.txt are skipped. The requested page cap is applied throughout discovery, including to the candidate pool, so an enormous sitemap cannot turn a personal study action into an unbounded crawl.

This is a deliberately small tool. Twenty-five pages is the default and fifty is the hard maximum. That is enough for a documentation section or handbook while keeping the work legible to you and respectful to the publisher.

• Same-origin pages only; the indexer never fans out across other domains.
• Robots.txt disallow rules are enforced for the multi-page crawl.
• A published crawl-delay is respected, with a polite minimum between page requests.
• Discovery and ingestion are rate-limited for both the requester and the target host.
• The crawl is capped at the limit you choose and can be stopped at any time.

Every page becomes a real project

After discovery, the browser sends one allowed URL at a time through the existing Read a web page endpoint. The server fetches that page with content-type, redirect, size, timeout, and private-address guards, then returns a normalised document. Your browser builds the knowledge project and saves it to IndexedDB. The next page does not begin until the required delay has passed.

Treating pages separately is important. It keeps each project's title, canonical URL, author, date, licence note, sections, and evidence anchors intact. A claim from the installation guide still points to the installation guide; it is not blended into a synthetic mega-document with provenance that becomes hard to untangle.

If a page has already been saved, the progress list marks it instead of creating a duplicate. A page that cannot be read is skipped with its own diagnostic while the rest of the crawl continues. Stop the run partway through and the projects already completed remain available.

Search across the library, then jump to the evidence

Saving pages would not solve the distributed-reading problem if you still had to open them one by one. Novus Learn maintains a local inverted index alongside the projects in IndexedDB. Open the Novus Hub with ⌘K or Ctrl-K and a search can match project titles as well as passages inside the pages you indexed. A passage result deep-links into the project near the supporting section.

The index is built on your device from the same claims and source blocks the study tools use. It gives title, entity, and claim matches useful boosts without asking a remote service to store or rank your reading history. Adding or removing a project updates the local records with it.

This is where a multi-page site starts feeling like one study resource. Search a configuration term, open the most relevant page, ask that page a follow-up question, and then move into its flashcards or sources without losing which document made the claim.

What stays local — and what still crosses the network

The durable library, project structure, search index, and study progress stay in this browser. There is no account, cloud library, or cross-device sync. The server is involved only because a browser cannot safely fetch arbitrary sites directly: it discovers robots and sitemap URLs, fetches each requested public page through the guarded ingestion route, and returns the parsed document. It does not keep the resulting project.

That boundary is worth understanding. Indexing a site is not an offline action because the original pages have to be requested. Once a project is saved, its extracted study material can be reopened locally, but clearing site data, using private browsing, or switching browser profiles can remove or hide it. The source publisher also remains responsible for the page's rights and accuracy; Novus Learn records the source URL and any available licence rather than claiming ownership.

A responsible indexing routine

Start with the smallest cap that covers the section you need. A documentation subsection with ten well-chosen pages is often more useful than fifty loosely related pages. Watch the discovered paths before the crawl gets far, stop if the site shape is not what you expected, and read any skipped-page notes instead of repeatedly retrying a publisher that is refusing or throttling access.

When the run finishes, open My Library and search for two or three terms you know the site uses. Remove navigational or thin pages that add no learning value. Then choose one central project, scan its summary, and follow its internal evidence before building a flashcard routine. The site indexer solves collection and retrieval; your final library still improves when you curate it.

Use the tool for public material you are permitted to access: documentation, open handbooks, multi-page tutorials, or your organisation's public knowledge base. It is not designed to bypass logins, paywalls, robots rules, or publisher limits, and its technical boundaries enforce that posture rather than leaving it as a suggestion.