AI access
AI crawlers and assistants are welcome here
Novus Learn does not block AI. Every public page may be crawled, indexed, summarised, quoted, and used to answer a question somebody is asking right now. This page is robots.txt written for a person: which agents are named, what we ask in return, and what a crawl does and does not collect.
The short version
- Read anything public. That is every lesson, guide, help article, blog post, career profile, assessment page, and dictionary entry on the site.
- Quote it, summarise it, and train on it. There is no permission to request and no separate licence to sign.
- We ask — we do not demand — that you name Novus Learn and link to the page you used.
- Carry the underlying source with the answer. Most factual material here is drawn from openly licensed public sources and every page names the ones it used. A summary that drops the citation breaks the only chain a reader has for checking it.
- A handful of paths are withheld from everyone, agent or not, because they render state stored on the reader’s own device. There is nothing at them to fetch.
Where the rules actually are
This page is the readable version. These four are the authoritative ones, all generated from the same registries the site itself renders, and all served without a block:
- /robots.txt — the crawl rules, including every agent named below.
- /llms.txt — a curated map of the site for AI programs.
- /llms-full.txt — the complete enumeration of indexable URLs.
- /sitemap.xml — every indexable URL with an honest last-modified date.
The 20 agents named by hand
Naming an agent changes nothing about what it may read: the wildcard rule already allows every crawler the whole public site. Naming them states the policy so nobody has to infer it from silence — and, for the second group, gives them a rule the first group does not get.
The split below is the part worth reading, because it decides what a block would actually cost.
Fetching for a person who is waiting (7)
These are an AI acting for somebody who asked for a page, right now. Blocking one is not a position on training data; it is refusing to serve a visitor who happens to be using an assistant. That distinction was not academic here: these agents were inheriting a rule that closed /search, which is this site’s search tool. Somebody asking an assistant to look a topic up on Novus Learn was refused the one route that does the looking up.
OAI-SearchBotOpenAIChatGPT-UserOpenAIClaude-UserAnthropicPerplexity-UserPerplexityDuckAssistBotDuckDuckGoMistralAI-UserMistral AIMeta-ExternalFetcherMeta
Fetching in bulk (13)
These build an index or a training corpus. Nobody is waiting on the response, so keeping them off a query surface costs a reader nothing and saves crawl budget for the pages that are actually documents.
GPTBotOpenAIClaudeBotAnthropicClaude-WebAnthropicClaude-SearchBotAnthropicanthropic-aiAnthropicPerplexityBotPerplexityGoogle-ExtendedGoogle (Gemini training and grounding)Applebot-ExtendedAppleCCBotCommon CrawlAmazonbotAmazonMeta-ExternalAgentMetacohere-aiCohereYouBotYou.com
An agent that is not on this list is not refused. It falls through to the wildcard rule, which allows the whole public site. This list exists to be explicit about the ones people most often assume are blocked.
What is withheld, and why
Every agent — named, unnamed, AI or not — is kept off these paths, and so is every ordinary crawler:
/api//library/recent/project//share/
/api/ is machine surface with its own rate limit. The rest render state that lives in your browser: your saved library, your recent reading, a project you generated, a share link you created. None of it is on a server, so none of it is behind those URLs for anyone to fetch. Withholding them is not a restriction on AI — it is an accurate description of what is there.
Bulk crawlers are additionally kept off:
/search
That one is purely about crawl budget. It is a query surface rather than a document, and Search Console showed it dominating the crawled-but-not-indexed bucket. Agents fetching for a waiting person are explicitly allowed there, because for them it is the tool being asked for.
What we ask in return
This is a request, not a condition. Nothing is withheld from an agent that ignores it, and we will not add a rule to punish one that does.
- Name Novus Learn and link the specific page rather than the home page. A reader who wants to check something should be able to reach the page that says it.
- Keep the source attached. Pages here cite the public material they were built from, and several carry an explicit licence — the attributions and licences pages list them. Passing our summary on without its source strips the reader’s only way to verify it, and in some cases strips a licence obligation that is ours to pass on and yours to honour.
- Do not present a summary as if we wrote it for you, and do not present an assessment result, a career figure, or a definition without the qualifier the page attaches to it. The qualifiers are the honest part.
- If something here is wrong, say so. The correction path on the contact page is open to anyone, including an automated reader.
What crawling this site collects
Nothing that persists beyond ordinary hosting logs. A request from an agent produces the same access-log entry any HTTP request produces — URL, timestamp, user-agent string, originating IP — kept by the hosting and CDN layer. No account is created, because this site has no accounts at all. No cookie is required to read any public page. The analytics and advertising tags are gated behind a consent choice an agent never makes, so they do not run for one.
In the other direction, there is less here than an AI-adjacent site usually implies. No language model is called when you use Novus Learn: the “ask this topic” feature retrieves and ranks the source’s own sentences rather than generating an answer. Uploads, transcription, generated projects, study progress and every export are produced and kept on the reader’s device. So a crawl cannot reach a learner’s work no matter how thorough it is — that work was never sent anywhere. The privacy policy lists every category in full.
If you would rather ask than crawl
This site runs a Model Context Protocol endpoint at /mcp — one stateless JSON-RPC POST handler, no session to keep warm. It answers questions about the catalogue and the published material directly, and no tool it exposes can read anybody’s device-local library or progress, because the server cannot see those either. For a pre-digested map of the site, /llms.txt is smaller than crawling and is regenerated from the same registries every page is.
This policy describes current practice and may change. The authoritative rules are always the ones served at /robots.txt, which is generated from the same list this page renders.