remember() or add(), and cognee asks Keenable for the page as Markdown instead of crawling the HTML itself. The rest of the pipeline (chunking, embeddings, entity extraction, recall) doesn’t change.
→ View the change on GitHub · cognee’s URL-ingestion guide
Install
Install cognee
The Keenable backend is part of cognee, so there’s no Keenable package to add:Set the key
cognee needs the key. It picks Keenable only if
KEENABLE_API_KEY is set, and it calls the keyed /v1/fetch endpoint. If the key is missing, cognee uses a different backend instead of calling Keenable without a key. Create a key at app.keenable.ai/console.Check the precedence
cognee chooses a fetch backend from the environment, in this order:- Tavily, if
TAVILY_API_KEYis set - Keenable, if
KEENABLE_API_KEYis set - The built-in crawler
TAVILY_API_KEY or select the backend explicitly.
Use
Give cognee a URL
Pass the URL toremember(), then query what cognee learned:
remember() fetches the page through Keenable, ingests the Markdown, and builds the graph. recall() answers from the graph without fetching the page again.
Fetch the current version of a page
By default Keenable returns its indexed copy of a page when it has one, and fetches the page from the source when it doesn’t. To fetch every page from the source, turn on live fetching:fetch.live. See what each call costs.
Select the backend explicitly
If you call cognee’s fetch layer directly, pass the backend as a parameter. The environment precedence doesn’t apply:Narrow the page with an extraction prompt
KeenableConfig carries the per-call options. prompt makes Keenable return only the part of the page you ask for instead of the whole document:
Track a page over time
cognee’s scheduled scraper runs through the same fetch path (it needsapscheduler):
WebPage nodes into the graph, linked is_part_of to a WebSite. The ScrapingJob links to the WebSite with is_scraping. Every page carries a SHA-256 content_hash, so the job can tell whether a page changed between runs. Unlike remember(), this path indexes the nodes directly and skips chunking and entity extraction.
Fetch behavior
- cognee sends
GET /v1/fetch?url=...withX-API-Key, pluslive=trueandprompt=...when set, and keeps thecontentfield of the response. - Up to 5 URLs are in flight at once (
KeenableConfig.concurrency), with a 30-second timeout per request (timeout, 1 to 120). - If a URL fails, cognee logs a warning, skips it and continues with the rest of the batch. If every URL fails, cognee raises the first error instead of returning an empty result, so a bad key shows up as an authentication error.
- Warnings identify a failed URL by its position in the batch and the exception class, never by the URL, so credentials in query strings can’t leak into logs.
Configuration
The environment variables cover the memory path (
remember(), add()). The KeenableConfig fields cover direct calls to fetch_page_content, web_scraper_task and cron_web_scraper_task, and override the environment for that call.
Related pages
Fetch quickstart
Turn any URL into clean markdown your model can read, or get only the part a prompt asks for. Try it with no API key. Code, cost per call and gotchas.
LlamaIndex
Give LlamaIndex agents web search and page reading with the Keenable ToolSpec.
Haystack
Add web search and page fetch to Haystack pipelines with the Keenable components.
GPT Researcher
Give GPT Researcher web search with no API key through gpt-researcher-keenable, a retriever plugin.