Skip to main content
Fetch backend From version 1.5.0, cognee can fetch URLs through Keenable, so pages enter its graph memory as clean Markdown. Pass a URL to remember() or add(), and cognee asks Keenable for the page as Markdown instead of crawling the HTML itself. The rest of the pipeline (chunking, embeddings, entity extraction, recall) doesn’t change. → View the change on GitHub · cognee’s URL-ingestion guide

Install

Install cognee

The Keenable backend is part of cognee, so there’s no Keenable package to add:

Set the key

cognee needs the key. It picks Keenable only if KEENABLE_API_KEY is set, and it calls the keyed /v1/fetch endpoint. If the key is missing, cognee uses a different backend instead of calling Keenable without a key. Create a key at app.keenable.ai/console.

Check the precedence

cognee chooses a fetch backend from the environment, in this order:
  1. Tavily, if TAVILY_API_KEY is set
  2. Keenable, if KEENABLE_API_KEY is set
  3. The built-in crawler
If both keys are set, cognee uses Tavily. To route URL ingestion through Keenable, unset TAVILY_API_KEY or select the backend explicitly.

Use

Give cognee a URL

Pass the URL to remember(), then query what cognee learned:
remember() fetches the page through Keenable, ingests the Markdown, and builds the graph. recall() answers from the graph without fetching the page again.

Fetch the current version of a page

By default Keenable returns its indexed copy of a page when it has one, and fetches the page from the source when it doesn’t. To fetch every page from the source, turn on live fetching:
With live fetching on, every request goes to the source page. Leave it off when the indexed copy is recent enough and you want the faster cached path. A fetch from the source is billed as fetch.live. See what each call costs.

Select the backend explicitly

If you call cognee’s fetch layer directly, pass the backend as a parameter. The environment precedence doesn’t apply:

Narrow the page with an extraction prompt

KeenableConfig carries the per-call options. prompt makes Keenable return only the part of the page you ask for instead of the whole document:

Track a page over time

cognee’s scheduled scraper runs through the same fetch path (it needs apscheduler):
Each run writes WebPage nodes into the graph, linked is_part_of to a WebSite. The ScrapingJob links to the WebSite with is_scraping. Every page carries a SHA-256 content_hash, so the job can tell whether a page changed between runs. Unlike remember(), this path indexes the nodes directly and skips chunking and entity extraction.

Fetch behavior

  • cognee sends GET /v1/fetch?url=... with X-API-Key, plus live=true and prompt=... when set, and keeps the content field of the response.
  • Up to 5 URLs are in flight at once (KeenableConfig.concurrency), with a 30-second timeout per request (timeout, 1 to 120).
  • If a URL fails, cognee logs a warning, skips it and continues with the rest of the batch. If every URL fails, cognee raises the first error instead of returning an empty result, so a bad key shows up as an authentication error.
  • Warnings identify a failed URL by its position in the batch and the exception class, never by the URL, so credentials in query strings can’t leak into logs.

Configuration

The environment variables cover the memory path (remember(), add()). The KeenableConfig fields cover direct calls to fetch_page_content, web_scraper_task and cron_web_scraper_task, and override the environment for that call.

Fetch quickstart

Turn any URL into clean markdown your model can read, or get only the part a prompt asks for. Try it with no API key. Code, cost per call and gotchas.

LlamaIndex

Give LlamaIndex agents web search and page reading with the Keenable ToolSpec.

Haystack

Add web search and page fetch to Haystack pipelines with the Keenable components.

GPT Researcher

Give GPT Researcher web search with no API key through gpt-researcher-keenable, a retriever plugin.