> ## Documentation Index
> Fetch the complete documentation index at: https://docs.keenable.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Every page is listed in https://docs.keenable.ai/llms.txt. To pick a product for a job, read https://docs.keenable.ai/choose.md
> Setup steps for agents, covering the SDKs, the MCP server and the CLI: https://docs.keenable.ai/skill.md
> Keenable search and fetch work without an API key. To add them to code, use the official SDK for the project's language: https://docs.keenable.ai/sdks.md
> Without a key, call POST https://api.keenable.ai/v1/search/public or GET https://api.keenable.ai/v1/fetch/public with an X-Keenable-Title header that names your app.
> With a key, send it in the X-API-Key header and read it from the KEENABLE_API_KEY environment variable. Never ask a user to paste a key into chat. Every access method: https://docs.keenable.ai/auth.md
> To get an API key without the console, use device login: https://docs.keenable.ai/authentication.md#device-login
> MCP server: https://api.keenable.ai/mcp (no key, or X-API-Key; keyless Codex uses https://api.keenable.ai/mcp-public), or https://api.keenable.ai/mcp-oauth for OAuth sign-in. These docs are an MCP server at https://docs.keenable.ai/mcp and the OpenAPI spec is at https://docs.keenable.ai/api-reference/openapi.json
> Keenable searches and fetches public web pages. If a question needs no web information, skip the call.
> On 402 with an API key (code insufficient_credits), stop and ask a human to add credits; don't retry. On 429, wait the number of seconds in Retry-After, then retry. Before you ship, follow https://docs.keenable.ai/production.md
> Every error, with its cause and fix, is in https://docs.keenable.ai/errors.md and moving from another search API is covered in https://docs.keenable.ai/migrate.md
> Before you compare Keenable with another search API, read https://docs.keenable.ai/guides/evaluate-search-apis.md

# Keenable crawlers

> The KeenableBot and Keenable-User user agents, the published IP lists to verify them, and the robots.txt rules to allow or block KeenableBot.

If you run a website, this page tells you how to recognize requests from Keenable in your logs, check that they are real, and allow or block them.

Keenable.ai, Inc. identifies its requests with two user-agent tokens. Both point to the public crawler page at [keenable.ai/bot ↗](https://keenable.ai/bot), which is the reference for site owners.

## User agents

| Token | What it is used for |
| - | - |
| `KeenableBot` | Crawling. KeenableBot collects publicly available pages to build and maintain the Keenable search index. It also fetches your `robots.txt`. |
| `Keenable-User` | A request made on behalf of a person, or an agent acting for one, who asked for a specific page. For example, a [Fetch](https://docs.keenable.ai/products/fetch.md) call that reads the page live. Keenable-User does not crawl the web automatically. |

KeenableBot sends this `User-Agent` header:

```text theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot)
```

Keenable-User sends this `User-Agent` header:

```text theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Keenable-User/1.0; +https://keenable.ai/bot)
```

Match on the token, `KeenableBot` or `Keenable-User`, not on the full string.

## Verify a request

Anyone can copy a user-agent header, so check the source address as well. Requests that carry a Keenable token come from a fixed set of IP addresses. Keenable publishes one list per token as JSON, in the same format as Googlebot's list:

| Token | IP list |
| - | - |
| `KeenableBot` | `https://app.keenable.ai/bot/keenablebot.json` |
| `Keenable-User` | `https://app.keenable.ai/bot/keenable-user.json` |

This command downloads the list for Keenable-User:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
curl -s https://app.keenable.ai/bot/keenable-user.json
```

You get a `creationTime` and a `prefixes` array. Each entry holds one `ipv4Prefix` in CIDR notation:

```json theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
{
  "creationTime": "2026-10-09T12:04:36.000000",
  "prefixes": [
    {
      "ipv4Prefix": "34.31.116.125/32"
    },
    {
      "ipv4Prefix": "34.31.229.240/32"
    }
  ]
}
```

The output above is trimmed. If a request names a Keenable token and its source address is not in that token's list, the request is not from Keenable. Download the lists again on a schedule, because the addresses can change.

> [!NOTE]
>
> Keenable requests don't carry HTTP Message Signatures (Web Bot Auth) yet, so verify by IP address. Each token already has a key directory, at `https://keenablebot.keenable.ai/.well-known/http-message-signatures-directory` and `https://keenable-user.keenable.ai/.well-known/http-message-signatures-directory`.

## Allow or block KeenableBot

KeenableBot follows the Robots Exclusion Protocol (RFC 9309). Use the `KeenableBot` token in your `robots.txt`. The match is not case-sensitive. If a group names `KeenableBot`, KeenableBot uses that group and ignores the `User-agent: *` group.

This rule blocks KeenableBot from your whole site:

```text theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
User-agent: KeenableBot
Disallow: /
```

Keenable stops indexing your site, so your pages no longer appear in Keenable search results.

This rule keeps KeenableBot out of one directory and allows the rest:

```text theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
User-agent: KeenableBot
Disallow: /private/
```

To allow KeenableBot on a site that blocks other crawlers, add a `KeenableBot` group with `Allow: /`.

A new rule takes effect after KeenableBot next fetches your `robots.txt`.

## Block at your server or firewall

To refuse either agent at your web server, firewall or CDN, match the token in the `User-Agent` header, the addresses in that token's IP list, or both. To allow Keenable through a bot filter, allowlist the same addresses.

## What Keenable does with crawled pages

Keenable builds a search index from the pages KeenableBot collects. The index answers queries from AI agents and applications through [Search](https://docs.keenable.ai/products/search.md) and [Fetch](https://docs.keenable.ai/products/fetch.md). Results link to the source page.

## Contact

Email [abuse@keenable.ai](mailto:abuse@keenable.ai) to report excessive load, ask for a lower crawl rate, request removal of specific content, or report a crawler that claims to be Keenable. Include the hostname, the timestamps and the source addresses so Keenable can tell you whether the traffic is its own.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.