Web Bot Auth Key Generator

Most websites still decide whether a visitor is a known crawler by looking at two signals: the User-Agent header and the IP address the request comes from. Both are weak. Any script can send a User-Agent that claims to be a famous search engine or AI assistant, and IP allowlists break whenever a bot operator moves to new cloud ranges or routes traffic through a proxy. Site owners end up either trusting liars or blocking legitimate bots by accident.

Web Bot Auth replaces those guesses with cryptography. The bot operator creates a key pair, publishes the public key at a well-known URL on a domain they control, and signs every outgoing request with the private key. A website (or the CDN in front of it) reads the signature headers, fetches the public key from the operator's key directory, and checks the math. If the signature verifies, the request really came from whoever holds that private key. Nobody else can produce a valid signature, no matter what User-Agent they send.

The signing itself is not new. It reuses HTTP Message Signatures, standardized as RFC 9421, which defines how to sign selected parts of an HTTP message and carry the result in the Signature-Input and Signature headers. Web Bot Auth adds a profile on top: which components bots should sign, a tag value of web-bot-auth, a Signature-Agent header that points to the key directory, and a JSON format for that directory.

Be aware that the Web Bot Auth pieces are still IETF Internet-Drafts, not finished RFCs. The architecture draft and the HTTP Message Signatures Directory draft can still change, and field names or required parameters may shift between revisions. Cloudflare already documents verification of these signatures, so there is real deployment, but you should follow the drafts and the verifier you care about and expect to adjust your signer over time.

This generator gives you everything needed to try the scheme end to end: an Ed25519 key, the directory to publish, and a signed example request you can compare against your own code.

You enter three things: the https origin that will host your key directory (for example https://bot.example.com), an example target host such as example.com, and a key validity of no expiry or 30, 90 or 365 days. Pressing Generate key pair creates a fresh Ed25519 key with your browser's Web Crypto API and fills in a set of result cards.

• Key summary. The headline is the key ID, a JWK thumbprint that every signature references. Below it you see the algorithm (ed25519), the full directory URL, the creation time and the expiry time (or "Never").
• Key directory. A JSON Web Key Set with one entry: kty, crv, kid, the public value x, use: "sig" and, if you chose a validity, nbf and exp. It contains the public key only and can be copied or downloaded as a JSON file.
• Directory response headers. The Content-Type, a Cache-Control of max-age=86400, and a sample Signature-Input and Signature pair that signs the directory response itself, valid for 24 hr.
• Private key. The same key as a JWK (with the secret d parameter) and as a PKCS#8 PEM block, each with its own download button.
• Public key (PEM). The SPKI PEM form, handy for verifiers and libraries that do not read JWK.
• Signed request example. Ready-made Signature-Agent, Signature-Input and Signature headers plus a curl command that sends a GET to the root of your target host with them.
• Signature base. The exact text that was signed for that request, line by line.

The target host is used only for the example request; it does not limit where you can use the key later. Changing any input clears the results, so what you see always matches the current form. The tool does not verify signatures from other software and does not host your directory. It is free, needs no sign up, and the key is generated on your device.

Verifiers find your public key through the Signature-Agent header your bot sends. Its value is your origin, and the directory lives at a fixed path under it: /.well-known/http-message-signatures-directory. With https://bot.example.com as the origin, the summary card shows the full URL you need to serve, and the key directory card holds the body.

The response should use the media type application/http-message-signatures-directory+json, not plain application/json. The directory response headers card lists it together with a Cache-Control value of max-age=86400, which lets verifiers cache the keys for a day instead of fetching them on every request.

The directory response is also signed with the same key. That proves the party serving the file actually holds the matching private key, so a copied or tampered directory on another host is easy to reject. The sample signature covers @authority (the host of your origin) under the tag http-message-signatures-directory and is valid for 24 hr from the moment you generated it. Treat it as a reference, not something to paste into a static config: once it expires, verifiers will refuse it. In production your server, edge function or build step should sign each directory response fresh with the private key.

The Key validity option controls the nbf (not before) and exp (expires) fields in the directory entry. Both are Unix timestamps in seconds; nbf is the creation time and exp adds 30, 90 or 365 days. Choosing No expiry leaves both fields out.

Rotation works by overlap. Generate a new key before the old one expires, add its entry to the keys array next to the existing one, start signing with the new key, and remove the old entry once nothing uses it. Because each entry carries its own kid, verifiers pick the right key from the keyid in each signature. The generator produces one key per run, so for rotation you merge the new entry into your existing directory by hand.

RFC 9421 never signs the whole request. The signer picks a list of covered components, builds a text block called the signature base from them, and signs that text. The example request from this tool covers two components:

• @authority, the host the request is going to (your example target host)
• signature-agent, the header that tells the verifier where your key directory lives

The Signature-Input header lists those components and then a set of parameters:

The signature label is sig1, so Signature-Input starts with sig1= followed by the component list and parameters, and Signature holds sig1= plus a value wrapped in colons. That value is the base64 encoded Ed25519 signature.

The signature base is built line by line. Each covered component becomes one line with its name in quotes, a colon, a space and its value. For signature-agent the value is the header exactly as sent, including its surrounding quotes. The last line is always "@signature-params" followed by the same component list and parameters that appear in Signature-Input. Lines are joined with a single newline and there is no trailing newline. The private key signs those exact bytes.

A verifier repeats the process: it reads Signature-Input, rebuilds the base from the request it received, fetches the key named by keyid from the directory, and checks the signature. It also rejects signatures past their expires time, and it can remember nonces to refuse replays. Comparing your own signer's output with the Signature base card is the fastest way to find a mismatch, because a single extra space or a missing quote changes the bytes and breaks verification.

A key generator is only worth trusting if it never sees your private key. This one runs entirely in your browser: Web Crypto creates the Ed25519 key pair on your device, the page builds every output locally, and nothing is uploaded to any server. There is no account and no storage, which also means the key exists only in that open tab.

That has a practical consequence. Reloading the page, closing the tab or pressing Reset discards the key for good. Before you leave, download the private key with the JWK or PEM button (or both). The JWK file is convenient for JavaScript and JOSE libraries; the PKCS#8 PEM file is what OpenSSL, most server languages and many secret stores expect. If you lose the private key after publishing the directory, you cannot sign with that key again. Generate a new pair and publish the new directory.

Store the private key where your bot's other credentials live: a secret manager, an encrypted environment variable in your deployment platform, or a key management service. Keep it out of source control, logs, screenshots and chat threads. Anyone who obtains it can sign requests that verifiers will attribute to your bot.

Know which parts are public and which are secret. In the private JWK, the d parameter is the secret; x is the public half and is safe to share. The key directory card contains only x, which is why it can be published. Never add d to the directory, and never publish the PEM block labelled as a private key. The Public key (PEM) card and the directory are the only outputs meant to be visible to others.

If a key leaks, remove its entry from your directory right away, generate a replacement, publish it and switch your signer over. A finite validity of 30 or 90 days limits how long a forgotten or leaked key can be abused, at the cost of having to rotate on a schedule.

Web Bot Auth answers one question: who is making this request. It does not say what that bot is allowed to read. Access rules still live in robots.txt, and a well behaved crawler should identify itself with a signature and also respect the site's rules. The two work together: signatures let a site trust that a request claiming to be your bot really is, and robots.txt tells your bot which paths it may fetch.

If you run a site rather than a bot, an AI crawler robots.txt generator helps you write allow and disallow rules for the user agents of AI crawlers. Once bots sign their requests, those rules become far easier to enforce, because a scraper can no longer get past them just by borrowing another bot's User-Agent.

A few of the formats in this kit appear in other tools on the site. The public key x, the key ID and the nonce are base64url strings, and the Signature value is standard base64; a Base64 encoder is useful when you want to see how raw bytes turn into those strings or compare the two alphabets. JWK comes from the same JOSE family of standards as JSON Web Tokens, so if your bot also handles OAuth access tokens, the JWT Decoder lets you inspect a token's header and claims without sending it anywhere.

A typical setup for a crawler operator looks like this:

1. Generate a key here and publish the directory at your well-known URL with fresh signatures.
2. Add signing to your HTTP client using the signature base as a reference.
3. Keep your crawler's robots.txt handling strict and log the keyid you sign with, so you can match it against any reports from site owners.

For site owners, the order is reversed: decide what you allow in robots.txt first, then use signature verification at your CDN or server to tell genuine bots from impostors.