Self-host Firecrawl for your agents
Your own Firecrawl API, private by default, with keyless LLM extraction and MCP.
Shelley handles the setup. About 15 minutes.
Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/firecrawl/prompt.txt | ssh exe.dev new --prompt=/dev/stdinFirecrawl turns web pages into what language models want: one call returns a page as clean Markdown, a crawl walks a whole site, and a JSON schema gets you structured data out of any page. With more than 200,000 stars on GitHub it's one of the most used tools for giving agents the web, mostly through Firecrawl Cloud.
The open-source version runs the same API on your own machine, and an exe.dev VM is a good one for it. The API stays private behind exe.dev's HTTPS proxy, with no Firecrawl auth to set up: you reach it with your exe.dev login, and scripts and agents elsewhere with a token for the VM. JSON extraction uses exe.dev's LLM integration, so there's no OpenAI key on the VM. Claude Code and Codex are already installed, and the Firecrawl MCP server connects them to your instance. There are no credits to run out of, and the VM is always on, so crawls finish while your laptop sleeps.
Once it's set up, an agent on the VM answers like this:
$ claude -p "Use the firecrawl_scrape tool to get https://books.toscrape.com/catalogue/category/books/mystery_3/index.html as markdown. List the five cheapest books on that page as 'title - price', cheapest first."
These are the five cheapest books on that page, cheapest first:
1. Tastes Like Fear (DI Marnie Rome #3) - £10.69
2. Hide Away (Eve Duncan #20) - £11.84
3. Playing with Fire - £13.71
4. That Darkness (Gardiner and Renner #1) - £13.92
5. A Murder in Time - £16.64
What you get#
- Firecrawl 2.11.809 (API, workers, Playwright, PostgreSQL, Redis, RabbitMQ) in Docker Compose, restarted after a reboot
- A scrape that returns Markdown from a page built with JavaScript
- JSON extraction through exe.dev's LLM integration, with no OpenAI key on the VM
- The Firecrawl MCP server connected to Claude Code and Codex on the VM
- A Claude Code run that scrapes books.toscrape.com through Firecrawl
What's running, and where
http://localhost:3002
the API, for agents and scripts on the VM
https://my-firecrawl.exe.xyz:3002/
the same API for your computer and other VMs, with a VM token
ssh my-firecrawl.exe.xyz
your terminal
Deploy with Shelley#
Click Deploy and Shelley will set up Firecrawl on a new VM, then check that it works. No config-file wrangling required.
Prefer a terminal?
curl -fsSL \
https://exe-staging.dev/deploy/firecrawl/prompt.txt \
| ssh exe.dev new --prompt=/dev/stdinWhat Shelley is told
Set up Firecrawl on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/firecrawl.md?via=deploy from top to bottom.
The recipe's example VM name is my-firecrawl. This VM's name is the output of `hostname`: use it wherever the recipe writes my-firecrawl. Skip the step that creates the VM; you are on it.
You are done when all of these are true. Check each one yourself:
- Firecrawl 2.11.809 (API, workers, Playwright, PostgreSQL, Redis, RabbitMQ) in Docker Compose, restarted after a reboot
- A scrape that returns Markdown from a page built with JavaScript
- JSON extraction through exe.dev's LLM integration, with no OpenAI key on the VM
- The Firecrawl MCP server connected to Claude Code and Codex on the VM
- A Claude Code run that scrapes books.toscrape.com through Firecrawl
Run the VM-side commands in "Set up by hand" only; skip creating the VM.
If `exeuntu configure` reports an existing user config, leave it alone,
and if `~/.codex/config.toml` already has `[mcp_servers.firecrawl]`,
don't add it twice. Verify with the checks in that section. Don't create
VM tokens unless the person asks. Finish by showing the Claude Code
answer, and tell the owner how to reach the API from their computer
("Use it from your computer or another VM").
Finish with the URLs I should open.
Also at /deploy/firecrawl/prompt.txt. The recipe it follows is /deploy/firecrawl.md.
Set up the VM#
Deploy does this for you. By hand, it's a few blocks of commands: start Firecrawl's Docker Compose stack at a pinned release, check a scrape and an extraction, and connect Claude Code and Codex to it.
Set up by hand
On your computer, create a VM and connect. Pick any free name; if you've already clicked Deploy or have a VM, skip the first command and use its name:
ssh exe.dev new --name=my-firecrawl
ssh my-firecrawl.exe.xyz
The default size (2 CPUs, 8 GB of memory, 25 GB of disk) is enough; see Size and speed for bigger jobs.
Start Firecrawl#
On the VM, turn on Docker at boot, then check out release 2.11.809.
Firecrawl's self-hosting guide
runs the compose file of an exact release, because main and the images
change several times a day:
sudo systemctl enable --now docker.service
git clone --depth 1 --branch v2.11.809 https://github.com/firecrawl/firecrawl.git ~/firecrawl
cd ~/firecrawl
Write the settings to .env, which only you can read. Authentication
stays off (the exe.dev proxy guards the port), the queue's database gets a
random password, and the three OPENAI_ and MODEL_ lines send
extraction to exe.dev's LLM integration:
(umask 077; cat > .env <<EOF
FIRECRAWL_VERSION=2.11.809
USE_DB_AUTHENTICATION=false
POSTGRES_USER=postgres
POSTGRES_PASSWORD=$(openssl rand -hex 24)
POSTGRES_DB=postgres
OPENAI_BASE_URL=https://llm.int.exe.xyz/v1
OPENAI_API_KEY=exe-llm-integration
MODEL_NAME=gpt-5.4-mini
NUQ_WORKER_COUNT=5
EOF
)
Then add a docker-compose.override.yaml, which Docker Compose merges
into Firecrawl's own file. It uses the release's prebuilt API image
instead of compiling it, drops the API's 4-CPU limit (Docker refuses to
start it on a 2-CPU VM), restarts everything after a reboot, and leaves
out FoundationDB, an optional queue backend:
cat > docker-compose.override.yaml <<'EOF'
services:
api:
image: ghcr.io/firecrawl/firecrawl:${FIRECRAWL_VERSION}
build: !reset null
cpus: !reset null
restart: unless-stopped
environment:
NUQ_WORKER_COUNT: ${NUQ_WORKER_COUNT:-5}
playwright-service:
image: firecrawl-playwright:${FIRECRAWL_VERSION}
restart: unless-stopped
nuq-postgres:
image: firecrawl-nuq-postgres:${FIRECRAWL_VERSION}
restart: unless-stopped
redis:
restart: unless-stopped
rabbitmq:
restart: unless-stopped
foundationdb:
profiles: [fdb]
foundationdb-init:
profiles: [fdb]
EOF
Build the two small images (the Playwright service and the queue's PostgreSQL) from the checkout, pull the API, and start it all. The build takes about two minutes, and the last line waits for the API to answer:
docker compose build
docker compose pull api
docker compose up -d
timeout 180 sh -c 'until curl -sf localhost:3002/v0/health/readiness; do sleep 2; done'; echo
Compose prints a warning for every optional variable that isn't set
(The "SUPABASE_URL" variable is not set, and so on); they're harmless.
The wait ends with {"status":"ok"}, and docker compose ps shows
api, playwright-service, nuq-postgres, rabbitmq and redis up.
Check a scrape and an extraction#
That health check doesn't touch the workers or the browser, so scrape a page, one that builds its content with JavaScript:
curl -s localhost:3002/v2/scrape -H 'Content-Type: application/json' \
-d '{"url": "https://quotes.toscrape.com/js/", "formats": ["markdown"]}' | head -c 400; echo
The answer starts with {"success":true,"data":{"markdown": and the page's
first quote, Einstein's about thinking. Then ask for JSON, which goes
through the LLM integration:
curl -s localhost:3002/v2/scrape -H 'Content-Type: application/json' -d '{
"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"formats": [{"type": "json", "prompt": "The five cheapest books on the page",
"schema": {"type": "object", "properties": {"books": {"type": "array",
"items": {"type": "object", "properties": {"title": {"type": "string"},
"price": {"type": "number"}}}}}}}]}' | jq .data.json
It prints five books, starting with Tastes Like Fear at 10.69.
Connect Claude Code and Codex#
The Firecrawl MCP server needs Node.js 22 or newer, and the VM's is older. Install Node.js 24 and the server, connect both agents to the LLM integration, then add the server to each, pointed at your instance:
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g firecrawl-mcp@3.41.0
exeuntu configure claude
exeuntu configure codex
claude mcp add --scope user firecrawl -e FIRECRAWL_API_URL=http://localhost:3002 -- firecrawl-mcp
cat >> ~/.codex/config.toml <<'EOF'
[mcp_servers.firecrawl]
command = "firecrawl-mcp"
env = { FIRECRAWL_API_URL = "http://localhost:3002" }
enabled_tools = ["firecrawl_scrape", "firecrawl_map", "firecrawl_crawl", "firecrawl_check_crawl_status", "firecrawl_search"]
default_tools_approval_mode = "approve"
EOF
The Codex lines limit it to the tools a self-hosted instance answers and
let it call them without asking. Check it end to end: claude mcp list
shows firecrawl: firecrawl-mcp - ✔ Connected, and Claude Code answers
with the five cheapest mystery books, starting with Tastes Like Fear at
£10.69:
mkdir -p ~/project && cd ~/project
claude mcp list
claude -p "Use the firecrawl_scrape tool to get https://books.toscrape.com/catalogue/category/books/mystery_3/index.html as markdown. List the five cheapest books on that page as 'title - price', cheapest first." --allowedTools "mcp__firecrawl__firecrawl_scrape"
Ask an agent to scrape#
On the VM, start claude or codex in a project folder and ask in plain
words. Naming the tool saves the agent a guess:
Use the firecrawl_scrape tool to fetch https://quotes.toscrape.com/js/ as markdown. How many quotes are on the page, and who wrote the first one?
The answer is ten quotes, the first by Albert Einstein. Codex works the
same way: codex exec --skip-git-repo-check "Use the firecrawl MCP server's firecrawl_scrape tool to ...".
The tools that work against a self-hosted instance:
| Tool | Use it for |
|---|---|
firecrawl_scrape |
one page as Markdown, or as JSON with a schema or prompt |
firecrawl_map |
the URLs a site lists in its sitemap and links |
firecrawl_crawl, firecrawl_check_crawl_status |
a whole site or section, as a job the agent checks on |
firecrawl_search |
a web search, optionally scraping the results |
The server offers Claude Code about thirty more, among them
firecrawl_agent, firecrawl_interact, the firecrawl_monitor_* and
firecrawl_research_* tools: those are Firecrawl Cloud products, and
against your instance they fail. If Claude Code reaches for one, name the
tool you want.
Scrape, crawl and extract from code#
The API is Firecrawl's v2 API,
so the official SDKs and any example written for Firecrawl Cloud work
once they point at http://localhost:3002. Leave the API key out; with
authentication off, Firecrawl doesn't check it.
A crawl runs as a job: start it, then ask for its status and pages.
crawl.py does both with the
Python SDK and saves each page as a Markdown file in ~/crawls/<site>/,
with an index.jsonl listing URLs, titles and status codes. On the VM:
mkdir -p ~/scripts && cd ~/scripts
curl -fsSLO https://exe-staging.dev/deploy/firecrawl/crawl.py
uv run crawl.py https://books.toscrape.com/ 50
It prints completed: 50 pages saved to /home/exedev/crawls/books.toscrape.com in 36 s.
A crawl stays under the starting URL's path unless you pass
crawlEntireDomain, follows includePaths and excludePaths (regular
expressions on the path), and skips what robots.txt disallows. Finished
jobs are kept in the queue's database for a while, but the files are
yours to keep: they're on the VM's disk, where your agents can read them.
Other calls, with curl:
# The links a site lists (its sitemap first)
curl -s localhost:3002/v2/map -H 'Content-Type: application/json' \
-d '{"url": "https://exe.dev/docs", "limit": 10}' | jq -r '.links[].url'
# Search, through DuckDuckGo
curl -s localhost:3002/v2/search -H 'Content-Type: application/json' \
-d '{"query": "firecrawl self-host", "limit": 3}' | jq -r '.data.web[].url'
# A summary instead of the whole page (uses the model)
curl -s localhost:3002/v2/scrape -H 'Content-Type: application/json' \
-d '{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "formats": ["summary"]}' | jq -r .data.summary
For structured data, use a json format on /v2/scrape, as in the setup
check. Firecrawl marks the older /v2/extract endpoint deprecated, and its
replacement for many pages, /v2/agent, is Cloud-only.
Use it from your computer or another VM#
The API is also at https://my-firecrawl.exe.xyz:3002/, behind exe.dev's
proxy, which answers 401 to anything without a login or a
token for the VM. On your computer, create
a token into a file only you can read:
(umask 077; ssh exe.dev ssh-key generate-api-key --vm=my-firecrawl --label=firecrawl > ~/.firecrawl-vm-token)
Send it as X-Exedev-Authorization, which the proxy removes before the
request reaches Firecrawl:
curl -s https://my-firecrawl.exe.xyz:3002/v2/scrape \
-H "X-Exedev-Authorization: Bearer $(cat ~/.firecrawl-vm-token)" \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com", "formats": ["markdown"]}'
The Firecrawl SDKs, the CLI and the MCP server can't add a header of your
choosing, but they send their API key as Authorization: Bearer, which
exe.dev's proxy also accepts. So give them the VM token as the API key.
crawl.py from above, run on your computer:
export FIRECRAWL_API_URL=https://my-firecrawl.exe.xyz:3002
export FIRECRAWL_API_KEY=$(cat ~/.firecrawl-vm-token)
uv run https://exe-staging.dev/deploy/firecrawl/crawl.py https://books.toscrape.com/ 20
And Claude Code on your computer (with Node.js 22 or newer) gets the MCP server the same way:
claude mcp add --scope user firecrawl \
-e FIRECRAWL_API_URL=https://my-firecrawl.exe.xyz:3002 \
-e FIRECRAWL_API_KEY="$(cat ~/.firecrawl-vm-token)" \
-- npx -y firecrawl-mcp@3.41.0
The scrapes run on the VM, from its IP address, and an agent on your laptop gets Markdown back.
For an agent on another of your exe.dev VMs, keep the token off that VM
with a VM-to-VM integration: exe.dev adds
the token at the edge. On your computer (my-agent-vm is the VM that
runs the agent):
ssh exe.dev integrations add http-proxy --name firecrawl \
--target https://my-firecrawl.exe.xyz:3002/ --peer --attach vm:my-agent-vm
On my-agent-vm, set FIRECRAWL_API_URL=https://firecrawl.int.exe.xyz
and no API key.
Warning
Whoever can call the API can make this VM fetch any URL, and with authentication off, Firecrawl doesn't tell callers apart. Hand out VM tokens like you'd hand out a shell.
ssh exe.dev ssh-key listshows each one by its label, andssh exe.dev ssh-key removerevokes it.
What self-hosting leaves out#
The open-source stack is Firecrawl's core: fetch, Playwright, the queue and the API. Firecrawl's comparison lists what Cloud adds; in practice, at 2.11.809:
- No Fire-engine. Firecrawl's own scraping engine is closed and not
in the stack, so the
screenshotformat andactions(click, type, scroll, wait) fail withSCRAPE_ALL_ENGINES_FAILED, and there's no stealth mode or rotating proxies. Pages come from a plain HTTP fetch or the bundled Chromium, from the VM's IP address. You can route them through your own proxy withPROXY_SERVER,PROXY_USERNAMEandPROXY_PASSWORDin.env. - No Cloud products: Agent, Browser, Interact, monitors, the dashboard, API keys and usage tracking.
- A smaller map and search.
maphas no index of the web behind it, so a site without a sitemap returns few links. Search scrapes DuckDuckGo's HTML results, which can refuse a busy IP; setSEARXNG_ENDPOINTto a SearXNG instance for something sturdier. - No per-crawl politeness. Self-hosted jobs skip the concurrency
queue that enforces a crawl's
delayandmaxConcurrency: in testing, eight pages with"delay": 5took 13 seconds. Pace crawls withNUQ_WORKER_COUNTinstead (below).
Size and speed#
On a default VM (2 CPUs, 8 GB), measured with release 2.11.809:
| Measure | Result |
|---|---|
| Memory at rest | 3.3 GB, 2.4 GB of it the API and its five workers |
| Memory crawling 200 pages | 3.7 GB at the peak |
| Crawl of 200 pages of books.toscrape.com | 118 seconds |
| Ten scrapes at once in the browser | 7 seconds |
| Disk for the images | about 6 GB |
The knob that matters is NUQ_WORKER_COUNT in .env: each worker
process scrapes one page at a time and takes about 250 MB, so it's both
how many pages are in flight and most of the memory. With one worker, the
same VM crawled 30 pages in 42 seconds and the stack used 2.1 GB, which
is also the way to crawl gently. The browser opens at most 10 pages at
once (CRAWL_CONCURRENT_REQUESTS). Compose's NUM_WORKERS_PER_QUEUE,
MAX_CONCURRENT_JOBS and BROWSER_POOL_SIZE aren't read by the API at
this release. After changing .env, apply it with docker compose up -d.
The crawl above kept both CPUs busy. For more workers, make a bigger VM
(ssh exe.dev new --name=my-firecrawl --cpu=4 --memory=16GB, within your
plan's limits).
Scrape responsibly#
A self-hosted Firecrawl is fast, and the requests come from your VM.
Check a site's terms before you crawl it, keep crawls to what you need
with limit and includePaths, and keep NUQ_WORKER_COUNT low for
sites that aren't yours. Crawls honor robots.txt unless you pass
ignoreRobotsTxt; don't, on sites that ask not to be crawled. The
sites in these examples, books.toscrape.com and quotes.toscrape.com, are
sandboxes built for scraping practice.
Update#
Firecrawl tags a release several times a day. To move to a newer one,
read its compose file
for changes, then check out the tag, rebuild, and start over with fresh
containers (down drops the queue's contents, which a new release may
lay out differently, so wait for running crawls to finish). On the VM,
with the release you want in V:
cd ~/firecrawl
OLD=$(sed -n 's/^FIRECRAWL_VERSION=//p' .env)
V=2.11.809
git fetch --depth 1 origin tag v$V && git checkout -q v$V
sed -i "s/^FIRECRAWL_VERSION=.*/FIRECRAWL_VERSION=$V/" .env
docker compose build && docker compose pull api
docker compose down -v && docker compose up -d
[ "$OLD" = "$V" ] || docker rmi ghcr.io/firecrawl/firecrawl:$OLD firecrawl-playwright:$OLD firecrawl-nuq-postgres:$OLD
docker builder prune -f
.env and docker-compose.override.yaml aren't part of the checkout, so
they stay. Run the scrape from the setup to check. To go back, do the
same with the old version in V. To update the MCP server,
sudo npm install -g firecrawl-mcp@<version>.
There's nothing else to back up: .env is the only configuration, and
the results you keep are files like ~/crawls.
Troubleshooting#
range of CPUs is from 0.01 to 2.00when starting: the API's 4-CPU limit from Firecrawl's compose file is still there. Check thatdocker-compose.override.yamlis in~/firecrawlwith itscpus: !reset nullline.- Readiness never answers:
docker compose ps -ashows which container stopped;docker compose logs --tail=100 apisays why. The API waits for PostgreSQL and RabbitMQ to be healthy first, which takes about 20 seconds. - A scrape fails or times out:
docker compose logs --tail=200 api playwright-service.SCRAPE_ALL_ENGINES_FAILEDon a screenshot or actions means Fire-engine (see above); on an ordinary page, the site is blocking the VM or is down. - JSON or summary formats fail: check the model from the VM with
curl -s https://llm.int.exe.xyz/v1/models | jq -r '.data[].id' | grep gpt-5.4-mini; if nothing prints, the LLM integration isn't attached to the VM (LLM integration). Any model in that list can go inMODEL_NAME. 401frommy-firecrawl.exe.xyz:3002: the VM token is missing, wrong, or for another VM. Firecrawl itself never answers401with authentication off.- Nothing after a reboot:
systemctl is-enabled dockershould sayenabled; the containers come back with Docker.
Frequently asked questions#
Is self-hosted Firecrawl free? What does this cost?
Firecrawl's API is open source under AGPL-3.0 (the SDKs are MIT), and a self-hosted instance has no credits or page limits. You pay for the VM as part of your exe.dev plan. JSON extraction and summaries call a model through exe.dev's LLM integration, billed to your exe.dev account (models and prices); plain scrapes and crawls call no model.
What does self-hosted Firecrawl lack compared with Firecrawl Cloud?
Fire-engine, Firecrawl's closed scraping engine, isn't included, so there are no screenshots, no page actions (click, type, scroll), and none of its anti-bot handling or proxies; pages are fetched directly or with the bundled Playwright Chromium. Cloud products such as Agent, Browser, Interact and monitoring aren't included either, map only knows what a site's sitemap and links show, and search goes through DuckDuckGo unless you add SearXNG. Scrape, crawl, map, search and LLM extraction work.
Do I need an OpenAI API key for extraction?
No. OPENAI_BASE_URL points Firecrawl at exe.dev's LLM integration, which adds the credentials at the network edge, and MODEL_NAME picks the model (gpt-5.4-mini here). OPENAI_API_KEY holds a placeholder only because the OpenAI client wants a value.
Is the API authenticated?
Firecrawl's own authentication is off (USE_DB_AUTHENTICATION=false); turning it on needs Supabase. exe.dev's HTTPS proxy is the lock instead: port 3002 is reachable only by you in a browser and by programs that hold a token for the VM, and ports other than the VM's main one can't be made public. Anything running on the VM itself can call localhost:3002 freely.
How big a VM does Firecrawl need?
A default VM (2 CPUs, 8 GB) is enough. With five queue workers, the stack used about 3.3 GB at rest and 3.7 GB at its peak while crawling 200 pages in two minutes, and the images take about 6 GB of disk. For big crawls or many agents at once, make the VM with --cpu=4 --memory=16GB and raise NUQ_WORKER_COUNT.
Can I use the Firecrawl SDKs and CLI with it?
Yes. Set the API URL to http://localhost:3002 on the VM, or to https://my-firecrawl.exe.xyz:3002 elsewhere with a VM token as the API key: the SDKs send it as Authorization: Bearer, which exe.dev's proxy accepts, and Firecrawl ignores it.
Published Nov 21, 2026 · Updated Nov 21, 2026 · tested end to end Nov 21, 2026 · Markdown · manifest.json · Something wrong? Tell us