Finding the costs of extractig a page

[JonasPapinigis]

I'm looking to use this library on a relatively large scale seeded extraction. I wanted to know there is a way to expose the amount of tokens used in a single-(or even multi-)page extraction/crawl? I know I get the CrawlResult for each page, but I don't see any input/output token consumption metrics anywhere (possibly in metadata?).

My inference provider does not have a way to view EXACT costs, but I want to be able to make something like this for myself: https://github.com/orkunkinay/openai_cost_calculator

Thanks for any help!

[hafezparast]

We just addressed this — PR #1874 adds a token_usage field to CrawlResult.

To try it now before it's merged:

pip install git+https://github.com/hafezparast/crawl4ai.git@fix/token-usage-in-crawl-result-1745

Then:

result = await crawler.arun(url, config=config)
print(result.token_usage)
# {'prompt_tokens': 1234, 'completion_tokens': 567, 'total_tokens': 1801}

If you prefer to stay on the current release, token usage is already tracked on the strategy object — it's just not exposed in CrawlResult yet:

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o", api_token="..."),
    instruction="...",
)
config = CrawlerRunConfig(extraction_strategy=strategy)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url, config=config)

# Token usage is on the strategy object
print(strategy.total_usage)           # Accumulated across all chunks
print(strategy.usages)                # Per-chunk breakdown
strategy.show_usage()                 # Pretty-printed summary

For cost calculation, multiply prompt_tokens and completion_tokens by your provider's per-token pricing.

[JonasPapinigis]

What a coincidence! Just today I went back to working at the extraction side of our pipeline and began exploring improvements/library alternatives for it. Ended up settling on Crawl4AI as the de-facto winner again.

Great to hear development is still very active, and very excited to continue integrating your guys' amazing project.

All the best.

[JonasPapinigis]

And of course, great fix

[domondi1]

@JonasPapinigis for a large seeded extraction, result.token_usage (from #1874) gives you counts per page. If you also want dollars and a hard stop per job, LLMConfig(base_url=...) plus extra_args={"extra_headers": {...}} is enough to route the extraction calls through a local gateway that prices each call and refuses one with HTTP 402 before it reaches the provider once the job's budget is used. Then inferrail work <job-id> prints calls, tokens and dollars for that job.

I wrote up the exact setup and what I tested here: #2342. Disclosure: I maintain that gateway (Inferrail, open source). For a provider without a built-in price you add one in the gateway config, so the dollars are your provider's real rates, not an estimate.