crt: request an explicit / path (blasthttp sends an empty path, which crt.sh rejects with 400)

[wandering-walnut]

On bbot 3.x, the crt module fails every query with Error parsing results for query "<domain>" (status code 400).

Cause: request_url builds its URL with add_get_params("https://crt.sh", params), which gives https://crt.sh?q=%25.<domain>&output=json. There's no path in that URL. blasthttp sends it exactly as given, so the request line is GET ?q=..., and crt.sh's nginx answers 400 Bad Request. httpx (bbot 2.x) normalised an empty path to /, which is why this only appeared with bbot 3's client.

With web.debug, bbot 3.0.2 logs blasthttp request: GET https://crt.sh?q=%25.<domain>&output=json, and the response is nginx's 400.

Reproduction, independent of bbot:

curl -s -o /dev/null -w '%{http_code}\n' --request-target '?q=%25.example.com&output=json' https://crt.sh    # 400
curl -s -o /dev/null -w '%{http_code}\n' --request-target '/?q=%25.example.com&output=json' https://crt.sh   # 200 (or crt.sh's usual transient 502)

Change:

• base_url = "https://crt.sh/" in bbot/modules/crt.py;
• the matching URL in test_module_crt.py's blasthttp mock.

Tested: with this change applied to 3.0.2, bbot -m crt against a live domain returns results again.

Possible follow-up: a more general fix would be for add_get_params or the blasthttp wrapper to normalise an empty path to /, since any other base_url without a path would hit the same thing.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[wandering-walnut]

I have read the CLA Document and I hereby sign the CLA

[liquidsec]

hi @wandering-walnut, good catch and thanks for the PR.

Lets remove the comment and leave the documentation of this for the commit messages, other than that this should be good to go.

I agree there should be more normalization to handle cases like this also, will open an issue for that.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (2889e12) to head (1723915).

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3428   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        459     459           
  Lines      48523   48523           
=====================================
+ Hits       43652   43653    +1     
+ Misses      4871    4870    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liquidsec]

also fixing at blacklanternsecurity/blasthttp#112