[Feature Request]: Docker egress proxy: opt-in hostname passthrough for public destinations (residential proxy providers require hostnames)

[sergeesteves]

What needs to be done?

Add an opt-in mode to the Docker server's egress pinning proxy so that, when
traffic is chained through an upstream proxy, the upstream receives
CONNECT <hostname>:<port> instead of CONNECT <pinned-ip>:<port> for public
destinations.

Today (0.9.3), every proxied request is an IP-only CONNECT. #2142 lists this as a
known limitation ("proxies that refuse CONNECT-to-an-IP are not supported yet")
and mentions a possible future opt-in hostname mode. This request is for that
mode, covering public-web crawling.

What problem does this solve?

Commercial residential proxy providers (e.g. Webshare) enforce trust & safety
rules that require the destination hostname on every request. My account was
suspended for "requests without hostname", and support asked me to reconfigure
the software to connect to hostnames instead of IP addresses.

With the current design this is impossible: the pinning proxy resolves locally
and always sends the pinned IP upstream. Commercial proxy providers with such
rules therefore cannot be used with the Docker server at all.

#2152 does not solve it either. It delegates hostnames only for an explicit list
of DNS suffixes, with wildcards rejected. That fits private networks with known
suffixes, but not public-web crawling, where target domains are arbitrary and
unknown in advance.

Target users/beneficiaries

• Docker server users who route crawls through commercial proxy providers
  (residential, ISP or datacenter) that require hostnames for compliance.
• Corporate deployments behind proxies with hostname/domain ACLs that refuse
  CONNECT to bare IPs (the limitation already listed in Docker: Chain egress proxy through upstream HTTP(S)_PROXY #2142).
• Anyone crawling arbitrary public sites through an upstream proxy, where a
  suffix allowlist (Feat/upstream proxy hostname passthrough #2152) cannot be maintained.

Current alternatives/workarounds

• An SNI-sniffing gateway between Crawl4AI and the provider (e.g. Xray-core with
  sniffing.destOverride: ["http","tls"]): it answers the CONNECT, reads the
  hostname from the TLS ClientHello (or the Host header for plain HTTP), then
  re-issues the CONNECT upstream with the hostname. It works, but it is an extra
  service to run and secure, it holds the proxy credentials, and upstream errors
  (e.g. 402/407) surface in Crawl4AI as closed connections instead of explicit
  failures.
• CRAWL4AI_ALLOW_INTERNAL_URLS=true: drops the entry-point check for every
  destination and still does not send hostnames upstream. Not a real option.
• Using the Python library directly with Playwright's own proxy settings: loses
  the Docker server and its security hardening.

Proposed approach

An opt-in env var on the upstream path only, e.g.
CRAWL4AI_UPSTREAM_PROXY_HOSTNAME_PASSTHROUGH=public:

1. Keep resolve_and_pin() as a validation step: resolve locally and reject
   non-global addresses, exactly as today.
2. If the target is global, send CONNECT <hostname>:<port> to the upstream
   instead of the pinned IP (and the absolute-form hostname for plain HTTP).
3. Restrict to ports 80/443 by default, as Feat/upstream proxy hostname passthrough #2152 does. IP literals keep the
   current pinned path. NO_PROXY behaviour is unchanged. The upstream is never
   bypassed and the target is never dialed directly.

Trust model: the connection is only ever handed to the operator-configured
upstream, so the crawler's own network stays unreachable even if a name
re-resolves differently on the upstream side. The remaining gap (the upstream
resolving to another address than the one validated) is the same deliberate
trust decision already accepted in #2152, but narrower: private destinations
are still rejected at validation time, and no suffix list is required.

It could also be a keyword of #2152's variable (e.g. accepting public as a
special value) rather than a separate variable, if maintainers prefer one knob.

Happy to test a branch.

[sergeesteves]

Update: instead of the SNI gateway mentioned above, I implemented the proposed
approach directly. It is a two-line change in deploy/docker/egress_proxy.py
(v0.9.3), on the upstream path only:

 # _handle_absolute(), upstream branch
-            out = f"{method} http://{_bracket(pin.ip)}:{port}{path} HTTP/1.1\r\n".encode("latin-1")
+            out = f"{method} http://{_bracket(pin.host)}:{port}{path} HTTP/1.1\r\n".encode("latin-1")
 # _dial()
-            dst = f"{_bracket(pin.ip)}:{port}"
+            dst = f"{_bracket(pin.host)}:{port}"

• resolve_and_pin() still runs first, so non-global targets are rejected
  exactly as before. The direct-dial paths (pin.ip) are untouched.
• For IP-literal URLs, pin.host is already the IP, so nothing changes there.
• Applied on top of the official 0.9.3 image with a small Dockerfile
  (USER root for the edit, then back to USER appuser).
• Tested against a logging upstream proxy: every request carried a hostname
  (CONNECT example.com:443, CONNECT en.wikipedia.org:443,
  GET http://neverssl.com:80/…), never an IP. Final check with the proxy
  provider is pending.

A proper contribution would add what this quick patch lacks: an opt-in env var
(default behaviour unchanged), the 80/443 port restriction, docs, and tests
alongside test_security_egress_proxy.py. Happy to open a PR if the approach
looks acceptable to maintainers.

[sergeesteves]

Follow-up: confirmed in production with the proxy provider. With the patch, their
activity log now shows destination hostnames instead of IP addresses, and the
account restriction has been lifted.