Proxy Client

proxy_client

Synchronous HTTP client for Scrape.do's Proxy Mode.

Defines ScrapeDoProxyClient, the proxy-mode counterpart of ScrapeDoClient. Configures httpx.Client instances with Scrape.do's Proxy Mode endpoint (proxy.scrape.do:8080) and reuses the same retry / hook / validator semantics as the API-mode client.

Because Scrape.do encodes per-request parameters into the proxy URL's password field, each unique (token, params) combination needs its own httpx.Client. This module maintains a bounded LRU pool of clients keyed on the formatted proxy URL — repeated requests with the same params reuse the same TCP / TLS / HTTP-2 connection state.

ScrapeDoProxyClient

Synchronous HTTP client for Scrape.do's Proxy Mode.

Counterpart of ScrapeDoClient that routes requests through proxy.scrape.do:8080 instead of calling api.scrape.do directly. Reuses the same RequestParameters model, the same retry strategy, the same event hooks (SyncClientEventHooks), and the same session-validation contract (SyncSessionValidator).

Features

• Local API parameter validation via the RequestParameters Pydantic model, plus the proxy-mode-specific cross-checks in validate_proxy_params.

• Status code error parsing and customisable retry intervals for rate-limited requests.

• Strongly-typed interface for responses via the ScrapeDoResponse Pydantic model.

• Connection reuse via a bounded LRU pool of httpx.Client instances keyed on the formatted proxy URL.

Concurrency Limit and Server Errors

This client intercepts and manages Scrape.do's specific gateway errors (429, 502, 510), automatically applying a customisable retry strategy before the error can reach the application.

SDK Event Hooks (event_hooks)

This client reuses the synchronous SyncClientEventHooks TypedDict — same shape, same lifecycle, same callback signatures as the API-mode client.

TLS Verification

• Scrape.do's Proxy Mode upgrades and forwards HTTPS requests on your behalf (MITM-style), so HTTPS target validation against the normal system CAs would fail

• The default verify value is DEFAULT_PROXY_SSL_CONTEXT, an ssl.SSLContext preloaded with system CAs + Scrape.do's bundled CA, so HTTPS targets validate correctly through the proxy

• Pass verify=True if you've installed Scrape.do's CA into your OS keychain, or verify=False to disable validation entirely (discouraged).

Additional httpx.Client Configuration

The following httpx.Client parameters can be provided as keyword arguments and will be passed directly to every pooled client.

• verify
• cert
• http1
• http2
• timeout
• limits
• transport
• default_encoding

Additionally, the following httpx.Client.request parameters can be provided as keyword arguments during request execution.

• timeout (r_timeout)
• extensions

For more information on their behaviour and default values, please consult the official httpx documentation.

Unsupported HTTPX Client Arguments

The underlying httpx.Client instances are strictly managed by the pool to prevent invalid configurations from being sent to Scrape.do. For this reason, arguments not listed in the previous section are intentionally blocked and shouldn't be changed.

Connection Pool

• Each unique formatted proxy URL gets its own httpx.Client

• Two requests with the same RequestParameters reuse the same pooled client (and therefore the same TCP / TLS / HTTP-2 state) for transport-level efficiency.

• Two requests with different parameters get different clients

• When max_pooled_clients is exceeded, the least-recently-used client is closed.

• Cookies are not preserved across requests on the pooled client - the jar is cleared after every call. Scrape.do owns the cookie lifecycle through setCookies (in), scrape.do-cookies / pureCookies=true (out), and sessionId (server-side session jars). Pooling is purely a transport concern.

Parameters:

Name	Type	Description	Default
api_token	Optional[str]	The Scrape.do API key. If omitted, the client will attempt to load it from the 'SCRAPE_DO_API_KEY' environment variable.	None
max_retries	int	The maximum number of retry attempts for retryable Scrape.do gateway errors (HTTP 429, 502, and 510).	3
retry_backoff	Union[float, Callable[[int], float]]	The strategy used to calculate the delay between retries. Can be a static float (seconds) or a callable that accepts the current attempt number (0-indexed) and returns a float. Defaults to a jittered exponential backoff when set to None.	None
event_hooks	Optional[SyncClientEventHooks]	A dictionary of SDK-native hooks to execute during different points of the request lifecycle.	None
max_pooled_clients	int	Maximum number of httpx.Client instances to keep alive in the LRU pool. Defaults to 16.	16
verify	Union[SSLContext, str, bool]	SSL verification configuration. Defaults to DEFAULT_PROXY_SSL_CONTEXT which trusts both system CAs and Scrape.do's bundled CA.	DEFAULT_PROXY_SSL_CONTEXT
cert	Optional[CertTypes]	Client-side certificates for mutual TLS authentication.	None
http1	bool	Enable HTTP/1.1 support.	True
http2	bool	Enable HTTP/2 multiplexing for higher concurrency.	False
timeout	TimeoutTypes	The default timeout (in seconds) applied to all network phases. Defaults to 60s.	60.0
limits	Limits	Configuration for maximum connection pool sizes within each pooled httpx.Client.	DEFAULT_LIMITS
transport	Optional[BaseTransport]	A completely custom transport engine.	None
default_encoding	Union[str, Callable[[bytes], str]]	The fallback text encoding used if a target website omits a charset header.	'utf-8'

_get_or_create_client(proxy_url)

Returns a pooled httpx.Client for the given proxy URL.

Pool Behavior

Cache hit

• Bumps the entry to the back of the LRU ordering (most recently used) and returns the existing client.

Cache miss + pool full

• evicts the least-recently-used entry, closes it, then constructs a new client.

Cache miss + pool has room

• constructs a new client and stores it.

Parameters:

Name	Type	Description	Default
proxy_url	str	The formatted proxy URL (with api_token inserted and parameters URL-encoded into the password field) returned by RequestParameters.to_proxy_url().format(...).	required

Returns:

Type	Description
Client	A pooled httpx.Client configured for the given proxy URL.

close()

Closes every httpx.Client in the pool and clears it.

It is recommended to use the client as a context manager to ensure resources are released automatically.

__enter__()

Context manager entry. Returns the ScrapeDoProxyClient instance. The pool starts empty.

Returns:

Type	Description
Self	The ScrapeDoProxyClient instance.

__exit__(exc_type, exc_val, exc_tb)

Closes every pooled httpx.Client without swallowing exceptions.

Parameters:

Name	Type	Description	Default
exc_type	Optional[type[BaseException]]	The type of the exception.	required
exc_val	Optional[BaseException]	The instance of the exception.	required
exc_tb	Optional[TracebackType]	The traceback information.	required

Returns:

Type	Description
Literal[False]	False, since no exceptions are swallowed.

execute(request, session_validator=None, *, r_timeout=USE_CLIENT_DEFAULT, extensions=None)

Executes a fully prepared and validated Scrape.do request through Proxy Mode.

Intended Usage

Use this method if you have manually constructed a PreparedScrapeDoRequest object for bulk routing, custom configurations, or task reusability.

Sessions (sessionId)

If you configure a request with a session_id, Scrape.do will attempt to route your traffic through the same proxy address. However, it can still silently rotate this address for various reasons. If it rotates during a multi-step scraping task, any target-specific WAF state or cookies accumulated will be lost, which may cause the task to fail.

Validating Sessions (session_validator)

• In order to prevent unexpected errors due to dropped sessions, you can pass a custom function to the client's execute method session_validator argument.

• This function will be called internally by the client after each stateful request (sessionId is not None) to determine whether or not a RotatedSessionError exception should be raised to signal that this session is no longer valid.

• The function should take the current request's ScrapeDoResponse object as its only argument, and return a single bool value.

• If the function evaluates to True, this method will raise the RotatedSessionError instead of returning the response object. (The request's ScrapeDoResponse object can still be accessed later on using the exception's response attribute.) Otherwise, no additional action is taken.

Parameters:

Name	Type	Description	Default
request	PreparedScrapeDoRequest	The validated request payload.	required
session_validator	Optional[SyncSessionValidator]	A custom function called to determine whether or not to raise a RotatedSessionError exception.	None
r_timeout	Union[TimeoutTypes, UseClientDefault]	A request-specific timeout override.	USE_CLIENT_DEFAULT
extensions	Optional[RequestExtensions]	Advanced HTTPX extensions for this specific request.	None

Returns:

Type	Description
ScrapeDoResponse	The ScrapeDoResponse object containing the target's data.

Raises:

Type	Description
APIConnectionError	If the underlying network transport drops entirely (e.g., DNS failure).
RotatedSessionError	If a session_validator is provided, the request was made with a session_id argument, and the session_validator returned True.
ValueError	If the request.api_params RequestParameters instance contains an invalid parameter configuration for Proxy Mode

request(method, target_url, params=None, session_validator=None, *, headers=None, body=None, payload_type='json', r_timeout=USE_CLIENT_DEFAULT, extensions=None, **api_kwargs)

Builds and executes a Scrape.do request through Proxy Mode.

Parameter Configuration

Like ScrapeDoClient.request, this method provides smart routing based on the arguments provided. You can configure the request in two distinct ways:

• Keyword Arguments (Default) : Pass the target URL and Scrape.do parameters directly as **api_kwargs (render=False, geoCode="us").

• Pre-built Parameters : Pass a fully validated RequestParameters object via the params argument.

Raw api.scrape.do URLs Are Not Accepted

• Unlike the API-mode client, proxy mode has no equivalent of a raw api.scrape.do/?... URL.

• Passing one as target_url simply targets that URL through the Scrape.do proxy, which is almost certainly not what you want.

Parameter Restrictions

To prevent silent overwrites, the client enforces that only one of the parameter configurations can be used at a time. Mixing params with **api_kwargs raises ValueError.

Pre-built Parameters Configuration

When passing an already constructed RequestParameters instance to the params argument, its url attribute will be ignored and replaced by the provided target_url.

Parameters:

Name	Type	Description	Default
method	HttpMethod	The HTTP method to forward to the target website.	required
target_url	str	The destination website URL.	required
params	Optional[RequestParameters]	A pre-validated parameter object.	None
session_validator	Optional[SyncSessionValidator]	A custom function called to determine whether or not to raise a RotatedSessionError exception. See ScrapeDoProxyClient.execute docstring for more information	None
headers	Optional[Dict[str, str]]	Custom HTTP headers to forward to the target.	None
body	Optional[Union[Dict[str, Any], str, bytes]]	The payload to send to the target website.	None
payload_type	PayloadType	Dictates how the client encodes the body.	'json'
r_timeout	Union[TimeoutTypes, UseClientDefault]	Request-specific timeout override.	USE_CLIENT_DEFAULT
extensions	Optional[RequestExtensions]	Advanced HTTPX extensions.	None
**api_kwargs	Unpack[RequestParametersDict]	Scrape.do API configuration parameters (e.g., render=False).	{}

Returns:

Type	Description
ScrapeDoResponse	The ScrapeDoResponse object containing the target's data.

Raises:

Type	Description
ValueError	If configuration constraints are violated
APIConnectionError	If the underlying network transport drops entirely (e.g., DNS failure).
RotatedSessionError	If a session_validator is provided, the request was made with a session_id argument, and the session_validator returned True.

get(url, params=None, session_validator=None, *, headers=None, r_timeout=USE_CLIENT_DEFAULT, extensions=None, **api_kwargs)

Wrapper for executing a GET request.

Inherits the smart routing logic, parameter validation, and execution constraints of the base request method.

Parameters:

Name	Type	Description	Default
url	str	The target website URL.	required
params	Optional[RequestParameters]	A pre-validated parameter object.	None
session_validator	Optional[SyncSessionValidator]	A custom function to be called in order to determine whether or not to raise a RotatedSessionError exception. (See ScrapeDoProxyClient.execute docstring for more information)	None
headers	Optional[Dict[str, str]]	Custom HTTP headers to forward.	None
r_timeout	Union[TimeoutTypes, UseClientDefault]	Request-specific timeout override.	USE_CLIENT_DEFAULT
extensions	Optional[RequestExtensions]	Advanced HTTPX extensions.	None
**api_kwargs	Unpack[RequestParametersDict]	Scrape.do API configuration parameters.	{}

Raises:

Type	Description
ValueError	If configuration constraints are violated.
APIConnectionError	If the underlying network transport drops entirely (e.g., DNS failure).
RotatedSessionError	If a session_validator is provided, the request was made with a session_id argument, and the session_validator returned True

Returns:

Type	Description
ScrapeDoResponse	The ScrapeDoResponse object containing the target's data.

post(url, params=None, session_validator=None, *, body=None, headers=None, payload_type='json', r_timeout=USE_CLIENT_DEFAULT, extensions=None, **api_kwargs)

Wrapper for executing a POST request.

Inherits the smart routing logic, parameter validation, and execution constraints of the base request method.

Parameters:

Name	Type	Description	Default
url	str	The target website URL.	required
params	Optional[RequestParameters]	A pre-validated parameter object.	None
session_validator	Optional[SyncSessionValidator]	A custom function to be called in order to determine whether or not to raise a RotatedSessionError exception. (See ScrapeDoProxyClient.execute docstring for more information)	None
body	Optional[Union[Dict[str, Any], str, bytes]]	The payload to send to the target website.	None
headers	Optional[Dict[str, str]]	Custom HTTP headers to forward.	None
payload_type	PayloadType	Dictates how the client encodes the body.	'json'
r_timeout	Union[TimeoutTypes, UseClientDefault]	Request-specific timeout override.	USE_CLIENT_DEFAULT
extensions	Optional[RequestExtensions]	Advanced HTTPX extensions.	None
**api_kwargs	Unpack[RequestParametersDict]	Scrape.do API configuration parameters.	{}

Raises:

Type	Description
ValueError	If configuration constraints are violated.
APIConnectionError	If the underlying network transport drops entirely (e.g., DNS failure).
RotatedSessionError	If a session_validator is provided, the request was made with a session_id argument, and the session_validator returned True

Returns:

Type	Description
ScrapeDoResponse	The ScrapeDoResponse object containing the target's data.