Roadmap

This Document Might Change

Items may be reordered or rescoped based on user feedback and design discoveries.

---

0.2 — Async + Proxy Mode

Status → Shipped

---

0.3 — Scrape.do Async API

Status → Shipped

---

0.4 — Google Plugin

Status → Planned

• Sub-package wrapping Scrape.do's Google Scraper API with new data models specific to search/results.

---

0.5 — Amazon Plugin

Status → Planned

• Sub-package wrapping Scrape.do's Amazon Scraper API with new data models specific to product/listing data.

---

1.0 — Surface Freeze

Status → Planned

Stability Commitment

• Stabilize the public API across sync, async, proxy, async-API, and plugin namespaces

• Post-1.0, breaking changes follow strict Semantic Versioning

---

Planned Package Layout

Speculative

• A starting point, not a commitment

• Each milestone may surface design constraints that justify deviation

• Version slots above are firmer than the file paths below

File Structure

src/scrape_do/
│
├─ __init__.py  # (1)!
├─ py.typed  # (2)!
├─ exceptions.py  # (3)!
├─ constants.py
├─ abc.py
│
├─ data/
│  │
│  ├─ __init__.py
│  └─ scrapedo_ca.crt # (18)!
│
│
│ # (4)!
├─ client.py  # (5)!
├─ async_client.py # (6)!
├─ proxy_client.py # (7)!
├─ async_proxy_client.py # (8)!
├─ models/ # (9)!
│  │
│  ├─ __init__.py
│  ├─ browser_actions.py
│  ├─ enums.py
│  ├─ parameters.py
│  ├─ request.py
│  ├─ response.py
│  └─ validators.py
│   
│
├─ async_api/ # (10)!
│  │
│  ├─ __init__.py
│  ├─ client.py
│  ├─ async_client.py
│  ├─ exceptions.py # (12)!
│  └─ models/ # (11)!
│     │
│     ├─ __init__.py
│     ├─ enums.py
│     ├─ parameters.py
│     ├─ response.py
│     └─ plugins/
│        │
│        ├─ __init__.py
│        ├─ additional.py
│        ├─ amazon.py
│        └─ google.py 
│
│
└─ plugins/ # (13)!
   │
   ├─ __init__.py
   ├─ google/ # (14)!
   │  │
   │  ├─ __init__.py
   │  ├─ client.py
   │  ├─ async_client.py
   │  └─ models/ # (15)!
   │     │
   │     ├─ __init__.py
   │     ├─ enums.py
   │     ├─ localization.py
   │     └─ parameters.py
   │
   │
   └─ amazon/ # (16)!
      │    
      ├─ __init__.py 
      ├─ client.py
      ├─ async_client.py
      └─ models/ # (17)!
         │
         ├─ __init__.py
         └─ parameters.py

1. Curated Public Re-Exports
2. PEP 561 Marker
3. Base Hierarchy (sub-packages may extend)
4. 0.1 + 0.2 - api.scrape.do + proxy.scrape.do
5. ScrapeDoClient (sync, api.scrape.do) — 0.1
6. AsyncScrapeDoClient — 0.2
7. ScrapeDoProxyClient (proxy.scrape.do) — 0.2
8. AsyncScrapeDoProxyClient — 0.2
9. Request / Response models for the four above
10. 0.3 - q.scrape.do — Server-Side Job Queue
11. job_id, polling, results, ...
12. AsyncAPI-Specific
13. 0.4 + 0.5 - Each plugin is a sub-package
14. 0.4
15. Endpoint-Specific
16. 0.5
17. Endpoint-Specific
18. Scrape.do certificate for Proxy Mode

---

Suggestions Are Welcome

Influence The Roadmap

• If a feature you need isn't here, open a Feature Request

• The roadmap reorders based on what real users need