Skip to content

Contributing

Thanks for your interest in scrape-do-python.

This page covers local setup, the test/lint pipeline, the docs build, and PR conventions.

Code of Conduct

By participating in this project you agree to abide by the Code of Conduct

Local Setup

git clone https://github.com/svdC1/scrape-do-python.git
cd scrape-do-python
pip install -e .[dev]
Python Versions
  • Requires Python 3.9+
  • CI runs on 3.13

Running Tests

  • Runs on Every Push and PR

  • Uses respx to mock HTTP calls

  • They're fast and don't need any external dependencies

pytest tests/unit/ --cov
Single-Test Invocation 
pytest tests/unit/test_client.py::TestName -k pattern
  • Live tests against the real Scrape.do API.
pytest tests/integration/ --cov
Requires SCRAPE_DO_API_KEY

  • Integration tests are skipped if SCRAPE_DO_API_KEY is not set in the environment

  • CI provides it via repo secrets

  • For local runs, make sure to export it first

Test target

  • Defaults to https://httpbin.co (Scrape.do's documented playground target)

  • Override by setting the HTTPBIN_BASE environemnt variable to a different URL

PRs
  • Both the unit and integration suites must pass for a PR to be merged
  • CI runs both on every PR

Lint & Type-Check

ruff check .
mypy src/
CI Gate
  • Both ruff and mypy must pass.
  • This project aims to ship strict typing, so try not to # type: ignore unless you've exhausted alternatives

Documentation

pip install -e .[docs]
mkdocs serve

The docs site is built with mkdocs-material and auto-generates the API reference from docstrings via mkdocstrings

Keep The API Reference Updated

Pull Request Guidelines

  • One concern per PR

  • Refactors and feature additions belong in separate PRs from bug fixes

  • Add unit tests for any new SDK behaviour or bug fix

  • Integration tests are reserved for changes that interact with Scrape.do's actual gateway behavior

  • Update CHANGELOG under a ## [Unreleased] section (create the section if it doesn't exist)

  • One bullet under ### Added / ### Changed / ### Fixed / ### Removed as appropriate

  • Update docstrings to keep this site's documentation updated

CHANGELOG Entries

  • Pre-1.0, the CHANGELOG is the only source of truth for "what changed and why" between releases

  • PR titles and commit messages are searchable but not consolidated

  • CHANGELOG is meant to provide users with relevant information about changes so that they can decide whether or not to upgrade

Reporting Bugs / Requesting Features

Use Templates
Security Issues

  • Don't open a public issue for security issues

  • See the Security Policy for private reporting channels.