Skip to content

Contributing

Thanks for your interest in scrape-do-python.

This page covers local setup, the test/lint pipeline, the docs build, and PR conventions.

Code of Conduct

By participating in this project you agree to abide by the Code of Conduct

Local Setup

git clone https://github.com/svdC1/scrape-do-python.git
cd scrape-do-python
pip install -e .[dev]
Python Versions
  • Requires Python 3.9+
  • Unit tests run on a matrix of 3.9 / 3.10 / 3.11 / 3.12 / 3.13
  • Lint, type-checking, and integration tests run on 3.13

Running Tests

  • Runs on Every Push and PR

  • Uses respx to mock HTTP calls

  • Fast and don't need any external dependencies

pytest tests/unit/ --cov

  • Runs on Every PR and Release

  • Uses a real API token to run live tests against Scrape.do

  • Much slower than unit tests

set SCRAPE_DO_API_KEY=***
pytest tests/integration/ --cov

export SCRAPE_DO_API_KEY="***"
pytest tests/integration/ --cov
Logs

  • All integration test runs are logged according to the Conftest Logging Setup

  • Log files are saved at scrape_do/logs/integration_{timestamp}.log

SCRAPE_DO_API_KEY

The entire suite is SKIPPED when SCRAPE_DO_API_KEY is not set as an environment variable

Test Target
PRs
  • Both the unit and integration suites must pass for a PR to be merged
  • CI runs both on every PR
Single-Test Invocation 
pytest tests/unit/test_client.py::TestName -k pattern

Lint & Type-Check

ruff check .
mypy src/
CI Gate
  • Both ruff and mypy must pass.
  • This project aims to ship strict typing, so try not to # type: ignore unless you've exhausted alternatives

Documentation

pip install -e .[docs]
mkdocs serve

The docs site is built with mkdocs-material and auto-generates the API reference from docstrings via mkdocstrings

Keep The API Reference Updated

Pull Request Guidelines

  • One concern per PR

  • Refactors and feature additions belong in separate PRs from bug fixes

  • Add unit tests for any new SDK behaviour or bug fix

  • Integration tests are reserved for changes that interact with Scrape.do's actual gateway behavior

  • Update CHANGELOG under a ## [Unreleased] section (create the section if it doesn't exist)

  • One bullet under ### Added / ### Changed / ### Fixed / ### Removed as appropriate

  • Update docstrings to keep this site's documentation updated

CHANGELOG Entries

  • Pre-1.0, the CHANGELOG is the only source of truth for "what changed and why" between releases

  • PR titles and commit messages are searchable but not consolidated

  • CHANGELOG is meant to provide users with relevant information about changes so that they can decide whether or not to upgrade

Reporting Bugs / Requesting Features

Use Templates
Security Issues

  • Don't open a public issue for security issues

  • See the Security Policy for private reporting channels.