Skip to content

Overview

plugins

Scrape.do Async-API Plugin Adapters

This sub-package defines pydantic models for the optional Plugin key that can be sent in the Async Job Creation JSON payload in order to create asynchronous tasks that use the Scrape.do's Plugins endpoints

What Do These Models Represent

  • Each one of the *AsyncPlugin models defined here represents a single Async API Job using the Plugins key

  • When creating an Async API Job that uses a Scrape.do Plugin, all of its tasks must be requests to the specified Scrape.do Plugin that will be executed asynchronously. Therefore, no Targets can be specified for a plugin job, which is why Targets and Plugin are mutually exclusive keys when creating an Async API Job

Constraints

  • Each *AsyncPlugin model enforces min_length=1, max_length=1000 on its params list

  • The cap matches Scrape.do's documented per-job maximum of 1000 tasks for plugin-jobs

  • Each *AsyncPlugin model has populate_by_name=True and extra="ignore" so additions on the server side don't break parsing

key + params Attributes

  • Every *AsyncPlugin model contains the key and params attributes

  • The key attribute is a literal string representing the name of the Scrape.do Plugin that the Async API Job will use. (Only one plugin can be used per Async API Job)

  • The params attribute is a list of Plugin-Specific Parameters that define a single task in that Async API Job (i.e, One request to the Scrape.do Plugin specified by the key)

Difference From Sync Plugins

While the Plugin-Specific Parameters accepted by the params attribute of the *AsyncPlugin models are the same pydantic *Parameters models used by the Sync Plugins, the following distinctions regarding the synchronous vs asynchronous usage of Scrape.do Plugins must be considered

  • Only a subset of the synchronous Scrape.do Plugins can be used asynchronously through the Scrape.do Async API

  • Some parameters that are accepted by the synchronous version of a plugin might be silently dropped or restricted by the server while using the Async API version of that plugin

  • Each *AsyncPlugin model known to have this behaviour will contain a warning with more information about what to keep in mind while using it

Async-Only Plugins

The walmart/store, lowes/store, shein, trip/search, and trip/detail plugins are only available through the Async API and do not have dedicated synchronous endpoints, which is why they don't exist in the Sync Plugins Sub-Package

Official Documentation

The structure of these models and all information contained here comes from manual testing and the official Scrape.do Async API Plugins, Scrape.do Amazon Plugin, Scrape.do Google Plugin, and Scrape.do ChatGPT Plugin documentation pages

AsyncPlugin module-attribute 

AsyncPlugin = Annotated[
    Union[
        AmazonPdpAsyncPlugin,
        AmazonSearchAsyncPlugin,
        AmazonOfferListingAsyncPlugin,
        ChatGPTChatAsyncPlugin,
        GoogleSearchAsyncPlugin,
        GoogleSearchAiModeAsyncPlugin,
        GoogleSearchAiOverviewAsyncPlugin,
        GoogleMapsSearchAsyncPlugin,
        GoogleMapsPlaceAsyncPlugin,
        GoogleMapsReviewsAsyncPlugin,
        GoogleShoppingAsyncPlugin,
        GoogleShoppingProductAsyncPlugin,
        GoogleShoppingProductStoresAsyncPlugin,
        GoogleFlightsAsyncPlugin,
        GoogleHotelsAsyncPlugin,
        GoogleHotelsDetailAsyncPlugin,
        GoogleNewsAsyncPlugin,
        GoogleTrendsAsyncPlugin,
        GoogleTrendingAsyncPlugin,
        GoogleYouTubeAsyncPlugin,
        GooglePlayStoreAsyncPlugin,
        GooglePlayStoreProductAsyncPlugin,
        GooglePlayStoreReviewsAsyncPlugin,
        WalmartStoreAsyncPlugin,
        LowesStoreAsyncPlugin,
        SheinAsyncPlugin,
        TripSearchAsyncPlugin,
        TripDetailAsyncPlugin,
    ],
    Field(discriminator="key"),
]

Discriminated union of every supported Async API Plugin adapter

Additional Information
New Plugins

  • If a new plugin lands server-side, its domain *Parameters model will be added in the Sync Plugins Sub-Package

  • After that, a new *AsyncPlugin adapter will be added to a new module in this sub-package and appended to the Union[...] above so that pydantic can pick up the new discriminator value