Batch Data Scraping (Web) ​

Function Description ​

TIP

Scrape batch data from web pages, commonly used for scraping list and detail pages, and can scrape multiple pages of data by setting pagination buttons

Batch Data Scraping (Web)

Configuration Item Description ​

General ​

Command Input

  • Web Page ObjectTBrowser: Input a web page object that is obtained or created by the 'Open Web Page' function

  • Operation TargetTTarget: The table element to be scraped

  • Number of Pages to ExtractInteger: Number of Pages to Extract

  • Next Page ButtonTTarget: Pagination element

  • Number of PagesInteger: The maximum number of pages to scrape

  • Save to Data TableBoolean: Whether to save the results to the data table

  • Preview in Data TableBoolean: Preview in Data Table

Command Output

  • Save Batch Data ToTDataTable: Save the batch data obtained from the web page

Advanced ​

  • Custom Column Namesstring: You can customize the column names, separated by commas.

  • Page Interval (milliseconds)Integer: Time interval between page turns

  • Simulate Manual Click on Pagination ButtonBoolean: Simulate a manual click event

  • Send Window MessagesBoolean: Send window messages to click

  • Automatically Scroll to Bottom of PageBoolean: Automatically scroll to load the page before scraping, can be cancelled

  • Simulate Pause During Page TurningBoolean: Simulate a real person's pause during page turning

  • Minimum Pause Duration (milliseconds)Integer: The shortest waiting time after execution is complete

  • Maximum Pause Duration (milliseconds)Integer: The longest waiting time after execution is complete

  • Delay Before (ms)Integer: The waiting time before instruction execution

  • TimeOutInteger:

Error Handling ​

  • Print Error LogsBoolean: Whether to print error logs to the "Logs" panel when the command fails. Default is checked.

  • Handling MethodInteger:

    • Terminate Process: If the command fails, terminate the process.

    • Ignore Exception and Continue Execution: If the command fails, ignore the exception and continue the process.

    • Retry This Command: If the command fails, retry the command a specified number of times with a specified interval between retries.

Usage Example ​

Process logic description:

Common Errors and Handling ​

None

Frequently Asked Questions ​

None