Rythme: Scraping module for automation and optimization of daily purchases

Introduction

Rythme, a startup specialized in optimizing the daily product shopping experience, called on Terros to create a high-performance scraping solution. The objective of this project is to develop an application allowing individuals to automate their daily purchases by gathering products from different supermarkets (GMS) and offering the possibility to compare prices. Terros is responsible for designing an autonomous scraping module, capable of building and regularly updating Rythme's product database.

Solution and project development

Rythme requested Terros to develop a custom scraping module capable of gathering and updating product data from partner supermarkets. The project presents several technical challenges, including source diversity (different GMS) and the need to maintain data completeness while ensuring regular updates.

The solution proposed by Terros is based on two types of scraping:

Catalog scraping: This module browses all catalog pages of platforms to retrieve and update product data.

Unit scraping: This module intervenes as a complement to update products that were not retrieved by the Catalog Scraper, processing product URLs individually.

Results and Feedback

The deployment of the scraping module allowed Rythme to effectively automate the constitution and updating of its product database, ensuring completeness and reliability of information. The main results include:

• Implementation of a complete database: Thanks to scraping, Rythme has an updated product base, including key information (URL, EAN, price...) from several supermarkets.

• Cost and time reduction: The optimization of reusable components has limited maintenance and development times, which had a positive impact on budgets and deployment speed.

• Real-time price comparison: End users of the Rythme application can now easily view and compare product prices, offering a better shopping experience for daily products.

• Update reliability: The dual approach (Catalog and unit scraping) ensures regular updating of product information, thus ensuring high data reliability for the Rythme application.