The Challenge
What PriceZ Was Facing
PriceZ aggregated pricing data from over 40 retailer sources — a mix of official APIs, third-party data feeds, and web scrape pipelines. Each source had different update frequencies, authentication schemes, and data structures. Prices from different sources for the same product frequently conflicted, with no systematic way to determine which was authoritative. End users were seeing stale or contradictory prices, which undermined the core value proposition of the platform.
The Solution
What We Built
We rebuilt the ingestion layer as a distributed pipeline with per-source adapters handling rate limiting, authentication refresh, and format normalisation. A conflict resolution engine was built to apply source-priority rules and staleness weighting when multiple sources disagreed on a price — with transparent audit logs showing which source each displayed price came from and when it was last verified. A scheduling system managed crawl frequency per source based on observed price volatility, reducing unnecessary load while improving freshness for high-change products.
Results