The Challenge
What DATwise Was Facing
DATwise needed to deliver BI dashboards to clients whose data lived across multiple operational databases — PostgreSQL, MySQL, Oracle, and legacy SQL Server instances — with no existing ETL infrastructure. Direct querying of production databases was not acceptable due to performance risk. Every client had a different schema, different data quality standards, and different refresh requirements, making a generic connector approach unworkable.
The Solution
What We Built
We built a client-configurable integration layer with schema-aware connectors for each supported database engine, using read-replica routing to eliminate load on production systems. A lightweight ETL pipeline handled per-client schema mapping, data quality validation, and incremental sync — transferring only changed records rather than full dumps. A metadata registry tracked column lineage so that BI dashboards could display data provenance information alongside each metric, giving analysts confidence in what they were looking at.
Results