So we're running BigQuery on both ends of our pipeline here in the US and the ETL layer between them has become a real bottleneck. Anyone dealt with BigQuery to BigQuery specifically and found an approach that holds up under real production load?
Thursday, July 30, 2026 11:10:57 AM
BigQuery ETL pipeline, what actually works in production?
The incremental point is exactly where teams underestimate the complexity. BigQuery ETL between two instances looks simple until you're dealing with late arriving data, deduplication logic and upstream schema changes hitting at the same time. Purpose built tooling for this specific route handles those scenarios out of the box rather than requiring custom code every time something unexpected happens. Worth looking at what's designed for this pipeline before building something from scratch. More detail on how this works is here http://datrise.com/en/pipeline/bigquery-to-bigquery