Sunday, September 13, 2026 5:18:44 PM

BigQuery ETL pipeline, what actually works in production?

Posted: one month ago
So we're running BigQuery on both ends of our pipeline here in the US and the ETL layer between them has become a real bottleneck. Anyone dealt with BigQuery to BigQuery specifically and found an approach that holds up under real production load?
Posted: one month ago
Incremental loading is where most setups fall apart in my experience. Full table scans every run kill performance and cost fast. Getting the incremental logic right from day one is what separates pipelines that scale from ones that become expensive problems.
Posted: one month ago
The incremental point is exactly where teams underestimate the complexity. BigQuery ETL between two instances looks simple until you're dealing with late arriving data, deduplication logic and upstream schema changes hitting at the same time. Purpose built tooling for this specific route handles those scenarios out of the box rather than requiring custom code every time something unexpected happens. Worth looking at what's designed for this pipeline before building something from scratch. More detail on how this works is here http://datrise.com/en/pipeline/bigquery-to-bigquery
Posted: 14 days ago
First produced for the Soviet Air Force in 1959, these cream dial chronographs were issued to cosmonauts by 1964 and worn by the first three-man crew on Voskhod 1 in October '64. The Strela “Arrow” chronographs were worn underneath the Berkut spacesuit during the very first spacewalk by Alexei Leonov in March 1965, and on Soyuz link spaceflight missions up to 1973.