Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
145 views
in Technique[技术] by (71.8m points)

Setup 2 youtube bigquery transfer (one for each owner id), use the same suffix, so the data goes to the same table. the transfer missing a lot of data

Usually, the data range I got each day should be over 100,000 records. But when I setup 2 bigquery transfer (one for each owner id), use the same suffix, so the data goes to the same table. the transfer is missing a lot of data.

The transfer status shows successful but somedays I only get a 4000-5000 records instead of 100,000+ records. But somedays I got 100,000 records.

Question 1:

Is that possible bigquery transfer only finishes transferring partial of the data? My understanding is bigquery transfer is "all or none". If it is successful, then all the transfer is done. If it fails, zero records transferred.

Although, I do see in some cases that bigquery failed but status shows complete.

Question 2:

When I set up two transfers, both transfers sending data to a same table, I can see the failure rate increases hugely.

Is it possible the two transfers somehow conflict with each other in the process?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)
Waitting for answers

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...