google cloud dataflow - How can I improve performance of TextIO or AvroIO when reading a very large number of files?

Question

Welcome To Ask or Share your Answers For Others

google cloud dataflow - How can I improve performance of TextIO or AvroIO when reading a very large number of files?

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

TextIO.read() and AvroIO.read() (as well as some other Beam IO's) by default don't perform very well in current Apache Beam runners when reading a filepattern that expands into a very large number of files - for example, 1M files.

How can I read such a large number of files efficiently?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

1.3k views

1 Answer

深蓝 · Answer 1 · 2021-10-23T20:04:43+0000

answered Oct 24, 2021 by 深蓝 (71.8m points)

Waitting for answers

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

Categories

google cloud dataflow - How can I improve performance of TextIO or AvroIO when reading a very large number of files?

Please log in or register to add a comment.

Please log in or register to answer this question.

1 Answer

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags