java.io.FileNotFoundException in NativeFileSorter #5765
Description
When calling method sortValues:
collection.sortValues(1024)
when underlying sorted items exceed memory buffer and spill to disk:
Caused by: java.lang.IllegalStateException: java.io.FileNotFoundException: /tmp/sort10431472593432919885seq (No such file or directory)
org.apache.beam.sdk.extensions.sorter.NativeFileSorter.lambda$mergeSortedFiles$1(NativeFileSorter.java:175)
org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$DecodingIterable.iterator(SortValues.java:194)
com.spotify.scio.extra.sorter.syntax.SorterOps$$anonfun$sortValues$1$$anonfun$apply$3$$anon$1$$anon$2.<init>(SCollectionSyntax.scala:75)
com.spotify.scio.extra.sorter.syntax.SorterOps$$anonfun$sortValues$1$$anonfun$apply$3$$anon$1.iterator(SCollectionSyntax.scala:74)
scala.collection.View$Map.iterator(View.scala:294)
scala.collection.immutable.List.prependedAll(List.scala:151)
It is hard to find more accurate conditions to reproduce, because this is not consistent but a race condition. Code needs to be analyzed and apparently some lazy evaluations are the root cause