hadoop - Will there be Shuffle and sort in Map only task?

Question

Does the shuffle and sort phase come before the end of the map task or does it come after the output is generated from the map task so that there is no look back to the map task anymore. This is a 'Map only task' case where I get confusion. If there is no Shuffle and sort in Map only task, can someone explain how is the data written into the final output files.

Answer 1

When you have a map-only task, there is not shuffling at all, which means that mappers will write the final output directly to the HDFS.

On the other hand, when you have a whole Map-Reduce program, with mappers and reducers, yes, shuffling can start before reduce-phase start.

Quoting this very nice answer in SO:

First of all shuffling is the process of transfering data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn't be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to save some time. That's why you can see a reduce status greater than 0% (but less than 33%) when the map status is not yet 100%.

Hope this answer had clarified your confusion.