hadoop - Append data to existing file in HDFS Java

Question

I'm having trouble to append data to an existing file in HDFS. I want that if the file exists then append a line, if not, create a new file with the name given.

Here's my method to write into HDFS.

if (!file.exists(path)){
   file.createNewFile(path);
}

FSDataOutputStream fileOutputStream = file.append(path); 
BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fileOutputStream));
br.append("Content: " + content + "
");
br.close();

Actually this method writes into HDFS and create a file but as I mention is not appending.

This is how I test my method:

RunTimeCalculationHdfsWrite.hdfsWriteFile("RunTimeParserLoaderMapperTest2", "Error message test 2.2", context, null);

The first param is the name of the file, the second the message and the other two params are not important.

So anyone have an idea what I'm missing or doing wrong?

Answer 1

Actually, you can append to a HDFS file:

From the perspective of Client, append operation firstly calls append of DistributedFileSystem, this operation would return a stream object FSDataOutputStream out. If Client needs to append data to this file, it could calls out.write to write, and calls out.close to close.

I checked HDFS sources, there is DistributedFileSystem#append method:

 FSDataOutputStream append(Path f, final int bufferSize, final Progressable progress) throws IOException

For details, see presentation.

Also you can append through command line:

hdfs dfs -appendToFile <localsrc> ... <dst>

Add lines directly from stdin:

echo "Line-to-add" | hdfs dfs -appendToFile - <dst>