stdout - Nohup and Python -u : it still doesn't log data in realtime

Question

When launching a Python process, in background, with

nohup python myscript.py > test.log 2>&1 < /dev/null &

the problem is that stdout is buffered: the data is not written in realtime to test.log. The common solution to this problem is to flush periodically with sys.stdout.flush(), or even better, as suggested by this answer, to use python -u :

nohup python -u myscript.py > test.log 2>&1 < /dev/null &

---

But this is not enough. I noticed that it worked during a few hours, and then, it stopped working, i.e. after a few hours test.log is not written in realtime anymore, even if I used nohup python -u ....

1) This is how to reproduce the problem (I have a standard Debian Jessie). Start this file:

import time
import datetime
while True:
    print datetime.datetime.now()
    time.sleep(60)

with nohup python -u myscript.py > test.log 2>&1 < /dev/null &. The log file will be updated during a few hours, and then after 3 or 4 hours, nothing anymore.

2) How to solve this problem, without having to insert stdout.flush() every 2 lines in the code (ugly solution) ?

Answer 1

If python -u does not seem to be working for you, the next suggestion would be to replace sys.stdout with a custom class that does not buffer output.

Magnus Lycka provided this solution in a mailing list

class Unbuffered(object):
    def __init__(self, stream):
        self.stream = stream
    def write(self, data):
        self.stream.write(data)
        self.stream.flush()
    def __getattr__(self, attr):
        return getattr(self.stream, attr)

import sys
sys.stdout = Unbuffered(sys.stdout)
print 'Unbuffered Output'

I can't see why python -u wouldn't work from your example, but this should solve the issue for you.