fbprophet - Python Too many open files - fbprohet

Question

I'm working on a project where I calculate multiple models with fbprophet. Fbprohet has the problem, that there is no default way to surpress the messages of pystan. So there is a workaround mentioned on github to use this as decorator:

class suppress_stdout_stderr(object):

    def __init__(self):
        # Open a pair of null files
        self.null_fds = [os.open(os.devnull, os.O_RDWR) for x in range(2)]
        # Save the actual stdout (1) and stderr (2) file descriptors.
        self.save_fds = (os.dup(1), os.dup(2))

    def __enter__(self):
        # Assign the null pointers to stdout and stderr.
        os.dup2(self.null_fds[0], 1)
        os.dup2(self.null_fds[1], 2)

    def __exit__(self, *_):
        # Re-assign the real stdout/stderr back to (1) and (2)
        os.dup2(self.save_fds[0], 1)
        os.dup2(self.save_fds[1], 2)
        # Close the null files
        os.close(self.null_fds[0])
        os.close(self.null_fds[1])

I calculate the model with that command:

with suppress_stdout_stderr():
     m = Prophet()
     m.fit(history)

I'm doing that for every model about 250 times and have about 600 single models. So Prophet() will be called about 150.000 time:

class Model:

    def __init__(stockticker):
        self.ticker = stockticker
        self.data = pd.read_excel(f'{self.ticker}.xlsx')


   def get_residuals(my_stock_resid_distr_fitting):
       for i in range(len(my_stock_resid_distr_fitting)):
            with suppress_stdout_stderr():
                 m = Prophet()
                 m.fit(history)

   def main():
      self.get_residuals(200)


tickers = ['AAPL',....]
for ticker in tickers:
   m = model(ticker)
   m.main() 

• list of stocktickers are about 600

After about 1024 iterations the script stops running with that error code:

OSError: [Errno 24] Too many open files

So as I understand the decorator correctly, it opens empty files to substitute the pystan warning. In my opinion the decorator closes the empty files in the function def __exit__(self, *_):. But this seems not to work correctly.

Does anybody sees the mistake or have another solution?

If not I would increase the max open files on my ubuntu server, but I hope you have a more pythonic way.

BR Lorenz

question from:https://stackoverflow.com/questions/66052283/python-too-many-open-files-fbprohet

Answer 1

Waitting for answers

Answer 2

try not use with suppress_stdout_stderr() and see if the same error shows again, my guess is that within suppress_stdout_stderr() os.dup2() leaves some files unclosed. If the number of unclosed files exceed certain number then the error pop up.