exception handling - Python not catching MemoryError

Question

I've wrapped some code that can run out of memory with a try/except block. However, though a MemoryError is generated, it is not caught.

I have the following code:

    while True:
        try:
            self.create_indexed_vocab( vocab )
            self.reset_weights()
            break;
        except MemoryError:
            # Stuff to reduce size of vocabulary
            self.vocab, self.index2word = None, None
            self.syn0, self.syn1 = None, None

            self.min_count += 1
            logger.info( ...format string here... )

I get the following Traceback:

File "./make_model_tagged_wmt11.py", line 39, in <module>
  model.build_vocab(sentences)
File "/root/CustomCompiledSoftware/gensim/gensim/models/word2vec.py", line 236, in build_vocab
  self.reset_weights()
File "/root/CustomCompiledSoftware/gensim/gensim/models/word2vec.py", line 347, in reset_weights
  self.syn0 += (random.rand(len(self.vocab), self.layer1_size) - 0.5) / self.layer1_size
File "mtrand.pyx", line 1044, in mtrand.RandomState.rand (numpy/random/mtrand/mtrand.c:6523)
File "mtrand.pyx", line 760, in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5713)
File "mtrand.pyx", line 137, in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1300)
MemoryError

I'm running Python 2.7.3 under Ubuntu 12.04

The reset_weights line self.syn0 is exactly the line I am expecting to raise the exception (it allocates a big array). The puzzling thing is that I can't catch the memory error and do things that will make the array size smaller.

Are there special circumstances that result in the MemoryError being unable to be caught?

Answer 1

Note that because of the underlying memory management architecture (C’s malloc() function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause.

(See the docs)

Usually, you can catch MemoryErrors nevertheless. Without knowing what exactly happens when a MemoryError gets thrown, I'd guess that you might not be able to catch it when shit really hit the fan and there's no more memory there to handle it.

Also, since you may not be able to really recover from it (see above), it probably wouldn't make that much sense to catch it. You should really avoid running out of memory and limiting the amount of memory your program uses by e.g. only allowing a list to have a limited size.