c - Having hard time tracking memory corruption - when running with Valgrind runs correctly with no errors

Question

We have a complex program that is working well on heavy duty input (any input actually) with no multithreading implemented.
We've implemented multithreading with a threadpool, and given these input parameters I get these results:
(Note: Where I say no errors, it means I've tested with valgrind -v and when I say no memory leaks, it means I've tested it with valgrind --leak-check=full -v).

1. small_file: Runs successfully with more than 1 workers (threads), no valgrind errors, no memory leaks
2. medium_file: With 1 worker it runs successfully, no errors/memory leaks. With > 1 workers, I get: a. usually heap-corruption error, b. double-free. When running with valgrind -v with > 1 workers the program completes successfully. Also, no errors are printed from valgrind, that is ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2).

Now that I don't get any errors from valgrind to start with, what can I do to find the memory corruption problem in this complex and big application?

DevelopmentEnvironment:
Ubuntu, 64bit, gcc version: 4.7.2 and 4.8.1 (different computers, newer version of Ubuntu).

Answer 1

Now that I don't get any errors from valgrind to start with, what can I do to find the memory corruption problem in this complex and big application?

Well let me describe to you what I did to find memory leaks in Microsoft's implementation of JavaScript back in the 1990s.

First I ensured that in the debug version of my program, as many memory allocations as possible were being routed to the same helper methods. That is, I redefined malloc, new, etc, to all be synonyms for an allocator that I wrote myself.

That allocator was just a thin shell around an operating system virtual heap memory allocator, but it had some extra smarts. It allocated extra memory at the beginning and end of the block and filled that with sentinel values, a threadsafe count of the number of allocations so far, and a threadsafe doubly-linked list of all allocations. The "free" routine would verify that the sentinel values on both sides were still intact; if not, then there's a memory corruption somewhere. It would unlink the block from the linked list and free it.

At any point I could ask the memory manager for a list of all the outstanding blocks in memory in the order they had been allocated. Any items left in the list when the DLL was unloaded were memory leaks.

Those tools enabled me to find memory leaks and memory corruptions in real time very easily.