python - What is the recursion depth and why is set to a fixed limit?

Question

I have biuld a function in python that replicates a circuit of three other functions (defined in the main function) that call eachother while iterating over a given list of numbers required in the main function (In the code there is a way to break out of the recursion circuit). Basically this structure:

def main_function(list_of_numbers, parameter_1, parameter_2):

    def block1():
        nonlocal result_1, result_2, var_1, var_2, var_3
        # Code
        block2()

    def block2():
        nonlocal result_1, result_2, var_1, var_2, var_3
        # Code
        if b:
            # Code
            block2()
        else:
            # Code
            block3()

    def block3():
        nonlocal result_1, result_2, var_1, var_2, var_3
        # Code
        if a:
            # Code
            bloc1()
        else:
            # Code
            if b:
                # Code
                if b:
                    # Code
                    if c:
                        # Code
                        block1()
                    else:
                        block2()
                else:
                    block2()
            else:
                block3()

    result_1 = []
    result_2 = []
    var_1 = 0
    var_2 = 0
    var_3 = 0
    block1()
    return result_1, result_2

The problem I'm having is that if the list of numbers is larger than 500 values then this error appears:

RecursionError: maximum recursion depth exceeded in comparison

I know I can change the limit of recursion but what does that mean, is my computer gonna be affected if I want to operate a list of half a million values?. The code inside the functions is quite simple and short.

Answer 1

A function is recursive if it calls itself either directly or via some other function that it calls. While recursion is a useful way to build an algorithm it can be a resource hog. Each time the function is called, it creates some local state (the local variables) and memory space remains in use until the function finally returns. If your recursive calls go a million deep, that's a million local frames in memory.

The risk is that recursion will blow up the stack frame or process memory and you will get a hard failure of the code. This is a really bad error as the entire program exits with no ability to clean up. Even if the program doesn't crash, it uses a lot of memory, impacting the rest of your system. There could be more swapping and worse case, the Linux oom killer could randomly kill processes in an attempt to get resources for the kernel.

To mitigate risk from either accidental recursion or a buggy algorithm that doesn't properly limit itself, python puts a limit on recursion. But developers can override that limit via sys.setrecursionlimit(some_huge_number) in which case anything bad is on their heads.