python 3.x - using Subprocess to avoid long-running task from disconnecting discord.py bot?

Question

I created a bot for my Discord server, that goes to the Reddit API for a given subreddit, and posts the Top 10 results for the Day in the Discord chat, based on the subreddit(s) that you input. It disregards self posts, and really only posts pictures and GIFs. The Discord message command would look something like this: =get funny awww news programming, posting the results for each subreddit as it gets them from the Reddit API (PRAW). THIS WORKS WITH NO PROBLEM. I know that the bot's ability to hit the API and post to discord works.

I added another command =getshuffled which puts all of the results from the subreddits in a large list, and then shuffles them before posting. This works really well with a request of up to ~50 subreddits.

This is what I need help with:

Because it can be such a large list of results, 1000+ results from 100+ subreddits, the bot is crashing on really big requests. Based on what help I got from my question yesterday, I understand what is going wrong. The bot is starting, it is talking to my Discord server, and when I pass it a long request, it stops talking to the server for too long while the Reddit API call is done, and it the Discord connection fails.

So, what I think I need to do, is have a subprocess for the code that goes to the Reddit API and pulls the results, (which I think will let the discord connection stay running), and then pass those results BACK to the bot when it is finished....

Or... this is something that Asyncio can handle on its own...

I'm having a hard time with the subprocess call, as I knew I would.

Basically, I either need help with this subprocess trickery, or need to know if I'm being an idiot and Asyncio can handle all of this for me. I think this is just one of those "I don't know what I don't know" instances.

So to recap: The bot worked fine with smaller amounts of subreddits being shuffled. It goes through the args sent (which are subreddits), grabbing info for each post, and then shuffling before posting the links to discord. The problem is when it is a larger set of subreddits of ~ 50+. In order to get it to work with the larger amount, I need to have the Reddit call NOT block the main discord connection, and that's why I'm trying to make a subprocess.

Python version is 3.6 and Discord.py version is 0.16.12 This bot is hosted and running on PythonAnywhere

Code:

from redditBot_auth import reddit

import discord
import asyncio
from discord.ext.commands import Bot
#from discord.ext import commands
import platform
import subprocess
import ast

client = Bot(description="Pulls posts from Reddit", command_prefix="=", pm_help = False)

@client.event
async def on_ready():
    return await client.change_presence(game=discord.Game(name='Getting The Dank Memes')) 

def is_number(s):
    try:
        int(s)
        return True
    except:
        pass

def show_title(s):
    try:
        if s == 'TITLES':
            return True
    except:
        pass

async def main_loop(*args, shuffled=False):
    print(type(args))

    q=10

    #This takes a integer value argument from the input string.
    #It sets the number variable,
    #Then deletes the number from the arguments list.
    title = False
    for item in args:
        if is_number(item):
            q = item
            q = int(q)
            if q > 15:
                q=15
            args = [x for x in args if not is_number(x)]

        if show_title(item):
            title = True
            args = [x for x in args if not show_title(x)]

    number_of_posts = q * len(args)
    results=[]

    TESTING = False #If this is turned to True, the subreddit of each post will be posted. Will use defined list of results


    if shuffled == False: #If they don't want it shuffled

        for item in args:
            #get subreddit results
            #post links into Discord as it gets them
            #The code for this works

    else: #if they do want it shuffled
        output = subprocess.run(["python3.6", "get_reddit.py", "*args"])
        results = ast.literal_eval(output.decode("ascii"))
        # ^^ this is me trying to get the results back from the other process.

. This is my get_reddit.py file:

#THIS CODE WORKS, JUST NEED TO CALL THE FUNCTION AND RETURN RESULTS
#TO THE MAIN_LOOP FUNCTION

from redditBot_auth import reddit
import random

def is_number(s):
    try:
        int(s)
        return True
    except:
        pass

def show_title(s):
    try:
        if s == 'TITLES':
            return True
    except:
        pass

async def get_results(*args, shuffled=False):

    q=10

    #This takes a integer value argument from the input string.
    #It sets the number variable,
    #Then deletes the number from the arguments list.
    title = False
    for item in args:
        if is_number(item):
            q = item
            q = int(q)
            if q > 15:
                q=15
            args = [x for x in args if not is_number(x)]

        if show_title(item):
            title = True
            args = [x for x in args if not show_title(x)]

    results=[]

    TESTING = False #If this is turned to True, the subreddit of each post will be posted. Will use defined list of results.
    NoGrabResults = False

    #This pulls the data and creates a list of links for the bot to post

    if NoGrabResults == False:
        for item in args:
            try:
                #get the posts
                #put them in results list    

            except Exception as e:
                #handle error
                pass

        try:
            #print('____SHUFFLED___')
            random.shuffle(results)
            random.shuffle(results)
            random.shuffle(results)

        except:
            #error stuff

        print(results)
#I should be able to read that print statement for the results, 
#and then use that in the main bot function to post the results.

.

@client.command()
async def get(*args, brief="say '=get' followed by a list of subreddits", description="To get the 10 Top posts from a subreddit, say '=get' followed by a list of subreddits:
'=get funny news pubg'
 would get the top 10 posts for today for each subreddit and post to the chat."):
    #sr = '+'.join(args)
    await main_loop(*args)

#THIS POSTS THE POSTS RANDOMLY   
@client.command()
async def getshuffled(*args, brief="say '=getshuffled' followed by a list of subreddits", description="Does the same thing as =get, but grabs ALL of the posts and shuffles them, before posting."):

    await main_loop(*args, shuffled=True)


client.run('my ID')

UPDATE: Following advice, I had the command passed through a ThreadPoolExecutor as shown:

async def main(*args, shuffled):

    if shuffled==True:

        with concurrent.futures.ThreadPoolExecutor() as pool:
            results = await asyncio.AbstractEventLoop().run_in_executor(
                executor=pool, func=await main_loop(*args, shuffled=True))
            print('custom thread pool', results)

but this still results in errors when the script tries to talk to Discord:

ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Client._run_event() running at /home/GageBrk/.local/lib/python3.6/site-packages/discord/client.py:307> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f28acd8db28>()]>>
Event loop is closed
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
...

It is sending the results correctly, but discord is still losing connection.

Answer 1

praw relies on the requests library, which is synchronous meaning that the code is blocking. This can cause your bot to freeze if the blocking code takes too long to execute.

To get around this, a separate thread can be created that handles the blocking code. Below is an example of this. Note how blocking_function will use time.sleep to block for 10 minutes (600 seconds). This should be more than enough to freeze and eventually crash the bot. However, since the function is in it's own thread using run_in_executor, the bot continues to operate as normal.

import time
import asyncio
from discord.ext import commands
from concurrent.futures import ThreadPoolExecutor

def blocking_function():
    print('entering blocking function')
    time.sleep(600)
    print('sleep has been completed')
    return 'Pong'

client = commands.Bot(command_prefix='!')

@client.event
async def on_ready():
    print('client ready')

@client.command()
async def ping():
    loop = asyncio.get_event_loop()
    block_return = await loop.run_in_executor(ThreadPoolExecutor(), blocking_function)
    await client.say(block_return)

client.run('token')