python - Problems with command using * wildcard in subprocess

Question

I'm trying to copy files from one location to another using subprocess library and Popen method. When runing following script I'm getting the error cp: cannot stat /some/dev_path/*. I was told that the * is not expanded to the file names and that's where the problem is. Also in some other posts people were suggesting to use call instead of Popen, but call will not return stderr as far as I know.

devPath = '/some/dev_path/'
productionPath = '/some/prod_path/'

p = subprocess.Popen(['cp', '-r', devPath + '*', productionPath], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
pout, perr = p.communicate()

if perr != '':
    sys.exit('Error: ' + perr)

Answer 1

Expanding the * (globbing) is a function of your shell, bash for example. Therefore you'd have to use the keyword argument shell=True in your subprocess.Popen call.

However, for this case I'd strongly suggest to use shutil.copytree instead.

(First of all, because it's much simpler (see Zen of Python) and less error-prone. Dealing with errors is much cleaner, you get nice exceptions including a list of errors (for multi file operations like yours), and you don't have to deal with spawning a subprocess and communicating with it. Second, it's an unnecessary waste of resources to fork a child process if you don't need to. Other issues include quoting / escaping and possibly introducing security vulnerabilities into your code if you fail to properly sanitize user input.)

For example:

from shutil import copytree
from shutil import Error

try:
   copytree('dir_a', 'dir_b')
except (Error, OSError), e:
    print "Attempt to copy failed: %s" % e

Also, you shouldn't build filesystem paths by concatenating strings together, but instead use os.path.join(). That will use the correct directory separator (os.sep) for the current OS and allow you to easily write portable code.

Example:

>>> import os
>>> os.path.join('/usr/lib', 'python2.7')
'/usr/lib/python2.7'

Note: os.path.join still only does (smart) string manipulation - it doesn't care if that path is accessible or even exists.