python - nightmare with relative imports, how does pep 366 work?

Question

I have a "canonical file structure" like that (I'm giving sensible names to ease the reading):

mainpack/

  __main__.py
  __init__.py 

  - helpers/
     __init__.py
     path.py

  - network/
     __init__.py
     clientlib.py
     server.py

  - gui/
     __init__.py
     mainwindow.py
     controllers.py

In this structure, for example modules contained in each package may want to access the helpers utilities through relative imports in something like:

# network/clientlib.py
from ..helpers.path import create_dir

The program is runned "as a script" using the __main__.py file in this way:

python mainpack/

Trying to follow the PEP 366 I've put in __main__.py these lines:

___package___ = "mainpack"
from .network.clientlib import helloclient 

But when running:

$ python mainpack 
Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "path/mainpack/__main__.py", line 2, in <module>
    from .network.clientlib import helloclient
SystemError: Parent module 'mainpack' not loaded, cannot perform relative import

What's wrong? What is the correct way to handle and effectively use relative imports?

I've tried also to add the current directory to the PYTHONPATH, nothing changes.

Answer 1

The "boilerplate" given in PEP 366 seems incomplete. Although it sets the __package__ variable, it doesn't actually import the package, which is also needed to allow relative imports to work. extraneon's solution is on the right track.

Note that it is not enough to simply have the directory containing the module in sys.path, the corresponding package needs to be explicitly imported. The following seems like a better boilerplate than what was given in PEP 366 for ensuring that a python module can be executed regardless of how it is invoked (through a regular import, or with python -m, or with python, from any location):

# boilerplate to allow running as script directly
if __name__ == "__main__" and __package__ is None:
    import sys, os
    # The following assumes the script is in the top level of the package
    # directory.  We use dirname() to help get the parent directory to add to
    # sys.path, so that we can import the current package.  This is necessary 
    # since when invoked directly, the 'current' package is not automatically
    # imported.
    parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    sys.path.insert(1, parent_dir)
    import mypackage
    __package__ = str("mypackage")
    del sys, os

# now you can use relative imports here that will work regardless of how this
# python file was accessed (either through 'import', through 'python -m', or 
# directly.

If the script is not at the top level of the package directory and you need to import a module below the top level, then the os.path.dirname has to be repeated until the parent_dir is the directory containing the top level.