How is super() in Python 3 implemented?

Question

I'm wondering how is the new super in Python 3 implemented.

This question was born in my head after I have made a small example and I got a strange error. I'm using Pyutilib Component architecture (PCA) and I've made my custom metaclass to drive the creation of another class:

from pyutilib.component.core import implements, SingletonPlugin, PluginMeta, Interface

class IPass(Interface):
    pass

class __MetaPlugin(PluginMeta):
    def __new__(cls, name, baseClasses, classdict):
        print(cls, name, baseClasses, classdict)
        if baseClasses:
            baseClasses += (SingletonPlugin,)
        return PluginMeta.__new__(cls, name, baseClasses, classdict)

class Pass(metaclass=__MetaPlugin):
    implements(IPass)

    def __init__(self, inputs=[], outputs=[]):
        self.inputs = []
        self.outputs = []


class A(Pass):
    def __init__(self):
        print(self.__class__) # <class '__main__.A'>
        print(self.__class__.__class__) # <class '__main__.__MetaPlugin'>
        print(PluginMeta.__class__) # <class 'type'>
        super().__init__() # SystemError: super(): empty __class__ cell
        #Pass.__init__(self) - this works

a = A()

I'm getting the following error:

super().__init__() SystemError: super(): empty __class__ cell

I'm wonderign what does exactly super(), that it raises error on super().__init__(), while all of self.__class__, self.__class__.__class__and PluginMeta.__class__ exist. Additional the "old way" - Pass.__init__(self) is working.

Answer 1

TL;DR: This "empty __class__ cell" error will happen when the metaclass tries to call a method in the defined class (or instantiate it) before it is done with its __new__ and __init__,and the called method uses super. The error will also happen if one writes a call to super() in a function defined outside of a class body, and tries to add this method to an existing class and use it. (update: this behavior has been fixed in Python 3.6)

Python 3 super makes an implicit reference to a "magic" __class__[*] name which behaves as a cell variable in the namespace of each class method.

This variable is created automatically at the end of the class creation mechanism - i.e. whenever there is a class body in Python, the metaclass's __new__ and __init__ are run - when __init__ finishes, the __class__ cell is populated and made available to the class's methods.

What is going on here, is that likely (I have not looked at all the code) in the PluginMeta initialization code, the class's __init__ is called, before the end of the metaclass __init__ - since one of the points of this metaclass is handling singletons - what is likely to happen is that the metaclass mechanism is instantiating the single-instance and populating __instance__ before returning from the metaclass' __init__. The implicit __class__ used by super does not exist at this point.

Thus, referencing the superclass by the hardcoded name, as one had to do prior to super in Python2 will work - and is the best way to achieve what you want there.

*- This is not the self.__class__ attribute of the instance, it is a __class__ variable actually available inside methods:

class A:
   def a(self):
      print ("Instance's class: {}, "
             "actual class where this line is coded: {}".format(
                 self.__class__, __class__))

class B(A):
   pass

And running this we have:

>>> B().a()
Instance's class: <class '__main__.B'>, actual class where this line is coded: <class '__main__.A'>
>>> 

From the Python datamodel:

__class__ is an implicit closure reference created by the compiler if any methods in a class body refer to either __class__ or super. This allows the zero argument form of super() to correctly identify the class being defined based on lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method.

For more details, please check PEP 3135