unit testing - Django tests complain of missing tables

Question

When I run my test dealing with my Customer model, I get the following error:

DatabaseError: (1146, "Table 'test_mcif2.customer' doesn't exist")

I'm not entirely surprised because I have my Django project connected to a "legacy" database. Since my tables weren't created "the Django way," it's not shocking that Django wouldn't be able to talk to them without some finagling. Here's my model:

from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel

class Customer(McifModel):

    class Meta:
        db_table = u'customer'
        app_name = 'mcif'

    id = models.BigIntegerField(primary_key=True)
    customer_number = models.CharField(unique=True, max_length=255)
    social_security_number = models.CharField(unique=True, max_length=33)
    name = models.CharField(unique=True, max_length=255)
    phone = models.CharField(unique=True, max_length=255)
    deceased = models.IntegerField(unique=True, null=True, blank=True)
    do_not_mail = models.IntegerField(null=True, blank=True)
    created_at = models.DateTimeField()
    updated_at = models.DateTimeField()

    def distinguishing_column_names(self):
        return ['name', 'customer_number', 'social_security_number', 'phone']

Any idea why exactly this isn't working?

Edit: Here's McifModel:

from django.db import models
from django.db import connection, transaction

class McifModel(models.Model):
    class Meta:
        abstract = True

    def upsert(self):
        cursor = connection.cursor()
        cursor.execute(self.upsert_sql())
        transaction.commit_unless_managed()
        return self

    def value_list(self):
        return ','.join(map(lambda column_name: "'{c}'".format(c=getattr(self, column_name)), self.distinguishing_column_names()))

    def upsert_sql(self):
        column_names = ','.join(self.distinguishing_column_names())
        return "INSERT IGNORE INTO {t} ({c}) VALUES ({v})".format(t=self._meta.db_table, c=column_names, v=self.value_list())

    @classmethod
    def save_from_row(cls, row):
        object = cls()
        map(lambda column_name: setattr(object, column_name, row.value(object._meta.db_table, column_name)), object.distinguishing_column_names())
        return object.upsert()

Edit: I took tarequeh's advice and put the contents of the Caktus file in mcif/utils.py. I also set TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'. If I go on the console I can verify that Customer is unmanaged:

>>> [m for m in get_models() if not m._meta.managed]
[<class 'mcif.models.customer.Customer'>]

However, my test still complains that the table doesn't exist. What am I missing?

Here's my settings.py:

# Django settings for mcifdjango project.

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
    ('Jason Swett', 'jason.swett@gmail.com'),
)

MANAGERS = ADMINS

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'xxxxx',                # Or path to database file if using sqlite3.
        'USER': 'xxxxx',                       # Not used with sqlite3.
        'PASSWORD': 'xxxxx',           # Not used with sqlite3.
        'HOST': '',                           # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                           # Set to empty string for default. Not used with sqlite3.
    }
}

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Chicago'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True

# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = ''

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = ''

# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/media/'

# Make this unique, and don't share it with anybody.
SECRET_KEY = '#7+qm%hqfe+z8ul5@x_i&sqmu!n=4sa0&i0_#)m99*w$fbk3%#'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
#     'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
)

ROOT_URLCONF = 'mcifdjango.urls'

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
)

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.admin',
    'django_extensions',
    'mcif',
    # Uncomment the next line to enable the admin:
    # 'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'

import os
ROOTDIR = os.path.abspath(os.path.dirname(__file__))
TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    ROOTDIR + '/mcif/templates',
)

Edit 2:

Here's my Customer class now:

from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel

class Customer(McifModel):

    class Meta:
        db_table = u'customer'
        managed = False

    id = models.BigIntegerField(primary_key=True)
    customer_number = models.CharField(unique=True, max_length=255)
    social_security_number = models.CharField(unique=True, max_length=33)
    name = models.CharField(unique=True, max_length=255)
    phone = models.CharField(unique=True, max_length=255)
    deceased = models.IntegerField(unique=True, null=True, blank=True)
    do_not_mail = models.IntegerField(null=True, blank=True)
    created_at = models.DateTimeField()
    updated_at = models.DateTimeField()

    def distinguishing_column_names(self):
        return ['name', 'customer_number', 'social_security_number', 'phone']

Here's what I get when I run the test:

$ ./manage.py test mcif.CustomerUpsertTest
Creating test database 'default'...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
Installing index for admin.LogEntry model
No fixtures found.
E
======================================================================
ERROR: test_upsert (mcif.tests.customer_upsert_test.CustomerUpsertTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jason/projects/mcifdjango/mcif/tests/customer_upsert_test.py", line 9, in test_upsert
    customer.upsert()
  File "/home/jason/projects/mcifdjango/mcif/models/mcif_model.py", line 11, in upsert
    cursor.execute(self.upsert_sql())
  File "/usr/lib/pymodules/python2.6/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
    raise errorclass, errorvalue
DatabaseError: (1146, "Table 'test_mcif_django.customer' doesn't exist")

----------------------------------------------------------------------
Ran 1 test in 3.724s

FAILED (errors=1)
Destroying test database 'default'...

Answer 1

Since you're using a legacy database, you are probably not adding the app name to INSTALLED_APPS. If an app is not included in INSTALLED_APPS, the tables for the apps' models will not get created on syncdb. This works for you in production since you already have a table, but not in test environment.

You can adopt any of the following:

• The supermonkeypatch way: Take out app_name from Customer class Meta, put the model in a models.py file inside a python module name mcif, and add mcif to INSTALLED_APPS - just for the sake of testing

• The nicer way: Extend DjangoTestSuiteRunner and override setup_test_environment to call super and then create your legacy table manually in the test DB.

• The nicest way: Put your model in properly named app module. Remove app_name from model Meta but add managed=False docs. Include app name in INSTALLED_APPS. Now django will not create table for that model. Then use this nice snippet the Caktus group folks have compiled to run your tests.

Cheers!

Edit - How to use the overridden DjangoTestSuiteRunner

You will need at least Django 1.2 for this.

Copy the code from here. Put it in utils.py inside the mcif app.

Add/edit the following in settings.py:

TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'

Now when you run tests, all unmanaged tables will be treated as managed table only for the duration of the test. So the tables will be created prior to running tests.

Notice this part of the code, thats where the magic happens.

self.unmanaged_models = [m for m in get_models() if not m._meta.managed]
for m in self.unmanaged_models:
    m._meta.managed = True

2nd Edit: Possible Gotchas

Make sure of the following:

• The DB user has privilege to create databases and not only tables because django will try to create a test database
• The test cases extend django.test.TransactionTestCase, since you have transactional behavior
• If none of the above applies, put a pdb in ManagedModelTestRunner's setup_test_environment just to make sure the code is being reached. Because if that code is reached, the table should get created

3rd Edit: Debugging Inside mcif.utils.ManagedModelTestRunner replace setup_test_environment function with the following and let me know if the output of your test changes:

def setup_test_environment(self, *args, **kwargs):
    print "Loading ManagedModelTestRunner"
    from django.db.models.loading import get_models
    self.unmanaged_models = [m for m in get_models()
                             if not m._meta.managed]
    for m in self.unmanaged_models:
        print "Modifying model %s to be managed for testing" % m
        m._meta.managed = True
    super(ManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)