Updating an old site to Python 3.*, Django 2.*, Wagtail 2.*

Carson Zhang
Senior Software Developer
Tue 23 April 2019 • 14 min read

0. Useful Links

Check if all the packages are upgradable.
- It is essentially important that you at least check all the major packages that are used as core packages to make sure they are upgradable. If not, you may have to choose not to upgrade the web at all, or find some other solutions
- Two very useful tools:
  - updatable – returns all available updates for the packages installed in the current environment (virtualenv or global)
  - pipdeptree – returns a tree of dependencies, which will help to remove installed packages that aren’t required anymore.
Check the version support table for python, django, and wagtail
- support relation: wagtail 2.0 --> django 1.11, 2.0 --> Python 3.4, 3.5, 3.6
Gochas:
- Any site using the wagtail elasticsearch 1.x. You’ll have to upgrade it before you make the switch.
- Any site using wagtail API 1.x. It’s also been removed and if upgrading isn’t possible you can use wagtailapi_legacy.
Warnings:
- Create a new git branch for upgrading in case anything went wrong
- Create a another db for the upgrade if possible, in case anything went wrong

The biggest trick is that you do not work in the current virtual environment (assume that you are using a virtual environment, and please do so if not). You should start a new environment with python 3 preinstalled so that you don't have to manually uninstall and install different versions of python.
- For example: mkvirtualenv env_name --python=python3
After switch to Python 3, we have to deal with the changes:
- print has become a function in python 3
- from six import u as unicode # Fix Python 2.x. fixes all the issues with unicode
Run the website and fix all the errors\warnings, and then go to the next step
- Double check that “Conservative python 3 porting guide” for any errors showing up

In order for the production server to be upgraded successfully, you should pay extra attention to how you make the branches as you have to manually change some code and how the migrations may affect your database
I suggest using the two-step process and make three branches:
- Step 1: the python 3 branch
  - You should commit any change for making the project run under python 3 in this branch
  - ATTENTATION: after your project is upgraded and running with python 3, you should do a django-admin makemigrations and django-admin migrate to make sure your database is up-to-date
- Step 2: wagtail and django 2 branch
  - Commit any changes for upgrading wagtail and django to this branch
So the process for reproducing the upgrade on the production server will be:
- Step 1:
  - make virtual environment with python 3
  - git checkout python_upgrade_3
  - pip install -r dev.txt
  - django-admin migrate (you must migrate first for the project to run)
  - django-admin runserver should work
- Step 2:
  - git checkout wagtail_django_upgrade
  - *pip freeze | xargs pip uninstall -y to delete all the packages first
  - *pip install -r dev.txt to re-install all the packages (the quickest way to get all the upgraded packages working)
  - add2virtualenv thirdparty_apps (if any packages are not maintained and you have to upgrade them locally)
  - django-admin migrate
  - django-admin runserver should work

You should upgrade django one version at a time and fix everything during upgrading until you are at 1.11. Chances are that the upgrade is much smoother to Django 2.0 if the project was built in 1.11.
- It is possible due to package dependencies, you cannot upgrade to 1.11, that's ok, but try to upgrade to the latest django version before 2.0
- Upgrade wagtail accordingly, to the version 1.13 as close as possible
- These errors will be written in red text if any are found.
- Run the application locally, and fix any deprecation warnings that it alerts you with.
After you are at Django 1.11 (or at least very close to this version) and before continuing, make sure the site doesn't have any missing migrations and you have made a backup of the database (just incase shit hits the fan).
- django-admin makemigrations to check if there is any migration changes
- django-admin migrate to actually make the migration happens at the database level
Squash every apps migrations and make sure to use the last migration number.
- NOTE that you have to do this under Python 3 environment otherwise nightmare will come to you after you are at Django 2.0
- For example: django-admin squashmigrations pages 0032
Delete all migration files except for the squashed ones.

Before doing the actual upgrading, double check that you are at (or at least close to) Django 1.11 and wagtail 1.13 as we mentioned before
Start upgrading the main ones:
- Upgrade django to 2.0. pip3 install django==2.*
- Upgrade wagtail to 2.0. pip3 install wagtail==2.*
- Upgrade DRF to 3.7. pip3 install djangorestframework==3.7.7
Next, run the wagtail module update script. This will rename any old imports to the new ones.
- wagtail updatemodulepaths --list Will list the files getting changed.
- wagtail updatemodulepaths --diff Will show the diffs.
- wagtail updatemodulepaths Will make the all changes.
If this doesn't work and gives you a unicode error. Wait a bit a run it again. I noticed docker takes some time to switch to django 2.

Django 2 made some changes to how urls and namespacing works. Known quick changes are:
- url(r'^clientadmin\', include(admin.site.urls)), to url(r'^clientadmin\', admin.site.urls),
Also make sure to add app_name to included urls that are namespaced. So if you see url(r'^forms\', include(snippetforms_urls, namespace="forms")), make sure apps\cms\snippetforms\urls.py is updated to have app_name = 'forms'

Django 2 or more specifically python3 doesn't have the unicode mess, instead it only has the str type which saves a lot headache. In the django models, change all the __unicode__ to __str__ and it'll correct all the display issue

Django 2 removed assignment_tag and is used in most of our template tags (blog, menu & pages). Do a quick find all and replace it to simple_tag.

Django 2 does not call process_request and process_response by default. This functionally has been replaced with call and the get_response function that is passed to init.
- A great solution is HERE
- Above solution should work, if not, check the accepted answer at HERE

If you encounter an error similar to 'ManyToManyField' object has no attribute 'rel', it is possible due to the fact that db_field.rel is deprecated, just change it to db_field.remote_field
- More details about this could be find HERE

For those third packages that has compatibility issues:
1. check if github has the latest version since Pypi can be out-dated sometimes
2. checkout github pull request as sometime people have already fixed it
3. If above two steps won't work, git clone the whole package and save into the thirdparty_apps folder inside the project and make the change locally. NOTE that by doing this it will be really tough to do further upgrade in the future.