Introduction
Having worked with CKAN for quite some time and while working on the migration process to CKAN 2.9, I decided to prepare a short guideline on how to migrate from CKAN 2.8 to CKAN 2.9. For those who are learning about CKAN now, the Comprehensive Knowledge Archive Network (CKAN), is a tool for creating open data websites. It helps you manage and publish collections of data. It is used by national and local governments, research institutions, and other organizations that collect a lot of data. Now, let`s take a look at the process of migration.
How to migrate from CKAN 2.8.9 to CKAN 2.9.4
There are 24 ckan extensions running on the site, some custom for this specific site, some taken from the open-source community. So, as is with any open source project the first place to check is the community for already migrated extensions from the upstream repositories. If an extension was already migrated it can be pulled from upstream and just merged with custom HTMLs and the translations for the extension. For the extensions that were not migrated there is this general blueprint that can be followed, tackling the migration one extension at a time:
- Migrate Python 2 to Python 3
- Migrate the web framework Pylons to Flask
- Integration of the key CKAN dependencies
- Upgrade of the UI in order to be compatible with the changes done through the migration of the web framework Pylons to Flask
- Migrate the current tests from nosetests to pytests
Migrating Python 2 to Python 3 is the simplest task, just simply change python 2 specific syntax to python 3. After this, there are some Python 3 compatibility issues for example bytes vs str handling, unicode, use of basestring, relative imports etc, but these errors can be handled while working on the next steps.
The next three steps are interlinked. With these steps comes code modernization. First, check whether any python libraries that the extension depended on are Python 2 only. If that is the case, you should find suitable replacements or updates. A useful tool for this is the caniusepython3 library, which can help in identifying dependencies that can’t run on Python 3. After this, comes the migration of all the controllers to flask views/blueprints. This means changing all BaseController controllers handled by the IRoutes interface to blueprints handled by IBlueprint interface. Simply put, the code went from looking like this:
import ckan.plugins as p
class MyPlugin(p.SingletonPlugin):
p.implements(p.IRoutes, inherit=True)
def before_map(self, _map):
# New route to custom action
m.connect(
'/foo',
controller='ckanext.my_ext.plugin:MyController',
action='custom_action')
# Overriding a core route
m.connect(
'/group',
controller='ckanext.my_ext.plugin:MyController',
action='custom_group_index')
return m
class MyController(p.toolkit.BaseController):
def custom_action(self):
# ...
def custom_group_index(self):
# ...
to looking like this:
import ckan.plugins as p
def custom_action():
# ...
def custom_group_index():
# ...
class MyPlugin(p.SingletonPlugin):
p.implements(p.IBlueprint)
def get_blueprint(self):
blueprint = Blueprint('foo', self.__module__)
rules = [
('/foo', 'custom_action', custom_action),
('/group', 'group_index', custom_group_index), ]
for rule in rules:
blueprint.add_url_rule(*rule)
return blueprint
If a plugin registered CLI commands it should be migrated from paste command to Click CLI commands. These migrated commands are integrated into the existing ecosystem via the new IClick interface.
class ExtPlugin(p.SingletonPlugin)
p.implements(p.IClick)
# IClick
def get_commands(self):
"""Call me via: `ckan hello`"""
import click
@click.command()
def hello():
click.echo('Hello, World!')
return [hello]
For migrating the frontend resources, the resources registered in resource.config or the files in the fantastic location such as ckanext/ext/fanstatic/script.js and ckanext/ext/fanstatic/style.css had to be loaded into any HTML through webassets.yaml. An example webassets the file looks like this:
ext_script: # name of the asset
filters: rjsmin # preprocessor for files. Optional
output: ckanext-ext/extension.js # actual path, where the generated file will be stored
contents: # list of files that are included in the asset
- script.js
ext_style: # name of asset
output: ckanext-ext/extension.css # actual path, where generated file will be stored
contents: # list of files that are included into asset
- style.css
In the case of fanstatic it is possible to load CSS and JS files into one asset. For webassets those files need to be split into two separate assets. So, after migration to webassets.yaml, the templates from loading resources are updated to assets.
{% block scripts %}
{{ super() }}
{% resource 'ckanext-ext/script.js' %}
{% endblock scripts %}
{% block styles %}
{{ super() }}
{% resource 'ckanext-ext/style.css' %}
{% endblock styles %}
{% block scripts %}
{{ super() }}
{% asset 'ckanext-ext/ext_script' %}
{% endblock scripts %}
{% block styles %}
{{ super() }}
{% asset 'ckanext-ext/ext_style' %}
{% endblock styles %}
After this step is completed, there is a functioning extension and with the final step of re-writing the tests, you can be sure that there were no new bugs introduced in the process. The tests were re-written from nosetests to pytests which meant removing imports from nose.tools to import pytest. The assert statements also had to be changed:
assert_equal(x, y)
eq_(x, y)
assert_in(x, y)
assert_raises(exc, func, *args)
with assert_raises(exc) as cm:
func(*args)
assert 'error msg' in cm.exception
assert x == y
assert x == y
assert x in y
with pytest.raises(exc):
func(*args)
with pytest.raises(RuntimeError) as excinfo:
func(*args)
assert 'error msg' in str(excinfo.value)
The old FunctionalTestBase was replaced by various pytest fixtures. These fixtures are decorators that were added directly to the test class or method. After this last step of migrating the tests was complete, the whole migration of the extension was completed.
Final Thoughts
To sum up, it is an interesting journey where I myself learned a lot and can`t wait for the next upgrades and new challenges. If you have any questions on the migration process, feel free to comment on this blog and we could catch up and discuss your project.
