Best Practices for Creating and Maintaining CKAN Extensions

CKAN (Comprehensive Knowledge Archive Network) is an open-source data management system. It is widely used by governments, organizations, and researchers to manage datasets. One of its greatest strengths is its extensibility, allowing users to add custom features via extensions. However, building CKAN extensions requires careful planning. Maintaining them involves following good practices. This approach ensures they remain stable, easy to use, and compatible with future CKAN versions.

In this post, we’ll walk through the best practices for maintaining CKAN extensions. We will cover how they are created. We will also discuss how they are referenced, organized, and updated.

CKAN Extensions and How They Are Initially Created

CKAN comes with some core extensions packaged within. These extensions don’t need to be installed before you can use them. They’re installed when you install CKAN. They can simply be enabled by following the setup instructions in each extension’s documentation.

In this post we’ll talk about external extensions. These are CKAN extensions that don’t come packaged with CKAN. They must be downloaded and installed separately.

Let’s start from the beginning. We will create the extension. This involves defining new functionality that integrates seamlessly with the CKAN ecosystem. Extensions can range from small changes. For example, you might add a new feature to a dataset or a custom template. They can also involve large integration, such as building a new plugin to handle complex data visualizations.

Steps for Creating a CKAN Extension

1. Identify the Functionality

Start by clearly defining the problem you’re solving or the feature you’re adding. This could involve a custom dataset or organization schema. It could also include templates and layout changes. Additionally, consider complex resource views and visualizations.

2. Create the Extension

We can use the cookiecutter command to create an “empty” extension from a template. Alternatively, use the CLI command: ckan generate extension. A thorough guide on creating and empty CKAN extension can be found in the official CKAN documentation.

3. Use CKAN’s Extension Template

CKAN offers a standard directory structure that developers can use to create extensions. This structure is essential for ensuring that your extension integrates smoothly with CKAN’s framework.

my_extension/
├── ckanext/
│   └── my_extension/
├── setup.py
├── MANIFEST.in
└── README.md

• ckanext/my_extension/ — This folder contains the core code of your extension
• setup.py — This is a Python script used to install the extension
• MANIFEST.in — A configuration file specifying which files should be included in the package
• README.md — A documentation file for users and developers

4. Develop the Extension

Write your Python code, templates, and any additional resources. Extensions can include models, views, and other components that integrate into CKAN’s web interface.

5. Register the Extension in CKAN

Once your extension is developed, you need to register it within CKAN’s configuration. This involves adding your extension to CKAN’s ckan.ini or production.ini configuration file in the ckan.plugins section.

How the Code is Structured and Organized

A well-structured CKAN extension is critical for maintainability, readability, and scalability. A typical CKAN extension follows the Model-View-Template architecture.

This is the suggested structure:

• Models — Define the core data structures and business logic. For example, we might create a custom model to interact with CKAN’s database or extend CKAN’s existing models
• Views — Handle user input and interaction with the models. If your extension requires custom routes, this is where they would be defined
• Templates — Where you define the HTML output for our extension. CKAN uses Jinja for templating. The file structure for the CKAN extension templates must match exactly. These templates are overwriting the default CKAN ones. We can also use snippets. They are small fragments of HTML code. These snippets can be pulled into any template. These are kept in a snippets directory within the same folder as the actions that are using them. More generic snippets are added to templates/snippets. Check the full documentation on templating for more details
• Assets — Contain custom JavaScript, CSS, and image files for the front-end customizations. Those are added using webassets . The assets folder contains a webassets.yml file where all the configuration for the custom JavaScript and CSS is done. Check the full documentation on assets for more details

A well-organized extension should use separate modules or files for different parts of the codebase. This keeps functions and classes clearly defined. They are easy to find. For example:

my_extension/
├── ckanext/
│   └── my_extension/
│       ├── plugin.py
│       ├── assets/
│       ├── public/
│       ├── i18n/
│       ├── templates/
│       ├── views/
│       ├── tests/     
└── setup.py

This structure from the example above allows easy expansion or modification of parts of your extension. This modification does not break other components.

How Basic CKAN Functionality Can Be Extended

CKAN offers numerous ways to extend its functionality without rewriting large sections of its code. Here are a few common approaches:

Adding Custom Actions

You can extend CKAN’s actions by defining new ones in your extension. Actions are Python functions that perform operations like creating, updating, or deleting data. For instance, you could create a custom action to process a dataset in a unique way.

from ckan.plugins import toolkit

def my_custom_action(context, data_dict):
    # Custom logic here
    return {'success': True}

Extending the Web Interface

CKAN’s web interface can be extended by creating custom controllers and views. You can add new pages, modify existing ones, or provide custom data visualizations. This is particularly useful for building custom dataset views, data entry forms or visualizations.

from ckan.plugins import toolkit

class MyController(toolkit.BaseController):
    def my_custom_view(self):
        # Return a custom template
        return toolkit.render('my_custom_template.html')

Versioning, Testing, and Automation

Maintaining CKAN extensions requires careful versioning, testing, and automation. These steps ensure that they work well with future versions of CKAN. They also remain backward-compatible and remain bug-free.

Versioning

When developing an extension, it’s important to follow Semantic Versioning (SemVer). This practice helps other developers and users of the extension. They can understand when backward-incompatible changes have been made. It also shows when there are new features or bug fixes.

• Major version — Introduce backward-incompatible changes
• Minor version — Add functionality in a backward-compatible manner
• Patch version — Fix backward-compatible bugs

For example, an update from 1.0.0 to 1.1.0 means new features were added, while a move from 1.0.0 to 2.0.0 means breaking changes were introduced.

We must ensure that the version number is reflected in both your setup.py and README.md files.

Testing

When developing a CKAN extension, testing is a crucial step. It ensures that the extension works as expected. It also integrates well with CKAN’s core functionality. CKAN extensions typically use testing frameworks, such as pytest and unittest, along with CKAN’s built-in testing utilities to facilitate development.

CKAN can be run in a Docker container to facilitate testing the extension in an isolated environment which allows us to set up CKAN, along with any dependencies (e.g., a database), for a clean and repeatable testing process.

Here is a list of some of the best practices for writing tests in the CKAN extension:

• Write Tests for Core Features — Focus on testing the core functionality of your extension, including actions, validation, and API endpoints
• Use Fixtures and Mocking — Leverage fixtures to set up test environments and mock external dependencies to isolate tests
• Keep Tests Isolated — Ensure each test is independent and doesn’t depend on the result of another test
• Test with Multiple Scenarios — Consider testing edge cases, error conditions, and expected behavior

For more information, please check the full documentation for CKAN testing coding standards.

Automation

Automating the build, testing, and deployment of your CKAN extension is essential for maintaining quality over time. We can use tools like GitHub Actions or Travis CI. These tools automatically run your tests. They ensure your extension works with every commit. We can also use pre-commit hooks. They ensure code quality checks are done before committing changes. These checks include linting, formatting, and running tests.

Here’s an example .github/workflows/ci.yml file for GitHub Actions:

name: CKAN Extension CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -e .

      - name: Run tests
        run: pytest

Additionally, we can consider setting up automated deployments to the CKAN instance after testing.

Documentation

Maintaining comprehensive and up-to-date documentation that covers installation, configuration, usage, and troubleshooting of the CKAN extension is crucial. It ensures that your extension can be easily understood, maintained, and extended by other developers. This will help both developers and users understand how to work with the extension effectively. As a good practice the documentation should include the following segments:

• Extension Summary
• Installation Instructions
• Examples of Usage
• Contribution Guidelines
• Changelog
• Common Issues and FAQ
• License and Legal Information

Well-maintained documentation will also help ensure the long-term success and usability of our extension in the CKAN ecosystem.

Conclusion

Maintaining CKAN extensions requires careful attention to code organization, versioning, testing, and automation. By following best practices, we can create CKAN’s functionality effectively. This allows us to build stable, high-quality extensions. These extensions remain compatible with future CKAN releases. They provide lasting value to your users. Always prioritize good code structure. Implement automated testing and maintain clear documentation. This ensures the long-term success of your CKAN extensions and provides value to the CKAN ecosystem for years to come.