Introduction
Before I share my story of “Me vs. Travis for CKAN’s datastore extension”, let me give you a bit of background into CKAN and me. I am Andrea Bozarevska, a software developer, dog lover and origami and crochet enthusiast. I’ve been part of Keitaro’s team for almost a year and in that time most of the projects I’ve worked on were CKAN based data solutions.
The Comprehensive Knowledge Archive Network (CKAN) is the world’s leading open source data management system, a powerful data portal platform used widely by governments, research institutions and other organizations that collect lots of data and want to make their data accessible and available to others and especially the general public.
Simply put, CKAN is a fully-featured software solution for creating open data websites and it provides tools making it easy to publish, share and use data. It is built with Python on the backend and Javascript on the frontend, and it maintains information about the data sets in PostgreSQL databases. CKAN has a modular architecture and offers a powerful API that allows extensions to be developed to provide additional features such as harvesting or data upload.
CKAN and the problem
The core of CKAN can be flexibly extended, meeting almost our every need. We can add social tools, data quality checking, integration with third party tools and services etc.
One of Keitaro’s main areas of expertise is developing and maintaining solutions built with CKAN, so naturally we have used it in many of the solutions suggested to clients, such as Amplus.
CKAN has many stable versions, however, people are solving CKAN bugs and are introducing new features and ideas almost every day. One of the problems I had while working on a CKAN project, was trying to make the datastore extension for CKAN work with Travis on git. After 30+ commits in which I was trying to find a way to reason with Travis, he finally accepted my suggestion.
While writing the Travis build/run scripts, we have to be cautious with the order of commands. Also, some of CKAN’s new versions require us to do specific things, like recreate the database tables, set the permissions again, etc.
Overcoming
In my case, I had trouble incorporating the datastore extension in the unit tests. The datastore extension is very easy to use with CKAN and works great with other extensions. However, when we move to the testing, it gets a bit more complicated, since we have to mock a few things.
The first thing I did was load and unload the extension at the beginning, this solving most of the errors I had (the program could not locate some functions and many datastore-related methods were inaccessible). Now I could continue working locally without any problems and it seemed to me that nothing else could stand in the way.
As I was committing my changes a few days later, I met Travis and we hung out the whole day (on a Friday!). Travis CI is a hosted continuous integration service and it is used to build and test projects on GitHub.
Travis did not care about the loading of the extension, or the other additional imports I made. I tried telling him the location again in the shell script, changed the directories a few times, even tried to force him to run the datastore service, by adding the start command in the Travis run script, but nothing worked. I even changed the roles for the created databases, created new users, but all I got from Travis were fatal errors colored in red.
So, I turned to other methods. Since I had to test the code on git, I did many (unsuccessful) commits. I tried sweet-talking Travis into accepting my code, by explicitly stating the location of the extension and a few other packages required. I even sang to him in the commit messages. And lucky for me, it worked.
The issue was resolved by setting datastore permissions again, right after the initialising of the databases. This is done because in the newer versions, it is required to set the permissions again, since some updates clear them. I also changed and reinstalled some packages that got installed with other extensions, and needed to be downgraded / upgraded. It was explicitly written which ones were the wrong version, so I only had to put them in the right place in the script.
In the end, there are two things we need to remember when encountering this kind of problem.
- Travis is a Queen fan and the song that got to him was “Don’t stop me now“.
- It is very important to read everything from CKAN’s documentation and be very careful of the versions you are using and know all the things that have been changed from one version to another.
