Author: Harry Newton

Nick Jackson and I attended Pycon UK 2012 in Coventry over the weekend of 28th – 30th September. The aim was to get to grips and obtain an understanding of the Python programming language.

The main reason we went was because CKAN was written in python and, as we will be extending it, it makes sense to understand the tools we will be using before we dive in and start to mess about without knowing what we are doing.

There were a number of talks given about how Python is currently being used, the future of the language as well as open spaces to network. I had gone along to a number of these talks, such as Raspberry Pi and Python, Python in a teaching environment and Python powered rich apps. Unfortunately, I missed the talk on practical Artificial Intelligence with Python as I was attending the beginners Python workshop. In a way, this probably proved more useful as I now have a better understanding of the language and its differences. These differences, for example, include how boolean values are case sensitive and the interesting way references work. Any small bit of syntax will be useful later on when programming.

Without expanding on how the programming language operates, I can say that it has been a very easy language to understand, and has been referenced as ‘Executable pseudo code’. Meaning that it basically does what it says on the tin. It is very human readable.

Networking also helped, as we met someone from the MET Office who was very interested in how we were managing research data, as they themselves were having to manage a lot of data. We may have inadvertently sold CKAN to them. We also met, and spoke with, Ross who works on CKAN. After speaking to both, we gained an understanding of what CKAN is like to develop on and how data is managed elsewhere in large quantities.

The Python community proved to be a valuable asset to the Orbital project, thanks to their friendliness towards newcomers to the language and helpfulness with learning it. A lot of speakers also happened to be ex-teachers, which may have helped. I foresee a useful set of connections with this community in the future.

I attended a CKAN workshop OKFest to get to grips with CKAN. It was an interactive session where people can suggest items to run through. The reason I attended this was to get a clearer understanding of how to write extensions for CKAN, especially as I will be writing the SWORD extension in Orbital. I’ve taken a lot of notes during this workshop, which are attached to this post for reference.

A lot was learnt from the afternoon, such as which files should be edited to change difference parts of CKAN. For example, the aptly named plugin.py is where the plugin for CKAN is written. An example of which can be found here – https://github.com/okfn/ckanext-iati/blob/master/ckanext/iati/plugin.py. Templates can be changes easily by changing one variable in plugins.py; def configure_template_directory. This tells CKAN to run these templates, overriding the default ones. Another variable, {% block secondary_content %}, overriders secondary block content, changing the location, colour, size etc of content in the secondary block. Other variables to do with the styling of pages can be chagned to alter the appearance of CKAN and theme it. There is also the web interface, found at /ckan-admin/config. This is just an interface to edit CKAN. So there are a number of different ways to theme CKAN, and I have learnt a lot about where these variables are, what to change them to and the general gist of how to theme and customise.

Extensions are a more important part of CKAN that I went to learn about. Some examples of which are found here: https://github.com/okfn. The plugin.py file also contains variable to do with extensions, such as roots. This is used in conjunction with interfaces.py to set the interfaces that will be used in the extension. These interfaces are used to literally interface with an extension. Packages can be added and modified depending on what the extensions is designed to do. IPackageController modifies the packages, and ‘package_show’package_show_iati – implements a package. These are variables that will be used often with creating an extension. I have only briefly run through these items as there is a lot more depth to them than a blog post should go into.

It was noted that a lot of extensions already exists and are open source, like CKAN. So to write an extension with no experience, it would be a good idea to look at ones already made to see how they are put together. I asked about how to go about making a SWORD extension, as it is likely that a similar extensions already exists. They decided that it was probably best to create it as a web service that CKAN can hook into via APIs. So as long at the web service has APIs that can take one format and spit out another to CKAN it should be fine. Especially as then any other CKAN instance can use it.

The final outcome of this was that the SWORD extension will now be created as a web instance rather than a direct extension of CKAN.

Here are the Notes I took during the workshop.

A working development environment of CKAN has been set up, thourgh it is disconcerting that so many failed attempts has to be worked through first. Below are subsequent attempts to recreate the working setup. They were, however, scuppered by errors encountered in earlier attempts.

This is a list of all the actions I undertook to get an instance of CKAN running on Ubuntu 10.04 (On a virtual box machine).

Install Ubuntu 10.04

Close Update Manager (Install no updates)

Open Firefox (Navigate to http://docs.ckan.org/en/ckan-1.7.1/install-from-source.html and http://docs.ckan.org/en/ckan-1.7.1/solr-setup.html#solr-single) For install instructions

Open Terminal

sudo apt-get install mercurial python-dev postgresql libpq-dev

Type password for sudo command

Press Y then return to confirm

sudo apt-get install libxml2-dev libxslt-dev python-virtualenv

Press Y then return to confirm

sudo apt-get install wget build-essential git-core subversion

Press Y then return to confirm

sudo apt-get install solr-jetty openjdk-6-jdk

Press Y then return to confirm

cd ~

virtualenv pyenv

. pyenv/bin/activate

Now line should look like: (pyenv)harry@harry-desktop:~$

Or of equivalent structure

easy_install pip

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan

pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt

pip install webob==1.0.8

sudo apt-get install python-pybabel python-psycopg2 python-lxml

sudo apt-get install python-pylons python-repoze.who

Y to continue

sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface

Y to continue

deactivate

. pyenv/bin/activate

sudo -u postgres createuser -S -D -R -P ckanuser

pass

pass

Typing ‘pass’ twice is to set the password for the created user

sudo -u postgres createdb -O ckanuser ckantest

cd pyenv/src/ckan

paster make-config ckan development.ini

sudo nano development.ini

CTRL + X

Setting Up Solr

sudo nano /etc/default/jetty

Change the following lines:

NO_START=1 to NO_START=0

#JETTY_HOST=$(uname -n) to JETTY_HOST=127.0.0.1

#JETTY_PORT=8080 to JETTY_PORT=8983

CTRL + O

<RETURN>

CTRL + X

sudo service jetty start

sudo ln -s /home/harry/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml /etc/solr/conf/schema.xml

sudo /etc/init.d/jetty stop

sudo /etc/init.d/jetty start

sudo nano development.ini

sudo nano ~/pyenv/src/ckan/development.ini

Change the following lines:

# ckan.site_id = ckan.net to ckan.site_id = my_ckan_instance

#solr_url = http://127.0.0.1:8983/solr to solr_url = http://127.0.0.1:8983/solr

CTRL + O

<RETURN>

CTRL + X

cd /home/harry/pyenv/src/ckan

Unauthorized. Password mismatch.

Passwords do not mismatch. It is unknown what is causing this error. Config files show that the passwords are correct and the error has no apparent means of being rectified

CKAN Install Aborted

Install Ubuntu 10.04

Close Update manager (Install no updates)

Open Terminal

Optional: Open Firefox, go to: http://docs.ckan.org/en/ckan-1.7.1/install-from-source.html to follow and copy paste commands from.

sudo apt-get install mercurial python-dev postgresql libpq-dev

Type password for sudo command

Press Y then return to confirm

sudo apt-get install libxml2-dev libxslt-dev python-virtualenv

Press Y then return to confirm

sudo apt-get install wget build-essential git-core subversion

Press Y then return to confirm

sudo apt-get install solr-jetty openjdk-6-jdk

Press Y then return to confirm

cd ~

virtualenv pyenv

. pyenv/bin/activate

Now line should look like: (pyenv)harry@harry-desktop:~$

Or of equivalent structure

easy_install pip

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan

pip install webob==1.0.8

sudo apt-get install python-pybabel python-psycopg2 python-lxml

Type password for sudo command

Press Y then return to confirm

sudo apt-get install python-pylons python-repoze.who

Press Y then return to confirm. Twice

sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface

Press Y then return to confirm. Twice

deactivate

. pyenv/bin/activate

sudo -u postgres psql -l

‘Encoding’ Column should show UTF-8 for all three records

Press ‘:’ (SHIFT + 😉 then ‘q’ to exit this view and back to the terminal

sudo -u postgres createuser -S -D -R -P ckanuser

pass

pass

‘Pass’ has been written twice as the password for the user we just created, and to confirm it.

sudo -u postgres createdb -O ckanuser ckantest

cd pyenv/src/ckan

paster make-config ckan development.ini

sudo nano development.ini

The line ‘sqlalchemy.url = postgresql://ckanuser:pass@localhost/ckantest’ shouild read as written. if not, change it to read as written.

There are two lines commented out under this line (By the #) also called ‘sqlalchemy.url’. Ignore these. Do not change them. The lines should now read as follows:

sqlalchemy.url = postgresql://ckanuser:pass@localhost/ckantest
#sqlalchemy.url = sqlite:///
#sqlalchemy.url = sqlite:///%(here)s/somedfb.db

CTRL + O

<RETURN>

CTRL + X

This saves the file and closes it, returning to the terminal screen.

Set up Solr

sudo apt-get install solr-jetty openjdk-6-jdk

This should attempt to install solr-jetty, but as it is already installed earlier on it will fail. This is just to ensure that it is installed properly.

sudo nano /etc/default/jetty

Change the following lines:

NO_START=1 to NO_START=0

#JETTY_HOST=$(uname -n) to JETTY_HOST=127.0.0.1

#JETTY_PORT=8080 to JETTY_PORT=8983

CTRL + O

<RETURN>

CTRL + X

sudo service jetty start

Open Firefox, go to http://127.0.0.1:8983

You should see the ‘Welcome to Jetty 6 on Debian’ page

Go to http://127.0.0.1:8983/solr

You should see the ‘Welcome to Solr’ page

Click on the ‘admin’ link (Or go to http://127.0.0.1:8983/solr/admin)

You should see ‘Solr Admin (Example)’.

Go back to the terminal

sudo service jetty stop

sudo service jetty start

sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.bak

sudo ln -s ~/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml /etc/solr/conf/schema.xml

cd ~

sudo service jetty stop

sudo service jetty start

sudo /etc/init.d/jetty stop

sudo /etc/init.d/jetty start

Go to http://127.0.0.1:8982/solr/admin

You should see ‘Solr Admin (ckan)’

sudo nano ~/pyenv/src/ckan/development.ini

Change the following lines:

# ckan.site_id = ckan.net to ckan.site_id = my_ckan_instance

#solr_url = http://127.0.0.1:8983/solr to solr_url = http://127.0.0.1:8983/solr

CTRL + O

<RETURN>

CTRL + X

cd /pyenv/src/ckan

Missing Genshi module

This error has been encountered before, and the module in question was installed along with each subsequent module missing. This got nowhere and eved with an error stating that a function was expecting one argument but was receiving two.

CKAN INSTALL ABORTED

CKAN

CKAN will be extended to allow applications to input into CKAN using SWORD. It has been decided that this extension will benefit not only Orbital, but several other projects for storing data via SWORD input in the wider CKAN community.

The setup process is outlined in the CKAN Documentation. This has been followed to setup the CKAN source install, which is needed to be able to create the CKAN SWORD extension.

CKAN Problems

I have been active in the IRC channel with other CKAN members and got their feedback on creating a CKAN source instance. This was mostly done to overcome a number of errors during the setup. This happened several times, such as with the version being run having a feature depreciated and so not being able to find a file. Or such as the database not being initialised due to a missing component. The install has not yet been completed due to these issues, but an instance of Solr has been implemented.

Versions

The documentaion at step 4 instructs to install CKAN source code. The first says how to install the latest code:

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git#egg=ckan

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan

Solr

The Solr instance was easy to follow and implement and connect to over localhost. However, when the schema.xml file was replaced with CKANs, it no longer worked only to throw an error that it could not find teh file when it existed.

The problem turned out to be that the location of thee schema.xml of CKAN was not in the directory specified in the documentation. It was, however, in the install so the correct directory was found and the file moved across. Solr did indeed work, and the problem of the file not existing was that a blank file was created due to the CKAN schema being moved not existing. The documented location of the file is as follows:

~/ckan/ckan/config/solr/schema-1.4.xml

The actual location of the file is thus:

~/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml

Now, the discrepancy between the two locations is not much , as the home directory is just replaced with the pyenv/srv directory. This, when following instructions, is not known as the schema.xml could be in either directory.

Database

Section 9 - Create database tables states to use the following command to initialise the database tables:

paster --plugin=ckan db init

The LNCD website was in need to updating, so I have taken the lead in creating the new site. Nick Jackson Provided the new LNCD logo, which was decided on via peer review along with the colour. A number of colours were looked at but there was no real logic to the colour to reflect the group as it was chosen mostly on the number of people who liked it. Majority rules, so it was chosen along with the white version of the logo. Using the CWD, I made a template for the site using the University crest and LNCD logo.

CWD

The CWD has recently been updated by Alex Bilbie to version 4.0. Now it is almost entirely twitter bootstrap, any additions to the bootstrap framework from Twitter can easily be implemented into future versions. So the LNCD site has used this to keep consistency with other sites in the University. The CWD used bootstrap before, but only partially. The new version uses the entirety of Bootstrap so all functionality is available. A custom css file is used in the LNCD site to overwrite any css from the CWD. This is simply for customisation for certain aspects of the site, but not so it is no longer recognised as using CWD.

Blogs and Projects

I have imported Blog posts from the Lincoln blogs platform along with the database of past and present projects. The website can now stay up to date with both these sources. Jamie Mahoney provided some initial content and the method form extracting the projects out of the database.

Changes

The content from the old site was ported driectly across to the new site, but a few changes, additions and deletions have been made.

Kept:

• Contributors
• Current Projects
• Past Projects

Removed:

• Funding
• Calendar

The Calendar is still on the site, but is the link to it us currently hidden as there are no upcoming events.