Category: Uncategorized

Orbital has come a long way since the start of the project. Now, with it being the hub of activity around a research project, it makes more sense for the front end to be called the ‘Researcher Dashboard’. Also named because it’s the interface to the researcher.

We have implemented CKAN more fully, with a few users already using its features. The same will soon be the case with the other technologies we are using. What this also means is that we are underway linking them to the Researcher Dashboard so that the centre point of project management can be created. Soon, the project page will have links to any CKAN datasets belonging to it along with the record in the AMS. Importing projects from the AMS is still on the to do list, but will be done once we get AMS access.

Things are coming together development-wise. It’s now a case of sticking things together, rather than building them from scratch. After realising the size of our workload, we managed to finish off a lot of things and hopefully the rest will follow at a similar pace.

In a previous post, I looked at extending CKAN with a SWORD extension to input and output data via SWORD. This had not gone to plan, as numerous difficulties were encountered.

Orbital Bridge has been developing, and since the last post it’s structure has been planned and more clearly laid out. Each technology it will interface with will be done so via a library. Each of these libraries will have the following functions: CREATE, READ, WRITE and DELETE. The CRUD functions are Create, to create something externally from a Bridge Object, Read to get something externally and transform it into a Bridge Object, Write to update something externally from a Bridge Object and Delete to delete something externally specified in the Bridge Object. This ‘Bridge Object’ is a standard format PHP object used in Orbital-Bridge. This is so it can be sent to and from any library to be used however the library wishes. This creates a standard that any new library can use to connect to external technologies. There will also be a fifth function, RECEIVE, which is a different function from the CRUD functions. Receive takes an object sent from a HTTP POST and changes it to a ‘Bridge Object’. More about this will be documented and posted about later when it has been implemented rather than conceptualised.

The main update to the CKAN/SWORD development is that they are now in two libraries, one for interfacing Orbital-Bridge with SWORD (the ePrints SWORD endpoint) and one for interfacing Orbital-Bridge with CKAN. Using the structure of Orbital-Bridge, deposits can be made in ePrints by using the CKAN library to READ and dataset, turning it into a bridge object, then using the SWORD library to  CREATE the object, after turning it into SWORD XML from the Bridge Object, in ePrints. This structure of Orbital-Bridge has changed the way CKAN and SWORD will talk to one another. Originally the SWORD extension of CKAN was just that, a CKAN extension. Since OKFest and talking to CKAN developers, and with the decision on how Orbital-Bridge will be structure, this is seen as the best way to interface the two technologies.

Nick Jackson and I attended Pycon UK 2012 in Coventry over the weekend of 28th – 30th September. The aim was to get to grips and obtain an understanding of the Python programming language.

The main reason we went was because CKAN was written in python and, as we will be extending it, it makes sense to understand the tools we will be using before we dive in and start to mess about without knowing what we are doing.

There were a number of talks given about how Python is currently being used, the future of the language as well as open spaces to network. I had gone along to a number of these talks, such as Raspberry Pi and Python, Python in a teaching environment and Python powered rich apps. Unfortunately, I missed the talk on practical Artificial Intelligence with Python as I was attending the beginners Python workshop. In a way, this probably proved more useful as I now have a better understanding of the language and its differences. These differences, for example, include how boolean values are case sensitive and the interesting way references work. Any small bit of syntax will be useful later on when programming.

Without expanding on how the programming language operates, I can say that it has been a very easy language to understand, and has been referenced as ‘Executable pseudo code’. Meaning that it basically does what it says on the tin. It is very human readable.

Networking also helped, as we met someone from the MET Office who was very interested in how we were managing research data, as they themselves were having to manage a lot of data. We may have inadvertently sold CKAN to them. We also met, and spoke with, Ross who works on CKAN. After speaking to both, we gained an understanding of what CKAN is like to develop on and how data is managed elsewhere in large quantities.

The Python community proved to be a valuable asset to the Orbital project, thanks to their friendliness towards newcomers to the language and helpfulness with learning it. A lot of speakers also happened to be ex-teachers, which may have helped. I foresee a useful set of connections with this community in the future.

I attended a CKAN workshop OKFest to get to grips with CKAN. It was an interactive session where people can suggest items to run through. The reason I attended this was to get a clearer understanding of how to write extensions for CKAN, especially as I will be writing the SWORD extension in Orbital. I’ve taken a lot of notes during this workshop, which are attached to this post for reference.

A lot was learnt from the afternoon, such as which files should be edited to change difference parts of CKAN. For example, the aptly named plugin.py is where the plugin for CKAN is written. An example of which can be found here – https://github.com/okfn/ckanext-iati/blob/master/ckanext/iati/plugin.py. Templates can be changes easily by changing one variable in plugins.py; def configure_template_directory. This tells CKAN to run these templates, overriding the default ones. Another variable, {% block secondary_content %}, overriders secondary block content, changing the location, colour, size etc of content in the secondary block. Other variables to do with the styling of pages can be chagned to alter the appearance of CKAN and theme it. There is also the web interface, found at /ckan-admin/config. This is just an interface to edit CKAN. So there are a number of different ways to theme CKAN, and I have learnt a lot about where these variables are, what to change them to and the general gist of how to theme and customise.

Extensions are a more important part of CKAN that I went to learn about. Some examples of which are found here: https://github.com/okfn. The plugin.py file also contains variable to do with extensions, such as roots. This is used in conjunction with interfaces.py to set the interfaces that will be used in the extension. These interfaces are used to literally interface with an extension. Packages can be added and modified depending on what the extensions is designed to do. IPackageController modifies the packages, and ‘package_show’package_show_iati – implements a package. These are variables that will be used often with creating an extension. I have only briefly run through these items as there is a lot more depth to them than a blog post should go into.

It was noted that a lot of extensions already exists and are open source, like CKAN. So to write an extension with no experience, it would be a good idea to look at ones already made to see how they are put together. I asked about how to go about making a SWORD extension, as it is likely that a similar extensions already exists. They decided that it was probably best to create it as a web service that CKAN can hook into via APIs. So as long at the web service has APIs that can take one format and spit out another to CKAN it should be fine. Especially as then any other CKAN instance can use it.

The final outcome of this was that the SWORD extension will now be created as a web instance rather than a direct extension of CKAN.

Here are the Notes I took during the workshop.

A working development environment of CKAN has been set up, thourgh it is disconcerting that so many failed attempts has to be worked through first. Below are subsequent attempts to recreate the working setup. They were, however, scuppered by errors encountered in earlier attempts.

 

This is a list of all the actions I undertook to get an instance of CKAN running on Ubuntu 10.04 (On a virtual box machine).

Install Ubuntu 10.04

Close Update Manager (Install no updates)

Open Firefox (Navigate to http://docs.ckan.org/en/ckan-1.7.1/install-from-source.html and http://docs.ckan.org/en/ckan-1.7.1/solr-setup.html#solr-single) For install instructions

Open Terminal

sudo apt-get install mercurial python-dev postgresql libpq-dev

Type password for sudo command

Press Y then return to confirm

sudo apt-get install libxml2-dev libxslt-dev python-virtualenv

Press Y then return to confirm

sudo apt-get install wget build-essential git-core subversion

Press Y then return to confirm

sudo apt-get install solr-jetty openjdk-6-jdk

Press Y then return to confirm

cd ~

virtualenv pyenv

. pyenv/bin/activate

Now line should look like: (pyenv)harry@harry-desktop:~$

Or of equivalent structure

easy_install pip

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan

pip install --ignore-installed -r pyenv/src/ckan/requires/lucid_missing.txt -r pyenv/src/ckan/requires/lucid_conflict.txt

pip install webob==1.0.8

sudo apt-get install python-pybabel python-psycopg2 python-lxml

sudo apt-get install python-pylons python-repoze.who

Y to continue

sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface

Y to continue

deactivate

. pyenv/bin/activate

sudo -u postgres createuser -S -D -R -P ckanuser

pass

pass

Typing ‘pass’ twice is to set the password for the created user

sudo -u postgres createdb -O ckanuser ckantest

cd pyenv/src/ckan

paster make-config ckan development.ini

sudo nano development.ini

CTRL + X

Setting Up Solr

sudo nano /etc/default/jetty

Change the following lines:

NO_START=1 to NO_START=0

#JETTY_HOST=$(uname -n) to JETTY_HOST=127.0.0.1

#JETTY_PORT=8080 to JETTY_PORT=8983

CTRL + O

<RETURN>

CTRL + X

sudo service jetty start

sudo ln -s /home/harry/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml /etc/solr/conf/schema.xml

sudo /etc/init.d/jetty stop

sudo /etc/init.d/jetty start

sudo nano development.ini

sudo nano ~/pyenv/src/ckan/development.ini

Change the following lines:

# ckan.site_id = ckan.net to ckan.site_id = my_ckan_instance

#solr_url = http://127.0.0.1:8983/solr to solr_url = http://127.0.0.1:8983/solr

CTRL + O

<RETURN>

CTRL + X

cd /home/harry/pyenv/src/ckan

Unauthorized. Password mismatch.

Passwords do not mismatch. It is unknown what is causing this error. Config files show that the passwords are correct and the error has no apparent means of being rectified

CKAN Install Aborted

 

 

 

Install Ubuntu 10.04

Close Update manager (Install no updates)

Open Terminal

Optional: Open Firefox, go to: http://docs.ckan.org/en/ckan-1.7.1/install-from-source.html to follow and copy paste commands from.

sudo apt-get install mercurial python-dev postgresql libpq-dev

Type password for sudo command

Press Y then return to confirm

sudo apt-get install libxml2-dev libxslt-dev python-virtualenv

Press Y then return to confirm

sudo apt-get install wget build-essential git-core subversion

Press Y then return to confirm

sudo apt-get install solr-jetty openjdk-6-jdk

Press Y then return to confirm

cd ~

virtualenv pyenv

. pyenv/bin/activate

Now line should look like: (pyenv)harry@harry-desktop:~$

Or of equivalent structure

easy_install pip

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan

pip install webob==1.0.8

sudo apt-get install python-pybabel python-psycopg2 python-lxml

Type password for sudo command

Press Y then return to confirm

sudo apt-get install python-pylons python-repoze.who

Press Y then return to confirm. Twice

sudo apt-get install python-repoze.who-plugins python-tempita python-zope.interface

Press Y then return to confirm. Twice

deactivate

. pyenv/bin/activate

sudo -u postgres psql -l

‘Encoding’ Column should show UTF-8 for all three records

Press ‘:’ (SHIFT + 😉 then ‘q’ to exit this view and back to the terminal

sudo -u postgres createuser -S -D -R -P ckanuser

pass

pass

‘Pass’ has been written twice as the password for the user we just created, and to confirm it.

sudo -u postgres createdb -O ckanuser ckantest

cd pyenv/src/ckan

paster make-config ckan development.ini

sudo nano development.ini

The line ‘sqlalchemy.url = postgresql://ckanuser:pass@localhost/ckantest’ shouild read as written. if not, change it to read as written.

There are two lines commented out under this line (By the #) also called ‘sqlalchemy.url’. Ignore these. Do not change them. The lines should now read as follows:

sqlalchemy.url = postgresql://ckanuser:pass@localhost/ckantest
#sqlalchemy.url = sqlite:///
#sqlalchemy.url = sqlite:///%(here)s/somedfb.db

CTRL + O

<RETURN>

CTRL + X

This saves the file and closes it, returning to the terminal screen.

 

Set up Solr

 

sudo apt-get install solr-jetty openjdk-6-jdk

This should attempt to install solr-jetty, but as it is already installed earlier on it will fail. This is just to ensure that it is installed properly.

sudo nano /etc/default/jetty

Change the following lines:

NO_START=1 to NO_START=0

#JETTY_HOST=$(uname -n) to JETTY_HOST=127.0.0.1

#JETTY_PORT=8080 to JETTY_PORT=8983

CTRL + O

<RETURN>

CTRL + X

sudo service jetty start

Open Firefox, go to http://127.0.0.1:8983

You should see the ‘Welcome to Jetty 6 on Debian’ page

Go to http://127.0.0.1:8983/solr

You should see the ‘Welcome to Solr’ page

Click on the ‘admin’ link (Or go to http://127.0.0.1:8983/solr/admin)

You should see ‘Solr Admin (Example)’.

Go back to the terminal

sudo service jetty stop

sudo service jetty start

sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.bak

sudo ln -s ~/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml /etc/solr/conf/schema.xml

cd ~

sudo service jetty stop

sudo service jetty start

sudo /etc/init.d/jetty stop

sudo /etc/init.d/jetty start

Go to http://127.0.0.1:8982/solr/admin

You should see ‘Solr Admin (ckan)’

sudo nano ~/pyenv/src/ckan/development.ini

Change the following lines:

# ckan.site_id = ckan.net to ckan.site_id = my_ckan_instance

#solr_url = http://127.0.0.1:8983/solr to solr_url = http://127.0.0.1:8983/solr

CTRL + O

<RETURN>

CTRL + X

cd /pyenv/src/ckan

Missing Genshi module

This error has been encountered before, and the module in question was installed along with each subsequent module missing. This got nowhere and eved with an error stating that a function was expecting one argument but was receiving two.

CKAN INSTALL ABORTED