Beginnings of a CKAN SWORD Extension

CKAN

CKAN will be extended to allow applications to input into CKAN using SWORD. It has been decided that this extension will benefit not only Orbital, but several other projects for storing data via SWORD input in the wider CKAN community.

The setup process is outlined in the CKAN Documentation. This has been followed to setup the CKAN source install, which is needed to be able to create the CKAN SWORD extension.

 

CKAN Problems

I have been active in the IRC channel with other CKAN members and got their feedback on creating a CKAN source instance. This was mostly done to overcome a number of errors during the setup. This happened several times, such as with the version being run having a feature depreciated and so not being able to find a file. Or such as the database not being initialised due to a missing component. The install has not yet been completed due to these issues, but an instance of Solr has been implemented.

 

Versions

The documentaion at step 4 instructs to install CKAN source code. The first says how to install the latest code:

pip install --ignore-installed -e git+https://github.com/okfn/ckan.git#egg=ckan
OR for a specific version e.g. for v1.5.1:
pip install --ignore-installed -e git+https://github.com/okfn/ckan.git@ckan-1.7#egg=ckan
The first problem arose here. Of course, the initial thing to think about was the latest code., this would mean that CKAN would be up to date and compatible. However, later in the documentation this is not so. After speaking in the IRC channel it turns out that the requires folder is no longer included in the install in the latest version, but 1.7.1 (Not 1.5.1 as labelled) does have the folder. So this is the version that should be installed.

 

Solr

The Solr instance was easy to follow and implement and connect to over localhost. However, when the schema.xml file was replaced with CKANs, it no longer worked only to throw an error that it could not find teh file when it existed.

The problem turned out to be that the location of thee schema.xml of CKAN was not in the directory specified in the documentation. It was, however, in the install so the correct directory was found and the file moved across. Solr did indeed work, and the problem of the file not existing was that a blank file was created due to the CKAN schema being moved not existing. The documented location of the file is as follows:

~/ckan/ckan/config/solr/schema-1.4.xml
The actual location of the file is thus:
~/pyenv/src/ckan/ckan/config/solr/schema-1.4.xml
Now, the discrepancy between the two locations is not much , as the home directory is just replaced with the pyenv/srv directory. This, when following instructions, is not known as the schema.xml could be in either directory.

Database
Section 9 - Create database tables states to use the following command to initialise the database tables:
paster --plugin=ckan db init
This produced an error, which said that a Ganshi module was missing. After installing the required module, a series of other errors appeared.
pyutilib was the second module missing. After installing this a string of other dependancies were also required.
Migrate.versioning.api was the next, followed by Big Integer. This one was not solved as a dependancy installed did not have the required programming for handling the BigInteger.
Working CKAN Instance
After all these error, a CKAN instance has finally been initialised. It’s unclear to me what what done differently, but it is now working. An Ubuntu image has been saved so the instance will not be lost.
This blog post will be followed up by an account of everything I type and click in order to get CKAN running. Creating a new virtual instance and starting from scratch will be the starting point of the documentation

LNCD Website

The LNCD website was in need to updating, so I have taken the lead in creating the new site. Nick Jackson Provided the new LNCD logo, which was decided on via peer review along with the colour. A number of colours were looked at but there was no real logic to the colour to reflect the group as it was chosen mostly on the number of people who liked it. Majority rules, so it was chosen along with the white version of the logo. Using the CWD, I made a template for the site using the University crest and LNCD logo.

CWD

The CWD has recently been updated by Alex Bilbie to version 4.0. Now it is almost entirely twitter bootstrap, any additions to the bootstrap framework from Twitter can easily be implemented into future versions. So the LNCD site has used this to keep consistency with other sites in the University. The CWD used bootstrap before, but only partially. The new version uses the entirety of Bootstrap so all functionality is available. A custom css file is used in the LNCD site to overwrite any css from the CWD. This is simply for customisation for certain aspects of the site, but not so it is no longer recognised as using CWD.

Blogs and Projects

I have imported Blog posts from the Lincoln blogs platform along with the database of past and present projects. The website can now stay up to date with both these sources. Jamie Mahoney provided some initial content and the method form extracting the projects out of the database.

Changes

The content from the old site was ported driectly across to the new site, but a few changes, additions and deletions have been made.

Kept:

  • Contributors
  • Current Projects
  • Past Projects

Removed:

  • Funding
  • Calendar

The Calendar is still on the site, but is the link to it us currently hidden as there are no upcoming events.

Orbital 0.2 release

The second release of Orbital has gone fairly smoothly. Joss has written an official 0.2 release, here I am commenting on how the release has gone personally. There were a few bugs which got ironed out after the meeting with Bingo, but otherwise Pivotal tracker has shown that we are on track. The bugs were only ones that were unlikely to appear in the final version, for example if no licences were enabled. This still means we needed to fix them, but it meant that it had been overlooked. The number of licences enabled needs to be checked for before displaying the dropdown selection in the edit pages or file upload. No licences meant no licence post request. And of course, breaks Orbital.

The main features of this release are thee dynamic data sets and file collections. Now, files can be organised into file collections, like a group of files, but can also be contained in multiple collections. This was one of the major features Orbital needed to implement, and a feature requested by Bingo. He is happy with the results and specifically noted the timeline. Now, each project you own or are a member of will have a timeline of activity you can view, or add comments to. Mostly, this is to track changes over time in the project. Thoguh we have also added a summary of all activity in all projects you have access to on the home page. Its also been colour coded so you can see which comments are public and private.

Though the clocked number of points tends to fluctuate, the releases have been on time. If visualised, the graph shows the projected work speed as a steady incline, whereas the actual workload appears more like steps. this is because of the amount of work being done increasing at some points and decreasing at others.

A patch, 0.2.1, has been release since. This was fixing the bugs found when testing. Mostly, the licences and other such things were improved. No interface overhaul as it was behind the scenes. Though I have just finished moving teh file upload  to a new page. it now has a nice interface! As a plus, this also fixes the invalid upload token issue that sometimes appears.

The Developer Mindset

Having now been working on the Orbital project for almost two months, I have been slowly getting into the developer mindset. Coming from an Artificial Intelligence and Robotics background, I have always been more focussed on my own work rather than writing about whats going on, blog posts and generally being more active in the developer community. This Blog page aims to be a repository for any thoughts on the project or other progressions or difficulties. A step, I guess, towards being more active.

Having gone to MRD Hack day, I have seen what the community of developers is like when they get together. It was an interesting experience, having made ‘Project Rainbow beam’, an activity data stream, and had a number of talks and collaborations between people on such project. Being more active in the developer community, such as seeking out conferences and hacks days that might benefit me, is encouraged and I intend to do just that.

Getting across what CERD are doing and projects being worked on that benefit the University, or anyone for that matter, is important. If no one know what is being done then the work will go unnoticed.