TDD saved me and I didn’t realise it

U+1F410 GOAT

Years ago at PyCon Ireland there was an enthusiastic workshop on Test-Driven Development (TDD) by Harry Percival. (He’s written a book on it, now in its second edition). The idea is that you write the tests first, show they fail, then write the minimum of code to fix them and then the tests pass. This way everything you write has a test to back it up. I really like the idea, and although I have to force myself to do it, I do like the warm fuzzy feeling of a proper set of tests covering every line I write before I even write it.

Anyway, a minor corner of my mostly-sysadmin job is web development. I’d written a Django project for internal use that used Django’s built-in admin pages as a database front end. I wrote it while Django 1.10 was current, but shortly after I finished it 1.11 was released. I could have left my project as 1.10, but its support life is pretty short, and 1.11 is “LTS”, long-term support, with support for three years.

I bumped my requirements.txt to pull slightly newer versions of three packages, including Django 1.11 instead of 1.10.*

I ran git push to our private GitLab instance, which triggers GitLab-CI to run my tests, and a minute or two later, boom, tests failed. Lots of them. Most of them, actually. I was vaguely reassured that so many had failed, because somewhat counter-intuitively when something is fundamentally broken it’s often quite easy to fix. I read release notes, changelogs, tried various combinations of I thought, hang on, GitLab is really actively developed (there’s a new version every week almost and we’re pretty good and keeping it up to date) so maybe something changed there.

Nothing worked. My buildfix git branch was getting increasingly desperate. The errors from the tests weren’t particularly helpful:

ProgrammingError? Really? Thanks.

It took a while (in my defence this was neither urgent nor the main part of my part-time day job), but I finally properly read the error above the exception:

I had thought until now this was a problem with py.test, or my PostgreSQL service definition in GitLab-CI. Finally I ran:

Well, F*!#, if it didn’t spit out a heap of database changes. Remember I’d updated some of the packages above? One of them was django-simple-history, a nice little package for adding change histories to any Django model. Updated from version 1.8.2 to 1.9, and it seems that changed its database tables. git add/commit/push, tests pass.


  • TDD was correct. My project really was broken, and the automatic tests meant it never got pushed into production.
  • Small version changes sometimes aren’t.
  • Treat all version changes like refactoring. Change one package version at a time.

* I’ll note here that I somewhat disagree with people who say to use the exact versions given by pip freeze, and instead use ranges for things like this:

This will pull the latest of the 1.11 series (1.11.7 today), so you get the latest bug fixes and security fixes, but it should all still work. Certainly in my case I’m doing nothing adventurous enough to cause trouble. And anyway, I’ve got tests!

Solr? Xapian? Something else?

The bibliography

Markup example
I’m not joking about the complex markup
One of the projects in work is a bibliography, a painstakingly-produced list of publications for a specific research area. It wouldn’t be so bad if it was all English-language, but much of it relates to linguistics. There are diacritics all over the place (sometimes stacked on top of each other), mixtures of languages and alphabets within the one title, individually-italicised words, small-caps, superscript, subscript, and combinations of the above. This project started 12 years ago with the intention that, when complete, it would be a printed book. As a result the markup chosen which could handle the mess of markup necessary was LaTeX (remember, this was well before the days of markdown or similar). There could be another ten years of work to go.

The website was designed with a simple alphabetical navigation system — you can browse through authors or journals, etc, by initial letter and it was easy enough to find what you wanted. There’s on-the-fly TeX-to-HTML conversion for the website which uses a recursive regular expression which makes me feel both guilty and proud. But now, after 12 years, there are about 13,000 entries and the navigation has become unwieldy. They want a real search.

Search engines

I threw together a very quick search using a few quick SQL searches (did I mention I work one day per week with support queries of ~20 users to keep happy?) that did a reasonably tolerable job, but really they want to be able to specify multiple fields, wildcards and all that sort of thing. A search of raw database tables won’t work because of all the tex markup.

Solr logoA quick google and my first stop was Apache Solr, which seems to be one of the best-known search engines there is. It has all the features I could think of, and lots of features I hadn’t thought of. My first thought was that it’s huge. It’s a standalone Java application, the zipped download alone is nearly 150MB, and there was a pile of documentation. I succeeded in building an XML-formatted output (including lots of embedded HTML) from the bibliography and got it into Solr, and tinkered with some searches.

In the end, though, I couldn’t shake the feeling that Solr was just too big. There was a lot of configuration. Just setting up Solr as a standalone daemon was a task in itself. Many of the potentially useful features were killed by the nature of our data; language-specific stemming and so on doesn’t work very well if you’ve got English, Irish and Greek mixed together in the one title.

Xapian logoSo I started to look around again and came across Xapian. It’s smaller and more lightweight, has the features I need, and has direct bindings for several languages, including PHP which is what I need. From what I can tell this means I don’t need a separate daemon, the documentation is written with code samples in Python (yay!), and I’m planning to use xml.etree.ElementTree to re-use my existing XML output and stick it into Xapian.

I said I’d have a very basic working example on one or two fields in about three weeks (which for me is the cumulative “spare” time from three busy working days). Wish me luck…

PHP and Unicode

In my last post about the testing goat I mentioned there’s now an official Unicode codepoint for “GOAT”, U+1F410.

At the time, I tried typing it in. Under Linux you just press ctrl-shift-u (you’ll see an underlined letter u), type the hex digits for the code you want, press space and continue on. Easy. Having installed the free Symbola font, I could see my little goat in the editor. Happy Days!

Until I went to preview the post, at which point my little goat, and everything after it, had disappeared. Fortunately it was the last thing in my post, but if it was higher up I’d have lost some of my work. Not good! It was late and I was tired so I left it out, a little disappointed.

So, looking again this evening I found that there’s a known problem that WordPress gets confused if it sees a Unicode character above U+FFFF. If you install the Full UTF-8 plugin, it works again. Without a doubt, this plugin, or something like it, should be merged into the core. Right now.

PHP and Unicode

In my job I have the dubious pleasure of maintaining a very old PHP application. Several hoops are jumped through to keep UTF-8 characters intact, but the hoops still work so I generally just leave it alone. This WordPress issue just had me googling again, and it seems to confirm that PHP (which is the language WordPress is written in) still doesn’t support Unicode natively. Really. In 2014.

It seems that Unicode support for PHP was first proposed in 2005 for what was planned to be PHP 6. Nine years later, and we’re just at 5.6.1. I came across this presentation on Slideshare from 2011 describing how the PHP+Unicode project reached a certain point and just ran out of steam. It seems nothing has happened since.

The nine years of bad history associated with the name “PHP 6” even has people suggesting that the next actual major release of PHP should be called “PHP 7”. It’s that bad.

Conclusion, for now

That PHP application I maintain is well over ten years old. It’s fairly stable, but has accumulated various bits of cruft over time. Adding new features is awkward and really it needs a rewrite. Since it uses lots of international characters I’d really like clean Unicode support, so I’m strongly drawn to using Python 3. It’s nearly 6 years old and supports Unicode properly. Now I’ve to pick a web framework. I’ll probably have a go with Django for now, simply because Harry’s TDD book uses it.

Oh and finally, just because I can, even though WordPress doesn’t want me to, here’s a goat: 🐐

Assuming you’ve got a font for it, of course!

The testing goat?

Test Driven Development with Python at PyConIE14
Test Driven Development with Python at PyConIE14

I’m not new to Python at all. I still have a copy of André Lessa’s Python Developer’s Handbook, which the receipt says I bought on 15th March 2001, and covers Python 1.6. Unfortunately in all the years since I’ve never used it much. My postgrad studies mostly used Verilog and my day job generally involves bash scripts and maintaining some really old PHP.

Still, it’s a language I feel I want to use a lot, and I’ve attended the last two PyCon’s in Ireland. At PyConIE 2013 I went to a tutorial on Test-Driven Development by Harry Percival, and at PyConIE 2014 I won a copy of his book Test-Driven Development in Python. I say won, but simply there were 40 books being given away (20 of these, and 20 of High Performance Python) and I was 40th in the queue. It feels like winning something, at least 🙂

Anyway, at the tutorial 2013 Harry made reference to the “Testing Goat”, and I thought it was just a whimsical idea of his, but the goat was back in 2014 and it’s on the cover of his book.

A bit of googling and it seems the Python Testing Goat is a thing.

As best as i can tell, the Testing Goat was a running joke at PyCon 2010 (see here) and it’s become a mascot for Python Testing ever since. There was even a successful campaign to have O’Reilly put a goat on Harry’s book instead of the expected snake.

U+1F410 GOAT
U+1F410 GOAT

2010 was also the year Unicode 6.0 was released, which added (amongst other things) U+1F410 GOAT. Surely not a coincidence?

Hello world!

After having a blank site here for (too) many years, it’s finally time to put something here. I’m just at the end of PyConIE 2014 and energised to try writing about programming and IT stuff in general. There’s the theory that if you can’t explain something you don’t really understand it, so I’m going to try that approach to writing posts here about whatever I’m working on. We’ll see how it works out!