Tuesday, 10 March 2009

Running debugger inside python unittests

I have been a fan of test driven development for a number of years now.

Where it really shines is in an environment with a library or set of libraries that interact with each other to give the programs actions.

The current place I work use a complicated system written in python. I have been asked to make some changes inside this. Naturally my first thought was to isolate the particular functions I have to change in a unit test

The change I have to make isn't important to know for this article but I may as well explain it briefly. The modules implement both client and server for a system transmitting data over a variety of transports. Currently the bulky part of the data is sent with UDP. For a number of reasons too tedious to recount this needs to be rewritten to use TCP/IP instead. As UDP is in many ways a simpler and more basic protocol the changes will be moderately complicated

After a bit of effort I managed to get a test program working that more or less compiles using the python "unittest" module. "unittest" works in the familar way that perl .t files work or Kent Becks xunit stuff or junit or...I could go on. The doctest method seems more popular in the python community but I prefer doing stand alone unittest programs

So I did a unittest program which like I said, compiled. But it didn't run. It was throwing an error somewhere in the guts of the modules. If I was trying to debug the production code directly ( I was doing this last week :) ), for example with an error at line 36 of the foo module then I'd run this


python -m pdb service
(Pdb) b foo.py:36
Breakpoint 1 at /home/jamie/fix/foo.py:36
(Pdb) c


The program would run and then break at where the error was so variables can be inspected etc. In some situations this is a very useful method

However, if I try this on a program that is a unittest it simply gives me a cryptic message about "The program exited via sys.exit(). Exit status: False", it doesn't stop at the requested line and gives no clues.

I tried to use the "debug" method mentioned in the unittest documentation. I tried all kinds of things in the (Pdb) shell but with no helpful effect.

I did STFW for a long time before I found a comment from "orip" at Stackoverflow
This suggests using the testoob python module. So I installed it and now, running

testoob --debug datachanneltest.py
Runs my unittest program AND opens a (Pdb) line when an error occurs, allowing variable inspection etc. Fantastic!

Of course a fix that marvellous and easy has it's problems. If there is an error in a library function like the socket getpeername() it can't quite manage to DTRT but seems to jam up inside the call and doesn't allow access to the python stack above. Maybe if i knew more Pdb voodoo it might.

Summary: testoob is recommended for running tests with the python unittest module

Thursday, 5 March 2009

Gentoo on a USB Stick

Recently I was given the job of installing gentoo Linux on a new piece of equipment. The target was a tablet PC. It had no keyboard but ran Windows XP on a touch screen.

The lack of the keyboard was easy to fix, it had a couple of USB ports.

To get it to run Linux was a bit more interesting. It didn't have a CDROM.
I had three options
  1. Copy install files to the XP disk and boot using syslinux or similar
  2. Boot from a USB CDROM
  3. Boot from a USB memory stick
I went for option #3

On the gentoo site there were notes on how to make gentoo work on a USB stick. In particular this page http://en.gentoo-wiki.com/wiki/Gentoo_minimal_CD_on_USB_stick has details on two methods for making a bootable gentoo. The images it aims to set up are install images from CDs. The first method uses the normal install CD with extlinux and the second a rescue CD with syslinux.

extlinux is a simple method for booting from a ext2 file system. syslinux is a simple method for booting from a fat/fvat/other MS filesystems.

On the target hardware I was using the first method with extlinux seemed to work well. The syslinux method didn't.

We wanted to keep the Windows XP installed and use just part of the disk for running Linux. Fortunately there was a lot of free space and there is a great howto for doing the operation I wanted here

The target machine is a little underpowered compared to my desktop. Also wasn't clear how easy it would be to connect the target machine to the network. So my preferred method of working would be to set up a fake environment on the desktop machine running gentoo "emerge" to get and compile packages and then copying that over to the target via the usb stick.

There are instructions for a similar method of working using the ASUS EE here
I used a script to copy the files to the usb stick as they changed.

Once I had this all in place it was just a case of ploughing through the listed devices and ensuring the kernel was doing the right thing and that user space things like bluez-tools were loaded

After a few days the machine was running- job done!

Tuesday, 17 February 2009

Feeling Flushed

I just had a little bug fix to do on our embedded product

It uses python with pyrex extensions to drive a small screen in an embedded device. One of the dialogs was not working.

First I wrote a test program that tried to display the non working "DecisionDialog" dialog.
Next, from one of the classes that "DecisionDialog" was using I added print statements.
Fortunately the test program was crashing with an error message too so I kept adding more print statements to hop to the next part of the classes to see where it got confused. Normally this isn't the greatest way of proceeding but the classes have no unit testing or other features that might help.

However, at one point I seemed to be adding more print statements and not getting any extra output

I guessed that there was some problem brought on by the code suddenly stopping with the buffering of the print statements. The code was in a mixture of C, python and python pyrex to bind it together. The problem was in a C code segment so I added

fflush(NULL);

Just after the printf statements that I hoped would help me.

This worked and eventually I managed to find the problem, an unassigned string to a graphics resource.

The bug fix for the dialog was pretty simple (one line in a __init__() function) so I checked out a clean copy of the code, loosing all the print statement rubbish, fixed the clean copy, checked in again and that was that.

When using this crude print method to trace bugs it is worth remembering the buffering layer

Thursday, 11 December 2008

Adding dojo tabbed forms to django admin

The django system provides, "out of the box" an admin system that gives webpages with CRUD functionality with next to no work. This is great for me as I have to implement an in house form management system.

Of course I'd like to take that admin system as a basis and make it do what I want- without having to rewrite it

My latest idea of how to do this is to add dojo into django and then have dojo tart up the forms. There is already a django distribution that does dojo integration called dojoango. I wasn't keen on this as it makes me dependant on a third party source who are in turn dependant on django and dojo. So while dojo and django are stable enough to use, the extra risk of a intermediate package doesn't seem worth it for what I want to do. Anyway...

I've used ajax style javascript / dhtml / css manipulation before but not in dojo. I was keen on using dojo as it abstracts a lot of the problems of javascript such as browser imcompatiblity. This is probably true of most javascript libraries. I thought that the docs for dojo were reasonable too.

However, after a week of trying to learn dojo purely from online sources I found two problems. First, all the docs seemed to assume that you were already a top notch javascript developer and/or familiar with dojo. Secondly, I couldn't Cargo Cultishly copy example programs as the versions of the library functions seem to alter quite briskly. Examples for version 1.0.0 from a year or so ago don't work with the current at the time of writing 1.2.2. This was essentially a problem of not being able to understand enough of this comprehensive framework to get started. But fortunately I talked to the right person on the freenode irc channel #dojo and they recommended the book "Mastering Dojo: JavaScript and Ajax Tools for Great Web Experiences". After a weekend reading this I had some idea of how to do what I wanted

The forms I have to implement with django are quite extensive and in the default admin interface come out as one large page. But with dojo it is relatively easy to split the page up into "tabs"

The tricky bit is to add the javascript that makes dojo work to the right template files.
To find the template files I looked at the html source of the existing admin interface. Then I grepped for unusual strings in the django contrib/admin/templates/ directory. Once I had more or less the right files I copied them to my own private template directory ( configured in django settings.py )

First I added a bunch of stuff to the "base.html" file so that the dojo libraries and styles were loaded before I did anything else:


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="{{ LANGUAGE_CODE }}" xml:lang="{{ LANGUAGE_CODE }}" {% if LANGUAGE_BIDI %}dir="rtl"{% endif %}>
<head>
<style type="text/css">
@import "http://localhost:8000/dijit/themes/nihilo/nihilo.css";
</style>

<script type="text/javascript" src="http://localhost:8000/dojo/dojo.js" djConfig="parseOnLoad:true, isDebug: true"></script>
<script type="text/javascript">
dojo.require("dojo.parser")
dojo.require("dijit.layout.TabContainer")
dojo.require("dijit.layout.ContentPane")
</script>
<style>
.formContainer{
width:800px;
height:700px;
}
label {
width:150px;
float:left;
}
</style>
<title>{% block title %}{% endblock %}</title>
<link rel="stylesheet" type="text/css" href="{% block stylesheet %}{% load adminmedia %}{% admin_media_prefix %}css/base.css{% endblock %}" />
{% if LANGUAGE_BIDI %}<link rel="stylesheet" type="text/css" href="{% block stylesheet_rtl %}{% admin_media_prefix %}css/rtl.css{% endblock %}" />{% endif %}
{% block extrastyle %}{% endblock %}
{% block extrahead %}{% endblock %}
{% block blockbots %}<meta name="robots" content="NONE,NOARCHIVE" />{% endblock %}
</head>
{% load i18n %}

<body class="nihilo">

<!-- Container -->
<div id="container">

The rest of the pre modified admin/templates/base.html file continues after the div id container tab

Next I altered the change_form.html container. Here is the fragment, starting from just before the repeated call to the fieldset template:

<div class="formContainer" dojoType="dijit.layout.TabContainer">
{% for fieldset in adminform %}
{% include "admin/includes/fieldset.html" %}
{% endfor %}

{% block after_field_sets %}{% endblock %}

{% for inline_admin_formset in inline_admin_formsets %}
{% include inline_admin_formset.opts.template %}
{% endfor %}

{% block after_related_objects %}{% endblock %}
</div> <!-- end of dijit tabcontainer -->


The above fragment sets up a context or container for enclosed dojo elements to be treated as tabs in a tabbed form.

Then I drilled into the admin/includes/fieldset.html and edit_inline/tabular.html templates and added the dojo magic to define the individual tabs. Basically for the fieldset.html I added this single line at the start of the file:

<div dojoType="dijit.layout.ContentPane" title="{{fieldset.name}}" >

and a closing div tag at the end of the file. I did a similar thing in the edit_inline/tabular.html file.

The only gotcha I found with trying to do this was the way that the dijit component did not like other elements as direct parents/children. If the container is immediately followed by a contentpane and then that contentpane is immediately followed by a another then there is no problem. If there are other bits and pieces in between then the container gets confused. Obviously when modifying a set of templates this can be an issue.

Next I had to alter the way that the admin form was declared to use fieldsets. This is covered in the django docs Each fieldset or inline group appears in its own tab.

Here's a screenshot of the changed form.


Monday, 8 December 2008

How to write a Linux Kernel ioctl

The company I'm working with at the moment have an embedded Linux product

The hardware is supplied by a third party and although they have supplied a pretty good, reliable and comprehensive set of drivers we wanted to add a method for detecting the cable being plugged in the ethernet

The ethernet driver lacked this of course but we have a memory map and thought that it might be available somewhere in there

First I had to find how to add the ioctl code to the device driver. This proved pretty easy! The difficult bit was working out what kernel call got me the data and how to copy the data back to user space.

First, find the init function for the module. Next add a line like this

dev->do_ioctl=&queryisrreg_ioctl;

Where queryisrreg_ioctl is the name of the function to carry out your ioctl needs

here is my function


static int queryisrreg_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) {
static short int c=0xbb;
switch(cmd) {
case SIOCDEVPRIVATE: /* allowed to use this for DIY ioctl */
c=__EVA_REG(MAC_MEM_OFFSET+0xc4);
rq->ifr_flags=c;
return 0;
default:
return -EOPNOTSUPP;
}
return 0; /* this should never be reached..*/
}


So it uses the magic device driver call "_EVA_REG(MAC_MEM_OFFSET+0xc4);" (which i hope is the ethernet cable detect!) and passes it back to user space in "ifr_flags" of the ioctl request. Initially I'd used the "copy_to_user() macros but they didn't seem to work for me, I don't know why.

I also found I had to declare the prototype of this function higher up in the source file.

To load and test the module as I modified it I used a script that modprobed the module in and out and brought the interface up and down.

To help test/debug and stuff it is most useful to print to the kernel log, there is a macro defined "DO_PRINT" in my driver to do this.

To test the driver I found a great example program here (thanks mr pierce) which I then modified to just call my ioctl:


#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/in.h>
#include <stdio.h>

#include <string.h>

/* Get a socket handle. */
sck = socket(AF_INET, SOCK_DGRAM, 0);
if(sck < 0)
{
perror("socket");
return 1;
}
/* run custom ioctl */
strcpy(myifreq.ifr_name,"eth0");

if(ioctl(sck, SIOCDEVPRIVATE, &myifreq) < 0)
{
printf("oh dear!");
perror("ioctl(SIOCDEVPRIVATE)");
return 1;
}
printf(", magic number %x\n", myifreq.ifr_flags);

return 0;

}


And that's it. What a short blog entry for a weeks work :)
Oh--I used the 2.4 kernel but I'm pretty sure it's similar for any post 2.1 kernel

Monday, 17 November 2008

Django form with preset defaults

I have just started to learn how to use the Django web application framework

It has excellent documentation including tutorials. With a framework that has enough flexibility to do most things not everything is explained for a beginner in the documentation.

The application I am using the framework to write is for setting up product order forms. Each order has a number of "peripherals" which are added to the product.

In order to experiment with framework features I also set up a test application which sets up "bags" (like orders) that contain "beans" (like peripherals)

The data model for the Bean Bag application looks like this:


class Bean(models.Model):
Name=models.CharField(max_length=30)
def __unicode__(self):
return self.Name

class Bag(models.Model):
iam=models.CharField(max_length=30)
stuff=models.ManyToManyField('Bean',through='Beanzd')
def __unicode__(self):
return self.iam

class Beanz(models.Model):
bean=models.ForeignKey(Bean)
bag=models.ForeignKey(Bag)
n=models.PositiveIntegerField(default=1)

The trick is to use the "through" option to mediate the Beans into the Bag. The mediating "Beanz" class also adds a quantity "n"

Django has a very clever system for adding CRUD functionality with very little work. Define the model, add a file with a few lines in it and you have a full featured create/read/update/delete to the database.



Then I thought.."what if I want to make the admin forms so I can set the default list of beans that are available for each bag?". I altered the model like this:


class Bean(models.Model):
Name=models.CharField(max_length=30)
list=models.BooleanField(default=True)
def __unicode__(self):
return self.Name


So there is a "list" field which implies that the Bean is listed by default on the form. But the problem is how to convince the admin page to do this. First I altered the model



def wah(self):
for i in (Bean.objects.filter(list=True)):
yield i.pk

class Beanz(models.Model):
q=wah()
bean=models.ForeignKey(Beans,default=q.next())
bag=models.ForeignKey(Bag)
n=models.PositiveIntegerField(default=1)



This didn't work for two reasons. First the model is not the admin form. I was causing a default in the wrong place. Second, the way that the form was created meant that a number of different instance objects were made from the class Beanz and they each made a new instance of
the "wah" iterator.



I found that the way to alter the form in the admin interface was to sub class the form that the admin page got to build the interface. This could be done in such a way that the default could be set. To get around the problem of the different instance objects continually reseting the iterator I used a singleton pattern

In the model I did this:


class Bingo(Singleton):
def wah(self):
for i in (Bean.objects.filter(list=True)):
yield i.pk
def __init__(self):
if not self.__dict__.__contains__('q'): self.q=self.wah()
def again(self):
try:
r=self.q.next()
except (StopIteration, NameError, UnboundLocalError,AttributeError):
self.q=self.wah()
r=self.q.next()
return r



class Beanz(models.Model):
bean=models.ForeignKey(Bean)
bag=models.ForeignKey(Bag)
n=models.PositiveIntegerField(default=1)



and in the admin interface I did this


from django.forms import ModelForm

class BagAdminForm(ModelForm):
def __init__(self,*args,**kwargs):
super(BagAdminForm,self).__init__(*args,**kwargs)
b=Bingo()
self.fields.get("bean").initial=b.again()

class Meta:
model = Beanz

class beandefaultinline(admin.TabularInline):
form= BagAdminForm
model=Beanz
extra=len(Bean.objects.filter(list=True))

class DefaultListingBag(admin.ModelAdmin):
inlines=(beandefaultinline,)

admin.site.register(Bag, DefaultListingBag)


The modifications to the model are perhaps a bit clumsy. The "Borg Idiom" pattern might be better than the Singleton. In the admin model note that super is called before the changes are made to the sub classed data in the instance. This is so that all the stuff in the object is set up and available. Here is a screen grab of the finished result, with default "beans" listed