Ubuntu 8.04 & ZODB

After a fresh install, saying:

sudo apt-get install python-zodb

..did not properly set up ZODB. Some kind of version conflict/Ubuntu bug. Had to add following lines to /etc/apt/sources.list:

deb http://ppa.launchpad.net/schooltool-owners/ubuntu hardy main
deb-src http://ppa.launchpad.net/schooltool-owners/ubuntu hardy main

..and say:

sudo apt-get install python-zodb
sudo apt-get install python-zope.interface
sudo apt-get install python-zope.proxy

See: https://bugs.launchpad.net/ubuntu/+source/zodb/+bug/226292/comments/5

Python shelve speed testing

import shelve, os, time, sha

class Foo:
    def __init__(self):
        self.data = sha.new(repr(time.time())).hexdigest()

nobjects = 100000

try:
    os.unlink('speedtest.db')
except OSError:
    print "creating db file"

start = time.time()

db = shelve.open('speedtest')

i = nobjects
while i > 0:
    db[str(i)] = Foo()
    db.sync()
    i = i - 1

db.close()

t = time.time() - start
print t, " sec for ", nobjects, " objects; ", t/nobjects, " sec/object"

On a 2 GHz MacBook with 1 GB of RAM and Python 2.3.5 the run time for this program is around 7 seconds. ("7.00173902512 sec for 100000 objects; 7.00173902512e-05 sec/object", the AnyDBM backend being "Berkeley DB 1.85 (Hash, version 2, native byte-order)".) Omitting the .sync() call shaves a few hundred milliseconds from that.

See also: http://codeidol.com/python/python3/Databases-and-Persistence/Shelve-Files/

mod_python

So, the Twisted engine turned out to be a bit more complicated than necessary for my needs. Maybe later..

A more conservative approach is needed, hence Apache/mod_python. To get it running in Ubuntu (assuming you have Apache already installed), do:

sudo apt-get install libapache2-mod-python

sudo apt-get install libapache2-mod-python-doc

sudo emacs /etc/apache2/sites-enabled/000-default

(Or whatever editor one is using.)

..and inside <Directory /var/www/> insert these lines:

AddHandler mod_python .py
PythonHandler mod_python.publisher
PythonDebug On

Exit Emacs and restart Apache:

sudo /etc/init.d/apache2 restart

Done.

Ok, not quite. The above enables mod_python Publisher handler which operates at a high level. Instead of, or in addition to above, put something like this into Apache config:

<Directory /var/www/python>
        AddHandler mod_python .py
        PythonHandler python
        PythonDebug On
</Directory>

Now you have more control over the response:

from mod_python import apache

def handler(req):
        import time

        req.content_type = "text/plain"
        req.headers_out['CacheControl'] = "no-cache"
        req.headers_out['Expires'] = "-1"
        req.headers_out['This-is-my-own-wacky-and-useless-header'] = "Pangalactic Gargleblaster"
        req.write(str(time.time()))

        return apache.OK

Python doc resources

Python.org doc page

Dive Into Python

The Python Papers

ActiveState Python Cookbook

Using Threads in Twisted

This Twisted thingy is seriously hard stuff to understand... In addition to that threading doc (post link), see howtos about deferreds and async programming.

What I'm trying to do here is plug Berkeley DB XML into a Twisted.Web2 application. This involves delegating DB access to worker threads outside the Twisted event loop ("reactor" pattern). A lot more stuff to figure out before coding starts...

Berkeley DB XML Python basics

In an earlier post a C++ snippet can be found where a DB XML container was created (or opened if already exists) and a document read from stdin was put into that container. That same snippet done in Python is pretty much identical:

from bsddb3.db import *
from dbxml import *

mgr = XmlManager(DBXML_ALLOW_EXTERNAL_ACCESS)
uc = mgr.createUpdateContext()

try:
        cont = mgr.openContainer("testcontainer.dbxml", DB_CREATE|DBXML_ALLOW_VALIDATION, XmlContainer.WholedocContainer)
        doc = mgr.createDocument()
        input = mgr.createStdInInputStream()
        doc.setContentAsXmlInputStream(input)
        cont.putDocument(doc, uc, DBXML_GEN_NAME)

except XmlException, inst:
        print "XmlException (", inst.ExceptionCode,"): ", inst.What
        if inst.ExceptionCode == DATABASE_ERROR:
                print "Database error code:",inst.DBError

Berkeley DB XML Python module building

A glitch appears when trying to build out of the box as instructed in src/python/README. Minor tweaking with types was necessary in the dbxml_python_wrap.cpp file. Here's output from diff(1):

2058c2058
<     const char *doc = (((PyCFunctionObject *)obj) -> m_ml -> ml_doc);
---
>     char *doc = (((PyCFunctionObject *)obj) -> m_ml -> ml_doc);
3001c3001
<     char *cstr; Py_ssize_t len;
---
>     char *cstr; int len;
36695c36695
<       const char *c = methods[i].ml_doc;
---
>       char *c = methods[i].ml_doc;
36699c36699
<         const char *name = c + 10;
---
>         char *name = c + 10;

This same problem was discussed in DB XML forums (which, for some reason, seem to be off line a lot of time; what's wrong, Oracle?). After those minor modifications, build went as advertised:

cd bsddb3-4.5.0

python setup.dbxml.py build

sudo python setup.dbxml.py install

cd ..

python setup.py build

sudo python setup.py install

After the build these basic tests passed:

>>> from bsddb3.db import *
>>> from dbxml import *
>>> print version()
(4, 5, 20)
>>> mgr=XmlManager()
>>> print mgr.get_version_string()
Oracle: Berkeley DB XML 2.3.10: (January 30, 2007)

Running the tests (python examples.py test) also passed just fine. Great!