- lxml

- Developing with lxml

- Extending lxml

- Developing lxml

# How to build lxml from source

To build lxml from source, you need libxml2 and libxslt properly installed, including the header files. These are likely shipped in separate `-dev` or `-devel` packages like `libxml2-dev`, which you need to install. The build process also requires setuptools. The lxml source distribution comes with a script called `ez_setup.py` that can be used to install them.

Contents

- Cython
- Subversion
- Setuptools
- Running the tests and reporting errors
- Contributing an egg
- Providing newer library versions on Mac-OS X
- Static linking on Windows
- Building Debian packages from SVN sources

# Cython

The lxml.etree and lxml.objectify modules are written in Cython. Since we distribute the Cython-generated .c files with lxml releases, however, you do not need Cython to build lxml from the normal release sources. We even encourage you to not install Cython for a normal release build, as the generated C code can vary quite heavily between Cython versions, which may or may not generate correct code for lxml. The pre-generated release sources were tested and therefore are known to work.

So, if you want a reliable build of lxml, we suggest to a) use a source release of lxml and b) disable or uninstall Cython for the build.

Only if you are interested in building lxml from a Subversion checkout (e.g. to test a bug fix that has not been release yet) or if want to be an lxml developer, then you do need a working Cython installation. You can use EasyInstall to install it:

    easy_install Cython==0.9.6.12

lxml currently requires Cython 0.9.6.11b or 0.9.6.12, later versions were not tested.

# Subversion

The lxml package is developed in a Subversion repository. You can retrieve the current developer version by calling:

    svn co http://codespeak.net/svn/lxml/trunk lxml

This will create a directory `lxml` and download the source into it. You can also browse the repository through the web or use your favourite SVN client to access it.

# Setuptools

Usually, building lxml is done through setuptools. Do a Subversion checkout (or download the source tar-ball and unpack it) and then type:

    python setup.py build

or:

    python setup.py bdist_egg

If you want to test lxml from the source directory, it is better to build it in-place like this:

    python setup.py  build_ext -i

or, in Unix-like environments:

    make

If you get errors about missing header files (e.g. `libxml/xmlversion.h`) then you need to make sure the development packages of both libxml2 and libxslt are properly installed. Try passing the following option to setup.py to make sure the right config is found:

    python setup.py build --with-xslt-config=/path/to/xslt-config

If this doesn't help, you may have to add the location of the header files to the include path like:

    python setup.py build_ext -i  -I /usr/include/libxml2

where the file is in `/usr/include/libxml2/libxml/xmlversion.h`

To use lxml.etree in-place, you can place lxml's `src` directory on your Python module search path (PYTHONPATH) and then import `lxml.etree` to play with it:

    # cd lxml
    # PYTHONPATH=src python
    Python 2.5.1
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from lxml import etree
    >>>

To recompile after changes, note that you may have to run `make clean` or delete the file `src/lxml/etree.c`. Distutils do not automatically pick up changes that affect files other than the main file `src/lxml/etree.pyx`.

# Running the tests and reporting errors

The source distribution (tgz) and the Subversion repository contain a test suite for lxml. You can run it from the top-level directory:

    python test.py

Note that the test script only tests the in-place build (see distutils building above), as it searches the `src` directory. You can use the following one-step command to trigger an in-place build and test it:

    make test

This also runs the ElementTree and cElementTree compatibility tests. To call them separately, make sure you have lxml on your PYTHONPATH first, then run:

    python selftest.py

and:

    python selftest2.py

If the tests give failures, errors, or worse, segmentation faults, we'd really like to know. Please contact us on the mailing list, and please specify the version of lxml, libxml2, libxslt and Python you were using, as well as your operating system type (Linux, Windows, MacOs, ...).

# Contributing an egg

This is the procedure to make an lxml egg for your platform:

- Download the lxml-x.y.tar.gz release. This contains the pregenerated C so that you can be sure you build exactly from the release sources. Unpack them and cd into the resulting directory.
- python setup.py build
- If you're on a unixy platform, cd into `build/lib.your.platform` and strip any `.so` file you find there. This reduces the size of the egg considerably.
- `python setup.py bdist_egg upload`

The last 'upload' step only works if you have access to the lxml cheeseshop entry. If not, you can just make an egg with `bdist_egg` and mail it to the lxml maintainer.

# Providing newer library versions on Mac-OS X

The Unix environment in Mac-OS X makes it relatively easy to install Unix/Linux style package management tools and new software. However, it seems to be hard to get libraries set up for exclusive usage that Mac-OS X ships in an older version. The result can be segfaults on this platform that are hard to track down.

To make sure the newer libxml2 and libxslt versions (e.g. those provided by fink or macports) are used at build time, you must take care that the script `xslt-config` is found from the newly installed version when running the build setup. The system libraries also provide this script, but the new one must come first in the PATH. The best way to make sure the right version is used is by passing the path to the script as an option to setup.py:

    python setup.py build --with-xslt-config=/path/to/xslt-config \
                          --with-xml2-config=/path/to/xml2-config

To make sure the newer libxml2 and libxslt versions are used at runtime, you should add all directories where the newer libraries are installed (i.e. libxml2, libxslt and libexslt) to the `DYLD_LIBRARY_PATH` environment variable when you use lxml (i.e. not only at build time). This seems to fix a lot of problems for users.

Please read this thread about experiences with MacOS-X if you encounter problems. It also has a buildout for lxml that you can use.

# Static linking on Windows

Most operating systems have proper package management that makes installing current versions of libxml2 and libxslt easy. The most famous exception is Microsoft Windows, which entirely lacks these capabilities. It can therefore be interesting to statically link the external libraries into lxml.etree to avoid having to install them separately.

Download lxml and all required libraries to the same directory. The iconv, libxml2, libxslt, and zlib libraries are all available from the ftp site ftp://ftp.zlatkovic.com/pub/libxml/.

Your directory should now have the following files in it (although most likely different versions):

    iconv-1.9.1.win32.zip
    libxml2-2.6.23.win32.zip
    libxslt-1.1.15.win32.zip
    lxml-1.0.0.tgz
    zlib-1.2.3.win32.zip

Now extract each of those files in the same directory. This should give you something like this:

    iconv-1.9.1.win32/
    iconv-1.9.1.win32.zip
    libxml2-2.6.23.win32/
    libxml2-2.6.23.win32.zip
    libxslt-1.1.15.win32/
    libxslt-1.1.15.win32.zip
    lxml-1.0.0/
    lxml-1.0.0.tgz
    zlib-1.2.3.win32/
    zlib-1.2.3.win32.zip

Go to the lxml directory and edit the file `setup.py`. There should be a section near the top that looks like this:

    STATIC_INCLUDE_DIRS = []
    STATIC_LIBRARY_DIRS = []
    STATIC_CFLAGS = []

Change this section to something like this, but take care to use the correct version numbers:

    STATIC_INCLUDE_DIRS = [
           "..\\libxml2-2.6.23.win32\\include",
           "..\\libxslt-1.1.15.win32\\include",
           "..\\zlib-1.2.3.win32\\include",
           "..\\iconv-1.9.1.win32\\include"
           ]

    STATIC_LIBRARY_DIRS = [
           "..\\libxml2-2.6.23.win32\\lib",
           "..\\libxslt-1.1.15.win32\\lib",
           "..\\zlib-1.2.3.win32\\lib",
           "..\\iconv-1.9.1.win32\\lib"
           ]

    STATIC_CFLAGS = []

Add any CFLAGS you might consider useful to the third list. As Ashish Kulkarni notes, you might have to add the standard Windows library `wsock32.dll` to the list of libraries to make `lxml.objectify` compile.

Now you should be able to pass the `--static` option to setup.py and everything should work well. Try calling:

    python setup.py bdist_wininst --static

This will create a windows installer in the `pkg` directory.

# Building Debian packages from SVN sources

Andreas Pakulat proposed the following approach.

- `apt-get source lxml`
- remove the unpacked directory
- tar.gz the lxml SVN version and replace the orig.tar.gz that lies in the directory
- check md5sum of created tar.gz file and place new sum and size in dsc file
- do `dpkg-source -x lxml-[VERSION].dsc` and cd into the newly created directory
- run `dch -i` and add a comment like "use trunk version", this will increase the debian version number so apt/dpkg won't get confused
- run `dpkg-buildpackage -rfakeroot -us -uc` to build the package

In case `dpkg-buildpackage` tells you that some dependecies are missing, you can either install them manually or run `apt-get build-dep lxml`.

That will give you .deb packages in the parent directory which can be installed using `dpkg -i`.