2 Installing wcd

2.1 Getting new versions of wcd

The latest version of wcd can be found at:

http://code.google.com/p/wcdest

You should also be able to find it at: ftp://ftp.cs.wits.ac.za/pub/research/software/

Please let me know if you would like to be told about new versions of wcd.

2.2 Changes from versions 0.3.x

There are a number of changes from 0.3 to 0.4 in memory organisation and algorithm. The first two are important to note if you used the options as the appropriate default values have changed.

• The sample word length is no longer a parameter. It's fixed as a constant at 8. This was done after extensive empirical testing. Fixing so that a sample word fits in a machine word has numerous computational advantages. The -B option therefore no longer exists. The default value for the -K is now 6.
• Our second heuristic, the so-called t/v heuristic has been replaced in favour of a common word heuristic. The the common word heuristic does seem better it is more expensive to implement. The -H or –common_word heuristic's default value is now 65.
• The SMP algorithm has been simplified. It is now possible to use the Pthreads and MPI parallelisation together. This may be useful when you have a cluster of multi-processor machines, but it may be easier just to run multiple MPI jobs.

2.3 Installing the distribution

The distribution comes as a gzipped, tarred file. The following commands should create a directory called ‘wcd_distrib’ with all the necessary files.

  gunzip wcd_distrib.tar.gz
  tar -xf wcd_distrib.tar

2.4 Compiling

There are two possible ways to compile and install wcd

2.4.1 Method 1: using configure

The INSTALL file gives a more detailed explanation and description. The following should work if you don't want anything fancy. If you do, or if you want to ensure that some compile flags are different, then read the INSTALL file.

./configure
make
make install

Note that the last instruction will attempt to install the executable and documentation in the appropriate places on your system and will probably require you to have root privilege or write permission for those standard directorites. If you don't have these, you can find the executable wcd in the src directory and the texinfo documentation wcd.info in the doc directory. Manually copy these to the correct place.

If you are using FreeBSD, you may have to make a small change to the common.h file.

If you want a manual you can print out, say make pdf. Otherwise you can read the on-line version using the info system. If the documentation is installed in the right place on your system you can just say info wcd, otherwise you will have to say info -f wcd.info.

wcd should work on 64-bit, little-endian and big-endian machines.

2.4.1.1 MPI Version

From version 0.3.0, wcd has an MPI version that allows parallelisation of the clustering on a cluster of workstations.

./configure --enable-MPI
make
make install

Initial experimentation has shown that using the MPI option when not necessary leads to a 2\% degradation in performance, so the main benefit of not using the enable-MPI option is that wcd will compile without the MPI libraries.

2.4.1.2 Pthreads

From version 0.3.0, wcd has an MPI version that allows parallelisation on SMP architectures

./configure --enable-PTHREADS
make
make install

In version 0.4, the two parallelisation techniques complement each other. You can run on a cluster of SMP machines. However, an assumption is made that each of the SMP machines has the same number of processors. In version 0.3, binaries supporting both techniques could be used but only one used in a particular run; this no longer holds in 0.4.

2.4.2 Method 2: Manually using make

The wcd program is written in reasonably standard C. If you don't want to or can't use the autoconf method described above, try the following. Change directory to the src directory.

make -f WcdMake wcd

This was tested on different versions of RedHat and Suse and worked without problem. wcd is written in reasonably standard C, and so should work on most systems, but unfortunately the vagaries of different libraries and compilers might make the standard compilation fail. See the troubleshooting section below for some suggestions.

To get documentation, running make -f WcdMake pdf in the src directory will produce a pdf version of the manual (placing output in the doc directory). make -f WcdMake info will produce texinfo version.

2.4.2.1 Trouble shooting

If you know something about your system and are happy mucking about with make files, please read the technical note below and make the necessary changes. If not, try the following suggestions. There are three likely causes of problems: (1) wcd supports long options (e.g. you can say --cluster_compare rather than -D); (2) your compiler is cc rather than gcc; and (3) your compiler does not support procedure inlining (inlining may make some performance benefit, but does not change functionality or use). The makefile WcdMake is the one to look at.

1. Try make -f WcdMake simple to see whether the problem is that you don't have the libraries that support long options. If this works, wcd will run as is, except that you cannot use the long options form for wcd, only the short-form options.
2. If your compiler is named cc rather than gcc, try saying make -f WcdMake ccsimple.
3. If that doesn't work, try switching off inlining as well by doing make reallysimple. In this case neither long options or inlining is used. If your compiler is cc then try make -f WcdMake ccreallysimple.

wcd was tested on a number of different systems. The table below shows what worked on those systems. With a little tweaking of the makefile to specify where gcc libraries on your machine are, you might get the standard make wcd to work too.

Linux RedHat and Suse

make wcd

Sun OS 5.10, Darwin, Aix, FreeBSD

make simple

Irix64, Alpha/OSF1 Tru64, Cray

make ccreallysimple

With the FreeBSD machine I managed to get the full version to work by finding the appropriate libraries and includes. The same thing probably applies to the others.

This section only need be read if make wcd did not work and you would like to have long options working.

The non-standard options of wcd are to use inlining and the long options library. If your system does not support inlining then the compiler must be given the -DNOINLINE flag when run. If your system does not provide a library for getting long options then the compiler must be given the -DNOLONGOPT flag.

If, however, your system does provide a long options library and you would like to use long options, then you must say where the include file (‘getopt.h’) and where the library will be. You may also have to explicitly tell the compiler to use the ‘libtextlib’ library. Find out where these things are, and change the variables in the WcdMake to the appropriate values and then run make try. For example, if the include file is in /usr/share/contrib/gcc and the library is in /usr/local/lib you might have

INCLUDES = /usr/share/contrib/gcc
LIBS     = /usr/lib
EXTRAFLAGS = gettextlib

You might or might not have the EXTRAFLAGS variable defined.

The makefile assumes that the compiler available is gcc. If it's not or if you wish to use a different compiler, then edit the file Makefile and change the definition of the variable CC to the appropriate compiler.

If this doesn't work, then it may be that your libraries are in non-standard places. Look at the Makefile and try the following things

• Add ‘-DNOLONGOPT’ to the CFLAGS variable. In this case, you can't use the long form options (e.g. you must say -g rather than --histogram)
• Try replacing ‘gcc’ by ‘cc’.
• Add to the CFLAGS the directories where the include and libraries can be found.

wcd's MPI and PTHREADS code are optional. In the code there are macros that protect all references to MPI and PTHREADS. Unless these macros are defined the compilier won't know about them. This enables wcd to be compiled without having either libraries.

If you are making manually rather than through configure, and you want to use MPI or PTHREADS (either or both), then you must make sure that either (or both) the MPI are PTHREADS macros are defined. You could either put the relevant defines in the wcd.h file, or pass it to gcc with something like -D MPI.

2.4.3 EMBOSS

EMBOSS wrappers are available from the same source as the wcd code itself.

2.4.4 Auxiliary programs

A number of auxiliary programs are included. They may be useful but are not essential. They are either Perl or Python See Auxiliary Programs, for more details. They should run as is, but if your version of perl or python is not in a standard place, edit the files and change their first lines to replace the line that says

#!/usr/bin/perl

#!/usr/local/bin/perl

In addition to these programs, there is a shell script wcd_wrapper.sh that allows wcd to be used as a replacement for d2_cluster in the stackPACK analysis pipeline.

2.5 Documentation

This info file can be read by saying

                  info -f wcd.info

To create the documentation say ‘make pdf, make html’, etc depending on what sort of document you want. This creates the following:

• ‘wcd.info’: the info file which can be read using the texinfo system. This is probably easiest for reference purposes. You can read the on-line version using the info system. If the documentation is installed in the right place on your system you can just say info wcd, otherwise you will have to say ‘info -f wcd.info’, or give the full path.
• ‘html’: a directory that contains all this help information in HTML files. Use your browser to read it.
• ‘wcd.pdf’: A PDF version of this help file, which is probably the best for a hard copy.

You can play around with ‘makeinfo’ etc. if you want different versions.