Bugzilla/SVN/Wiki Integration - Segetech Open Source Rendezvous

Introduction

Software projects are getting larger and more complicated year after year stressing the need for efficient and reliable tools and agile project management methods. Widely used and recognized tools in Open Source development, Subversion (SVN) and Bugzilla, have been proven to be suitable for most kind of software projects and have lately been accompanied in many projects with an emerging document and content management system, Wiki. These tools are usually used separately, rarely integrated with each other to enable easier understanding and management of the whole process flow from specifications and code to releases and maintenance. Integrated tools would also provide more effective and seamless environment for software development with improved quality and productivity.

This document provides a detailed configuration guide to integrate incrementally several Open Source development tools creating a seamless development enviroment without unnecessary complexity or overhead. Bugzilla/SVN/MediaWiki are all linked to each other and they together with mailing lists and various SVN utilities provide easily accessible way to see changes to documentation and source code, handling of bug reports, and people contributing to projects without any additional tools or preparations in matter of minutes whenever needed. All this is achieved by using three special magic identifiers to create references to other tools. The identifier BugID: creates references to Bugzilla, VCS: creates references to version control system, SVN, and Wiki: creates references to Wiki. Whenever writing a Wiki document, a bug report, or an SVN commit message, these identifiers can be used and they are then automatically turned into hyperlinks pointing to the tools referenced. This enables smooth browsing from specifications to code, from code to bug reports, and back from bug reports to code changes and documentation. Additionally, all activities in projects are archived in mailing lists, SVN utilities provide graphical statistics, and bug reports contain references to actual changes or documentation that permitted closing the bug. The instructions below proceed incrementally by configuring one tool after another until the whole environment is set up. Provided screenshots illustrate results of integration. These instructions can be used of course separately to set up only parts of the presented environment or to integrate with the tools already in use. Reader is expected to have basic knowledge about the tools and their purpose.

Mailman Configuration

The following instructions to configure Mailman mailing list manager are based on Red Hat packaged Mailman RPMs that have some paths different compared to official Mailman releases but otherwise the instructions can be used with any version of Mailman. It is of course necessary to run httpd and sendmail (or some other MTA) at the server where Mailman archives are located.

First, Apache is configured to redirect queries to /mailman to a list info page. This setting is recommended for all Mailman installations and requires only adding one directive to the file /etc/httpd/conf.d/mailman.conf that is installed from the Mailman RPM. Obviously, you must change the hostname to match your local environment.

RedirectMatch ^/mailman[/]*$ http://www.dom.ain/mailman/listinfo

This setting will be effective after the service httpd has been restarted:

/sbin/service httpd restart

Actual Mailman configuration begins by setting Mailman site password with the command:

/usr/lib/mailman/bin/mmsitepass <your-site-password>

Main configuration file for Mailman is /etc/mailman/mm_cfg.py. There are dozens of configuration directives that could be set but here only two mandatory settings and few recommended ones are listed. The mandatory settings are:

DEFAULT_URL_HOST   = 'www.dom.ain'
DEFAULT_EMAIL_HOST = 'dom.ain'

The settings must, of course, be adjusted to match your local environment. If you use localhost.localdomain as DEFAULT_EMAIL_HOST then you should also add the following line just after the previous directives:

VIRTUAL_HOSTS.clear()

Other recommended and/or useful settings include the following:

ALLOW_SITE_ADMIN_COOKIES = Yes  # For secure intranets only
DEFAULT_CHARSET          = 'utf-8'
DEFAULT_MAX_MESSAGE_SIZE = 4096 # kB
# These are needed if archives are accessed only by using HTTPS
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
PUBLIC_ARCHIVE_URL  = 'https://%(hostname)s/pipermail/%(listname)s'

To update Mailman configuration after configuring needed directives the following command must be executed:

/usr/lib/mailman/bin/update

Next, for administration purposes the mailing list mailman must be created:

/usr/lib/mailman/bin/newlist mailman

The command will ask for an e-mail address for person running the list. The e-mail address must contain proper host and domain parts also for local users so enter an e-mail address like [email protected]. Follow the provided instructions on how to update the file /etc/aliases to properly process e-mails sent to the list. After updating /etc/aliases, the alias database must be updated with the command:

/usr/bin/newaliases

The service mailman that updates the archives when needed can now be started:

/sbin/chkconfig mailman on
/sbin/service mailman start

Now the actual lists for Bugzilla and SVN can finally be created. Two lists will be used, one for archiving Bugzilla activities and one for SVN activities. For Bugzilla activities, the list shall be prj-bugs and for SVN commits prj-vcs.

To create a new mailing list open the URL http://www.dom.ain/mailman/create in a browser and follow the instructions to create the list. Mandatory information that must be provided when creating a list is the name of the list, initial list owner e-mail address, and initial list password. For other settings, default values are usually suitable. Note that after creating the mailing lists the file /etc/aliases must updated as instructed in the e-mails sent to the list creator and after the update the command newaliases(1) must be run.

After both the lists have been created configuration should be changed to allow anyone to post to the lists. Especially with the list prj-vcs this eases maintenance, as there is no need to duplicate changes to SVN rights in the mailing list configuration. To allow all postings to a list, visit list's admin page and go to Privacy Options->Sender Filter and change settings By default, should new list member postings be moderated? to No and Action to take for postings from non-members for which no explicit action is defined. to Accept.

The last step is to enable the magic identifiers in the mailing list archives. Applying this patch and restarting the mailman service is all what is needed to enable them. The patch was generated against Mailman 2.1.5 but it should apply also against later versions in the 2.1.x series. Once done, the mailing list configuration is ready.

Available screenshots: commit message in the prj-bugs list and commit message in the prj-vcs list.

MySQL Configuration

Both Bugzilla and MediaWiki use MySQL database to store data so before installing Bugzilla or MediaWiki, MySQL must be installed and configured. These instructions provide only quick steps to get MySQL 4 or later ready for Bugzilla and MediaWiki, for any performance optimizations or other advanced adjustments you should refer to MySQL documentation. MySQL should be installed from a package provided with your distribution. Note that before configuring MySQL the MySQL database server (mysqld) must be running.

The very first configuration step after MySQL installation is to set MySQL root user password. This can be done with the following command. Note that the command will fail if the password is already set so this command can be executed more than once if in doubt.

mysqladmin -uroot password <root-password>

Actually, this is all what is needed for MediaWiki as its installer will set up the rest. But for Bugzilla the user bugs must also be added:

mysql -uroot -p<root-password> mysql
GRANT SELECT, INSERT, UPDATE, DELETE, INDEX, ALTER,
  CREATE, LOCK TABLES, CREATE TEMPORARY TABLES, DROP,
  REFERENCES ON bugs.* TO bugs@localhost
  IDENTIFIED BY '<bugs-password>';
FLUSH PRIVILEGES;

Those are the only mandatory configurations needed for Bugzilla and MediaWiki but you probably also want to configure mysqld to start during system bootup. Other MySQL configuration options can be studied from the MySQL documentation. For reference, however, few of the most useful options are also provided here for download. To apply these settings add them to the file /etc/mysql.cnf into the [mysqld] section.

Bugzilla Configuration

Bugzilla configuration usually requires a bit of work but the actual procedure is rather simple. Before installing Bugzilla, make sure that you have installed Perl and MySQL is configured. If Perl is not installed, use your distribution's packages to install it. Then, download latest stable Bugzilla release, unpack the downloaded archive, and move the extracted directory as /var/www/html/bugzilla. To start Bugzilla configuration, enter the Bugzilla directory, fix owner of the files and directories, and run the checksetup.pl script with the parameter --check-modules:

cd /var/www/html/bugzilla
chown -R root:root . *
./checksetup.pl --check-modules

The script will most likely complain about missing Perl modules. You must install all the required Perl modules. Whether you should also install all or some of the optional modules mentioned depends on many things. If you eat Perl for breakfast and already know how to install Perl modules then it is surely no problem to install all the optional modules, too. But if you just want to get Bugzilla up and running and perhaps later investigate the features provided by the optional modules then just follow the instructions provided by the checksetup.pl script to install the required modules. Note that some of the modules may be found from your distribution's package collections. If the module installation gets complicated, you may find some help from the Bugzilla documentation. When checksetup.pl is finally happy, rerun the script without any parameters:

./checksetup.pl

This time, checksetup.pl will instruct you to edit the file localconfig. Most likely the only setting you want to change is $db_pass. Set it to match the bugs user password you used earlier during MySQL configuration. When done, rerun the script checksetup.pl again without any parameters:

./checksetup.pl

The script should proceed until it asks for an administrator. Please enter a valid e-mail address, administrator's real name, and a proper password. After checksetup.pl finishes, one final step is remaining; Apache must be configured to allow CGI scripts to be executed in the Bugzilla installation directory. This can be done by adding the following lines to Apache configuration file, usually /etc/httpd/conf/httpd.conf:

<Directory /var/www/html/bugzilla>
  AddHandler cgi-script .cgi
  Options +Indexes +ExecCGI
  DirectoryIndex index.cgi
  AllowOverride Limit
</Directory>

Setting will be effective after the service httpd has been restarted:

/sbin/service httpd restart

Bugzilla is now installed and mostly configured and can be used with a browser at the URL http://www.dom.ain/bugzilla/. Log in as administrator and go to Parameters page. There are several configuration options that could be tweaked. At least the following two options should be changed: maintainer should contain valid e-mail address of the Bugzilla installation maintainer and urlbase should contain the URL of the Bugzilla installation. More information about all the options can be found from the Bugzilla documentation.

After saving parameter changes, you must add two new users for integration purposes. As administrator go to administration page Users->add a new user and then create users [email protected] and [email protected]. You may want to use descriptive real names like Bugmail and VCS Commits but more importantly disable the accounts by stating their purpose in the Disable field. The accounts will be used to send bug activity e-mails to the prj-bugs mailing list and to append SVN commits log messages to bug reports.

To enable automatic bug activity e-mails Default CC List must be set for each component for which the feature is wanted. Components can be found under the menu Products->Edit Product. Same account can be reused for as many components as wanted and typically all components of a product have the same CC List when used for activity e-mail generation. Example product TestProduct and its component TestComponent that are automatically created during Bugzilla installation can be used to see how the automated e-mailing works by setting Default CC List account of the TestComponent as [email protected]. Whenever a bug is created against TestComponent or status of a TestComponent bug changes an e-mail is sent to the prj-bugs mailing list and archived.

The last required step is to enable the magic identifiers in Bugzilla. Applying this patch is all what is needed to enable them. The patch was generated against Bugzilla 3.0.2 but it should apply also against later Bugzilla 3.x releases.

At this stage you might want to test your Bugzilla installation a bit by creating a test bug or two against the example product TestProduct and see how linking with the magic identifiers and automated e-mails work. You may also want to adjust some other options, such as the e-mail templates and time tracking group, in the Parameters page to better suit your needs. After Bugzilla seems to be your liking, you can proceed to the next section.

Available screenshots: commit message in Bugzilla.

SVN Configuration

Basic set up of a Subversion repository is rather straightforward task. The configuration presented here uses the directory /var/lib/svnroot as the repository and expects that all SVN users belong to the group users. More information about other settings can be found from the book Version Control with Subversion.

To initialize a new repository and set file permissions correctly the following commands must be executed:

export SVNROOT=/var/lib/svnroot
svnadmin create $SVNROOT
chgrp -R users $SVNROOT/db $SVNROOT/locks
chmod -R g+w $SVNROOT/db $SVNROOT/locks

These commands create and initialize the repository and make sure the files have sane permissions. Users are then able to use the created repository via file://, svn://, and svn+ssh:// URLs. More information about repository internals and access methods can be found from the SVN book but during this integration deeper knowledge about them is not essential. However, the presented integration works with other access methods, too.

To add a test module prj with some dummy content the following commands should be executed. Please the SVN book for the rationale of the repository directory structure.

mkdir -p prj/trunk
mkdir -p prj/tags
mkdir -p prj/branches
svn import prj file://$SVNROOT/prj -m "Initial repository layout."
rm -rf prj
svn co file://$SVNROOT/prj
cd prj/trunk
mkdir -p test
echo foo > foo
echo bar > test/bar
svn add foo test
svn ci -m "Imported sources."
cd ../..
rm -rf prj

Now, to start the actual integration work, ViewVC (formerly ViewCVS) is configured and patched so that the SVN repository can be accessed with a browser and the magic identifiers are shown as links. Either install ViewVC using the official sources or use a ViewVC package available for your distribution. Then, apply a minor patch to the script viewvc/lib/viewvc.py to enable the magic identifiers. The patch was generated against ViewVC 1.0.0 but it should apply also to later versions. In addition, some minor changes to the default viewvc.conf configuration file must be done so that ViewVC properly uses the repository created earlier. For convenience, both the patched files are also available: viewvc.py and viewvc.conf.

Before proceeding any further, please make sure that at this stage your SVN repository is properly set up, checkouts and checkins can be done, and that the repository can be accessed with a browser via ViewVC. You should also do some test commits using the magic identifiers in the log message to see that ViewVC produces correct links for them. An example log entry could be something like:

During integration testing a problem was reported at BugID: 1
that although the specification at Wiki: The_Spec says that we
should do bar, VCS: prj/trunk/foo was doing foo. Fixed to do bar.

If both the SVN repository and the ViewVC interface are working properly then it is time to begin perhaps the most crucial part of the integration. The Subversion repository will be configured to send commit e-mails to the prj-vcs mailing list and to integrate with Bugzilla. All this is done by adding and configuring a set of hook scripts under the directory /var/lib/svnroot/hooks. Since there are some interdependencies between the scripts it is probably best to do all required changes at a go, then test the changes, and fix any problems that might arise.

First, svnmailer needs to be installed and configured. Svnmailer is a tool written in Python to post commit information by e-mail, news, or XML (via CIA). To enable some rather useful features of recent svnmailer releases, version 1.1 or later should be used. Earliest tested version is 1.1.0-dev-r1373. The hook scripts below expect that or a later version. Either before or after the installation of svnmailer, apply this patch to enable Bugzilla/Subversion integration based on the magic identifiers. Then, a configuration file should be placed into the directory /var/lib/svnroot/hooks. Please review the configuration file and edit it to match your environment.

After svnmailer is installed and configured, it can be enabled with a set of hook scripts. The scripts must be placed into the hooks directory /var/lib/svnroot/hooks and they must have proper permissions allowing their execution (i.e., mode 0755). Please also check that the paths defined in the scripts match your environment. The needed scripts are provided below with some explanations.

pre-commit and post-commit define actions to do before and after a commit. pre-commit makes sure that the commit log message contains some text and post-commit sends a commit e-mail based on svnmailer configuration and the commit log message. If the log message contains the magic identifier BugID: referencing to a bug report, the commit log information is sent to the Bugzilla instance as a properly labeled e-mail.

pre-revprop-change makes sure that ordinary users are able to change only the svn:log revision property of the files. post-revprop-change just sends an e-mail notification about such an event. These two scripts are optional from the integration point of view but their usage is recommended for administrative reasons.

post-lock and post-unlock are used to send a notification e-mail after a file is locked or unlocked. These two scripts are also optional from the integration point of view but their usage is recommended to allow easier tracking of file lockings.

For more information about these and additional hook scripts please refer to the SVN book.

Since the script post-commit sends the commit messages to Bugzilla as e-mails, they must be received and processed properly. The processing is done by the script email_in.pl that is part of the Bugzilla installation. The script must be slightly patched to enable it to find all required Perl modules. The following line must be added to the file /etc/aliases to automatically invoke the script when sendmail receives commit e-mails. Note that the recipient must match to the one defined in the svnmailer configuration file.

bugzilla:"|/var/www/html/bugzilla/email_in.pl"

And, once again, the command newaliases(1) must be run after the alias database has been updated. After the update, sendmail executes the script for all e-mails sent to the address bugzilla. However, by default sendmail is not allowed to run random commands while processing e-mails so the script must be listed as an allowed command. Also the permissions in the Bugzilla installation must be changed so that the user mail under which sendmail is running is allowed to read and execute all the needed scripts. The following commands allow sendmail to invoke the script email_in.pl and change permissions so that the user mail is actually able to run it.

cd /etc/smrsh
ln -s /var/www/html/bugzilla/email_in.pl .
cd /var/www/html/bugzilla
chmod g+w data/mimedump-tmp
chown -R apache:mail . *

It must be noted that running again the script checksetup.pl or editing Bugzilla parameters will change the file permissions so that they must be adjusted again. However, as these are rather rare events it does not cause too much maintenance burden but this is well worth keeping in mind in case of any integration issues.

Before actually testing the integration, please make sure that all Perl modules required by email_in.pl have been installed. This can be verified by running the script from the command line. If any error messages are printed, some modules, such as MIME-tools, are probably missing. If no errors are printed, the Bugzilla/SVN integration is complete! You may now experiment with commits containing the magic identifiers and see how the commit messages are appended to bug reports alongside with hyperlinks to actual changes in SVN. If everything seems to be working you are ready to proceed to the next section.

Available screenshots: commit message in repository web interface.

SVN Utilities Configuration

This section provides quick instructions on how to use some rather useful utilities to generate statistics from your SVN repository. Although they are not absolutely required from the integration point of view, they all fall into the low-hanging fruit category and are thus worth configuring.

CvsGraph is a utility to make graphical representations of CVS repository. No similar utility usable with SVN is available for the time being. Once such utility exists, instructions to configure it will be added here.

CVSHistory is a utility written in Python that is used to track development activity on CVS using software projects. No similar utility usable with SVN is available for the time being. Once such utility exists, instructions to configure it will be added here.

svn2cl can be used to generate formatted changelogs based on SVN activities. It is a shell script that uses libxslt. It is not compatible with cl2html but it can also produce HTML pages. Installation of svn2cl is simply done by unpackaging the downloaded archive, moving the extracted directory as /usr/local/share/svn2cl, fixing permissions, and creating a needed symbolic link svn2cl in a directory contained in PATH, e.g., /usr/local/bin:

cd /usr/local/share/svn2cl
chown -R root:root .
chmod a+rx .
chmod a+r *
cd /usr/local/bin
ln -s /usr/local/share/svn2cl/svn2cl.sh svn2cl

There are several command line parameters that could be used but for convenience a shell script is provided that creates GNU/HTML/XML formatted changelogs from the Subversion repository and also a nice web page to access them. Please refer to the documentation of the scripts for more information about their usage. Note that the shell script requires the file $SVNROOT/conf/users to be created and if it contains valid mappings for names those are used when generating the changelogs. As can be seen from the script the statictics web page created by default is /prj-svn-stats/index.html.

StatSVN is a tool written in Java that is used to create exhaustive amount of statistics from projects in an SVN repository. Installation is simply done by placing the JAR file from the StatSVN package to a proper directory, e.g., /usr/local/share and running it with the help of java(1) command. There are several command line parameters that could be used but for convenience the previously provided shell script also contains needed commands to create StatSVN statistics that also contain links to the ViewVC interface. Please refer to StatSVN documentation for more information about its usage and example statistics.

Other related CVS utilities that might be mentioned here are LXR, a general purpose source code indexer and cross-referencer, and Bonsai, a tool that lets one to perform queries on the contents of a CVS archive. No similar utilities usable with SVN are available for the time being. Once such utilities exist, instructions to configure them might be added here.

Available screenshots: links to statistics generated from the repository.

MediaWiki Configuration

MediaWiki is the easiest piece in this puzzle so installing it will take only a while. Before installation, make sure that you have MySQL configured and PHP and PHP's MySQL module installed. They are usually included in distributions' package collections and require no or very little configuration. Once PHP is enabled, get latest stable MediaWiki release, unpack the downloaded archive, and move the extracted directory as /var/www/html/wiki. You must then prepare directory permissions for configuration with the commands:

cd /var/www/html/wiki
chown -R root:root . *
chmod 0777 config

MediaWiki configuration is then done with a browser simply by opening the URL http://www.dom.ain/wiki/ and answering to configuration queries. Sufficient help for most queries is available at the configuration page so most of the items are not covered here.

Shared memory caching would provide performance speedup but in this basic configuration, no caching is available. For all e-mail related queries you should answer depending on your own needs. Since MediaWiki automatically maintains history of recent changes and also provides them as an RSS feed so e-mailing them to a mailing list would not be very useful. Finally, in the Database config section the provided default values are suitable. For the field DB password use a proper password that is different from the MySQL root password that was set during MySQL installation. The previously set MySQL root password must then be entered into the field Superuser password. Note that you must also set the option Use superuser accounts. For Database charset it is recommended to select Backward-compatible UTF-8.

After hitting the Install! wait a minute until the next page is loaded. Check that it says Installation successful! and to finish the configuration enter the following commands:

cd /var/www/html/wiki
mv config/LocalSettings.php .
chmod 0700 config

MediaWiki is now installed and configured. To allow usage of the magic identifiers also in Wiki you need only to apply this patch. The patch was generated againt MediaWiki version 1.11.0 but it should apply also against later versions in the 1.11.x series. After applying the patch to the installed MediaWiki the magic identifiers turn into hyperlinks.

Now, after all your hard work, the integration should be finally completed, congratulations! Bugzilla/SVN/MediaWiki are all linked to each other and they together with the mailing lists and various SVN utilities provide easily accessible way to see how projects have evolved and improved. Changes to documentation and source code, handling of change requests and people involved in the project can be found out without any additional tools or preparations in matter of minutes whenever needed.

Available screenshots: magic identifiers in a Wiki page.

Conclusion

This document provided a detailed guide to integrate Bugzilla, SVN, MediaWiki, several SVN utilities, and mailing lists together. After integration, these standard tools provide an effective and seamless software configuration management (SCM) environment for professional software development allowing developers and managers to keep on track of projects and quality issues. Environment where everything from specifications and code to bug reports and code modifications are all integrated and easily accessible improves tremendously both quality and productivity of a project. Environment could be further enhanced with other tools such as a source tree monitoring system, a test case management system, a code review system, an integrated IDE, and other similar features. In a corporate environment, access restrictions and more sophisticated user management methods might also be needed. These kinds of configurations are, however, outside of scope of this document.

All the configuration files mentioned are browsable and available for download.

Comments and feedback can be sent to oss(at)segetech.com. We would kindly like to note that we cannot promise a personal reply to all e-mails received, especially to those asking for help in some unrelated configuration problems. Inquiries about possible cooperation should be done via addresses listed at our Contact page.

This document was last updated on 2007-11-13.