Introduction
Software projects are getting larger and more complicated year after year stressing the need for efficient and reliable tools and agile project management methods. Widely used and recognized tools in Open Source development, Subversion (SVN) and Bugzilla, have been proven to be suitable for most kind of software projects and have lately been accompanied in many projects with an emerging document and content management system, Wiki. These tools are usually used separately, rarely integrated with each other to enable easier understanding and management of the whole process flow from specifications and code to releases and maintenance. Integrated tools would also provide more effective and seamless environment for software development with improved quality and productivity.
This document provides a detailed configuration guide to integrate incrementally several Open Source development tools creating a seamless development enviroment without unnecessary complexity or overhead. Bugzilla/SVN/MediaWiki are all linked to each other and they together with mailing lists and various SVN utilities provide easily accessible way to see changes to documentation and source code, handling of bug reports, and people contributing to projects without any additional tools or preparations in matter of minutes whenever needed. All this is achieved by using three special magic identifiers to create references to other tools. The identifier BugID: creates references to Bugzilla, VCS: creates references to version control system, SVN, and Wiki: creates references to Wiki. Whenever writing a Wiki document, a bug report, or an SVN commit message, these identifiers can be used and they are then automatically turned into hyperlinks pointing to the tools referenced. This enables smooth browsing from specifications to code, from code to bug reports, and back from bug reports to code changes and documentation. Additionally, all activities in projects are archived in mailing lists, SVN utilities provide graphical statistics, and bug reports contain references to actual changes or documentation that permitted closing the bug. The instructions below proceed incrementally by configuring one tool after another until the whole environment is set up. Provided screenshots illustrate results of integration. These instructions can be used of course separately to set up only parts of the presented environment or to integrate with the tools already in use. Reader is expected to have basic knowledge about the tools and their purpose.
Mailman Configuration
The following instructions to configure Mailman mailing list
manager are based on Red Hat
packaged Mailman RPMs that
have some paths different compared to official Mailman releases but
otherwise the instructions can be used with any version of Mailman. It
is of course necessary to run httpd
and
sendmail
(or some other MTA) at the server where Mailman archives are located.
First, Apache is
configured to redirect queries to /mailman
to a list info
page. This setting is recommended for all Mailman installations and
requires only adding one directive to the file
/etc/httpd/conf.d/mailman.conf
that is installed from the
Mailman RPM. Obviously, you
must change the hostname to match your local environment.
RedirectMatch ^/mailman[/]*$ http://www.dom.ain/mailman/listinfo
This setting will be effective after the service httpd
has
been restarted:
/sbin/service httpd restart
Actual Mailman configuration begins by setting Mailman site password with the command:
/usr/lib/mailman/bin/mmsitepass <your-site-password>
Main configuration file for Mailman is
/etc/mailman/mm_cfg.py
. There are dozens of configuration
directives that could be set but here only two mandatory settings and
few recommended ones are listed. The mandatory settings are:
DEFAULT_URL_HOST = 'www.dom.ain' DEFAULT_EMAIL_HOST = 'dom.ain'
The settings must, of course, be adjusted to match your local
environment. If you use localhost.localdomain
as
DEFAULT_EMAIL_HOST
then you should also add the following
line just after the previous directives:
VIRTUAL_HOSTS.clear()
Other recommended and/or useful settings include the following:
ALLOW_SITE_ADMIN_COOKIES = Yes # For secure intranets only DEFAULT_CHARSET = 'utf-8' DEFAULT_MAX_MESSAGE_SIZE = 4096 # kB # These are needed if archives are accessed only by using HTTPS DEFAULT_URL_PATTERN = 'https://%s/mailman/' PUBLIC_ARCHIVE_URL = 'https://%(hostname)s/pipermail/%(listname)s'
To update Mailman configuration after configuring needed directives the following command must be executed:
/usr/lib/mailman/bin/update
Next, for administration purposes the mailing list mailman
must be created:
/usr/lib/mailman/bin/newlist mailman
The command will ask for an e-mail address for person running the list.
The e-mail address must contain proper host and domain parts also for
local users so enter an e-mail address like
[email protected]
. Follow the provided
instructions on how to update the file /etc/aliases
to
properly process e-mails sent to the list. After updating
/etc/aliases
, the alias database must be updated with the
command:
/usr/bin/newaliases
The service mailman
that updates the archives when needed
can now be started:
/sbin/chkconfig mailman on /sbin/service mailman start
Now the actual lists for Bugzilla and SVN can finally be created. Two lists will be
used, one for archiving Bugzilla activities and one for SVN activities. For Bugzilla activities, the
list shall be prj-bugs
and for SVN commits prj-vcs
.
To create a new mailing list open the URL http://www.dom.ain/mailman/create
in a
browser and follow the instructions to create the list. Mandatory
information that must be provided when creating a list is the name of
the list, initial list owner e-mail address, and initial list password.
For other settings, default values are usually suitable. Note that after
creating the mailing lists the file /etc/aliases
must
updated as instructed in the e-mails sent to the list creator and after
the update the command newaliases(1) must be run.
After both the lists have been created configuration should be changed
to allow anyone to post to the lists. Especially with the list
prj-vcs
this eases maintenance, as there is no need to
duplicate changes to SVN rights in the
mailing list configuration. To allow all postings to a list, visit
list's admin page and go to Privacy Options->Sender Filter
and change settings By default, should new list member postings be
moderated? to No and Action to take for
postings from non-members for which no explicit action is
defined. to Accept.
The last step is to enable the magic identifiers in the mailing list
archives. Applying this
patch and restarting the mailman
service is all what is
needed to enable them. The patch was generated against Mailman 2.1.5
but it should apply also against later versions in the 2.1.x series.
Once done, the mailing list configuration is ready.
Available screenshots: commit message
in the prj-bugs
list and commit message
in the prj-vcs
list.
MySQL Configuration
Both Bugzilla and MediaWiki use MySQL database to store data so before
installing Bugzilla or MediaWiki, MySQL must be installed and
configured. These instructions provide only quick steps to get MySQL 4
or later ready for Bugzilla and MediaWiki, for any performance
optimizations or other advanced adjustments you should refer to MySQL
documentation. MySQL should be installed from a package provided
with your distribution. Note that before configuring MySQL the MySQL
database server (mysqld
) must be running.
The very first configuration step after MySQL installation is to set
MySQL root
user password. This can be done with the
following command. Note that the command will fail if the password is
already set so this command can be executed more than once if in doubt.
mysqladmin -uroot password <root-password>
Actually, this is all what is needed for MediaWiki as its installer will
set up the rest. But for Bugzilla the user bugs
must also
be added:
mysql -uroot -p<root-password> mysql GRANT SELECT, INSERT, UPDATE, DELETE, INDEX, ALTER, CREATE, LOCK TABLES, CREATE TEMPORARY TABLES, DROP, REFERENCES ON bugs.* TO bugs@localhost IDENTIFIED BY '<bugs-password>'; FLUSH PRIVILEGES;
Those are the only mandatory configurations needed for Bugzilla and
MediaWiki but you probably also want to configure mysqld
to
start during system bootup. Other MySQL configuration options can be
studied from the MySQL documentation. For reference, however, few of the
most useful options are also provided here for download. To apply these settings add
them to the file /etc/mysql.cnf
into the
[mysqld]
section.
Bugzilla Configuration
Bugzilla
configuration usually requires a bit of work but the actual procedure is
rather simple. Before installing Bugzilla, make sure that you have
installed Perl and MySQL is configured. If Perl is not installed, use
your distribution's packages to install it. Then, download latest stable Bugzilla
release, unpack the downloaded archive, and move the extracted
directory as /var/www/html/bugzilla
. To start Bugzilla
configuration, enter the Bugzilla directory, fix owner of the files and
directories, and run the checksetup.pl
script with the
parameter --check-modules
:
cd /var/www/html/bugzilla chown -R root:root . * ./checksetup.pl --check-modules
The script will most likely complain about missing Perl modules. You
must install all the required Perl modules. Whether you should
also install all or some of the optional modules mentioned depends on
many things. If you eat Perl for breakfast and already know how to
install Perl modules then it is surely no problem to install all the
optional modules, too. But if you just want to get Bugzilla up and
running and perhaps later investigate the features provided by the
optional modules then just follow the instructions provided by the
checksetup.pl
script to install the required modules. Note
that some of the modules may be found from your distribution's package
collections. If the module installation gets complicated, you may find
some help from the Bugzilla documentation. When
checksetup.pl
is finally happy, rerun the script without
any parameters:
./checksetup.pl
This time, checksetup.pl
will instruct you to edit the file
localconfig
. Most likely the only setting you want to
change is $db_pass
. Set it to match the bugs
user password you used earlier during MySQL
configuration. When done, rerun the script
checksetup.pl
again without any parameters:
./checksetup.pl
The script should proceed until it asks for an administrator. Please
enter a valid e-mail address, administrator's real name, and a proper
password. After checksetup.pl
finishes, one final step is
remaining; Apache must be configured to allow CGI scripts to be executed in the Bugzilla
installation directory. This can be done by adding the following lines
to Apache configuration file, usually
/etc/httpd/conf/httpd.conf
:
<Directory /var/www/html/bugzilla> AddHandler cgi-script .cgi Options +Indexes +ExecCGI DirectoryIndex index.cgi AllowOverride Limit </Directory>
Setting will be effective after the service httpd
has been
restarted:
/sbin/service httpd restart
Bugzilla is now installed and mostly configured and can be used with a
browser at the URL
http://www.dom.ain/bugzilla/
. Log in as administrator and
go to Parameters
page. There are several configuration
options that could be tweaked. At least the following two options should
be changed: maintainer
should contain valid e-mail address
of the Bugzilla installation maintainer and urlbase
should
contain the URL of the
Bugzilla installation. More information about all the options can be
found from the Bugzilla documentation.
After saving parameter changes, you must add two new users for
integration purposes. As administrator go to administration page
Users->add a new user
and then create users
[email protected]
and
[email protected]
. You may want to use
descriptive real names like Bugmail
and VCS
Commits
but more importantly disable the accounts by stating
their purpose in the Disable
field. The accounts will be
used to send bug activity e-mails to the prj-bugs
mailing
list and to append SVN commits log
messages to bug reports.
To enable automatic bug activity e-mails Default CC List
must be set for each component for which the feature is wanted.
Components can be found under the menu Products->Edit
Product
. Same account can be reused for as many components as
wanted and typically all components of a product have the same CC List when used for activity e-mail
generation. Example product TestProduct
and its component
TestComponent
that are automatically created during
Bugzilla installation can be used to see how the automated e-mailing
works by setting Default CC List
account of the
TestComponent
as
[email protected]
. Whenever a bug is created
against TestComponent
or status of a
TestComponent
bug changes an e-mail is sent to the
prj-bugs
mailing list and archived.
The last required step is to enable the magic identifiers in Bugzilla. Applying this patch is all what is needed to enable them. The patch was generated against Bugzilla 3.0.2 but it should apply also against later Bugzilla 3.x releases.
At this stage you might want to test your Bugzilla installation a bit by
creating a test bug or two against the example product
TestProduct
and see how linking with the magic identifiers
and automated e-mails work. You may also want to adjust some other
options, such as the e-mail templates and time tracking group, in the
Parameters
page to better suit your needs. After Bugzilla
seems to be your liking, you can proceed to the next
section.
Available screenshots: commit message in Bugzilla.
SVN Configuration
Basic set up of a Subversion repository is rather
straightforward task. The configuration presented here uses the
directory /var/lib/svnroot
as the repository and expects
that all SVN users belong to the group
users
. More information about other settings can be found
from the book Version Control with Subversion.
To initialize a new repository and set file permissions correctly the following commands must be executed:
export SVNROOT=/var/lib/svnroot svnadmin create $SVNROOT chgrp -R users $SVNROOT/db $SVNROOT/locks chmod -R g+w $SVNROOT/db $SVNROOT/locks
These commands create and initialize the repository and make sure the
files have sane permissions. Users are then able to use the created
repository via file://
, svn://
, and
svn+ssh://
URLs. More information about repository internals and
access methods can be found from the SVN book but during this
integration deeper knowledge about them is not essential. However, the
presented integration works with other access methods, too.
To add a test module prj
with some dummy content the
following commands should be executed. Please the SVN book for the rationale of
the repository
directory structure.
mkdir -p prj/trunk mkdir -p prj/tags mkdir -p prj/branches svn import prj file://$SVNROOT/prj -m "Initial repository layout." rm -rf prj svn co file://$SVNROOT/prj cd prj/trunk mkdir -p test echo foo > foo echo bar > test/bar svn add foo test svn ci -m "Imported sources." cd ../.. rm -rf prj
Now, to start the actual integration work, ViewVC (formerly ViewCVS) is
configured and patched so that the SVN
repository can be accessed with a browser and the magic identifiers are
shown as links. Either install ViewVC using the official sources or use
a ViewVC package available for your distribution. Then, apply a minor patch to the
script viewvc/lib/viewvc.py
to enable the magic
identifiers. The patch was generated against ViewVC
1.0.0 but it should apply also to later versions. In addition, some
minor changes to the
default viewvc.conf
configuration file must be done so that
ViewVC properly uses the repository created earlier. For convenience,
both the patched files are also available: viewvc.py and viewvc.conf.
Before proceeding any further, please make sure that at this stage your SVN repository is properly set up, checkouts and checkins can be done, and that the repository can be accessed with a browser via ViewVC. You should also do some test commits using the magic identifiers in the log message to see that ViewVC produces correct links for them. An example log entry could be something like:
During integration testing a problem was reported at BugID: 1 that although the specification at Wiki: The_Spec says that we should do bar, VCS: prj/trunk/foo was doing foo. Fixed to do bar.
If both the SVN repository and the
ViewVC interface are working properly then it is time to begin perhaps
the most crucial part of the integration. The Subversion repository will
be configured to send commit e-mails to the prj-vcs
mailing
list and to integrate with Bugzilla. All this is done by adding and
configuring a set of hook scripts under the directory
/var/lib/svnroot/hooks
. Since there are some
interdependencies between the scripts it is probably best to do all
required changes at a go, then test the changes, and fix any problems
that might arise.
First, svnmailer needs to be
installed and configured. Svnmailer is a tool written in Python to post commit
information by e-mail, news, or XML (via CIA). To enable some rather useful
features of recent svnmailer releases, version 1.1 or later should be
used. Earliest tested version is 1.1.0-dev-r1373. The hook scripts below
expect that or a later version. Either before or after the installation
of svnmailer, apply this patch to enable
Bugzilla/Subversion integration based on the magic identifiers. Then, a configuration file should be
placed into the directory /var/lib/svnroot/hooks
. Please
review the configuration file and edit it to match your environment.
After svnmailer is installed and configured, it can be enabled with a
set of hook scripts. The scripts must be placed into the hooks directory
/var/lib/svnroot/hooks
and they must have proper
permissions allowing their execution (i.e., mode 0755
).
Please also check that the paths defined in the scripts match your
environment. The needed scripts are provided below with some
explanations.
pre-commit and post-commit define actions to do
before and after a commit. pre-commit
makes sure that the
commit log message contains some text and post-commit
sends
a commit e-mail based on svnmailer configuration and the commit log
message. If the log message contains the magic identifier
BugID:
referencing to a bug report, the commit log
information is sent to the Bugzilla instance as a properly labeled
e-mail.
pre-revprop-change makes
sure that ordinary users are able to change only the
svn:log
revision property of the files. post-revprop-change just
sends an e-mail notification about such an event. These two scripts are
optional from the integration point of view but their usage is
recommended for administrative reasons.
post-lock and post-unlock are used to send a notification e-mail after a file is locked or unlocked. These two scripts are also optional from the integration point of view but their usage is recommended to allow easier tracking of file lockings.
For more information about these and additional hook scripts please refer to the SVN book.
Since the script post-commit
sends the commit messages to
Bugzilla as e-mails, they must be received and processed properly. The
processing is done by the script email_in.pl
that is part
of the Bugzilla installation. The script must be slightly patched to
enable it to find all required Perl modules. The following line must be
added to the file /etc/aliases
to automatically invoke the
script when sendmail
receives commit e-mails. Note that the
recipient must match to the one defined in the svnmailer configuration
file.
bugzilla:"|/var/www/html/bugzilla/email_in.pl"
And, once again, the command newaliases(1) must be run after
the alias database has been updated. After the update,
sendmail
executes the script for all e-mails sent to the
address bugzilla
. However, by default sendmail
is not allowed to run random commands while processing e-mails so the
script must be listed as an allowed command. Also the permissions in the
Bugzilla installation must be changed so that the user mail
under which sendmail
is running is allowed to read and
execute all the needed scripts. The following commands allow
sendmail
to invoke the script email_in.pl
and
change permissions so that the user mail
is actually able
to run it.
cd /etc/smrsh ln -s /var/www/html/bugzilla/email_in.pl . cd /var/www/html/bugzilla chmod g+w data/mimedump-tmp chown -R apache:mail . *
It must be noted that running again the script
checksetup.pl
or editing Bugzilla parameters will change
the file permissions so that they must be adjusted again. However, as
these are rather rare events it does not cause too much maintenance
burden but this is well worth keeping in mind in case of any integration
issues.
Before actually testing the integration, please make sure that all Perl
modules required by email_in.pl
have been installed. This
can be verified by running the script from the command line. If any
error messages are printed, some modules, such as MIME-tools, are
probably missing. If no errors are printed, the Bugzilla/SVN integration is complete! You may now
experiment with commits containing the magic identifiers and see how the
commit messages are appended to bug reports alongside with hyperlinks to
actual changes in SVN. If everything
seems to be working you are ready to proceed to the
next section.
Available screenshots: commit message in repository web interface.
SVN Utilities Configuration
This section provides quick instructions on how to use some rather
useful utilities to generate statistics from your SVN repository. Although they are not
absolutely required from the integration point of view, they all fall
into the low-hanging fruit
category and are thus worth
configuring.
CvsGraph is a utility to make graphical representations of CVS repository. No similar utility usable with SVN is available for the time being. Once such utility exists, instructions to configure it will be added here.
CVSHistory is a utility written in Python that is used to track development activity on CVS using software projects. No similar utility usable with SVN is available for the time being. Once such utility exists, instructions to configure it will be added here.
svn2cl can be used to
generate formatted changelogs based on SVN activities. It is a shell script that uses
libxslt. It is not compatible
with cl2html
but it can also produce HTML pages. Installation of svn2cl
is
simply done by unpackaging the downloaded archive, moving the extracted
directory as /usr/local/share/svn2cl
, fixing permissions,
and creating a needed symbolic link svn2cl
in a directory
contained in PATH
, e.g., /usr/local/bin
:
cd /usr/local/share/svn2cl chown -R root:root . chmod a+rx . chmod a+r * cd /usr/local/bin ln -s /usr/local/share/svn2cl/svn2cl.sh svn2cl
There are several command line parameters that could be used but for
convenience a shell script is
provided that creates GNU/HTML/XML formatted changelogs from the Subversion
repository and also a nice web page to access them. Please refer to the
documentation of the scripts for more information about their usage.
Note that the shell script requires the file $SVNROOT/conf/users to be
created and if it contains valid
mappings for names those are used when generating the changelogs. As
can be seen from the script the statictics web page created by default
is /prj-svn-stats/index.html
.
StatSVN is a tool
written in Java that is used to
create exhaustive amount of statistics from projects in an SVN repository. Installation is simply done by
placing the JAR file from the StatSVN package to a proper directory,
e.g., /usr/local/share
and running it with the help of
java
(1) command. There are several command line parameters
that could be used but for convenience the previously provided shell script also contains needed
commands to create StatSVN statistics that also contain links to the
ViewVC interface. Please refer to StatSVN documentation for
more information about its usage and example statistics.
Other related CVS utilities that might be mentioned here are LXR, a general purpose source code indexer and cross-referencer, and Bonsai, a tool that lets one to perform queries on the contents of a CVS archive. No similar utilities usable with SVN are available for the time being. Once such utilities exist, instructions to configure them might be added here.
Available screenshots: links to statistics generated from the repository.
MediaWiki Configuration
MediaWiki is the
easiest piece in this puzzle so installing it will take only a while.
Before installation, make sure that you have MySQL
configured and PHP
and PHP's MySQL module
installed. They are usually included in distributions' package
collections and require no or very little configuration. Once PHP is enabled, get latest
stable MediaWiki release, unpack the downloaded archive, and move
the extracted directory as /var/www/html/wiki
. You must
then prepare directory permissions for configuration with the commands:
cd /var/www/html/wiki chown -R root:root . * chmod 0777 config
MediaWiki configuration is then done with a browser simply by opening
the URL
http://www.dom.ain/wiki/
and answering to configuration
queries. Sufficient help for most queries is available at the
configuration page so most of the items are not covered here.
Shared memory caching would provide performance speedup but in this basic configuration, no caching is available. For all e-mail related queries you should answer depending on your own needs. Since MediaWiki automatically maintains history of recent changes and also provides them as an RSS feed so e-mailing them to a mailing list would not be very useful. Finally, in the Database config section the provided default values are suitable. For the field DB password use a proper password that is different from the MySQL root password that was set during MySQL installation. The previously set MySQL root password must then be entered into the field Superuser password. Note that you must also set the option Use superuser accounts. For Database charset it is recommended to select Backward-compatible UTF-8.
After hitting the Install! wait a minute until the next page is loaded. Check that it says Installation successful! and to finish the configuration enter the following commands:
cd /var/www/html/wiki mv config/LocalSettings.php . chmod 0700 config
MediaWiki is now installed and configured. To allow usage of the magic identifiers also in Wiki you need only to apply this patch. The patch was generated againt MediaWiki version 1.11.0 but it should apply also against later versions in the 1.11.x series. After applying the patch to the installed MediaWiki the magic identifiers turn into hyperlinks.
Now, after all your hard work, the integration should be finally completed, congratulations! Bugzilla/SVN/MediaWiki are all linked to each other and they together with the mailing lists and various SVN utilities provide easily accessible way to see how projects have evolved and improved. Changes to documentation and source code, handling of change requests and people involved in the project can be found out without any additional tools or preparations in matter of minutes whenever needed.
Available screenshots: magic identifiers in a Wiki page.
Conclusion
This document provided a detailed guide to integrate Bugzilla, SVN, MediaWiki, several SVN utilities, and mailing lists together. After integration, these standard tools provide an effective and seamless software configuration management (SCM) environment for professional software development allowing developers and managers to keep on track of projects and quality issues. Environment where everything from specifications and code to bug reports and code modifications are all integrated and easily accessible improves tremendously both quality and productivity of a project. Environment could be further enhanced with other tools such as a source tree monitoring system, a test case management system, a code review system, an integrated IDE, and other similar features. In a corporate environment, access restrictions and more sophisticated user management methods might also be needed. These kinds of configurations are, however, outside of scope of this document.
All the configuration files mentioned are browsable and available for download.
Comments and feedback can be sent to oss(at)segetech.com. We would kindly like to note that we cannot promise a personal reply to all e-mails received, especially to those asking for help in some unrelated configuration problems. Inquiries about possible cooperation should be done via addresses listed at our Contact page.
This document was last updated on 2007-11-13.