Migrating a Perl CGI application such as Bugzilla

Intro

At the bottom of the page is the full app.psgi for bugzilla.

A fairly common situation is running a perl CGI application that has been converted to run under mod_perl. There are a few ways to do this, especially running under OpenBSD.

slowcgi -- The standard way to have a cgi-bin on OpenBSD This works great for the C CGI that come with OpenBSD, but there are some downsides for perl CGI. For example, it re-compiles any scripts each time, you have to copy perl and any required libraries and modules into to chroot or run it un-chrooted.
Plack::App::CGIBin -- Another way to do the same thing, but possibly more portable to non OpenBSD systems. The difference here is that if the CGI is perl it will use CGI::Compile to load the script the first time it is run and keep it in memory to make subsequent accesses faster.

There are likely other ways to accomplish this, but the one described here uses a similar method to Plack::App::CGIBin to pre-compile an entire application, load all the required modules and then chroot into a directory and drop privileges. This makes it run faster, although not as fast as a native Plack app, and also saves memory due to Copy on Write.

Getting Started

Bugzilla is a good example because in order to run persistently it requires both a setup and teardown for each request and has many dependencies, yet has already had the hard work of writing those things done for mod_perl.

This page assumes a basic understanding of perl and plack but will hopefully provide pointers to to enough resources that you can learn enough to be successful.

Requirements

Apart from the requirements specifically for your application, just running a "Hello World" application in this way requires some modules, all of which are available as OpenBSD packages:

CGI::Compile -- Compiles a perl .cgi script into a persistent package.
CGI::Emulate::PSGI -- Turns the compiled cgi into a Plack App
Plack::App::File -- To load static files when running under the Plack standalone server
Plack::Builder -- That allows routing of all the static file apps and compiled cgi apps to be routed to correctly.
Class::Load -- could use eval "require $class" but it was already a dependency for Bugzilla and works better.
POSIX - To drop privileges with setuid and setgid

Pre-loading Modules

Because we are going to be chrooting into the Bugzilla directory, we need to load all the modules that will be needed before we do. Bugzilla has a lot of requirements just to get it going with more as you enable additional features.

There are some other interesting modules that need to be preloaded, but which ones depend on your use case. For example, depending on which DateTime::TimeZones you want to use, you will need to preload those as well as the Plack Handler and related modules you want to use and any database DBD::* modules you will be using.

In our case, some of the modules we learned from the mod_perl handler, while others were trial-and-error, running it and seeing the errors that were thrown.

# Pre-load required modules for two reasons
# 1. so we can chroot
# 2. so when we fork we can share the memory with CoW
my @requirements = qw(
	CGI::Cookie
	DateTime::TimeZone
	DateTime::TimeZone::Local::Unix
	Email::Send
	Email::Send::SMTP
	File::Compare
	Net::SMTP
	PerlIO::scalar
	Template::Context
	Template::Iterator
	Template::Namespace::Constants
	Template::Parser
	Template::Plugin
	Template::Plugins
	Template::Stash::XS
	URI
	URI::_foreign
);

foreach my $module (@requirements) {
	load_class($module);
}

# annoyingly not a class
require "bytes_heavy.pl";

# Much of this was stolen from mod_perl.cgi
# if you don't have a mod_perl enabled script, you may just have to guess at what needs to be loaded.

# This loads most of our modules.
use Bugzilla ();
# Loading Bugzilla.pm doesn't load this, though, and we want it preloaded.
use Bugzilla::BugMail ();
use Bugzilla::CGI ();
use Bugzilla::Extension ();
use Bugzilla::Install::Requirements ();
use Bugzilla::Util ();
use Bugzilla::RNG ();

# Pre-compile the CGI.pm methods that we're going to use.
Bugzilla::CGI->compile(qw(:cgi :push));

$Bugzilla::extension_packages = Bugzilla::Extension->load_all();

Loading the CGI scripts

Early in the script we chdir into the Bugzilla CGI directory so that all the modules and scripts are in the correct relative location. We find all the cgi scripts that bugzilla uses, ask if they are in use and if so, load them as a Plack application using CGI::Compile and CGI::Emulate::PSGI.

#my $cgi_path = Bugzilla::Constants::bz_locations()->{'cgi_path'};
opendir my $dh, "." or die "opendir: $!";
my @cgis = grep { $_ =~ /\.cgi$/ } readdir $dh;
closedir $dh;

my $feature_files = Bugzilla::Install::Requirements::map_files_to_features();
foreach my $script (@cgis) {
	if (my $feature = $feature_files->{$script}) {
		next if !Bugzilla->feature($feature);
	}
	Bugzilla::Util::trick_taint($script);

	# In order to chroot, need to fake CGI::Compile to use relative paths
	my $sub = do {
		no warnings 'redefine';
		local *Cwd::abs_path = sub { return join '/', '.', @_ };
		CGI::Compile->compile($script);
	};

	my $app = CGI::Emulate::PSGI->handler($sub);
	$builder->mount( "/$script" => $app );
}

Chroot and Dropping Privileges

This is one of the benefits that pre-loading all the modules gets us, the ability to chroot into the $root directory where Bugzilla lives and drop root privileges. In order to do that, you need to start app.psgi as root.

daemonize();

# In a sub so it's easy to comment out
sub daemonize {
	my $uid = getpwnam($user);
	my $gid = getgrnam($group);

	# We need to fake what mailers are available because
	# Module::Pluggable is confused by our chroot.
	{ no warnings 'redefine';
	  *Email::Send::all_mailers = sub { qw( SMTP ) };
	}

	chroot $root || die "Couldn't chroot to $root: $!";
	setgid($gid) || die "Couldn't setgid $group [$gid]: $!";
	setuid($uid) || die "Couldn't setuid $user [$uid]: $!";
}

The last thing in the file, what loading the file returns, needs to be the Plack app that we want to serve.

$builder->to_app;

Startup and Teardown

Bugzilla needs to execute some code before and after handling the request and Plack provides a very convenient place for this with a Plack middleware.

Fortunately the mod_perl script for Bugzilla told us what needed to happen, so we just duplicate that around calling $self->app($env).

There are some other important things here that I want to point out.

localize $SIG{__DIE__} because Plack has its own die handler and I prefer that to Bugzilla's.
Set $env->{PATH_INFO} because for some reason it wasn't set for me although I see the code that is supposed to do it in httpd.
Ignore $SIG{__TERM__} because Bugzilla calls exit and this seemed to stop the app from exiting. I didn't do enough testing to actually prove that though.

package Plack::Middleware::BugzillaInit;
use parent qw(Plack::Middleware);
 
sub call {
	my($self, $env) = @_;

	# pre-processing
	{ no warnings 'redefine';
	  local *lib::import = sub {};
	  local $SIG{__DIE__}; # don't let Bugzilla take it
	  Bugzilla::init_page();
	}

	# OpenBSD's httpd doesn't seem to set this properly
	$env->{PATH_INFO} ||= $env->{SCRIPT_NAME};

	# Don't let Bugzilla exit on us
	local $SIG{TERM} = 'IGNORE';

	my $res = $self->app->($env);

	# post-processing
	Bugzilla::_cleanup();

	return $res;
}

I put this Middleware at the top of the app.psgi file by faking %INC to know that it was loaded.

# Fake that we loaded the Middleware
$INC{'Plack/Middleware/BugzillaInit.pm'} = 1;

Added it to the Builder like this:

$builder->add_middleware('BugzillaInit');

Patches

Bugzilla gets the random seed for its RNG from /dev/urandom or /dev/random normally. On OpenBSD, rand() is backed by arc4random() so I was able to replace in a good way that so we could chroot. Probably their entire RNG should just use OpenBSD's arc4random() backed rand() but I haven't done that yet.

--- Bugzilla/RNG.pm.orig	Sat Feb 14 14:57:38 2015
+++ Bugzilla/RNG.pm	Sat Feb 14 15:00:34 2015
@@ -129,6 +129,7 @@ sub _check_seed {
 }
 
 sub _get_seed {
+    return CORE::rand();
     return _windows_seed() if ON_WINDOWS;
 
     if (-r '/dev/urandom') {

Running

With the below #! line to run plackup, you can just execute the app.psgi directly, that will normally start it with the Standalone server listening on a high port. Getting it to listen as an FCGI is fairly easy.

/path/to/bugzilla/app.psgi --proc-title bugzilla-fcgi -s FCGI --socket /run/bugzilla.sock -E deployment

Unfortunately you can't use the FCGI -D daemonize setting because we no longer have access to /dev/null to reopen STDIN and STDOUT. That could become part of the above daemonize subroutine but the OpenBSD rc system does a fine job of redirecting the output for us.

All together app.psgi

This assumes Bugzilla is in /var/www/bugzilla/.

#!/usr/bin/env plackup
use strict;
use warnings;

my ($user, $group) = qw( _bugzilla _bugzilla );
my $root;

# chdir early because we load Bugzilla.pm
# from a relative path
BEGIN {
	$root = '/var/www/bugzilla';
	chdir $root || die "Unable to chdir $root: $!";
}

package Plack::Middleware::BugzillaInit;
use parent qw(Plack::Middleware);
use Plack::Util;
 
sub call {
	my($self, $env) = @_;

	# pre-processing
	{ no warnings 'redefine';
	  local *lib::import = sub {};
	  local $SIG{__DIE__}; # don't let Bugzilla take it
	  Bugzilla::init_page();
	}

	# OpenBSD's httpd doesn't seem to set this properly
	$env->{PATH_INFO} ||= $env->{SCRIPT_NAME};

	# Don't let Bugzilla exit on us
	local $SIG{TERM} = 'IGNORE';

	my $res = $self->app->($env);

	# post-processing
	Bugzilla::_cleanup();

	return $res;
}
# Fake that we loaded the Middleware
$INC{'Plack/Middleware/BugzillaInit.pm'} = 1;

package main;

use CGI::Compile;
use CGI::Emulate::PSGI;
use Class::Load qw( load_class );
use POSIX qw( setuid setgid );
use Plack::App::File;
use Plack::Builder;

# Pre-load required modules for two reasons
# 1. so we can chroot
# 2. so when we fork we can share the memory with CoW
my @requirements = qw(
	CGI::Cookie
	DateTime::TimeZone
	DateTime::TimeZone::Local::Unix
	Email::Send
	Email::Send::SMTP
	File::Compare
	Net::SMTP
	PerlIO::scalar
	Template::Context
	Template::Iterator
	Template::Namespace::Constants
	Template::Parser
	Template::Plugin
	Template::Plugins
	Template::Stash::XS
	URI
	URI::_foreign
);

# Additional modules for different requirements go in this file
if (-e 'preload_modules') {
	open my $fh, '<', 'preload_modules' or die $!;
	while (<$fh>) {
		chomp;
		s/\s*\#.*$//; # not valid in module names
		next unless /\w/;
		push @requirements, $_;
	}
	close $fh;
}

foreach my $module (@requirements) {
	load_class($module);
}

# annoyingly not a class
require "bytes_heavy.pl";

# Much of this was stolen from mod_perl.cgi

# This loads most of our modules.
use Bugzilla ();
# Loading Bugzilla.pm doesn't load this, though, and we want it preloaded.
use Bugzilla::BugMail ();
use Bugzilla::CGI ();
use Bugzilla::Extension ();
use Bugzilla::Install::Requirements ();
use Bugzilla::Util ();
use Bugzilla::RNG ();

# Pre-compile the CGI.pm methods that we're going to use.
Bugzilla::CGI->compile(qw(:cgi :push));

$Bugzilla::extension_packages = Bugzilla::Extension->load_all();

my $builder = Plack::Builder->new;
$builder->add_middleware('BugzillaInit');

#my $cgi_path = Bugzilla::Constants::bz_locations()->{'cgi_path'};
opendir my $dh, "." or die "opendir: $!";
my @cgis = grep { $_ =~ /\.cgi$/ } readdir $dh;
closedir $dh;

my $feature_files = Bugzilla::Install::Requirements::map_files_to_features();
foreach my $script (@cgis) {
	if (my $feature = $feature_files->{$script}) {
		next if !Bugzilla->feature($feature);
	}
	Bugzilla::Util::trick_taint($script);

	# In order to chroot, need to fake CGI::Compile to use relative paths
	my $sub = do {
		no warnings 'redefine';
		local *Cwd::abs_path = sub { return join '/', '.', @_ };
		CGI::Compile->compile($script);
	};

	my $app = CGI::Emulate::PSGI->handler($sub);
	$builder->mount( "/$script" => $app );
}

# Below should be served by httpd
# but when running standalone, we need to do it

# Send requests for / to index.cgi
$builder->mount('/' => sub {
	return [ 301, [ Location => '/index.cgi' ], [] ];
});

foreach my $doc (qw( docs skins images js )) {
	$builder->mount("/$doc" => Plack::App::File->new(
	     root => $doc )->to_app );
}

daemonize();

# In a sub so it's easy to comment out
sub daemonize {
	my $uid = getpwnam($user);
	my $gid = getgrnam($group);

	# We need to fake what mailers are available because
	# Module::Pluggable is confused by our chroot.
	{ no warnings 'redefine';
	  *Email::Send::all_mailers = sub { qw( SMTP ) };
	}

	chroot $root || die "Couldn't chroot to $root: $!";
	setgid($gid) || die "Couldn't setgid $group [$gid]: $!";
	setuid($uid) || die "Couldn't setuid $user [$uid]: $!";
}

$builder->to_app;

preload_modules -- the extra modules needed:

# This file is where you add additional modules to preload
# for running bugzilla chrooted.
# It also helps keep memory usage low with Copy on Write.

# You will need to run /var/www/bugzilla/checksetup.pl
# which will provide further instructions on what to install

# To use SQLite as the backend:
DBD::SQLite
# And set `$db_driver = 'sqlite';` in localconfig

# You also need to copy /usr/share/zoneinfo into /var/www/bugzilla
# and create a relative symlink from etc/lcoaltime to the one you
# want in the directory where it is chrooted.
# and pre-load any TimeZones you want to use, such as:
# DateTime::TimeZone::Europe::Vienna


# For Postgres or mySQL you will need to preload those modules
# although I haven't gotten to testing those yet.

# Any other features you add will also need to be pre-loaded

# For testing with the standalone server:
Plack::Handler::Standalone

# For running as an FCGI under httpd we need these:
FCGI::ProcManager
Plack::Handler::FCGI

Example httpd.conf

ext_addr="*"

# A bugzilla server
server "default" {
        listen on $ext_addr port 80

        location "*.cgi" {
                fastcgi socket "/bugzilla/run/bugzilla.sock"
        }

	location "/" {
		block return 301 "/index.cgi"
	}

        root "/bugzilla"
}

Please direct questions and feedback to @AFresh1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrating a Perl CGI application such as Bugzilla

Intro

Getting Started

Requirements

Pre-loading Modules

Loading the CGI scripts

Chroot and Dropping Privileges

Startup and Teardown

Patches

Running

All together app.psgi

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally