Add files using upload-large-folder tool

mosesdecoder/contrib/combine-ptables/README.md

@@ -0,0 +1,147 @@
+`combine-ptables.pl`: fill-up and other techniques of translation models combination.
+
+Author: 
+Arianna Bisazza bisazza[AT]fbk.eu
+
+ABOUT
+-----
+This tool implements "fill-up" and other operations that are useful to combine translation and reordering tables.
+In the "fill-up" approach, the weights of out-domain data sources are estimated directly by MERT along with the 
+other model weights.
+
+This tool also supports linear interpolation, but weights must be provided by the user.
+If you want to automatically estimate linear interpolation weights, use `contrib/tmcombine` instead.
+
+
+REFERENCE
+---------
+When using this script, please cite: 
+Arianna Bisazza, Nick Ruiz, and Marcello Federico. 2011. 
+"Fill-up versus Interpolation Methods for Phrase-based SMT Adaptation."
+In International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA.
+
+
+FILL-UP
+-------
+
+This combination technique is useful when the relevance of the models is known a priori,
+e.g. when one is trained on in-domain data and the others on out-of-domain data.
+
+This mode preserves all the entries and scores coming from the first model, and adds
+entries from the other models only if new.
+If more than two tables are provided, each entry is taken only from the first table 
+that contains it.
+
+Moreover, a binary feature is added for each additional table to denote the provenance
+of an entry. For in-domain entries, the binary features are all set to 1 (=exp(0)).
+Entries coming from the 2nd table will have the 1st binary feature set to 2.718 (=exp(1)).
+
+This technique was proposed in the following works:
+
+Preslav Nakov. 2008. 
+"Improving English-Spanish Statistical Machine Translation: Experiments in Domain 
+Adaptation, Sentence Paraphrasing, Tokenization, and Recasing."
+In Workshop on Statistical Machine Translation.
+
+Arianna Bisazza, Nick Ruiz, and Marcello Federico. 2011. 
+"Fill-up versus Interpolation Methods for Phrase-based SMT Adaptation."
+In International Workshop on Spoken Language Translation (IWSLT), San Francisco, CA.
+
+The latter paper contains details about the present implementation as well as an empirical
+evaluation of fill-up against other combination techniques.
+Reordering model fill-up, cascaded fill-up and pruning criteria are also discussed in the 
+same paper.
+
+Among the findings of this paper, pruning new (out-of-domain) phrases with more than 4
+source words appeared to be beneficial on the Arabic-English TED task when combining the
+in-domain models with MultiUn models.
+This corresponds to the option:
+   `--newSourceMaxLength=4`
+
+
+BACKOFF
+-------
+
+This combination technique is a simplified version of the fill-up technique.
+With respect to fill-up technique, the backoff technique does not add the 
+binary additional feature denoting the provenance of an entry.
+
+
+LINEAR INTERPOLATION
+--------------------
+
+This combination technique consists in linearly combining the feature values coming
+from all tables. The combination weights should be provided by the user, otherwise
+uniform weights are assumed.
+When a phrase pair is absent from a table, a constant value (epsilon) is assumed for 
+the corresponding feature values. You may want to set your own epsilon.
+
+See [Bisazza et al. 2011] for an empirical comparison of uniformly weighted linear 
+interpolation against fill-up and decoding-time log-linear interpolation. In that paper, 
+epsilon was always set to 1e-06.
+
+
+UNION
+-----
+
+This combination technique creates the union of all phrase pairs and assigns to each
+of them the concatenation of all tables scores. 
+
+
+INTERSECTION
+------------
+
+This combination technique creates the intersection of all phrase pairs: each phrase 
+pair that occurs in all phrase tables is output along with the feature vector taken 
+from the *first* table.
+The intersection can be used to prune the reordering table in order to match the 
+entries of a corresponding pruned phrase table.
+
+
+USAGE
+-----
+
+Get statistics about overlap of entries:
+    `combine-ptables.pl --mode=stats ptable1 ptable2 ... ptableN > ptables-overlap-stats`
+
+Interpolate phrase tables...
+- with uniform weights:
+    `combine-ptables.pl --mode=interp --phpenalty-at=4 ptable1 ptable2 ptable3 > interp-ptable.X`
+
+- with custom weights:
+    `combine-ptables.pl --mode=interp --phpenalty-at=4 --weights=0.8,0.1,0.1 ptable1 ptable2 ptable3 > interp-ptable.Y`
+
+- with custom epsilon:
+    `combine-ptables.pl --mode=interp --phpenalty-at=4 --epsilon=1e-05 ptable1 ptable2 ptable3 > interp-ptable.Z`
+
+
+Fillup phrase tables...
+- unpruned:
+    `combine-ptables.pl --mode=fillup ptable1 ptable2 ... ptableN > fillup-ptable`
+
+- pruned (new phrases only with max. 4 source words):
+    `combine-ptables.pl --mode=fillup --newSourceMaxLength=4 ptable1 ptable2 ... ptableN > fillup-ptable`
+
+
+Given a pruned phrase table, prune the corresponding reordering table:
+    `combine-ptables.pl --mode=intersect1 reotable1-unpruned ptable1-pruned > reotable1-pruned`
+
+
+NOTES
+-----
+
+The script works only with textual (non-binarized) phrase or reordering tables 
+that were *previously sorted* with `LC_ALL=C sort`
+
+The resulting combined tables are also textual and need to binarized normally.
+
+The script combine-ptables.pl can be used on lexicalized reordering tables as well.
+
+Input tables can be gzipped.
+
+When integrating filled up models into a Moses system, remember to:
+ - specify the correct number of features (typically 6) under [ttable-file] in the configuration file `moses.ini`
+ - add a weight under [weight-t] in `moses.ini`
+ - if you binarize the models, provide the correct number of features to the command:
+    `$moses/bin/processPhraseTable -ttable 0 0 - -nscores $nbFeatures`
+

mosesdecoder/contrib/combine-ptables/combine-ptables.pl

@@ -0,0 +1,449 @@
+#! /usr/bin/perl
+
+#******************************************************************************
+# Arianna Bisazza @ FBK-irst. March 2012
+#******************************************************************************
+# combine-ptables.pl : Combine Moses-style phrase tables, using different approaches
+
+
+use strict;
+use open ':utf8';
+binmode STDIN, ':utf8';
+binmode STDOUT, ':utf8';
+
+use Getopt::Long "GetOptions";
+
+sub main {
+my $usage = "
+USAGE
+-----
+combine-ptables.pl --mode=(interp|union|fillup|backoff|intersect1|stats) ptable1 ptable2 ... ptableN > combined-ptable
+combine-ptables.pl --mode=intersect1 reotable-unpruned ptable-pruned > reotable-pruned
+-----
+#
+# This scripts reads two or more *sorted* phrase tables and combines them in different modes.
+#
+# (Note: if present, word alignments are ignored).
+#
+# ----------------
+# OPTIONS
+# ----------------
+#
+# Required:
+# --mode			fillup:	    Each entry is taken only from the first table that contains it.
+#		                            A binary feature is added from each table except the first.
+# 				backoff:    Each entry is taken only from the first table that contains it.
+#		                            NO binary feature is added.
+#				interp:     Linear interpolation.
+#				union:	    Union of entries, feature vectors are concatenated.
+#                               intersect1: Intersection of entries, feature vectors taken from the first table.
+#                               stats:      Only compute some statistics about tables overlap. No table is produced.
+#
+#                               NOTE: if present, additional fields such as word alignment, phrase counts etc. are always
+#                                     taken from the first table.
+#
+# Generic options:
+# --phpenalty=FLOAT             Constant value for phrase penalty. Default is exp(1)=2.718
+# --phpenalty-at=N              The (N+1)th score of each table is considered as phrase penalty with a constant value.
+#                               In 'interp' mode, the corresponding feature is not interpolated but simply set to the constant.
+#                               In 'union' mode, the ph.penalty (constant) is output only once, after all the other scores.
+#                               By default, no score is considered as phrase penalty.
+#
+#
+# Options for 'fillup':
+# --newSourceMaxLength=INT      Don't include \"new\" source phrases if longer than INT words.
+#
+# Options for 'backoff':
+# --newSourceMaxLength=INT      Don't include \"new\" source phrases if longer than INT words.
+#
+# Options for 'interp':
+# --weights=W1,W2,...WN		Weights for interpolation. By default, uniform weights are applied.
+# --epsilon=X			Score to assume when a phrase pair is not contained in a table (in 'interp' and 'union' modes).
+#                               Default epsilon is 1e-06.
+#
+# Options for 'union':
+#
+#
+";
+
+my $combination_mode = '';
+my $debug = '';
+my $weights_str = '';
+my $epsilon = 0.000001;
+my $phPenalty = 2.718;	# exp(1)
+my $phPenalty_idx = -1; 
+my $delim= " ||| ";
+my $delim_RE= ' \|\|\| ';
+my $exp_one = 2.718;
+my $exp_zero = 1;
+my $newSourceMaxLength = -1;
+my $help = '';
+
+GetOptions ('debug' => \$debug, 
+	    'mode=s' => \$combination_mode,
+	    'weights=s' => \$weights_str,
+            'epsilon=f' => \$epsilon,
+	    'phpenalty=f' => \$phPenalty,
+            'phpenalty-at=i' => \$phPenalty_idx,
+	    'newSourceMaxLength=i' => \$newSourceMaxLength,
+            'help' => \$help);
+
+if($help) { die "$usage\n\n"; }
+
+if($combination_mode!~/(interp|union|fillup|backoff|intersect1|stats)/) {die "$usage\nUnknown combination mode!\n"}; 
+
+if(@ARGV < 2) {die "$usage\n\n Please provide at least 2 tables to combine \n\n";}
+
+print STDERR "
+WARNING: Your phrase tables must be sorted (with LC_ALL=C) !!
+******************************
+Combination mode is [$combination_mode]
+******************************
+";
+
+my @tables = @ARGV;
+my $nbtables = scalar(@tables);
+
+###########################################
+
+# The newSourceMaxLength option requires reading all the first PT before starting the combination
+my %sourcePhrasesPT1; 
+if((($combination_mode eq "fillup") || ($combination_mode eq "backoff")) && $newSourceMaxLength>-1) {
+    my $table1=$tables[0];
+    $table1 =~ s/(.*\.gz)\s*$/gzip -dc < $1|/;
+    open(TABLE1, "$table1") or die "Cannot open $table1: ($!)\n";
+    while(my $line=<TABLE1>) {
+	$line=~m/^(.*?)$delim_RE/;
+	$sourcePhrasesPT1{$1}++;
+    }
+    close(TABLE1);
+}
+
+my @table_files=();
+foreach my $table (@tables) {
+    $table =~ s/(.*\.gz)\s*$/gzip -dc < $1|/;
+    #localize the file glob, so FILE is unique to the inner loop.
+    local *FILE;
+    open(FILE, "$table") or die "Cannot open $table: ($!)\n";
+    push(@table_files, *FILE);
+}
+
+
+# Read first line from all tables to find number of weights (and sanity checks)
+my @read_ppairs=();
+my $nbscores = &read_line_from_tables(\@table_files, \@read_ppairs);
+print STDERR "Each phrase table contains $nbscores features.\n";
+
+###########################################
+
+if($phPenalty_idx!=-1) {
+    if($phPenalty_idx<0 || $phPenalty_idx>=$nbscores) {
+	die "Invalid value for option phpenalty-at! Should be in the range [0,($nbscores-1)]\n\n";
+    }
+    else { print STDERR "Phrase penalty at index $phPenalty_idx\n"; }
+}
+
+#if($weights_str ne "") { die "Weights option NOT supported yet. Can only use uniform (1/nbscores)\n\n"; }
+#my $unifw = 1/$nbtables;
+
+my @weights=(); # Array of arrays each containing the feature weights for a phrase table
+if($combination_mode eq "interp") {
+    my @table_level_weights=();
+    if($weights_str eq "") {
+	@table_level_weights= ((1/$nbtables) x $nbtables);   # assuming uniform weights
+    }
+    else {
+	@table_level_weights= split(/,/, $weights_str);
+	if(scalar(@table_level_weights) != $nbtables) {
+	    die "$usage\n Invalid string for option --weights! Must be a comma-separated list of floats, one per ph.table.\n";
+	}
+    }
+
+    for(my $i=0; $i<$nbtables; $i++) {
+	my @weights_pt = (($table_level_weights[$i]) x $nbscores);
+	if($phPenalty_idx!=-1) {
+	    $weights_pt[$phPenalty_idx]=0;
+	}
+	print STDERR "WEIGHTS-PT_$i: ", join(" -- ", @weights_pt), "\n";
+	$weights[$i] = \@weights_pt;
+    }
+    print STDERR "EPSILON: $epsilon \n";
+}
+
+
+###########################################
+
+my @empty_ppair=("");
+my @epsilons = (($epsilon) x $nbscores);
+if($phPenalty_idx>-1) {
+    pop @epsilons;
+}
+
+my $nbPpairs_inAll=0;
+my @nbPairs_found_only_in=((0) x $nbtables);
+my $MINSCORE=1;
+
+print STDERR "Working...\n\n";
+while(1) {  
+    my $min_ppair="";
+    my $reached_end_of_tables=1;
+    my @tablesContainingPpair=((0) x $nbtables);
+    for(my $i=0; $i<$nbtables; $i++) {
+	my $ppair=$read_ppairs[$i]->[0];
+	if($ppair ne "") {
+	    $reached_end_of_tables=0;
+	    if($min_ppair eq "" || $ppair lt $min_ppair) {
+		$min_ppair=$ppair;
+		@tablesContainingPpair=((0) x $nbtables);
+		$tablesContainingPpair[$i]=1;
+	    }
+	    elsif($ppair eq $min_ppair) {
+                $tablesContainingPpair[$i]=1;
+	    }
+	}
+    }
+    last if($reached_end_of_tables);
+
+    ## Actual combination is performed here:
+    &combine_ppair(\@read_ppairs, \@tablesContainingPpair);
+
+    &read_line_from_tables(\@table_files, \@read_ppairs, \@tablesContainingPpair);
+    
+}
+
+print STDERR "...done!\n";
+
+print STDERR "The minimum score in all tables is $MINSCORE\n";
+
+if($combination_mode eq "stats") {
+my $tot_ppairs=0;
+print "
+# entries
+found in all tables:   $nbPpairs_inAll\n";
+
+for(my $i=0; $i<$nbtables; $i++) {
+    print "found only in PT_$i:    $nbPairs_found_only_in[$i]\n";
+}
+
+}
+
+####################################
+sub combine_ppair(PPAIRS_REFARRAY, TABLE_INDICES_REFARRAY) {
+    my $ra_ppairs=shift;   # 1st item: phrase-pair key (string); 
+                           # 2nd item: ref.array of scores;
+                           # 3rd item: additional info (string, may be empty)
+
+    my $ra_toRead=shift;   # Important: this says which phrase tables contain the ph.pair currently processed
+
+    my $ppair="";
+    my @scores=();
+    my $additional_info="";
+
+    my $to_print=1;
+
+    if($debug) {
+	print STDERR "combine_ppair:\n";
+	for(my $i=0; $i<$nbtables; $i++) {
+	    if($ra_toRead->[$i]) {
+		print STDERR "ppair_$i= ", join (" // ", @{$ra_ppairs->[$i]}), "\n";
+	    }
+	}
+    }
+
+    if($combination_mode eq "stats") {
+	$to_print=0;
+	my $found_in=-1;
+	my $nb_found=0;
+	for(my $i=0; $i<$nbtables; $i++) {
+	    if($ra_toRead->[$i]) {
+		$found_in=$i;
+		$nb_found++;
+	    }
+	}
+	if($nb_found==1) { $nbPairs_found_only_in[$found_in]++; }
+	elsif($nb_found==$nbtables) { $nbPpairs_inAll++; }
+    }
+    ### Fill-up + additional binary feature
+    elsif($combination_mode eq "fillup") {
+	my @bin_feats=(($exp_zero) x ($nbtables-1));
+	for(my $i=0; $i<$nbtables; $i++) {
+	    if($ra_toRead->[$i]) {
+		$ppair= shift(@{$ra_ppairs->[$i]});
+		# pruning criteria are applied here:
+		if($i>0 && $newSourceMaxLength>-1) {
+		    $ppair=~m/^(.*?)$delim_RE/;
+		    if(scalar(split(/ +/, $1)) > $newSourceMaxLength &&
+			!defined($sourcePhrasesPT1{$1})) 
+		       { $to_print=0; }
+		}
+#		@scores= @{$ra_ppairs->[$i]};
+		@scores = @{shift(@{$ra_ppairs->[$i]})};
+                # binary feature for ph.pair provenance fires here
+		if($i>0) { $bin_feats[$i-1]=$exp_one; } 
+		$additional_info=shift(@{$ra_ppairs->[$i]});
+		last;
+	    }
+	}
+	push(@scores, @bin_feats);
+    }
+    ### Backoff
+    elsif($combination_mode eq "backoff") {
+        #my @bin_feats=(($exp_zero) x ($nbtables-1));
+        for(my $i=0; $i<$nbtables; $i++) {
+            if($ra_toRead->[$i]) {
+                $ppair= shift(@{$ra_ppairs->[$i]});
+                # pruning criteria are applied here:
+                if($i>0 && $newSourceMaxLength>-1) {
+                    $ppair=~m/^(.*?)$delim_RE/;
+                    if(scalar(split(/ +/, $1)) > $newSourceMaxLength &&
+                        !defined($sourcePhrasesPT1{$1}))
+                       { $to_print=0; }
+                }
+		@scores = @{shift(@{$ra_ppairs->[$i]})};
+ 		$additional_info=shift(@{$ra_ppairs->[$i]});
+                last;
+            }
+	}
+    }
+    ### Linear interpolation
+    elsif($combination_mode eq "interp") {
+	my $firstPpair=-1;
+	@scores=((0) x $nbscores);
+	for(my $i=0; $i<$nbtables; $i++) {
+	    if($ra_toRead->[$i]) {
+		if($firstPpair==-1) { $firstPpair=$i; }
+		$ppair= shift(@{$ra_ppairs->[$i]});
+		my @scoresPT = @{shift(@{$ra_ppairs->[$i]})};
+		for(my $j=0; $j<$nbscores; $j++) {
+#		    $scores[$j]+= $weights[$i]->[$j]* $ra_ppairs->[$i][$j];
+		    $scores[$j]+= $weights[$i]->[$j]* $scoresPT[$j];
+		}
+	    }
+	    else {
+		for(my $j=0; $j<$nbscores; $j++) {
+                    $scores[$j]+= $weights[$i]->[$j]* $epsilon;
+                }
+	    }
+	    if($phPenalty_idx!=-1) {
+		$scores[$phPenalty_idx]= $phPenalty;
+	    }
+	}
+	if($debug) { print STDERR "..taking info from ptable_$firstPpair\n"; }
+	$additional_info= shift(@{$ra_ppairs->[$firstPpair]});
+    }
+    ### Union + feature concatenation
+    elsif($combination_mode eq "union") {
+        my $firstPpair=-1;
+	for(my $i=0; $i<$nbtables; $i++) {
+	    if($ra_toRead->[$i]) { 
+		if($firstPpair==-1) { $firstPpair=$i; }
+		$ppair= shift(@{$ra_ppairs->[$i]});
+		my @scoresPT= @{shift(@{$ra_ppairs->[$i]})};
+		if($phPenalty_idx!=-1) {
+#		    splice(@{$ra_ppairs->[$i]}, $phPenalty_idx, 1);
+		    splice(@scoresPT, $phPenalty_idx, 1);
+		}
+#		push(@scores, @{$ra_ppairs->[$i]});
+	        push(@scores, @scoresPT);
+	    } 
+	    else { 
+		push(@scores, @epsilons);
+	    }
+	}
+	if($phPenalty_idx!=-1) { 
+	    push(@scores, $phPenalty);
+	}
+	if($debug) { print STDERR "..taking info from ptable_$firstPpair\n"; }
+        $additional_info= shift(@{$ra_ppairs->[$firstPpair]});
+    }
+    ### Intersect + features from first table
+    elsif($combination_mode eq "intersect1") {
+        $to_print=0;
+        my $found_in_all=1;
+        for(my $i=0; $i<$nbtables; $i++) {
+            if(!$ra_toRead->[$i]) {
+		$found_in_all=0;
+		last;
+            }
+        }
+        if($found_in_all) { 
+	    $to_print=1;
+	    $ppair= shift(@{$ra_ppairs->[0]});
+#	    @scores= @{$ra_ppairs->[0]};
+	    @scores= @{shift(@{$ra_ppairs->[0]})};
+	    $additional_info= shift(@{$ra_ppairs->[0]});
+	}
+    }
+    else {
+	die "$usage\nUnknown combination mode!\n";
+    }
+
+
+    if($to_print) {
+	if($additional_info eq "") {
+	    print $ppair, join(" ", @scores), "\n";
+	}else {
+	    print $ppair, join(" ", @scores), $delim, $additional_info, "\n";
+	}
+    }
+}
+
+####################################
+# Read lines from all filehandles given in FILES_REFARRAY, 
+# or from the files whose indices are assigned 1 in the array TABLE_INDICES_REFARRAY
+# Parse each of them as a phrase pair entry and stores it to the corresponding position of PPAIRS_REFARRAY
+sub read_line_from_tables(FILES_REFARRAY, PPAIRS_REFARRAY, TABLE_INDICES_REFARRAY) {
+    my $ra_files=shift;
+    my $ra_ppairs=shift;
+
+    my $ra_toRead=shift;
+    my @toRead=((1) x $nbtables);   # by default read from all files
+    if($ra_toRead ne "") { 
+	@toRead=@$ra_toRead;
+    }
+
+    my $nbscores=-1;
+    my $key=""; my $additional_info="";
+    for(my $i=0; $i<$nbtables; $i++) {
+	next if($toRead[$i]==0);
+	my @ppair=();
+	my $file=$ra_files->[$i];
+	if(my $line = <$file>) { 
+	    chomp $line;
+	    my @fields = split(/$delim_RE/, $line);
+	    if(scalar(@fields)<3) {
+                die "Invalid phrase table entry:\n$line\n";
+	    }
+	    my @scores = split(/\s+/, $fields[2]);
+	    foreach my $score (@scores) {
+		if($score<$MINSCORE) { $MINSCORE=$score; }
+	    }
+	    # Get nb of scores from the 1st table. Check that all tables provide the same nb of scores,
+	    # unless mode is 'intersect' (then it doesn't matter as scores are taken only from 1st table)
+	    if($nbscores==-1) {
+		$nbscores=scalar(@scores);
+	    } elsif($nbscores!=scalar(@scores) && $combination_mode ne "intersect1") {
+		die "Wrong number of scores in table-$i! Should be $nbscores\n";
+	    }
+	    # Get additional fields if any (word aligment, phrase counts etc.)
+	    if(scalar(@fields)>3) {
+		$additional_info=join($delim, splice(@fields,3));
+		#print STDOUT "additional_info:__{$additional_info}__\n";
+	    }
+	    my $key = "$fields[0]$delim$fields[1]$delim";  ## IMPORTANT: the | delimiter at the end of the phrase pair is crucial to preserve sorting!!
+	    push(@ppair, $key, \@scores, $additional_info);
+	}
+	else { 
+	    push(@ppair, "");
+	}
+	$ra_ppairs->[$i]=\@ppair;
+    }
+
+    return $nbscores;
+}
+
+#########
+}
+
+
+&main;

mosesdecoder/contrib/m4m/Jamfile

@@ -0,0 +1,12 @@
+external-lib bzip2 ;
+external-lib zlib ;
+
+exe merge-sorted : 
+util/merge-sorted.cc 
+$(TOP)/moses/TranslationModel/UG/mm//mm 
+$(TOP)/moses/TranslationModel/UG/generic//generic 
+$(TOP)//boost_iostreams 
+$(TOP)//boost_program_options 
+; 
+
+install $(PREFIX)/bin :  merge-sorted ; 

mosesdecoder/contrib/m4m/Makefile

@@ -0,0 +1,2 @@
+merge-sorted:
+	g++ -O3 -I ../.. util/merge-sorted.cc ../../moses/TranslationModel/UG/generic/file_io/ug_stream.cpp -o $@ -lboost_iostreams -lboost_program_options -lbz2 -lz 

mosesdecoder/contrib/m4m/README

@@ -0,0 +1,24 @@
+M4M - Makefiles for Moses
+
+Makefiles for Moses (M4M) is a collection of make file modules designed to
+facilitate building and experimenting with Moses MT systems. The long story is
+in the paper (m4m.pdf).
+
+You'll need the following additional tools (in your PATH):
+- wget if you want to use m4m to download and install WMT data sets 
+- Gnu parallel: http://www.gnu.org/software/parallel/
+- a word aligner (mgiza or fastalign or Berkeley Aligner)
+
+******************************************************************************
+*        MAKEFILES FOR MOSES IS STILL UNDER INTENSIVE DEVELOPMENT.           *
+*  BEAR WITH ME WHILE BUGS ARE BEING FIXED AND FUNCTIONALITY IS ADDED.       *
+******************************************************************************
+
+KNOWN ISSUES: 
+
+- giza phrase table creation sometimes fails when make is run in
+  multi-threaded mode. Running make without the -j switch seems to work.
+
+
+
+Ulrich Germann

mosesdecoder/contrib/picaro/zh/README

@@ -0,0 +1,4 @@
+Chinese-English sample
+sample.f	Chinese text
+sample.e 	English text
+sample.a	Alignment file with links in f-e format

mosesdecoder/contrib/picaro/zh/alternate.aln

@@ -0,0 +1 @@
+0-0 1-1 2-6 3-2 3-3 4-4 4-5 5-7

mosesdecoder/contrib/picaro/zh/sample.aln

@@ -0,0 +1 @@
+0-0 1-1 2-6 3-2 3-3 3-4 4-5 5-7

mosesdecoder/contrib/picaro/zh/sample.e

@@ -0,0 +1 @@
+i want to go to china tomorrow .

mosesdecoder/contrib/picaro/zh/sample.f

@@ -0,0 +1 @@
+我 想 明天 去 中国 .

mosesdecoder/contrib/promix/test_data/esen.ep.model.filtered/phrase-table.0-0.1.1.binphr.srcvoc

@@ -0,0 +1,8 @@
+7 del
+6 lados
+5 ambos
+4 en
+3 políticos
+2 los
+1 a
+0 Atlántico

mosesdecoder/contrib/server/Translation-web/src/conf/MANIFEST.MF

@@ -0,0 +1,2 @@
+Manifest-Version: 1.0
+

mosesdecoder/contrib/server/Translation-web/src/java/com/hpl/mt/Translate.java

@@ -0,0 +1,129 @@
+package com.hpl.mt;
+
+/*
+ * To change this template, choose Tools | Templates
+ * and open the template in the editor.
+ */
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.util.HashMap;
+import java.util.logging.Level;
+import java.util.logging.Logger;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.xmlrpc.XmlRpcException;
+import org.apache.xmlrpc.client.XmlRpcClient;
+import org.apache.xmlrpc.client.XmlRpcClientConfigImpl;
+
+/**
+ *
+ * @author ulanov
+ */
+public class Translate extends HttpServlet {
+
+    /**
+     * Processes requests for both HTTP
+     * <code>GET</code> and
+     * <code>POST</code> methods.
+     *
+     * @param request servlet request
+     * @param response servlet response
+     * @throws ServletException if a servlet-specific error occurs
+     * @throws IOException if an I/O error occurs
+     */
+    protected void processRequest(HttpServletRequest request, HttpServletResponse response)
+            throws ServletException, IOException {
+        response.setContentType("text/html;charset=UTF-8");
+        System.out.println("before" + request.getCharacterEncoding());
+        request.setCharacterEncoding("UTF-8");
+        System.out.println("after" + request.getCharacterEncoding());
+        PrintWriter out = response.getWriter();
+        try {
+            /*
+             * TODO output your page here. You may use following sample code.
+             */
+                // Create an instance of XmlRpcClient
+                String textToTranslate = request.getParameter("text");
+                XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
+                config.setServerURL(new URL("http://localhost:9008/RPC2"));
+                XmlRpcClient client = new XmlRpcClient();
+                client.setConfig(config);
+                // The XML-RPC data type used by mosesserver is <struct>. In Java, this data type can be represented using HashMap.
+                HashMap<String,String> mosesParams = new HashMap<String,String>();
+                mosesParams.put("text", textToTranslate);
+                mosesParams.put("align", "true");
+                mosesParams.put("report-all-factors", "true");
+                // The XmlRpcClient.execute method doesn't accept Hashmap (pParams). It's either Object[] or List. 
+                Object[] params = new Object[] { null };
+                params[0] = mosesParams;
+                // Invoke the remote method "translate". The result is an Object, convert it to a HashMap.
+                HashMap result;
+                try {
+                    result = (HashMap)client.execute("translate", params);
+                } catch (XmlRpcException ex) {
+                    Logger.getLogger(Translate.class.getName()).log(Level.SEVERE, null, ex);
+                    throw new IOException("XML-RPC failed");
+                }
+                // Print the returned results
+                String textTranslation = (String)result.get("text");
+                System.out.println("Input : "+textToTranslate);
+                System.out.println("Translation : "+textTranslation);
+                out.write(textTranslation);
+                if (result.get("align") != null){ 
+                        Object[] aligns = (Object[])result.get("align");
+                        System.out.println("Phrase alignments : [Source Start:Source End][Target Start]"); 
+                        for ( Object element : aligns) {
+                                HashMap align = (HashMap)element;	
+                                System.out.println("["+align.get("src-start")+":"+align.get("src-end")+"]["+align.get("tgt-start")+"]");
+                        }
+                }				
+        } finally {            
+            out.close();
+        }
+    }
+
+    // <editor-fold defaultstate="collapsed" desc="HttpServlet methods. Click on the + sign on the left to edit the code.">
+    /**
+     * Handles the HTTP
+     * <code>GET</code> method.
+     *
+     * @param request servlet request
+     * @param response servlet response
+     * @throws ServletException if a servlet-specific error occurs
+     * @throws IOException if an I/O error occurs
+     */
+    @Override
+    protected void doGet(HttpServletRequest request, HttpServletResponse response)
+            throws ServletException, IOException {
+        processRequest(request, response);
+    }
+
+    /**
+     * Handles the HTTP
+     * <code>POST</code> method.
+     *
+     * @param request servlet request
+     * @param response servlet response
+     * @throws ServletException if a servlet-specific error occurs
+     * @throws IOException if an I/O error occurs
+     */
+    @Override
+    protected void doPost(HttpServletRequest request, HttpServletResponse response)
+            throws ServletException, IOException {
+        processRequest(request, response);
+    }
+
+    /**
+     * Returns a short description of the servlet.
+     *
+     * @return a String containing servlet description
+     */
+    @Override
+    public String getServletInfo() {
+        return "Short description";
+    }// </editor-fold>
+}

mosesdecoder/contrib/server/Translation-web/web/META-INF/context.xml

@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Context antiJARLocking="true" path="/Translation"/>

mosesdecoder/contrib/server/Translation-web/web/WEB-INF/web.xml

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
+    <servlet>
+        <servlet-name>Translate</servlet-name>
+        <servlet-class>com.hpl.mt.Translate</servlet-class>
+    </servlet>
+    <servlet-mapping>
+        <servlet-name>Translate</servlet-name>
+        <url-pattern>/Translate</url-pattern>
+    </servlet-mapping>
+    <session-config>
+        <session-timeout>
+            30
+        </session-timeout>
+    </session-config>
+</web-app>

mosesdecoder/contrib/server/Translation-web/web/css/common.css

@@ -0,0 +1,22 @@
+/* 
+    Document   : common
+    Created on : Jul 31, 2012, 11:53:29 AM
+    Author     : ulanov
+    Description:
+        Purpose of the stylesheet follows.
+*/
+
+root { 
+    display: block;
+}
+
+body {font-size:small; font-family: Verdana,Arial,sans-serif;height:auto; width: auto;}
+span {font-size:medium;}
+
+#north_tab {height: 10%; width: 100%; float: top;}
+#south_tab {height: 80%; width: 100%; float: bottom;}
+
+#input_text {height: 50%; width: 30%; margin-right: 10px; float: left;}
+#output_text {height: 50%; width: 30%; margin-right: 10px; float: left;}
+
+#translate {float: left; margin-right: 10px;}

mosesdecoder/jam-files/boost-build/tools/boostbook.jam

@@ -0,0 +1,730 @@
+# Copyright 2003, 2004, 2005 Dave Abrahams 
+# Copyright 2003, 2004, 2005 Douglas Gregor 
+# Copyright 2005, 2006, 2007 Rene Rivera 
+# Copyright 2003, 2004, 2005 Vladimir Prus 
+# Distributed under the Boost Software License, Version 1.0. 
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt) 
+
+#  This module defines rules to handle generation of documentation
+#  from BoostBook sources.
+#
+#  The type of output is controlled by the <format> feature which can
+#  have the following values::
+#
+#   * html: Generates html documention.  This is the default.
+#   * xhtml: Generates xhtml documentation
+#   * htmlhelp: Generates html help output.
+#   * onehtml: Generates a single html page.
+#   * man: Generates man pages.
+#   * pdf: Generates pdf documentation.
+#   * ps: Generates postscript output.
+#   * docbook: Generates docbook XML.
+#   * fo: Generates XSL formating objects.
+#   * tests: Extracts test cases from the boostbook XML.
+#
+#  format is an implicit feature, so typing pdf on the command
+#  line (for example) is a short-cut for format=pdf.
+
+import "class" : new ;
+import common ;
+import errors ;
+import targets ;
+import feature ;
+import generators ;
+import print ;
+import property ;
+import project ;
+import property-set ;
+import regex ;
+import scanner ;
+import sequence ;
+import make ;
+import os ;
+import type ;
+import modules path project ;
+import build-system ;
+
+import xsltproc : xslt xslt-dir ;
+
+# Make this module into a project.
+project.initialize $(__name__) ;
+project boostbook ;
+
+
+feature.feature format : html xhtml htmlhelp onehtml man pdf ps docbook fo tests
+  : incidental implicit composite propagated ;
+
+type.register DTDXML : dtdxml ;
+type.register XML : xml ;
+type.register BOOSTBOOK : boostbook : XML ;
+type.register DOCBOOK : docbook : XML ; 
+type.register FO : fo : XML ;
+type.register PDF : pdf ;
+type.register PS : ps ;
+type.register XSLT : xsl : XML ;
+type.register HTMLDIR ;
+type.register XHTMLDIR ;
+type.register HTMLHELP ;
+type.register MANPAGES ;
+type.register TESTS : tests ;
+# Artificial target type, used to require invocation of top-level
+# BoostBook generator.
+type.register BOOSTBOOK_MAIN ;
+
+
+# Initialize BoostBook support.
+rule init (
+      docbook-xsl-dir ? # The DocBook XSL stylesheet directory. If not
+                        # provided, we use DOCBOOK_XSL_DIR from the environment
+                        # (if available) or look in standard locations.
+                        # Otherwise, we let the XML processor load the
+                        # stylesheets remotely.
+            
+    : docbook-dtd-dir ? # The DocBook DTD directory. If not provided, we use
+                        # DOCBOOK_DTD_DIR From the environment (if available) or
+                        # look in standard locations.  Otherwise, we let the XML
+                        # processor load the DTD remotely.
+
+    : boostbook-dir ?   # The BoostBook directory with the DTD and XSL subdirs.
+)
+{
+
+  if ! $(.initialized) 
+  {
+    .initialized = true ;
+    
+    check-boostbook-dir $(boostbook-dir) ;
+    find-tools $(docbook-xsl-dir) : $(docbook-dtd-dir) : $(boostbook-dir) ;
+
+    # Register generators only if we've were called via "using boostbook ; "
+    generators.register-standard boostbook.dtdxml-to-boostbook : DTDXML : XML ;
+    generators.register-standard boostbook.boostbook-to-docbook : XML : DOCBOOK ;
+    generators.register-standard boostbook.boostbook-to-tests : XML : TESTS ;
+    generators.register-standard boostbook.docbook-to-onehtml : DOCBOOK : HTML ;
+    generators.register-standard boostbook.docbook-to-htmldir : DOCBOOK : HTMLDIR ;
+    generators.register-standard boostbook.docbook-to-xhtmldir : DOCBOOK : XHTMLDIR ;
+    generators.register-standard boostbook.docbook-to-htmlhelp : DOCBOOK : HTMLHELP ;
+    generators.register-standard boostbook.docbook-to-manpages : DOCBOOK : MANPAGES ;
+    generators.register-standard boostbook.docbook-to-fo : DOCBOOK : FO ;
+    
+    # The same about Jamfile main target rules.
+    IMPORT $(__name__) : boostbook : : boostbook ;
+  }
+  else
+  {
+    if $(docbook-xsl-dir) 
+    {
+      modify-config ;
+      .docbook-xsl-dir = [ path.make $(docbook-xsl-dir) ] ;
+      check-docbook-xsl-dir ;
+    }
+    if $(docbook-dtd-dir) 
+    {
+      modify-config ;
+      .docbook-dtd-dir = [ path.make $(docbook-dtd-dir) ] ;
+      check-docbook-dtd-dir ;
+    }
+    if $(boostbook-dir)
+    {
+      modify-config ;
+      check-boostbook-dir $(boostbook-dir) ;
+      local boostbook-xsl-dir = [ path.glob $(boostbook-dir) : xsl ] ;
+      local boostbook-dtd-dir = [ path.glob $(boostbook-dir) : dtd ] ;
+      .boostbook-xsl-dir = $(boostbook-xsl-dir[1]) ;
+      .boostbook-dtd-dir = $(boostbook-dtd-dir[1]) ;
+      check-boostbook-xsl-dir ;
+      check-boostbook-dtd-dir ;
+    }
+  }
+}
+
+rule lock-config ( )
+{
+  if ! $(.initialized)
+  {
+    errors.user-error "BoostBook has not been configured." ;
+  }
+  if ! $(.config-locked)
+  {
+    .config-locked = true ;
+  }
+}
+
+rule modify-config ( )
+{
+  if $(.config-locked)
+  {
+    errors.user-error "BoostBook configuration cannot be changed after it has been used." ;
+  }
+}
+
+rule find-boost-in-registry ( keys * )
+{
+  local boost-root = ;
+  for local R in $(keys) 
+  {
+    local installed-boost = [ W32_GETREG
+      "HKEY_LOCAL_MACHINE\\SOFTWARE\\$(R)"
+      : "InstallRoot" ] ;
+    if $(installed-boost)
+    {
+      boost-root += [ path.make $(installed-boost) ] ;
+    }
+  }
+  return $(boost-root) ;
+}
+
+rule check-docbook-xsl-dir ( )
+{
+  if $(.docbook-xsl-dir)
+  {
+    if ! [ path.glob $(.docbook-xsl-dir) : common/common.xsl ]
+    {
+      errors.user-error "BoostBook: could not find docbook XSL stylesheets in:" [ path.native $(.docbook-xsl-dir) ] ;
+    }
+    else
+    {
+      if --debug-configuration in [ modules.peek : ARGV ]
+      {
+        ECHO "notice: BoostBook: found docbook XSL stylesheets in:" [ path.native $(.docbook-xsl-dir) ] ;
+      }
+    }
+  }
+}
+
+rule check-docbook-dtd-dir ( )
+{
+  if $(.docbook-dtd-dir)
+  {
+    if ! [ path.glob $(.docbook-dtd-dir) : docbookx.dtd ]
+    {
+      errors.user-error "error: BoostBook: could not find docbook DTD in:" [ path.native $(.docbook-dtd-dir) ] ;
+    }
+    else
+    {
+      if --debug-configuration in [ modules.peek : ARGV ]
+      {
+        ECHO "notice: BoostBook: found docbook DTD in:" [ path.native $(.docbook-dtd-dir) ] ;
+      }
+    }
+  }
+}
+
+rule check-boostbook-xsl-dir ( )
+{
+  if ! $(.boostbook-xsl-dir)
+  {
+    errors.user-error "error: BoostBook: could not find boostbook XSL stylesheets." ;
+  }
+  else if ! [ path.glob $(.boostbook-xsl-dir) : docbook.xsl ]
+  {
+    errors.user-error "error: BoostBook: could not find docbook XSL stylesheets in:" [ path.native $(.boostbook-xsl-dir) ] ;
+  }
+  else
+  {
+    if --debug-configuration in [ modules.peek : ARGV ]
+    {
+      ECHO "notice: BoostBook: found boostbook XSL stylesheets in:" [ path.native $(.boostbook-xsl-dir) ] ;
+    }
+  }
+}
+
+rule check-boostbook-dtd-dir ( )
+{
+  if ! $(.boostbook-dtd-dir)
+  {
+    errors.user-error "error: BoostBook: could not find boostbook DTD." ;
+  }
+  else if ! [ path.glob $(.boostbook-dtd-dir) : boostbook.dtd ]
+  {
+    errors.user-error "error: BoostBook: could not find boostbook DTD in:" [ path.native $(.boostbook-dtd-dir) ] ;
+  }
+  else
+  {
+    if --debug-configuration in [ modules.peek : ARGV ]
+    {
+      ECHO "notice: BoostBook: found boostbook DTD in:" [ path.native $(.boostbook-dtd-dir) ] ;
+    }
+  }
+}
+
+rule check-boostbook-dir ( boostbook-dir ? )
+{
+  if $(boostbook-dir) && ! [ path.glob $(boostbook-dir) : xsl ]
+  {
+    errors.user-error "error: BoostBook: could not find boostbook in:" [ path.native $(boostbook-dir) ] ;
+  }
+}
+
+rule find-tools ( docbook-xsl-dir ? : docbook-dtd-dir ? : boostbook-dir ? )
+{
+  docbook-xsl-dir ?= [ modules.peek : DOCBOOK_XSL_DIR ] ;
+  docbook-dtd-dir ?= [ modules.peek : DOCBOOK_DTD_DIR ] ;
+  boostbook-dir ?= [ modules.peek : BOOSTBOOK_DIR ] ;
+
+  # Look for the boostbook stylesheets relative to BOOST_ROOT
+  # and Boost.Build.
+  local boost-build-root = [ path.make [ build-system.location ] ] ;
+  local boostbook-search-dirs = [ path.join $(boost-build-root) .. .. ] ;
+
+  local boost-root =  [ modules.peek : BOOST_ROOT ] ;
+  if $(boost-root)
+  {
+    boostbook-search-dirs += [ path.join [ path.make $(boost-root) ] tools ] ;
+  }
+  boostbook-dir ?= [ path.glob $(boostbook-search-dirs) : boostbook* ] ; 
+
+  # Try to find the tools in platform specific locations
+  if [ os.name ] = NT
+  {
+    # If installed by the Boost installer.
+    local boost-root = ;
+
+    local boost-installer-versions = snapshot cvs 1.33.0 ;
+    local boost-consulting-installer-versions = 1.33.1 1.34.0 1.34.1 ;
+    local boostpro-installer-versions =
+        1.35.0 1.36.0 1.37.0 1.38.0 1.39.0 1.40.0 1.41.0 1.42.0
+        1.43.0 1.44.0 1.45.0 1.46.0 1.47.0 1.48.0 1.49.0 1.50.0 ;
+
+    local old-installer-root = [ find-boost-in-registry Boost.org\\$(boost-installer-versions) ] ;
+
+    # Make sure that the most recent version is searched for first
+    boost-root += [ sequence.reverse
+      [ find-boost-in-registry
+        Boost-Consulting.com\\$(boost-consulting-installer-versions)
+        boostpro.com\\$(boostpro-installer-versions) ] ] ;
+
+    # Plausible locations.
+    local root = [ PWD ] ;
+    while $(root) != $(root:D) { root = $(root:D) ; }
+    root = [ path.make $(root) ] ;
+    local search-dirs = ;
+    local docbook-search-dirs = ;
+    for local p in $(boost-root) {
+      search-dirs += [ path.join $(p) tools ] ;
+    }
+    for local p in $(old-installer-root)
+    {
+      search-dirs += [ path.join $(p) share ] ;
+      docbook-search-dirs += [ path.join $(p) share ] ;
+    }
+    search-dirs += [ path.join $(root) Boost tools ] ;
+    search-dirs += [ path.join $(root) Boost share ] ;
+    docbook-search-dirs += [ path.join $(root) Boost share ] ;
+
+    docbook-xsl-dir ?= [ path.glob $(docbook-search-dirs) : docbook-xsl* ] ;
+    docbook-dtd-dir ?= [ path.glob $(docbook-search-dirs) : docbook-xml* ] ;
+    boostbook-dir ?= [ path.glob $(search-dirs) : boostbook* ] ;
+  }
+  else
+  {
+    # Plausible locations.
+
+    local share = /usr/local/share /usr/share /opt/share /opt/local/share ;
+    local dtd-versions = 4.2 ;
+
+    docbook-xsl-dir ?= [ path.glob $(share) : docbook-xsl* ] ;
+    docbook-xsl-dir ?= [ path.glob $(share)/sgml/docbook : xsl-stylesheets ] ;
+    docbook-xsl-dir ?= [ path.glob $(share)/xsl : docbook* ] ;
+
+    docbook-dtd-dir ?= [ path.glob $(share) : docbook-xml* ] ;
+    docbook-dtd-dir ?= [ path.glob $(share)/sgml/docbook : xml-dtd-$(dtd-versions)* ] ;
+    docbook-dtd-dir ?= [ path.glob $(share)/xml/docbook : $(dtd-versions) ] ;
+
+    boostbook-dir ?= [ path.glob $(share) : boostbook* ] ;
+
+    # Ubuntu Linux
+    docbook-xsl-dir ?= [ path.glob /usr/share/xml/docbook/stylesheet : nwalsh ] ;
+    docbook-dtd-dir ?= [ path.glob /usr/share/xml/docbook/schema/dtd : $(dtd-versions) ] ;
+
+    # SUSE
+    docbook-xsl-dir ?= [ path.glob /usr/share/xml/docbook/stylesheet/nwalsh : current ] ;
+  }
+
+  if $(docbook-xsl-dir) 
+  {
+    .docbook-xsl-dir = [ path.make $(docbook-xsl-dir[1]) ] ;
+  }
+  if $(docbook-dtd-dir) 
+  {
+    .docbook-dtd-dir = [ path.make $(docbook-dtd-dir[1]) ] ;
+  }
+
+  if --debug-configuration in [ modules.peek : ARGV ] 
+  {
+    ECHO "notice: Boost.Book: searching XSL/DTD in" ;
+    ECHO "notice:" [ sequence.transform path.native : $(boostbook-dir) ] ;
+  }    
+  local boostbook-xsl-dir ;
+  for local dir in $(boostbook-dir) {
+    boostbook-xsl-dir += [ path.glob $(dir) : xsl ] ;
+  }
+  local boostbook-dtd-dir ; 
+  for local dir in $(boostbook-dir) {
+    boostbook-dtd-dir += [ path.glob $(dir) : dtd ] ;
+  }
+  .boostbook-xsl-dir = $(boostbook-xsl-dir[1]) ;
+  .boostbook-dtd-dir = $(boostbook-dtd-dir[1]) ;
+
+  check-docbook-xsl-dir ;
+  check-docbook-dtd-dir ;
+  check-boostbook-xsl-dir ;
+  check-boostbook-dtd-dir ; 
+}
+
+rule xsl-dir
+{
+  lock-config ;
+  return $(.boostbook-xsl-dir) ;
+}
+
+rule dtd-dir
+{
+  lock-config ;
+  return $(.boostbook-dtd-dir) ;
+}
+
+rule docbook-xsl-dir
+{
+  lock-config ;
+  return $(.docbook-xsl-dir) ;
+}
+
+rule docbook-dtd-dir
+{
+  lock-config ;
+  return $(.docbook-dtd-dir) ;
+}
+
+rule dtdxml-to-boostbook ( target : source : properties * )
+{
+  lock-config ;
+  xslt $(target) : $(source) "$(.boostbook-xsl-dir)/dtd/dtd2boostbook.xsl" 
+                 : $(properties) ;
+}
+
+rule boostbook-to-docbook ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/docbook.xsl ] ;
+  xslt $(target) : $(source) $(stylesheet) : $(properties) ;
+}
+
+rule docbook-to-onehtml ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/html-single.xsl ] ;
+  xslt $(target) : $(source) $(stylesheet) : $(properties) ;
+}
+
+rule docbook-to-htmldir ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/html.xsl ] ;
+  xslt-dir $(target) : $(source) $(stylesheet) : $(properties) : html ;
+}
+
+rule docbook-to-xhtmldir ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/xhtml.xsl ] ;
+  xslt-dir $(target) : $(source) $(stylesheet) : $(properties) : xhtml ;
+}
+
+rule docbook-to-htmlhelp ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/html-help.xsl ] ;
+  xslt-dir $(target) : $(source) $(stylesheet) : $(properties) : htmlhelp ;
+}
+
+rule docbook-to-manpages ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/manpages.xsl ] ;
+  xslt-dir $(target) : $(source) $(stylesheet) : $(properties) : man ;
+}
+
+rule docbook-to-fo ( target : source : properties * )
+{
+  lock-config ;
+  local stylesheet = [ path.native $(.boostbook-xsl-dir)/fo.xsl ] ;
+  xslt $(target) : $(source) $(stylesheet) : $(properties) ;
+}
+
+rule format-catalog-path ( path )
+{
+    local result = $(path) ;
+    if [ xsltproc.is-cygwin ]
+    {
+        if [ os.name ] = NT
+        {
+            drive = [ MATCH ^/(.):(.*)$ : $(path) ] ;
+            result = /cygdrive/$(drive[1])$(drive[2]) ;
+        }
+    }
+    else
+    {
+        if [ os.name ] = CYGWIN
+        {
+            local native-path = [ path.native $(path) ] ;
+            result = [ path.make $(native-path:W) ] ;
+        }
+    }
+    return [ regex.replace $(result) " " "%20" ] ;
+}
+
+rule generate-xml-catalog ( target : sources * : properties * )
+{
+  print.output $(target) ;
+
+  # BoostBook DTD catalog entry
+  local boostbook-dtd-dir = [ boostbook.dtd-dir ] ;
+  if $(boostbook-dtd-dir)
+  {      
+    boostbook-dtd-dir = [ format-catalog-path $(boostbook-dtd-dir) ] ;
+  }
+    
+  print.text
+    "<?xml version=\"1.0\"?>"
+    "<!DOCTYPE catalog "
+    "  PUBLIC \"-//OASIS/DTD Entity Resolution XML Catalog V1.0//EN\""
+    "  \"http://www.oasis-open.org/committees/entity/release/1.0/catalog.dtd\">"
+    "<catalog xmlns=\"urn:oasis:names:tc:entity:xmlns:xml:catalog\">"
+    "  <rewriteURI uriStartString=\"http://www.boost.org/tools/boostbook/dtd/\" rewritePrefix=\"file://$(boostbook-dtd-dir)/\"/>"
+    : true ;
+
+  local docbook-xsl-dir = [ boostbook.docbook-xsl-dir ] ;    
+  if ! $(docbook-xsl-dir) 
+  {
+    ECHO "BoostBook warning: no DocBook XSL directory specified." ;
+    ECHO "  If you have the DocBook XSL stylesheets installed, please " ;
+    ECHO "  set DOCBOOK_XSL_DIR to the stylesheet directory on either " ;
+    ECHO "  the command line (via -sDOCBOOK_XSL_DIR=...) or in a " ;
+    ECHO "  Boost.Jam configuration file. The DocBook XSL stylesheets " ;
+    ECHO "  are available here: http://docbook.sourceforge.net/ " ;
+    ECHO "  Stylesheets will be downloaded on-the-fly (very slow!) " ;
+  }
+  else 
+  {
+    docbook-xsl-dir = [ format-catalog-path $(docbook-xsl-dir) ] ;      
+    print.text "  <rewriteURI uriStartString=\"http://docbook.sourceforge.net/release/xsl/current/\" rewritePrefix=\"file://$(docbook-xsl-dir)/\"/>" ;
+  }
+
+  local docbook-dtd-dir = [ boostbook.docbook-dtd-dir ] ;  
+  if ! $(docbook-dtd-dir)
+  {
+    ECHO "BoostBook warning: no DocBook DTD directory specified." ;
+    ECHO "  If you have the DocBook DTD installed, please set " ;
+    ECHO "  DOCBOOK_DTD_DIR to the DTD directory on either " ;
+    ECHO "  the command line (via -sDOCBOOK_DTD_DIR=...) or in a " ;
+    ECHO "  Boost.Jam configuration file. The DocBook DTD is available " ;
+    ECHO "  here: http://www.oasis-open.org/docbook/xml/4.2/index.shtml" ;
+    ECHO "  The DTD will be downloaded on-the-fly (very slow!) " ;
+  }
+  else 
+  {
+    docbook-dtd-dir = [ format-catalog-path $(docbook-dtd-dir) ] ;      
+    print.text "  <rewriteURI uriStartString=\"http://www.oasis-open.org/docbook/xml/4.2/\" rewritePrefix=\"file://$(docbook-dtd-dir)/\"/>" ;
+  }
+
+  print.text "</catalog>" ;
+}
+
+rule xml-catalog ( )
+{
+    if ! $(.xml-catalog)
+    {
+        # The target is created as part of the root project. But ideally
+        # it would be created as part of the boostbook project. This is not
+        # current possible as such global projects don't inherit things like
+        # the build directory.
+        
+        # Find the root project.
+        local root-project = [ project.current ] ;
+        root-project = [ $(root-project).project-module ] ;
+        while
+            [ project.attribute $(root-project) parent-module ] &&
+            [ project.attribute $(root-project) parent-module ] != user-config &&
+            [ project.attribute $(root-project) parent-module ] != project-config
+        {
+            root-project = [ project.attribute $(root-project) parent-module ] ;
+        }
+        .xml-catalog = [ new file-target boostbook_catalog
+            : XML
+            : [ project.target $(root-project) ]
+            : [ new action : boostbook.generate-xml-catalog ]
+            :
+            ] ;
+        .xml-catalog-file = [ $(.xml-catalog).path ] [ $(.xml-catalog).name ] ;
+        .xml-catalog-file = $(.xml-catalog-file:J=/) ;
+    }
+    return $(.xml-catalog) $(.xml-catalog-file) ;
+}
+
+class boostbook-generator : generator
+{
+    import feature ;
+    import virtual-target ;
+    import generators ;
+    import boostbook ;
+      
+    
+    rule __init__ ( * : * )
+    {
+        generator.__init__ $(1) : $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ;
+    }
+    
+    rule run ( project name ? : property-set : sources * )
+    {
+        # Generate the catalog, but only once...
+        local global-catalog = [ boostbook.xml-catalog ] ;
+        local catalog = $(global-catalog[1]) ;
+        local catalog-file = $(global-catalog[2]) ;
+        local targets ;
+       
+        # Add the catalog to the property set
+        property-set = [ $(property-set).add-raw <catalog>$(catalog-file) ] ;
+
+        local type = none ;
+        local manifest ; 
+        local format = [ $(property-set).get <format> ] ;
+        switch $(format) 
+        {
+            case html    : 
+            {
+                type = HTMLDIR ;
+                manifest = HTML.manifest ;
+            }
+            case xhtml    : 
+            {
+                type = XHTMLDIR ;
+                manifest = HTML.manifest ;
+            }
+            case htmlhelp    : 
+            {
+                type = HTMLHELP ;
+                manifest = HTML.manifest ;
+            }
+            
+            case onehtml : type = HTML ;
+            
+            case man : 
+            {
+                type = MANPAGES ;
+                manifest = man.manifest ;
+            }
+            
+            case docbook : type = DOCBOOK ;
+            case fo      : type = FO ;
+            case pdf     : type = PDF ;
+            case ps      : type = PS ;
+            case tests   : type = TESTS ;
+        }
+        
+        if $(manifest)
+        {
+            # Create DOCBOOK file from BOOSTBOOK sources.
+            local base-target = [ generators.construct $(project) 
+              : DOCBOOK : $(property-set) : $(sources) ] ;
+            base-target = $(base-target[2]) ;
+            $(base-target).depends $(catalog) ;
+            
+            # Generate HTML/PDF/PS from DOCBOOK.
+            local target = [ generators.construct $(project) $(name)_$(manifest)
+                : $(type)
+                : [ $(property-set).add-raw
+                    <xsl:param>manifest=$(name)_$(manifest) ]
+                : $(base-target) ] ;
+            local name = [ $(property-set).get <name> ] ;
+            name ?= $(format) ;
+            $(target[2]).set-path $(name) ;
+            $(target[2]).depends $(catalog) ;            
+
+            targets += $(target[2]) ;
+        }
+        else {
+            local target = [ generators.construct $(project)
+              : $(type) : $(property-set) : $(sources) ] ;
+            
+            if ! $(target)
+            {
+                errors.error "Cannot build documentation type '$(format)'" ;
+            }
+            else 
+            {
+                $(target[2]).depends $(catalog) ;
+                targets += $(target[2]) ;
+            }
+        }
+        
+        return $(targets) ;
+    }
+}
+
+generators.register [ new boostbook-generator boostbook.main : : BOOSTBOOK_MAIN ] ;
+
+# Creates a boostbook target.
+rule boostbook ( target-name : sources * : requirements * : default-build * )
+{ 
+  local project = [ project.current ] ;
+    
+  targets.main-target-alternative 
+    [ new typed-target $(target-name) : $(project) : BOOSTBOOK_MAIN
+        : [ targets.main-target-sources $(sources) : $(target-name) ] 
+        : [ targets.main-target-requirements $(requirements) : $(project) ]
+        : [ targets.main-target-default-build $(default-build) : $(project) ] 
+    ] ;
+}
+
+#############################################################################
+# Dependency scanners
+#############################################################################
+# XInclude scanner. Mostly stolen from c-scanner :)
+# Note that this assumes an "xi" prefix for XIncludes. This isn't always the
+# case for XML documents, but we'll assume it's true for anything we encounter.
+class xinclude-scanner : scanner 
+{
+    import virtual-target ;
+    import path ;
+    import scanner ;
+        
+    rule __init__ ( includes * )
+    {
+        scanner.__init__ ;
+        self.includes = $(includes) ;
+    }
+    
+  rule pattern ( )
+  {
+    return "xi:include[ ]*href=\"([^\"]*)\"" ;
+  }
+
+  rule process ( target : matches * : binding )
+  {
+    local target_path = [ NORMALIZE_PATH $(binding:D) ] ;
+
+    NOCARE $(matches) ;
+    INCLUDES $(target) : $(matches) ;
+    SEARCH on $(matches) = $(target_path) $(self.includes:G=) ;
+    
+    scanner.propagate $(__name__) : $(matches) : $(target) ;     
+  }
+}
+
+scanner.register xinclude-scanner : xsl:path ;
+type.set-scanner XML : xinclude-scanner ;
+
+rule boostbook-to-tests ( target : source : properties * )
+{
+  lock-config ;
+  local boost_root = [ modules.peek : BOOST_ROOT ] ;
+  local native-path =
+    [ path.native [ path.join $(.boostbook-xsl-dir) testing Jamfile ] ] ;
+  local stylesheet = $(native-path:S=.xsl) ;
+  xslt $(target) : $(source) $(stylesheet) 
+                 : $(properties) <xsl:param>boost.root=$(boost_root) 
+                 ;
+}
+
+

mosesdecoder/jam-files/boost-build/tools/como-linux.jam

@@ -0,0 +1,103 @@
+# Copyright 2004, 2005, 2006 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# The following #// line will be used by the regression test table generation
+# program as the column heading for HTML tables. Must not include a version
+# number.
+#//<a href="http://www.comeaucomputing.com/">Comeau<br>C++</a>
+
+import toolset ;
+import feature ;
+import toolset : flags ;
+import common ;
+import generators ;
+
+import unix ;
+import como ;
+
+feature.extend-subfeature toolset como : platform : linux ;
+
+toolset.inherit-generators como-linux
+     <toolset>como <toolset-como:platform>linux : unix ;
+generators.override como-linux.prebuilt : builtin.lib-generator ;
+generators.override como-linux.searched-lib-generator : searched-lib-generator ;
+toolset.inherit-flags como-linux : unix ;
+toolset.inherit-rules como-linux : gcc ;
+
+generators.register-c-compiler como-linux.compile.c++ : CPP : OBJ
+    : <toolset>como <toolset-como:platform>linux ;
+generators.register-c-compiler como-linux.compile.c : C : OBJ
+    : <toolset>como <toolset-como:platform>linux ;
+
+
+rule init ( version ? : command * : options * )
+{
+    local condition = [ common.check-init-parameters como-linux
+        : version $(version) ] ;
+
+    command = [ common.get-invocation-command como-linux : como
+        : $(command) ] ;
+
+    common.handle-options como-linux : $(condition) : $(command) : $(options) ;
+}
+
+
+flags como-linux C++FLAGS <exception-handling>off : --no_exceptions ;
+flags como-linux C++FLAGS <exception-handling>on : --exceptions ;
+
+flags como-linux CFLAGS <inlining>off : --no_inlining ;
+flags como-linux CFLAGS <inlining>on <inlining>full : --inlining ;
+
+flags como-linux CFLAGS <optimization>off : -O0 ;
+flags como-linux CFLAGS <optimization>speed : -O3 ;
+flags como-linux CFLAGS <optimization>space : -Os ;
+
+flags como-linux CFLAGS <debug-symbols>on : -g ;
+flags como-linux LINKFLAGS <debug-symbols>on : -g ;
+
+flags como-linux FINDLIBS : m ;
+flags como-linux FINDLIBS : rt ;
+
+flags como-linux CFLAGS <cflags> ;
+flags como-linux C++FLAGS <cxxflags> ;
+flags como-linux DEFINES <define> ;
+flags como-linux UNDEFS <undef> ;
+flags como-linux HDRS <include> ;
+flags como-linux STDHDRS <sysinclude> ;
+flags como-linux LINKFLAGS <linkflags> ;
+flags como-linux ARFLAGS <arflags> ;
+
+flags como-linux.link LIBRARIES <library-file> ;
+flags como-linux.link LINKPATH <library-path> ;
+flags como-linux.link FINDLIBS-ST <find-static-library> ;
+flags como-linux.link FINDLIBS-SA <find-shared-library> ;
+
+flags como-linux.link RPATH <dll-path> ;
+flags como-linux.link RPATH_LINK <xdll-path> ;
+
+
+actions link bind LIBRARIES
+{
+    $(CONFIG_COMMAND) $(LINKFLAGS) -o "$(<[1])" "$(>)" -L"$(LINKPATH)" -Wl,-R$(SPACE)-Wl,"$(RPATH)" -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)"  "$(LIBRARIES)"  "$(LIBRARIES)" -l$(FINDLIBS-SA) -l$(FINDLIBS-ST) 2>&1
+}
+
+actions link.dll bind LIBRARIES
+{
+    $(CONFIG_COMMAND) $(LINKFLAGS) -shared -o "$(<[1])" "$(>)" -L"$(LINKPATH)" -Wl,-R$(SPACE)-Wl,"$(RPATH)" -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" "$(LIBRARIES)"  "$(LIBRARIES)" -l$(FINDLIBS-SA) -l$(FINDLIBS-ST) 2>&1
+}
+
+actions compile.c
+{
+    $(CONFIG_COMMAND) -c --c99 --long_long -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) -I"$(HDRS)" -I"$(STDHDRS)" -o "$(<)" "$(>)" 2>&1
+}
+
+actions compile.c++
+{
+    $(CONFIG_COMMAND) -tused -c --long_long -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) $(C++FLAGS) -I"$(HDRS)" -I"$(STDHDRS)"  -o "$(<)" "$(>)" 2>&1
+}
+
+actions archive
+{
+    ar rcu $(<) $(>)
+}

mosesdecoder/jam-files/boost-build/tools/gfortran.jam

@@ -0,0 +1,39 @@
+# Copyright (C) 2004 Toon Knapen
+#
+#  Use, modification and distribution is subject to the Boost Software
+#  License Version 1.0. (See accompanying file LICENSE_1_0.txt or
+#  http://www.boost.org/LICENSE_1_0.txt)
+
+import toolset : flags ;
+import feature ;
+import fortran ;
+
+rule init ( version ? : command * : options * )
+{
+}
+
+# Declare flags and action for compilation
+flags gfortran OPTIONS <fflags> ;
+
+flags gfortran OPTIONS <optimization>off : -O0 ;
+flags gfortran OPTIONS <optimization>speed : -O3 ;
+flags gfortran OPTIONS <optimization>space : -Os ;
+
+flags gfortran OPTIONS <debug-symbols>on : -g ;
+flags gfortran OPTIONS <profiling>on : -pg ;
+
+flags gfortran OPTIONS <link>shared/<main-target-type>LIB : -fPIC ;
+
+flags gfortran DEFINES <define> ;
+flags gfortran INCLUDES <include> ;
+
+rule compile.fortran
+{
+}
+
+actions compile.fortran
+{
+  gcc -Wall $(OPTIONS) -D$(DEFINES) -I$(INCLUDES) -c -o "$(<)" "$(>)" 
+}
+
+generators.register-fortran-compiler gfortran.compile.fortran : FORTRAN FORTRAN90 : OBJ ; 

mosesdecoder/jam-files/boost-build/tools/message.jam

@@ -0,0 +1,55 @@
+# Copyright 2008 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# Defines main target type 'message', that prints a message when built for the
+# first time.
+
+import project ;
+import "class" : new ;
+import targets ;
+import property-set ;
+
+class message-target-class : basic-target
+{
+    rule __init__ ( name-and-dir : project : * )
+    {
+        basic-target.__init__ $(name-and-dir) : $(project) ;
+        self.3 = $(3) ;
+        self.4 = $(4) ;
+        self.5 = $(5) ;
+        self.6 = $(6) ;
+        self.7 = $(7) ;
+        self.8 = $(8) ;        
+        self.9 = $(9) ;        
+        self.built = ;
+    }
+    
+    rule construct ( name : source-targets * : property-set )
+    {
+        if ! $(self.built)
+        {
+            for i in 3 4 5 6 7 8 9
+            {
+                if $(self.$(i))
+                {
+                    ECHO $(self.$(i)) ;
+                }
+            }
+            self.built = 1 ;
+        }
+        
+        return [ property-set.empty ] ;
+    }
+}
+
+
+rule message ( name : * )
+{
+    local project = [ project.current ] ;
+
+    targets.main-target-alternative
+      [ new message-target-class $(name) : $(project) 
+        : $(2) : $(3) : $(4) : $(5) : $(6) : $(7) ] ;
+}
+IMPORT $(__name__) : message : : message ;

mosesdecoder/jam-files/boost-build/tools/pgi.jam

@@ -0,0 +1,147 @@
+#  Copyright Noel Belcourt 2007.
+#  Distributed under the Boost Software License, Version 1.0.
+#    (See accompanying file LICENSE_1_0.txt or copy at
+#          http://www.boost.org/LICENSE_1_0.txt)
+
+import property ;
+import generators ;
+import os ;
+import toolset : flags ;
+import feature ;
+import fortran ;
+import type ;
+import common ;
+import gcc ;
+
+feature.extend toolset : pgi ;
+toolset.inherit pgi : unix ;
+generators.override pgi.prebuilt : builtin.lib-generator ;
+generators.override pgi.searched-lib-generator : searched-lib-generator ;
+
+# Documentation and toolchain description located
+# http://www.pgroup.com/resources/docs.htm
+
+rule init ( version ? : command * : options * ) 
+{
+  local condition = [ common.check-init-parameters pgi : version $(version) ] ;
+
+  local l_command = [ common.get-invocation-command pgi : pgCC : $(command) ] ;
+
+  common.handle-options pgi : $(condition) : $(l_command) : $(options) ;
+    
+  command_c = $(command_c[1--2]) $(l_command[-1]:B=cc) ;
+
+  toolset.flags pgi CONFIG_C_COMMAND $(condition) : $(command_c) ;
+
+  flags pgi.compile DEFINES $(condition) :
+    [ feature.get-values <define> : $(options) ] : unchecked ;
+
+  # IOV_MAX support
+  flags pgi.compile DEFINES $(condition) : __need_IOV_MAX : unchecked ;
+
+  # set link flags
+  flags pgi.link FINDLIBS-ST : [
+    feature.get-values <find-static-library> : $(options) ] : unchecked ;
+
+  # always link lib rt to resolve clock_gettime()
+  flags pgi.link FINDLIBS-SA : rt [
+    feature.get-values <find-shared-library> : $(options) ] : unchecked ;
+
+  gcc.init-link-flags pgi gnu $(condition) ;
+}
+
+# Declare generators
+generators.register-c-compiler pgi.compile.c : C : OBJ : <toolset>pgi ;
+generators.register-c-compiler pgi.compile.c++ : CPP : OBJ : <toolset>pgi ;
+generators.register-fortran-compiler pgi.compile.fortran : FORTRAN : OBJ : <toolset>pgi ;
+
+# Declare flags and actions for compilation
+flags pgi.compile OPTIONS : -Kieee ;
+flags pgi.compile OPTIONS <link>shared : -fpic -fPIC ;
+flags pgi.compile OPTIONS <debug-symbols>on : -gopt ;
+flags pgi.compile OPTIONS <profiling>on : -xprofile=tcov ;
+flags pgi.compile OPTIONS <optimization>speed : -fast -Mx,8,0x10000000 ;
+flags pgi.compile OPTIONS <optimization>space : -xO2 -xspace ;
+# flags pgi.compile OPTIONS <threading>multi : -mt ;
+
+flags pgi.compile OPTIONS <warnings>off : -Minform=severe ;
+flags pgi.compile OPTIONS <warnings>on : -Minform=warn ;
+
+flags pgi.compile.c++ OPTIONS <inlining>off : -INLINE:none ;
+
+flags pgi.compile OPTIONS <cflags> ;
+flags pgi.compile.c++ OPTIONS <cxxflags> ;
+flags pgi.compile DEFINES <define> ;
+flags pgi.compile INCLUDES <include> ;
+
+flags pgi.compile.fortran OPTIONS <fflags> ;
+
+actions compile.c
+{
+    "$(CONFIG_C_COMMAND)" $(OPTIONS) -D$(DEFINES) -I"$(INCLUDES)" -c -o "$(<)" "$(>)"
+}
+
+actions compile.c++
+{
+    "$(CONFIG_COMMAND)" $(OPTIONS) -D$(DEFINES) -I"$(INCLUDES)" -c -o "$(<)" "$(>)"
+}
+
+actions compile.fortran
+{
+    "$(CONFIG_F_COMMAND)" $(OPTIONS) -D$(DEFINES) -I"$(INCLUDES)" -c -o "$(<)" "$(>)"
+}
+
+# Declare flags and actions for linking
+flags pgi.link OPTIONS <debug-symbols>on : -gopt ;
+# Strip the binary when no debugging is needed
+flags pgi.link OPTIONS <debug-symbols>off : -s ;
+flags pgi.link OPTIONS <profiling>on : -xprofile=tcov ;
+flags pgi.link OPTIONS <linkflags> ;
+flags pgi.link OPTIONS <link>shared : -fpic -fPIC ;
+flags pgi.link LINKPATH <library-path> ;
+flags pgi.link FINDLIBS-ST <find-static-library> ;
+flags pgi.link FINDLIBS-SA <find-shared-library> ;
+flags pgi.link FINDLIBS-SA <threading>multi : pthread rt ;
+flags pgi.link LIBRARIES <library-file> ;
+flags pgi.link LINK-RUNTIME <runtime-link>static : static ;
+flags pgi.link LINK-RUNTIME <runtime-link>shared : dynamic ;
+flags pgi.link RPATH <dll-path> ;
+
+# On gcc, there are separate options for dll path at runtime and
+# link time. On Solaris, there's only one: -R, so we have to use
+# it, even though it's bad idea.
+flags pgi.link RPATH <xdll-path> ;
+
+rule link ( targets * : sources * : properties * )
+{
+    SPACE on $(targets) = " " ;
+}
+
+# reddish can only link statically and, somehow, the presence of -Bdynamic on the link line 
+# marks the executable as a dynamically linked exec even though no dynamic libraries are supplied.
+# Yod on redstorm refuses to load an executable that is dynamically linked.
+# removing the dynamic link options should get us where we need to be on redstorm.
+# "$(CONFIG_COMMAND)" $(OPTIONS) -L"$(LINKPATH)" -R"$(RPATH)" -o "$(<)" "$(>)" "$(LIBRARIES)" -Bdynamic -l$(FINDLIBS-SA) -Bstatic -l$(FINDLIBS-ST) -B$(LINK-RUNTIME)
+actions link bind LIBRARIES
+{
+    "$(CONFIG_COMMAND)" $(OPTIONS) -L"$(LINKPATH)" -R"$(RPATH)" -o "$(<)" "$(>)" "$(LIBRARIES)" -Bstatic -l$(FINDLIBS-ST) -Bdynamic -l$(FINDLIBS-SA) -B$(LINK-RUNTIME)
+}
+
+# Slight mods for dlls
+rule link.dll ( targets * : sources * : properties * )
+{
+    SPACE on $(targets) = " " ;
+}
+
+# "$(CONFIG_COMMAND)" $(OPTIONS) -L"$(LINKPATH)" -R"$(RPATH)" -o "$(<)" -h$(<[1]:D=) -G "$(>)" "$(LIBRARIES)" -Bdynamic -l$(FINDLIBS-SA) -Bstatic -l$(FINDLIBS-ST) -B$(LINK-RUNTIME)
+
+actions link.dll bind LIBRARIES
+{
+    "$(CONFIG_COMMAND)" $(OPTIONS) -shared -L"$(LINKPATH)" -R"$(RPATH)" -o "$(<)" "$(>)" -Wl,-h -Wl,$(<[1]:D=) "$(LIBRARIES)" -Bdynamic -l$(FINDLIBS-SA) -Bstatic -l$(FINDLIBS-ST) -B$(LINK-RUNTIME)
+}
+
+actions updated together piecemeal pgi.archive
+{
+    ar -rc$(ARFLAGS:E=) "$(<)" "$(>)"
+}
+

mosesdecoder/jam-files/boost-build/tools/quickbook-config.jam

@@ -0,0 +1,44 @@
+#~ Copyright 2005 Rene Rivera.
+#~ Distributed under the Boost Software License, Version 1.0.
+#~ (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# Automatic configuration for BoostBook tools. To use, just import this module.
+
+import os ;
+import toolset : using ;
+
+if [ os.name ] = NT
+{
+    local boost-dir = ;
+    for local R in snapshot cvs 1.33.0
+    {
+        boost-dir += [ W32_GETREG
+            "HKEY_LOCAL_MACHINE\\SOFTWARE\\Boost.org\\$(R)"
+            : "InstallRoot" ] ;
+    }
+    local quickbook-path = [ GLOB "$(boost-dir)\\bin" "\\Boost\\bin" : quickbook.exe ] ;
+    quickbook-path = $(quickbook-path[1]) ;
+    
+    if $(quickbook-path)
+    {
+        if --debug-configuration in [ modules.peek : ARGV ]
+        {
+            ECHO "notice:" using quickbook ":" $(quickbook-path) ;
+        }
+        using quickbook : $(quickbook-path) ;
+    }
+}
+else
+{
+    local quickbook-path = [ GLOB "/usr/local/bin" "/usr/bin" "/opt/bin" : quickbook ] ;
+    quickbook-path = $(quickbook-path[1]) ;
+    
+    if $(quickbook-path)
+    {
+        if --debug-configuration in [ modules.peek : ARGV ]
+        {
+            ECHO "notice:" using quickbook ":" $(quickbook-path) ;
+        }
+        using quickbook : $(quickbook-path) ;
+    }
+}

mosesdecoder/jam-files/boost-build/tools/quickbook.jam

@@ -0,0 +1,361 @@
+#
+#   Copyright (c) 2005 Jo�o Abecasis
+#   Copyright (c) 2005 Vladimir Prus
+#   Copyright (c) 2006 Rene Rivera
+#
+#   Distributed under the Boost Software License, Version 1.0. (See
+#   accompanying file LICENSE_1_0.txt or copy at
+#   http://www.boost.org/LICENSE_1_0.txt)
+#
+
+# This toolset defines a generator to translate QuickBook to BoostBook. It can
+# be used to generate nice (!) user documentation in different formats
+# (pdf/html/...), from a single text file with simple markup.
+#
+# The toolset defines the QUICKBOOK type (file extension 'qbk') and
+# a QUICKBOOK to XML (BOOSTBOOK) generator.
+#
+#
+#   ===========================================================================
+#   Q & A
+#   ===========================================================================
+#
+#   If you don't know what this is all about, some Q & A will hopefully get you
+#   up to speed with QuickBook and this toolset.
+#
+#
+#   What is QuickBook ?
+#
+#       QuickBook is a WikiWiki style documentation tool geared towards C++
+#       documentation using simple rules and markup for simple formatting tasks.
+#       QuickBook extends the WikiWiki concept. Like the WikiWiki, QuickBook
+#       documents are simple text files. A single QuickBook document can
+#       generate a fully linked set of nice HTML and PostScript/PDF documents
+#       complete with images and syntax-colorized source code.
+#
+#
+#   Where can I get QuickBook ?
+#
+#       Quickbook can be found in Boost's repository, under the tools/quickbook
+#       directory it was added there on Jan 2005, some time after the release of
+#       Boost v1.32.0 and has been an integral part of the Boost distribution
+#       since v1.33.
+#
+#       Here's a link to the SVN repository:
+#           https://svn.boost.org/svn/boost/trunk/tools/quickbook
+#
+#       And to QuickBook's QuickBook-generated docs:
+#           http://www.boost.org/doc/libs/release/tools/quickbook/index.html
+#
+#
+#   How do I use QuickBook and this toolset in my projects ?
+#
+#       The minimal example is:
+#
+#           using boostbook ;
+#           import quickbook ;
+#
+#           boostbook my_docs : my_docs_source.qbk ;
+#
+#       where my_docs is a target name and my_docs_source.qbk is a QuickBook
+#       file. The documentation format to be generated is determined by the
+#       boostbook toolset. By default html documentation should be generated,
+#       but you should check BoostBook's docs to be sure.
+#
+#
+#   What do I need ?
+#
+#       You should start by setting up the BoostBook toolset. Please refer to
+#       boostbook.jam and the BoostBook documentation for information on how to
+#       do this.
+#
+#       A QuickBook executable is also needed. The toolset will generate this
+#       executable if it can find the QuickBook sources. The following
+#       directories will be searched:
+#
+#           BOOST_ROOT/tools/quickbook/
+#           BOOST_BUILD_PATH/../../quickbook/
+#
+#       (BOOST_ROOT and BOOST_BUILD_PATH are environment variables)
+#
+#       If QuickBook sources are not found the toolset will then try to use
+#       the shell command 'quickbook'.
+#
+#
+#   How do I provide a custom QuickBook executable ?
+#
+#       You may put the following in your user-config.jam or site-config.jam:
+#
+#           using quickbook : /path/to/quickbook ;
+#
+#       or, if 'quickbook' can be found in your PATH,
+#
+#           using quickbook : quickbook ;
+#
+#
+#   For convenience three alternatives are tried to get a QuickBook executable:
+#
+#       1.  If the user points us to the a QuickBook executable, that is used.
+#
+#       2.  Otherwise, we search for the QuickBook sources and compile QuickBook
+#           using the default toolset.
+#
+#       3.  As a last resort, we rely on the shell for finding 'quickbook'.
+#
+
+import boostbook ;
+import "class" : new ;
+import feature ;
+import generators ;
+import toolset ;
+import type ;
+import scanner ;
+import project ;
+import targets ;
+import build-system ;
+import path ;
+import common ;
+import errors ;
+
+# The one and only QUICKBOOK type!
+type.register QUICKBOOK : qbk ;
+
+# <quickbook-binary> shell command to run QuickBook
+# <quickbook-binary-dependencies> targets to build QuickBook from sources.
+feature.feature <quickbook-binary> : : free ;
+feature.feature <quickbook-binary-dependencies> : : free dependency ;
+feature.feature <quickbook-define> : : free ;
+feature.feature <quickbook-indent> : : free ;
+feature.feature <quickbook-line-width> : : free ;
+
+
+# quickbook-binary-generator handles generation of the QuickBook executable, by
+# marking it as a dependency for QuickBook docs.
+#
+# If the user supplied the QuickBook command that will be used.
+#
+# Otherwise we search some sensible places for the QuickBook sources and compile
+# from scratch using the default toolset.
+#
+# As a last resort we rely on the shell to find 'quickbook'.
+#
+class quickbook-binary-generator : generator
+{
+    import modules path targets quickbook ;
+
+    rule run ( project name ? : property-set : sources * : multiple ? )
+    {
+        quickbook.freeze-config ;
+        # QuickBook invocation command and dependencies.
+        local quickbook-binary = [ modules.peek quickbook : .quickbook-binary ] ;
+        local quickbook-binary-dependencies ;
+
+        if ! $(quickbook-binary)
+        {
+            # If the QuickBook source directory was found, mark its main target
+            # as a dependency for the current project. Otherwise, try to find
+            # 'quickbook' in user's PATH
+            local quickbook-dir = [ modules.peek quickbook : .quickbook-dir ] ;
+            if $(quickbook-dir)
+            {
+                # Get the main-target in QuickBook directory.
+                local quickbook-main-target = [ targets.resolve-reference $(quickbook-dir) : $(project) ] ;
+
+                # The first element are actual targets, the second are
+                # properties found in target-id. We do not care about these
+                # since we have passed the id ourselves.
+                quickbook-main-target =
+                    [ $(quickbook-main-target[1]).main-target quickbook ] ;
+
+                quickbook-binary-dependencies =
+                    [ $(quickbook-main-target).generate [ $(property-set).propagated ] ] ;
+
+                # Ignore usage-requirements returned as first element.
+                quickbook-binary-dependencies = $(quickbook-binary-dependencies[2-]) ;
+
+                # Some toolsets generate extra targets (e.g. RSP). We must mark
+                # all targets as dependencies for the project, but we will only
+                # use the EXE target for quickbook-to-boostbook translation.
+                for local target in $(quickbook-binary-dependencies)
+                {
+                    if [ $(target).type ] = EXE
+                    {
+                        quickbook-binary = 
+                            [ path.native 
+                                [ path.join
+                                    [ $(target).path ]
+                                    [ $(target).name ]
+                                ]
+                            ] ;
+                    }
+                }
+            }
+        }
+
+        # Add $(quickbook-binary-dependencies) as a dependency of the current
+        # project and set it as the <quickbook-binary> feature for the
+        # quickbook-to-boostbook rule, below.
+        property-set = [ $(property-set).add-raw
+            <dependency>$(quickbook-binary-dependencies)
+            <quickbook-binary>$(quickbook-binary)
+            <quickbook-binary-dependencies>$(quickbook-binary-dependencies)
+        ] ;
+
+        return [ generator.run $(project) $(name) : $(property-set) : $(sources) : $(multiple) ] ;
+    }
+}
+
+
+# Define a scanner for tracking QBK include dependencies.
+#
+class qbk-scanner : common-scanner
+{
+    rule pattern ( )
+    {
+        return "\\[[ ]*include[ ]+([^]]+)\\]" 
+        "\\[[ ]*include:[a-zA-Z0-9_]+[ ]+([^]]+)\\]" 
+        "\\[[ ]*import[ ]+([^]]+)\\]" ;
+    }
+}
+
+
+scanner.register qbk-scanner : include ;
+
+type.set-scanner QUICKBOOK : qbk-scanner ;
+
+
+# Initialization of toolset.
+#
+# Parameters:
+#   command ?    -> path to QuickBook executable.
+#
+# When command is not supplied toolset will search for QuickBook directory and
+# compile the executable from source. If that fails we still search the path for
+# 'quickbook'.
+#
+rule init (
+        command ?   # path to the QuickBook executable.
+    )
+{
+    if $(command)
+    {
+        if $(.config-frozen)
+        {
+            errors.user-error "quickbook: configuration cannot be changed after it has been used." ;
+        }
+        .command = $(command) ;
+    }
+}
+
+rule freeze-config ( )
+{
+    if ! $(.config-frozen)
+    {
+        .config-frozen = true ;
+
+        # QuickBook invocation command and dependencies.
+
+        .quickbook-binary = $(.command) ;
+
+        if $(.quickbook-binary)
+        {
+            # Use user-supplied command.
+            .quickbook-binary = [ common.get-invocation-command quickbook : quickbook : $(.quickbook-binary) ] ;
+        }
+        else
+        {
+            # Search for QuickBook sources in sensible places, like
+            #   $(BOOST_ROOT)/tools/quickbook
+            #   $(BOOST_BUILD_PATH)/../../quickbook
+
+            # And build quickbook executable from sources.
+
+            local boost-root = [ modules.peek : BOOST_ROOT ] ;
+            local boost-build-path = [ build-system.location ] ;
+
+            if $(boost-root)
+            {
+                .quickbook-dir += [ path.join $(boost-root) tools ] ;
+            }
+
+            if $(boost-build-path)
+            {
+                .quickbook-dir += $(boost-build-path)/../.. ;
+            }
+
+            .quickbook-dir = [ path.glob $(.quickbook-dir) : quickbook ] ;
+
+            # If the QuickBook source directory was found, mark its main target
+            # as a dependency for the current project. Otherwise, try to find
+            # 'quickbook' in user's PATH
+            if $(.quickbook-dir)
+            {
+                .quickbook-dir = [ path.make $(.quickbook-dir[1]) ] ;
+            }
+            else
+            {
+                ECHO "QuickBook warning: The path to the quickbook executable was" ;
+                ECHO "  not provided. Additionally, couldn't find QuickBook" ;
+                ECHO "  sources searching in" ;
+                ECHO "    * BOOST_ROOT/tools/quickbook" ;
+                ECHO "    * BOOST_BUILD_PATH/../../quickbook" ;
+                ECHO "  Will now try to find a precompiled executable by searching" ;
+                ECHO "  the PATH for 'quickbook'." ;
+                ECHO "  To disable this warning in the future, or to completely" ;
+                ECHO "  avoid compilation of quickbook, you can explicitly set the" ;
+                ECHO "  path to a quickbook executable command in user-config.jam" ;
+                ECHO "  or site-config.jam with the call" ;
+                ECHO "    using quickbook : /path/to/quickbook ;" ;
+
+                # As a last resort, search for 'quickbook' command in path. Note
+                # that even if the 'quickbook' command is not found,
+                # get-invocation-command will still return 'quickbook' and might
+                # generate an error while generating the virtual-target.
+
+                .quickbook-binary = [ common.get-invocation-command quickbook : quickbook ] ;
+            }
+        }
+    }
+}
+
+
+generators.register [ new quickbook-binary-generator quickbook.quickbook-to-boostbook : QUICKBOOK : XML ] ;
+
+
+# <quickbook-binary> shell command to run QuickBook
+# <quickbook-binary-dependencies> targets to build QuickBook from sources.
+toolset.flags quickbook.quickbook-to-boostbook QB-COMMAND      <quickbook-binary> ;
+toolset.flags quickbook.quickbook-to-boostbook QB-DEPENDENCIES <quickbook-binary-dependencies> ;
+toolset.flags quickbook.quickbook-to-boostbook INCLUDES        <include> ;
+toolset.flags quickbook.quickbook-to-boostbook QB-DEFINES      <quickbook-define> ;
+toolset.flags quickbook.quickbook-to-boostbook QB-INDENT       <quickbook-indent> ;
+toolset.flags quickbook.quickbook-to-boostbook QB-LINE-WIDTH   <quickbook-line-width> ;
+
+
+rule quickbook-to-boostbook ( target : source : properties * )
+{
+    # Signal dependency of quickbook sources on <quickbook-binary-dependencies>
+    # upon invocation of quickbook-to-boostbook.
+    DEPENDS $(target) : [ on $(target) return $(QB-DEPENDENCIES) ] ;
+}
+
+
+actions quickbook-to-boostbook
+{
+    "$(QB-COMMAND)" -I"$(INCLUDES)" -D"$(QB-DEFINES)" --indent="$(QB-INDENT)" --linewidth="$(QB-LINE-WIDTH)" --output-file="$(1)" "$(2)"
+}
+
+
+# Declare a main target to convert a quickbook source into a boostbook XML file.
+#
+rule to-boostbook ( target-name : sources * : requirements * : default-build * )
+{ 
+  local project = [ project.current ] ;
+    
+  targets.main-target-alternative 
+    [ new typed-target $(target-name) : $(project) : XML
+        : [ targets.main-target-sources $(sources) : $(target-name) ] 
+        : [ targets.main-target-requirements $(requirements) : $(project) ]
+        : [ targets.main-target-default-build $(default-build) : $(project) ] 
+    ] ;
+}

mosesdecoder/jam-files/boost-build/tools/stage.jam

@@ -0,0 +1,524 @@
+# Copyright 2003 Dave Abrahams
+# Copyright 2005, 2006 Rene Rivera
+# Copyright 2002, 2003, 2004, 2005, 2006 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# This module defines the 'install' rule, used to copy a set of targets to a
+# single location.
+
+import targets ;
+import "class" : new ;
+import errors ;
+import type ;
+import generators ;
+import feature ;
+import project ;
+import virtual-target ;
+import path ;
+import types/register ;
+
+
+feature.feature <install-dependencies> : off on : incidental      ;
+feature.feature <install-type>         :        : free incidental ;
+feature.feature <install-source-root>  :        : free path       ;
+feature.feature <so-version>           :        : free incidental ;
+
+# If 'on', version symlinks for shared libraries will not be created. Affects
+# Unix builds only.
+feature.feature <install-no-version-symlinks> : on : optional incidental ;
+
+
+class install-target-class : basic-target
+{
+    import feature ;
+    import project ;
+    import type ;
+    import errors ;
+    import generators ;
+    import path ;
+    import stage ;
+    import "class" : new ;
+    import property ;
+    import property-set ;
+
+    rule __init__ ( name-and-dir : project : sources * : requirements * : default-build * )
+    {
+        basic-target.__init__ $(name-and-dir) : $(project) : $(sources) :
+            $(requirements) : $(default-build) ;
+    }
+
+    # If <location> is not set, sets it based on the project data.
+    #
+    rule update-location ( property-set )
+    {
+        local loc = [ $(property-set).get <location> ] ;
+        if ! $(loc)
+        {
+            loc = [ path.root $(self.name) [ $(self.project).get location ] ] ;
+            property-set = [ $(property-set).add-raw $(loc:G=<location>) ] ;
+        }
+
+        return $(property-set) ;
+    }
+
+    # Takes a target that is installed and a property set which is used when
+    # installing.
+    #
+    rule adjust-properties ( target : build-property-set )
+    {
+        local ps-raw ;
+        local a = [ $(target).action ] ;
+        if $(a)
+        {
+            local ps = [ $(a).properties ] ;
+            ps-raw = [ $(ps).raw ] ;
+
+            # Unless <hardcode-dll-paths>true is in properties, which can happen
+            # only if the user has explicitly requested it, nuke all <dll-path>
+            # properties.
+            if [ $(build-property-set).get <hardcode-dll-paths> ] != true
+            {
+                ps-raw = [ property.change $(ps-raw) : <dll-path> ] ;
+            }
+
+            # If any <dll-path> properties were specified for installing, add
+            # them.
+            local l = [ $(build-property-set).get <dll-path> ] ;
+            ps-raw += $(l:G=<dll-path>) ;
+
+            # Also copy <linkflags> feature from current build set, to be used
+            # for relinking.
+            local l = [ $(build-property-set).get <linkflags> ] ;
+            ps-raw += $(l:G=<linkflags>) ;
+
+            # Remove the <tag> feature on original targets.
+            ps-raw = [ property.change $(ps-raw) : <tag> ] ;
+
+            # And <location>. If stage target has another stage target in
+            # sources, then we shall get virtual targets with the <location>
+            # property set.
+            ps-raw = [ property.change $(ps-raw) : <location> ] ;
+        }
+
+        local d = [ $(build-property-set).get <dependency> ] ;
+        ps-raw += $(d:G=<dependency>) ;
+
+        local d = [ $(build-property-set).get <location> ] ;
+        ps-raw += $(d:G=<location>) ;
+
+        local ns = [ $(build-property-set).get <install-no-version-symlinks> ] ;
+        ps-raw += $(ns:G=<install-no-version-symlinks>) ;
+
+        local d = [ $(build-property-set).get <install-source-root> ] ;
+        # Make the path absolute: we shall use it to compute relative paths and
+        # making the path absolute will help.
+        if $(d)
+        {
+            d = [ path.root $(d) [ path.pwd ] ] ;
+            ps-raw += $(d:G=<install-source-root>) ;
+        }
+
+        if $(ps-raw)
+        {
+            return [ property-set.create $(ps-raw) ]  ;
+        }
+        else
+        {
+            return [ property-set.empty ] ;
+        }
+    }
+
+    rule construct ( name : source-targets * : property-set )
+    {
+        source-targets = [ targets-to-stage $(source-targets) :
+            $(property-set) ] ;
+
+        property-set = [ update-location $(property-set) ] ;
+
+        local ename = [ $(property-set).get <name> ] ;
+
+        if $(ename) && $(source-targets[2])
+        {
+            errors.error "When <name> property is used in 'install', only one"
+                "source is allowed" ;
+        }
+
+        local result ;
+        for local i in $(source-targets)
+        {
+            local staged-targets ;
+
+            local new-properties = [ adjust-properties $(i) :
+                $(property-set) ] ;
+
+            # See if something special should be done when staging this type. It
+            # is indicated by the presence of a special "INSTALLED_" type.
+            local t = [ $(i).type ] ;
+            if $(t) && [ type.registered INSTALLED_$(t) ]
+            {
+                if $(ename)
+                {
+                    errors.error "In 'install': <name> property specified with target that requires relinking." ;
+                }
+                else
+                {
+                    local targets = [ generators.construct $(self.project)
+                        $(name) : INSTALLED_$(t) : $(new-properties) : $(i) ] ;
+                    staged-targets += $(targets[2-]) ;
+                }
+            }
+            else
+            {
+                staged-targets = [ stage.copy-file $(self.project) $(ename) :
+                    $(i) : $(new-properties) ] ;
+            }
+
+            if ! $(staged-targets)
+            {
+                errors.error "Unable to generate staged version of " [ $(source).str ] ;
+            }
+
+            for t in $(staged-targets)
+            {
+                result += [ virtual-target.register $(t) ] ;
+            }
+        }
+
+        return [ property-set.empty ] $(result) ;
+    }
+
+    # Given the list of source targets explicitly passed to 'stage', returns the
+    # list of targets which must be staged.
+    #
+    rule targets-to-stage ( source-targets * : property-set )
+    {
+        local result ;
+
+        # Traverse the dependencies, if needed.
+        if [ $(property-set).get <install-dependencies> ] = "on"
+        {
+            source-targets = [ collect-targets $(source-targets) ] ;
+        }
+
+        # Filter the target types, if needed.
+        local included-types = [ $(property-set).get <install-type> ] ;
+        for local r in $(source-targets)
+        {
+            local ty = [ $(r).type ] ;
+            if $(ty)
+            {
+                # Do not stage searched libs.
+                if $(ty) != SEARCHED_LIB
+                {
+                    if $(included-types)
+                    {
+                        if [ include-type $(ty) : $(included-types) ]
+                        {
+                            result += $(r) ;
+                        }
+                    }
+                    else
+                    {
+                        result += $(r) ;
+                    }
+                }
+            }
+            else if ! $(included-types)
+            {
+                # Don't install typeless target if there is an explicit list of
+                # allowed types.
+                result += $(r) ;
+            }
+        }
+
+        return $(result) ;
+    }
+
+    # CONSIDER: figure out why we can not use virtual-target.traverse here.
+    #
+    rule collect-targets ( targets * )
+    {
+        # Find subvariants
+        local s ;
+        for local t in $(targets)
+        {
+            s += [ $(t).creating-subvariant ] ;
+        }
+        s = [ sequence.unique $(s) ] ;
+        
+        local result = [ new set ] ;
+        $(result).add $(targets) ;
+        
+        for local i in $(s)
+        {
+            $(i).all-referenced-targets $(result) ;
+        }
+        local result2 ;
+        for local r in [ $(result).list ]
+        {
+            if $(r:G) != <use>
+            {
+                result2 += $(r:G=) ;
+            }
+        }
+        DELETE_MODULE $(result) ;
+        return [ sequence.unique $(result2) ] ;
+    }
+
+    # Returns true iff 'type' is subtype of some element of 'types-to-include'.
+    #
+    local rule include-type ( type : types-to-include * )
+    {
+        local found ;
+        while $(types-to-include) && ! $(found)
+        {
+            if [ type.is-subtype $(type) $(types-to-include[1]) ]
+            {
+                found = true ;
+            }
+            types-to-include = $(types-to-include[2-]) ;
+        }
+
+        return $(found) ;
+    }
+}
+
+
+# Creates a copy of target 'source'. The 'properties' object should have a
+# <location> property which specifies where the target must be placed.
+#
+rule copy-file ( project name ? : source : properties )
+{
+    name ?= [ $(source).name ] ;
+    local relative ;
+
+    local new-a = [ new non-scanning-action $(source) : common.copy :
+        $(properties) ] ;
+    local source-root = [ $(properties).get <install-source-root> ] ;
+    if $(source-root)
+    {
+        # Get the real path of the target. We probably need to strip relative
+        # path from the target name at construction.
+        local path = [ $(source).path ] ;
+        path = [ path.root $(name:D) $(path) ] ;
+        # Make the path absolute. Otherwise, it would be hard to compute the
+        # relative path. The 'source-root' is already absolute, see the
+        # 'adjust-properties' method above.
+        path = [ path.root $(path) [ path.pwd ] ] ;
+
+        relative = [ path.relative-to $(source-root) $(path) ] ;
+    }
+
+    # Note: Using $(name:D=$(relative)) might be faster here, but then we would
+    # need to explicitly check that relative is not ".", otherwise we might get
+    # paths like '<prefix>/boost/.', try to create it and mkdir would obviously
+    # fail.
+    name = [ path.join $(relative) $(name:D=) ] ;
+
+    return [ new file-target $(name) exact : [ $(source).type ] : $(project) :
+        $(new-a) ] ;
+}
+
+
+rule symlink ( name : project : source : properties )
+{
+    local a = [ new action $(source) : symlink.ln : $(properties) ] ;
+    return [ new file-target $(name) exact : [ $(source).type ] : $(project) :
+        $(a) ] ;
+}
+
+
+rule relink-file ( project : source : property-set  )
+{
+    local action = [ $(source).action ] ;
+    local cloned-action = [ virtual-target.clone-action $(action) : $(project) :
+        "" : $(property-set) ] ;
+    return [ $(cloned-action).targets ] ;
+}
+
+
+# Declare installed version of the EXE type. Generator for this type will cause
+# relinking to the new location.
+type.register INSTALLED_EXE : : EXE ;
+
+
+class installed-exe-generator : generator
+{
+    import type ;
+    import property-set ;
+    import modules ;
+    import stage ;
+
+    rule __init__ ( )
+    {
+        generator.__init__ install-exe : EXE : INSTALLED_EXE ;
+    }
+
+    rule run ( project name ? : property-set : source : multiple ? )
+    {
+        local need-relink ;
+        
+        if [ $(property-set).get <os> ] in NT CYGWIN ||
+            [ $(property-set).get <target-os> ] in windows cygwin
+        {
+        }
+        else
+        {
+            # See if the dll-path properties are not changed during
+            # install. If so, copy, don't relink.
+            local a = [ $(source).action ] ;
+            local p = [ $(a).properties ] ;
+            local original = [ $(p).get <dll-path> ] ;
+            local current = [ $(property-set).get <dll-path> ] ;
+            
+            if $(current) != $(original)
+            {
+                need-relink = true ;
+            }            
+        }
+        
+            
+        if $(need-relink)
+        {
+            return [ stage.relink-file $(project)
+              : $(source) : $(property-set) ] ;
+        }
+        else
+        {
+            return [ stage.copy-file $(project)
+              : $(source) : $(property-set) ] ;
+        }
+    }
+}
+
+
+generators.register [ new installed-exe-generator ] ;
+
+
+# Installing a shared link on Unix might cause a creation of versioned symbolic
+# links.
+type.register INSTALLED_SHARED_LIB : : SHARED_LIB ;
+
+
+class installed-shared-lib-generator : generator
+{
+    import type ;
+    import property-set ;
+    import modules ;
+    import stage ;
+
+    rule __init__ ( )
+    {
+        generator.__init__ install-shared-lib : SHARED_LIB
+          : INSTALLED_SHARED_LIB ;
+    }
+
+    rule run ( project name ? : property-set : source : multiple ? )
+    {
+        if [ $(property-set).get <os> ] in NT CYGWIN ||
+            [ $(property-set).get <target-os> ] in windows cygwin
+        {
+            local copied = [ stage.copy-file $(project) : $(source) :
+                $(property-set) ] ;
+            return [ virtual-target.register $(copied) ] ;
+        }
+        else
+        {
+            local a = [ $(source).action ] ;
+            local copied ;
+            if ! $(a)
+            {
+                # Non-derived file, just copy.
+                copied = [ stage.copy-file $(project) : $(source) :
+                    $(property-set) ] ;
+            }
+            else
+            {
+                local cp = [ $(a).properties ] ;
+                local current-dll-path = [ $(cp).get <dll-path> ] ;
+                local new-dll-path = [ $(property-set).get <dll-path> ] ;
+
+                if $(current-dll-path) != $(new-dll-path)
+                {
+                    # Rpath changed, need to relink.
+                    copied = [ stage.relink-file $(project) : $(source) :
+                        $(property-set) ] ;
+                }
+                else
+                {
+                    copied = [ stage.copy-file $(project) : $(source) :
+                        $(property-set) ] ;
+                }
+            }
+
+            copied = [ virtual-target.register $(copied) ] ;
+
+            local result = $(copied) ;
+            # If the name is in the form NNN.XXX.YYY.ZZZ, where all 'X', 'Y' and
+            # 'Z' are numbers, we need to create NNN.XXX and NNN.XXX.YYY
+            # symbolic links.
+            local m = [ MATCH (.*)\\.([0123456789]+)\\.([0123456789]+)\\.([0123456789]+)$
+                : [ $(copied).name ] ] ;
+            if $(m)
+            {
+                # Symlink without version at all is used to make
+                # -lsome_library work.
+                result += [ stage.symlink $(m[1]) : $(project) : $(copied) :
+                    $(property-set) ] ;
+
+                # Symlinks of some libfoo.N and libfoo.N.M are used so that
+                # library can found at runtime, if libfoo.N.M.X has soname of
+                # libfoo.N. That happens when the library makes some binary
+                # compatibility guarantees. If not, it is possible to skip those
+                # symlinks.
+                local suppress =
+                    [ $(property-set).get <install-no-version-symlinks> ] ;
+
+                if $(suppress) != "on"
+                {
+                    result += [ stage.symlink $(m[1]).$(m[2]) : $(project)
+                      : $(copied) : $(property-set) ] ;
+                    result += [ stage.symlink $(m[1]).$(m[2]).$(m[3])  : $(project)
+                      : $(copied) : $(property-set) ] ;
+                }
+            }
+
+            return $(result) ;
+        }
+    }
+}
+
+generators.register [ new installed-shared-lib-generator ] ;
+
+
+# Main target rule for 'install'.
+#
+rule install ( name : sources * : requirements * : default-build * )
+{
+    local project = [ project.current ] ;
+
+    # Unless the user has explicitly asked us to hardcode dll paths, add
+    # <hardcode-dll-paths>false in requirements, to override default value.
+    if ! <hardcode-dll-paths>true in $(requirements)
+    {
+        requirements += <hardcode-dll-paths>false ;
+    }
+
+    if <tag> in $(requirements:G)
+    {
+        errors.user-error
+            "The <tag> property is not allowed for the 'install' rule" ;
+    }
+
+    targets.main-target-alternative
+      [ new install-target-class $(name) : $(project)
+        : [ targets.main-target-sources $(sources) : $(name) ]
+        : [ targets.main-target-requirements $(requirements) : $(project) ]
+        : [ targets.main-target-default-build $(default-build) : $(project) ]
+      ] ;
+}
+
+
+IMPORT $(__name__) : install : : install ;
+IMPORT $(__name__) : install : : stage ;

mosesdecoder/jam-files/boost-build/tools/testing-aux.jam

@@ -0,0 +1,210 @@
+# This module is imported by testing.py. The definitions here are
+# too tricky to do in Python
+
+# Causes the 'target' to exist after bjam invocation if and only if all the
+# dependencies were successfully built.
+#
+rule expect-success ( target : dependency + : requirements * )
+{
+    **passed** $(target) : $(sources) ;
+}
+IMPORT testing : expect-success : : testing.expect-success ;
+
+# Causes the 'target' to exist after bjam invocation if and only if all some of
+# the dependencies were not successfully built.
+#
+rule expect-failure ( target : dependency + : properties * )
+{
+    local grist = [ MATCH ^<(.*)> : $(dependency:G) ] ;
+    local marker = $(dependency:G=$(grist)*fail) ;
+    (failed-as-expected) $(marker) ;
+    FAIL_EXPECTED $(dependency) ;
+    LOCATE on $(marker) = [ on $(dependency) return $(LOCATE) ] ;
+    RMOLD $(marker) ;
+    DEPENDS $(marker) : $(dependency) ;
+    DEPENDS $(target) : $(marker) ;
+    **passed** $(target) : $(marker) ;
+}
+IMPORT testing : expect-failure : : testing.expect-failure ;
+
+# The rule/action combination used to report successful passing of a test.
+#
+rule **passed**
+{
+    # Force deletion of the target, in case any dependencies failed to build.
+    RMOLD $(<) ;
+}
+
+
+# Used to create test files signifying passed tests.
+#
+actions **passed**
+{
+    echo passed > "$(<)"
+}
+
+
+# Used to create replacement object files that do not get created during tests
+# that are expected to fail.
+#
+actions (failed-as-expected)
+{
+    echo failed as expected > "$(<)"
+}
+
+# Runs executable 'sources' and stores stdout in file 'target'. Unless
+# --preserve-test-targets command line option has been specified, removes the
+# executable. The 'target-to-remove' parameter controls what should be removed:
+#   - if 'none', does not remove anything, ever
+#   - if empty, removes 'source'
+#   - if non-empty and not 'none', contains a list of sources to remove.
+#
+rule capture-output ( target : source : properties * : targets-to-remove * )
+{
+    output-file on $(target) = $(target:S=.output) ;
+    LOCATE on $(target:S=.output) = [ on $(target) return $(LOCATE) ] ;
+
+    # The INCLUDES kill a warning about independent target...
+    INCLUDES $(target) : $(target:S=.output) ;
+    # but it also puts .output into dependency graph, so we must tell jam it is
+    # OK if it cannot find the target or updating rule.
+    NOCARE $(target:S=.output) ;
+
+    # This has two-fold effect. First it adds input files to the dependendency
+    # graph, preventing a warning. Second, it causes input files to be bound
+    # before target is created. Therefore, they are bound using SEARCH setting
+    # on them and not LOCATE setting of $(target), as in other case (due to jam
+    # bug).
+    DEPENDS $(target) : [ on $(target) return $(INPUT_FILES) ] ;
+
+    if $(targets-to-remove) = none
+    {
+        targets-to-remove = ;
+    }
+    else if ! $(targets-to-remove)
+    {
+        targets-to-remove = $(source) ;
+    }
+
+    if [ on $(target) return $(REMOVE_TEST_TARGETS) ]
+    {
+        TEMPORARY $(targets-to-remove) ;
+        # Set a second action on target that will be executed after capture
+        # output action. The 'RmTemps' rule has the 'ignore' modifier so it is
+        # always considered succeeded. This is needed for 'run-fail' test. For
+        # that test the target will be marked with FAIL_EXPECTED, and without
+        # 'ignore' successful execution will be negated and be reported as
+        # failure. With 'ignore' we do not detect a case where removing files
+        # fails, but it is not likely to happen.
+        RmTemps $(target) : $(targets-to-remove) ;
+    }
+}
+
+
+if [ os.name ] = NT
+{
+    .STATUS        = %status% ;
+    .SET_STATUS    = "set status=%ERRORLEVEL%" ;
+    .RUN_OUTPUT_NL = "echo." ;
+    .STATUS_0      = "%status% EQU 0 (" ;
+    .STATUS_NOT_0  = "%status% NEQ 0 (" ;
+    .VERBOSE       = "%verbose% EQU 1 (" ;
+    .ENDIF         = ")" ;
+    .SHELL_SET     = "set " ;
+    .CATENATE      = type ;
+    .CP            = copy ;
+}
+else
+{
+    .STATUS        = "$status" ;
+    .SET_STATUS    = "status=$?" ;
+    .RUN_OUTPUT_NL = "echo" ;
+    .STATUS_0      = "test $status -eq 0 ; then" ;
+    .STATUS_NOT_0  = "test $status -ne 0 ; then" ;
+    .VERBOSE       = "test $verbose -eq 1 ; then" ;
+    .ENDIF         = "fi" ;
+    .SHELL_SET     = "" ;
+    .CATENATE      = cat ;
+    .CP            = cp ;
+}
+
+
+.VERBOSE_TEST = 0 ;
+if --verbose-test in [ modules.peek : ARGV ]
+{
+    .VERBOSE_TEST = 1 ;
+}
+
+
+.RM = [ common.rm-command ] ;
+
+
+actions capture-output bind INPUT_FILES output-file
+{
+    $(PATH_SETUP)
+    $(LAUNCHER) "$(>)" $(ARGS) "$(INPUT_FILES)" > "$(output-file)" 2>&1
+    $(.SET_STATUS)
+    $(.RUN_OUTPUT_NL) >> "$(output-file)"
+    echo EXIT STATUS: $(.STATUS) >> "$(output-file)"
+    if $(.STATUS_0)
+        $(.CP) "$(output-file)" "$(<)"
+    $(.ENDIF)
+    $(.SHELL_SET)verbose=$(.VERBOSE_TEST)
+    if $(.STATUS_NOT_0)
+        $(.SHELL_SET)verbose=1
+    $(.ENDIF)
+    if $(.VERBOSE)
+        echo ====== BEGIN OUTPUT ======
+        $(.CATENATE) "$(output-file)"
+        echo ====== END OUTPUT ======
+    $(.ENDIF)
+    exit $(.STATUS)
+}
+
+IMPORT testing : capture-output : : testing.capture-output ;
+
+
+actions quietly updated ignore piecemeal together RmTemps
+{
+    $(.RM) "$(>)"
+}
+
+
+.MAKE_FILE = [ common.file-creation-command ] ;
+
+actions unit-test
+{
+    $(PATH_SETUP)
+    $(LAUNCHER) $(>) $(ARGS) && $(.MAKE_FILE) $(<)
+}
+
+rule record-time ( target : source : start end user system )
+{
+    local src-string = [$(source:G=:J=",")"] " ;
+    USER_TIME on $(target) += $(src-string)$(user) ;
+    SYSTEM_TIME on $(target) += $(src-string)$(system) ;
+}
+
+# Calling this rule requests that Boost Build time how long it taks to build the
+# 'source' target and display the results both on the standard output and in the
+# 'target' file.
+#
+rule time ( target : source : properties *  )
+{
+    # Set up rule for recording timing information.
+    __TIMING_RULE__ on $(source) = testing.record-time $(target) ;
+
+    # Make sure that the source is rebuilt any time we need to retrieve that
+    # information.
+    REBUILDS $(target) : $(source) ;
+}
+
+
+actions time
+{
+    echo user: $(USER_TIME)
+    echo system: $(SYSTEM_TIME)
+
+    echo user: $(USER_TIME)" seconds" > "$(<)"
+    echo system: $(SYSTEM_TIME)" seconds" >> "$(<)"
+}

mosesdecoder/jam-files/boost-build/tools/vacpp.jam

@@ -0,0 +1,150 @@
+# Copyright Vladimir Prus 2004.
+# Copyright Toon Knapen 2004.
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt
+# or copy at http://www.boost.org/LICENSE_1_0.txt)
+
+#
+# Boost.Build V2 toolset for the IBM XL C++ compiler
+#
+
+import toolset : flags ;
+import feature ;
+import common ;
+import generators ;
+import os ;
+
+feature.extend toolset : vacpp ;
+toolset.inherit vacpp : unix ;
+generators.override vacpp.prebuilt : builtin.prebuilt ;
+generators.override vacpp.searched-lib-generator : searched-lib-generator ;
+
+# Configure the vacpp toolset
+rule init ( version ? : command * : options * )
+{
+    local condition = [ 
+        common.check-init-parameters vacpp : version $(version) ] ;
+              
+    command = [ common.get-invocation-command vacpp : xlC
+        : $(command) : "/usr/vacpp/bin/xlC" ] ;
+        
+    common.handle-options vacpp : $(condition) : $(command) : $(options) ;
+}
+
+# Declare generators
+generators.register-c-compiler vacpp.compile.c : C : OBJ : <toolset>vacpp ;
+generators.register-c-compiler vacpp.compile.c++ : CPP : OBJ : <toolset>vacpp ;
+
+# Allow C++ style comments in C files
+flags vacpp CFLAGS : -qcpluscmt ;
+
+# Declare flags
+flags vacpp CFLAGS <optimization>off : -qNOOPTimize ;
+flags vacpp CFLAGS <optimization>speed : -O3 -qstrict ;
+flags vacpp CFLAGS <optimization>space : -O2 -qcompact ;
+
+# Discretionary inlining (not recommended)
+flags vacpp CFLAGS <inlining>off : -qnoinline ;
+flags vacpp CFLAGS <inlining>on : -qinline ;
+#flags vacpp CFLAGS <inlining>full : -qinline ;
+flags vacpp CFLAGS <inlining>full : ;
+
+# Exception handling
+flags vacpp C++FLAGS <exception-handling>off : -qnoeh ;
+flags vacpp C++FLAGS <exception-handling>on : -qeh ;
+
+# Run-time Type Identification
+flags vacpp C++FLAGS <rtti>off : -qnortti ;
+flags vacpp C++FLAGS <rtti>on : -qrtti ;
+
+# Enable 64-bit memory addressing model
+flags vacpp CFLAGS <address-model>64 : -q64 ;
+flags vacpp LINKFLAGS <address-model>64 : -q64 ;
+flags vacpp ARFLAGS <target-os>aix/<address-model>64 : -X 64 ;
+
+# Use absolute path when generating debug information
+flags vacpp CFLAGS <debug-symbols>on : -g -qfullpath ;
+flags vacpp LINKFLAGS <debug-symbols>on : -g -qfullpath ;
+flags vacpp LINKFLAGS <debug-symbols>off : -s ;
+
+if [ os.name ] = AIX
+{
+    flags vacpp.compile C++FLAGS : -qfuncsect ;
+
+    # The -bnoipath strips the prepending (relative) path of libraries from
+    # the loader section in the target library or executable. Hence, during
+    # load-time LIBPATH (identical to LD_LIBRARY_PATH) or a hard-coded
+    # -blibpath (*similar* to -lrpath/-lrpath-link) is searched. Without
+    # this option, the prepending (relative) path + library name is
+    # hard-coded in the loader section, causing *only* this path to be
+    # searched during load-time. Note that the AIX linker does not have an
+    # -soname equivalent, this is as close as it gets.
+    #
+    # The above options are definately for AIX 5.x, and most likely also for
+    # AIX 4.x and AIX 6.x. For details about the AIX linker see:
+    # http://download.boulder.ibm.com/ibmdl/pub/software/dw/aix/es-aix_ll.pdf
+    #
+    flags vacpp.link LINKFLAGS <link>shared : -bnoipath ;
+
+    # Run-time linking
+    flags vacpp.link EXE-LINKFLAGS <link>shared : -brtl ;
+}
+else
+{
+    # Linux PPC
+    flags vacpp.compile CFLAGS <link>shared : -qpic=large ;
+    flags vacpp FINDLIBS : rt ;
+}
+
+# Profiling
+flags vacpp CFLAGS <profiling>on : -pg ;
+flags vacpp LINKFLAGS <profiling>on : -pg ;
+
+flags vacpp.compile OPTIONS <cflags> ;
+flags vacpp.compile.c++ OPTIONS <cxxflags> ;
+flags vacpp DEFINES <define> ;
+flags vacpp UNDEFS <undef> ;
+flags vacpp HDRS <include> ;
+flags vacpp STDHDRS <sysinclude> ;
+flags vacpp.link OPTIONS <linkflags> ;
+flags vacpp ARFLAGS <arflags> ;
+
+flags vacpp LIBPATH <library-path> ;
+flags vacpp NEEDLIBS <library-file> ;
+flags vacpp FINDLIBS <find-shared-library> ;
+flags vacpp FINDLIBS <find-static-library> ;
+
+# Select the compiler name according to the threading model.
+flags vacpp VA_C_COMPILER  <threading>single : xlc   ;
+flags vacpp VA_C_COMPILER  <threading>multi : xlc_r ;
+flags vacpp VA_CXX_COMPILER <threading>single : xlC   ;
+flags vacpp VA_CXX_COMPILER <threading>multi : xlC_r ;
+
+SPACE = " " ;
+
+flags vacpp.link.dll HAVE_SONAME <target-os>linux : "" ;
+
+actions vacpp.link bind NEEDLIBS
+{
+    $(VA_CXX_COMPILER) $(EXE-LINKFLAGS) $(LINKFLAGS) -o "$(<[1])" -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS) $(USER_OPTIONS)
+}
+
+actions vacpp.link.dll bind NEEDLIBS
+{
+    xlC_r -G $(LINKFLAGS) -o "$(<[1])" $(HAVE_SONAME)-Wl,-soname$(SPACE)-Wl,$(<[-1]:D=) -L$(LIBPATH) -L$(STDLIBPATH) "$(>)" "$(NEEDLIBS)" "$(NEEDLIBS)" -l$(FINDLIBS) $(OPTIONS) $(USER_OPTIONS) 
+}
+
+actions vacpp.compile.c
+{
+    $(VA_C_COMPILER) -c $(OPTIONS) $(USER_OPTIONS) -I$(BOOST_ROOT) -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) -I"$(HDRS)" -I"$(STDHDRS)" -o "$(<)" "$(>)"
+}
+
+actions vacpp.compile.c++
+{
+    $(VA_CXX_COMPILER) -c $(OPTIONS) $(USER_OPTIONS) -I$(BOOST_ROOT) -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) $(C++FLAGS) -I"$(HDRS)" -I"$(STDHDRS)" -o "$(<)" "$(>)"
+}
+
+actions updated together piecemeal vacpp.archive
+{
+    ar $(ARFLAGS) ru "$(<)" "$(>)"
+}

mosesdecoder/jam-files/boost-build/util/container.jam

@@ -0,0 +1,339 @@
+# Copyright 2003 Dave Abrahams
+# Copyright 2002, 2003 Rene Rivera
+# Copyright 2002, 2003, 2004 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+# Various container classes.
+
+# Base for container objects. This lets us construct recursive structures. That
+# is containers with containers in them, specifically so we can tell literal
+# values from node values.
+#
+class node
+{
+    rule __init__ (
+        value ?  # Optional value to set node to initially.
+    )
+    {
+        self.value = $(value) ;
+    }
+
+    # Set the value of this node, passing nothing will clear it.
+    #
+    rule set ( value * )
+    {
+        self.value = $(value) ;
+    }
+
+    # Get the value of this node.
+    #
+    rule get ( )
+    {
+        return $(self.value) ;
+    }
+}
+
+
+# A simple vector. Interface mimics the C++ std::vector and std::list, with the
+# exception that indices are one (1) based to follow Jam standard.
+#
+# TODO: Possibly add assertion checks.
+#
+class vector : node
+{
+    import numbers ;
+    import utility ;
+    import sequence ;
+
+    rule __init__ (
+        values *  # Initial contents of vector.
+    )
+    {
+        node.__init__ ;
+        self.value = $(values) ;
+    }
+
+    # Get the value of the first element.
+    #
+    rule front ( )
+    {
+        return $(self.value[1]) ;
+    }
+
+    # Get the value of the last element.
+    #
+    rule back ( )
+    {
+        return $(self.value[-1]) ;
+    }
+
+    # Get the value of the element at the given index, one based. Access to
+    # elements of recursive structures is supported directly. Specifying
+    # additional index values recursively accesses the elements as containers.
+    # For example: [ $(v).at 1 : 2 ] would retrieve the second element of our
+    # first element, assuming the first element is a container.
+    #
+    rule at (
+        index  # The element index, one based.
+        : *  # Additional indices to access recursively.
+    )
+    {
+        local r = $(self.value[$(index)]) ;
+        if $(2)
+        {
+            r = [ $(r).at $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ] ;
+        }
+        return $(r) ;
+    }
+
+    # Get the value contained in the given element. This has the same
+    # functionality and interface as "at" but in addition gets the value of the
+    # referenced element, assuming it is a "node".
+    #
+    rule get-at (
+        index  # The element index, one based.
+        : *  # Additional indices to access recursively.
+    )
+    {
+        local r = $(self.value[$(index)]) ;
+        if $(2)
+        {
+            r = [ $(r).at $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ] ;
+        }
+        return [ $(r).get ] ;
+    }
+
+    # Insert the given value into the front of the vector pushing the rest of
+    # the elements back.
+    #
+    rule push-front (
+        value  # Value to become first element.
+    )
+    {
+        self.value = $(value) $(self.value) ;
+    }
+
+    # Remove the front element from the vector. Does not return the value. No
+    # effect if vector is empty.
+    #
+    rule pop-front ( )
+    {
+        self.value = $(self.value[2-]) ;
+    }
+
+    # Add the given value at the end of the vector.
+    #
+    rule push-back (
+        value  # Value to become back element.
+    )
+    {
+        self.value += $(value) ;
+    }
+
+    # Remove the back element from the vector. Does not return the value. No
+    # effect if vector is empty.
+    #
+    rule pop-back ( )
+    {
+        self.value = $(self.value[1--2]) ;
+    }
+
+    # Insert the given value at the given index, one based. The values at and to
+    # the right of the index are pushed back to make room for the new value.
+    # If the index is passed the end of the vector the element is added to the
+    # end.
+    #
+    rule insert (
+        index  # The index to insert at, one based.
+        : value  # The value to insert.
+    )
+    {
+        local left = $(self.value[1-$(index)]) ;
+        local right = $(self.value[$(index)-]) ;
+        if $(right)-is-not-empty
+        {
+            left = $(left[1--2]) ;
+        }
+        self.value = $(left) $(value) $(right) ;
+    }
+
+    # Remove one or more elements from the vector. The range is inclusive, and
+    # not specifying an end is equivalent to the [start, start] range.
+    #
+    rule erase (
+        start  # Index of first element to remove.
+        end ?  # Optional, index of last element to remove.
+    )
+    {
+        end ?= $(start) ;
+        local left = $(self.value[1-$(start)]) ;
+        left = $(left[1--2]) ;
+        local right = $(self.value[$(end)-]) ;
+        right = $(right[2-]) ;
+        self.value = $(left) $(right) ;
+    }
+
+    # Remove all elements from the vector.
+    #
+    rule clear ( )
+    {
+        self.value = ;
+    }
+
+    # The number of elements in the vector.
+    #
+    rule size ( )
+    {
+        return [ sequence.length $(self.value) ] ;
+    }
+
+    # Returns "true" if there are NO elements in the vector, empty otherwise.
+    #
+    rule empty ( )
+    {
+        if ! $(self.value)-is-not-empty
+        {
+            return true ;
+        }
+    }
+
+    # Returns the textual representation of content.
+    #
+    rule str ( )
+    {
+        return "[" [ sequence.transform utility.str : $(self.value) ] "]" ;
+    }
+
+    # Sorts the vector inplace, calling 'utility.less' for comparisons.
+    #
+    rule sort ( )
+    {
+        self.value = [ sequence.insertion-sort $(self.value) : utility.less ] ;
+    }
+
+    # Returns true if content is equal to the content of other vector. Uses
+    # 'utility.equal' for comparison.
+    #
+    rule equal ( another )
+    {
+        local mismatch ;
+        local size = [ size ] ;
+        if $(size) = [ $(another).size ]
+        {
+            for local i in [ numbers.range 1 $(size) ]
+            {
+                if ! [ utility.equal [ at $(i) ] [ $(another).at $(i) ] ]
+                {
+                    mismatch = true ;
+                }
+            }
+        }
+        else
+        {
+            mismatch = true ;
+        }
+
+        if ! $(mismatch)
+        {
+            return true ;
+        }
+    }
+}
+
+
+rule __test__ ( )
+{
+    import assert ;
+    import "class" : new ;
+
+    local v1 = [ new vector ] ;
+    assert.true $(v1).equal $(v1) ;
+    assert.true $(v1).empty ;
+    assert.result 0 : $(v1).size ;
+    assert.result "[" "]" : $(v1).str ;
+    $(v1).push-back b ;
+    $(v1).push-front a ;
+    assert.result "[" a b "]" : $(v1).str ;
+    assert.result a : $(v1).front ;
+    assert.result b : $(v1).back ;
+    $(v1).insert 2 : d ;
+    $(v1).insert 2 : c ;
+    $(v1).insert 4 : f ;
+    $(v1).insert 4 : e ;
+    $(v1).pop-back ;
+    assert.result 5 : $(v1).size ;
+    assert.result d : $(v1).at 3 ;
+    $(v1).pop-front ;
+    assert.result c : $(v1).front ;
+    assert.false $(v1).empty ;
+    $(v1).erase 3 4 ;
+    assert.result 2 : $(v1).size ;
+
+    local v2 = [ new vector q w e r t y ] ;
+    assert.result 6 : $(v2).size ;
+    $(v1).push-back $(v2) ;
+    assert.result 3 : $(v1).size ;
+    local v2-alias = [ $(v1).back ] ;
+    assert.result e : $(v2-alias).at 3 ;
+    $(v1).clear ;
+    assert.true $(v1).empty ;
+    assert.false $(v2-alias).empty ;
+    $(v2).pop-back ;
+    assert.result t : $(v2-alias).back ;
+
+    local v3 = [ new vector ] ;
+    $(v3).push-back [ new vector 1 2 3 4 5 ] ;
+    $(v3).push-back [ new vector a b c ] ;
+    assert.result "[" "[" 1 2 3 4 5 "]" "[" a b c "]" "]" : $(v3).str ;
+    $(v3).push-back [ new vector [ new vector x y z ] [ new vector 7 8 9 ] ] ;
+    assert.result 1 : $(v3).at 1 : 1 ;
+    assert.result b : $(v3).at 2 : 2 ;
+    assert.result a b c : $(v3).get-at 2 ;
+    assert.result 7 8 9 : $(v3).get-at 3 : 2 ;
+
+    local v4 = [ new vector 4 3 6 ] ;
+    $(v4).sort ;
+    assert.result 3 4 6 : $(v4).get ;
+    assert.false $(v4).equal $(v3) ;
+
+    local v5 = [ new vector 3 4 6 ] ;
+    assert.true $(v4).equal $(v5) ;
+    # Check that vectors of different sizes are considered non-equal.
+    $(v5).pop-back ;
+    assert.false $(v4).equal $(v5) ;
+
+    local v6 = [ new vector [ new vector 1 2 3 ] ] ;
+    assert.true $(v6).equal [ new vector [ new vector 1 2 3 ] ] ;
+
+    local v7 = [ new vector 111 222 333 ] ;
+    assert.true $(v7).equal $(v7) ;
+    $(v7).insert 4 : 444 ;
+    assert.result 111 222 333 444 : $(v7).get ;
+    $(v7).insert 999 : xxx ;
+    assert.result 111 222 333 444 xxx : $(v7).get ;
+
+    local v8 = [ new vector "" "" "" ] ;
+    assert.true $(v8).equal $(v8) ;
+    assert.false $(v8).empty ;
+    assert.result 3  : $(v8).size ;
+    assert.result "" : $(v8).at 1 ;
+    assert.result "" : $(v8).at 2 ;
+    assert.result "" : $(v8).at 3 ;
+    assert.result    : $(v8).at 4 ;
+    $(v8).insert 2 : 222 ;
+    assert.result 4 : $(v8).size ;
+    assert.result "" 222 "" "" : $(v8).get ;
+    $(v8).insert 999 : "" ;
+    assert.result 5 : $(v8).size ;
+    assert.result "" 222 "" "" "" : $(v8).get ;
+    $(v8).insert 999 : xxx ;
+    assert.result 6 : $(v8).size ;
+    assert.result "" 222 "" "" "" xxx : $(v8).get ;
+
+    # Regression test for a bug causing vector.equal to compare only the first
+    # and the last element in the given vectors.
+    local v9 = [ new vector 111 xxx 222 ] ;
+    local v10 = [ new vector 111 yyy 222 ] ;
+    assert.false $(v9).equal $(v10) ;
+}

mosesdecoder/jam-files/boost-build/util/indirect.jam

@@ -0,0 +1,115 @@
+# Copyright 2003 Dave Abrahams
+# Copyright 2003 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+import modules ;
+import numbers ;
+
+
+# The pattern that indirect rules must match: module%rule
+.pattern = ^([^%]*)%([^%]+)$ ;
+
+
+#
+# Type checking rules.
+#
+local rule indirect-rule ( x )
+{
+    if ! [ MATCH $(.pattern) : $(x) ]
+    {
+        return "expected a string of the form module%rule, but got \""$(x)"\" for argument" ;
+    }
+}
+
+
+# Make an indirect rule which calls the given rule. If context is supplied it is
+# expected to be the module in which to invoke the rule by the 'call' rule
+# below. Otherwise, the rule will be invoked in the module of this rule's
+# caller.
+#
+rule make ( rulename bound-args * : context ? )
+{
+    context ?= [ CALLER_MODULE ] ;
+    context ?= "" ;
+    return $(context)%$(rulename) $(bound-args) ;
+}
+
+
+# Make an indirect rule which calls the given rule. 'rulename' may be a
+# qualified rule; if so it is returned unchanged. Otherwise, if frames is not
+# supplied, the result will be invoked (by 'call', below) in the module of the
+# caller. Otherwise, frames > 1 specifies additional call frames to back up in
+# order to find the module context.
+#
+rule make-qualified ( rulename bound-args * : frames ? )
+{
+    if [ MATCH $(.pattern) : $(rulename) ]
+    {
+        return $(rulename) $(bound-args) ;
+    }
+    else
+    {
+        frames ?= 1 ;
+        # If the rule name includes a Jamfile module, grab it.
+        local module-context = [ MATCH ^(Jamfile<[^>]*>)\\..* : $(rulename) ] ;
+        
+        if ! $(module-context)
+        {                    
+            # Take the first dot-separated element as module name. This disallows
+            # module names with dots, but allows rule names with dots.
+            module-context = [ MATCH ^([^.]*)\\..* : $(rulename) ] ;
+        }        
+        module-context ?= [ CALLER_MODULE $(frames) ] ;
+        return [ make $(rulename) $(bound-args) : $(module-context) ] ;
+    }
+}
+
+
+# Returns the module name in which the given indirect rule will be invoked.
+#
+rule get-module ( [indirect-rule] x )
+{
+    local m = [ MATCH $(.pattern) : $(x) ] ;
+    if ! $(m[1])
+    {
+        m = ;
+    }
+    return $(m[1]) ;
+}
+
+
+# Returns the rulename that will be called when x is invoked.
+#
+rule get-rule ( [indirect-rule] x )
+{
+    local m = [ MATCH $(.pattern) : $(x) ] ;
+    return $(m[2]) ;
+}
+
+
+# Invoke the given indirect-rule.
+#
+rule call ( [indirect-rule] r args * : * )
+{
+    return [ modules.call-in [ get-module $(r) ] : [ get-rule $(r) ] $(args)
+        : $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ] ;
+}
+
+
+rule __test__
+{
+    import assert ;
+
+    rule foo-barr! ( x )
+    {
+        assert.equal $(x) : x ;
+    }
+
+    assert.equal [ get-rule [ make foo-barr! ] ] : foo-barr! ;
+    assert.equal [ get-module [ make foo-barr! ] ] : [ CALLER_MODULE ] ;
+
+    call [ make foo-barr! ] x ;
+    call [ make foo-barr! x ] ;
+    call [ make foo-barr! : [ CALLER_MODULE ] ] x ;
+}

mosesdecoder/jam-files/boost-build/util/option.jam

@@ -0,0 +1,109 @@
+#  Copyright (c) 2005 Vladimir Prus.
+#
+#  Use, modification and distribution is subject to the Boost Software
+#  License Version 1.0. (See accompanying file LICENSE_1_0.txt or
+#  http://www.boost.org/LICENSE_1_0.txt)
+
+import modules ;
+
+# Set a value for a named option, to be used when not overridden on the command
+# line.
+rule set ( name : value ? )
+{
+    .option.$(name) = $(value) ;
+}
+
+rule get ( name : default-value ? : implied-value ? )
+{
+    local m = [ MATCH --$(name)=(.*) : [ modules.peek : ARGV ] ] ;
+    if $(m)
+    {
+        return $(m[1]) ;
+    }
+    else  
+    {        
+        m = [ MATCH (--$(name)) : [ modules.peek : ARGV ] ] ;
+        if $(m) && $(implied-value)
+        {
+            return $(implied-value) ;
+        }
+        else if $(.option.$(name))
+        {
+            return $(.option.$(name)) ;
+        }    
+        else
+        {
+            return $(default-value) ;
+        }    
+    }              
+}
+
+
+# Check command-line args as soon as possible. For each option try to load
+# module named after option. Is that succeeds, invoke 'process' rule in the
+# module. The rule may return "true" to indicate that the regular build process
+# should not be attempted.
+#
+# Options take the general form of: --<name>[=<value>] [<value>]
+#
+rule process ( )
+{
+    local ARGV = [ modules.peek : ARGV ] ;
+    local BOOST_BUILD_PATH = [ modules.peek : BOOST_BUILD_PATH ] ;
+
+    local dont-build ;
+    local args = $(ARGV) ;
+    while $(args)
+    {
+        local arg = [ MATCH ^--(.*) : $(args[1]) ] ;
+        while $(args[2-]) && ! $(arg)
+        {
+            args = $(args[2-]) ;
+            arg = [ MATCH ^--(.*) : $(args[1]) ] ;
+        }
+        args = $(args[2-]) ;
+
+        if $(arg)
+        {
+            local split = [ MATCH ^(([^-=]+)[^=]*)(=?)(.*)$ : $(arg) ] ;
+            local full-name = $(split[1]) ;
+            local prefix = $(split[2]) ;
+            local values ;
+
+            if $(split[3])
+            {
+                values = $(split[4]) ;
+            }
+            if $(args) && ! [ MATCH ^(--).* : $(args[1]) ]
+            {
+                values += $(args[1]) ;
+                args = $(args[2-]) ;
+            }
+
+            # Jook in options subdirectories of BOOST_BUILD_PATH for modules
+            # matching the full option name and then its prefix.
+            local plugin-dir = options ;
+            local option-files = [ GLOB $(plugin-dir:D=$(BOOST_BUILD_PATH)) :
+                $(full-name).jam $(prefix).jam ] ;
+
+            if $(option-files)
+            {
+                # Load the file into a module named for the option.
+                local f = $(option-files[1]) ;
+                local module-name = --$(f:D=:S=) ;
+                modules.load $(module-name) : $(f:D=) : $(f:D) ;
+
+                # If there is a process rule, call it with the full option name
+                # and its value (if any). If there was no "=" in the option, the
+                # value will be empty.
+                if process in [ RULENAMES $(module-name) ]
+                {
+                    dont-build += [ modules.call-in $(module-name) : process
+                        --$(full-name) : $(values) ] ;
+                }
+            }
+        }
+    }
+
+    return $(dont-build) ;
+}

mosesdecoder/jam-files/boost-build/util/os.jam

@@ -0,0 +1,171 @@
+# Copyright 2001, 2002, 2003, 2005 Dave Abrahams
+# Copyright 2006 Rene Rivera
+# Copyright 2003, 2005 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+import modules ;
+import string ;
+
+
+# Return the value(s) of the given environment variable(s) at the time bjam was
+# invoked.
+rule environ ( variable-names + )
+{
+    return [ modules.peek .ENVIRON : $(variable-names) ] ;
+}
+
+.name = [ modules.peek : OS ] ;
+.platform = [ modules.peek : OSPLAT ] ;
+.version = [ modules.peek : OSVER ] ;
+
+
+local rule constant ( c : os ? )
+{
+    os ?= $(.name) ;
+    # First look for a platform-specific name, then the general value.
+    local variables = .$(c)-$(os) .$(c) ;
+    local result = $($(variables)) ;
+    return $(result[1]) ;
+}
+
+rule get-constant ( os ? )
+{
+    # Find the name of the constant being accessed, which is equal to the name
+    # used to invoke us.
+    local bt = [ BACKTRACE 1 ] ;
+    local rulename = [ MATCH ([^.]*)$ : $(bt[4]) ] ;
+    return [ constant $(rulename) : $(os) ] ;
+}
+
+
+# export all the common constants
+.constants = name platform version shared-library-path-variable path-separator executable-path-variable executable-suffix ;
+for local constant in $(.constants)
+{
+    IMPORT $(__name__) : get-constant : $(__name__) : $(constant) ;
+}
+EXPORT $(__name__) : $(.constants) ;
+
+.executable-path-variable-NT = PATH ;
+# On Windows the case and capitalization of PATH is not always predictable, so
+# let's find out what variable name was really set.
+if $(.name) = NT
+{
+    for local n in [ VARNAMES .ENVIRON ]
+    {
+        if $(n:L) = path
+        {
+            .executable-path-variable-NT = $(n) ;
+        }
+    }
+}
+
+# Specific constants for various platforms.  There's no need to define any
+# constant whose value would be the same as the default, below.
+.shared-library-path-variable-NT = $(.executable-path-variable-NT) ;
+.path-separator-NT = ";" ;
+.expand-variable-prefix-NT = % ;
+.expand-variable-suffix-NT = % ;
+.executable-suffix-NT = .exe ;
+
+.shared-library-path-variable-CYGWIN = PATH ;
+
+.shared-library-path-variable-MACOSX = DYLD_LIBRARY_PATH ;
+
+.shared-library-path-variable-AIX = LIBPATH ;
+
+# Default constants
+.shared-library-path-variable = LD_LIBRARY_PATH ;
+.path-separator = ":" ;
+.expand-variable-prefix = $ ;
+.expand-variable-suffix = "" ;
+.executable-path-variable = PATH ;
+.executable-suffix = "" ;
+
+
+# Return a list of the directories in the PATH. Yes, that information is (sort
+# of) available in the global module, but jam code can change those values, and
+# it isn't always clear what case/capitalization to use when looking. This rule
+# is a more reliable way to get there.
+rule executable-path ( )
+{
+    return [ string.words [ environ [ constant executable-path-variable ] ]
+        : [ constant path-separator ] ] ;
+}
+
+
+# Initialize the list of home directories for the current user depending on the
+# OS.
+if $(.name) = NT
+{
+    local home = [ environ HOMEDRIVE HOMEPATH ] ;
+    .home-directories = $(home[1])$(home[2]) [ environ HOME ] [ environ USERPROFILE ] ;
+}
+else
+{
+    .home-directories = [ environ HOME ] ;
+}
+
+
+# Can't use 'constant' mechanism because it only returns 1-element values.
+rule home-directories ( )
+{
+    return $(.home-directories) ;
+}
+
+
+# Return the string needed to represent the expansion of the named shell
+# variable.
+rule expand-variable ( variable )
+{
+    local prefix = [ constant expand-variable-prefix ] ;
+    local suffix = [ constant expand-variable-suffix ] ;
+    return $(prefix)$(variable)$(suffix) ;
+}
+
+
+# Returns true if running on windows, whether in cygwin or not.
+rule on-windows ( )
+{
+    local result ;
+    if [ modules.peek : NT ]
+    {
+        result = true ;
+    }
+    else if [ modules.peek : UNIX ]
+    {
+        switch [ modules.peek : JAMUNAME ]
+        {
+            case CYGWIN* :
+            {
+                result = true ;
+            }
+        }
+    }
+    return $(result) ;
+}
+
+
+if ! [ on-windows ]
+{
+    .on-unix = 1 ;
+}
+
+
+rule on-unix
+{
+    return $(.on-unix) ;
+}
+
+
+rule __test__
+{
+    import assert ;
+    if ! ( --quiet in [ modules.peek : ARGV ] )
+    {
+        ECHO os: name= [ name ] ;
+        ECHO os: version= [ version ] ;
+    }
+    assert.true name ;
+}

mosesdecoder/jam-files/boost-build/util/path.jam

@@ -0,0 +1,934 @@
+# Copyright Vladimir Prus 2002-2006.
+# Copyright Dave Abrahams 2003-2004.
+# Copyright Rene Rivera 2003-2006.
+#
+# Distributed under the Boost Software License, Version 1.0.
+#    (See accompanying file LICENSE_1_0.txt or copy at
+#          http://www.boost.org/LICENSE_1_0.txt)
+
+# Performs various path manipulations. Paths are always in a 'normalized'
+# representation. In it, a path may be either:
+#
+#     - '.', or
+#
+#     - ['/'] [ ( '..' '/' )* (token '/')* token ]
+#
+# In plain english, path can be rooted, '..' elements are allowed only at the
+# beginning, and it never ends in slash, except for path consisting of slash
+# only.
+
+import errors ;
+import modules ;
+import regex ;
+import sequence ;
+import set ;
+import version ;
+
+
+os = [ modules.peek : OS ] ;
+if [ modules.peek : UNIX ]
+{
+    local uname = [ modules.peek : JAMUNAME ] ;
+    switch $(uname)
+    {
+        case CYGWIN* : os = CYGWIN ;
+        case *       : os = UNIX ;
+    }
+}
+
+
+# Converts the native path into normalized form.
+#
+rule make ( native )
+{
+    return [ make-$(os) $(native) ] ;
+}
+
+
+# Builds native representation of the path.
+#
+rule native ( path )
+{
+    return [ native-$(os) $(path) ] ;
+}
+
+
+# Tests if a path is rooted.
+#
+rule is-rooted ( path )
+{
+    return [ MATCH "^(/)" : $(path) ] ;
+}
+
+
+# Tests if a path has a parent.
+#
+rule has-parent ( path )
+{
+    if $(path) != /
+    {
+        return 1 ;
+    }
+    else
+    {
+        return ;
+    }
+}
+
+
+# Returns the path without any directory components.
+#
+rule basename ( path )
+{
+    return [ MATCH "([^/]+)$" : $(path) ] ;
+}
+
+
+# Returns parent directory of the path. If no parent exists, error is issued.
+#
+rule parent ( path )
+{
+    if [ has-parent $(path) ]
+    {
+        if $(path) = .
+        {
+            return .. ;
+        }
+        else
+        {
+            # Strip everything at the end of path up to and including the last
+            # slash.
+            local result = [ regex.match "((.*)/)?([^/]+)" : $(path) : 2 3 ] ;
+
+            # Did we strip what we shouldn't?
+            if $(result[2]) = ".."
+            {
+                return $(path)/.. ;
+            }
+            else
+            {
+                if ! $(result[1])
+                {
+                    if [ is-rooted $(path) ]
+                    {
+                        result = / ;
+                    }
+                    else
+                    {
+                        result = . ;
+                    }
+                }
+                return $(result[1]) ;
+            }
+        }
+    }
+    else
+    {
+        errors.error "Path '$(path)' has no parent" ;
+    }
+}
+
+
+# Returns path2 such that "[ join path path2 ] = .". The path may not contain
+# ".." element or be rooted.
+#
+rule reverse ( path )
+{
+    if $(path) = .
+    {
+        return $(path) ;
+    }
+    else
+    {
+        local tokens = [ regex.split $(path) "/" ] ;
+        local tokens2 ;
+        for local i in $(tokens)
+        {
+            tokens2 += .. ;
+        }
+        return [ sequence.join $(tokens2) : "/" ] ;
+    }
+}
+
+
+# Concatenates the passed path elements. Generates an error if any element other
+# than the first one is rooted. Skips any empty or undefined path elements.
+#
+rule join ( elements + )
+{
+    if ! $(elements[2-])
+    {
+        return $(elements[1]) ;
+    }
+    else
+    {
+        for local e in $(elements[2-])
+        {
+            if [ is-rooted $(e) ]
+            {
+                errors.error only the first element may be rooted ;
+            }
+        }
+        if [ version.check-jam-version 3 1 17 ]
+        {
+            return [ NORMALIZE_PATH "$(elements)" ] ;
+        }
+        else
+        {
+            # Boost Jam prior to version 3.1.17 had problems with its
+            # NORMALIZE_PATH rule in case you passed it a leading backslash
+            # instead of a slash, in some cases when you sent it an empty
+            # initial path element and possibly some others. At least some of
+            # those cases were being hit and relied upon when calling this rule
+            # from the path.make-NT rule.
+            if ! $(elements[1]) && $(elements[2])
+            {
+                return [ NORMALIZE_PATH "/" "$(elements[2-])" ] ;
+            }
+            else
+            {
+                return [ NORMALIZE_PATH "$(elements)" ] ;
+            }
+        }
+    }
+}
+
+
+# If 'path' is relative, it is rooted at 'root'. Otherwise, it is unchanged.
+#
+rule root ( path root )
+{
+    if [ is-rooted $(path) ]
+    {
+        return $(path) ;
+    }
+    else
+    {
+        return [ join $(root) $(path) ] ;
+    }
+}
+
+
+# Returns the current working directory.
+#
+rule pwd ( )
+{
+    if ! $(.pwd)
+    {
+        .pwd = [ make [ PWD ] ] ;
+    }
+    return $(.pwd) ;
+}
+
+
+# Returns the list of files matching the given pattern in the specified
+# directory. Both directories and patterns are supplied as portable paths. Each
+# pattern should be non-absolute path, and can't contain "." or ".." elements.
+# Each slash separated element of pattern can contain the following special
+# characters:
+#   - '?', which match any character
+#   - '*', which matches arbitrary number of characters.
+# A file $(d)/e1/e2/e3 (where 'd' is in $(dirs)) matches pattern p1/p2/p3 if and
+# only if e1 matches p1, e2 matches p2 and so on.
+#
+# For example:
+#   [ glob . : *.cpp ]
+#   [ glob . : */build/Jamfile ]
+#
+rule glob ( dirs * : patterns + : exclude-patterns * )
+{
+    local result ;
+    local real-patterns ;
+    local real-exclude-patterns ;
+    for local d in $(dirs)
+    {
+        for local p in $(patterns)
+        {
+            local pattern = [ path.root $(p) $(d) ] ;
+            real-patterns += [ path.native $(pattern) ] ;
+        }
+
+        for local p in $(exclude-patterns)
+        {
+            local pattern = [ path.root $(p) $(d) ] ;
+            real-exclude-patterns += [ path.native $(pattern) ] ;
+        }
+    }
+
+    local inc = [ GLOB-RECURSIVELY $(real-patterns) ] ;
+    inc = [ sequence.transform NORMALIZE_PATH : $(inc) ] ;
+    local exc = [ GLOB-RECURSIVELY $(real-exclude-patterns) ] ;
+    exc = [ sequence.transform NORMALIZE_PATH : $(exc) ] ;
+
+    return [ sequence.transform path.make : [ set.difference $(inc) : $(exc) ] ]
+        ;
+}
+
+
+# Recursive version of GLOB. Builds the glob of files while also searching in
+# the subdirectories of the given roots. An optional set of exclusion patterns
+# will filter out the matching entries from the result. The exclusions also
+# apply to the subdirectory scanning, such that directories that match the
+# exclusion patterns will not be searched.
+#
+rule glob-tree ( roots * : patterns + : exclude-patterns * )
+{
+    return [ sequence.transform path.make : [ .glob-tree [ sequence.transform
+        path.native : $(roots) ] : $(patterns) : $(exclude-patterns) ] ] ;
+}
+
+
+local rule .glob-tree ( roots * : patterns * : exclude-patterns * )
+{
+    local excluded ;
+    if $(exclude-patterns)
+    {
+        excluded = [ GLOB $(roots) : $(exclude-patterns) ] ;
+    }
+    local result = [ set.difference [ GLOB $(roots) : $(patterns) ] :
+        $(excluded) ] ;
+    local subdirs ;
+    for local d in [ set.difference [ GLOB $(roots) : * ] : $(excluded) ]
+    {
+        if ! ( $(d:D=) in . .. ) && ! [ CHECK_IF_FILE $(d) ]
+        {
+            subdirs += $(d) ;
+        }
+    }
+    if $(subdirs)
+    {
+        result += [ .glob-tree $(subdirs) : $(patterns) : $(exclude-patterns) ]
+            ;
+    }
+    return $(result) ;
+}
+
+
+# Returns true is the specified file exists.
+#
+rule exists ( file )
+{
+    return [ path.glob $(file:D) : $(file:D=) ] ;
+}
+NATIVE_RULE path : exists ;
+
+
+# Find out the absolute name of path and returns the list of all the parents,
+# starting with the immediate one. Parents are returned as relative names. If
+# 'upper_limit' is specified, directories above it will be pruned.
+#
+rule all-parents ( path : upper_limit ? : cwd ? )
+{
+    cwd ?= [ pwd ] ;
+    local path_ele = [ regex.split [ root $(path) $(cwd) ] "/" ] ;
+
+    if ! $(upper_limit)
+    {
+        upper_limit = / ;
+    }
+    local upper_ele = [ regex.split [ root $(upper_limit) $(cwd) ] "/" ] ;
+
+    # Leave only elements in 'path_ele' below 'upper_ele'.
+    while $(path_ele) && ( $(upper_ele[1]) = $(path_ele[1]) )
+    {
+        upper_ele = $(upper_ele[2-]) ;
+        path_ele = $(path_ele[2-]) ;
+    }
+
+    # Have all upper elements been removed ?
+    if $(upper_ele)
+    {
+        errors.error "$(upper_limit) is not prefix of $(path)" ;
+    }
+    
+    # Create the relative paths to parents, number of elements in 'path_ele'.
+    local result ;
+    for local i in $(path_ele)
+    {
+        path = [ parent $(path) ] ;
+        result += $(path) ;
+    }
+    return $(result) ;
+}
+
+
+# Search for 'pattern' in parent directories of 'dir', up till and including
+# 'upper_limit', if it is specified, or till the filesystem root otherwise.
+#
+rule glob-in-parents ( dir : patterns + : upper-limit ? )
+{
+    local result ;
+    local parent-dirs = [ all-parents $(dir) : $(upper-limit) ] ;
+
+    while $(parent-dirs) && ! $(result)
+    {
+        result = [ glob $(parent-dirs[1]) : $(patterns) ] ;
+        parent-dirs = $(parent-dirs[2-]) ;
+    }
+    return $(result) ;
+}
+
+
+# Assuming 'child' is a subdirectory of 'parent', return the relative path from
+# 'parent' to 'child'.
+#
+rule relative ( child parent : no-error ? )
+{
+    local not-a-child ;
+    if $(parent) = "."
+    {
+        return $(child) ;
+    }
+    else
+    {
+        local split1 = [ regex.split $(parent) / ] ;
+        local split2 = [ regex.split $(child) / ] ;
+
+        while $(split1)
+        {
+            if $(split1[1]) = $(split2[1])
+            {
+                split1 = $(split1[2-]) ;
+                split2 = $(split2[2-]) ;
+            }
+            else
+            {
+                not-a-child = true ;
+                split1 = ;
+            }
+        }
+        if $(split2)
+        {
+            if $(not-a-child)
+            {
+                if $(no-error)
+                {
+                    return not-a-child ;
+                }
+                else
+                {
+                    errors.error $(child) is not a subdir of $(parent) ;
+                }                        
+            }
+            else
+            {
+                return [ join $(split2) ] ;    
+            }        
+        }
+        else
+        {
+            return "." ;
+        }
+    }
+}
+
+
+# Returns the minimal path to path2 that is relative path1.
+#
+rule relative-to ( path1 path2 )
+{
+    local root_1 = [ regex.split [ reverse $(path1) ] / ] ;
+    local split1 = [ regex.split $(path1) / ] ;
+    local split2 = [ regex.split $(path2) / ] ;
+
+    while $(split1) && $(root_1)
+    {
+        if $(split1[1]) = $(split2[1])
+        {
+            root_1 = $(root_1[2-]) ;
+            split1 = $(split1[2-]) ;
+            split2 = $(split2[2-]) ;
+        }
+        else
+        {
+            split1 = ;
+        }
+    }
+    return [ join . $(root_1) $(split2) ] ;
+}
+
+
+# Returns the list of paths which are used by the operating system for looking
+# up programs.
+#
+rule programs-path ( )
+{
+    local result ;
+    local raw = [ modules.peek : PATH Path path ] ;
+    for local p in $(raw)
+    {
+        if $(p)
+        {
+            result += [ path.make $(p) ] ;
+        }
+    }
+    return $(result) ;
+}
+
+rule makedirs ( path )
+{
+    local result = true ;
+    local native = [ native $(path) ] ;
+    if ! [ exists $(native) ]
+    {
+        if [ makedirs [ parent $(path) ] ]
+        {            
+            if ! [ MAKEDIR $(native) ]
+            {
+                errors.error "Could not create directory '$(path)'" ;
+                result = ;
+            }
+        }        
+    }    
+    return $(result) ;
+}
+
+# Converts native Windows paths into our internal canonic path representation.
+# Supports 'invalid' paths containing multiple successive path separator
+# characters.
+#
+# TODO: Check and if needed add support for Windows 'X:file' path format where
+# the file is located in the current folder on drive X.
+#
+rule make-NT ( native )
+{
+    local result ;
+
+    if [ version.check-jam-version 3 1 17 ]
+    {
+        result = [ NORMALIZE_PATH $(native) ] ;
+    }
+    else
+    {
+        # This old implementation is really fragile due to a not so clear way
+        # NORMALIZE_PATH rule worked in Boost.Jam versions prior to 3.1.17. E.g.
+        # path.join would mostly ignore empty path elements but would root the
+        # joined path in case the initial two path elements were empty or some
+        # similar accidental wierdness.
+        result = [ path.join [ regex.split $(native) "[/\\]" ] ] ;
+    }
+
+    # We need to add an extra '/' in front in case this is a rooted Windows path
+    # starting with a drive letter and not a path separator character since the
+    # builtin NORMALIZE_PATH rule has no knowledge of this leading drive letter
+    # and treats it as a regular folder name.
+    if [ regex.match "(^.:)" : $(native) ]
+    {
+        result = /$(result) ;
+    }
+
+    return $(result) ;
+}
+
+
+rule native-NT ( path )
+{
+    local result ;
+    if [ is-rooted $(path) ] && ! [ regex.match "^/(.:)" : $(path) ]
+    {
+        result = $(path) ;
+    }
+    else
+    {
+        result = [ MATCH "^/?(.*)" : $(path) ] ;
+    }
+    result = [ sequence.join [ regex.split $(result) "/" ] : "\\" ] ;
+    return $(result) ;
+}
+
+
+rule make-UNIX ( native )
+{
+    # VP: I have no idea now 'native' can be empty here! But it can!
+    if ! $(native)
+    {
+        errors.error "Empty path passed to 'make-UNIX'" ;
+    }
+    else
+    {
+        return [ NORMALIZE_PATH $(native:T) ] ;
+    }
+}
+
+
+rule native-UNIX ( path )
+{
+    return $(path) ;
+}
+
+
+rule make-CYGWIN ( path )
+{
+    return [ make-NT $(path) ] ;
+}
+
+
+rule native-CYGWIN ( path )
+{
+    local result = $(path) ;
+    if [ regex.match "(^/.:)" : $(path)  ]  # Windows absolute path.
+    {
+        result = [ MATCH "^/?(.*)" : $(path) ] ;  # Remove leading '/'.
+    }
+    return [ native-UNIX $(result) ] ;
+}
+
+
+# split-path-VMS: splits input native path into device dir file (each part is
+# optional).
+#
+# example:
+#
+# dev:[dir]file.c => dev: [dir] file.c
+#
+rule split-path-VMS ( native )
+{
+    local matches = [ MATCH ([a-zA-Z0-9_-]+:)?(\\[[^\]]*\\])?(.*)?$   : $(native) ] ;
+    local device = $(matches[1]) ;
+    local dir = $(matches[2]) ;
+    local file = $(matches[3]) ;
+
+    return $(device) $(dir) $(file) ;
+}
+
+
+# Converts a native VMS path into a portable path spec.
+#
+# Does not handle current-device absolute paths such as "[dir]File.c" as it is
+# not clear how to represent them in the portable path notation.
+#
+# Adds a trailing dot (".") to the file part if no extension is present (helps
+# when converting it back into native path).
+#
+rule make-VMS ( native )
+{
+    if [ MATCH ^(\\[[a-zA-Z0-9]) : $(native) ]
+    {
+        errors.error "Can't handle default-device absolute paths: " $(native) ;
+    }
+
+    local parts = [ split-path-VMS $(native) ] ;
+    local device = $(parts[1]) ;
+    local dir = $(parts[2]) ;
+    local file = $(parts[3]) ;
+    local elems ;
+
+    if $(device)
+    {
+        #
+        # rooted
+        #
+        elems = /$(device) ;
+    }
+
+    if $(dir) = "[]"
+    {
+        #
+        # Special case: current directory
+        #
+        elems = $(elems) "." ;
+    }
+    else if $(dir)
+    {
+        dir = [ regex.replace $(dir) "\\[|\\]" "" ] ;
+        local dir_parts = [ regex.split $(dir) \\. ]  ;
+
+        if $(dir_parts[1]) = ""
+        {
+            #
+            # Relative path
+            #
+            dir_parts = $(dir_parts[2--1]) ;
+        }
+
+        #
+        # replace "parent-directory" parts (- => ..)
+        #
+        dir_parts = [ regex.replace-list $(dir_parts) : - : .. ] ;
+
+        elems = $(elems) $(dir_parts) ;
+    }
+
+    if $(file)
+    {
+        if ! [ MATCH (\\.) : $(file) ]
+        {
+            #
+            # Always add "." to end of non-extension file.
+            #
+            file = $(file). ;
+        }
+        elems = $(elems) $(file) ;
+    }
+
+    local portable = [ path.join $(elems) ] ;
+
+    return $(portable) ;
+}
+
+
+# Converts a portable path spec into a native VMS path.
+#
+# Relies on having at least one dot (".") included in the file name to be able
+# to differentiate it from the directory part.
+#
+rule native-VMS ( path )
+{
+    local device = "" ;
+    local dir = $(path) ;
+    local file = "" ;
+    local native ;
+    local split ;
+
+    #
+    # Has device ?
+    #
+    if [ is-rooted $(dir) ]
+    {
+        split = [ MATCH ^/([^:]+:)/?(.*) : $(dir) ] ;
+        device = $(split[1]) ;
+        dir = $(split[2]) ;
+    }
+
+    #
+    # Has file ?
+    #
+    # This is no exact science, just guess work:
+    #
+    # If the last part of the current path spec
+    # includes some chars, followed by a dot,
+    # optionally followed by more chars -
+    # then it is a file (keep your fingers crossed).
+    #
+    split = [ regex.split $(dir) / ] ;
+    local maybe_file = $(split[-1]) ;
+
+    if [ MATCH ^([^.]+\\..*) : $(maybe_file) ]
+    {
+        file = $(maybe_file) ;
+        dir = [ sequence.join $(split[1--2]) : / ] ;
+    }
+
+    #
+    # Has dir spec ?
+    #
+    if $(dir) = "."
+    {
+        dir = "[]" ;
+    }
+    else if $(dir)
+    {
+        dir = [ regex.replace $(dir) \\.\\. - ] ;
+        dir = [ regex.replace $(dir) / . ] ;
+
+        if $(device) = ""
+        {
+            #
+            # Relative directory
+            #
+            dir = "."$(dir) ;
+        }
+        dir = "["$(dir)"]" ;
+    }
+
+    native = [ sequence.join $(device) $(dir) $(file) ] ;
+
+    return $(native) ;
+}
+
+
+rule __test__ ( )
+{
+    import assert ;
+    import errors : try catch ;
+
+    assert.true is-rooted "/" ;
+    assert.true is-rooted "/foo" ;
+    assert.true is-rooted "/foo/bar" ;
+    assert.result : is-rooted "." ;
+    assert.result : is-rooted "foo" ;
+    assert.result : is-rooted "foo/bar" ;
+
+    assert.true has-parent "foo" ;
+    assert.true has-parent "foo/bar" ;
+    assert.true has-parent "." ;
+    assert.result : has-parent "/" ;
+
+    assert.result "." : basename "." ;
+    assert.result ".." : basename ".." ;
+    assert.result "foo" : basename "foo" ;
+    assert.result "foo" : basename "bar/foo" ;
+    assert.result "foo" : basename "gaz/bar/foo" ;
+    assert.result "foo" : basename "/gaz/bar/foo" ;
+
+    assert.result "." : parent "foo" ;
+    assert.result "/" : parent "/foo" ;
+    assert.result "foo/bar" : parent "foo/bar/giz" ;
+    assert.result ".." : parent "." ;
+    assert.result ".." : parent "../foo" ;
+    assert.result "../../foo" : parent "../../foo/bar" ;
+
+    assert.result "." : reverse "." ;
+    assert.result ".." : reverse "foo" ;
+    assert.result "../../.." : reverse "foo/bar/giz" ;
+
+    assert.result "foo" : join "foo" ;
+    assert.result "/foo" : join "/" "foo" ;
+    assert.result "foo/bar" : join "foo" "bar" ;
+    assert.result "foo/bar" : join "foo/giz" "../bar" ;
+    assert.result "foo/giz" : join "foo/bar/baz" "../../giz" ;
+    assert.result ".." : join "." ".." ;
+    assert.result ".." : join "foo" "../.." ;
+    assert.result "../.." : join "../foo" "../.." ;
+    assert.result "/foo" : join "/bar" "../foo" ;
+    assert.result "foo/giz" : join "foo/giz" "." ;
+    assert.result "." : join lib2 ".." ;
+    assert.result "/" : join "/a" ".." ;
+
+    assert.result /a/b : join /a/b/c .. ;
+
+    assert.result "foo/bar/giz" : join "foo" "bar" "giz" ;
+    assert.result "giz" : join "foo" ".." "giz" ;
+    assert.result "foo/giz" : join "foo" "." "giz" ;
+
+    try ;
+    {
+        join "a" "/b" ;
+    }
+    catch only first element may be rooted ;
+
+    local CWD = "/home/ghost/build" ;
+    assert.result : all-parents . : . : $(CWD) ;
+    assert.result . .. ../.. ../../..  : all-parents "Jamfile" : "" : $(CWD) ;
+    assert.result foo . .. ../.. ../../.. : all-parents "foo/Jamfile" : "" : $(CWD) ;
+    assert.result ../Work .. ../.. ../../.. : all-parents "../Work/Jamfile" : "" : $(CWD) ;
+
+    local CWD = "/home/ghost" ;
+    assert.result . .. : all-parents "Jamfile" : "/home" : $(CWD) ;
+    assert.result . : all-parents "Jamfile" : "/home/ghost" : $(CWD) ;
+
+    assert.result "c/d" : relative "a/b/c/d" "a/b" ;
+    assert.result "foo" : relative "foo" "." ;
+
+    local save-os = [ modules.peek path : os ] ;
+    modules.poke path : os : NT ;
+
+    assert.result "foo/bar/giz" : make "foo/bar/giz" ;
+    assert.result "foo/bar/giz" : make "foo\\bar\\giz" ;
+    assert.result "foo" : make "foo/" ;
+    assert.result "foo" : make "foo\\" ;
+    assert.result "foo" : make "foo/." ;
+    assert.result "foo" : make "foo/bar/.." ;
+    assert.result "foo" : make "foo/bar/../" ;
+    assert.result "foo" : make "foo/bar/..\\" ;
+    assert.result "foo/bar" : make "foo/././././bar" ;
+    assert.result "/foo" : make "\\foo" ;
+    assert.result "/D:/My Documents" : make "D:\\My Documents" ;
+    assert.result "/c:/boost/tools/build/new/project.jam" : make "c:\\boost\\tools\\build\\test\\..\\new\\project.jam" ;
+    
+    #   Test processing 'invalid' paths containing multiple successive path
+    # separators.
+    assert.result "foo" : make "foo//" ;
+    assert.result "foo" : make "foo///" ;
+    assert.result "foo" : make "foo\\\\" ;
+    assert.result "foo" : make "foo\\\\\\" ;
+    assert.result "/foo" : make "//foo" ;
+    assert.result "/foo" : make "///foo" ;
+    assert.result "/foo" : make "\\\\foo" ;
+    assert.result "/foo" : make "\\\\\\foo" ;
+    assert.result "/foo" : make "\\/\\/foo" ;
+    assert.result "foo/bar" : make "foo//\\//\\\\bar//\\//\\\\\\//\\//\\\\" ;
+    assert.result "foo" : make "foo/bar//.." ;
+    assert.result "foo/bar" : make "foo/bar/giz//.." ;
+    assert.result "foo/giz" : make "foo//\\//\\\\bar///\\\\//\\\\////\\/..///giz\\//\\\\\\//\\//\\\\" ;
+    assert.result "../../../foo" : make "..///.//..///.//..////foo///" ;
+
+    #   Test processing 'invalid' rooted paths with too many '..' path elements
+    # that would place them before the root.
+    assert.result : make "/.." ;
+    assert.result : make "/../" ;
+    assert.result : make "/../." ;
+    assert.result : make "/.././" ;
+    assert.result : make "/foo/../bar/giz/.././././../../." ;
+    assert.result : make "/foo/../bar/giz/.././././../.././" ;
+    assert.result : make "//foo/../bar/giz/.././././../../." ;
+    assert.result : make "//foo/../bar/giz/.././././../.././" ;
+    assert.result : make "\\\\foo/../bar/giz/.././././../../." ;
+    assert.result : make "\\\\foo/../bar/giz/.././././../.././" ;
+    assert.result : make "/..///.//..///.//..////foo///" ;
+
+    assert.result "foo\\bar\\giz" : native "foo/bar/giz" ;
+    assert.result "foo" : native "foo" ;
+    assert.result "\\foo" : native "/foo" ;
+    assert.result "D:\\My Documents\\Work" : native "/D:/My Documents/Work" ;
+
+    modules.poke path : os : UNIX ;
+
+    assert.result "foo/bar/giz" : make "foo/bar/giz" ;
+    assert.result "/sub1" : make "/sub1/." ;
+    assert.result "/sub1" : make "/sub1/sub2/.." ;
+    assert.result "sub1" : make "sub1/." ;
+    assert.result "sub1" : make "sub1/sub2/.." ;
+    assert.result "/foo/bar" : native "/foo/bar" ;
+
+    modules.poke path : os : VMS ;
+
+    #
+    # Don't really need to poke os before these
+    #
+    assert.result "disk:" "[dir]"  "file" : split-path-VMS "disk:[dir]file" ;
+    assert.result "disk:" "[dir]"  ""     : split-path-VMS "disk:[dir]" ;
+    assert.result "disk:" ""       ""     : split-path-VMS "disk:" ;
+    assert.result "disk:" ""       "file" : split-path-VMS "disk:file" ;
+    assert.result ""      "[dir]"  "file" : split-path-VMS "[dir]file" ;
+    assert.result ""      "[dir]"  ""     : split-path-VMS "[dir]" ;
+    assert.result ""      ""       "file" : split-path-VMS "file" ;
+    assert.result ""      ""       ""     : split-path-VMS "" ;
+
+    #
+    # Special case: current directory
+    #
+    assert.result ""      "[]"     ""     : split-path-VMS "[]" ;
+    assert.result "disk:" "[]"     ""     : split-path-VMS "disk:[]" ;
+    assert.result ""      "[]"     "file" : split-path-VMS "[]file" ;
+    assert.result "disk:" "[]"     "file" : split-path-VMS "disk:[]file" ;
+
+    #
+    # Make portable paths
+    #
+    assert.result "/disk:" : make "disk:" ;
+    assert.result "foo/bar/giz" : make "[.foo.bar.giz]" ;
+    assert.result "foo" : make "[.foo]" ;
+    assert.result "foo" : make "[.foo.bar.-]" ;
+    assert.result ".." : make "[.-]" ;
+    assert.result ".." : make "[-]" ;
+    assert.result "." : make "[]" ;
+    assert.result "giz.h" : make "giz.h" ;
+    assert.result "foo/bar/giz.h" : make "[.foo.bar]giz.h" ;
+    assert.result "/disk:/my_docs" : make "disk:[my_docs]" ;
+    assert.result "/disk:/boost/tools/build/new/project.jam" : make "disk:[boost.tools.build.test.-.new]project.jam" ;
+
+    #
+    # Special case (adds '.' to end of file w/o extension to
+    # disambiguate from directory in portable path spec).
+    #
+    assert.result "Jamfile." : make "Jamfile" ;
+    assert.result "dir/Jamfile." : make "[.dir]Jamfile" ;
+    assert.result "/disk:/dir/Jamfile." : make "disk:[dir]Jamfile" ;
+
+    #
+    # Make native paths
+    #
+    assert.result "disk:" : native "/disk:" ;
+    assert.result "[.foo.bar.giz]" : native "foo/bar/giz" ;
+    assert.result "[.foo]" : native "foo" ;
+    assert.result "[.-]" : native ".." ;
+    assert.result "[.foo.-]" : native "foo/.." ;
+    assert.result "[]" : native "." ;
+    assert.result "disk:[my_docs.work]" : native "/disk:/my_docs/work" ;
+    assert.result "giz.h" : native "giz.h" ;
+    assert.result "disk:Jamfile." : native "/disk:Jamfile." ;
+    assert.result "disk:[my_docs.work]Jamfile." : native "/disk:/my_docs/work/Jamfile." ;
+
+    modules.poke path : os : $(save-os) ;
+}

mosesdecoder/jam-files/boost-build/util/print.jam

@@ -0,0 +1,488 @@
+# Copyright 2003 Douglas Gregor 
+# Copyright 2002, 2003, 2005 Rene Rivera 
+# Copyright 2002, 2003, 2004, 2005 Vladimir Prus 
+# Distributed under the Boost Software License, Version 1.0. 
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt) 
+
+# Utilities for generating format independent output. Using these
+# will help in generation of documentation in at minimum plain/console
+# and html.
+
+import modules ;
+import numbers ;
+import string ;
+import regex ;
+import "class" ;
+import scanner ;
+import path ;
+
+# The current output target. Defaults to console.
+output-target = console ;
+
+# The current output type. Defaults to plain. Other possible values are "html".
+output-type = plain ;
+
+# Whitespace.
+.whitespace = [ string.whitespace ] ;
+
+
+# Set the target and type of output to generate. This sets both the destination
+# output and the type of docs to generate to that output. The target can be
+# either a file or "console" for echoing to the console. If the type of output
+# is not specified it defaults to plain text.
+#
+rule output (
+    target  # The target file or device; file or "console".
+    type ?  # The type of output; "plain" or "html".
+)
+{
+    type ?= plain ;
+    if $(output-target) != $(target)
+    {
+        output-target = $(target) ;
+        output-type = $(type) ;
+        if $(output-type) = html
+        {
+            text
+                "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">"
+                "<html>"
+                "<head>"
+                "</head>"
+                "<body link=\"#0000ff\" vlink=\"#800080\">"
+                : true
+                : prefix ;
+            text
+                "</body>"
+                "</html>"
+                :
+                : suffix ;
+        }
+    }
+}
+
+
+# Generate a section with a description. The type of output can be controlled by
+# the value of the 'output-type' variable.
+#
+rule section (
+    name  # The name of the section.
+    description *  # A number of description lines.
+)
+{
+    if $(output-type) = plain
+    {
+        lines [ split-at-words $(name): ] ;
+        lines ;
+    }
+    else if $(output-type) = html
+    {
+        name = [ escape-html $(name) ] ;
+        text <h3>$(name)</h3> <p> ;
+    }
+    local pre = ;
+    while $(description)
+    {
+        local paragraph = ;
+        while $(description) && [ string.is-whitespace $(description[1]) ] { description = $(description[2-]) ; }
+        if $(pre)
+        {
+            while $(description) && (
+                $(pre) = " $(description[1])" ||
+                ( $(pre) < [ string.chars [ MATCH "^([$(.whitespace)]*)" : " $(description[1])" ] ] )
+                )
+                { paragraph += $(description[1]) ; description = $(description[2-]) ; }
+            while [ string.is-whitespace $(paragraph[-1]) ] { paragraph = $(paragraph[1--2]) ; }
+            pre = ;
+            if $(output-type) = plain
+            {
+                lines $(paragraph) "" : "  " "  " ;
+            }
+            else if $(output-type) = html
+            {
+                text <blockquote> ;
+                lines $(paragraph) ;
+                text </blockquote> ;
+            }
+        }
+        else
+        {
+            while $(description) && ! [ string.is-whitespace $(description[1]) ]
+                { paragraph += $(description[1]) ; description = $(description[2-]) ; }
+            if $(paragraph[1]) = :: && ! $(paragraph[2])
+            {
+                pre = " " ;
+            }
+            if $(paragraph[1]) = ::
+            {
+                if $(output-type) = plain
+                {
+                    lines $(paragraph[2-]) "" : "  " "  " ;
+                    lines ;
+                }
+                else if $(output-type) = html
+                {
+                    text <blockquote> ;
+                    lines $(paragraph[2-]) ;
+                    text </blockquote> ;
+                }
+            }
+            else
+            {
+                local p = [ MATCH "(.*)(::)$" : $(paragraph[-1]) ] ;
+                local pws = [ MATCH "([ 	]*)$" : $(p[1]) ] ;
+                p = [ MATCH "(.*)($(pws))($(p[2]))$" :  $(paragraph[-1]) ] ;
+                if $(p[3]) = ::
+                {
+                    pre = [ string.chars [ MATCH "^([$(.whitespace)]*)" : " $(p[1])" ] ] ;
+                    if ! $(p[2]) || $(p[2]) = "" { paragraph = $(paragraph[1--2]) $(p[1]): ; }
+                    else { paragraph = $(paragraph[1--2]) $(p[1]) ; }
+                    if $(output-type) = plain
+                    {
+                        lines [ split-at-words " " $(paragraph) ] : "  " "  " ;
+                        lines ;
+                    }
+                    else if $(output-type) = html
+                    {
+                        text </p> <p> [ escape-html $(paragraph) ] ;
+                    }
+                }
+                else
+                {
+                    if $(output-type) = plain
+                    {
+                        lines [ split-at-words " " $(paragraph) ] : "  " "  " ;
+                        lines ;
+                    }
+                    else if $(output-type) = html
+                    {
+                        text </p> <p> [ escape-html $(paragraph) ] ;
+                    }
+                }
+            }
+        }
+    }
+    if $(output-type) = html
+    {
+        text </p> ;
+    }
+}
+
+
+# Generate the start of a list of items. The type of output can be controlled by
+# the value of the 'output-type' variable.
+#
+rule list-start ( )
+{
+    if $(output-type) = plain
+    {
+    }
+    else if $(output-type) = html
+    {
+        text <ul> ;
+    }
+}
+
+
+# Generate an item in a list. The type of output can be controlled by the value
+# of the 'output-type' variable.
+#
+rule list-item (
+    item +  # The item to list.
+)
+{
+    if $(output-type) = plain
+    {
+        lines [ split-at-words "*" $(item) ] : "    " "  " ;
+    }
+    else if $(output-type) = html
+    {
+        text <li> [ escape-html $(item) ] </li> ;
+    }
+}
+
+
+# Generate the end of a list of items. The type of output can be controlled by
+# the value of the 'output-type' variable.
+#
+rule list-end ( )
+{
+    if $(output-type) = plain
+    {
+        lines ;
+    }
+    else if $(output-type) = html
+    {
+        text </ul> ;
+    }
+}
+
+
+# Split the given text into separate lines, word-wrapping to a margin. The
+# default margin is 78 characters.
+#
+rule split-at-words (
+    text +  # The text to split.
+    : margin ?  # An optional margin, default is 78.
+)
+{
+    local lines = ;
+    text = [ string.words $(text:J=" ") ] ;
+    text = $(text:J=" ") ;
+    margin ?= 78 ;
+    local char-match-1 = ".?" ;
+    local char-match = "" ;
+    while $(margin) != 0
+    {
+        char-match = $(char-match)$(char-match-1) ;
+        margin = [ numbers.decrement $(margin) ] ;
+    }
+    while $(text)
+    {
+        local s = "" ;
+        local t = "" ;
+        # divide s into the first X characters and the rest
+        s = [ MATCH "^($(char-match))(.*)" : $(text) ] ;
+        
+        if $(s[2])
+        {
+            # split the first half at a space
+            t = [ MATCH "^(.*)[\\ ]([^\\ ]*)$" : $(s[1]) ] ;
+        }
+        else
+        {
+            t = $(s) ;
+        }
+        
+        if ! $(t[2])
+        {
+            t += "" ;
+        }
+        
+        text = $(t[2])$(s[2]) ;
+        lines += $(t[1]) ;
+    }
+    return $(lines) ;
+}
+
+
+# Generate a set of fixed lines. Each single item passed in is output on a
+# separate line. For console this just echos each line, but for html this will
+# split them with <br>.
+#
+rule lines (
+    text *  # The lines of text.
+    : indent ?  # Optional indentation prepended to each line after the first one.
+    outdent ?  # Optional indentation to prepend to the first line.
+)
+{
+    text ?= "" ;
+    indent ?= "" ;
+    outdent ?= "" ;
+    if $(output-type) = plain
+    {
+        text $(outdent)$(text[1]) $(indent)$(text[2-]) ;
+    }
+    else if $(output-type) = html
+    {
+        local indent-chars = [ string.chars $(indent) ] ;
+        indent = "" ;
+        for local c in $(indent-chars)
+        {
+            if $(c) = " " { c = "&nbsp;" ; }
+            else if $(c) = "	" { c = "&nbsp;&nbsp;&nbsp;&nbsp;" ; }
+            indent = $(indent)$(c) ;
+        }
+        local html-text = [ escape-html $(text) : "&nbsp;" ] ;
+        text $(html-text[1])<br> $(indent)$(html-text[2-])<br> ;
+    }
+}
+
+
+# Output text directly to the current target. When doing output to a file, one
+# can indicate if the text should be output to "prefix" it, as the "body"
+# (default), or "suffix" of the file. This is independant of the actual
+# execution order of the text rule. This rule invokes a singular action, one
+# action only once, which does the build of the file. Therefore actions on the
+# target outside of this rule will happen entirely before and/or after all
+# output using this rule.
+#
+rule text (
+    strings *  # The strings of text to output.
+    : overwrite ?  # true to overwrite the output (if it is a file)
+    : prefix-body-suffix ?  # Indication to output prefix, body, or suffix (for a file).
+)
+{
+    prefix-body-suffix ?= body ;
+    if $(output-target) = console
+    {
+        if ! $(strings)
+        {
+            ECHO ;
+        }
+        else
+        {
+            for local s in $(strings)
+            {
+                ECHO $(s) ;
+            }
+        }
+    }
+    if ! $($(output-target).did-action)
+    {
+        $(output-target).did-action = yes ;
+        $(output-target).text-prefix = ;
+        $(output-target).text-body = ;
+        $(output-target).text-suffix = ;
+        
+        nl on $(output-target) = "
+" ;
+        text-redirect on $(output-target) = ">>" ;
+        if $(overwrite)
+        {
+            text-redirect on $(output-target) = ">" ;
+        }
+        text-content on $(output-target) = ;
+        
+        text-action $(output-target) ;
+
+        if $(overwrite) && $(output-target) != console
+        {
+            check-for-update $(output-target) ;
+        }
+    }
+    $(output-target).text-$(prefix-body-suffix) += $(strings) ;
+    text-content on $(output-target) =
+        $($(output-target).text-prefix)
+        $($(output-target).text-body)
+        $($(output-target).text-suffix) ;
+}
+
+
+# Outputs the text to the current targets, after word-wrapping it.
+#
+rule wrapped-text ( text + )
+{
+    local lines = [ split-at-words $(text) ] ;
+    text $(lines) ;
+}
+
+
+# Escapes text into html/xml printable equivalents. Does not know about tags and
+# therefore tags fed into this will also be escaped. Currently escapes space,
+# "<", ">", and "&".
+#
+rule escape-html (
+    text +  # The text to escape.
+    : space ?  # What to replace spaces with, defaults to " ".
+)
+{
+    local html-text = ;
+    while $(text)
+    {
+        local html = $(text[1]) ;
+        text = $(text[2-]) ;
+        html = [ regex.replace $(html) "&" "&amp;" ] ;
+        html = [ regex.replace $(html) "<" "&lt;" ] ;
+        html = [ regex.replace $(html) ">" "&gt;" ] ;
+        if $(space)
+        {
+            html = [ regex.replace $(html) " " "$(space)" ] ;
+        }
+        html-text += $(html) ;
+    }
+    return $(html-text) ;
+}
+
+
+# Outputs the text strings collected by the text rule to the output file.
+#
+actions quietly text-action
+{
+    @($(STDOUT):E=$(text-content:J=$(nl))) $(text-redirect) "$(<)"
+}
+
+
+rule get-scanner ( )
+{
+    if ! $(.scanner)
+    {
+        .scanner = [ class.new print-scanner ] ;
+    }
+    return $(.scanner) ;
+}
+
+
+# The following code to update print targets when their contents
+# change is a horrible hack.  It basically creates a target which
+# binds to this file (print.jam) and installs a scanner on it
+# which reads the target and compares its contents to the new
+# contents that we're writing.
+#
+rule check-for-update ( target )
+{
+    local scanner = [ get-scanner ] ;
+    local file = [ path.native [ modules.binding $(__name__) ] ] ;
+    local g = [ MATCH <(.*)> : $(target:G) ] ;
+    local dependency-target = $(__file__:G=$(g:E=)-$(target:G=)-$(scanner)) ;
+    DEPENDS $(target) : $(dependency-target) ;
+    SEARCH on $(dependency-target) = $(file:D) ;
+    ISFILE $(dependency-target) ;
+    NOUPDATE $(dependency-target) ;
+    base on $(dependency-target) = $(target) ;
+    scanner.install $(scanner) : $(dependency-target) none ;
+    return $(dependency-target) ;
+}
+
+
+class print-scanner : scanner
+{
+    import path ;
+    import os ;
+
+    rule pattern ( )
+    {
+        return "(One match...)" ;
+    }
+
+    rule process ( target : matches * : binding )
+    {
+        local base = [ on $(target) return $(base) ] ;
+        local nl = [ on $(base) return $(nl) ] ;
+        local text-content = [ on $(base) return $(text-content) ] ;
+        local dir = [ on $(base) return $(LOCATE) ] ;
+        if $(dir)
+        {
+            dir = [ path.make $(dir) ] ;
+        }
+        local file = [ path.native [ path.join $(dir) $(base:G=) ] ] ;
+        local actual-content ;
+        if [ os.name ] = NT
+        {
+            actual-content = [ SHELL "type \"$(file)\" 2>nul" ] ;
+        }
+        else
+        {
+            actual-content = [ SHELL "cat \"$(file)\" 2>/dev/null" ] ;
+        }
+        if $(text-content:J=$(nl)) != $(actual-content)
+        {
+            ALWAYS $(base) ;
+        }
+    }
+}
+
+
+rule __test__ ( )
+{
+    import assert ;
+    
+    assert.result one two three   : split-at-words one two three : 5 ;
+    assert.result "one two" three : split-at-words one two three : 8 ;
+    assert.result "one two" three : split-at-words one two three : 9 ;
+    assert.result "one two three" : split-at-words one two three ;
+
+    # VP, 2004-12-03 The following test fails for some reason, so commenting it
+    # out.
+    #assert.result "one&nbsp;two&nbsp;three" "&amp;&lt;&gt;" :
+    #    escape-html "one two three" "&<>" ;
+}

mosesdecoder/jam-files/boost-build/util/sequence.jam

@@ -0,0 +1,335 @@
+# Copyright 2001, 2002, 2003 Dave Abrahams 
+# Copyright 2006 Rene Rivera 
+# Copyright 2002, 2003 Vladimir Prus 
+# Distributed under the Boost Software License, Version 1.0. 
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt) 
+
+import assert ;
+import numbers ; 
+import modules ;
+
+
+# Note that algorithms in this module execute largely in the caller's module
+# namespace, so that local rules can be used as function objects. Also note that
+# most predicates can be multi-element lists. In that case, all but the first
+# element are prepended to the first argument which is passed to the rule named
+# by the first element.
+
+
+# Return the elements e of $(sequence) for which [ $(predicate) e ] has a
+# non-null value.
+#
+rule filter ( predicate + : sequence * )
+{
+    local caller = [ CALLER_MODULE ] ;
+    local result ;
+
+    for local e in $(sequence)
+    {
+        if [ modules.call-in $(caller) : $(predicate) $(e) ]
+        {
+            result += $(e) ;
+        }
+    }
+    return $(result) ;
+}
+
+
+# Return a new sequence consisting of [ $(function) $(e) ] for each element e of
+# $(sequence).
+#
+rule transform ( function + : sequence * )
+{
+    local caller = [ CALLER_MODULE ] ;
+    local result ;
+
+    for local e in $(sequence)
+    {
+        result += [ modules.call-in $(caller) : $(function) $(e) ] ;
+    }
+    return $(result) ;
+}
+
+
+rule reverse ( s * )
+{
+    local r ;
+    for local x in $(s)
+    {
+        r = $(x) $(r) ;
+    }
+    return $(r) ;
+}
+
+
+rule less ( a b )
+{
+    if $(a) < $(b)
+    {
+        return true ;
+    }
+}
+
+
+# Insertion-sort s using the BinaryPredicate ordered.
+#
+rule insertion-sort ( s * : ordered * )
+{
+    if ! $(ordered)
+    {
+        return [ SORT $(s) ] ;
+    }
+    else 
+    {           
+        local caller = [ CALLER_MODULE ] ;
+        ordered ?= sequence.less ;
+        local result = $(s[1]) ;
+        if $(ordered) = sequence.less
+        {
+            local head tail ;
+            for local x in $(s[2-])
+            {
+                head = ;
+                tail = $(result) ;
+                while $(tail) && ( $(tail[1]) < $(x) )
+                {
+                    head += $(tail[1]) ;
+                    tail = $(tail[2-]) ;
+                }
+                result = $(head) $(x) $(tail) ;
+            }
+        }
+        else
+        {
+            for local x in $(s[2-])
+            {
+                local head tail ;
+                tail = $(result) ;
+                while $(tail) && [ modules.call-in $(caller) : $(ordered) $(tail[1]) $(x) ]
+                {
+                    head += $(tail[1]) ;
+                    tail = $(tail[2-]) ;
+                }
+                result = $(head) $(x) $(tail) ;
+            }
+        }
+        
+        return $(result) ;
+    }    
+}
+
+
+# Merge two ordered sequences using the BinaryPredicate ordered.
+#
+rule merge ( s1 * : s2 * : ordered * )
+{
+    ordered ?= sequence.less ;
+    local result__ ;
+    local caller = [ CALLER_MODULE ] ;
+
+    while $(s1) && $(s2)
+    {
+        if [ modules.call-in $(caller) : $(ordered) $(s1[1]) $(s2[1]) ]
+        {
+            result__ += $(s1[1]) ;
+            s1 = $(s1[2-]) ;
+        }
+        else if [ modules.call-in $(caller) : $(ordered) $(s2[1]) $(s1[1]) ]
+        {
+            result__ += $(s2[1]) ;
+            s2 = $(s2[2-]) ;
+        }
+        else 
+        {           
+            s2 = $(s2[2-]) ;
+        }
+        
+    }
+    result__ += $(s1) ;
+    result__ += $(s2) ;
+
+    return $(result__) ;
+}
+
+
+# Join the elements of s into one long string. If joint is supplied, it is used
+# as a separator.
+#
+rule join ( s * : joint ? )
+{
+    joint ?= "" ;
+    return $(s:J=$(joint)) ;
+}
+
+
+# Find the length of any sequence.
+#
+rule length ( s * )
+{
+    local result = 0 ;
+    for local i in $(s)
+    {
+        result = [ CALC $(result) + 1 ] ;
+    }
+    return $(result) ;
+}
+
+
+rule unique ( list * : stable ? )
+{
+    local result ;
+    local prev ;
+    if $(stable)
+    {
+        for local f in $(list)
+        {
+            if ! $(f) in $(result)
+            {
+                result += $(f) ;
+            }
+        }
+    }
+    else
+    {
+        for local i in [ SORT $(list) ]
+        {
+            if $(i) != $(prev)
+            {
+                result += $(i) ;
+            }
+            prev = $(i) ;
+        }
+    }
+    return $(result) ;
+}
+
+
+# Returns the maximum number in 'elements'. Uses 'ordered' for comparisons or
+# 'numbers.less' if none is provided.
+#
+rule max-element ( elements + : ordered ? )
+{
+    ordered ?= numbers.less ;
+
+    local max = $(elements[1]) ;
+    for local e in $(elements[2-]) 
+    {
+        if [ $(ordered) $(max) $(e) ] 
+        {
+            max = $(e) ;
+        }
+    }
+    return $(max) ;           
+}
+
+
+# Returns all of 'elements' for which corresponding element in parallel list
+# 'rank' is equal to the maximum value in 'rank'.
+#
+rule select-highest-ranked ( elements * : ranks * )
+{
+    if $(elements)
+    {        
+        local max-rank = [ max-element $(ranks) ] ;
+        local result ;
+        while $(elements) 
+        {
+            if $(ranks[1]) = $(max-rank)
+            {
+                result += $(elements[1]) ;
+            }
+            elements = $(elements[2-]) ;
+            ranks = $(ranks[2-]) ;
+        }
+        return $(result) ;
+    }    
+}
+NATIVE_RULE sequence : select-highest-ranked ;
+
+
+rule __test__ ( )
+{
+    # Use a unique module so we can test the use of local rules.
+    module sequence.__test__
+    {
+        import assert ;
+        import sequence ;
+        
+        local rule is-even ( n )
+        {
+            if $(n) in 0 2 4 6 8
+            {
+                return true ;
+            }
+        }
+
+        assert.result 4 6 4 2 8 : sequence.filter is-even : 1 4 6 3 4 7 2 3 8 ;
+
+        # Test that argument binding works.
+        local rule is-equal-test ( x y )
+        {
+            if $(x) = $(y)
+            {
+                return true ;
+            }
+        }
+
+        assert.result 3 3 3 : sequence.filter is-equal-test 3 : 1 2 3 4 3 5 3 5 7 ;
+
+        local rule append-x ( n )
+        {
+            return $(n)x ;
+        }
+
+        assert.result 1x 2x 3x : sequence.transform append-x : 1 2 3 ;
+
+        local rule repeat2 ( x )
+        {
+            return $(x) $(x) ;
+        }
+
+        assert.result 1 1 2 2 3 3 : sequence.transform repeat2 : 1 2 3 ;
+
+        local rule test-greater ( a b )
+        {
+            if $(a) > $(b)
+            {
+                return true ;
+            }
+        }
+        assert.result 1 2 3 4 5 6 7 8 9 : sequence.insertion-sort 9 6 5 3 8 7 1 2 4 ;
+        assert.result 9 8 7 6 5 4 3 2 1 : sequence.insertion-sort 9 6 5 3 8 7 1 2 4 : test-greater ;
+        assert.result 1 2 3 4 5 6 :  sequence.merge 1 3 5 : 2 4 6 ;
+        assert.result 6 5 4 3 2 1 :  sequence.merge 5 3 1 : 6 4 2 : test-greater ;
+        assert.result 1 2 3 : sequence.merge 1 2 3 : ;
+        assert.result 1 : sequence.merge 1 : 1 ;
+
+        assert.result foo-bar-baz : sequence.join foo bar baz : - ;
+        assert.result substandard : sequence.join sub stan dard ;
+        assert.result 3.0.1 : sequence.join 3.0.1 : - ;
+
+        assert.result 0 : sequence.length ;
+        assert.result 3 : sequence.length a b c ;
+        assert.result 17 : sequence.length 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 ;
+
+        assert.result 1 : sequence.length a ;
+        assert.result 10 : sequence.length a b c d e f g h i j ;
+        assert.result 11 : sequence.length a b c d e f g h i j k ;
+        assert.result 12 : sequence.length a b c d e f g h i j k l ;
+
+        local p2 = x ;
+        for local i in 1 2 3 4 5 6 7 8
+        {
+            p2 = $(p2) $(p2) ;
+        }
+        assert.result 256 : sequence.length $(p2) ;
+
+        assert.result 1 2 3 4 5 : sequence.unique 1 2 3 2 4 3 3 5 5 5 ;
+                
+        assert.result 5 : sequence.max-element 1 3 5 0 4 ;
+        
+        assert.result e-3 h-3 : sequence.select-highest-ranked e-1 e-3 h-3 m-2 : 1 3 3 2 ;
+        
+        assert.result 7 6 5 4 3 2 1 : sequence.reverse 1 2 3 4 5 6 7 ;
+    }
+}

mosesdecoder/jam-files/boost-build/util/set.jam

@@ -0,0 +1,93 @@
+# Copyright 2001, 2002 Dave Abrahams
+# Copyright 2003 Vladimir Prus
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+class set
+{
+    rule __init__ ( )
+    {
+    }  
+    
+    rule add ( elements * )
+    {
+        for local e in $(elements)
+        {
+            if ! $($(e))
+            {
+                $(e) = 1 ;
+                self.result += $(e) ;
+            }            
+        }        
+    }    
+    
+    rule contains ( element )
+    {
+        return $($(element)) ;
+    }
+    
+    rule list ( )
+    {
+        return $(self.result) ;
+    }            
+}
+
+
+
+# Returns the elements of set1 that are not in set2.
+#
+rule difference ( set1 * : set2 * )
+{
+    local result = ;
+    for local element in $(set1)
+    {
+        if ! ( $(element) in $(set2) )
+        {
+            result += $(element) ;
+        }
+    }
+    return $(result) ;
+}
+
+NATIVE_RULE set : difference ;
+
+
+# Removes all the items appearing in both set1 & set2.
+#
+rule intersection ( set1 * : set2 * )
+{
+    local result ;
+    for local v in $(set1)
+    {
+        if $(v) in $(set2)
+        {
+            result += $(v) ;
+        }
+    }
+    return $(result) ;
+}
+
+
+# Returns whether set1 & set2 contain the same elements. Note that this ignores
+# any element ordering differences as well as any element duplication.
+#
+rule equal ( set1 * : set2 * )
+{
+    if $(set1) in $(set2) && ( $(set2) in $(set1) )
+    {
+        return true ;
+    }
+}
+
+
+rule __test__ ( )
+{
+    import assert ;
+
+    assert.result 0 1 4 6 8 9 : difference   0 1 2 3 4 5 6 7 8 9 : 2 3 5 7 ;
+    assert.result 2 5 7       : intersection 0 1 2   4 5 6 7 8 9 : 2 3 5 7 ;
+
+    assert.true  equal         :         ;
+    assert.true  equal 1 1 2 3 : 3 2 2 1 ;
+    assert.false equal 2 3     : 3 2 2 1 ;
+}

mosesdecoder/jam-files/boost-build/util/string.jam

@@ -0,0 +1,189 @@
+# Copyright 2002 Dave Abrahams
+# Copyright 2002, 2003 Rene Rivera
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+import regex ;
+
+
+# Characters considered whitespace, as a list.
+.whitespace-chars = " " "	" "
+" ;
+
+# Characters considered whitespace, as a single string.
+.whitespace = $(.whitespace-chars:J="") ;
+
+
+# Returns the canonical set of whitespace characters, as a list.
+#
+rule whitespace-chars ( )
+{
+    return $(.whitespace-chars) ;
+}
+
+
+# Returns the canonical set of whitespace characters, as a single string.
+#
+rule whitespace ( )
+{
+    return $(.whitespace) ;
+}
+
+
+# Splits the given string into a list of strings composed of each character of
+# the string in sequence.
+#
+rule chars (
+    string  # The string to split.
+    )
+{
+    local result ;
+    while $(string)
+    {
+        local s = [ MATCH (.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.*) : $(string) ] ;
+        string = $(s[9]) ;
+        result += $(s[1-8]) ;
+    }
+
+    # Trim off empty strings.
+    while $(result[1]) && ! $(result[-1])
+    {
+        result = $(result[1--2]) ;
+    }
+
+    return $(result) ;
+}
+
+
+# Apply a set of standard transformations to string to produce an abbreviation
+# no more than 5 characters long.
+#
+rule abbreviate ( string )
+{
+    local r = $(.abbreviated-$(string)) ;
+    if $(r)
+    {
+        return $(r) ;
+    }
+    # Anything less than 4 characters gets no abbreviation.
+    else if ! [ MATCH (....) : $(string) ]
+    {
+        .abbreviated-$(string) = $(string) ;
+        return $(string) ;
+    }
+    else
+    {
+        # Separate the initial letter in case it's a vowel.
+        local s1 = [ MATCH ^(.)(.*) : $(string) ] ;
+
+        # Drop trailing "ing".
+        local s2 = [ MATCH ^(.*)ing$ : $(s1[2]) ] ;
+        s2 ?= $(s1[2]) ;
+
+        # Reduce all doubled characters to one.
+        local last = "" ;
+        for local c in [ chars $(s2) ]
+        {
+            if $(c) != $(last)
+            {
+                r += $(c) ;
+                last = $(c) ;
+            }
+        }
+        s2 = $(r:J="") ;
+
+        # Chop all vowels out of the remainder.
+        s2 = [ regex.replace $(s2) [AEIOUaeiou] "" ] ;
+
+        # Shorten remaining consonants to 4 characters.
+        s2 = [ MATCH ^(.?.?.?.?) : $(s2) ] ;
+
+        # Glue the initial character back on to the front.
+        s2 = $(s1[1])$(s2) ;
+
+        .abbreviated-$(string) = $(s2) ;
+        return $(s2) ;
+    }
+}
+
+
+# Concatenates the given strings, inserting the given separator between each
+# string.
+#
+rule join (
+    strings *  # The strings to join.
+    : separator ?  # The optional separator.
+    )
+{
+    separator ?= "" ;
+    return $(strings:J=$(separator)) ;
+}
+
+
+# Split a string into whitespace separated words.
+#
+rule words (
+    string  # The string to split.
+    : whitespace *  # Optional, characters to consider as whitespace.
+    )
+{
+    whitespace = $(whitespace:J="") ;
+    whitespace ?= $(.whitespace) ;
+    local w = ;
+    while $(string)
+    {
+        string = [ MATCH "^[$(whitespace)]*([^$(whitespace)]*)(.*)" : $(string) ] ;
+        if $(string[1]) && $(string[1]) != ""
+        {
+            w += $(string[1]) ;
+        }
+        string = $(string[2]) ;
+    }
+    return $(w) ;
+}
+
+
+# Check that the given string is composed entirely of whitespace.
+#
+rule is-whitespace (
+    string ?  # The string to test.
+    )
+{
+    if ! $(string) { return true ; }
+    else if $(string) = "" { return true ; }
+    else if [ MATCH "^([$(.whitespace)]+)$" : $(string) ] { return true ; }
+    else { return ; }
+}
+
+rule __test__ ( )
+{
+    import assert ;
+    assert.result a b c : chars abc ;
+
+    assert.result rntm : abbreviate runtime ;
+    assert.result ovrld : abbreviate overload ;
+    assert.result dbg : abbreviate debugging ;
+    assert.result async : abbreviate asynchronous ;
+    assert.result pop : abbreviate pop ;
+    assert.result aaa : abbreviate aaa ;
+    assert.result qck : abbreviate quack ;
+    assert.result sttc : abbreviate static ;
+
+    # Check boundary cases.
+    assert.result a : chars a ;
+    assert.result : chars "" ;
+    assert.result a b c d e f g h : chars abcdefgh ;
+    assert.result a b c d e f g h i : chars abcdefghi ;
+    assert.result a b c d e f g h i j : chars abcdefghij ;
+    assert.result a b c d e f g h i j k : chars abcdefghijk ;
+
+    assert.result a//b/c/d : join a "" b c d : / ;
+    assert.result abcd : join  a "" b c d ;
+
+    assert.result a b c : words "a b	c" ;
+
+    assert.true is-whitespace "     	" ;
+    assert.false is-whitespace "  a b c	" ;
+    assert.true is-whitespace "" ;
+    assert.true is-whitespace ;
+}

mosesdecoder/jam-files/boost-build/util/utility.jam

@@ -0,0 +1,235 @@
+# Copyright 2001, 2002 Dave Abrahams
+# Copyright 2002, 2003, 2004, 2005 Vladimir Prus
+# Copyright 2008 Jurko Gospodnetic
+# Distributed under the Boost Software License, Version 1.0.
+# (See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)
+
+import "class" : is-instance ;
+import errors ;
+
+
+# For all elements of 'list' which do not already have 'suffix', add 'suffix'.
+#
+rule apply-default-suffix ( suffix : list * )
+{
+    local result ;
+    for local i in $(list)
+    {
+        if $(i:S) = $(suffix)
+        {
+            result += $(i) ;
+        }
+        else
+        {
+            result += $(i)$(suffix) ;
+        }
+    }
+    return $(result) ;
+}
+
+
+# If 'name' contains a dot, returns the part before the last dot. If 'name'
+# contains no dot, returns it unmodified.
+#
+rule basename ( name )
+{
+    if $(name:S)
+    {
+        name = $(name:B) ;
+    }
+    return $(name) ;
+}
+
+
+# Return the file of the caller of the rule that called caller-file.
+#
+rule caller-file ( )
+{
+    local bt = [ BACKTRACE ] ;
+    return $(bt[9]) ;
+}
+
+
+# Tests if 'a' is equal to 'b'. If 'a' is a class instance, calls its 'equal'
+# method. Uses ordinary jam's comparison otherwise.
+#
+rule equal ( a b )
+{
+    if [ is-instance $(a) ]
+    {
+        return [ $(a).equal $(b) ] ;
+    }
+    else
+    {
+        if $(a) = $(b)
+        {
+            return true ;
+        }
+    }
+}
+
+
+# Tests if 'a' is less than 'b'. If 'a' is a class instance, calls its 'less'
+# method. Uses ordinary jam's comparison otherwise.
+#
+rule less ( a b )
+{
+    if [ is-instance $(a) ]
+    {
+        return [ $(a).less $(b) ] ;
+    }
+    else
+    {
+        if $(a) < $(b)
+        {
+            return true ;
+        }
+    }
+}
+
+
+# Returns the textual representation of argument. If it is a class instance,
+# class its 'str' method. Otherwise, returns the argument.
+#
+rule str ( value )
+{
+    if [ is-instance $(value) ]
+    {
+        return [ $(value).str ] ;
+    }
+    else
+    {
+        return $(value) ;
+    }
+}
+
+
+# Accepts a list of gristed values and returns them ungristed. Reports an error
+# in case any of the passed parameters is not gristed, i.e. surrounded in angle
+# brackets < and >.
+#
+rule ungrist ( names * )
+{
+    local result ;
+    for local name in $(names)
+    {
+        local stripped = [ MATCH ^<(.*)>$ : $(name) ] ;
+        if ! $(stripped)
+        {
+            errors.error "in ungrist $(names) : $(name) is not of the form <.*>" ;
+        }
+        result += $(stripped) ;
+    }
+    return $(result) ;
+}
+
+
+# If the passed value is quoted, unquotes it. Otherwise returns the value
+# unchanged.
+#
+rule unquote ( value ? )
+{
+    local match-result = [ MATCH ^(\")(.*)(\")$ : $(value) ] ;
+    if $(match-result)
+    {
+        return $(match-result[2]) ;
+    }
+    else
+    {
+        return $(value) ;
+    }
+}
+
+
+rule __test__ ( )
+{
+    import assert ;
+    import "class" : new ;
+    import errors : try catch ;
+
+    assert.result 123 : str 123 ;
+
+    class test-class__
+    {
+        rule __init__ (   ) {                            }
+        rule str      (   ) { return "str-test-class"  ; }
+        rule less     ( a ) { return "yes, of course!" ; }
+        rule equal    ( a ) { return "not sure"        ; }
+    }
+
+    assert.result "str-test-class" : str [ new test-class__ ] ;
+    assert.true less 1 2 ;
+    assert.false less 2 1 ;
+    assert.result "yes, of course!" : less [ new test-class__ ] 1 ;
+    assert.true equal 1 1 ;
+    assert.false equal 1 2 ;
+    assert.result "not sure" : equal [ new test-class__ ] 1 ;
+
+    assert.result foo.lib foo.lib : apply-default-suffix .lib : foo.lib foo.lib
+        ;
+
+    assert.result foo : basename foo ;
+    assert.result foo : basename foo.so ;
+    assert.result foo.so : basename foo.so.1 ;
+
+    assert.result         : unquote ;
+    assert.result ""      : unquote "" ;
+    assert.result foo     : unquote foo ;
+    assert.result \"foo   : unquote \"foo ;
+    assert.result foo\"   : unquote foo\" ;
+    assert.result foo     : unquote \"foo\" ;
+    assert.result \"foo\" : unquote \"\"foo\"\" ;
+
+    assert.result         : ungrist ;
+    assert.result foo     : ungrist <foo> ;
+    assert.result <foo>   : ungrist <<foo>> ;
+    assert.result foo bar : ungrist <foo> <bar> ;
+
+    try ;
+    {
+        ungrist "" ;
+    }
+    catch "in ungrist  :  is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist <> ;
+    }
+    catch "in ungrist <> : <> is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist foo ;
+    }
+    catch "in ungrist foo : foo is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist <foo ;
+    }
+    catch "in ungrist <foo : <foo is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist foo> ;
+    }
+    catch "in ungrist foo> : foo> is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist foo bar ;
+    }
+    catch "in ungrist foo : foo is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist foo <bar> ;
+    }
+    catch "in ungrist foo : foo is not of the form <.*>" ;
+
+    try ;
+    {
+        ungrist <foo> bar ;
+    }
+    catch "in ungrist bar : bar is not of the form <.*>" ;
+}

mosesdecoder/moses/TranslationModel/CYKPlusParser/ChartRuleLookupManagerCYKPlus.h

@@ -0,0 +1,67 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2012 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#pragma once
+#ifndef moses_ChartRuleLookupManagerCYKPlus_h
+#define moses_ChartRuleLookupManagerCYKPlus_h
+
+#include "moses/ChartRuleLookupManager.h"
+#include "moses/StackVec.h"
+
+namespace Moses
+{
+
+class DottedRule;
+class TargetPhraseCollection;
+class Range;
+
+/** @todo what is this?
+ */
+class ChartRuleLookupManagerCYKPlus : public ChartRuleLookupManager
+{
+public:
+  ChartRuleLookupManagerCYKPlus(const ChartParser &parser,
+                                const ChartCellCollectionBase &cellColl)
+    : ChartRuleLookupManager(parser, cellColl) {}
+
+protected:
+  void AddCompletedRule(
+    const DottedRule &dottedRule,
+    const TargetPhraseCollection &tpc,
+    const Range &range,
+    ChartParserCallback &outColl);
+
+  StackVec m_stackVec;
+};
+
+// struct that caches cellLabel, its end position and score for quicker lookup
+struct ChartCellCache {
+  ChartCellCache(size_t endPos, const ChartCellLabel* cellLabel, float score)
+    : endPos(endPos)
+    , cellLabel(cellLabel)
+    , score(score) {}
+
+  size_t endPos;
+  const ChartCellLabel* cellLabel;
+  float score;
+};
+
+}  // namespace Moses
+
+#endif

mosesdecoder/moses/TranslationModel/CYKPlusParser/ChartRuleLookupManagerExample.cpp

@@ -0,0 +1,107 @@
+/***********************************************************************
+  Moses - factored phrase-based language decoder
+  Copyright (C) 2011 University of Edinburgh
+
+  This library is free software; you can redistribute it and/or
+  modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either
+  version 2.1 of the License, or (at your option) any later version.
+
+  This library is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+
+#include <iostream>
+#include "ChartRuleLookupManagerExample.h"
+#include "DotChartInMemory.h"
+
+#include "moses/Util.h"
+#include "moses/ChartParser.h"
+#include "moses/InputType.h"
+#include "moses/ChartParserCallback.h"
+#include "moses/StaticData.h"
+#include "moses/NonTerminal.h"
+#include "moses/ChartCellCollection.h"
+#include "moses/TranslationModel/PhraseDictionaryMemory.h"
+#include "moses/TranslationModel/ExamplePT.h"
+
+using namespace std;
+
+namespace Moses
+{
+
+ChartRuleLookupManagerExample::ChartRuleLookupManagerExample(
+  const ChartParser &parser,
+  const ChartCellCollectionBase &cellColl,
+  const ExamplePT &skeletonPt)
+  : ChartRuleLookupManager(parser, cellColl)
+  , m_skeletonPT(skeletonPt)
+{
+  cerr << "starting ChartRuleLookupManagerExample" << endl;
+}
+
+ChartRuleLookupManagerExample::~ChartRuleLookupManagerExample()
+{
+  // RemoveAllInColl(m_tpColl);
+}
+
+void ChartRuleLookupManagerExample::GetChartRuleCollection(
+  const InputPath &inputPath,
+  size_t last,
+  ChartParserCallback &outColl)
+{
+  //m_tpColl.push_back(TargetPhraseCollection());
+  //TargetPhraseCollection &tpColl = m_tpColl.back();
+  TargetPhraseCollection::shared_ptr tpColl(new TargetPhraseCollection);
+  m_tpColl.push_back(tpColl);
+
+  const Range &range = inputPath.GetWordsRange();
+
+  if (range.GetNumWordsCovered() == 1) {
+    const ChartCellLabel &sourceWordLabel = GetSourceAt(range.GetStartPos());
+    const Word &sourceWord = sourceWordLabel.GetLabel();
+    TargetPhrase *tp = CreateTargetPhrase(sourceWord);
+    tpColl->Add(tp);
+  }
+
+  outColl.Add(*tpColl, m_stackVec, range);
+}
+
+TargetPhrase *
+ChartRuleLookupManagerExample::
+CreateTargetPhrase(const Word &sourceWord) const
+{
+  // create a target phrase from the 1st word of the source, prefix with 'ChartManagerExample:'
+  string str = sourceWord.GetFactor(0)->GetString().as_string();
+  str = "ChartManagerExample:" + str;
+
+  TargetPhrase *tp = new TargetPhrase(&m_skeletonPT);
+  Word &word = tp->AddWord();
+  word.CreateFromString(Output, m_skeletonPT.GetOutput(), str, false);
+
+  // create hiero-style non-terminal for LHS
+  Word *targetLHS = new Word();
+  targetLHS->CreateFromString(Output, m_skeletonPT.GetOutput(), "X", true);
+  tp->SetTargetLHS(targetLHS);
+
+  // word alignement
+  tp->SetAlignmentInfo("0-0");
+
+  // score for this phrase table
+  vector<float> scores(m_skeletonPT.GetNumScoreComponents(), 1.3);
+  tp->GetScoreBreakdown().PlusEquals(&m_skeletonPT, scores);
+
+  // score of all other ff when this rule is being loaded
+  Phrase sourcePhrase;
+  sourcePhrase.AddWord(sourceWord);
+  //tp->Evaluate(sourcePhrase, m_skeletonPT.GetFeaturesToApply());
+
+  return tp;
+}
+}  // namespace Moses

mosesdecoder/moses/TranslationModel/CYKPlusParser/ChartRuleLookupManagerExample.h

@@ -0,0 +1,57 @@
+/***********************************************************************
+  Moses - factored phrase-based language decoder
+  Copyright (C) 2011 University of Edinburgh
+
+  This library is free software; you can redistribute it and/or
+  modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either
+  version 2.1 of the License, or (at your option) any later version.
+
+  This library is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+
+#pragma once
+
+#include <vector>
+#include "moses/ChartRuleLookupManager.h"
+#include "moses/StackVec.h"
+
+namespace Moses
+{
+class TargetPhraseCollection;
+class ChartParserCallback;
+class DottedRuleColl;
+class Range;
+class ExamplePT;
+
+class ChartRuleLookupManagerExample  : public ChartRuleLookupManager
+{
+public:
+  ChartRuleLookupManagerExample(const ChartParser &parser,
+                                const ChartCellCollectionBase &cellColl,
+                                const ExamplePT &skeletonPt);
+
+  ~ChartRuleLookupManagerExample();
+
+  virtual void GetChartRuleCollection(
+    const InputPath &inputPath,
+    size_t last,
+    ChartParserCallback &outColl);
+
+private:
+  TargetPhrase *CreateTargetPhrase(const Word &sourceWord) const;
+
+  StackVec m_stackVec;
+  std::vector<TargetPhraseCollection::shared_ptr > m_tpColl;
+  const ExamplePT &m_skeletonPT;
+};
+
+}  // namespace Moses
+

mosesdecoder/moses/TranslationModel/CYKPlusParser/ChartRuleLookupManagerMemory.cpp

@@ -0,0 +1,270 @@
+/***********************************************************************
+  Moses - factored phrase-based language decoder
+  Copyright (C) 2011 University of Edinburgh
+
+  This library is free software; you can redistribute it and/or
+  modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either
+  version 2.1 of the License, or (at your option) any later version.
+
+  This library is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+
+#include <iostream>
+#include "ChartRuleLookupManagerMemory.h"
+
+#include "moses/ChartParser.h"
+#include "moses/InputType.h"
+#include "moses/Terminal.h"
+#include "moses/ChartParserCallback.h"
+#include "moses/StaticData.h"
+#include "moses/NonTerminal.h"
+#include "moses/ChartCellCollection.h"
+#include "moses/FactorCollection.h"
+#include "moses/TranslationModel/PhraseDictionaryMemory.h"
+
+using namespace std;
+
+namespace Moses
+{
+
+ChartRuleLookupManagerMemory::ChartRuleLookupManagerMemory(
+  const ChartParser &parser,
+  const ChartCellCollectionBase &cellColl,
+  const PhraseDictionaryMemory &ruleTable)
+  : ChartRuleLookupManagerCYKPlus(parser, cellColl)
+  , m_ruleTable(ruleTable)
+  , m_softMatchingMap(StaticData::Instance().GetSoftMatches())
+{
+
+  size_t sourceSize = parser.GetSize();
+  size_t ruleLimit  = parser.options()->syntax.rule_limit;
+  m_completedRules.resize(sourceSize, CompletedRuleCollection(ruleLimit));
+
+  m_isSoftMatching = !m_softMatchingMap.empty();
+}
+
+void ChartRuleLookupManagerMemory::GetChartRuleCollection(
+  const InputPath &inputPath,
+  size_t lastPos,
+  ChartParserCallback &outColl)
+{
+  const Range &range = inputPath.GetWordsRange();
+  size_t startPos = range.GetStartPos();
+  size_t absEndPos = range.GetEndPos();
+
+  m_lastPos = lastPos;
+  m_stackVec.clear();
+  m_stackScores.clear();
+  m_outColl = &outColl;
+  m_unaryPos = absEndPos-1; // rules ending in this position are unary and should not be added to collection
+
+  // create/update data structure to quickly look up all chart cells that match start position and label.
+  UpdateCompressedMatrix(startPos, absEndPos, lastPos);
+
+  const PhraseDictionaryNodeMemory &rootNode = m_ruleTable.GetRootNode();
+
+  // all rules starting with terminal
+  if (startPos == absEndPos) {
+    GetTerminalExtension(&rootNode, startPos);
+  }
+  // all rules starting with nonterminal
+  else if (absEndPos > startPos) {
+    GetNonTerminalExtension(&rootNode, startPos);
+  }
+
+  // copy temporarily stored rules to out collection
+  CompletedRuleCollection & rules = m_completedRules[absEndPos];
+  for (vector<CompletedRule*>::const_iterator iter = rules.begin(); iter != rules.end(); ++iter) {
+    outColl.Add((*iter)->GetTPC(), (*iter)->GetStackVector(), range);
+  }
+
+  rules.Clear();
+
+}
+
+// Create/update compressed matrix that stores all valid ChartCellLabels for a given start position and label.
+void ChartRuleLookupManagerMemory::UpdateCompressedMatrix(size_t startPos,
+    size_t origEndPos,
+    size_t lastPos)
+{
+
+  std::vector<size_t> endPosVec;
+  size_t numNonTerms = FactorCollection::Instance().GetNumNonTerminals();
+  m_compressedMatrixVec.resize(lastPos+1);
+
+  // we only need to update cell at [startPos, origEndPos-1] for initial lookup
+  if (startPos < origEndPos) {
+    endPosVec.push_back(origEndPos-1);
+  }
+
+  // update all cells starting from startPos+1 for lookup of rule extensions
+  else if (startPos == origEndPos) {
+    startPos++;
+    for (size_t endPos = startPos; endPos <= lastPos; endPos++) {
+      endPosVec.push_back(endPos);
+    }
+    //re-use data structure for cells with later start position, but remove chart cells that would break max-chart-span
+    for (size_t pos = startPos+1; pos <= lastPos; pos++) {
+      CompressedMatrix & cellMatrix = m_compressedMatrixVec[pos];
+      cellMatrix.resize(numNonTerms);
+      for (size_t i = 0; i < numNonTerms; i++) {
+        if (!cellMatrix[i].empty() && cellMatrix[i].back().endPos > lastPos) {
+          cellMatrix[i].pop_back();
+        }
+      }
+    }
+  }
+
+  if (startPos > lastPos) {
+    return;
+  }
+
+  // populate compressed matrix with all chart cells that start at current start position
+  CompressedMatrix & cellMatrix = m_compressedMatrixVec[startPos];
+  cellMatrix.clear();
+  cellMatrix.resize(numNonTerms);
+  for (std::vector<size_t>::iterator p = endPosVec.begin(); p != endPosVec.end(); ++p) {
+
+    size_t endPos = *p;
+    // target non-terminal labels for the span
+    const ChartCellLabelSet &targetNonTerms = GetTargetLabelSet(startPos, endPos);
+
+    if (targetNonTerms.GetSize() == 0) {
+      continue;
+    }
+
+#if !defined(UNLABELLED_SOURCE)
+    // source non-terminal labels for the span
+    const InputPath &inputPath = GetParser().GetInputPath(startPos, endPos);
+
+    // can this ever be true? Moses seems to pad the non-terminal set of the input with [X]
+    if (inputPath.GetNonTerminalSet().size() == 0) {
+      continue;
+    }
+#endif
+
+    for (size_t i = 0; i < numNonTerms; i++) {
+      const ChartCellLabel *cellLabel = targetNonTerms.Find(i);
+      if (cellLabel != NULL) {
+        float score = cellLabel->GetBestScore(m_outColl);
+        cellMatrix[i].push_back(ChartCellCache(endPos, cellLabel, score));
+      }
+    }
+  }
+}
+
+// if a (partial) rule matches, add it to list completed rules (if non-unary and non-empty), and try find expansions that have this partial rule as prefix.
+void ChartRuleLookupManagerMemory::AddAndExtend(
+  const PhraseDictionaryNodeMemory *node,
+  size_t endPos)
+{
+
+  TargetPhraseCollection::shared_ptr tpc = node->GetTargetPhraseCollection();
+  // add target phrase collection (except if rule is empty or a unary non-terminal rule)
+  if (!tpc->IsEmpty() && (m_stackVec.empty() || endPos != m_unaryPos)) {
+    m_completedRules[endPos].Add(*tpc, m_stackVec, m_stackScores, *m_outColl);
+  }
+
+  // get all further extensions of rule (until reaching end of sentence or max-chart-span)
+  if (endPos < m_lastPos) {
+    if (!node->GetTerminalMap().empty()) {
+      GetTerminalExtension(node, endPos+1);
+    }
+    if (!node->GetNonTerminalMap().empty()) {
+      GetNonTerminalExtension(node, endPos+1);
+    }
+  }
+}
+
+
+// search all possible terminal extensions of a partial rule (pointed at by node) at a given position
+// recursively try to expand partial rules into full rules up to m_lastPos.
+void ChartRuleLookupManagerMemory::GetTerminalExtension(
+  const PhraseDictionaryNodeMemory *node,
+  size_t pos)
+{
+
+  const Word &sourceWord = GetSourceAt(pos).GetLabel();
+  const PhraseDictionaryNodeMemory::TerminalMap & terminals = node->GetTerminalMap();
+
+  // if node has small number of terminal edges, test word equality for each.
+  if (terminals.size() < 5) {
+    for (PhraseDictionaryNodeMemory::TerminalMap::const_iterator iter = terminals.begin(); iter != terminals.end(); ++iter) {
+      const Word & word = iter->first;
+      if (TerminalEqualityPred()(word, sourceWord)) {
+        const PhraseDictionaryNodeMemory *child = & iter->second;
+        AddAndExtend(child, pos);
+        break;
+      }
+    }
+  }
+  // else, do hash lookup
+  else {
+    const PhraseDictionaryNodeMemory *child = node->GetChild(sourceWord);
+    if (child != NULL) {
+      AddAndExtend(child, pos);
+    }
+  }
+}
+
+// search all nonterminal possible nonterminal extensions of a partial rule (pointed at by node) for a variable span (starting from startPos).
+// recursively try to expand partial rules into full rules up to m_lastPos.
+void ChartRuleLookupManagerMemory::GetNonTerminalExtension(
+  const PhraseDictionaryNodeMemory *node,
+  size_t startPos)
+{
+
+  const CompressedMatrix &compressedMatrix = m_compressedMatrixVec[startPos];
+
+  // non-terminal labels in phrase dictionary node
+  const PhraseDictionaryNodeMemory::NonTerminalMap & nonTermMap = node->GetNonTerminalMap();
+
+  // make room for back pointer
+  m_stackVec.push_back(NULL);
+  m_stackScores.push_back(0);
+
+  // loop over possible expansions of the rule
+  PhraseDictionaryNodeMemory::NonTerminalMap::const_iterator p;
+  PhraseDictionaryNodeMemory::NonTerminalMap::const_iterator end = nonTermMap.end();
+  for (p = nonTermMap.begin(); p != end; ++p) {
+    // does it match possible source and target non-terminals?
+#if defined(UNLABELLED_SOURCE)
+    const Word &targetNonTerm = p->first;
+#else
+    const Word &targetNonTerm = p->first.second;
+#endif
+    const PhraseDictionaryNodeMemory *child = &p->second;
+    //soft matching of NTs
+    if (m_isSoftMatching && !m_softMatchingMap[targetNonTerm[0]->GetId()].empty()) {
+      const std::vector<Word>& softMatches = m_softMatchingMap[targetNonTerm[0]->GetId()];
+      for (std::vector<Word>::const_iterator softMatch = softMatches.begin(); softMatch != softMatches.end(); ++softMatch) {
+        const CompressedColumn &matches = compressedMatrix[(*softMatch)[0]->GetId()];
+        for (CompressedColumn::const_iterator match = matches.begin(); match != matches.end(); ++match) {
+          m_stackVec.back() = match->cellLabel;
+          m_stackScores.back() = match->score;
+          AddAndExtend(child, match->endPos);
+        }
+      }
+    } // end of soft matches lookup
+
+    const CompressedColumn &matches = compressedMatrix[targetNonTerm[0]->GetId()];
+    for (CompressedColumn::const_iterator match = matches.begin(); match != matches.end(); ++match) {
+      m_stackVec.back() = match->cellLabel;
+      m_stackScores.back() = match->score;
+      AddAndExtend(child, match->endPos);
+    }
+  }
+  // remove last back pointer
+  m_stackVec.pop_back();
+  m_stackScores.pop_back();
+}
+
+}  // namespace Moses

mosesdecoder/moses/TranslationModel/CYKPlusParser/CompletedRuleCollection.cpp

@@ -0,0 +1,129 @@
+/***********************************************************************
+  Moses - factored phrase-based language decoder
+  Copyright (C) 2014 University of Edinburgh
+
+  This library is free software; you can redistribute it and/or
+  modify it under the terms of the GNU Lesser General Public
+  License as published by the Free Software Foundation; either
+  version 2.1 of the License, or (at your option) any later version.
+
+  This library is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+  Lesser General Public License for more details.
+
+  You should have received a copy of the GNU Lesser General Public
+  License along with this library; if not, write to the Free Software
+  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+
+#include <iostream>
+#include "CompletedRuleCollection.h"
+
+#include "moses/StaticData.h"
+
+using namespace std;
+
+namespace Moses
+{
+
+CompletedRuleCollection::CompletedRuleCollection(size_t rule_limit)
+  : m_ruleLimit(rule_limit)
+{
+  m_scoreThreshold = numeric_limits<float>::infinity();
+}
+
+CompletedRuleCollection::~CompletedRuleCollection()
+{
+  Clear();
+}
+
+// copies some functionality (pruning) from ChartTranslationOptionList::Add
+void CompletedRuleCollection::Add(const TargetPhraseCollection &tpc,
+                                  const StackVec &stackVec,
+                                  const ChartParserCallback &outColl)
+{
+  if (tpc.IsEmpty()) {
+    return;
+  }
+
+  const TargetPhrase &targetPhrase = **(tpc.begin());
+  float score = targetPhrase.GetFutureScore();
+  for (StackVec::const_iterator p = stackVec.begin(); p != stackVec.end(); ++p) {
+    float stackScore = (*p)->GetBestScore(&outColl);
+    score += stackScore;
+  }
+
+  // If the rule limit has already been reached then don't add the option
+  // unless it is better than at least one existing option.
+  if (m_ruleLimit && m_collection.size() > m_ruleLimit && score < m_scoreThreshold) {
+    return;
+  }
+
+  CompletedRule *completedRule = new CompletedRule(tpc, stackVec, score);
+  m_collection.push_back(completedRule);
+
+  // If the rule limit hasn't been exceeded then update the threshold.
+  if (!m_ruleLimit || m_collection.size() <= m_ruleLimit) {
+    m_scoreThreshold = (score < m_scoreThreshold) ? score : m_scoreThreshold;
+  }
+
+  // Prune if bursting
+  if (m_ruleLimit && m_collection.size() == m_ruleLimit * 2) {
+    NTH_ELEMENT4(m_collection.begin(),
+                 m_collection.begin() + m_ruleLimit - 1,
+                 m_collection.end(),
+                 CompletedRuleOrdered());
+    m_scoreThreshold = m_collection[m_ruleLimit-1]->GetScoreEstimate();
+    for (size_t i = 0 + m_ruleLimit; i < m_collection.size(); i++) {
+      delete m_collection[i];
+
+    }
+    m_collection.resize(m_ruleLimit);
+  }
+}
+
+
+// copies some functionality (pruning) from ChartTranslationOptionList::Add
+void CompletedRuleCollection::Add(const TargetPhraseCollection &tpc,
+                                  const StackVec &stackVec,
+                                  const std::vector<float> &stackScores,
+                                  const ChartParserCallback &outColl)
+{
+  if (tpc.IsEmpty()) {
+    return;
+  }
+
+  const TargetPhrase &targetPhrase = **(tpc.begin());
+  float score = std::accumulate(stackScores.begin(), stackScores.end(), targetPhrase.GetFutureScore());
+
+  // If the rule limit has already been reached then don't add the option
+  // unless it is better than at least one existing option.
+  if (m_collection.size() > m_ruleLimit && score < m_scoreThreshold) {
+    return;
+  }
+
+  CompletedRule *completedRule = new CompletedRule(tpc, stackVec, score);
+  m_collection.push_back(completedRule);
+
+  // If the rule limit hasn't been exceeded then update the threshold.
+  if (m_collection.size() <= m_ruleLimit) {
+    m_scoreThreshold = (score < m_scoreThreshold) ? score : m_scoreThreshold;
+  }
+
+  // Prune if bursting
+  if (m_collection.size() == m_ruleLimit * 2) {
+    NTH_ELEMENT4(m_collection.begin(),
+                 m_collection.begin() + m_ruleLimit - 1,
+                 m_collection.end(),
+                 CompletedRuleOrdered());
+    m_scoreThreshold = m_collection[m_ruleLimit-1]->GetScoreEstimate();
+    for (size_t i = 0 + m_ruleLimit; i < m_collection.size(); i++) {
+      delete m_collection[i];
+
+    }
+    m_collection.resize(m_ruleLimit);
+  }
+}
+
+}

mosesdecoder/moses/TranslationModel/CYKPlusParser/DotChartInMemory.cpp

@@ -0,0 +1,40 @@
+/***********************************************************************
+ Moses - statistical machine translation system
+ Copyright (C) 2006-2011 University of Edinburgh
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#include "DotChartInMemory.h"
+
+#include "moses/Util.h"
+
+#include <algorithm>
+
+namespace Moses
+{
+
+DottedRuleColl::~DottedRuleColl()
+{
+#ifdef USE_BOOST_POOL
+  // Do nothing.  DottedRule objects are stored in object pools owned by
+  // the sentence-specific ChartRuleLookupManagers.
+#else
+  std::for_each(m_coll.begin(), m_coll.end(),
+                RemoveAllInColl<CollType::value_type>);
+#endif
+}
+
+}

mosesdecoder/moses/TranslationModel/CYKPlusParser/DotChartOnDisk.cpp

@@ -0,0 +1,57 @@
+// $Id$
+/***********************************************************************
+ Moses - factored phrase-based language decoder
+ Copyright (C) 2010 Hieu Hoang
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+#include <algorithm>
+#include "DotChartOnDisk.h"
+#include "moses/Util.h"
+#include "OnDiskPt/PhraseNode.h"
+
+using namespace std;
+
+namespace Moses
+{
+DottedRuleStackOnDisk::DottedRuleStackOnDisk(size_t size)
+  :m_coll(size)
+{
+  for (size_t ind = 0; ind < size; ++ind) {
+    m_coll[ind] = new DottedRuleCollOnDisk();
+  }
+}
+
+DottedRuleStackOnDisk::~DottedRuleStackOnDisk()
+{
+  RemoveAllInColl(m_coll);
+  RemoveAllInColl(m_savedNode);
+}
+
+class SavedNodesOderer
+{
+public:
+  bool operator()(const SavedNodeOnDisk* a, const SavedNodeOnDisk* b) const {
+    bool ret = a->GetDottedRule().GetLastNode().GetCount(0) > b->GetDottedRule().GetLastNode().GetCount(0);
+    return ret;
+  }
+};
+
+void DottedRuleStackOnDisk::SortSavedNodes()
+{
+  sort(m_savedNode.begin(), m_savedNode.end(), SavedNodesOderer());
+}
+
+};

mosesdecoder/moses/TranslationModel/CYKPlusParser/DotChartOnDisk.h

@@ -0,0 +1,186 @@
+// $Id$
+/***********************************************************************
+ Moses - factored phrase-based language decoder
+ Copyright (C) 2010 Hieu Hoang
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ ***********************************************************************/
+#pragma once
+
+#include <vector>
+#include "DotChart.h"
+
+namespace OnDiskPt
+{
+class PhraseNode;
+}
+
+namespace Moses
+{
+
+/** @todo what is this?
+ */
+class DottedRuleOnDisk : public DottedRule
+{
+public:
+  // used only to init dot stack.
+  explicit DottedRuleOnDisk(const OnDiskPt::PhraseNode &lastNode)
+    : DottedRule()
+    , m_lastNode(lastNode)
+    , m_done(false) {}
+
+  DottedRuleOnDisk(const OnDiskPt::PhraseNode &lastNode,
+                   const ChartCellLabel &cellLabel,
+                   const DottedRuleOnDisk &prev)
+    : DottedRule(cellLabel, prev)
+    , m_lastNode(lastNode)
+    , m_done(false) {}
+
+  const OnDiskPt::PhraseNode &GetLastNode() const {
+    return m_lastNode;
+  }
+
+  bool Done() const {
+    return m_done;
+  }
+  void Done(bool value) const {
+    m_done = value;
+  }
+
+private:
+  const OnDiskPt::PhraseNode &m_lastNode;
+  mutable bool m_done;
+};
+
+class DottedRuleCollOnDisk
+{
+protected:
+  typedef std::vector<const DottedRuleOnDisk*> CollType;
+  CollType m_coll;
+
+public:
+  typedef CollType::iterator iterator;
+  typedef CollType::const_iterator const_iterator;
+
+  const_iterator begin() const {
+    return m_coll.begin();
+  }
+  const_iterator end() const {
+    return m_coll.end();
+  }
+  iterator begin() {
+    return m_coll.begin();
+  }
+  iterator end() {
+    return m_coll.end();
+  }
+
+  const DottedRuleOnDisk &Get(size_t ind) const {
+    return *m_coll[ind];
+  }
+
+  void Add(const DottedRuleOnDisk *dottedRule) {
+    m_coll.push_back(dottedRule);
+  }
+  void Delete(size_t ind) {
+    //delete m_coll[ind];
+    m_coll.erase(m_coll.begin() + ind);
+  }
+
+  size_t GetSize() const {
+    return m_coll.size();
+  }
+
+};
+
+class SavedNodeOnDisk
+{
+  const DottedRuleOnDisk *m_dottedRule;
+
+public:
+  SavedNodeOnDisk(const DottedRuleOnDisk *dottedRule)
+    :m_dottedRule(dottedRule) {
+    UTIL_THROW_IF2(m_dottedRule == NULL, "Dotted rule is null");
+  }
+
+  ~SavedNodeOnDisk() {
+    delete m_dottedRule;
+  }
+
+  const DottedRuleOnDisk &GetDottedRule() const {
+    return *m_dottedRule;
+  }
+};
+
+class DottedRuleStackOnDisk
+{
+  // coll of coll of processed rules
+public:
+  typedef std::vector<SavedNodeOnDisk*> SavedNodeColl;
+
+protected:
+  typedef std::vector<DottedRuleCollOnDisk*> CollType;
+  CollType m_coll;
+
+  SavedNodeColl m_savedNode;
+
+public:
+  typedef CollType::iterator iterator;
+  typedef CollType::const_iterator const_iterator;
+
+  const_iterator begin() const {
+    return m_coll.begin();
+  }
+  const_iterator end() const {
+    return m_coll.end();
+  }
+  iterator begin() {
+    return m_coll.begin();
+  }
+  iterator end() {
+    return m_coll.end();
+  }
+
+  DottedRuleStackOnDisk(size_t size);
+  ~DottedRuleStackOnDisk();
+
+  const DottedRuleCollOnDisk &Get(size_t pos) const {
+    return *m_coll[pos];
+  }
+  DottedRuleCollOnDisk &Get(size_t pos) {
+    return *m_coll[pos];
+  }
+
+  const DottedRuleCollOnDisk &back() const {
+    return *m_coll.back();
+  }
+
+  void Add(size_t pos, const DottedRuleOnDisk *dottedRule) {
+    UTIL_THROW_IF2(dottedRule == NULL, "Dotted rule is null");
+
+    m_coll[pos]->Add(dottedRule);
+    m_savedNode.push_back(new SavedNodeOnDisk(dottedRule));
+  }
+
+  const SavedNodeColl &GetSavedNodeColl() const {
+    return m_savedNode;
+  }
+
+  void SortSavedNodes();
+
+};
+
+}
+

mosesdecoder/moses/TranslationModel/CompactPT/CanonicalHuffman.h

@@ -0,0 +1,326 @@
+// $Id$
+// vim:tabstop=2
+/***********************************************************************
+Moses - factored phrase-based language decoder
+Copyright (C) 2006 University of Edinburgh
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this library; if not, write to the Free Software
+Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+***********************************************************************/
+
+#ifndef moses_CanonicalHuffman_h
+#define moses_CanonicalHuffman_h
+
+#include <string>
+#include <algorithm>
+#include <boost/dynamic_bitset.hpp>
+#include <boost/unordered_map.hpp>
+
+#include "ThrowingFwrite.h"
+
+namespace Moses
+{
+
+template <typename Data>
+class CanonicalHuffman
+{
+private:
+  std::vector<Data> m_symbols;
+  std::vector<size_t> m_firstCodes;
+  std::vector<size_t> m_lengthIndex;
+
+  typedef boost::unordered_map<Data, boost::dynamic_bitset<> > EncodeMap;
+  EncodeMap m_encodeMap;
+
+  struct MinHeapSorter {
+    std::vector<size_t>& m_vec;
+
+    MinHeapSorter(std::vector<size_t>& vec) : m_vec(vec) { }
+
+    bool operator()(size_t a, size_t b) {
+      return m_vec[a] > m_vec[b];
+    }
+  };
+
+  template <class Iterator>
+  void CalcLengths(Iterator begin, Iterator end, std::vector<size_t>& lengths) {
+    size_t n = std::distance(begin, end);
+    std::vector<size_t> A(2 * n, 0);
+
+    m_symbols.resize(n);
+    size_t i = 0;
+    for(Iterator it = begin; it != end; it++) {
+      m_symbols[i] = it->first;
+
+      A[i] = n + i;
+      A[n + i] = it->second;
+      i++;
+    }
+
+    if(n == 1) {
+      lengths.push_back(1);
+      return;
+    }
+
+    MinHeapSorter hs(A);
+    std::make_heap(A.begin(), A.begin() + n, hs);
+
+    // marked volatile to prevent the intel compiler from generating bad code
+    volatile size_t h = n;
+    volatile size_t m1, m2;
+    while(h > 1) {
+      m1 = A[0];
+      std::pop_heap(A.begin(), A.begin() + h, hs);
+
+      h--;
+
+      m2 = A[0];
+      std::pop_heap(A.begin(), A.begin() + h, hs);
+
+      A[h] = A[m1] + A[m2];
+      A[h-1] = h;
+      A[m1] = A[m2] = h;
+
+      std::push_heap(A.begin(), A.begin() + h, hs);
+    }
+
+    A[1] = 0;
+    for(size_t i = 2; i < 2*n; i++)
+      A[i] = A[A[i]] + 1;
+
+    lengths.resize(n);
+    for(size_t i = 0; i < n; i++)
+      lengths[i] = A[i + n];
+  }
+
+  void CalcCodes(std::vector<size_t>& lengths) {
+    std::vector<size_t> numLength;
+    for(std::vector<size_t>::iterator it = lengths.begin();
+        it != lengths.end(); it++) {
+      size_t length = *it;
+      if(numLength.size() <= length)
+        numLength.resize(length + 1, 0);
+      numLength[length]++;
+    }
+
+    m_lengthIndex.resize(numLength.size());
+    m_lengthIndex[0] = 0;
+    for(size_t l = 1; l < numLength.size(); l++)
+      m_lengthIndex[l] = m_lengthIndex[l - 1] + numLength[l - 1];
+
+    size_t maxLength = numLength.size() - 1;
+
+    m_firstCodes.resize(maxLength + 1, 0);
+    for(size_t l = maxLength - 1; l > 0; l--)
+      m_firstCodes[l] = (m_firstCodes[l + 1] + numLength[l + 1]) / 2;
+
+    std::vector<Data> t_symbols;
+    t_symbols.resize(lengths.size());
+
+    std::vector<size_t> nextCode = m_firstCodes;
+    for(size_t i = 0; i < lengths.size(); i++) {
+      Data data = m_symbols[i];
+      size_t length = lengths[i];
+
+      size_t pos = m_lengthIndex[length]
+                   + (nextCode[length] - m_firstCodes[length]);
+      t_symbols[pos] = data;
+
+      nextCode[length] = nextCode[length] + 1;
+    }
+
+    m_symbols.swap(t_symbols);
+  }
+
+  void CreateCodeMap() {
+    for(size_t l = 1; l < m_lengthIndex.size(); l++) {
+      size_t intCode = m_firstCodes[l];
+      size_t num = ((l+1 < m_lengthIndex.size()) ? m_lengthIndex[l+1]
+                    : m_symbols.size()) - m_lengthIndex[l];
+
+      for(size_t i = 0; i < num; i++) {
+        Data data = m_symbols[m_lengthIndex[l] + i];
+        boost::dynamic_bitset<> bitCode(l, intCode);
+        m_encodeMap[data] = bitCode;
+        intCode++;
+      }
+    }
+  }
+
+  const boost::dynamic_bitset<>& Encode(Data data) const {
+    typename EncodeMap::const_iterator it = m_encodeMap.find(data);
+    UTIL_THROW_IF2(it == m_encodeMap.end(), "Cannot find symbol in encoding map");
+    return it->second;
+  }
+
+  template <class BitWrapper>
+  void PutCode(BitWrapper& bitWrapper, const boost::dynamic_bitset<>& code) {
+    for(int j = code.size()-1; j >= 0; j--)
+      bitWrapper.Put(code[j]);
+  }
+
+public:
+
+  template <class Iterator>
+  CanonicalHuffman(Iterator begin, Iterator end, bool forEncoding = true) {
+    std::vector<size_t> lengths;
+    CalcLengths(begin, end, lengths);
+    CalcCodes(lengths);
+
+    if(forEncoding)
+      CreateCodeMap();
+  }
+
+  CanonicalHuffman(std::FILE* pFile, bool forEncoding = false) {
+    Load(pFile);
+
+    if(forEncoding)
+      CreateCodeMap();
+  }
+
+  template <class BitWrapper>
+  void Put(BitWrapper& bitWrapper, Data data) {
+    PutCode(bitWrapper, Encode(data));
+  }
+
+  template <class BitWrapper>
+  Data Read(BitWrapper& bitWrapper) {
+    if(bitWrapper.TellFromEnd()) {
+      size_t intCode = bitWrapper.Read();
+      size_t len = 1;
+      while(intCode < m_firstCodes[len]) {
+        intCode = 2 * intCode + bitWrapper.Read();
+        len++;
+      }
+      return m_symbols[m_lengthIndex[len] + (intCode - m_firstCodes[len])];
+    }
+    return Data();
+  }
+
+  size_t Load(std::FILE* pFile) {
+    size_t start = std::ftell(pFile);
+    size_t read = 0;
+
+    size_t size;
+    read += std::fread(&size, sizeof(size_t), 1, pFile);
+    m_symbols.resize(size);
+    read += std::fread(&m_symbols[0], sizeof(Data), size, pFile);
+
+    read += std::fread(&size, sizeof(size_t), 1, pFile);
+    m_firstCodes.resize(size);
+    read += std::fread(&m_firstCodes[0], sizeof(size_t), size, pFile);
+
+    read += std::fread(&size, sizeof(size_t), 1, pFile);
+    m_lengthIndex.resize(size);
+    read += std::fread(&m_lengthIndex[0], sizeof(size_t), size, pFile);
+
+    return std::ftell(pFile) - start;
+  }
+
+  size_t Save(std::FILE* pFile) {
+    size_t start = std::ftell(pFile);
+
+    size_t size = m_symbols.size();
+    ThrowingFwrite(&size, sizeof(size_t), 1, pFile);
+    ThrowingFwrite(&m_symbols[0], sizeof(Data), size, pFile);
+
+    size = m_firstCodes.size();
+    ThrowingFwrite(&size, sizeof(size_t), 1, pFile);
+    ThrowingFwrite(&m_firstCodes[0], sizeof(size_t), size, pFile);
+
+    size = m_lengthIndex.size();
+    ThrowingFwrite(&size, sizeof(size_t), 1, pFile);
+    ThrowingFwrite(&m_lengthIndex[0], sizeof(size_t), size, pFile);
+
+    return std::ftell(pFile) - start;
+  }
+};
+
+template <class Container = std::string>
+class BitWrapper
+{
+private:
+  Container& m_data;
+
+  typename Container::iterator m_iterator;
+  typename Container::value_type m_currentValue;
+
+  size_t m_valueBits;
+  typename Container::value_type m_mask;
+  size_t m_bitPos;
+
+public:
+
+  BitWrapper(Container &data)
+    : m_data(data), m_iterator(m_data.begin()), m_currentValue(0),
+      m_valueBits(sizeof(typename Container::value_type) * 8),
+      m_mask(1), m_bitPos(0) { }
+
+  bool Read() {
+    if(m_bitPos % m_valueBits == 0) {
+      if(m_iterator != m_data.end())
+        m_currentValue = *m_iterator++;
+    } else
+      m_currentValue = m_currentValue >> 1;
+
+    m_bitPos++;
+    return (m_currentValue & m_mask);
+  }
+
+  void Put(bool bit) {
+    if(m_bitPos % m_valueBits == 0)
+      m_data.push_back(0);
+
+    if(bit)
+      m_data[m_data.size()-1] |= m_mask << (m_bitPos % m_valueBits);
+
+    m_bitPos++;
+  }
+
+  size_t Tell() {
+    return m_bitPos;
+  }
+
+  size_t TellFromEnd() {
+    if(m_data.size() * m_valueBits < m_bitPos)
+      return 0;
+    return m_data.size() * m_valueBits - m_bitPos;
+  }
+
+  void Seek(size_t bitPos) {
+    m_bitPos = bitPos;
+    m_iterator = m_data.begin() + int((m_bitPos-1)/m_valueBits);
+    m_currentValue = (*m_iterator) >> ((m_bitPos-1) % m_valueBits);
+    m_iterator++;
+  }
+
+  void SeekFromEnd(size_t bitPosFromEnd) {
+    size_t bitPos = m_data.size() * m_valueBits - bitPosFromEnd;
+    Seek(bitPos);
+  }
+
+  void Reset() {
+    m_iterator = m_data.begin();
+    m_currentValue = 0;
+    m_bitPos = 0;
+  }
+
+  Container& GetContainer() {
+    return m_data;
+  }
+};
+
+}
+
+#endif