OpenGrm Advanced Usage
Below are a variety of topics covered in greater depth or of more specialized interest than found in the Quick Tour. Reading the
Quick Tour first is recommended.
Distributed Computation
The C++ operations in OpenGrm offer extensive distributed computation support. N-gram counting can readily be parallelized by
sharding the data and producing a
count FST Md for each data shard
d. These can then be
count-merged to produce a single count FST. Alternatively and with more parallelism,
each
Md can be further split by
context c, which restricts each sub-model
Md,c to a specific range of n-grams. The
Md,c in the same context
c can then be count-merged to produce
one model
Mc for each context. The
Mc are constructed to be in proper
n-gram model format and so can be processed in parallel by the estimation and pruning operations
and then the pruned model components can be
model-merged into a single model at the end of this pipeline.
We have implemented a complete distributed version of OpenGrm NGram in
C++ Flume, however this system is currently not released. Instead, we provide here some added
functionality to our convenience script,
ngramdisttrain.sh. While this does not perform parallel computation, it can construct a pruned model as described above using data and context sharding. This allows processing corpora that would otherwise exceed available memory provided adequate disk space (under
$TMPDIR
) and computation time are provided. The script also could serve as a starting point for a fully distributed implementation by parallelizing the calls internal to the script, which should linearly speed up the pipeline with the degree of parallelism. An implementation that didn't use the file system for all data sharing/transfer like
ngram.sh
would also help greatly.
Multiple data shards are supported by specifying multiple input files to
ngramdisttrain.sh
with
--ifile="in.txt[0-9]"
. Multiple contexts are supported by specifying a context file with
--contexts=context.txt
. The best way to create a context file with, say, ten contexts balanced in size is with:
ngramcontext --context=10 lm.fst context.txt
where
lm.fst
is a n-gram LM that was built on a sample of the corpus (e..g, small enough to build unshared). Note you must provide a context file (even if it only has one context) if you want to use data sharding.
If you wish the shared context models to be merged when the pipeline finishes, you should provide the
--merge_contexts
flag, otherwise the component models will be returned.
--
MichaelRiley - 07 Aug 2013