OpenGrm Advanced Usage

Below are a variety of topics covered in greater depth or of more specialized interest than found in the Quick Tour. Reading the Quick Tour first is recommended.

Distributed Computation

The C++ operations in OpenGrm offer extensive distributed computation support. N-gram counting can readily be parallelized by sharding the data and producing a count FST M_d for each data shard d. These can then be count-merged to produce a single count FST. Alternatively and with more parallelism, each M_d can be further split by context c, which restricts each sub-model M_d,c to a specific range of n-grams. The M_d,c in the same context c can then be count-merged to produce one model M_c for each context. The M_c are constructed to be in proper n-gram model format and so can be processed in parallel by the estimation and pruning operations and then the pruned model components can be model-merged into a single model at the end of this pipeline.

We have implemented a complete distributed version of OpenGrm NGram in C++ Flume, however this system is currently not released. Instead, we provide here some added functionality to our convenience script, ngramdisttrain.sh. While this does not perform parallel computation, it can construct a pruned model as described above using data and context sharding. This allows processing corpora that would otherwise exceed available memory provided adequate disk space (under $TMPDIR) and computation time are provided. The script also could serve as a starting point for a fully distributed implementation by parallelizing the calls internal to the script, which should linearly speed up the pipeline with the degree of parallelism. An implementation that didn't use the file system for all data sharing/transfer like ngram.sh would also help greatly.

Multiple data shards are supported by specifying multiple input files to ngramdisttrain.sh with --ifile="in.txt[0-9]". Multiple contexts are supported by specifying a context file with --contexts=context.txt. The best way to create a context file with, say, ten contexts balanced in size is with:

 
ngramcontext --context=10 lm.fst context.txt

where lm.fst is a n-gram LM that was built on a sample of the corpus (e..g, small enough to build unshared). Note you must provide a context file (even if it only has one context) if you want to use data sharding. If you wish the shared context models to be merged when the pipeline finishes, you should provide the --merge_contexts flag, otherwise the component models will be returned.

-- MichaelRiley - 07 Aug 2013