NGramMerge

Description

Usage

ngrammerge [--options] in1.fst in2.fst [out.fst]
  --alpha: type = double, default = 1.0, weight for in1.fst in real semiring
  --beta: type = double, default = 1.0, weight for in2.fst in real semiring
  --normalize: type = bool, default = false, whether to normalize the resulting model
  --use_smoothing: type = bool, default = false, whether to use model smoothing when merging
  --fixedorder: type = bool, default false, whether to merge in the given argument order
 
 class NGramMerge(StdMutableFst *infst1, StdMutableFst *infst2, double alpha, double beta);
 

Examples

Suppose we split our corpus up into two parts, earnest.aa and earnest.ab, e.g., by using split:

$ split -844 earnest.txt earnest.

If we count each half independently, we can then merge the counts to get the same counts as derived above from the full corpus (earnest.cnts):

$ farcompilestrings -symbols=earnest.syms earnest.aa >earnest.aa.far
$ ngramcount -order=5 earnest.aa.far >earnest.aa.cnts
$ farcompilestrings -symbols=earnest.syms earnest.ab >earnest.ab.far
$ ngramcount -order=5 earnest.ab.far >earnest.ab.cnts
$ ngrammerge earnest.aa.cnts earnest.ab.cnts >earnest.merged.cnts
$ fstequal earnest.cnts earnest.merged.cnts

Note that, unlike our example merging unnormalized counts above, merging two smoothed models that have been built from half a corpus each will result in a different model than one built from the corpus as a whole, due to the smoothing and mixing.

Each of the two model or count FSTs can be weighted, using the --alpha switch for the first input FST, and the --beta switch for the second input FST. These weights are interpreted in the real semiring and both default to one, meaning that by default the original counts or probabilities are not scaled. To triple the contribution of the first model and double the contribution of the second:

$ ngrammerge --alpha=3 --beta=2 earnest.aa.mod earnest.ab.mod >earnest.merged.mod

Caveats

References

-- MichaelRiley - 09 Dec 2011

Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2011-12-13 - BrianRoark
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback