Myanmar Searching

Using Nutch with Myanmar


Nutch uses Lucene to index and search documents on the Web. Normally, it tokenizes text based on whitespace. However, it is easy to specify language specific tokenizers. An implementation of a Myanmar Tokenizer and associated files is available below.


Download Nutch and Lucene from Apache and then download the following Myanmar specific files.

/var/www/ThanLwinSoft/Downloads/Searching/ not found

These instructions assume that you have a suitable Java JDK (1.4 or higher) and ant and installed. Create a directory to contain the files e.g. /opt/nutch and place the downloaded files in it, then run this script:

## This script assumes that it is run as a user who has write permission in /opt/nutch
## Update these variables to the versions that you are using
export BASE_DIR=/opt/nutch
export LUCENE=lucene-1.9.1
export ORIG_LUCENE_VER=1.9-rc1
export NEW_LUCENE_VER=1.9.2
export NUTCH=nutch-0.8.1
export MY_NUTCH_DATE=20070302
export CATALINA_HOME=/opt/apache-tomcat-6.0.10

if ! (test -f $BASE_DIR/$LUCENE-src.tar.gz) then missing=1; fi
if ! (test -f $BASE_DIR/$NUTCH.tar.gz) then missing=1; fi
if ! (test -f $BASE_DIR/nutch-analysis-my-$ then missing=1; fi
if ! (test -f $BASE_DIR/lucene-analysis-my-$ then missing=1; fi
if ! (test -f $BASE_DIR/myNgram-$ then missing=1; fi
if test $missing
echo WARNING: $BASE_DIR/$LUCENE-src.tar.gz $BASE_DIR/$NUTCH.tar.gz $BASE_DIR/nutch-analysis-my-$ 
echo $BASE_DIR$BASE_DIR/lucene-analysis-my-$ $BASE_DIR/myNgram-$ not found.
return 1

tar -zxvf $LUCENE-src.tar.gz
cd $BASE_DIR/$LUCENE/contrib/analyzers/src/java/
unzip $BASE_DIR/lucene-analysis-my-$
cd $BASE_DIR/$LUCENE/contrib/analyzers
cd $BASE_DIR/$LUCENE/contrib/miscellaneous

tar -zxvf $NUTCH.tar.gz
## use your locally built lucene-analyzers
cd $BASE_DIR/$NUTCH/src/plugin/lib-lucene-analyzers
cp plugin.xml plugin.xml.orig
## update the version to the one that you are using
cat plugin.xml.orig | sed s/$ORIG_LUCENE_VER/$NEW_LUCENE_VER/g > plugin.xml
cd lib
ln -s ../../../../../$LUCENE/build/contrib/analyzers/lucene-analyzers-$NEW_LUCENE_VER-dev.jar
## build the myanmar nutch analysis wrapper
cd $BASE_DIR/$NUTCH/src/plugin
unzip $BASE_DIR/nutch-analysis-my-$
cd analysis-my
cd ../
## You may want to create your own ngrams
## e.g. if you have a large UTF-8 encoded Myanmar text file called myanmar.txt in your base directory
# cd $BASE_DIR/$NUTCH/src/plugin/languageidentifier/src/java/org/apache/nutch/analysis/lang/
# java -cp $BASE_DIR/$NUTCH/build/plugins/language-identifier/language-identifier.jar:$BASE_DIR/$NUTCH/lib/commons-logging-1.0.4.jar  org.apache.nutch.analysis.lang.NGramProfile -create my $BASE_DIR/myanmar.txt  UTF-8
## otherwise you can just use this one:
unzip $BASE_DIR/myNgram-$
cd languageidentifier
mkdir oldlib
mv lib/lucene-core*.jar oldlib
mv lib/lucene-misc*.jar oldlib
ln -s ../../$LUCENE/build/lucene-core-$NEW_LUCENE_VER-dev.jar
ln -s ../../$LUCENE/build/contrib/misc/lucene-misc-$NEW_LUCENE_VER-dev.jar
## get rid of the original versions of the plugins that we have rebuilt
## so they don't get picked up on the class path accidentally
rm -rf plugins/lib-lucene-analyzers
rm -rf plugins/language-identifier

echo Edit nutch-site.xml as appropriate
echo Make sure that the plugin.includes property value includes 
echo language-identifier\|analysis-my
gedit conf/nutch-site.xml
echo Edit src/web/include/style.html to include a suitable Myanmar font
echo e.g. add a style: "
* {
  font-family: Padauk, Myanmar3, Arial, Helvetica, sans-serif;
  line-height: 1.5em; 
gedit src/web/include/style.html
## build the war file with the configuration data
ant war
if test -f $CATALINA_HOME/webapps
    cp $NUTCH/build/$NUTCH.war $CATALINA_HOME/webapps
echo now copy $NUTCH/build/$NUTCH.war into your Tomcat catalina webapps directory
echo Follow the instructions at
echo to build an index e.g. in /opt/nutch/crawl and then test searching.

When this script runs, you will be asked to edit a couple of files. You will need to update your nutch-0.8.1/conf/nutch-site.xml to specify the language plugins and your crawl directory as in the example below. The searcher.dir property must match the one that you use in the -dir option to bin/nutch/crawl as described in the Nutch Tutorial.

  <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,
  and basic indexing and search plugins.

You will probably want to edit the styles in nutch-0.8.1/src/web/include/style.html e.g.

/* Default to a Myannmar font compliant to the latest Unicode proposal */
* {
  font-family: PadaukOT, Myanmar2, Padauk, Arial, Helvetica, sans-serif;
  line-height: 1.5em;
  font-size: 12px;
/* Underline doesn't look great with Myanmar, so use a bottom border instead */
a {
  text-decoration: none;
  border-bottom-style: solid;
  border-bottom-width: 1px;

The Apache Tomcat $CATALINA_HOME/conf/server.xml file needs to be modified to support UTF-8 queries. Make sure that the HTTP connector has the URIEncoding attribute set. e.g.

<Connector port="8080" protocol="HTTP/1.1" 
               maxThreads="150" connectionTimeout="20000" 
               URIEncoding="UTF-8" />

You are now ready to follow the instructions in the Nutch Tutorial to create an index and then test searching using the Tomcat interface.