当前位置:首页 > 开发 > 编程语言 > 搜索 > 正文

使用Lucene3和IKAnalyzer对一段文本进行分词

发表于: 2010-12-23   作者:hotsunshine   来源:转载   浏览:
摘要: import java.io.IOException; import java.io.StringReader; import java.util.ArrayList; import java.util.List; import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.TokenSt
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
import org.wltea.analyzer.lucene.IKAnalyzer;

/**
 * 对一段文字进行分词
 * @author Administrator
 */
public class IkAnalyzerWord {

  private String resource;
  private List<String> result = new ArrayList<String>();

  public IkAnalyzerWord(String resource) throws IOException {
    this.resource = resource;
    analyzer();
  }

  private void analyzer() throws IOException {
    Analyzer analyzer = new IKAnalyzer();
    TokenStream ts = analyzer.tokenStream("*", new StringReader(resource));
    ts.addAttribute(TermAttribute.class);
    //public <A extends Attribute> A addAttribute(Class<A> attClass)
    //The caller must pass in a Class<? extends Attribute> value.
    //This method first checks if an instance of that class is already in this AttributeSource and returns it.
    //Otherwise a new instance is created, added to this AttributeSource and returned.
    while (ts.incrementToken()) {
      TermAttribute ta = ts.getAttribute(TermAttribute.class);
      //public <A extends Attribute> A getAttribute(Class<A> attClass)
      //The caller must pass in a Class<? extends Attribute> value. 
      //Returns the instance of the passed in Attribute contained in this AttributeSource
      result.add(ta.term());
      //Returns the Token's term text.
    }
  }

  public List<String> getResult() {
    return this.result;
  }

  public static void main(String[] args) throws IOException {
    IkAnalyzerWord ik = new IkAnalyzerWord("今天的大风终于小了,但是又起雾了今天的大风终于小了,但是又起雾了");
    System.out.println(ik.getResult());
  }
}


输出结果(我设置了stopword词典)

[大风, 终于, 小了, 雾, 大风, 终于, 小了, 雾]

使用Lucene3和IKAnalyzer对一段文本进行分词

  • 0

    开心

    开心

  • 0

    板砖

    板砖

  • 0

    感动

    感动

  • 0

    有用

    有用

  • 0

    疑问

    疑问

  • 0

    难过

    难过

  • 0

    无聊

    无聊

  • 0

    震惊

    震惊

版权所有 IT知识库 CopyRight © 2009-2015 IT知识库 IT610.com , All Rights Reserved. 京ICP备09083238号