Package com.easydesk.lucene.analyzer

Class CJKAnalyzer

java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.StopwordAnalyzerBase
com.easydesk.lucene.analyzer.CJKAnalyzer
All Implemented Interfaces:
Closeable, AutoCloseable
public final class CJKAnalyzer extends org.apache.lucene.analysis.StopwordAnalyzerBase
EasyDesk 기본 Analyzer.

EasyDeskTokenizer 로 tokenize 하고, CJKWidthFilter 로 정규화, LowerCaseFilter 로 소문자화, CJKBigramFilter 로 한글/한자/일본어를 bigram 으로 만든 뒤 StopFilter 로 불용어를 제거한다.

Lucene 의 org.apache.lucene.analysis.cjk.CJKAnalyzer 와 같은 구성이지만 tokenizer 가 MathJax 수식을 하나의 token 으로 유지한다.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer

    org.apache.lucene.analysis.Analyzer.ReuseStrategy, org.apache.lucene.analysis.Analyzer.TokenStreamComponents

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_STOPWORD_FILE
    File containing default CJK stopwords.

    Fields inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase

    stopwords

    Fields inherited from class org.apache.lucene.analysis.Analyzer

    GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY

  • Constructor Summary

    Constructors
    Constructor
    Description
    CJKAnalyzer()
    Builds an analyzer which removes words in getDefaultStopSet().
    CJKAnalyzer(org.apache.lucene.analysis.CharArraySet stopwords)
    Builds an analyzer with the given stop words

  • Method Summary

    Modifier and Type
    Method
    Description
    protected org.apache.lucene.analysis.Analyzer.TokenStreamComponents
    createComponents(String fieldName)
     
    static org.apache.lucene.analysis.CharArraySet
    getDefaultStopSet()
    Returns an unmodifiable instance of the default stop-words set.
    protected org.apache.lucene.analysis.TokenStream
    normalize(String fieldName, org.apache.lucene.analysis.TokenStream in)
     

    Methods inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase

    getStopwordSet, loadStopwordSet, loadStopwordSet

    Methods inherited from class org.apache.lucene.analysis.Analyzer

    attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReader, initReaderForNormalization, normalize, tokenStream, tokenStream

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_STOPWORD_FILE

      public static final String DEFAULT_STOPWORD_FILE
      File containing default CJK stopwords.

      Currently it contains some common English words that are not usually useful for searching.

      See Also:

  • Constructor Details

    • CJKAnalyzer

      public CJKAnalyzer()
      Builds an analyzer which removes words in getDefaultStopSet().

    • CJKAnalyzer

      public CJKAnalyzer(org.apache.lucene.analysis.CharArraySet stopwords)
      Builds an analyzer with the given stop words
      Parameters:
      stopwords - a stopword set

  • Method Details

    • getDefaultStopSet

      public static org.apache.lucene.analysis.CharArraySet getDefaultStopSet()
      Returns an unmodifiable instance of the default stop-words set.
      Returns:
      an unmodifiable instance of the default stop-words set.

    • createComponents

      protected org.apache.lucene.analysis.Analyzer.TokenStreamComponents createComponents(String fieldName)
      Specified by:
      createComponents in class org.apache.lucene.analysis.Analyzer

    • normalize

      protected org.apache.lucene.analysis.TokenStream normalize(String fieldName, org.apache.lucene.analysis.TokenStream in)
      Overrides:
      normalize in class org.apache.lucene.analysis.Analyzer