Package com.easydesk.lucene.analyzer

Class EasyDeskTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
com.easydesk.lucene.analyzer.EasyDeskTokenizer
All Implemented Interfaces:
Closeable, AutoCloseable
public final class EasyDeskTokenizer extends org.apache.lucene.analysis.Tokenizer
JFlex 로 생성한 grammar 기반 tokenizer.

Lucene StandardTokenizer 와 같이 Unicode Standard Annex #29 의 Word Break rule 을 구현하며, 추가로 MathJax inline 수식 ($...$) 을 하나의 <MATH_EXPRESSION> token 으로 처리한다.

Token type 문자열은 StandardTokenizer.TOKEN_TYPES 의 instance 를 그대로 사용하므로 CJKBigramFilter 등 Lucene filter 와 함께 사용할 수 있다.

Author:
조 영 호

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

    org.apache.lucene.util.AttributeSource.State

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    ALPHANUM
    Alpha/numeric token type
    static final int
    DEFAULT_MAX_TOKEN_LENGTH
    Default maximum allowed token length
    static final int
    EMOJI
    Emoji token type
    static final int
    HANGUL
    Hangul token type
    static final int
    HIRAGANA
    Hiragana token type
    static final int
    IDEOGRAPHIC
    Ideographic token type
    static final int
    KATAKANA
    Katakana token type
    static final int
    MATH_EXPRESSION
    MathJax Latex token type
    static final int
    NUM
    Numeric token type
    static final int
    SOUTHEAST_ASIAN
    Southeast Asian token type
    static final String[]
    TOKEN_TYPES
    String token types that correspond to token type int constants

    Fields inherited from class org.apache.lucene.analysis.Tokenizer

    input

    Fields inherited from class org.apache.lucene.analysis.TokenStream

    DEFAULT_TOKEN_ATTRIBUTE_FACTORY

  • Constructor Summary

    Constructors
    Constructor
    Description
    EasyDeskTokenizer()
    Creates a new instance of the EasyDeskTokenizer.
    EasyDeskTokenizer(org.apache.lucene.util.AttributeFactory factory)
    Creates a new EasyDeskTokenizer with a given AttributeFactory

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    void
    end()
     
    int
    getMaxTokenLength()
     
    boolean
    incrementToken()
     
    void
    reset()
     
    void
    setMaxTokenLength(int length)
    Set the max allowed token length.

    Methods inherited from class org.apache.lucene.analysis.Tokenizer

    correctOffset, setReader, setReaderTestPoint

    Methods inherited from class org.apache.lucene.util.AttributeSource

    addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

  • Field Details

    • ALPHANUM

      public static final int ALPHANUM
      Alpha/numeric token type
      See Also:

    • NUM

      public static final int NUM
      Numeric token type
      See Also:

    • SOUTHEAST_ASIAN

      public static final int SOUTHEAST_ASIAN
      Southeast Asian token type
      See Also:

    • IDEOGRAPHIC

      public static final int IDEOGRAPHIC
      Ideographic token type
      See Also:

    • HIRAGANA

      public static final int HIRAGANA
      Hiragana token type
      See Also:

    • KATAKANA

      public static final int KATAKANA
      Katakana token type
      See Also:

    • HANGUL

      public static final int HANGUL
      Hangul token type
      See Also:

    • EMOJI

      public static final int EMOJI
      Emoji token type
      See Also:

    • MATH_EXPRESSION

      public static final int MATH_EXPRESSION
      MathJax Latex token type
      See Also:

    • TOKEN_TYPES

      public static final String[] TOKEN_TYPES
      String token types that correspond to token type int constants

    • DEFAULT_MAX_TOKEN_LENGTH

      public static final int DEFAULT_MAX_TOKEN_LENGTH
      Default maximum allowed token length
      See Also:

  • Constructor Details

    • EasyDeskTokenizer

      public EasyDeskTokenizer()
      Creates a new instance of the EasyDeskTokenizer.

    • EasyDeskTokenizer

      public EasyDeskTokenizer(org.apache.lucene.util.AttributeFactory factory)
      Creates a new EasyDeskTokenizer with a given AttributeFactory
      Parameters:
      factory - attribute factory

  • Method Details

    • setMaxTokenLength

      public void setMaxTokenLength(int length)
      Set the max allowed token length. Any token longer than this is skipped.
      Parameters:
      length - max token length

    • getMaxTokenLength

      public int getMaxTokenLength()
      Returns:
      max token length
      See Also:

    • incrementToken

      public boolean incrementToken() throws IOException
      Specified by:
      incrementToken in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • end

      public void end() throws IOException
      Overrides:
      end in class org.apache.lucene.analysis.TokenStream
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class org.apache.lucene.analysis.Tokenizer
      Throws:
      IOException

    • reset

      public void reset() throws IOException
      Overrides:
      reset in class org.apache.lucene.analysis.Tokenizer
      Throws:
      IOException