de.pangaea.metadataportal.search
Class MoreLikeThisQuery

java.lang.Object
  org.apache.lucene.search.Query
      de.pangaea.metadataportal.search.MoreLikeThisQuery

All Implemented Interfaces:

Serializable, Cloneable

public final class MoreLikeThisQuery
extends Query

This implementation of a "more like this" is based on an algorithm from Lucene Contrib and optimized for use with panFMP. The constructor needs a panFMP document ID and optional a field name (which must have term vectors enabled). If you use the default field for similarity calculation, the default field must have term vectors.

Lucene does let you access the document frequency of terms, with IndexReader.docFreq(). Term frequencies can be computed by re-tokenizing the text, which, for a single document, is usually fast enough. But looking up the docFreq() of every term in the document is probably too slow.

You can use some heuristics to prune the set of terms, to avoid calling docFreq() too much, or at all. Since you're trying to maximize a tf*idf score, you're probably most interested in terms with a high tf. Choosing a tf threshold even as low as two or three will radically reduce the number of terms under consideration. Another heuristic is that terms with a high idf (i.e., a low df) tend to be longer. So you could threshold the terms by the number of characters, not selecting anything less than, e.g., six or seven characters. With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms that do a pretty good job of characterizing a document.

It all depends on what you're trying to do. If you're trying to eek out that last percent of precision and recall regardless of computational difficulty so that you can win a TREC competition, then the techniques I mention above are useless. But if you're trying to provide a "more like this" button on a search results page that does a decent job and has good performance, such techniques might be useful.

Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated:

• setMinTermFreq(...)
• setMinDocFreq(...)
• setMinWordLen(...)
• setMaxWordLen(...)
• setMaxQueryTerms(...)
• setFractionTermsToMatch(...)

At rewrite(org.apache.lucene.index.IndexReader) time the reader is used to construct the actual BooleanQuery containing the relevant terms.

Author:

Uwe Schindler, David Spencer, Bruce Ritchie, Mark Harwood

See Also:

Serialized Form

Field Summary

static boolean	DEFAULT_BOOST_BY_SCORE Boost terms in query based on score.
static float	DEFAULT_FRACTION_TERMS_TO_MATCH How many terms must match a similar document (fraction, 1.0 means all).
static int	DEFAULT_MAX_QUERY_TERMS Return a Query with no more than this many terms.
static int	DEFAULT_MAX_WORD_LENGTH Ignore words greater than this length or if 0 then this has no effect.
static int	DEFAULT_MIN_DOC_FREQ Ignore words which do not occur in at least this many docs.
static int	DEFAULT_MIN_TERM_FREQ Ignore terms with less than this frequency in the source doc.
static int	DEFAULT_MIN_WORD_LENGTH Ignore words less than this length or if 0 then this has no effect.

Constructor Summary

MoreLikeThisQuery(String docIdentifier)
Creates a MoreLikeThisQuery instance finding similar documents of a panFMP document ID.

MoreLikeThisQuery(String docIdentifier, String matchingField)
Creates a MoreLikeThisQuery instance finding similar documents of a panFMP document ID.

Method Summary

boolean	equals(Object o)
float	getFractionTermsToMatch() Gets the percentage of terms to match in similar documents.
int	getMaxQueryTerms() Returns the maximum number of query terms that will be included in the rewritten query.
int	getMaxWordLen() Returns the maximum word length above which words will be ignored.
int	getMinDocFreq() Returns the frequency at which words will be ignored which do not occur in at least this many docs.
int	getMinTermFreq() Returns the frequency below which terms will be ignored in the source doc.
int	getMinWordLen() Returns the minimum word length below which words will be ignored.
int	hashCode()
boolean	isBoostByScore() Returns whether to boost terms in query based on "score" or not.
Query	rewrite(IndexReader reader)
void	setBoostByScore(boolean boostByScore) Sets whether to boost terms in query based on "score" or not.
void	setFractionTermsToMatch(float fractionTermsToMatch) Sets the percentage of terms to match in similar documents.
void	setMaxQueryTerms(int maxQueryTerms) Sets the maximum number of query terms that will be included in the rewritten query.
void	setMaxWordLen(int maxWordLen) Sets the maximum word length above which words will be ignored.
void	setMinDocFreq(int minDocFreq) Sets the frequency at which words will be ignored which do not occur in at least this many docs.
void	setMinTermFreq(int minTermFreq) Sets the frequency below which terms will be ignored in the source doc.
void	setMinWordLen(int minWordLen) Sets the minimum word length below which words will be ignored.
String	toString(String field)

Methods inherited from class org.apache.lucene.search.Query

clone, combine, createWeight, extractTerms, getBoost, getSimilarity, mergeBooleanQueries, setBoost, toString, weight

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_MIN_TERM_FREQ

public static final int DEFAULT_MIN_TERM_FREQ

Ignore terms with less than this frequency in the source doc.

See Also:

getMinTermFreq(), setMinTermFreq(int), Constant Field Values

DEFAULT_MIN_DOC_FREQ

public static final int DEFAULT_MIN_DOC_FREQ

Ignore words which do not occur in at least this many docs.

See Also:

getMinDocFreq(), setMinDocFreq(int), Constant Field Values

DEFAULT_MIN_WORD_LENGTH

public static final int DEFAULT_MIN_WORD_LENGTH

Ignore words less than this length or if 0 then this has no effect.

See Also:

getMinWordLen(), setMinWordLen(int), Constant Field Values

DEFAULT_MAX_WORD_LENGTH

public static final int DEFAULT_MAX_WORD_LENGTH

Ignore words greater than this length or if 0 then this has no effect.

See Also:

getMaxWordLen(), setMaxWordLen(int), Constant Field Values

DEFAULT_MAX_QUERY_TERMS

public static final int DEFAULT_MAX_QUERY_TERMS

Return a Query with no more than this many terms.

See Also:

BooleanQuery.getMaxClauseCount(), getMaxQueryTerms(), setMaxQueryTerms(int), Constant Field Values

DEFAULT_BOOST_BY_SCORE

public static final boolean DEFAULT_BOOST_BY_SCORE

Boost terms in query based on score.

See Also:

isBoostByScore(), setBoostByScore(boolean), Constant Field Values

DEFAULT_FRACTION_TERMS_TO_MATCH

public static final float DEFAULT_FRACTION_TERMS_TO_MATCH

How many terms must match a similar document (fraction, 1.0 means all).

See Also:

getFractionTermsToMatch(), setFractionTermsToMatch(float), Constant Field Values

Constructor Detail

MoreLikeThisQuery

public MoreLikeThisQuery(String docIdentifier)

Creates a MoreLikeThisQuery instance finding similar documents of a panFMP document ID. The default field must have term vectors enabled in panFMP's configuration.

MoreLikeThisQuery

public MoreLikeThisQuery(String docIdentifier,
                         String matchingField)

Creates a MoreLikeThisQuery instance finding similar documents of a panFMP document ID. This variant exspects a field name, which term vector contents are used to build the query. This is optimal, if you want to find similarities only by comparing e.g. titles of documents.

Method Detail

getMinTermFreq

public int getMinTermFreq()

Returns the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ.

Returns:

the frequency below which terms will be ignored in the source doc.

setMinTermFreq

public void setMinTermFreq(int minTermFreq)

Sets the frequency below which terms will be ignored in the source doc.

Parameters:

minTermFreq - the frequency below which terms will be ignored in the source doc.

getMinDocFreq

public int getMinDocFreq()

Returns the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ.

Returns:

the frequency at which words will be ignored which do not occur in at least this many docs.

setMinDocFreq

public void setMinDocFreq(int minDocFreq)

Sets the frequency at which words will be ignored which do not occur in at least this many docs.

Parameters:

minDocFreq - the frequency at which words will be ignored which do not occur in at least this many docs.

isBoostByScore

public boolean isBoostByScore()

Returns whether to boost terms in query based on "score" or not.

Returns:

whether to boost terms in query based on "score" or not.

See Also:

setBoostByScore(boolean)

setBoostByScore

public void setBoostByScore(boolean boostByScore)

Sets whether to boost terms in query based on "score" or not.

Parameters:

boostByScore - true to boost terms in query based on "score", false otherwise. The default is DEFAULT_BOOST_BY_SCORE.

See Also:

isBoostByScore()

getMinWordLen

public int getMinWordLen()

Returns the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH.

Returns:

the minimum word length below which words will be ignored.

setMinWordLen

public void setMinWordLen(int minWordLen)

Sets the minimum word length below which words will be ignored.

Parameters:

minWordLen - the minimum word length below which words will be ignored.

getMaxWordLen

public int getMaxWordLen()

Returns the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH.

setMaxWordLen

public void setMaxWordLen(int maxWordLen)

Sets the maximum word length above which words will be ignored.

getMaxQueryTerms

public int getMaxQueryTerms()

Returns the maximum number of query terms that will be included in the rewritten query. The default is DEFAULT_MAX_QUERY_TERMS.

setMaxQueryTerms

public void setMaxQueryTerms(int maxQueryTerms)

Sets the maximum number of query terms that will be included in the rewritten query. A hard limit is BooleanQuery.getMaxClauseCount().

getFractionTermsToMatch

public float getFractionTermsToMatch()

Gets the percentage of terms to match in similar documents.

setFractionTermsToMatch

public void setFractionTermsToMatch(float fractionTermsToMatch)

Sets the percentage of terms to match in similar documents. Defaults to DEFAULT_FRACTION_TERMS_TO_MATCH.

rewrite

public Query rewrite(IndexReader reader)
              throws IOException

Overrides:

rewrite in class Query

Throws:

IOException

toString

public String toString(String field)

Specified by:

toString in class Query

equals

public final boolean equals(Object o)

Overrides:

equals in class Query

hashCode

public final int hashCode()

Overrides:

hashCode in class Query