Text Processing in Information Retreival

Search Engines process texts before applying any ranking model (Vector Space Model, Probabilistic Models or Inference Network Model ) to find the information about documents and terms contained in the document. Some text preprocessing techniques that are commonly used in information retrieval systems are:

Tokenization

Tokenization is the technique of converting a character stream into tokens by detecting word boundaries. The most common word boundary is one or white space character (which can be new line, tab, carriage return and form feed) in continuation.

Normalization

Normalization is a technique which is applied to terms to minimize the number of variations of a word.The syntactical variations may result in drop of recall value if an exact match against the query keywords is required. The two most common methods used for normalization are:

Stemming

The technique is used to convert a word into root word or stem based on some rules. The stem is what is left after any prefixes or suffixes have been removed. Foe eg. connect which can be the stem for connected, connecting, connection, and connections. The four widely used stemming algorithms are affix removal, table lookup, successor variety, n-grams and Porter Stemmer. The main disadvantage of stemming is that the stem may not be a real word and therefore it should be used with caution. Light Stemming is another variation of stemming where only plural forms are stemmed. For example cars -> car. 

Lemmatization

Lemmatization is another normalization technique that is based around dictionary lookups. An advantage of lemmatization is that it will give valid words, but that in this case the context is required to be known is advance.

 Stop Words Removal

Stop words are frequently used words (for example the, is, or, and) that provides very little information about the content. The removal of stop words will decrease index size drastically but will also decrease recall in case of exact matches. Another  problem is that common stop words like may, can, and will are homonyms for rarer nouns. For example may is a stop words, but it is also the name of a month, May.

After the above processes are executed in the order the information regarding documents verses terms are maintained which is used by various mathematical models to select and rank the documents based on the information needs of the users. 

How Apache Solr Search Works?

Apache Solr is serverization  of Lucene Search engine, which is a information retrieval system and works on the principles of Vector Space Model. In this model each document is represented as a vector with each dimension representing a separate term. Set of such documents is called a corpus. A query string is also considered a vector and the similarity between the query document and a particular document in the corpora is calculated as cosine between two vector which gives the score for that document against the given query string.

Consider a  document in a corpora and a query string that will be used to search over documents. These can be represented as following vectors

d1=(w1,w2,..,wn)
q=(z1,z2,...zt)

w1,w2..wn and z1,z2...zn represents as weights of various terms in the above documents. These weights are calculated as tf*idf (term frequency and inverse document frequency) weights. The tf*idf weight (term frequency–inverse document frequency) is a numerical statistic which reflects how important a word is to a document in a collection or corpus. Term frequency is number of times a term occurs in a document whereas inverse document frequency finds out the more relevant documents over a given query. These can be calculated by the following formulas:

tf  =  (number of times a term occurs in a document)/ (total number of terms in that document)
idf =  log { Total documents/ (Total documents containing those terms+1) }

The score of a particular document against a given query can be calculated as:

Score of document=cosine=d1.q/ |d1||q| where |d1| is vector norm and can be calculated as : square root of (w1^2+w2^2+....+wn^2)

This can be inferred that if a term occurs in most of the documents then the idf for that term will be less and it will contribute less in the score.