A Similarity Measure for Text Classification and Clustering

IJSRD - International Journal for Scientific Research & Development| Vol. 3, Issue 08, 2015 | ISSN (online): 2321-0613

A Similarity Measure for Text Classification and Clustering P.Mayilsamy1 P.Elango (M.C.A., M.Phil)2 1 Research Scholar 2Assistant Professor 1,2 Department of Computer Science 1,2 Gobi Arts & Science College (Autonomous), Gobichettipalayam Abstract—This paper introduces a live of similarity between two clustering’s of constant dataset made by two completely different algorithms, or maybe constant algorithmic rule.

Mensuration the similarity between documents is a crucial operation within the text process field. This paper projected a replacement similarity measure. To compute the similarity between two documents with relevance a feature, the proposed measure takes the subsequent three cases into account: a) The feature seems in each documents, b) The feature seems in just one document, and c) The feature seems in none of the documents. For the primary case, the similarity will increase because the distinction between the two concerned feature values decreases. Moreover, the contribution of the distinction is generally scaled. For the second case, a hard and fast worth is contributed to the similarity. For the last case, the feature has no contribution to the similarity. The effectiveness of the live is evaluated on many real-world knowledge sets for text classification and bunch issues.

Key words: Clustering Algorithms, Clustering Applications, Similarity Measures, Text Clustering

similarities between the info points. Locality Preserving Indexing (LPI) methodology could be a completely different spectral bunch methodology supported graph partitioning theory. The LPI methodology applies a weighted operate to every pairwise distance trying to specialise in capturing the similarity structure, instead of the difference structure of the documents. However, it does not overcome the essential limitation of geometer distance. Moreover, the choice of the weighted functions is commonly a troublesome task.

Simultaneously, bunch still needs additional sturdy unsimilarity or similarity measures. The thesis is motivated by investigations from the on top of and similar analysis findings. It seems that the character of similarity live plays a really necessary role within the success or failure of a bunch methodology. The first objective is to derive a completely unique methodology for activity similarity between knowledge objects in distributed and high-dimensional domain, notably text documents. From the projected similarity live new bunch criterion functions ar developed and individual bunch algorithms ar introduced, that ar quick and climbable like k-means, however also are capable of providing high-quality and consistent performance [2].

I. INTRODUCTION Study of similarity of agglomeration was ab initio intended by an exploration on automatic text categorization of foreign language texts, as explained below. Supported varied distance measures, variety of ways are planned to handle document agglomeration. A typical and wide used distance live is that the geometrician distance. The k-means methodology is one amongst the ways that use the geometrician distance, that minimizes the add of the square geometrician distance between the information points and their corresponding cluster centers. Since the document house is often of high spatiality, it is desirable to seek out a low-dimensional illustration of the documents to scale back computation complexness.

Low computation value is achieved in spectral bunch ways, during which the documents are first projected into a low-dimensional linguistics area and so a standard bunch rule is applied to finding document clusters. Latent Semantic Indexing (LSI) is one amongst the effective spectral bunch ways, aimed toward finding the most effective approximation to the first document space by minimizing the world reconstruction error (Euclidean distance). However, due to the high spatiality of the document area, a definite illustration of documents typically reside on a nonlinear manifold embedded within the similarities between the info points. Unfortunately, the geometer distance could be a difference live that describes the dissimilarities instead of similarities between the documents. Thus, it is powerless to effectively capture the nonlinear manifold structure embedded within the similarities between them. A good document bunch methodology should be ready to realize a low-dimensional illustration of the documents that may best preserve the

II. MOTIVATION Developing strategies to prepare massive amounts of unstructured text documents into a smaller variety of meaty clusters would be terribly useful as document clump is important to such tasks as compartmentalization, filtering, automated meta knowledge generation, signified illumination, population of ranked catalogs of internet resources and application requiring document organization [1]. To spot an appropriate tool and algorithmic rule for clump that produces the most effective clump solutions, it becomes necessary to own a way for comparison the results of various clump algorithms. Though considerable work has been drained coming up with clustering algorithms, not abundant analysis has been done on formulating a live for the similarity of two completely different clustering algorithms [2].

Thus, the most goal of this paper is to: first, propose Associate in Nursing formula for activity similarity analysis among totally different cluster algorithms; second, apply the formula to calculate similarity of varied pairs of cluster strategies applied to a Portuguese corpus and therefore the Iris dataset; finally, to cross- validate the results of similarity analysis with the geometrician (centroids) distances and Pearson parametric statistic, exploitation a similar datasets.

III. RELATED WORKS Gianni Amati et al the authors were explicit that the introduce and build a framework for derivation probabilistic models of information retrieval. The models square measure statistic models of IR obtained within the language model approach. It derive term-weighting models by mensuration

All rights reserved by www.ijsrd.com

563

A Similarity Measure for Text Classification and Clustering (IJSRD/Vol. 3/Issue 08/2015/145)

the divergence of the particular term distribution from that obtained beneath a random method. Among the random processes to check the Bernoulli distribution and Bose– Einstein statistics. They outline two sorts of term frequency standardisation for standardization term weights within the document query matching method [4]. The primary standardisation assumes that documents have an equivalent length and measures the knowledge gain with the ascertained term once it is been accepted as an honest descriptor of the ascertained document. The second standardisation is expounded to the document length and to alternative statistics. These two standardisation strategies square measure applied to the fundamental models in succession to get coefficient formula. Results show that the framework produces totally different statistic models forming baseline alternatives to the quality tf-idf model [3].

J. A. Aslam et al the authors were expressed that the similarity is a crucial and wide used conception.

Previous definitions of similarity square measure tied to a specific application or a kind of information illustration.

They gift associate degree information-theoretic definition of similarity that is applicable as long as there is a probabilistic model. They demonstrate however this definition will be accustomed live the similarity in an exceedingly range of various domains. Similarity may be a elementary and wide used conception. Several similarity measures are planned, like info content mutual information, Dice constant, cos constant, distance-based measurements and have distinction model. McGill etc. surveyed and compared sixty seven similarity measures utilized in info retrieval. A drag with previous similarity measures is that every of them is tied to a specific application or assumes a specific domain model [2].

D. Cai, X. et al the authors were expressed that the propose a unique document cluster technique that aims to cluster the documents into totally different linguistics categories. The document area is usually of high spatial property and cluster in such a high dimensional area is commonly impossible thanks to the curse of spatial property [8]. By victimization Locality Preserving Indexing (LPI), the documents are often projected into a lower-dimensional linguistics area within which the documents associated with an equivalent linguistics square measure near one another.

Totally different from previous document cluster strategies supported Latent Semantic Indexing (LSI) or Nonnegative Matrix Factorization (NMF), this technique tries to find each the geometric and discriminating structures of the document area [5].

S. Clinchant and E. Gaussier the authors were declared that the family of information based models for spontanepous information retrieval. These models draw their inspiration from a long standing hypothesis in IR, specifically the very fact that the distinction within the behaviors of a word at the document and assortment levels brings info on the importance of the word for the document.

This hypothesis has been exploited within the two-poisson mixture models, within the notion of eliteness in BM25, and additional recently in DFR models. It show here that combined with notions associated with burstiness, it will cause easier and higher models [6].

I. S. Dhillon et al the authors were declared that the feature choice may be a basic step within the construction of

a vector house or bag of words model [BB99]. specifically, once the process task is to partition a given document assortment into clusters of comparable documents a alternative smart|of excellent|of fine} options at the side of good bunch algorithms is of predominant importance. This chapter suggests two techniques for feature or term choice at the side of variety of bunch methods. The choice techniques considerably scale back the dimension of the vector house model. Examples that illustrate the effectiveness of the projected algorithms area unit provided [6].

Inderjit S. et al the authors were expressed the unlabelled document collections are getting progressively common and available; mining such knowledge sets represents a serious modern challenge. Exploitation words as options, text documents area unit usually delineated as high-dimensional and thin vectors-a few thousand dimensions and a spareness of ninety five to ninety nine is typical [6]. During this paper, they study an explicit spherical k-means rule for clump such document vectors.

The rule outputs k disjoint clusters every with an inspiration vector that is the centre of mass of the cluster normalized to own unit euclidian norm. As this first contribution, it by trial and error demonstrate that, because of the highdimensionality and spareness of the text knowledge, the clusters created by the rule have an explicit “fractal-like” and “self-similar” behavior [7].

IV. EXISTING METHODOLOGY The existing system presents a replacement spectral clump technique referred to as Correlation Preserving Indexing (CPI), that is performed within the correlation similarity live area. During this system, the documents ar projected into a low-dimensional linguistics area during which the correlations between the documents within the native patches are maximized whereas the correlations between the documents outside these patches ar reduced at the same time [8]. Since the intrinsic geometrical structure of the document area is usually embedded within the similarities between the documents, correlation as a similarity live is a lot of appropriate for police work the intrinsic geometrical structure of the document area than geometrician distance [12]. The existing system is enforced within the planned system additionally. Additionally, for higher cluster, knowledge cleansing method like stemming and stop word removal is applied. Then equivalent word word replacement is formed that will increase the scientific theory among the documents. Two dimensions ar averaged into one dimension then three dimension knowledge is born-again into two dimension knowledge then clump is applied [9].

A. Drawbacks of Existing Methodology     

Preprocessing steps like stemming, stop word removal and word replacement do not seem to be thought-about.

Only text documents area unit taken. Websites or XML files don't seem to be thought-about.

How they work on different forms of distributed and high-dimensional knowledge isn't thought-about.

Considers single read purpose solely.

Two dimensional knowledge feature is taken into thought.

All rights reserved by www.ijsrd.com

564

A Similarity Measure for Text Classification and Clustering (IJSRD/Vol. 3/Issue 08/2015/145)

V. PROPOSED METHODOLOGY Text process plays a vital role in info. In text process, the bag-of-words model is often used. A document is sometimes delineate as a vector during which every element indicates the worth of the corresponding feature within the document.

The feature price is term frequency, relative term frequency (the quantitative relation between the term frequency and also the total range of occurrences of all the terms within the document set), or tf-idf. Usually, the spatiality of a document is massive and also the ensuing vector is distributed, i.e., most of the feature values within the vector square measure zero [10]. The planned system given the new live for computing the similarity between two documents. Many characteristics embedded during this measure. It is a biradial live. The distinction between presence and absence of a feature is taken into account additional essential than the distinction between the values related to a gift feature [14].

The similarity will increase because the distinction between the two values related to a gift feature decreases. what is more, the contribution of the distinction is generally scaled. The similarity decreases once the quantity of presence-absence options will increase [11]. Associate in Nursing absent feature has no contribution to the similarity.

The projected live is extended to measure the similarity between two sets of documents. The live is applied in many text applications, together with single label classification, multi-label classification and hierarchal agglomerate cluster, and also the results obtained demonstrate the effectiveness of the projected similarity live [15].  To check connected document with varied content word word is provided.  To find similarity between the documents relation specific or dependent and freelance similarity is enforced.  For clustering method lexical pattern clump construct is enforced.  Stemming is applied before text documents ar taken.  Stop word removal is applied that reduces the content size.  Synonym word replacement is applied in order that connected documents with varied contents conjointly coincide.  Two documents could have an exact price of cos similarity, however if neither of them is within the alternative one neighborhood, they need no VI. CONCLUSION The thesis presents a unique similarity live between two documents. Many fascinating properties embedded during this measure. For instance, the similarity live is symmetrical.

The presence or absence of a feature is taken into account additional essential than the distinction between the values related to a gift feature. The similarity degree will increase once the amount of presence-absence options try decreases.

The project proposes Associate in Nursing empirical methodology to estimate linguistics similarity victimisation page counts and text snippets retrieved from an online programme for two words. Specifically, it defines numerous word co-occurrence measures victimisation page counts and integrates those with lexical patterns extracted from text

snippets. Both page counts based co-occurrence measures and lexical pattern clusters were used to define features for a word pair.

REFERENCES [1] G. Amati and C. J. V. Rijsbergen, “Probabilistic models of information retrieval based on measuring the divergence from randomness,” ACM Trans. Inform.

Syst., vol. 20, no. 4, pp. 357–389, 2002.

[2] J. A. Aslam and M. Frost, “An information-theoretic measure for document similarity,” in Proc. 26th SIGIR, Toronto, ON, Canada, 2003, pp. 449–450.

[3] D. Cai, X. He, and J. Han, “Document clustering using locality preserving indexing,” IEEE Trans. Knowl.

Data Eng., vol. 17, no. 12, pp. 1624–1637, Dec. 2005.

[4] S. Clinchant and E. Gaussier, “Information-based models for ad hoc IR,” in Proc. 33rd SIGIR, Geneva, Switzerland, 2010, pp. 234–241.

[5] I. S. Dhillon, J. Kogan, and C. Nicholas, “Feature selection and document clustering,” in A Comprehensive Survey of Text Mining, M. W. Berry, Ed. Heidelberg, Germany: Springer, 2003.

[6] I.S. Dhillon and D.S. Modha.Concept decompositions for large sparse text data using clustering.Machine Learning, 42(1):143–175, January 2001.Also appears as IBM Research Report RJ 10147, July 1999.

[7] I. S. Dhillon and D. S. Modha, “Concept decompositions for large sparse text data using clustering,” Mach. Learn., vol. 42, no. 1, pp. 143–175, 2001.

[8] Broder, A. Z., S. C. Glassman, M. S. Manasse, and G. Zweig: 1997, `Syntactic clustering of the web'.

Technical Report 1997-015, Digital Systems Research Center.

[9] Dhillon, I. S., D. S. Modha, and W. S. Spangler: 1998,`Visualizing Class Structure of Multidimensional Data'. In: S.Weisberg (ed.): Proceedings of the 30th Symposium on the Interface: Computing Science and Statistics, Vol. 30. Minneapolis, MN, pp. 488-493.

[10] H. Fang, T. Tao, and C. Zhai, “A formal study of heuristic retrieval constraints,” in Proc. 27th SIGIR, Sheffield, South Yorkshire, U.K., 2004, pp. 49–56.

[11] G. Salton. Automatic Text Processing: The Transformation, Analysis and Retrieval of Information

by Computer. Addison-Wesley, 1989.

[12] J. Lafferty and C. Zhai. Probabilistic relevance models based on document and query generation. In W. B. Croft and J. Lafferty, editors, Language Modeling and Information Retrieval. Kluwer Academic Publishers, 2003.

[13] N. Fuhr. Language models and uncertain inference in information retrieval. In Proceedings of the Language Modeling and IR workshop.

[14] C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of SIGIR’01, pages 334–342, Sept 2001.

[15] C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, “Fair and Balanced?: Bias in Bug-Fix Data Sets,” Proc. Seventh European Software Eng. Conf. and Symp. Foundations of Software Eng., pp. 121-

All rights reserved by www.ijsrd.com

565