The effects of stemming, stopwords and multi-word phrases on Zipfian distributions of text
Public Deposited- Last Modified
- February 28, 2019
- Creator
-
Sharek, Zachariah
- Affiliation: School of Information and Library Science
- Abstract
- This study examines two questions: Does the application of stemming, stopwords and multi-word phrases, alone and in combination preserve the a text's Zipfian distribution? Further, can we predict the changes to the coefficients of the Zipfian distribution when these operations are applied? The study finds that stemming, stopwords and multi-word phrases preserve the Zipfian distribution of the experimental texts. A model, predicting the changes to the Zipfian distribution under these operations is produced, and tested against further texts. Using statistical analysis, it is shown that the effects of stemming and stopwords can be predicted, but the effects of multi-word phrases cannot.
- Date of publication
- November 2002
- Subject
- DOI
- Resource type
- Rights statement
- In Copyright
- Advisor
- Losee, Robert
- Degree granting institution
- University of North Carolina at Chapel Hill
- Language
- Extent
- 65 p.
- Access right
- Open access
- Date uploaded
- December 10, 2010
Relations
- Parents:
This work has no parents.
Items
| Thumbnail | Title | Date Uploaded | Visibility | Actions |
|---|---|---|---|---|
|
|
The effects of stemming, stopwords and multi-word phrases on Zipfian distributions of text | 2019-05-13 | Public | Download |