The Negative Binomial Model of Word Usage


Abstract


How people make texts is a rarely asked but central question in linguistics. From the language engineering perspective texts are outcomes of stochastic processes. A cognitive approach in linguistics holds that the speaker’s intention is the key to text formation. We propose a biologically inspired statistical formulation of word usage that brings together these views. We have observed that in several multilingual text collections1 word frequency distributions in a majority of non-rare words fit the negative binomial distribution (NBD). However, word counts in artificially randomized (permuted) texts agree with the geometric distribution. We conclude that the parameters of NBD deal with linguistic features of words. Specifically, named entities tend to have similar parameter values. We suggest that the NDB in natural texts accounts for intentionality of the speaker.

DOI Code: 10.1285/i20705948v6n1p84

Keywords: word frequency; negative binomial distribution (NBD); text randomization

References


.Von Humboldt, Wilheim. (2010). Ankündigung einer Schrift über die Vaskische Sprache und Nation, nebst Angabe des Geschichtspunctes und Inhalts derselben. In Werke. Vol. 5. Berlin: Wissenschaftliche Buchgesellschaft.

.Driem, George. (2008). The Language Organism: Parasite or Mutualist? In Evidence and counter-evidence: essays in honour of Frederik Kortlandt, ed. Frederik Kortlandt and Alexander Lubotsky, 101-113. Amsterdam; New York: Podopi.

.Propp, Vladimir. (1968). Morphology of the folktale. Publications of the American Folklore Society, v. 9. Austin, University of Texas Press.

.Joseph, Brian D., and Richard D. Janda. (2003). The handbook of historical linguistics. Wiley-Blackwell.

.Steiner, Peter, and Sergej Davydov. (1977). The Biological Metaphor in Russian Formalism: The Concept of Morphology. SubStance 6/7, no. 16 (July 1), 149-158. doi:10.2307/3684133.

.Gillman, Michael, and Rosemary Hails. (1997). An introduction to ecological modelling: putting practice into theory. Wiley-Blackwell.

.Katz, Slava M. (1996). Distribution of content words and phrases in text and language modelling. Natural Language Engineering 2 (March), 15–59. doi:10.1017/S1351324996001246.

.Leontiev, A.A. (1997). Osnovy psikholingvistiki. Moskva: Nauka.

.Sakharnyj, L.V., and A.S. Shtern. (1988). Nabor kl'uchevyh slov kak tip teksta. In Leksicheskije aspekty v sisteme professionalno-orientirovannogo obuchenija, 34-51. Perm, Permskij politekhnicheskij universitet.

.Stubbs, Michael. (2001). Words and phrases: corpus studies of lexical semantics. Oxford [England]: Blackwell.

.Li, Wentian. (1992). Random texts exhibit Zipf's-law-like word frequency distribution. IEEE TRANSACTIONS ON INFORMATION THEORY: 1842-1845.

.Bart, A.G. (2003). Analiz mediko-biologicheskikh sistem. Sankt-Peterburg: Sankt-Peterburgskij Gos.Universitet.

.Bart, A.G., and J. Minář. (1984). Basic regulatory parameters of the host-parasite system for warble flies of farm animals using Hypoderma bovis as an example. Folia Parasitologica (Praha) 31, 277-287.

.Breev, K.A. (1968). O raspredelenii lichinok podkozhnykh ovodov v stadakh krupnogo rogatogo skota: Negativnoje binomialnoje raspredelenije kak model. Parasitologija, no. 2, 381-394.

.Popescu, Ioan-Ioviț, Ján Mačutek, and Gabriel Altmann. (2009). Aspects of word frequencies. Lüdenscheid: RAM-Verlag.

.Afifi, Abdelmonem A., and Azen, S. P. (1979). Statistical analysis: a computer oriented approach. Academic Press.

.Garside, R., and N. Smith. (1997). A hybrid grammatical tagger: CLAWS4. In Corpus Annotation: Linguistic Information from Computer Text Corpora, ed. R. Garside, G. Leech, and A. McEnery, 102-121. London: Longman.

.Wittgenstein, Ludwig. 1942. The Blue and Brown Books. London: HarperCollins.

.Aufrecht, Theodor, ed. 1877. Die Hymnen des Rigveda. Bonn: A. Marcus.

.Elizarenkova, Tatyana, trans. 1989ff. Rigveda. Moskva: Nauka

.Griffith, Ralph. 1963. The Hymns of the Rigveda. 2 vols. Varanasi: C.S.S.

.The St. Petersburg Times Archive used in this article is available at http://times.spb.ru/index.php?action_id=5&i_number=1652&year=2010 (accessed May 01, 2011)


Full Text: PDF
Bookmark and Share


Creative Commons License
This work is licensed under a Creative Commons Attribuzione - Non commerciale - Non opere derivate 3.0 Italia License.