---
abstract: 'Many academic journals ask their authors to provide a list of about five to fifteen keywords, to appear on the first page of each article. Since these key words are often phrases of two or more words, we prefer to call them keyphrases. There is a wide variety of tasks for which keyphrases are useful, as we discuss in this paper. We approach the problem of automatically extracting keyphrases from text as a supervised learning task. We treat a document as a set of phrases, which the learning algorithm must learn to classify as positive or negative examples of keyphrases. Our first set of experiments applies the C4.5 decision tree induction algorithm to this learning task. We evaluate the performance of nine different configurations of C4.5. The second set of experiments applies the GenEx algorithm to the task. We developed the GenEx algorithm specifically for automatically extracting keyphrases from text. The experimental results support the claim that a custom-designed algorithm (GenEx), incorporating specialized procedural domain knowledge, can generate better keyphrases than a general-purpose algorithm (C4.5). Subjective human evaluation of the keyphrases generated by GenEx suggests that about 80% of the keyphrases are acceptable to human readers. This level of performance should be satisfactory for a wide variety of applications. '
altloc:
- http://extractor.iit.nrc.ca/reports/IR2000.html
chapter: ~
commentary: ~
commref: ~
confdates: ~
conference: ~
confloc: ~
contact_email: ~
creators_id: []
creators_name:
- family: Turney
given: Peter
honourific: ''
lineage: ''
date: 2000
date_type: published
datestamp: 2001-09-13
department: ~
dir: disk0/00/00/17/97
edit_lock_since: ~
edit_lock_until: ~
edit_lock_user: ~
editors_id: []
editors_name: []
eprint_status: archive
eprintid: 1797
fileinfo: /style/images/fileicons/application_postscript.png;/1797/1/IR2000.ps|/style/images/fileicons/application_pdf.png;/1797/5/IR2000.pdf
full_text_status: public
importid: ~
institution: ~
isbn: ~
ispublished: pub
issn: ~
item_issues_comment: []
item_issues_count: 0
item_issues_description: []
item_issues_id: []
item_issues_reported_by: []
item_issues_resolved_by: []
item_issues_status: []
item_issues_timestamp: []
item_issues_type: []
keywords: 'machine learning, summarization, indexing, keywords, keyphrase extraction.'
lastmod: 2011-03-11 08:54:47
latitude: ~
longitude: ~
metadata_visibility: show
note: ~
number: 4
pagerange: 303-336
pubdom: FALSE
publication: Information Retrieval
publisher: Kluwer
refereed: TRUE
referencetext: |-
Brandow, R., Mitze, K., and Rau, L.R. (1995). The automatic condensation of electronic publications
by sentence selection. Information Processing and Management, 31 (5), 675-685.
Breiman, L. (1996a). Arcing Classifiers. Technical Report 460, Statistics Department, University of
California at Berkeley.
Breiman, L. (1996b). Bagging predictors. Machine Learning, 24 (2), 123-140.
Buntine, W. (1989). Stratifying samples to improve learning. In Proceedings of the IJCAI-89 Work-shop
on Knowledge Discovery in Databases. Detroit, Michigan.
Carter, C., and Catlett, J. (1987). Assessing credit card applications using machine learning. IEEE
Expert, Fall issue, 71-79.
Catlett, J. (1991). Megainduction: Machine Learning on Very Large Databases. Ph.D. Dissertation,
Basser Department of Computer Science, University of Sydney.
Croft, W.B., Turtle, H., and Lewis, D. (1991). The use of phrases and structured queries in information
retrieval. SIGIR-91: Proceedings of the 14th Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval, pp. 32-45, New York: ACM.
Deming, W.E. (1978). Sample surveys: The field. In William H. Kruskal and Judith M. Tanur (Ed.),
International Encyclopedia of Statistics. New York: Free Press.
Edmundson, H.P. (1969). New methods in automatic extracting. Journal of the Association for Com-puting
Machinery, 16 (2), 264-285.
Fagan, J.L. (1987). Experiments in Automatic Phrase Indexing for Document Retrieval: A Comparison
of Syntactic and Non-Syntactic Methods. Ph.D. Dissertation, Department of Computer Science,
Cornell University, Report #87-868, Ithaca, New York.
Feelders, A. and Verkooijen, W. (1995). Which method learns the most from data? Methodological
issues in the analysis of comparative studies. Fifth International Workshop on Artificial Intelli-gence
and Statistics, Ft. Lauderdale, Florida, pp. 219-225.
Field, B.J. (1975). Towards automatic indexing: Automatic assignment of controlled-language index-ing
and classification from free indexing. Journal of Documentation, 31 (4), 246-265.
Frank, E., Paynter, G.W., Witten, I.H., Gutwin, C., and Nevill-Manning, C.G. (1999). Domain-specific
keyphrase extraction. Proceedings of the Sixteenth International Joint Conference on Artificial
Intelligence (IJCAI-99), pp. 668-673. California: Morgan Kaufmann.
Fraser, D.A.S. (1976). Probability and Statistics: Theory and Applications. Massachusetts: Duxbury
Press.
Freund, Y., and Schapire, R.E. (1996). Experiments with a new boosting algorithm. Machine Learn-ing:
Proceedings of the Thirteenth International Conference (ICML-96), pp. 148-156. California:
Morgan Kaufmann.
Ginsberg, A. (1993). A unified approach to automatic indexing and information retrieval. IEEE
Expert, 8, 46-56.
Grefenstette, J.J. (1983). A user’s guide to GENESIS. Technical Report CS-83-11, Computer Science
Department, Vanderbilt University.
Grefenstette, J.J. (1986). Optimization of control parameters for genetic algorithms. IEEE Transac-tions
on Systems, Man, and Cybernetics, 16, 122-128.
Gutwin, C., Paynter, G.W., Witten, I.H., Nevill-Manning, C.G., and Frank, E. (1999). Improving
browsing in digital libraries with keyphrase indexes. Decision Support Systems. In press.
Jang, D.-H., and Myaeng, S.H. (1997). Development of a document summarization system for effec-tive
information services. RIAO 97 Conference Proceedings: Computer-Assisted Information
Searching on Internet, pp. 101-111. Montreal, Canada.
Johnson, F.C., Paice, C.D., Black, W.J., and Neal, A.P. (1993). The application of linguistic process-ing
to automatic abstract generation. Journal of Document and Text Management, 1, 215-241.
Krovetz, R. (1993). Viewing morphology as an inference process. Proceedings of the Sixteenth Annual
International ACM SIGIR Conference on Research and Development in Information Retrieval,
SIGIR'93, 191-203.
Krulwich, B., and Burkey, C. (1996). Learning user information interests through the extraction of
semantically significant phrases. In M. Hearst and H. Hirsh, editors, AAAI 1996 Spring Sympo-sium
on Machine Learning in Information Access. California: AAAI Press.
Krupka, G. (1995). SRA: Description of the SRA system as used for MUC-6. Proceedings of the Sixth
Message Understanding Conference. California: Morgan Kaufmann.
Kubat, M., Holte, R., and Matwin, S. (1998). Machine learning for the detection of oil spills in satel-lite
radar images. Machine Learning, 30 (2/3), 195-215.
Kupiec, J., Pedersen, J., and Chen, F. (1995). A trainable document summarizer. In E.A. Fox, P. Ingw-ersen,
and R. Fidel, editors, SIGIR-95: Proceedings of the 18th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval, pp. 68-73, New York: ACM.
Leung, C.-H., and Kan, W.-K. (1997). A statistical learning approach to automatic indexing of con-trolled
index terms. Journal of the American Society for Information Science, 48 (1), 55-66.
Lovins, J.B. (1968). Development of a stemming algorithm. Mechanical Translation and Computa-tional
Linguistics, 11, 22-31.
Luhn, H.P. (1958). The automatic creation of literature abstracts. I.B.M. Journal of Research and
Development, 2 (2), 159-165.
Maclin, R., and Opitz, D. (1997). An empirical evaluation of bagging and boosting. Proceedings of the
Fourteenth National Conference on Artificial Intelligence (AAAI-97), pp 546-551. AAAI Press.
Marsh, E., Hamburger, H., and Grishman, R. (1984). A production rule system for message summari-zation.
In AAAI-84, Proceedings of the American Association for Artificial Intelligence, pp. 243-
246. Cambridge, MA: AAAI Press/MIT Press.
Mathieu, J. (1999). Adaptation of a keyphrase extractor for Japanese text. Proceedings of the 27th
Annual Conference of the Canadian Association for Information Science (CAIS-99), Sherbrooke,
Quebec, pp. 182-189.
MUC-3. (1991). Proceedings of the Third Message Understanding Conference. California: Morgan
Kaufmann.
MUC-4. (1992). Proceedings of the Fourth Message Understanding Conference. California: Morgan
Kaufmann.
MUC-5. (1993). Proceedings of the Fifth Message Understanding Conference. California: Morgan
Kaufmann.
MUC-6. (1995). Proceedings of the Sixth Message Understanding Conference. California: Morgan
Kaufmann.
Muñoz, A. (1996). Compound key word generation from document databases using a hierarchical
clustering ART model. Intelligent Data Analysis, 1 (1), Amsterdam: Elsevier.
Nakagawa, H. (1997). Extraction of index words from manuals. RIAO 97 Conference Proceedings:
Computer-Assisted Information Searching on Internet, pp. 598-611. Montreal, Canada.
Paice, C.D. (1990). Constructing literature abstracts by computer: Techniques and prospects. Informa-tion
Processing and Management, 26 (1), 171-186.
Paice, C.D., and Jones, P.A. (1993). The identification of important concepts in highly structured tech-nical
papers. SIGIR-93: Proceedings of the 16th Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval, pp. 69-78, New York: ACM.
Porter, M.F. (1980). An algorithm for suffix stripping. Program; Automated Library and Information
Systems, 14 (3), 130-137.
Quinlan, J.R. (1987). Decision trees as probabilistic classifiers. In Langley, P. (Ed.), Proceedings of
the Fourth International Workshop on Machine Learning, pp. 31-37. California: Morgan Kauf-mann.
Quinlan, J.R. (1990). Probabilistic decision trees. In Y. Kodratoff and R.S. Michalski, (Eds.), Machine
Learning: An Artificial Intelligence Approach, Volume III, pp. 140-152, California: Morgan Kauf-mann.
Quinlan, J.R. (1993). C4.5: Programs for machine learning. California: Morgan Kaufmann.
Quinlan, J.R. (1996). Bagging, boosting, and C4.5. In Proceedings of the Thirteenth National Confer-ence
on Artificial Intelligence (AAAI-96), pp. 725-730. AAAI Press.
Salton, G. (1988). Syntactic approaches to automatic book indexing. Proceedings of the 26th Annual
Meeting of the Association for Computational Linguistics, pp. 120-138. New York: ACM.
Salton, G., Allan, J., Buckley, C., and Singhal, A. (1994). Automatic analysis, theme generation, and
summarization of machine-readable texts. Science, 264, 1421-1426.
Soderland, S., and Lehnert, W. (1994). Wrap-Up: A trainable discourse module for information
extraction. Journal of Artificial Intelligence Research, 2, 131-158.
Sparck Jones, K. (1973). Does indexing exhaustivity matter? Journal of the American Society for
Information Science, September-October, 313-316.
Steier, A. M., and Belew, R. K. (1993). Exporting phrases: A statistical analysis of topical language.
In R. Casey and B. Croft, editors, Second Symposium on Document Analysis and Information
Retrieval, pp. 179-190.
Turney, P.D. (1997). Extraction of Keyphrases from Text: Evaluation of Four Algorithms. National
Research Council, Institute for Information Technology, Technical Report ERB-1051.
Turney, P.D. (1999). Learning to Extract Keyphrases from Text. National Research Council, Institute
for Information Technology, Technical Report ERB-1057.
Whitley, D. (1989). The GENITOR algorithm and selective pressure. Proceedings of the Third Inter-national
Conference on Genetic Algorithms (ICGA-89), pp. 116-121. California: Morgan Kauf-mann.
relation_type: []
relation_uri: []
reportno: ~
rev_number: 14
series: ~
source: ~
status_changed: 2007-09-12 16:40:37
subjects:
- comp-sci-lang
- comp-sci-mach-learn
- comp-sci-stat-model
succeeds: ~
suggestions: ~
sword_depositor: ~
sword_slug: ~
thesistype: ~
title: Learning algorithms for keyphrase extraction
type: journalp
userid: 2175
volume: 2