Spaces:
Sleeping
Sleeping
File size: 3,330 Bytes
b028d48 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
This code is the lexical complexity analyzer described in
Lu, Xiaofei (2012). The relationship of lexical richnes to the quality
of ESL speakers' oral narratives. The Modern Language Journal, 96(2), 190-208.
Version 1.1 Released on February 12, 2013
Copyright (C) 2013 Xiaofei Lu
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 2 of the License, or (at your option) any later
version.
This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
details.
You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 59 Temple
Place, Suite 330, Boston, MA 02111-1307 USA
To download the latest version of this software, follow the appropriate link
at
http://www.personal.psu.edu/xxl13/download.html
1. About
This tool computes the lexical complexity of English texts using 25 different
measures. Information on the measures can be found in Lu (2012). This
tool uses frequency lists derived from the British National Corpus and the
American National Corpus.
2. Running the tool
2.1 Input files: All input files must be POS-tagged and lemmatized first and
must be in the following format (see files in the samples folder for
examples). The file should contain a minumum of 50 words.
lemma_pos lemma_pos lemma_pos ...
or
lemma_pos
lemma_pos
lemma_pos
You can use any POS tagger and lemmatizer, as long as the Penn Treebank POS
tagset is adopted and the input file is appropriately formatted. In Lu
(2012), the following POS tagger and lemmaitzer were used:
The Stanford POS tagger:
http://nlp.stanford.edu/software/tagger.shtml
MORPHA:
http://www.informatics.susx.ac.uk/research/groups/nlp/carroll/morph.html
2.2 Analyzing a single file: To get the lexical complexity of a single file,
run the following from this directory. Replace input_file with the actual
name of your input file and output_file with the desired name of your output
file.
python lc.py input_file > output_file
e.g.,
python lc.py samples/1.lem > 1.lex
To use the American National Corpus (ANC) wordlist instead of the BNC wordlist
for lexical sophistication analysis, use the lc-anc.py script, e.g.,
python lc-anc.py samples/1.lem > 1-anc.lex
2.3 Analyzing multiple files: To get the lexical complexity of two or more
files within a single folder, run the following from this directory. Replace
path_to_folder with the actual path to the folder that contains your files
and output_file with the desired name of your output file. The folder should
only contain the files you want to analyze.
python folder-lc.py path_to_folder > output_file
e.g.,
python folder-lc.py samples/ > samples.lex
To use the American National Corpus (ANC) wordlist instead of the BNC wordlist
for lexical sophistication analysis, use the folder-lc-anc.py script, e.g.,
python folder-lc-anc.py samples/ > samples-anc.lex
2.4 Using the output: The output file is comma-delimited and can be loaded to
excel and spss directly for analysis.
|