Spaces:
Runtime error
Runtime error
| trec_eval is the standard tool used by the TREC community for | |
| evaluating an ad hoc retrieval run, given the results file and a | |
| standard set of judged results. | |
| ------------------------------------------------------------------------------ | |
| Installation: Should be as easy as typing "make" in the source directory. | |
| If you wish the trec_eval binary to be placed in a standard location, alter | |
| the first line of Makefile appropriately. | |
| ------------------------------------------------------------------------------ | |
| Testing: sample input and output files are included in the directory test. | |
| "make quicktest" will perform some sample simple evaluations and compare | |
| the results. | |
| ------------------------------------------------------------------------------ | |
| Usage: Most options can be ignored. The only one most folks will need | |
| is the "-q" flag, to indicate whether to output official results for individual | |
| queries as well as the averages over all queries. Official TREC usage | |
| might be something like | |
| trec_eval -q -c -M1000 official_qrels submitted_results | |
| to ensure correct evaluation if submitted_results doesn't have results | |
| for all queries, or returns more than 1000 documents per query. | |
| If you wish to output only one particular measure: | |
| trec_eval -m measure[.params] rel_info_file results_file | |
| ------------------------------------------------------------------------------ | |
| Change Log (only recent) | |
| ------------------------------------------------------------------------------ | |
| 12/31/08 | |
| comments and documentation of Zscore file format corrected. | |
| trec_eval.c get_zscores.c | |
| 2/25/08 Version 9.0alpha. | |
| Complete rewrite of entire trec_eval (needed for a long time!). | |
| Complete separation of individual measure calculations - | |
| computers are now fast enough so can afford recalculation of lots | |
| of intermediate values. | |
| Should be much easier to add measures to, and much easier to add | |
| new input file formats with associated measures. | |
| Parameters for measures (eg, cutoffs for P) can be specified on the | |
| command line. | |
| Choice of measures can be specifed on the command line. | |
| An initial set of preference evaluation measures (with their own | |
| input rel_info format) have been added. | |
| Help now gives targeted measure and format descriptions. Try | |
| trec_eval -h -m all_prefs -R prefs | |
| to get info on preference measures and formats, for instance. | |
| All internal calculations are in double rather than float. Yields | |
| minor variations in output at rare times; mostly when going from a | |
| double percentage to a corresponding doc cutoff (eg, in iprec_at_recall). | |
| All globally known procedure names or variables now begin with 'te_' | |
| to allow incorporation of procedures in other programs. | |
| Measures added: | |
| ndcg, ndcg_cut, set_F, success, map_avgjg, P_avgjg, | |
| various preference evaluation measures. | |
| Measures renamed: | |
| Rprec-mult_* was *R-prec | |
| set_P was exact_prec | |
| set_recall was exact_recall | |
| set_relative_P was exact_relative_prec | |
| set_recall was exact_recall | |
| set_map was exact_unranked_avg_prec | |
| gm_map was gm_ap | |
| 11pt_avg was 11-pt_avg | |
| P_* was P* | |
| recall_* was recall* | |
| relative_P_* was relative_P* | |
| iprec_at_recall_* was ircl_prn.* | |
| Measures dropped for now: | |
| 3-pt_avg | |
| avg_doc_prec | |
| avg_relative_prec | |
| exact_relative_unranked_avg_prec | |
| map_at_R | |
| int_map | |
| exact_int_R_rcl_prec | |
| int_map_at_R | |
| unranked_avg_prec* | |
| relative_unranked_avg_prec* | |
| rcl_at_142_nonrel | |
| fallout_recall_* | |
| int_*R-prec | |
| micro_prec | |
| micro_recall | |
| micro_bpref | |
| bpref variants | |
| time base measures. | |
| Input formats added: | |
| prefs - allows expression of preferences | |
| qrels_prefs - same as standard qrels, except treated as prefs | |
| qrels_jg - same as standard qrels, except allows judgment sets | |
| from multiple users (judgment groups). | |
| Version 8.1, Added infAP, minor bug fixes | |
| 7/24/06 Improved infAP comments (implementation verified by Yilmaz). | |
| trec_eval_help.c: allow longer measure explanations. | |
| 6/27/06 get_opt.c Fixed error message | |
| 6/22/06 Added measure infAP (Aslam et al) to allow judging only sample | |
| of pools. -1 for rel in qrels file interpreted as pool doc not judged. | |
| 6/22/06 trvec_teval.c: fixed bugs in calculation of bpref if multiple | |
| relevance levels were used and a non-default relevance level | |
| was given. (Eg. A doc with rel level of 2 was counted as unjudged | |
| rather than judged nonrel if a relevance level of 3 was needed | |
| to consider relevant.) | |
| 4/5/06 Changed comments in README, trec_eval.c, trec_eval_help.c files | |
| which incorrectly claimed queries with no relevant docs are | |
| ignored (this was true with very old versions of trec_eval). Now | |
| reads that queries with no relevance information are ignored. | |
| Giorgio Di Nunzio and Nicola Ferro, | |
| ------------------------------------------------------------------------------ | |
| Version 8.0, full bpref bug fix, see file bpref_bug. I decided to up the | |
| version number since bpref results are incompatible with previous | |
| results (though the changes are small). | |
| ------------------------------------------------------------------------------ | |
| ------------------------------------------------------------------------------ | |
| Files: | |
| Main procedure: | |
| trec_eval.c | |
| --- | |
| Procedures to read input files of various formats: | |
| formats.c Mapping names of input file formats to input procedures | |
| get_qrels.c Read the standard judged documents (qrels format) | |
| get_qrels_jg.c Read qrels format with multiple judgment groups per query | |
| get_prefs.c Read preferences judgments instead of doc judgments - see | |
| formats.c for full description. | |
| get_qrels_prefs.c Read qrels_jg format file, interpret as prefs file. | |
| get_trec_results.c Read the standard result file (trec_results format). | |
| --- | |
| Procedures to merge rel_info and results from input form into form that measures | |
| can easily use, if they wish: | |
| form_res_rels.c 'qrels' and 'trec_results' into RES_RELS format. | |
| form_res_rels_jg.c 'qrels_jg' and 'trec_results' into RES_RELS_JG format. | |
| form_pref_counts.c ('prefs' or 'qrels_prefs') and 'trec_results' format | |
| --- | |
| The actual measures: | |
| measures.c Associates measure name with parameters and | |
| init, calculation, accumulation, printing procedures | |
| meas_*.c Common procedures used by many measures for init, acc, printing. | |
| m_<measure_name>.c measure specific procedures | |
| --- | |
| Miscellaneous: | |
| Makefile Compile and test trec_eval | |
| README This file | |
| CHANGELOG Recent changes | |
| test Directory of collection of sample input and output for trec_eval | |
| trec_eval.h Basic evaluation structures. | |
| functions.h Prototype decorations of measure procedures. | |
| sysfunc.h | |
| common.h | |
| bpref_bug: Description of bug in bpref that existed in trec_eval versions 6 | |
| through 7.3. | |
| ------------------------------------------------------------------------------ | |
| ------------------------------------------------------------------------------ | |
| Adding a new measure. | |
| Assuming it uses standard input files: | |
| 1. In m_<new_measure>.c, write any needed measure specific procedures needed to | |
| initialize measure | |
| calculate measure | |
| accumulate measure (adding one topic's value to summary value over topics) | |
| calculate the ending average for a measure. | |
| print single | |
| print final query measure value (and cleanup if needed) | |
| Most measures require only a new calculate measure procedure - the | |
| other procedures are generic and already implemented depending on the type of | |
| measure (has cutoffs and value for each cutoff, parameters, is a float, etc). | |
| See functions.h to see fit for these generic procedures. | |
| 2. In same file, construct TREC_MEAS te_meas* entry pointing to above | |
| procedures and any default cutoffs or parameters. | |
| 3. Add pointer to that TREC_MEAS entry in "measures.c" | |
| 4. Add measure source file to Makefile | |
| ------------------------------------------------------------------------------ | |
| Adding a new file format | |
| 1. Implement reading of new format in get_<new_format>.c, with returned top | |
| level output of type ALL_REL_INFO or ALL_RESULTS. The individual topic | |
| returned values will be in a format dependent form which will be passed | |
| to the appropriate measures. | |
| 2. Add format to appropriate format list in formats.c | |
| 3. Add measures to take advantage of format (see above) | |
| 4. To use, invoke trec_eval with -R or -T values, and -m measures that | |
| are appropriate. | |