How to install
- Download from official site
- Extract
- run make
Usage
$ ./trec_eval [-q] [-m measure] qrel.txt results.txt
where:
- qrel.txt contains the correct answers
- results.txt are the results of your search engine
Your program should construct results.txt
qrel.txt:
query-id 0 document-id relevance
- query-id: is the query id
- 0: is reserved and is not used
- document-id: is the document id
- relevance: should be 1 for relevant documents and 0 for non relevant documents
results.txt:
query-id Q0 document-id rank score STANDARD
- query-id: is the query id
- Q0: is the run id (unique for the document)
- document-id: is the document id of a relevant document
- rank: is the rank of the id (unique, ascending)
- score: is the similarity score (descending)
- STANDARD: is a custom tag (ignored)
Evaluation measures:
| runid | Name of the run (is the name given on the last field of the results file) |
| num_q | Total number of evaluated queries |
| num_ret | Total number of retrieved documents |
| num_rel | Total number of relevant documents (according to the qrels file) |
| num_rel_ret | Total number of relevant documents retrieved (in the results file) |
| map | Mean average precision (map) |
| gm_map | Average precision. Geometric mean |
| Rprec | Precision of the first R documents, where R are the number os relevants |
| bpref | Binary preference |
| recip_rank | Reciprical Rank |
| iprec_at_recall_0.00 | Interpolated Recall – Precision Averages at 0.00 recall |
| iprec_at_recall_0.10 | Interpolated Recall – Precision Averages at 0.10 recall |
| iprec_at_recall_0.20 | Interpolated Recall – Precision Averages at 0.20 recall |
| iprec_at_recall_0.30 | Interpolated Recall – Precision Averages at 0.30 recall |
| iprec_at_recall_0.40 | Interpolated Recall – Precision Averages at 0.40 recall |
| iprec_at_recall_0.50 | Interpolated Recall – Precision Averages at 0.50 recall |
| iprec_at_recall_0.60 | Interpolated Recall – Precision Averages at 0.60 recall |
| iprec_at_recall_0.70 | Interpolated Recall – Precision Averages at 0.70 recall |
| iprec_at_recall_0.80 | Interpolated Recall – Precision Averages at 0.80 recall |
| iprec_at_recall_0.90 | Interpolated Recall – Precision Averages at 0.90 recall |
| iprec_at_recall_1.00 | Interpolated Recall – Precision Averages at 1.00 recall |
| P_5 | Precision of the 5 first documents |
| P_10 | Precision of the 10 first documents |
| P_15 | Precision of the 15 first documents |
| P_20 | Precision of the 20 first documents |
| P_30 | Precision of the 30 first documents |
| P_100 | Precision of the 100 first documents |
| P_200 | Precision of the 200 first documents |
| P_500 | Precision of the 500 first documents |
| P_1000 | Precision of the 1000 first documents |
• MemExt - Online notes • 