autoevaluator
HF Staff
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator
691d364
| license: | |
| - apache-2.0 | |
| - bsd-3-clause | |
| tags: | |
| - summarization | |
| - summary | |
| - booksum | |
| - long-document | |
| - long-form | |
| datasets: | |
| - kmfoda/booksum | |
| metrics: | |
| - rouge | |
| inference: false | |
| model-index: | |
| - name: pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP13 | |
| results: | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: samsum | |
| type: samsum | |
| config: samsum | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 24.4101 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjhmM2NiMDQ1NjI3Zjk4YjkyMTVkMmUwZDU2YWMwZjc4ZmIzMjA1OGZiYzRmNjI3NDk3OWNmOTlkZDMxZmViMyIsInZlcnNpb24iOjF9.wS774e7vxQrf2gCcPhySsET3UaiUsj8E7mQmBS84wz86aT9j1yCqVX-8ozuj896K5wMygbL-TpUbydRIyyHTDw | |
| - type: rouge | |
| value: 5.003 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTYyNTFkNWZhOTgwNDg5ZWU5Y2U5NGI4Y2Y2YTMxNjUzOWI0ZWNlNDE1OGYzMjA1YTBmNDE4ZjcyOTZmODE4NiIsInZlcnNpb24iOjF9.AuqDkCgUgDWl8vMyrjTh59QW741UssGxdBqj3GZKy5e5gKadClUA709qgKbpxPIbMEyk38yvXYGplaJf5CnCCA | |
| - type: rouge | |
| value: 17.2544 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTBmODZmNWRhMzBhY2MzOGRkZWQzNjAzMGViOGMxYWYyZjNlZmM4YzgzMjkxNTk3M2E1ODAwZjY1M2I2MDZkYyIsInZlcnNpb24iOjF9.Md52aHjujvkxaW-ubJNquiHHHgi-OfRav0ZElVvYhIpU_k0iKEaQZRcw9JYjtG5vZJbQeiWbMzcCOJ999DhrAA | |
| - type: rouge | |
| value: 20.9183 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMDJjNDc1OTZjY2VmNWRhNmYwZjRjY2JmNTAyNmIwZjRhYjMyMTdlNzY2M2Q4OGQwNTEyYTU0NGVhYWI2ZTk3NSIsInZlcnNpb24iOjF9.nlqol0HEeEjU7509-B9eyohf3CP3EZTibJ1lTvOx3wt8rU5LzEdwFazOTHjpWlcK_rik7jcySdUDe4fGjJtKAQ | |
| - type: loss | |
| value: 3.194674015045166 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzRiYmRiYjFkZDhlNGIwYTg3NDUwZTEzZjc5MjllNmJmODQ1YzBjNDM4MzQwNmMzNmNkMzk5N2M2MzZlOWY4MyIsInZlcnNpb24iOjF9._YJqPY9p_N2n7UxAkTeGenH1sVAkC_Z5HzZ6NbzlQoa8-RXTfbEPLw7fSKmlsGNyZxj7L_Bs4COIWzwAMxZSAA | |
| - type: gen_len | |
| value: 58.9951 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDJhYzU2Zjg4ZmIyOGRmNTU4MDM2NGZiNzc0NDk3YzZkOTQwMWMwNjMzZDQzZTZiZjk4ZDdmMmI2ODRkYjk3OCIsInZlcnNpb24iOjF9.MG1rcM_qpUhQmAYrsBxyNpcLUrPZw6V_uzYzDAo01kQyZEwJClWgMRVgpsSEnY93Mlu1445QLxkJEByUrfD3BQ | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: billsum | |
| type: billsum | |
| config: default | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 37.3648 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWU4ZmZmYzllMzQxM2I4YTUxMjkwYjEzNDk1NjRlYjJiZjYyYWNiNzM4ODMxMGJjMzdhYjFhMzhlNTE5YmYyMiIsInZlcnNpb24iOjF9.9NTlO_5zLC8Y3mkwstviPb9WmMqPmXfWfEN0yONA6WYhh1jPy0gECEb5uF0G6wBMhTPDTqGMWOYIAF2vMeNbDA | |
| - type: rouge | |
| value: 12.3316 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTJhZTcxMDc5ODljMDBjYzFmYWIzNTA4M2NiZDUwYTMwNTVjZTUyZTU2M2IwYWE2YjkzMzMzMjg1MDU1OWE1NSIsInZlcnNpb24iOjF9.FRsoRao8qj6A8W7OeIVAoZCEc1HCZEzmKOs0CPkUceF19pk1ngaXt5K6kcPJ-5fYJydtfSuSnuG3aqlOEJeYDQ | |
| - type: rouge | |
| value: 22.075 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2FjNTMxMGYyNjgyNjk2YTQwZjM4MTM4Yjg0MTkyN2RmNDE5YTU5ZDNkZDFhZDM2YWRlNDI4M2JlMWYxNDQ3ZCIsInZlcnNpb24iOjF9.wsLUEYGJyMSJPPclOzb1hcRdE-VrZex2Sd5er_XVbe6bY1cRO5DdIn69sE9hmAcltefu4ikpHu2ihbv7qvj4Aw | |
| - type: rouge | |
| value: 31.1679 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTUyODVkZGIxYzMyZDczNzU5YjVkNTliZmM4ZTdiYWE2ZjJhNGM3ZDgzMWE3ZjA2MDBhZWQ1ZGY1YzNmZDMwNiIsInZlcnNpb24iOjF9.fPgMnnXY5oPdCn1STZ0HwUiil8OlLZ8ZWZZav_chDIQ7Kh1RKeLy0EG2vEhrB6IlyP7uZ3RmdT9VHM1_khrEAw | |
| - type: loss | |
| value: 2.745267391204834 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNWQ2NDVmODI2ZTQyNmVjZjRkZDdlMTdiODBkZTlkNTFkODBjNjViMTZhMDVkYTkwYWIyNDFkZWZhZmJhODEwMyIsInZlcnNpb24iOjF9.9JWTqdGEhztS--N8grHY6q2a8taVu65Lr17ocXgudp4imhqr9Bhau2X2G5SLN7c1oYieKtyKcWdDAmVzHyTbDw | |
| - type: gen_len | |
| value: 157.3126 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWNiODFmMWQ1ZTkzZGNjNDkwM2ZiZjJlZmQ3N2ExNWJhYmUxYWM2ZGNiYzlhYTY5Y2RhOGVlZDhmN2ZmODQwYSIsInZlcnNpb24iOjF9.sRA9iBS4vzFDZtwM4Vs6Kevj3eiTkS5akApUWTZBCt58YSW8mpoKqsWcnQFEjDCCec-FfV_451OLIetcmDZiCA | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: xsum | |
| type: xsum | |
| config: default | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 18.2975 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjJhMjQ0Yzc4ZmNkOWI5YjhmOTlmOTA4MTE0NWM4NGRlNjE0NDIwOTY2ZmQyNjA0ZmE5MjM2YjAyZDZiNWFkNiIsInZlcnNpb24iOjF9.2UJ48OcezjnfMC0dGjksZpAiXRGNAOHniHdN-tQmQPo0vXwRYNTyPrVULnVoBZUvSdycTYvjl0jDKNhZmtGfCA | |
| - type: rouge | |
| value: 2.6806 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTlkMmQwZTRmN2JlOTQ0N2I0YjdhOTBmYmU3MzEwNzE2ZjFiOTM4OWMyMWRhNmZjNTBkZWY5OGMwYTZhZDRhYSIsInZlcnNpb24iOjF9.7D-IR1aBxx1goOkbeA3Tzd1Wu0Zfi0yQVSG8HWSboM7J67TBHblFsFCVJE7Z2wZRbBW4WtuDIGAcl1d1_Wu_Aw | |
| - type: rouge | |
| value: 11.9453 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZGZjNmY5NmU5ODBmMDQyMjhhNzY3NzBlNDEyMTE3NjY1ZmRkZDZkZWI1YTA0ZTA0NzU1MjMzOTNjZDA3YWM1MCIsInZlcnNpb24iOjF9.SlI42pwrWc_OlcBKOPtrYNzvK_DUk6IJlzrrtjvkZX7k1S7bguekAV-_rWHfn_82k8rJ1FQAReasGHu1dZ0aBw | |
| - type: rouge | |
| value: 14.2121 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2E2MGE0MTQ1YmU0MTJkOWY3ZDhhODIwYWNhNTE3YWJkZTFhYzM1ZjBmNGExODIzYmU2YzE1ODg4ZjdhZWMwMiIsInZlcnNpb24iOjF9.K5FEsZtSph0FqF5zwetkE-X5AKOlj5g_02DPdl-kEe1azKrBBZy9sDiS0WfIGfwHiRdNvKGKi8t3PAGPsfQwCQ | |
| - type: loss | |
| value: 4.836681365966797 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzhlYjA0YzZmYjdmYWQwNDFhNzIzYWNkYzM4OGFlOWJiY2EzYTkxYjk3ZmJmNGQyMGE1ZmYzMDU2MzhhMmVkMiIsInZlcnNpb24iOjF9.uHYwqPBg6K63exBvqt__c82gKi52OhPTRSrcIKHOECCmoXJLJKgFJCuIXGWMJ7UP4HG375e9uqunJB0XwC20DA | |
| - type: gen_len | |
| value: 96.2584 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjNjYzQzNmM5NTY2YzVhNzRkZjMxMzhiYTU1MDBiOGZkYjA4YTg0MmQzYzQ3YTk3N2YwMDA5MWNlM2Y4YTFmZiIsInZlcnNpb24iOjF9.dirG9kG6OdNi-YEMWHv0UMrHTjEt6VS9i6fRbbUeZd1OoP2fl6XcKoDIk6Us-cdiyVnCyyhWsMNsUufMAqLtDA | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: launch/gov_report | |
| type: launch/gov_report | |
| config: plain_text | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 37.3609 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMGExYjM5ODRiNThlZTU4ZTdhM2ZlZWRlNTgzNzc3N2ZjODk2ZjdlOGZlMDkzNmU2Yjk1NzQzZjQ5YzkwODllMCIsInZlcnNpb24iOjF9.JQIeaQkG-IlinWoyc6FKJZUgpWfqOsDhludqm5MgVsw68gsjo0nSPp_Y_1q26Y4dulZOLlQLyBAm3mlCA8s5Ag | |
| - type: rouge | |
| value: 8.6943 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWFjNzJkMzViOGM5YWQ0OGQ4ZTg3NTE5MzU1MjZkZjZiZmVkYTk0ZDhkYjAxMjZiZDVkZTYyYjk4MzRjNTQ3YiIsInZlcnNpb24iOjF9.9XJZ2UF6XyZNNrtp-XOEXC6etoDOFLq1xlIoMFEM9Jinisq3kWguXBiqPQWImLKra5WBm7jU_QIX-Fvn8sP-DA | |
| - type: rouge | |
| value: 17.9106 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNWQ1MTdmNThiM2FiOGRmZWRmOTNlYWMwYTU1YjRiNTRlMGEwYjBmMmQ0YjQ4MDBhNzMzZmZkNjk3NjU0YzRhMSIsInZlcnNpb24iOjF9.040nGV6pig0Rzq9vkN83ZVWQzyjcVi13L36v0QF-Nhziol_dPPhuvghTlGWXWHwj6amsKzyh8M7rNfwL2TcsAQ | |
| - type: rouge | |
| value: 33.8022 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDYwOGRmYzg4ODc2NDExNjhhMjI5MDg3MjI0YTQ5ZDRhM2NjN2Q2ZjM5YTIwZDIxNmY3Y2JlMmMxYTE5MDE4ZiIsInZlcnNpb24iOjF9.S1nynUjLz7z4gf-0WFfPs-ZuZubhN9kXyVSrYNzOdT2gTJmByQWasKreZkVSWus-HNAHR8DhzL6UUWxuDMmAAQ | |
| - type: loss | |
| value: 3.4974069595336914 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzkyNmU5NTRhMTkxNjA1M2E1MjdiMTE0MzQyMDc4ODBkNmM1NDg1ZDk4OTNjODk2MThlZGZiYzQxOGE1YzgwMiIsInZlcnNpb24iOjF9.H9Oo0VKvcqAHcVNvjeEPEhQe5HP0v614suyCv75tfFGaPSKTIe3UlBNDdGOtqfUxb2zUNaBQ8MkA66C_Fkq6CA | |
| - type: gen_len | |
| value: 243.3453 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWE1MGQzMDc2NDViOGM5ZmVkZjk0NmY0NzliOTBhMmE3NmY5MmUxMTI3NGE2OTQzM2Y1NjdmN2NlZGFlODFlYiIsInZlcnNpb24iOjF9.635fcTp_czTabJUVR_dwpzdkntb4cxEbODAC9MMTKrLKEf9NHqDBJXQ-nBOieW05iCSYzw_tEi8O-QW-sRxDAw | |
| - task: | |
| type: summarization | |
| name: Summarization | |
| dataset: | |
| name: kmfoda/booksum | |
| type: kmfoda/booksum | |
| config: kmfoda--booksum | |
| split: test | |
| metrics: | |
| - type: rouge | |
| value: 35.2043 | |
| name: ROUGE-1 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTRlZTdjZDRlZGMxYzA2NmRkYjBiMzZkY2Q1ODUyYjJkM2QwOTRmMzA3ZmU5MDI5ZmM1MmZkZDUwNzc0NjhmNyIsInZlcnNpb24iOjF9.zrskApkmkhbfQLtlgjf_n6i3WmZcmkDH7Sd-JTzOYAU3yk1_Zl4paGdmpXvyQY48M71qWsBYtEKkhnzrkvCGBA | |
| - type: rouge | |
| value: 5.746 | |
| name: ROUGE-2 | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2FlMjU2MzU1MTljZjM0ZmFhMmJlZDAxMTcwZDk3YWE5NjVjYjE0YmEyMTgzY2UyMTVmZDY5ZWM1YmM1ZDA5NSIsInZlcnNpb24iOjF9.5nDuOwa98pon3VW1TazB2Vw1uJgh6pfFMorzgLMJFvhgwYz6_MvLR1dDUeffP4eyw7rGZjBmf039AM7CyKEgCg | |
| - type: rouge | |
| value: 15.6794 | |
| name: ROUGE-L | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjRmYzk3NWFhZDVlODA4YWRiMDU1ZWFhZmMwMWE4MmNkNmNjZWM3ZjUwYzI3MWIxM2Y4MTlhZDk2ZTg5YjkyYSIsInZlcnNpb24iOjF9.TLflM2CYNgz4DNt-TwjgdkTL8ebKckTNnlPVsGLUUGqNI1CvSswzsPedqmntCfKVsH2YAsKsR4ZUb1HtJFsSAw | |
| - type: rouge | |
| value: 32.1129 | |
| name: ROUGE-LSUM | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzdhNWE1YjRjNGUzYWYyNzM4MjIyYThiODJhODU2OGVlOTYxOGNhZmQ4Mjk2ZDUwNmU0MGQwNjQ5NTk2MzU4ZiIsInZlcnNpb24iOjF9.5yvTmPktBuyzoVNHn7UHcci3OrZLTm7e9d_lQkJq8UwzUuso1wHoy_gdvnvpn2DvUfdcBi5sXgG4mtFnVnGgBw | |
| - type: loss | |
| value: 2.945225238800049 | |
| name: loss | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTgxNGRiN2RkMzQ5MjI2OGI0MTljZTY5ZDQyMzc5MjhmNzdhZWQ2NmJhYTgzOTRlMGY2YzkzZWE2NzVkYzVmNCIsInZlcnNpb24iOjF9.VkkP4-S6ZoozLj-iuY7tdsrSR0q1JLQXfgPv_0u2sJuv6x9RYMdCpfJHbqYbirV63b9w28USSwaAAMnz-LoJAA | |
| - type: gen_len | |
| value: 307.5493 | |
| name: gen_len | |
| verified: true | |
| verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMmQ1YTgxYmRhYWViYjhhNmYzNjdlYzVhMTNmZTBkY2RiOTRlMTUzNTIzY2RjOTNhMjRmNGRmYjQyNTBmZWRiMiIsInZlcnNpb24iOjF9.7ItU-AQXB4EEj9U9kJceteBQbA5MkZoegeLhCdpZepEaXzqr6Zg3yHLCD9zL_6Svb9uxuin678KOT5Zf-2YWCQ | |
| # long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP13 | |
| > Evaluating some metric results before merging with the "main" wip version | |
| This model is a fine-tuned version of [pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP12](https://huggingface.co/pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP12) on the `kmfoda/booksum`. | |
| The "base" checkpoint that I update when a training session is productive is [here](https://huggingface.co/pszemraj/long-t5-tglobal-large-pubmed-3k-booksum-16384-WIP) | |
| ## Model description | |
| More information needed | |
| ## Intended uses & limitations | |
| More information needed | |
| ## Training and evaluation data | |
| More information needed | |
| ## Training procedure | |
| ### Training hyperparameters | |
| The following hyperparameters were used during training: | |
| - learning_rate: 0.0006 | |
| - train_batch_size: 2 | |
| - eval_batch_size: 1 | |
| - seed: 42 | |
| - distributed_type: multi-GPU | |
| - gradient_accumulation_steps: 64 | |
| - total_train_batch_size: 128 | |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 | |
| - lr_scheduler_type: cosine | |
| - lr_scheduler_warmup_ratio: 0.01 | |
| - num_epochs: 1.1 | |
| ### Framework versions | |
| - Transformers 4.21.2 | |
| - Pytorch 1.10.0+cu113 | |
| - Datasets 2.4.0 | |
| - Tokenizers 0.12.1 | |