Repository not working with Inference Endpoints
Got twice the same error - First on using a small instance with a single GPU, as per the article on his article (https://www.philschmid.de/deploy-t5-11b), and second time on using a large instance:
-- TypeError: dispatch_model() got an unexpected keyword argument 'offload_index'
This seems to be related to how the instance is being set-up, not part of this repository code. May have to try to set-up everything from scratch separately from Hugging Face Inference Endpoints (which did work with simpler models).
Any feedback would be greatly appreciated, as I would prefer the lazy way, ie using Hugging Face Inference Endpoints.
-- the most recent model (as of Feb 6, 2023) from https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16 is working fine on a medium/large instance, so it could just be an old image. As such, closing this.