Install:
pip install deepsparse[llm]==1.7
Usage:
>>> from deepsparse import TextGeneration
>>> model = TextGeneration("nm-testing/granite-7b-lab-pruned50-quant-ds")
>>> model("Hello my name is")
TextGenerationOutput(
created=datetime.datetime(2024, 5, 1, 17, 41, 13, 176274),
prompts='Hello my name is',
generations=[GeneratedText(text='Alex and I am a senior in high school. I am taking a dual credit biology class this semester and I really need help with the lab. I am having trouble with the lab on dissociation and I was hoping that you could help me. The lab requires that we set up a reaction between a strong acid and a strong base and measure the pH of the solution at various points in the reaction.\n\nHere is a summary of the lab that I was given:\n\n1', score=None, finished=True, finished_reason='max_new_tokens')],
input_tokens=None
)
- Downloads last month
- 15
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.