OpenVINO Tokenizers: Incorporate Text Processing Into OpenVINO Pipelines

OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline tokenizer conversion for seamless integration into your projects. It supports Python and C++ environments and is compatible with all major platforms: Linux, Windows, and MacOS.

Notebook Contents

The tutorial consists of the following steps:

Explain the basics of tokenization
Install OpenVINO Tokenizers
Convert tokenizer from HuggingFace Hub using CLI and Python API
Create a Text Generation pipeline with OpenVINO tokenizer and detokenizer
Combine an OpenVINO tokenizer with a classification model

Installation Instructions

This is a self-contained example that relies solely on its own code.
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start. For details, please refer to Installation Guide.