docs/tutorial.processors.rst

Adding Processors
################################################

This is a tutorial on adding new processors using ``lavis.processors`` module. 

The LAVIS library includes a standard processor module that preprocesses data e.g. image transformation and sequence concatenation.
The ``lavis.processors`` module is designed such that any processors can be added, specifically to the requirements of corresponding models of interest. 
In this tutorial, we will replicate the steps to add visual and textual processors specifically for `video-grounded dialogue tasks <https://arxiv.org/pdf/1901.09107.pdf>`_. 
In addition, we also want the processors to have processing features to make the data samples compatible with GPT-style models.

Base Processor ``lavis.processors.base_processors``
*****************************************************

Note that any new processor definition should inherit the base processor class ``BaseProcessor``:

.. code-block:: python

    from omegaconf import OmegaConf
    
    class BaseProcessor:
        def __init__(self):
            self.transform = lambda x: x
            return
    
        def __call__(self, item):
            return self.transform(item)
    
        @classmethod
        def from_config(cls, cfg=None):
            return cls()
    
        def build(self, **kwargs):
            cfg = OmegaConf.create(kwargs)
    
            return self.from_config(cfg)

This allows us to standardize operations of processors across all processor classes while still allowing customization of processors specifically to data and model types. 
We encourage users not to modify the implementation of the base processor class as this will have an impact on all existing processor subclasses.

GPT-style Processors ``lavis.processors.gpt_processors``
**************************************************************
In this step, we can define new processor classes, e.g. under ``lavis.processors.gpt_processors``, for GPT models designed specifically for video-grounded dialogues. 
First, we want to process video features by defining ``GPTVideoFeatureProcessor`` class.
In this tutorial, we assume video features are extracted beforehand and this processor simply loads the features from ``npy`` files.
Other methods that are specifically defined are ``padding`` (which is used by dataset instances to pad multiple video samples) and ``get_attention_mask`` (which creates an attention mask for Transformer attention in GPT models). 

.. code-block:: python 

    SPECIAL_TOKENS_DICT = {'bos_token': "<bos>", 'eos_token': "<eos>", 'additional_special_tokens': ["<speaker1>", "<speaker2>", "<video>", "<cap>"], 'pad_token': "<pad>"}
    ...

    @registry.register_processor("gpt_video_ft")
    class GPTVideoFeatureProcessor(BaseProcessor):
        def __init__(self, visual_ft, audio_ft):

            self.visual_ft = visual_ft
            self.audio_ft = audio_ft

            self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
            self.tokenizer.add_special_tokens(SPECIAL_TOKENS_DICT) 
                    
        def padding(self, seq):
            padded_seq = torch.nn.utils.rnn.pad_sequence(seq, batch_first=True, padding_value=1.0) 
            return padded_seq
        
        def get_attention_mask(self, seq):
            return torch.sum(seq != 1, dim=2) != 0
    
        def __call__(self, ft_root, vname):
            all_ft = []
            
            for ft_name in self.visual_ft:
                ft_path = os.path.join(ft_root, ft_name, vname)
                all_ft.append(np.load(ft_path + '.npy'))
            
            for ft_name in self.audio_ft: 
                ft_path = os.path.join(ft_root, ft_name, vname)
                all_ft.append(np.load(ft_path + '.npy'))
            
            min_len = min([len(ft) for ft in all_ft])
            
            sampled_ft = [ft[:min_len] for ft in all_ft]
            sampled_ft = np.concatenate(sampled_ft, axis=1)
            item = {} 
            item['video_fts'] = torch.Tensor(sampled_ft) 
            
            video_type_token = self.tokenizer.convert_tokens_to_ids('<video>')
            item['token_type_ids'] = torch.Tensor([video_type_token] * len(sampled_ft)).long() 
            
            return item 
    
        @classmethod
        def from_config(cls, cfg=None):
            if cfg is None:
                cfg = OmegaConf.create()
            
            visual_ft = cfg.get("visual_ft", ["i3d_rgb"])
            audio_ft = cfg.get("audio_ft", ["vggish"])
            
            return cls(
                visual_ft=visual_ft,
                audio_ft=audio_ft
            )

Another processor class that will be useful to have is to process dialogue data. Here we can define a ``GPTDialogueProcessor`` class.
This processor class receives raw annotations and constructs inputs as a concatenation of input sequences (questions, dialogue contexts, and responses) to facilitate application in GPT models. 
Other methods that are specifically defined are ``padding`` (which is used by dataset instances to pad multiple sequence samples) and ``get_attention_mask`` (which creates an attention mask for Transformer attention in GPT models). 

.. code-block:: python 

    SPECIAL_TOKENS_DICT = {'bos_token': "<bos>", 'eos_token': "<eos>", 'additional_special_tokens': ["<speaker1>", "<speaker2>", "<video>", "<cap>"], 'pad_token': "<pad>"}
    ...

    @registry.register_processor("gpt_dialogue")
    class GPTDialogueProcessor(BaseProcessor):
        def __init__(self, max_turns=3, use_caption=True):
            self.max_turns = max_turns 
            self.use_caption = use_caption 
            self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
            self.tokenizer.add_special_tokens(SPECIAL_TOKENS_DICT) 
            
        def sample_sequence(self, caption, history, answer):
            bos, eos, speaker1, speaker2, cap = self.tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS[:-2])
            instance = {}
            sequence = [caption] + history + [answer]
            sequence = [s + [eos] for s in sequence] 
    
            instance["input_ids"] = list(chain(*sequence))
            instance["token_type_ids"] = [cap] * len(sequence[0]) + [speaker2 if i % 2 else speaker1 for i, s in enumerate(sequence[1:]) for _ in s]
            instance["labels"] = ([-1]*sum(len(s) for s in sequence[:-1])) + sequence[-1]
            
            assert len(instance["input_ids"])==len(instance["token_type_ids"])
            assert len(instance["token_type_ids"])==len(instance["labels"])
            
            for k,v in instance.items():
                instance[k] = torch.Tensor(v).long() 
            
            return instance 
        
        def padding(self, seq, pad_token=-1):
            if pad_token==-1: pad_token = self.tokenizer.pad_token_id 
            padded_seq = torch.nn.utils.rnn.pad_sequence(seq, batch_first=True, padding_value=pad_token) 
            return padded_seq
        
        def get_attention_mask(self, seq, pad_token=-1):
            if pad_token==-1: pad_token = self.tokenizer.pad_token_id 
            return seq != pad_token
        
        def __call__(self, ann):
            if self.use_caption:
                caption = ' '.join([ann['caption'], ann['summary']])
                caption = self.tokenizer.encode(caption)
            else:
                caption = []
                
            dial_history = []
            for turn in ann['dialog'][-self.max_turns:]:
                dial_history.append(turn['question'])
                dial_history.append(turn['answer'])
            dial_history.append(ann['question'])
            dial_history = [self.tokenizer.encode(t) for t in dial_history]
            
            answer = self.tokenizer.encode(ann['answer'])
            
            item = self.sample_sequence(caption, dial_history, answer)
            
            return item 
    
        @classmethod
        def from_config(cls, cfg=None):
            if cfg is None:
                cfg = OmegaConf.create()
    
            use_caption = cfg.get("use_caption", True)
            max_turns = cfg.get("max_turns", 3)
    
            return cls(max_turns=max_turns, use_caption=use_caption)

Registering New Processors ``lavis.processors.__init__``
**************************************************************

Finally, any new processor must be officially registered as part of the ``lavis.processors`` module. 
For instance, to add processor classes for GPT-based dialogue models, including one for dialogue data ``GPTDialogueProcessor`` and one for video features ``GPTVideoFeatureProcessor``, we can modify the ``__init__.py`` as follows: 

.. code-block:: python

    from lavis.processors.gpt_processors import (
        GPTVideoFeatureProcessor,
        GPTDialogueProcessor,
    )
    
    __all__ = [
        ...
        # GPT
        "GPTVideoFeatureProcessor",
        "GPTDialogueProcessor"
    ]

Assigning Processors 
**************************************************************
From the above example of processor classes, note that we define a ``from_config`` method for each class. 
This method will process a configuration file and pass specific parameters e.g. ``max_turns``, ``visual_ft``, to initialize the processor classes properly. 
To do this, we can assign/ associate the correct registry of processor classes in a configuration file.
For instance, the following should be specified in a configuration file e.g. ``dialogue_avsd_ft.yaml``:

.. code-block:: yaml 

    datasets:
      avsd_dialogue: # name of the dataset builder
        vis_processor:
            train:
              name: "gpt_video_ft" # name of the visual processor for training data
              visual_ft: ["i3d_flow", "i3d_rgb"]  
              audio_ft: ["vggish"]    
            eval:
              name: "gpt_video_ft" # name of the visual processor for evaluation data
              visual_ft: ["i3d_flow", "i3d_rgb"]  
              audio_ft: ["vggish"]   
        text_processor:
            train:
              name: "gpt_dialogue" # name of the textual processor for training data
              max_turns:  3
              use_caption: True 
            eval:
              name: "gpt_dialogue" # name of the textual processor for evaluation data
              max_turns:  3
              use_caption: True 

Subsequently, any processes (e.g. training) should load this configuration file to assign the correct processors.

.. code-block:: sh

    python train.py --cfg-path dialogue_avsd_ft.yaml