Problem:

Extracting text features from BERT can be accomplished by the provided example script extract_features.py. However, the problem arises if we want to initialize the words or phrases by the tokenizers that are not integrated in BERT.

Scenario:

For example, in multimodal learning, it is a trend to align audios and videos to the corresponding transcripts at word-level. P2FA enables the Forced Alignment between audios and transcripts, where the transcripts are tokenized by its own tokenizer. For convinience, we call this kind of tokens as P2FA-tokens, which can be easily initialized by Word2vec or GloVe pre-trained word embeddings because there is no tokenizer requirements. BERT is an architecture, which means we should encode the text by its tokenizer to ensure the correct mapping between the tokens and the features.

Solution:

The solution for aligning the P2FA-tokens to BERT-tokens is as below:

• Feature Extraction: We reconstruct each utterance from P2FA-tokens and save a .txt file for each video. Then, call the extract_features() function from extract_functions.py to extract BERT features and save into a .jsonl file.