Natural Language Processing - Bag of Word && Embedding

发表于 更新于 分类于
本文字数： 21k 阅读时长 ≈ 19 分钟

Introduction

近年来，自然语言处理（NLP）作为一个领域经历了快速增长，主要是因为语言模型的性能取决于其 "理解 "文本的整体能力，并且可以在大型文本库上以无监督的方式进行训练。因此，预训练的文本模型，如BERT，简化了许多NLP任务，并极大地提高了性能。

本文专注于在PyTorch中把NLP表示为张量的基本方面，以及经典的NLP架构，如 bag-of-words, embeddings and recurrent neural networks

Natural Language Tasks

  • Text Classification: 常用于需要将文本片段分类为几个预定义类之一的场景，例如，垃圾邮件检测、新闻分类、将支持请求分配给其中一个类别等等。

  • Intent Classification：Text Classification 的特例，将对话式人工智能系统中的输入语料映射到代表该短语实际含义的意图之一，或用户的 intent。

  • Sentiment Analysis：一个回归任务，我们想了解给定文本的消极程度。我们可能想把数据集中的文本从最负面的（-1）到最正面的（+1）进行标记，并训练一个模型，输出一个文本的 "正面性 "数字。

  • Named Entity Recognition (NER)：从文本中提取一些实体的任务，如日期、地址、人名等。与意图分类一起，NER经常被用于对话系统，从用户的话语中提取参数。

  • keyword extraction ：在文本中找到最有意义的词，然后可以将其用作标签。

  • Text Summarization：提取最有意义的文本片段，为用户提供包含大部分含义的摘要

  • Question/Answer： This model gets text fragment and a question as an input, and needs to find exact place within the text that contains answer. For example, the text "John is a 22 year old student who loves to use Microsoft Learn", and the question How old is John should provide us with the answer 22.

Text Classification

  • Understand how text is processed for natural language processing tasks
  • Get introduced to Recurrent Neural Networks (RNNs) and Generative Neural Networks (GNNs)
  • Learn about Attention Mechanisms
  • Learn how to build text classification models

Representing text as Tensors

In order to solve NLP tasks with neural networks, we need some way to represent text as tensors. Computers already represent textual characters as numbers that map to fonts on your screen using encodings such as ASCII or UTF-8

Computers by themselves do not have such an understanding of what each letter represents, neural network has to learn the meaning during train

Approaches of representing text:

  • Character-level representation: treating each charater as a number. Given that we have C different characters in our text corpus, the word Hello would be represented by 5xC tensor (One-Hot encoding).
  • Word-level representation: creating a vocabulary of all words in text, then representing words using one-hot encoding. This approach is somehow better, because each letter by itself does not have much meaning, and thus by using higher-level semantic concepts - words - we simplify the task for the neural network. However, given large dictionary size, we need to deal with high-dimensional sparse tensor.

Dataset

Start with a simple text classification task based on AG_NEWS dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech. This dataset is built into torchtext module

1
2
os.makedirs(root, exist_ok=True)
training_iter, test_iter = torchtext.datasets.AG_NEWS(root=root)

train_dataset and test_dataset contain iterators that return pairs of label (number of class) and text respectively, if we want to use the data multiple times we need to convert it to list

1
2
3
4
5
6
7
8
9
10
11
from torchtext.data.functional import to_map_style_dataset

builtins.classes = ['World', 'Sports', 'Business', 'Sci/Tech']
builtins.training_data, test_data = to_map_style_dataset(training_iter), to_map_style_dataset(test_iter)
num_train = int(len(builtins.training_data) * 0.8)
# 获取训练集的 vocab
builtins.vocab = get_vocab()
builtins.vocab_size = len(builtins.vocab)
# 训练集细分训练集和验证集
training_data, valid_data = random_split(builtins.training_data, [num_train, len(builtins.training_data) - num_train])

Convert text into numbers so that can be represented as tensor. (Word-level representation)

  • use tokenizer to split text into tokens
  • build a vocabulary of these tokens
1
2
3
4
5
6
7
8
9
10
11
12
13
def get_vocab(type='bow'):
    counter = collections.Counter()
    for (label, line) in training_data:
        if type == 'bow':
            counter.update(tokenizer(line))
        elif type == 'ngram':
            counter.update(torchtext.data.utils.ngrams_iterator(line, ngrams=2))
    vocab = torchtext.vocab.vocab(counter, min_freq=1)
    return vocab
  
# Using vocabulary, we can easily encode out tokenized string into a set of numbers
[vocab.get_stoi()[i] for i in tokenizer("I love u")]
# [599, 3279, 294]

bag of words text representatiion

Because words represent meaning, sometimes we can figure out the meaning of a text by just looking at the individual words, regardless of their order in the sentence. For example, when classifying news, words like weather, snow are likely to indicate weather forecast, while words like stocks, dollar would count towards financial news.

Bag of Words (BoW) vector representation is the most commonly used traditional vector representation. Each word is linked to a vector index, vector element contains the number of occurrences of a word in a given document. But, BoW does not consider the sequence of words in text

Example:

John likes to watch movies. Mary likes too.

John also likes to watch football games.

Dictionary:

{"John": 1, "likes": 2, "to": 3, "watch": 4, "movies": 5, "also": 6, "football": 7, "games": 8, "Mary": 9, "too": 10}

Transform : the vector element means the number of occurrences of a word in a given sentence

[1, 2, 1, 1, 1, 0, 0, 0, 1, 1]

[1, 1, 1, 1, 0, 1, 1, 1, 0, 0]

1
2
3
4
5
6
7
8
9
def bag_on_words(x):
    vocab_size = len(vocab)
    # each word location in vocab
    location_on_vocab = [vocab[i] for i in tokenizer(x)]
    bow = torch.zeros(vocab_size, dtype=torch.float32)
    for l in location_on_vocab:
        if l < vocab_size:
            bow[l] += 1
    return bow

Since often vocabulary size is pretty big, we can limit the size of the vocabulary to most frequent words.

Training BoW classifier

convert dataset for training in such a way, that all positional vector representations are converted to bag-of-words representation. This can be achieved by passing bowify function as collate_fn parameter to standard torch DataLoader:

1
2
3
4
5
6
7
8
9
10
11
# this collate function gets list of batch_size tuples, and needs to
# return a pair of label-feature tensors for the whole minibatch
def bowify(x):
    return (
        torch.stack([bag_on_words(t[1]) for t in x]),
        torch.LongTensor([t[0] - 1 for t in x])
    )

builtins.training_dataloader = DataLoader(training_data, batch_size=batch_size, collate_fn=bowify, shuffle=True)
builtins.valid_dataloader = DataLoader(valid_data, batch_size=batch_size, collate_fn=bowify, shuffle=True)

One linear layer

Now let's define a simple classifier neural network that contains one linear layer. The size of the input vector equals to vocab_size, and output size corresponds to the number of classes (4). Because we are solving classification task, the final activation function is LogSoftmax().

1
2
3
4
5
6
7
8
9
10
class OneLinearLayer(nn.Module):
    def __init__(self):
        super(OneLinearLayer, self).__init__()
        self.linear = nn.Sequential(
            nn.Linear(in_features=vocab_size, out_features=4),
            nn.LogSoftmax(dim=1)
        )

    def forward(self, x):
        return self.linear(x)

training

Standard training loop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
def train(net, training_dataloader, test_dataloader, optimizer=None, loss_fn=nn.CrossEntropyLoss()):
    hist = {'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc': []}
    for i in range(1, epoch + 1):
        print(f"epoch: {i}\n------------------------------------------")
        net.train()
        size, acc, total_loss, batch = len(training_dataloader.dataset), 0, 0, 0
        optimizer = op.Adam(net.parameters(), lr=learning_rate) if optimizer is None else optimizer
        for batch, (x, y) in enumerate(training_dataloader):
            pred_y = net(x)
            loss = loss_fn(pred_y, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            total_loss += loss.item()
            acc += (pred_y.argmax(1) == y).type(torch.float).sum().item()

            if batch % 10 == 0:
                print(f"train loss: {loss:>5f}   [{batch * batch_size}/{size}]")
        print(f"train_loss: {total_loss / batch:>5f}     train_acc: {acc / size:>5f}     {size}")
        hist['train_loss'].append(total_loss / batch)
        hist['train_acc'].append(acc / size)
        net.eval()
        size, acc, total_loss, count = len(test_dataloader.dataset), 0, 0, 0
        with torch.no_grad():
            for x, y in test_dataloader:
                pred_y = net(x)
                total_loss += loss_fn(pred_y, y).item()
                acc += (pred_y.argmax(dim=1) == y).type(torch.float).sum().item()
                count += 1
            print(f"val_loss: {total_loss / count:>5f}     val_acc: {acc / size:>5f}     {size}\n")
            hist['val_loss'].append(total_loss / count)
            hist['val_acc'].append(acc / size)
    return hist
  

if __name__ == "__main__":
    load_data()
    model = OneLinearLayer()
    summary(model, input_size=(1, vocab_size))
    hist = train(model, training_dataloader, valid_dataloader, print_step=1000)
    plot_acc_loss(hist)

1

BiGrams, TriGrams and N-Grams

Bag-of-Words 方法的一个局限性是，有些词是多词表达的一部分，例如，"热狗 "这个词与其他语境中的 "热 "和 "狗 "的含义完全不同。如果我们总是用相同的向量来表示 "热 "和 "狗 "这两个词，就会使我们的模型混乱。

N-gram 表示经常用于文档分类的方法中，其中每个word、bi-word 或 tri-word 的频率是训练分类器的有用特征。例如，在大词表征中，除了原始词之外，我们将把所有的词对加入到vocabulary中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\b\w+\b', min_df=1)
corpus = [
        'I like hot dogs.',
        'The dog ran fast.',
        'Its hot outside.',
    ]
bigram_vectorizer.fit_transform(corpus)
print("Vocabulary:\n",bigram_vectorizer.vocabulary_)
bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()


---------------------------------------------------------------------
Vocabulary:
 {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}

array([[1, 0, 1, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

N-gram方法的主要缺点是，词汇量开始极速增长。在实践中，我们需要将N-gram表示法与一些降维技术结合起来，如embeddings 技术。

1
2
3
4
5
tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
counter = collections.Counter()
for (label, line) in training_data:
		counter.update(torchtext.data.utils.ngrams_iterator(line, ngrams=2))
bi_vocab = torchtext.vocab.vocab(counter, min_freq=1)

Term Frequency Inverse Document Frequency (TF-IDF)

在BoW表示法中，无论单词本身如何，单词出现的权重是均匀的。然而，很明显，频繁出现的词，如a、in等，对于分类来说，其重要性远远低于专业术语。事实上，在大多数NLP任务中，有些词比其他词更有意义。

TF-IDF是 bag of words 的一个变种，用一个浮点值代替表示一个词在文档中出现的二进制0/1值，它与语料库中的词出现频率有关。

the weight \(w_{ij}\) of a word \(i\) in the document \(j\) is defined as:

\[ w_{ij} = tf_{ij}\times\log({N\over df_i}) \]

where

  • \(tf_{ij}\) is the number of occurrences of \(i\) in \(j\), i.e. the BoW value we have seen before
  • \(N\) is the number of documents in the collection
  • \(df_i\) is the number of documents containing the word \(i\) in the whole collection

TF-IDF value \(w_{ij}\) increases proportionally to the number of times a word appears in a document and is offset by the number of documents in the corpus that contains the word, which helps to adjust for the fact that some words appear more frequently than others.

For example, if the word appears in every document in the collection, \(df_i=N\), then \(w_{ij}=0\), and those terms would be completely disregarded.

然而，即使TF-IDF表示法为不同的词提供频率权重，它们也无法表示意义或顺序。正如著名的语言学家J.R.Firth在1935年所说："一个词的完整意义总是与上下文有关的，离开了上下文，对意义的研究就无法认真对待"。

Represent words with embeddings

One-hot 表示法不具有记忆效率，此外，每个词都是独立处理的，即 one-hot 编码向量不表达词之间的任何语义相似性。Enbedding 的想法是用低维的密集向量来表示 word ，这在某种程度上反映了一个词的语义。我们将在后面讨论如何建立有意义的单词嵌入，但现在我们只把嵌入看作是降低单词向量维度的一种方法。

embedding layer 将接受一个词作为输入，并产生一个 embedding_size 的输出向量。从某种意义上说，它与线性层非常相似，但它不是取一个 one-hot 的向量，而是取一个 word number ( word 在vocabulary 中出现的位置 ) 作为输入。

embedding向量的性质是能使距离相近的向量对应的物体有相近的含义，比如 Embedding(复仇者联盟)和Embedding(钢铁侠)之间的距离就会很接近，但 Embedding(复仇者联盟)和Embedding(乱世佳人)的距离就会远一些。

通过使用embedding layer作为我们网络中的第一层，我们可以从 bag of words model转换为embedding bag model ，我们首先将文本中的每个字转换为相应的embeddings，然后计算所有这些 embeddings 的一些集合函数，如sum、average或最max。

1

1
2
3
4
5
6
7
8
9
10
11
class EmbedClassifier(torch.nn.Module):
    def __init__(self, vocab_size, embed_dim, num_class):
        super().__init__()
        self.embedding = torch.nn.Embedding(vocab_size, embed_dim)
        self.fc = torch.nn.Linear(embed_dim, num_class)

    def forward(self, x):
        x = self.embedding(x)
        x = torch.mean(x,dim=1)
        return self.fc(x)

Dealing with variable sequence size

由于这种架构，对我们的网络的minibatches需要以某种方式创建。在上一个单元中，当使用bag of words时，不管我们的文本序列的实际长度如何，minibatch中的所有BoW张量都具有相同的大小vocab_size。

一旦我们使用 embedding，我们最终会发现每个文本样本中的字数是不一样的，当把这些样本合并成minibatch时，我们必须应用一些填充。这可以通过向数据源提供collate_fn函数的相同技术来完成。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def padify(b):
    # b is the list of tuples of length batch_size
    #   - first element of a tuple = label,
    #   - second = feature (text sequence)
    # build vectorized sequence
    v = [encode(x[1]) for x in b]
    # first, compute max length of a sequence in this minibatch
    l = max(map(len, v))
    return (  # tuple of two tensors - labels and features
        torch.stack(
            [torch.nn.functional.pad(torch.tensor(t), (0, l - len(t)), mode='constant', value=0) for t in v]),
        torch.LongTensor([t[0] - 1 for t in b]),
    )

train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)

train

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
if __name__ == "__main__":
    load_data()
    # Embedding 到 32 dimension
    model = EmbedClassifier(32, len(classes))
    summary(model)    # print(model)
    hist = train(model, training_dataloader, valid_dataloader, print_step=1000)
    plot_acc_loss(hist)
    
---------------------------------------------------------
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
EmbedClassifier                          --
├─Embedding: 1-1                         3,065,920
├─Linear: 1-2                            132
=================================================================
Total params: 3,066,052
Trainable params: 3,066,052
Non-trainable params: 0
=================================================================
....
epoch: 10
------------------------------------------
train loss: 0.118139   [0/96000]
train loss: 0.083000   [32000/96000]
train loss: 0.061542   [64000/96000]
train_loss: 0.131295     train_acc: 0.959406     96000
val_loss: 0.258924     val_acc: 0.916708     24000

1

Embedding Bag Layer and Variable-Length Sequence Representation

上面，我们需要将所有序列填充到相同的长度，以便将它们装入 minibatch 中。 这不是表示可变长度序列的最有效的方法 - 另一个apporach是使用 offset vector，这将所有序列的偏移存储在一个大的向量。

1

To work with offset representation, we use EmbeddingBag layer. It is similar to Embedding, but it takes content vector and offset vector as input, and it also includes averaging layer, which can be mean, sum or max.

1
2
3
4
5
6
7
8
9
class EmbedClassifier(torch.nn.Module):
    def __init__(self, vocab_size, embed_dim, num_class):
        super().__init__()
        self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim)
        self.fc = torch.nn.Linear(embed_dim, num_class)

    def forward(self, text, off):
        x = self.embedding(text, off)
        return self.fc(x)

To prepare the dataset for training, we need to provide a conversion function that will prepare the offset vector:

1
2
3
4
5
6
7
8
9
10
11
12
13
def offsetify(b):
    # first, compute data tensor from all sequences
    x = [torch.tensor(encode(t[1])) for t in b]
    # now, compute the offsets by accumulating the tensor of sequence lengths
    o = [0] + [len(t) for t in x]
    o = torch.tensor(o[:-1]).cumsum(dim=0)
    return ( 
        torch.LongTensor([t[0]-1 for t in b]), # labels
        torch.cat(x), # text 
        o
    )

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=offsetify, shuffle=True)

Note, that unlike in all previous examples, our network now accepts two parameters: data vector and offset vector, which are of different sizes. Sililarly, our data loader also provides us with 3 values instead of 2: both text and offset vectors are provided as features. Therefore, we need to slightly adjust our training function to take care of that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
net = EmbedClassifier(vocab_size,32,len(classes)).to(device)

def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):
    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
    loss_fn = loss_fn.to(device)
    net.train()
    total_loss,acc,count,i = 0,0,0,0
    for labels,text,off in dataloader:
        optimizer.zero_grad()
        labels,text,off = labels.to(device), text.to(device), off.to(device)
        out = net(text, off)
        loss = loss_fn(out,labels) #cross_entropy(out,labels)
        loss.backward()
        optimizer.step()
        total_loss+=loss
        _,predicted = torch.max(out,1)
        acc+=(predicted==labels).sum()
        count+=len(labels)
        i+=1
        if i%report_freq==0:
            print(f"{count}: acc={acc.item()/count}")
        if epoch_size and count>epoch_size:
            break
    return total_loss.item()/count, acc.item()/count


train_epoch_emb(net,train_loader, lr=4, epoch_size=25000)



1

Semantic Embeddings: Word2Vec

在之前的例子中，模型的嵌入层学会了将单词映射为矢量表示，然而，这种表示并没有太多的语义意义。如果能学会这样的向量表示，即类似的词或症状词将对应于在某些向量距离（例如，欧几里得距离）上彼此接近的向量。

要做到这一点，我们需要以一种特定的方式在大量的文本集合上预训练我们的嵌入模型。训练语义嵌入的最初方式之一被称为Word2Vec。它是基于两个主要的架构，用来产生单词的分布式表示。

- Continuous bag-of-words (CBoW) — in this architecture, we train the model to predict a word from surrounding context. Given the ngram \((W_{-2},W_{-1},W_0,W_1,W_2)\), the goal of the model is to predict \(W_0\) from \((W_{-2},W_{-1},W_1,W_2)\).

- Continuous skip-gram is opposite to CBoW. The model uses surrounding window of context words to predict the current word.

1

To experiment with word2vec embedding pre-trained on Google News dataset, we can use gensim library. Below we find the words most similar to 'neural'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
w2v = api.load('word2vec-google-news-300')
    for w, p in w2v.most_similar('neural'):
        print(f"{w} -> {p}")
      
----------------------------------------------------------
neuronal -> 0.7804799675941467
neurons -> 0.7326499819755554
neural_circuits -> 0.7252851128578186
neuron -> 0.7174385786056519
cortical -> 0.6941086649894714
brain_circuitry -> 0.6923246383666992
synaptic -> 0.669911801815033
neural_circuitry -> 0.6638562679290771
neurochemical -> 0.6555314660072327
neuronal_activity -> 0.6531826257705688

We can also extract vector embeddings from the word, to be used in training classification model (we only show first 20 components of the vector for clarity):

1
2
3
4
5
6
>>> print(w2v.get_vector('play')[:20])
<input>:1: DeprecationWarning: Call to deprecated `word_vec` (Use get_vector instead).
[ 0.01226807  0.06225586  0.10693359  0.05810547  0.23828125  0.03686523
  0.05151367 -0.20703125  0.01989746  0.10058594 -0.03759766 -0.1015625
 -0.15820312 -0.08105469 -0.0390625  -0.05053711  0.16015625  0.2578125
  0.10058594 -0.25976562]

Great thing about semantical embeddings is that you can manipulate vector encoding to change the semantics. For example, we can ask to find a word, whose vector representation would be as close as possible to words king and woman, and as far away from the word man:

1
2
>>> w2v.most_similar(positive=['king','woman'],negative=['man'])[0]
('queen', 0.7118193507194519)

Both CBOW and Skip-Grams are “predictive” embeddings, in that they only take local contexts into account. Word2Vec does not take advantage of global context.

FastText, builds on Word2Vec by learning vector representations for each word and the charachter n-grams found within each word. The values of the representations are then averaged into one vector at each training step. While this adds a lot of additional computation to pre-training it enables word embeddings to encode sub-word information.

Another method, GloVe, leverages the idea of co-occurence matrix, uses neural methods to decompose co-occurrence matrix into more expressive and non linear word vectors.

Using Pre-Trained Embeddings in PyTorch

We can modify the example above to pre-populate the matrix in our embedding layer with semantical embeddings, such as Word2Vec. We need to take into account that vocabularies of pre-trained embedding and our text corpus will likely not match, so we will initialize weights for the missing words with random values:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
class PreTrainedW2VEmbed(nn.Module):
    def __init__(self, w2v, embed_dim, num_class):
        super(PreTrainedW2VEmbed, self).__init__()
        self.embed_size = len(w2v.get_vector('Hello'))
        print(f'Embedding size: {self.embed_size}')
        self.embed_classifier = EmbedOffsetClassifier(embed_dim, num_class)
        print('Populating matrix, this will take some time...', end='')
        found, not_found = 0, 0
        for i, w in enumerate(vocab.get_stoi()):
            try:
                self.embed_classifier.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))
                found += 1
            except:
                self.embed_classifier.embedding.weight[i].data = torch.normal(0.0, 1.0, (self.embed_size,))
                not_found += 1
        print(f"Done, found {found} words, {not_found} words missing")
        self.embed_classifier.to(device)

    def forward(self, x, o):
        return self.embed_classifier(x, o)
      
def train_w2v_embed_off():
    load_data('embedding_offset')
    w2v = api.load('word2vec-google-news-300')
    # Embedding 到 32 dimension
    model = PreTrainedW2VEmbed(w2v, 32, len(classes))
    summary(model)  # print(model)
    hist = train_embed_offset(model, training_dataloader, valid_dataloader, print_step=1000)
    plot_acc_loss(hist)


if __name__ == "__main__":
    train_w2v_embed_off()
   
Embedding size: 300
Populating matrix, this will take some time...Done, found 41080 words, 54730 words missing
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
PreTrainedW2VEmbed                       --
├─EmbedOffsetClassifier: 1-1             --
│    └─EmbeddingBag: 2-1                 3,065,920
│    └─Linear: 2-2                       132
=================================================================
Total params: 3,066,052
Trainable params: 3,066,052
Non-trainable params: 0
=================================================================

1

In our case we do not see huge increase in accuracy, which is likely to quite different vocalularies. To overcome the problem of different vocabularies, we can use one of the following solutions:

  • Re-train word2vec model on our vocabulary
  • Load our dataset with the vocabulary from the pre-trained word2vec model. Vocabulary used to load the dataset can be specified during loading.

The latter approach seems easiter, especially because PyTorch torchtext framework contains built-in support for embeddings. We can, for example, instantiate GloVe-based vocabulary in the following manner:

1
vocab = torchtext.vocab.GloVe(name='6B', dim=50)

Loaded vocabulary has the following basic operations:

  • vocab.stoi dictionary allows us to convert word into its dictionary index
  • vocab.itos does the opposite - converts number into word
  • vocab.vectors is the array of embedding vectors, so to get the embedding of a word s we need to use vocab.vectors[vocab.stoi[s]]
1
2
3
4
5
6
7
8
9
10
11
12
def train_with_gloVe_vocab():
    # Process finished with exit code 137 (interrupted by signal 9: SIGKILL) 内存不足
    vocab = torchtext.vocab.GloVe(name='6B', dim=50)
    load_data('embedding_offset', vocab=vocab)
    net = EmbedOffsetClassifier(vocab_size, len(classes))
    net.embedding.weight.data = vocab.vectors
    hist = train_embed_offset(net,training_dataloader, valid_dataloader)
    plot_acc_loss(hist)


if __name__ == "__main__":
    train_with_gloVe_vocab()

训练过程电脑内存不足，vocab_size=400000，参数过多