Sentiment Classification with Word Embeddings and RNNs

Sentiment classification determines the attitude expressed in a piece of text. A model may predict whether a review is positive or negative, or assign a more detailed rating such as one to five stars.

Word embeddings are particularly helpful when the labeled sentiment dataset is limited. They transfer linguistic structure learned from much larger collections of text into the classifier.

What Is Sentiment Classification?

Given text \(x\), the model predicts a sentiment label \(y\):\[ x\longrightarrow y. \]

For example:

Review	Possible rating
“The dessert is excellent.”	4 or 5 stars
“Service was quite slow.”	2 stars
“Good for a quick meal, but nothing special.”	3 stars
“Completely lacking in good taste, service, and ambiance.”	1 star

The precise label depends on the annotation scheme. Common formulations include:

Positive versus negative
Negative, neutral, or positive
One-to-five-star classification
A continuous sentiment score
Several emotion categories
Aspect-specific sentiment

Practical Uses

Sentiment models can help analyze:

Product reviews
Customer-support messages
Survey responses
Restaurant reviews
Social-media posts
Employee feedback
Public comments
Brand mentions

Aggregated predictions can help identify trends, but individual outputs should be interpreted carefully because language can contain sarcasm, ambiguity, cultural references, and mixed opinions.

Why Labeled Data Can Be a Bottleneck

A high-quality sentiment dataset requires text paired with reliable labels. Producing those labels may require:

Manual annotation
Clear rating criteria
Agreement between annotators
Domain expertise
Quality control

Unlabeled text is generally much more abundant. Word embeddings can be learned from that larger source and transferred into a sentiment model.

This is useful when the target dataset is too small to learn strong token representations from scratch.

Representing Review Tokens

Suppose a tokenized review is\[ w_1,w_2,\ldots,w_T. \]

Each token is mapped to an embedding:\[ e_t=E[w_t], \]

where\[ E\in\mathbb{R}^{V\times d} \]

is the embedding matrix, \(V\) is the vocabulary size, and \(d\) is the embedding dimension.

The resulting review representation begins as a sequence:\[ e_1,e_2,\ldots,e_T. \]

A sentiment classifier must combine these vectors into a form suitable for predicting one label.

A Simple Baseline: Average the Embeddings

The simplest model averages all token embeddings:\[ r = \frac{1}{T} \sum_{t=1}^{T}e_t. \]

The vector\[ r\in\mathbb{R}^{d} \]

has the same dimension regardless of review length.

A classifier then computes\[ z=Wr+b, \]

followed by a softmax:\[ \hat{y} = \operatorname{softmax}(z). \]

For five-star prediction,\[ \hat{y} = \begin{pmatrix} P(y=1\mid x)\\ P(y=2\mid x)\\ P(y=3\mid x)\\ P(y=4\mid x)\\ P(y=5\mid x) \end{pmatrix}. \]

The predicted class is\[ \hat{c} = \arg\max_k \hat{y}_k. \]

Sum Versus Average

The embeddings can be summed:\[ r_{\text{sum}} = \sum_{t=1}^{T}e_t, \]

or averaged:\[ r_{\text{avg}} = \frac{1}{T} \sum_{t=1}^{T}e_t. \]

A sum retains a signal related to sequence length, while an average normalizes for length.

Method	Advantage	Limitation
Sum	Retains magnitude related to token count	Long reviews can have much larger norms
Average	More comparable across lengths	Removes explicit length information

Length can also be supplied separately as an additional feature.

Why the Averaging Model Can Work

The average embedding provides a rough summary of the words appearing in the review.

Positive reviews may contain representations associated with words such as:

excellent
delicious
friendly
wonderful
enjoyable

Negative reviews may contain:

terrible
slow
disappointing
rude
bland

If the embeddings capture relationships between these words, a linear classifier can learn a useful decision boundary in the averaged space.

This model is:

Fast
Compact
Easy to train
Compatible with variable-length text
A valuable baseline for smaller datasets

Masking Padding Tokens

Reviews in a batch are often padded to the same stored length. Padding must not contribute to the average.

Let\[ m_t = \begin{cases} 1, & \text{if position }t\text{ contains a real token},\\ 0, & \text{if it contains padding}. \end{cases} \]

Then the masked average is\[ r = \frac{ \sum_{t=1}^{T}m_te_t }{ \sum_{t=1}^{T}m_t }. \]

A numerical safeguard is needed in case an example contains no valid tokens.

The Main Limitation: Word Order Is Lost

Averaging treats the review as an unordered collection of words.

For any permutation \(\pi\),\[ \frac{1}{T}\sum_{t=1}^{T}e_t = \frac{1}{T}\sum_{t=1}^{T}e_{\pi(t)}. \]

Therefore, the model cannot distinguish sequences that contain the same tokens in different orders.

This is a major problem because sentiment often depends on composition.

Compare:

“The service was good.”
“The service was not good.”

The word “good” appears in both, but negation reverses the sentiment.

A Failure Case for Averaging

Consider:

“Completely lacking in good taste, good service, and good ambiance.”

The word “good” appears several times. An averaging model may give those positive embeddings substantial influence and miss the fact that “lacking in” changes the meaning of the entire phrase.

Other difficult patterns include:

“Not worth the price”
“Hardly impressive”
“I expected it to be good”
“It was good until the ending”
“The food was great, but the service was awful”
“Great—another two-hour delay”

These examples require sensitivity to syntax, contrast, scope, or sarcasm.

A Many-to-One RNN Classifier

An RNN preserves sequential structure by processing embeddings in order:\[ h_t = f(h_{t-1},e_t). \]

After the final token, the hidden state represents the sequence:\[ r=h_T. \]

The sentiment prediction is\[ \hat{y} = \operatorname{softmax}(Wh_T+b). \]

This is a many-to-one architecture:\[ (e_1,e_2,\ldots,e_T) \longrightarrow \hat{y}. \]

The model receives many token inputs and produces one sentiment label.

Why an RNN Handles Composition Better

The recurrent state depends on both the current word and the preceding sequence:\[ h_t=f(h_{t-1},e_t). \]

When the model reads “not,” it can update its state so that the later word “good” is interpreted differently from an isolated positive word.

Similarly, after reading “lacking in,” the model can learn that subsequent positive nouns or adjectives occur under a negative construction.

An RNN does not guarantee perfect understanding of negation or sarcasm, but it has representational access to word order that an average does not.

Using GRUs or LSTMs

A basic RNN may struggle with long-range dependencies. A GRU or LSTM is generally a stronger recurrent choice.

For a GRU:\[ h_t = \operatorname{GRU}(e_t,h_{t-1}). \]

For an LSTM:\[ (h_t,c_t) = \operatorname{LSTM} (e_t,h_{t-1},c_{t-1}). \]

The final representation may be\[ r=h_T. \]

Gated recurrent cells can better preserve information across longer reviews, although they still compress the sequence into a fixed-size final state.

Bidirectional Sequence Encoding

When the complete review is available, a bidirectional encoder can use both preceding and following context.

The forward state is\[ \overrightarrow{h}_t = f_{\rightarrow} (\overrightarrow{h}_{t-1},e_t), \]

and the backward state is\[ \overleftarrow{h}_t = f_{\leftarrow} (\overleftarrow{h}_{t+1},e_t). \]

A review-level representation can combine the terminal states:\[ r = [ \overrightarrow{h}_T; \overleftarrow{h}_1 ]. \]

Alternatively, all contextual states can be pooled.

Bidirectionality is appropriate for offline review classification because the complete text is known before the prediction is made.

Pooling Across Recurrent States

Using only the final hidden state may create a bottleneck for long reviews. Instead, define contextual states\[ h_1,h_2,\ldots,h_T \]

and pool them.

Mean pooling

\[ r = \frac{1}{T} \sum_{t=1}^{T}h_t. \]

Max pooling

For each feature dimension \(j\),\[ r_j = \max_t h_{t,j}. \]

Attention pooling

Learn a score for each position:\[ s_t = v^\top\tanh(Wh_t+b), \]

then normalize:\[ \alpha_t = \frac{\exp(s_t)} {\sum_j\exp(s_j)}. \]

The review representation becomes\[ r = \sum_{t=1}^{T}\alpha_th_t. \]

Attention pooling allows the classifier to emphasize especially informative phrases.

Training the Classifier

For \(K\) sentiment classes, let the target be a one-hot vector \(y\), and let the prediction be \(\hat{y}\).

The cross-entropy loss is\[ \mathcal{L} = -\sum_{k=1}^{K} y_k\log\hat{y}_k. \]

For a batch of \(N\) examples,\[ \mathcal{L}_{\text{batch}} = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}^{(i)}. \]

Backpropagation updates:

The output classifier
The recurrent encoder
Optionally, the embedding matrix

Frozen Versus Trainable Embeddings

Pretrained embeddings can be frozen or fine-tuned.

Frozen embeddings

The embedding matrix remains fixed:\[ E=E_{\text{pretrained}}. \]

This is useful when:

The sentiment dataset is small
Overfitting is a concern
Computation is limited
The source and target domains are similar

Trainable embeddings

The embedding matrix is updated:\[ E \leftarrow E-\eta\frac{\partial\mathcal{L}}{\partial E}. \]

This may help when:

The labeled dataset is larger
Domain-specific word meanings matter
The source corpus differs from the target domain
Sentiment usage is specialized

A common strategy is to train the classifier with frozen embeddings first and then unfreeze them using a smaller learning rate.

Generalizing to Rare Words

Suppose “absent” never appears in the labeled sentiment dataset, but it appears frequently in the corpus used to learn the embeddings.

If its representation is related to words such as

lacking
missing
without

then a sentiment classifier may transfer what it learned about those familiar terms.

This is one of the main benefits of pretrained representations:

A word does not need to appear frequently in the labeled dataset if useful information about it was learned elsewhere.

This benefit depends on vocabulary coverage. A word-level model still needs a strategy for tokens missing from the embedding vocabulary.

Subword Representations

Rare and unseen words can be represented using subword units.

For example, an uncommon word may be divided into reusable pieces. This allows its representation to share information with other words containing similar components.

Subword tokenization is useful for:

Misspellings
Names
Technical terminology
Morphologically rich languages
Newly created words
Informal online text

It generally provides better coverage than a fixed word-only vocabulary.

Contextual Representations

Static embeddings assign one vector to each word regardless of context. This creates difficulty for polysemous words.

For example, “cold” may describe:

Temperature
Illness
Personality
Color tone

A contextual encoder computes\[ e_t=f(w_t,w_1,\ldots,w_T), \]

so the representation changes with the sentence.

A modern sentiment classifier often uses:

A pretrained contextual encoder
A pooled sequence representation
A task-specific classification head

This usually handles composition and ambiguity better than static embedding averages, although it requires more computation.

One-to-Five Stars Are Ordinal

A five-star rating is often treated as five-way classification. However, the labels have an order:\[ 1<2<3<4<5. \]

Predicting five stars when the truth is four is usually less severe than predicting one star.

Ordinary softmax cross-entropy does not encode this ordering directly. It treats all incorrect classes as categorically different.

Alternatives include:

Regression
Ordinal classification
Cumulative threshold models
Earth-mover-style losses
A hybrid classification and regression objective

The best formulation depends on whether calibrated class probabilities or ordinal distance matters more.

Mixed Sentiment and Aspect-Level Analysis

A review may express different opinions about different aspects:

“The food was excellent, but the service was painfully slow.”

A single label compresses this into one overall rating and may lose important detail.

Aspect-based sentiment analysis can produce outputs such as:

Aspect	Sentiment
Food	Positive
Service	Negative
Ambiance	Not mentioned

This requires detecting both the aspect and the sentiment associated with it.

Class Imbalance

Sentiment datasets may contain many more positive reviews than negative ones, or few neutral examples.

Accuracy can be misleading in this setting. A model can achieve high accuracy by favoring the majority class.

Possible responses include:

Class-weighted loss
Stratified sampling
Balanced minibatches
Macro-averaged metrics
Per-class recall analysis
Careful threshold selection

For class \(k\), a weighted loss can be written as\[ \mathcal{L} = -\sum_{k=1}^{K} \lambda_k y_k\log\hat{y}_k. \]

Minority classes receive larger weights when appropriate.

Evaluation Metrics

Useful metrics include:

Accuracy
Macro precision
Macro recall
Macro F1
Confusion matrix
Mean absolute error for ratings
Calibration error
Per-domain performance
Robustness to negation and spelling variation

For ordered ratings, mean absolute error is\[ \operatorname{MAE} = \frac{1}{N} \sum_{i=1}^{N} |\hat{y}^{(i)}-y^{(i)}|. \]

A confusion matrix reveals whether errors occur mostly between neighboring ratings or across opposite sentiments.

A Mean-Embedding Baseline

import torch
import torch.nn as nn

class MeanEmbeddingClassifier(nn.Module):
    def __init__(
        self,
        vocabulary_size,
        embedding_size,
        number_of_classes,
        padding_index
    ):
        super().__init__()

        self.padding_index = padding_index

        self.embedding = nn.Embedding(
            vocabulary_size,
            embedding_size,
            padding_idx=padding_index
        )

        self.classifier = nn.Linear(
            embedding_size,
            number_of_classes
        )

    def forward(self, token_ids):
        vectors = self.embedding(token_ids)

        mask = (
            token_ids != self.padding_index
        ).unsqueeze(-1)

        masked_vectors = vectors * mask

        lengths = mask.sum(dim=1).clamp(min=1)

        pooled = (
            masked_vectors.sum(dim=1)
            / lengths
        )

        return self.classifier(pooled)

import torch
import torch.nn as nn

class MeanEmbeddingClassifier(nn.Module):
    def __init__(
        self,
        vocabulary_size,
        embedding_size,
        number_of_classes,
        padding_index
    ):
        super().__init__()

        self.padding_index = padding_index

        self.embedding = nn.Embedding(
            vocabulary_size,
            embedding_size,
            padding_idx=padding_index
        )

        self.classifier = nn.Linear(
            embedding_size,
            number_of_classes
        )

    def forward(self, token_ids):
        vectors = self.embedding(token_ids)

        mask = (
            token_ids != self.padding_index
        ).unsqueeze(-1)

        masked_vectors = vectors * mask

        lengths = mask.sum(dim=1).clamp(min=1)

        pooled = (
            masked_vectors.sum(dim=1)
            / lengths
        )

        return self.classifier(pooled)

This provides a fast and interpretable baseline. More complex models should be compared against it to confirm that their added cost produces meaningful gains.

A Bidirectional LSTM Classifier

import torch
import torch.nn as nn

class BiLSTMSentimentClassifier(nn.Module):
    def __init__(
        self,
        vocabulary_size,
        embedding_size,
        hidden_size,
        number_of_classes,
        padding_index
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocabulary_size,
            embedding_size,
            padding_idx=padding_index
        )

        self.encoder = nn.LSTM(
            input_size=embedding_size,
            hidden_size=hidden_size,
            batch_first=True,
            bidirectional=True
        )

        self.classifier = nn.Linear(
            2 * hidden_size,
            number_of_classes
        )

    def forward(self, token_ids):
        vectors = self.embedding(token_ids)
        _, (hidden, _) = self.encoder(vectors)

        forward_final = hidden[-2]
        backward_final = hidden[-1]

        representation = torch.cat(
            [forward_final, backward_final],
            dim=1
        )

        return self.classifier(representation)

import torch
import torch.nn as nn

class BiLSTMSentimentClassifier(nn.Module):
    def __init__(
        self,
        vocabulary_size,
        embedding_size,
        hidden_size,
        number_of_classes,
        padding_index
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocabulary_size,
            embedding_size,
            padding_idx=padding_index
        )

        self.encoder = nn.LSTM(
            input_size=embedding_size,
            hidden_size=hidden_size,
            batch_first=True,
            bidirectional=True
        )

        self.classifier = nn.Linear(
            2 * hidden_size,
            number_of_classes
        )

    def forward(self, token_ids):
        vectors = self.embedding(token_ids)
        _, (hidden, _) = self.encoder(vectors)

        forward_final = hidden[-2]
        backward_final = hidden[-1]

        representation = torch.cat(
            [forward_final, backward_final],
            dim=1
        )

        return self.classifier(representation)

For padded batches, a production implementation should use actual sequence lengths or packed sequences so padding does not distort terminal states.

Common Failure Modes

Negation

“The meal was not good.”

The model must understand the scope of “not.”

Contrast

“The food was great, but I would never return.”

Later clauses may dominate the overall judgment.

Sarcasm

“Wonderful—another hour waiting for cold food.”

The literal positive word conflicts with the intended sentiment.

Domain-specific polarity

“Unpredictable” may be positive for a thriller but negative for a vehicle.

Mixed sentiment

A review may praise one feature and criticize another.

Rating inconsistency

Two people may express similar opinions but choose different star ratings.

Distribution shift

A model trained on restaurant reviews may perform poorly on financial commentary or medical feedback.

Common Mistakes

Evaluating only the training loss

A sentiment model must be measured on held-out data from a realistic target distribution.

Ignoring padding in average pooling

Padding vectors can distort the review representation.

Assuming pretrained embeddings understand negation

Static embeddings represent individual words. Composition must be learned by the sequence model.

Fine-tuning embeddings too aggressively

A small dataset can damage useful pretrained structure.

Using accuracy alone on imbalanced data

Macro metrics and per-class errors may reveal poor minority-class performance.

Treating star ratings as unrelated categories

Their ordinal structure may matter to the application.

Assuming one global sentiment captures every opinion

Aspect-level analysis may be necessary for mixed reviews.

Key Takeaway

Sentiment classification maps a variable-length text sequence to a sentiment label. Pretrained word embeddings improve data efficiency by transferring linguistic relationships learned from much larger text collections.

A simple model averages token embeddings:\[ r = \frac{1}{T} \sum_{t=1}^{T}e_t, \]

and passes the result to a classifier. This is fast and useful as a baseline, but it ignores word order and can fail on negation or contrast.

A many-to-one RNN, GRU, LSTM, or bidirectional encoder processes embeddings sequentially:\[ h_t=f(h_{t-1},e_t), \qquad \hat{y}=g(h_T), \]

allowing the model to represent phrases whose meaning depends on composition. Contextual encoders extend this idea further by computing token representations from the complete sentence.