Bayesian Neural Networks (BNNs)
Bayesian neural networks average predictions over a weight posterior. Prior, likelihood, inference approximation, and observation noise shape their uncertainty.
-
-
Model Distillation (Knowledge Distillation)
Teacher 0.9815, student on hard labels 0.9060, student on teacher probabilities 0.9105. Distillation is a training signal, not extra capacity.
-
Epochs
An epoch counts data exposure. Batch size, accumulation, sampling, and validation determine what that exposure means for updates and useful learning.
-
Model Weights
Coefficient magnitude depends on input units. With fixed L2 regularization, rescaling a feature changes the fitting objective as well as the coefficient units.
-
FLOPs
A matrix-vector product and a scaled vector addition each do 33,554,432 FLOPs at intensities of 0.50 and 0.17 FLOPs per byte.
-
Autoencoder
A five-unit linear bottleneck and five-component PCA both reach error 0.0677 here. Reconstruction and downstream usefulness are separate evaluations.
-
Frozen Encoder
A mismatched encoder stayed near 0.85–0.87 accuracy over the tested sample sizes. This curve does not prove that more labels can never improve the result.
-
LSTM — Long Short-Term Memory Networks
LSTMs retain and update a cell state through learned gates. Their gradient path, state handling, and sequential execution define both benefits and limits.
-
Neural Networks
Neural networks learn parameterized transformations. Layers, nonlinearities, loss, gradients, and optimization play different roles in fitting a useful model.
-
Loss Functions
Twenty-five contaminating values pulled the squared-error fit from 99.73 to 142.60, while absolute error stayed at 100.02 and Huber at 100.21.
-
Softmax Function
The same logits read [0.9817, 0.0180, 0.0003] at temperature 0.25 and [0.3672, 0.3322, 0.3006] at temperature 10.
-
Sigmoid Function
At x = 9, a local slope estimate suggests -81.05 for a 0.01 output decrease. The exact input change is about -4.42; the derivative is only local.
-
Squashing Function
Ten maximal sigmoid derivatives multiply to 9.537e-07 and forty to 8.272e-25. These are activation-only factors; the latter is still representable in float32.
-
Embedding Similarity
The dot product ties two candidates at 10.0000 that cosine separates at 1.0000 and 0.9806. The metric is not a formatting choice.
-
Computer Vision (CV)
Computer vision connects pixels to task-specific outputs. Representation, labels, image processing, and deployment evaluation determine what a model can do.
-
Natural Language Processing (NLP)
NLP connects language representations to tasks such as classification, extraction, retrieval, and generation. Context, data, and evaluation shape reliability.
-
Deep Ensembles
Deep ensembles average separately trained networks. Error correlation limits gains, and member disagreement captures only part of predictive uncertainty.
-
Quantization
Quantization trades numerical resolution for storage and possible speed gains. Scale, clipping, activation ranges, and runtime support determine the result.
-
ONNX (Open Neural Network Exchange)
ONNX carries a computation graph and tensors. Export, operator support, numerical checks, and preprocessing determine whether that graph is deployable.
-
TPU Clusters
TPU clusters combine accelerator memory and compute through communication. Parallelism, batch size, input supply, and recovery determine useful scaling.
-
Logits
The logits [2, 1, 0] and [102, 101, 100] produce identical probabilities of [0.6652, 0.2447, 0.0900]. Only the gaps between them are identified.
-
Connectionist Temporal Classification (CTC)
Connectionist temporal classification is a training loss and decoding rule for sequence models whose input has at least as many frames as the output has labels, and whose alignment is unknown.
-
Cross-Attention
Cross-attention is attention in which the queries come from one sequence and the keys and values come from another.
-
Log-Mel Spectrogram
A log-mel spectrogram is a two-dimensional summary of an audio signal: a matrix with one row per short time step and one column per frequency band, where each entry is the logarithm of the energy in that band at that moment.
-
Replay and Retention Sets
Replay and retention sets are the two halves of handling catastrophic forgetting when a model is fine-tuned.
-
InfoNCE Loss
InfoNCE is the contrastive loss behind most embedding models and CLIP-style image-text models.
-
Rotary Position Embedding (RoPE)
Self-attention compares tokens by a dot product between a query vector and a key vector, and that dot product knows nothing about where the tokens sit in the sequence.
-
Embedding
Understand embeddings as learned vector representations, how similarity search works, and why similarity does not prove an answer is correct.