👋🏻Welcome to Yuyang’s Blog

Reinforcement Learning in LLM

Transformer

Large Language Model

Reinforcement Learning

In this blog, we will explore how reinforcement learning is applied in large language models (LLMs) to enhance their performance and adaptability. It will start with mapping LLMs to the RL framework, then we will dive into different RL algorithms used in LLMs (RLHF), such as PPO and DPO. Than we will explore the the RLVR(Reinforcement Learning with Verifiable Rewards) technique and explore the algorithms such as GRPO, GSPO, DAPO, SAPO and the most SOTA GDPO. Than we will discuss current LLM-RL training framework such as Verl. Finally, we will discuss the challenges and future directions of applying RL in LLMs.

From Entropy to KL Divergence: A Comprehensive Guide

Mathematics

KL Divergence, also known as Kullback-Leibler Divergence, is a fundamental concept in information theory and statistics. In this blog post, we will first explore the concept of entropy, which is the foundation of KL Divergence. Then, we will delve into the concept of KL Divergence, its mathematical formulation, how to approximate use three different Monte Carlo Estimator, and finally its applications in various fields such as classification, generative modeling, knowledge disllation, reinforcement learning, and large language model. By the end of this post, you will have a solid understanding of KL Divergence and how to apply it in your own projects.

All kinds of “Learnings” in Deep Learning

Deep Learning

Summary

In the blog, I will cover all kind of learnings in Deep Learning, including supervised learning, unsupervised learning, self-supervised learning, reinforcement learning, and more. By the end, you will have a solid understanding of different learning paradigms in Deep Learning.

The Evolution of Position Encoding in the Transformer

Transformer

Large Language Model

In the blog, I will go through different position encoding machnism used in the Transformer-based neural network architecture. We will explore why the position encoding is important and different types of position encoding. In the end, we will extend the position encoding to visual field, such as in the Image and Video to unify the position encoding.

All About Diffusion & Flow Models

Generative-Model

Diffusion-Model

This article offers a comprehensive overview of diffusion models from multiple perspectives. We begin with the foundations—DDPM, DDIM, and Score Matching—and explore their relationships. From there, we introduce the ODE/SDE framework, showing how DDPM can be derived from stochastic differential equations and how this connects to Flow Matching. We then highlight key model variants such as Stable Diffusion and Movie Gen, discussing their architectures and applications. Finally, we broaden the scope to examine how diffusion models are being adapted beyond image generation, including diffusion policies in reinforcement learning and their emerging role in large language models (LLMs).

LLM Part2: Inference

Large Language Model

Inference

In this blog, we will going through the inference process of LLMs, including how to effectively use them for various tasks. We will explore techniques such as prompt engineering, few-shot learning, and more. By the end, you will have a solid understanding of how to leverage LLMs for your own applications.

LLM Part1: Transformer Architecture & Training Stages

Transformer

Large Language Model

This is the part 1 of the LLM series, architecture. In this blog, we will going through different transformer architectures that power LLMs. Such as position encoding, attention mechanisms, mixture of experts, and more. In the end, we will explore some SOTA LLM architectures, and study how to they combine those techniques to achieve better performance. We are going from bottom to up: Position Encoding, Attention Mechanism, Feed Forward Network & Mixture of Experts, Output Layer. Besides that, we will also explore different normalization techniques, such as Layer Normalization and RMS Normalization, and the different position of the normalization layers within the architecture. With the architecture in place, we will going through different training stages, such as pre-training stage, mid-training stage, and post-training training stage. By the end of this blog, you will have a comprehensive understanding of the transformer architecture and how to get your own LLM up and running.

Speed Up Training for Neural Networks

Training Tricks

Training large neural networks can be time-consuming and resource-intensive. This blog post explores various techniques to speed up the training process, including optimizing calculating in the single GPUs, and parallelizing the training across multiple GPUs.

Categories