Tag

#reinforcement-learning

Every item tagged reinforcement-learning, newest first.

25 items

i post-trained a model to reliably roll a die

A Reddit user post-trained a model to reliably roll a die, achieving roughly equal probability for each number. This demonstrates overcoming a common issue in reinforcement learning where models often rely on known strategies rather than exploring new actions. The experiment shows that with post-training, a model can learn to generate truly random outputs. This has implications for builders working on applications requiring unpredictable behavior.

Key takeaways

Model post-trained to roll a die with roughly equal probability for each number.
Demonstrates overcoming common RL issue of model relying on known strategies.
Experiment shows post-training can achieve truly random outputs.

rr/LocalLLaMA#reinforcement-learning #randomness #post-training

research14h

Learning User Simulators with Turing Rewards

Researchers propose Turing-RL, a reinforcement learning approach for training user simulator models based on the Turing Test. This method trains large language models to simulate human users by maximizing their ability to fool a human evaluator into thinking they are real. The approach aims to improve simulator realism and usefulness across applications like agent training and personalization evaluation.

Key takeaways

Turing-RL uses a Turing-Test-based reward to train user simulators.
Goal is to improve simulator realism for applications like agent training.
Method trains LLMs to fool human evaluators into thinking they are real users.

aarXiv#reinforcement-learning #user-simulation #turing-test

research16h

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Researchers analyzed token-level entropy dynamics in GRPO, a post-training paradigm for LLMs, and found a credit assignment mismatch causing policy entropy collapse. They propose STARE, a method to reweight advantages and stabilize policy entropy. This addresses a key limitation of GRPO, enabling more stable training of complex reasoning in LLMs. You can apply STARE to improve GRPO's performance in your own LLM training workflows.

Key takeaways

GRPO suffers from policy entropy collapse due to token-level credit assignment mismatch.
STARE reweights advantages to stabilize policy entropy in GRPO.
STARE improves stability of complex reasoning training in LLMs.

aarXiv#reinforcement-learning #llm-training #entropy-stability

research17h

Forecasting what Matters: Decision-Focused RL for Controlled EV Charging with Unknown Departure Times

Researchers propose a decision-focused reinforcement learning approach for controlling electric vehicle charging when departure times are unknown. The method learns effective charging policies from historical data despite missing key features. This approach aims to reduce peak demand and grid instability caused by EV adoption.

Key takeaways

Decision-focused RL handles unknown departure times in EV charging.
Method learns from historical data to optimize charging policies.
Aims to mitigate peak demand and grid instability from EV growth.

aarXiv#reinforcement-learning #energy-systems #electric-vehicles

research17h

Pareto Q-Learning with Reward Machines

Researchers introduced Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm that combines Pareto Q-Learning and Q-Learning with Reward Machines. PQLRM approximates the Pareto front by maintaining sets of vector-valued Q-estimates and exploits the factored automaton structure of the reward signal. This algorithm enables efficient handling of complex reward structures in multi-objective tasks. You can explore the approach in a new research paper.

Key takeaways

PQLRM combines Pareto Q-Learning and Q-Learning with Reward Machines.
Approximates Pareto front with vector-valued Q-estimates.
Exploits factored automaton structure of reward signal.

aarXiv#multi-objective #reinforcement-learning #research

research18h

Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems

This paper compares model-free controllers on a nonlinear system under cyberattacks, analyzing four RL reward types for accuracy, cost, and resilience. The Lyapunov reward offers the best resilience with low tracking error. Results inform builders on selecting RL controllers for resilient cyber-physical systems. The study provides insights into trade-offs between resilience, accuracy, and cost.

Key takeaways

Lyapunov reward offers best resilience with low tracking error
Exponential mode provides good trade-offs under moderate training
Progressive and linear rewards converge faster but are less robust

aarXiv#model-free #reinforcement-learning #cyber-physical-systems

research19h

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Researchers propose RODS, a method for online data synthesis that prioritizes high-variance, reward-driven samples for multi-turn tool-use agents. The approach targets the agent's capability boundary where successes and failures are balanced, yielding large policy gradients. RODS improves sample efficiency in reinforcement learning by focusing on the most informative data. You can apply this method to optimize data collection for your own RL agents.

Key takeaways

RODS synthesizes data online, prioritizing high-reward-variance samples.
Targets the agent's capability boundary for large policy gradients.
Improves sample efficiency in multi-turn tool-use RL.

aarXiv#reinforcement-learning #data-synthesis #multi-turn-tool-use

research20h

Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training

Researchers propose combining seed exploration and spot GPUs to reduce DiT RL post-training costs. Seed exploration selects high-contrast samples to improve convergence, while spot GPUs offer 69-77% lower costs. By synergizing both, the approach reduces overall training costs without increasing wall-clock time. This method benefits builders working with resource-intensive DiT models.

Key takeaways

Combining seed exploration and spot GPUs reduces DiT RL post-training costs.
Spot GPUs can be 69-77% cheaper than high-end GPUs.
Synergized approach doesn't increase wall-clock time.

aarXiv#reinforcement-learning #diffusion-models #cost-optimization

research1d

Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents

Researchers propose a method to learn red agent policy from observations for neurosymbolic autonomous cyber agents. The approach uses reinforcement learning and behavior trees with learning-enabled components. This method aims to improve autonomous cyber-defense in partially observable systems. You can apply this approach to develop more adaptive security systems.

Key takeaways

Uses reinforcement learning and behavior trees with learning-enabled components.
Aims to improve autonomous cyber-defense in partially observable systems.
Method learns red agent policy from observations.

aarXiv#reinforcement-learning #neurosymbolic-ai #cyber-security

research1d

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Researchers propose Zone of Proximal Policy Optimization (ZPPO), a method that combines knowledge distillation and reinforcement learning to improve small student model generalization. ZPPO uses teacher guidance in prompts rather than gradients, reducing mode concentration and improving performance on out-of-distribution tasks. The approach helps small models generalize better beyond their training data. You can explore ZPPO's potential applications in your own projects.

Key takeaways

ZPPO combines knowledge distillation and reinforcement learning for small student models.
Teacher guidance is provided in prompts, not gradients.
ZPPO improves generalization on out-of-distribution tasks.

aarXiv#reinforcement-learning #knowledge-distillation #small-models

researchJun 8

The Open Source Community is backing OpenEnv for Agentic RL

The open source community is rallying behind OpenEnv, a new framework for agentic reinforcement learning. OpenEnv aims to make it easier for developers to build and train agentic RL models. The framework has gained support from researchers and developers in the open source community. You can explore OpenEnv on Hugging Face.

Key takeaways

OpenEnv is a new framework for agentic reinforcement learning.
The framework has gained support from the open source community.
OpenEnv is available on Hugging Face.

HHugging Face Blog#open-source #agentic-rl #reinforcement-learning

researchMay 6

vLLM V0 to V1: Correctness Before Corrections in RL

The vLLM library upgraded from V0 to V1, shifting focus from post-hoc error correction to ensuring correctness in reinforcement learning from human feedback. The new version prioritizes accurate model outputs over subsequent corrections. This change aims to improve the reliability of AI systems by addressing errors at the source.

Key takeaways

vLLM library upgraded to V1 with new focus on correctness.
Prioritizes accurate model outputs over post-hoc corrections.
Aims to improve AI system reliability by addressing errors at the source.

HHugging Face Blog#reinforcement-learning #ai-reliability #library-update

researchMar 10

Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

Sixteen open-source reinforcement learning libraries were analyzed for their async training capabilities. The study highlights design patterns and challenges in building scalable async RL systems. You can apply these insights to improve the performance and efficiency of your own RL projects. Key findings include the need for better support of heterogeneous compute resources and more efficient data transfer mechanisms.

Key takeaways

16 open-source RL libraries were studied for async training.
Async RL systems require better heterogeneous compute support.
Efficient data transfer is a key challenge in async RL.

HHugging Face Blog#open-source #reinforcement-learning #async-training

modelsAug 14

Kimina-Prover-RL

The Kimina-Prover-RL model is a new open-source tool for automated theorem proving. It is based on reinforcement learning and has shown promising results in experiments. The model is available on the Hugging Face platform for developers to explore and build upon. You can access and integrate it into your projects.

Key takeaways

Kumina-Prover-RL uses reinforcement learning for theorem proving.
The model is open-source and available on Hugging Face.
It has shown promising results in initial experiments.

HHugging Face Blog#reinforcement-learning #automated-theorem-proving #open-source

researchJul 10

Kimina-Prover: Applying Test-time RL Search on Large Formal Reasoning Models

Researchers applied test-time reinforcement learning search to large formal reasoning models, improving performance on mathematical proof generation. The Kimina-Prover system was released on the Hugging Face platform. This development may interest builders working on AI-assisted formal verification and proof generation. The approach could enhance the efficiency of formal reasoning tasks.

Key takeaways

Test-time RL search improves performance on mathematical proof generation.
Kimina-Prover system released on Hugging Face platform.
Potential applications in AI-assisted formal verification.

HHugging Face Blog#formal-reasoning #reinforcement-learning #hugging-face

modelsJan 31

Mini-R1: Reproduce Deepseek R1 „aha moment“ a RL tutorial

The Mini-R1 project on Hugging Face provides a simplified reproduction of Deepseek's R1 'aha moment' using a reinforcement learning tutorial. This project allows you to explore and understand the concepts behind R1 through a hands-on game-like experience. The tutorial is designed to be accessible and educational, enabling builders to learn about reinforcement learning in a practical way. By engaging with Mini-R1, you can gain insights into the R1 model's capabilities and limitations.

Key takeaways

Mini-R1 reproduces Deepseek's R1 'aha moment' in a simplified tutorial.
Hands-on reinforcement learning experience provided.
Educational project for builders to learn about R1 and reinforcement learning.

HHugging Face Blog#reinforcement-learning #tutorial #open-source

researchJun 12

Putting RL back in RLHF

Researchers propose RLOOP, a modification to the popular RLHF framework that incorporates reinforcement learning from human feedback. RLOOP aims to improve model performance by leveraging human feedback more effectively. The approach has shown promising results in preliminary experiments. You can explore the RLOOP implementation on the Hugging Face platform.

Key takeaways

RLOOP modifies RLHF to better leverage human feedback.
Preliminary experiments show promising results.
Implementation available on Hugging Face platform.

HHugging Face Blog#reinforcement-learning #rlhf #human-feedback

researchOct 24

The N Implementation Details of RLHF with PPO

The blog post from Hugging Face details the implementation of RLHF with PPO, a technique used to fine-tune large language models. It provides a comprehensive overview of the process, including the mathematical formulation and practical considerations. Builders can use this information to implement RLHF with PPO in their own projects. The post aims to facilitate understanding and adoption of this technique.

Key takeaways

RLHF with PPO is a technique for fine-tuning large language models.
The process involves mathematical formulation and practical considerations.
Hugging Face provides a comprehensive overview of the implementation.

HHugging Face Blog#reinforcement-learning #large-language-models #fine-tuning

researchDec 9

Illustrating Reinforcement Learning from Human Feedback (RLHF)

The Hugging Face blog post explains Reinforcement Learning from Human Feedback (RLHF), a technique for training AI models to align with human preferences. RLHF involves collecting human feedback, training a reward model, and fine-tuning the AI model. This approach enables builders to create more accurate and relevant models.

Key takeaways

RLHF involves collecting human feedback to train AI models.
A reward model is trained to predict human preferences.
The AI model is fine-tuned based on the reward model.

HHugging Face Blog#reinforcement-learning #human-feedback #fine-tuning

modelsSep 8

Train your first Decision Transformer

You can train your first Decision Transformer using Hugging Face's open-source library. The library provides pre-built components and examples to get started with training Decision Transformers. This model type is suitable for sequential decision-making tasks. Builders can leverage these models for tasks like robotic control and automated planning.

Key takeaways

Hugging Face provides an open-source library for training Decision Transformers.
Decision Transformers are suitable for sequential decision-making tasks.
The library includes pre-built components and examples for getting started.

HHugging Face Blog#open-source #decision-transformers #reinforcement-learning

tutorialsJun 30

Policy Gradient with PyTorch

The Hugging Face blog post explains how to implement policy gradient methods using PyTorch. Policy gradient is a type of reinforcement learning algorithm. You can use it to train agents to make decisions in complex environments. The post provides a practical example of training an agent using PyTorch.

Key takeaways

Policy gradient is a type of reinforcement learning algorithm.
PyTorch can be used to implement policy gradient methods.
The Hugging Face blog post provides a practical example of training an agent.

HHugging Face Blog#reinforcement-learning #pytorch

researchMay 4

An Introduction to Deep Reinforcement Learning

This blog post provides an introduction to deep reinforcement learning, covering key concepts and techniques. It aims to help readers understand the basics of deep RL and its applications. You can learn about the fundamental components, including agents, environments, and rewards. The post is suitable for builders looking to explore RL in their projects.

Key takeaways

Covers key concepts and techniques in deep RL.
Suitable for readers new to deep reinforcement learning.
Explores applications and fundamental components of deep RL.

HHugging Face Blog#deep-learning #reinforcement-learning #intro

modelsMar 28

Introducing Decision Transformers on Hugging Face 🤗

Hugging Face has introduced Decision Transformers, a new library for decision-making tasks. This library enables builders to implement transformer-based models for complex decision-making scenarios. Decision Transformers can be used for tasks such as reinforcement learning and planning. You can access the library on the Hugging Face platform.

Key takeaways

Decision Transformers library is now available on Hugging Face.
Enables transformer-based models for decision-making tasks.
Supports reinforcement learning and planning applications.

HHugging Face Blog#transformers #decision-making #reinforcement-learning

modelsJan 21

Welcome Stable-baselines3 to the Hugging Face Hub 🤗

Stable-baselines3 has joined the Hugging Face Hub, providing a centralized location for reinforcement learning algorithms and pre-trained models. This integration enables easy access to stable-baselines3's implementations of popular algorithms like PPO, A2C, and DQN. You can now browse, use, and contribute to stable-baselines3 models directly within the Hugging Face ecosystem. The addition expands the Hub's offerings for builders working on AI projects.

Key takeaways

Stable-baselines3 algorithms and models are now on the Hugging Face Hub.
Enables easy access and contribution to PPO, A2C, DQN implementations.
Centralized location for reinforcement learning resources.

HHugging Face Blog#hugging-face #reinforcement-learning #open-source

researchDec 2

Introducing Snowball Fight ☃️, our first ML-Agents environment

Hugging Face introduced Snowball Fight, a new ML-Agents environment for training agents in a simulated snowball fight scenario. This environment allows researchers and developers to train and test reinforcement learning models in a fun and interactive way. The release provides a unique tool for exploring AI applications in gaming and simulation. You can access Snowball Fight on the Hugging Face platform.

Key takeaways

Snowball Fight is an ML-Agents environment for training agents in a simulated snowball fight.
It enables training and testing of reinforcement learning models in an interactive scenario.
Available on the Hugging Face platform for researchers and developers.

HHugging Face Blog#ml-agents #reinforcement-learning #gaming-simulation