Stanford CS329A | Self-Improving AI Agents

2026/8/21 22:41:28 · 来源

#ai-agents#knowledge#reinforcement-learning#stanford#test-time-compute#reasoning

这篇文章还没有中文版——当前显示英文原文。

This article outlines Stanford's CS329A course on self-improving AI agents, covering test-time compute, verification, reinforcement learning, reasoning, planning, and open-ended evolution frameworks.

Stanford CS329A | Self-Improving AI Agents

Course Overview Autumn 2025

    This course covers the latest techniques and applications of AI agents 
    that can continuously improve themselves through interaction with themselves and the environment. 
    The course will start with self-improvement techniques for LLMs, such as constitutional AI, 
    using verifiers, scaling test-time compute, combining search with LLMs, and train time scaling with RL. 
    We will then discuss the latest research in augmenting LLMs with tool use, code, and memory, 
    and orchestrating AI capabilities with multimodal interaction. 
    We will next discuss multi-step reasoning and planning problems for agentic workflows, 
    and the challenges in building robust evaluation frameworks. 
    
    Our goal is that the students learn from the latest research papers, 
    discuss the suggested readings in each class, work on an original research project in this area,
    and learn from invited academic and industry speakers about applications in building coding agents, 
    research assistants in STEM, and autonomous systems in robotics. 
    
  
      
      Schedule

      
          
        
          
            #
            Date
            Description
            Paper Readings*
            Deadlines
          
        
        
          
             1 
             Mon Sep 22 
            Course Overview
            
            
          
          
             2 
            Fri Sep 26
            Test-time Compute Scaling
            
                          
                              Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (Brown et al. 2024)
                              Archon: An Architecture Search Framework for Inference-Time Techniques (Saad-Falcon et al. 2024) 
                              Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al. 2024)
                              How Do Large Language Monkeys Get Their Power (Laws)?
                          
                      
            
          
          
3
Mon Sep 29

Robust Verification

         Shrinking the Generation-Verification Gap with Weak Verifiers
         Training Verifiers to Solve Math Word Problems (Cobbe et al. 2021)
         Let's Verify step by step (Lightman et al. 2023)
         Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (Wang et al. 2023)
             
    



          
             4 
            Fri Oct 3
            Learning from feedback with tools/code
            
                          
                               ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al. 2022)
                               RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
                               Constitutional AI: Harmlessness from AI Feedback
                          
                      
            
                Homework 1 out (due Oct 13)
            
          
          
             5 
             Mon Oct 6 
            Multi-step Reasoning/Planning
            
                          
                               SWiRL: Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
                               Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (Zhou et al. 2023)
                               SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models
                               ADaPT: As-Needed Decomposition and Planning with Language Models (Prasad et al. 2024)
                               Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search
                          
                      
                          
          
          
             6 
            Fri Oct 10
            Train Time Scaling/Scaling RL
            
                
                  STaR: Bootstrapping Reasoning With Reasoning (Zelikman et al. 2022)
                     DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
                     DAPO: An Open-Source LLM Reinforcement Learning System at Scale
                 
            
            Project Proposal due
          
          
             7 
            Mon Oct 13
            Open-Ended Evolution of Self-Improving Agents
            
                
                     Automated design of agentic systems
                     The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (Lu et al. 2024)
                     AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms
                
            
            
                Homework 2 out on Oct 14 (due Oct 22)
            
          
          
             8 
            Fri Oct 17
            Self improvement with Search & Deep Research Agents
            
                
                     Competition-Level Code Generation with AlphaCode
                     AlphaCode 2 Technical Report
                     Search-o1: Agentic Search-Enhanced Large Reasoning Models
                
            
            
          
          
             9 
            Mon Oct 20
            
                Guest Lecture
              Melvin Johnson (Google DeepMind)
            
            
              Evolution of Post-training from Chatbots to Agents
            
             
              
                     
                       
10
Fri Oct 24
Mid term presentations
  
                    
  Homework 3 out (due Nov 7)


          
             11 
            Mon Oct 27
            Mid term presentations
            
            
          

          
             12 
            Fri Oct 31
            Mid term presentations
            
            
          
          
             13 
            Mon Nov 3
            Agentic Frameworks for Software Engineering
               
                 
                              CodeMonkeys: Scaling Test-Time Compute for Software Engineering
                              KernelBench: Can LLMs Write Efficient GPU Kernels?
                               Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces
                          
                      
            
            

          
          
             14 
            Fri Nov 7
            
                 Augmenting Agents with Memory
               Guest Lecturer: Junchen Jiang (LMCache, UChicago)
            
                
                          
                               Cartridges: Lightweight and general-purpose long context representations via self-study
                               MemGPT: Towards LLMs as Operating Systems (Packer et al, 2023)
                               CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
                          
                      
            
          
          
             15 
            Mon Nov 10
            
                Guest Lecture
              Denny Zhou, Google DeepMind
            
            
              LLM Reasoning
            

            
          
          
             16 
            Fri Nov 14
            
Guest Lecture
Thang Luong, Google DeepMind

             Towards AI Superhuman Reasoning: AlphaProof, AlphaGeometry & Gemini IMO Gold Medal
            
            
          
          
             17 
            Mon Nov 17
            Agentic Evaluations & Long-Horizon Tasks
            
              
                               Measuring AI Ability to Complete Long Tasks
                               GDPVal: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
                               DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
                
            
            
          
          
             18 
            Fri Nov 21
             
                Guest Lecture
               Misha Laskin (Reflection AI)
            
            
              Building Agentic Systems for Autonomy: Lessons & Open questions 
            
            
          
          
             
            Mon Nov 24
            Holiday
            
            
          
          
             
            Fri Nov 28
            Holiday
            
            
          
          
             19 
            Mon Dec 1
            
                 Guest Lecture
Danny Driess (Physical Intelligence)
            
            
              Multimodal AI Agents in Robotics
            
            
          
          
             20 
            Fri Dec 5
            Future Research Areas
            
            
          
          
             
            Wed Dec 10
            Final Project Due
            
            Final project due (EoD)
          
          
             
            Fri Dec 12
            Final Project Poster Presentation
            
            
          
        
      
      *Paper readings may be updated closer to the class date.

    
  
  Homework Assignments
  There will be three homework assignments. Homework 1 will be released on Oct 3. Homework 2 will be released on Oct 13. Homework 3 will be released on Oct 23. The homeworks will help develop intuition for the basics of self-improvement, multi-step reasoning, and tool use techniques.

  Research Projects
  As a graduate seminar, research is a big part of the class. Students will work in teams of 2 to 4 to complete original 
  research. These projects should be broadly around research areas discussed in class and benchmarks related to agent workflows. Students will receive API credits to support their 
  development work.

  Course Policies
  
  Late Policy
  
    All students have 4 free late days for the quarter.
    You may use up to 2 late days per assignment with no penalty.
    Late days can be used for assignments, project proposal, and project milestone.
    Late days cannot be used for the final project report.
    There will be a 25% penalty for each additional late day.
    No exceptions to this policy.
  

  Audit Policy
  Audits are not allowed for this course.

  Communication with Course Staff
  
    Please read the course documentation carefully before asking general questions.
    Questions should be asked during office hours or on EdStem.

Source: https://cs329a.stanford.edu/

Stanford CS329A | Self-Improving AI Agents

Paper Authors Poster

      AppBench: Benchmarking AI-Generated Web Applications
      Ethan Hellman, Brendan McLaughlin, Abhinav Lalwani, Belinda Mo
      Here
    
    
      PRIME: Planning with Reflective Iterative Multi-Agentic Exploration
      Chelsea Zou, Samuel Liu, Jui Khankari
      Here
    
    
      Enhancing Mathematical Reasoning in Large Language Models through Reasoning Distillation, GRPO, and Multi-agent PRM Reranking
      Abhinav Agarwal, Carlo Baronio, Shree Reddy, Shubhra Mishra
      Here
    
    
      Towards Effective Meta-Agents: Designing Agents that Design Agents
      Batu El, Mert Yuksekgonul, James Zou
      Here
    
    
      ARCHON Augmented: Planning and Web-Enhanced Components
      Megan Mou, Sherry Xie, Emily Zhang, Andrew Park
      Here

Source: https://cs329a.stanford.edu/pastprojects.html

CS329A Self-Improving AI Agents

CS329A Self-Improving AI Agents - YouTube

Source: https://www.youtube.com/playlist?list=PLangBM27OtEA

学习地图

Stage 1: Foundations of Inference Scaling

Learn test-time compute scaling techniques to understand how repeated sampling and search improve model outputs.
Explore robust verification methods like process-based reward models to bridge the generation-verification gap.

Stage 2: Reasoning, Planning, and Tool Integration

Study multi-step reasoning frameworks like ReAct and tree search to enable complex agentic workflows.
Investigate train-time scaling and reinforcement learning systems for math and code generation.

Stage 3: Autonomous Evolution and Systems

Examine open-ended evolution systems and automated agent design paradigms.
Build long-horizon evaluation benchmarks and deploy memory augmentations like MemGPT.

动手实践——分步指南

Set up a Python virtual environment and install standard LLM orchestration libraries like LangChain or LlamaIndex.
Implement a basic repeated sampling script that generates multiple candidate answers for a math problem.
Write a simple verifier function or rule-based evaluator to score the correctness of the candidate outputs.
Combine the generator and verifier into a primitive best-of-N search loop to scale inference compute.
Test your pipeline with an open-source model API and evaluate the performance gains.

三大推荐资源

1
Stanford CS329A Official Website
The official course homepage featuring schedules, lecture topics, and required paper readings.
https://cs329a.stanford.edu/
2
Let's Verify Step by Step
A landmark paper on process-based supervision for training verifiers in mathematical reasoning.
https://arxiv.org/abs/2305.20050
3
MemGPT GitHub Repository
An open-source framework treating LLMs as operating systems with memory management capabilities.
https://github.com/cpacker/MemGPT

链接由 AI 推荐——使用前建议快速核实。