AI Safety at UCLA Intro Fellowship: Governance Track

Table of Contents

Part 1: Introduction AI Safety

  1. Week 0: Overview, Ethos, and Social
  2. Week 1: Artificial Intelligence — How it Works and What it Can Achieve
  3. Week 2: Catastrophic Risk from AI
  4. Week 3: AI Safety — Goals and Challenges

Part 2: Introduction to AI Governance

  1. Week 4: AI Policy Levers
  2. Week 5: Existing Approaches — Corporate Governance & Open vs. Closed Source
  3. Week 6: New Approaches — Compute Governance & International Approaches
  4. Week 7: Looking Ahead

Part 1: Introduction to AI Safety

Week 0: Overview, Ethos, and Social

Core Content (~10 min):

  1. Fellowship Curriculum (this page)

Optional Additional Content:

  1. Scope Insensitivity (10 min)
  2. Biden’s Executive Order on Safe, Secure, and Trustworthy AI (10 min)

Learning Goals:

  1. Get to know the fellowship and each other!
  2. Get a flavor for the ethos and motivations of the fellowship
  3. Optionally, get acquainted with recent AI legislation

Week 1: Artificial Intelligence — How it Works and What it Can Achieve

Core Content (~60 min):

  1. Andrej Karpathy - Intro to Large Language Models (41 min)
  2. The AI Triad and What it Means for National Security Strategy by Ben Buchanan (intro + section 1) (20 min)

Optional Additional Content:

  1. AI, Machine Learning, and Deep Learning (10 min)
  2. Gradient Descent: How Neural Networks Learn | Chapter 2, Deep Learning (20 min)
  3. Visualizing the Deep Learning Revolution by Richard Ngo (20 min)
  4. The Transformative Potential of AGI – and When It Might Arrive by Shane Legg and Chris Anderson

Learning Goals:

  1. Understand the technical backbone of modern AI models
  2. Build a sense of why these technical details are relevant to AI policy, in the context of the AI triad (compute, data, and algorithms)
  3. Begin to think about what risks might be posed by AI

Week 2: Catastrophic Risk from AI

Core Content (~75 min):

  1. Existential Risk from Power-Seeking AI (60 min)
  2. The True Story of How GPT-2 Became Maximally Lewd (14 min)

Optional Additional Content:

First Principles AI Safety

  1. AGI Safety from First Principles
  2. Why Would AI Want to do Bad Things? Instrumental Convergence
  3. Why AI Alignment Could Be Hard with Modern Deep Learning by Ajeya Cotra (20 min)

Surveys of AI Risks

  1. AI Risks that Could Lead to Catastrophe | CAIS (25 min)
  2. Preventing an AI-Related Catastrophe - 80,000 Hours (60 min)

Concrete Scenarios

  1. What Failure Looks Like by Paul Christiano (20 min)
  2. Auto-GPT and AI Race Acceleration by The AI Beat (10 min)

Learning Goals:

  1. Understand the core arguments for existential risk from AI
  2. Begin to form an idea of the different paths to reducing AI risk
  3. Visualize how the techniques used to train AI directly contribute to potential bad outcomes

Week 3: AI Safety — Goals and Challenges

Core Content (~60 min):

  1. What is AI Alignment? – BlueDot Impact (10 min)
  2. Avoiding Extreme Global Vulnerability as a Core AI Governance Problem (10 min)
  3. AI Safety Seems Hard to Measure (18 min)
  4. Racing Through a Minefield: the AI deployment problem (18 min)

Optional Additional Content:

AI Safety Neglectedness

  1. Nobody’s on the Ball on AI Alignment (15 min)
  2. The Need for Work on Technical AI Alignment by Daniel Eth (25 min)

What Could Go Wrong

  1. AGI Ruin: A List of Lethalities (20 min)
  2. Rogue AIs by the Center for AI Safety (35 min)

Paths to Success

  1. Paradigms of AI Alignment: Components and Enablers (34 min)
  2. Managing Extreme AI Risks Amid Rapid Progress (20 min)

Learning Goals:

  1. Understand the term “AI alignment” — what it means, and paths to achieving it
  2. Understand the difficulties that arise when trying to align powerful AI models
  3. Build a framework for the various factors that exascerbate AI risk, and how each of these could potentially be mitigated

Part 2: Introduction to AI Governance

Week 4: AI Policy Levers

Core Content (~60 min):

  1. The AI Triad and What It Means for National Security Strategy (pgs 11-15) (15 min)
  2. Primer on Safety Standards and Regulations for Industrial-Scale AI Development (15 min)
  3. Historical Case Studies of Technology Governance and International Agreements (30 min)

Optional Additional Content:

  1. Strengthening Resilience to AI Risk: A Guide for UK Policymakers (30 min)
  2. The Policy Playbook: Building a Systems-Oriented Approach to Technology and National Security Policy (45 min)
  3. The Convergence of Artificial Intelligence and the Life Sciences: Safeguarding Technology (8 min)

Learning Goals:

  1. Understand the direct implications of the AI triad for policy
  2. Learn about existing standards for AI
  3. Gain historical context on the successes and failures of past technology governance

Week 5: Existing Approaches — Corporate Governance & Open vs. Closed Source

Core Content (~65 min):

  1. AI Index Report 2024, Chapter 7: Policy and Governance (20 min)
  2. Open Sourcing Highly Capable Foundation Models (45 min)

Optional Additional Content:

US AI Policy

  1. Recent U.S. Efforts on AI Policy (8 min)
  2. President Biden’s Executive Order on AI (10 min)

Principles and Recommendations

  1. The Bletchley Declaration (10 min)
  2. UNESCO’s Recommendation on the Ethics of AI (20 min)
  3. OECD AI Principles (10 min)

Open vs. Closed Source

  1. A Pro-Innovation Approach to AI Regulation (30 min)
  2. The Case for Uncensored Models (5 min)

Learning Goals:

  1. Understand the term “corporate governance,” and existing policies held by AI labs
  2. Weigh the pros and cons of open-sourcing model weights
  3. Understand the government’s role in the regulation of AI companies

Week 6: New Approaches — Compute Governance & International Approaches

Core Content (~60 min):

  1. Primer on AI Chips and AI Governance (20 min)
  2. International Institutions for Advanced AI (20 min)
  3. China’s AI Regulations and How They Get Made (20 min)

Optional Additional Content:

Compute Governance

  1. Choking Off China’s Access to the Future of AI (15 min)
  2. Computing Power and the Governance of AI (45 min)

Institutions and Policies 2. Driving U.S. Innovation in Artificial Intelligence: A Roadmap for AI Policy (30 min) 3. High-Level Summary of the AI Act (10 min) 4. Vision Statement of the US AI Safety Institute (15 min)

AI Control 5. Model Evaluation for Extreme Risks by Toby Shevlane (35 min) 6. Societal Adaptation to Advanced AI (40 min)

Learning Goals:

  1. Understand the term “compute governance,” and why regulating compute is a promising path for mitigating AI risk
  2. Survey existing international instiutions for AI, and proposals for new institutions
  3. Gain context on the state of AI in China, the US’s primary competitor in the field

Week 7: Looking Ahead

Core Content (~60 min):

  1. Career Profile: AI Governance and Policy by 80000 Hours (15 min)
  2. AI Governance Project Ideas – BlueDot Impact (10 min)
  3. 12 Tentative Ideas for U.S. AI Policy by Muehlhauser (5 min)
  4. Advice for Undergraduates (15 min)
  5. Career Resources on US AI Policy (5 min)
  6. Career Resources on US AI Strategy Research (12 min)

Optional Additional Content:

Skills to Build

  1. 10 skills you need to grow your public policy career
  2. Aptitudes for AI Governance Work

Advice and Ideas

  1. Advice for Seeking Full-Time Roles (8 min)
  2. Collection of AI Governance Research Ideas
  3. So You Want to Be a Policy Entrepreneur? by Michael Mintrom (40 min)

Career Resources 4. AI Policy Resources 7. AI Safety Career Opportunities Job Board

Learning Goals:

  1. Understand what careers exist in the AI governance space, and the skills required for each
  2. Browse proposals for AI governance and take note of the ones you may be interested in pursuing
  3. Understand the resources available to you if you are interested in pursuing AI governance