AI Safety at UCLA Intro Fellowship: Governance Track

Table of Contents

Part 1: Introduction AI Safety

  1. Week 0: Overview, Ethos, and Social
  2. Week 1: Artificial Intelligence — How it Works and What it Can Achieve
  3. Week 2: Catastrophic Risk from AI
  4. Week 3: AI Safety — Goals and Challenges

Part 2: Introduction to AI Governance

  1. Week 4: AI Policy Levers
  2. Week 5: Existing Approaches — Corporate Governance & Open vs. Closed Source
  3. Week 6: New Approaches — Compute Governance & International Approaches
  4. Week 7: Looking Ahead

Part 1: Introduction to AI Safety

Week 0: Overview, Ethos, and Social

Core Content (~10 min):

  1. Fellowship Curriculum (this page)

Optional Additional Content:

  1. Scope Insensitivity (10 min)
  2. Biden’s Executive Order on Safe, Secure, and Trustworthy AI (10 min)

Learning Goals:

  1. Get to know the fellowship and each other!
  2. Get a flavor for the ethos and motivations of the fellowship
  3. Optionally, get acquainted with recent AI legislation

Week 1: Artificial Intelligence — How it Works and What it Can Achieve

Core Content (~60 min):

  1. Andrej Karpathy - Intro to Large Language Models (60 min)

Optional Additional Content:

AI History

  1. Visualizing the Deep Learning Revolution by Richard Ngo (20 min)
  2. Nvidia: The chip maker that became an AI superpower (5 min)
  3. Compute Trends Across Three Eras of Machine Learning (10 min)

How AI Works

  1. AI, Machine Learning, and Deep Learning (10 min)
  2. Gradient Descent: How Neural Networks Learn | Chapter 2, Deep Learning (20 min)
  3. What is Self Supervised Learning? (5 min)

AI Futures

  1. The economic potential of generative AI: The next productivity frontier
  2. Forecasting Transformative AI, Part 1: What Kind of AI?
  3. The Transformative Potential of AGI – and When It Might Arrive by Shane Legg and Chris Anderson

Learning Goals:

  1. Understand the technical backbone of modern AI models
  2. Build a sense of why these technical details are relevant to AI policy, in the context of the AI triad (compute, data, and algorithms)
  3. Begin to think about what risks might be posed by AI

Week 2: Catastrophic Risk from AI

Core Content (~60 min):

  1. Existential Risk from Power-Seeking AI (60 min)

Optional Additional Content:

First Principles AI Safety

  1. AGI Safety from First Principles
  2. Why Would AI Want to do Bad Things? Instrumental Convergence
  3. Why AI Alignment Could Be Hard with Modern Deep Learning by Ajeya Cotra (20 min)

Surveys of AI Risks

  1. AI Risks that Could Lead to Catastrophe | CAIS (25 min)
  2. Preventing an AI-Related Catastrophe - 80,000 Hours (60 min)

Concrete Scenarios

  1. What Failure Looks Like by Paul Christiano (20 min)
  2. Auto-GPT and AI Race Acceleration by The AI Beat (10 min)
  3. The True Story of How GPT-2 Became Maximally Lewd (14 min)

Learning Goals:

  1. Understand the core arguments for existential risk from AI
  2. Begin to form an idea of the different paths to reducing AI risk
  3. Visualize how the techniques used to train AI directly contribute to potential bad outcomes

Week 3: AI Safety — Goals and Challenges

Core Content (~60 min):

  1. Paradigms of AI Alignment: Components and Enablers (34 min)
  2. Avoiding Extreme Global Vulnerability as a Core AI Governance Problem (10 min)
  3. AI Safety Seems Hard to Measure (16 min)

Optional Additional Content:

AI Safety Neglectedness

  1. Nobody’s on the Ball on AI Alignment (15 min)
  2. The Need for Work on Technical AI Alignment by Daniel Eth (25 min)

What Could Go Wrong

  1. AGI Ruin: A List of Lethalities (20 min)
  2. Rogue AIs by the Center for AI Safety (35 min)
  3. Racing through a minefield: the AI deployment problem (18 min)

Paths to Success

  1. Paradigms of AI Alignment: Components and Enablers (34 min)
  2. Managing Extreme AI Risks Amid Rapid Progress (20 min)
  3. What is AI Alignment? – BlueDot Impact (10 min)

Learning Goals:

  1. Understand the term “AI alignment” — what it means, and paths to achieving it
  2. Understand the difficulties that arise when trying to align powerful AI models
  3. Build a framework for the various factors that exascerbate AI risk, and how each of these could potentially be mitigated

Part 2: Introduction to AI Governance

Week 4: AI Policy Levers

Core Content (~60 min):

  1. The AI Triad and What It Means for National Security Strategy (45 min)
  2. Primer on Safety Standards and Regulations for Industrial-Scale AI Development (15 min)

Optional Additional Content:

  1. Strengthening Resilience to AI Risk: A Guide for UK Policymakers (30 min)
  2. The Policy Playbook: Building a Systems-Oriented Approach to Technology and National Security Policy (45 min)
  3. The Convergence of Artificial Intelligence and the Life Sciences: Safeguarding Technology (8 min)
  4. Historical Case Studies of Technology Governance and International Agreements (30 min)

Learning Goals:

  1. Understand the direct implications of the AI triad for policy
  2. Learn about existing standards for AI
  3. Gain historical context on the successes and failures of past technology governance

Week 5: Existing Approaches — Corporate Governance & Open vs. Closed Source

Core Content (~60 min):

  1. AI Index Report 2024, Chapter 7: Policy and Governance (20 min)
  2. Meta’s Chief AI Scientist Yann LeCun talks about the future of artificial intelligence (6 min)
  3. Open Sourcing Highly Capable Foundation Models (34 min)

Optional Additional Content:

US AI Policy

  1. Recent U.S. Efforts on AI Policy (8 min)
  2. President Biden’s Executive Order on AI (10 min)

Principles and Recommendations

  1. The Bletchley Declaration (10 min)
  2. UNESCO’s Recommendation on the Ethics of AI (20 min)
  3. OECD AI Principles (10 min)

Open vs. Closed Source

  1. A Pro-Innovation Approach to AI Regulation (30 min)
  2. The Case for Uncensored Models (5 min)

Learning Goals:

  1. Understand the term “corporate governance,” and existing policies held by AI labs
  2. Weigh the pros and cons of open-sourcing model weights
  3. Understand the government’s role in the regulation of AI companies

Week 6: New Approaches — Compute Governance & International Approaches

Core Content (~60 min):

  1. Computing Power and the Governance of AI (20 min)
  2. What Should be Internationalized in AI Governance? (40 min)

Optional Additional Content:

Compute Governance

  1. Choking Off China’s Access to the Future of AI (15 min)
  2. Computing Power and the Governance of AI (45 min)
  3. Primer on AI Chips and AI Governance (20 min)

Institutions and Policies

  1. China’s AI Regulations and How They Get Made (20 min)
  2. Driving U.S. Innovation in Artificial Intelligence: A Roadmap for AI Policy (30 min)
  3. High-Level Summary of the AI Act (10 min)
  4. Vision Statement of the US AI Safety Institute (15 min)
  5. International Institutions for Advanced AI (20 min)

AI Control 5. Model Evaluation for Extreme Risks by Toby Shevlane (35 min) 6. Societal Adaptation to Advanced AI (40 min)

Learning Goals:

  1. Understand the term “compute governance,” and why regulating compute is a promising path for mitigating AI risk
  2. Survey existing international instiutions for AI, and proposals for new institutions
  3. Gain context on the state of AI in China, the US’s primary competitor in the field

Week 7: Looking Ahead

Core Content (~60 min):

  1. Career Profile: AI Governance and Policy by 80000 Hours (15 min)
  2. AI Governance Project Ideas – BlueDot Impact (10 min)
  3. Collection of AI Governance Research Ideas - Markus Anderlung (20 min)
  4. Advice for Undergraduates (15 min)

Optional Additional Content:

Skills to Build

  1. 10 skills you need to grow your public policy career
  2. Aptitudes for AI Governance Work

Advice and Ideas

  1. Advice for Seeking Full-Time Roles (8 min)
  2. 12 Tentative Ideas for U.S. AI Policy by Muehlhauser (5 min)
  3. So You Want to Be a Policy Entrepreneur? by Michael Mintrom (40 min)
  4. What’s next in AI Governance (58 min)

Career Resources

  1. Career Resources on US AI Policy (5 min)
  2. Career Resources on US AI Strategy Research (12 min)
  3. AI Policy Resources
  4. AI Safety Career Opportunities Job Board

Learning Goals:

  1. Understand what careers exist in the AI governance space, and the skills required for each
  2. Browse proposals for AI governance and take note of the ones you may be interested in pursuing
  3. Understand the resources available to you if you are interested in pursuing AI governance