Link Search Menu Expand Document

Meeting #7

Monday, 5/9/2022

Table of Contents

  1. Lineup
  2. Papers Discussed
  3. Presentation Slides
  4. Recording
  5. Notes
  6. Photos


AndreMeasuring Intelligence

Papers Discussed

Presentation Slides

Open in a new tab here.




Awesome notes courtesy of Janna!


How do people measure intelligence?

  • Framework to ascribe human qualities to machines and programs

What does it mean to say that a machine is “conscious”?

  • Equation for intelligence
  • ARC is a concrete dataset where we can optimize intelligence

Turing: Computing Machinery and Intelligence (1950)

Can machines think?

  • Do not answer on the basis on democracy or intuition
  • Imitation Game


How do we build machines to do imitation game?

What is the machine vs. human?

  • Originally articulated in terms of gender
  • Most people during this time thought that machines can’t think
  • Identified learning as a key to AI


  • Only binary yes or no answer

Justifying the Imitation Game

  • Seperates physical and intellectual capacities
    • Truly measure the intelligence of the system
      • I.e. what does it mean to have a language to think things?
        • Does it know what an “apple” is if it hasn’t experienced yet?
  • If you could artithmetic faster, you are a thinking machine

Only care about input → output

  • How can we have a machine that demonstrate human thinking behavior but not be thinking?
  • General philosophy has been favorable in scope of history
    • Anti model agnostic
      • “There’s no way that this race or animal can think like we do!”

Digital Computers

  • Can do anything that humans can do — human computers are just higher than today’s calculators
  • Digital computers vs. Nervous system
    • Both electrical
    • Thoughts from nervous system are just complex hierarchies of digital ops

Learning Machines

  • Simulate a child‘s mind
    • Simulate its education and evolution into an adult brain

McCarthy: Ascribing Mental Qualities to Machines

How can we ascribe mental qualities like beliefs, intentions, and wants to machines?

What is legitimate to ascribe?

  • You can say that a machine is “conscious”
    • Term for the “self”

→ Why ascribe things at all?

  • Helps understand the structure of the machine and its temporal behavior
  • Want to describe the machine and its state
  • Need language of mental qualities to describe machines that represent higher level organization
  • Computers can perform abstracted tasks
    • Go one level further where its demonstrating complex qualities

Separate mental qualities from motivational structures

  • When you think of “feeling”, you might associate that with motivational structures, however
  • Use mental qualities to understand the internals of the system

Systems with mental qualities

  • Thermostats, self-reproducing cellular automata, computer time-sharing systems, programs designed to reason

Chollet: On the Measure of Intelligence

We need explicit intelligence metrics

  • Concrete metric → easier to measure → move towards AI

uring test and variants

  • So much influence over intelligence of machines
  • Intelligence is tied to how much the interrogator knows
  • As we become more acquainted to non human intelligence
    • Generator fooling discriminator

Two different understandings of intelligence:

1) Task specific skill

  • Mind is an arrangement of ~static specialized mechanisms fine-tuned through evolution
    • i.e. How do our eyes evolve to see well?

2) Generality and adaptation

  • Mind is a general purpose algorithm
    • Arbitrary experience → knowledge and skills

Chollet believes that both views are flawed.

Task specific skills of DL:

  • Primary way is bench marking to standardized metrics
  • Metrics are key to the modern deep learning

Metrics for task specific skill:

  • “AI Effect”
    • When AI does something new, it’s not really “thinking”

There is no single task X such that skill in X demonstrates intelligence

  • When the machine beats world champion, we don’t think that machine is intelligent
    • Machine built to do that one thing is not very “intelligent”
    • Narrow skills are impressive in the context of generality
    • Humans playing chess is impressive because it’s built upon cognitive skills

Generalization in AI:

System-centric generalization

  • Interpolation
    • Can the system generalize stuff based on what’s given?

Developer-aware generalization

  • Extrapolation
    • Can the machine see beyond the system?
      • Give machine a new point (x=) which is beyond what the system has seen

Current Efforts for Broad AI Evaluation:

Generalization in Reinforcement Learning

  • Exposed to environment via exploration
    • Only for trivial modifications
  • Multitask benchmarks
    • How does the machine perform with other skills?

Where do we fall short?

Surpassing humans in skill

  • “Moonshots”, AlphaGo & AlphaZero, DotA2, “AI beats human”

Developing broad abilities

  • Learning to learn
  • Acquiring new skills
  • General, flexible

New Perspectives

How do we evaluate a skill?

  • Chess can be abstract when humans play it, but it does not need abstraction
  • Think of the difficulty of abstraction
  • Learning hard-coded knowledge from data
    • Rigorously control the priors, experiences, and g-word (generalization)

Universal intelligence is a scam; must be anthropocentric

  • Obtain universal intelligence vs. Simulated human intelligence
    • Progress should be benchmarked against human intelligence
      • Recognize that this anthropocentrism is not greedy
        • Property not restrictive to humans
        • When comparing AI, compare it to human mode of thinking intelligence

Priors - where intelligence starts from

  • Most intelligence is acquired; it is not innate
    • Acquired via interaction through environment
      • Learning algorithm
  • Human cognitive priors
    • Low level — teeth chatter
    • Meta-learning — causality (making sound when one hits the table)
    • Knowledge — visual object-ness, Euclidean spaces, goals, social

Intellectual progress paved so far

  • Quantify the strength of adaptability
  • General AI must be benchmarked against human
    • There is no meaningful concept of intelligence that humans can pursue

    The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty.

    • Experience → Skill
      • Great way to think about intelligence


Also courtesy of Janna






I2 - Fusing neuroscience and AI to study intelligent computational systems. Contact us at