# Meeting #4

Friday, 3/2/2022

## Table of Contents

## Agenda

- Presentations, 2 groups + discussion
`~20 min each`

- Research/project ideation work time
- Develop a possible project list based on presentations

Meeting Recording for 1st Half, Meeting Recording for 2nd Half

## Deliverables for Next Week

Groups that have not presented this week will be giving presentations next week.

## Notes

- Motivation - algorithms for computing don’t learn like humans do, how can we make them more like humans?
- Take ideas and inspiration from neuroscience and manifest them in computing

### Reinforcement Learning

- Reinforcement - when you do something over and over again.
- Learning - when information goes into your brain.
- Combining reinforcement and learning - put information into your brain, make sure it stays there, and repeat.
- Oxford Dictionary - defines intelligence as the ability to acquire and apply knowledge and skills
- Reinforcement Learning - acquires knowledge as machines find the best possible behavior, done by learning from mistakes.
- Ties to psychology - operant conditioning. Positive vs negative reinforcement.
- Positive reinforcement gives a stimuli (either a reward or punishment) after an event
- Negative reinforcement just doesn’t respond after an event / takes away the stimuli

- Reinforcement learning in the field of Computer Science.
- Reward hypothesis - any goal can be formalized as the outcome of maximizing a cumulative reward.
- Formulate optimization problems as Markov Decision Processes - nodes represent states; an agent can take an action with some probability of doing that to get to a different node / state
- Recent advancements in Reinforcement Learning.
- Applications: self-driving cars, games (tetris, snake, etc.)

- Deep Q Learning to estimate values of possible actions given the state

Access slides here

### Neuromorphic Computing

See Chris Kang’s video on Neuromorphic Computing.

- Goal - quick survey of the field and research. Opportunities for neuromorphic computing algorihtms and applications.
- Brains have desirable properties - energy efficient, fast at learning, use unique computational operations
- Hardware and software need to be co-designed: they can’t exist in isolation
- Can be pretty foreign for most computer scientists.
- If we change the underlying hardware, we must change the paradigm
- Optimizing algorithms on phyiscal manisfestations

- Applications - why do we care?
- Edge computing (energetically efficient)
- Machine learning (rapidly training and flexibility)
- Coprocessor in heterogeneous systems

- Edge computing - Loihi graph
- ANNs to GPU are mathematical and abstractive approaches - we can try to run native simulations in hopes of being more efficient.
- Set up the neural network on-device in which the neurons are built into the hardware
*on the chip*. - Physical systems through hardware can be directly executed
- If we do work with these, will be using simulations of neuromorphic chips.
- Coprocessor for novel domains - how can neuromorphic computing be used for differential equation solving, graph problems, optimization, etc.
- We can continue the spirit of Moore’s law using heterogeneous systems
- Spiking Neural Network - generalizes to very broad neural networks, all nerual networks.
- Use spike-based inputs instead of typical one-hot vectors or scalar values.
- Spikes are temporal
- Focus on being temporal and event-driven
- Signals that propagate take time to reach their destinations
- The focus is being on being able to obtain a threshold and sending a signal once it is reached

- Theoretical guarantees of Spiking Neural Networks - SNNs are a superset of ANN functionality.
- Hardware neurmophic architectures: silicon-based, exotic materials (eggs, etc.)
- Co-design algorithms to the hardware
- SNNs are generalized ANNs. ML: quasi-backpropagation. Turn existing ANns into SNNs. Resevoir appraoch - take a soup of neurons and have a normal-ML interpret at the end.
- Non-ML question - how to map existing algorithms to graph theory and optimization algorithms.
- How to engineer neuronal properties in physical materials - lots of research in materials science.
- Open questions
- Applications
- Algorithms - theoretical guarantees on SNNs, convergence - SNNs are a superset of ANNs, but just because SNNs can represent ANN functions does not necessarily mean that they can converge to those functions. Increasing the space dimensionality makes the problem harder.
- Co-design - engineering relevant properties into the hardware to open up exploitation of desired properties.
- Hardware - materials science and material discovery, architecture desing and fabrication

- Neuron - can be mapped as an electrical circuit. ANN is modeled by the RC circuit properties of a neuron.
- Newman computing - CPU, memory, stream bits back and forth. Neuromorphic computing - event-based, more distributed and agglomeration of neurons, no separation of computation and memory