WS 2005/2006 Computational Neuroscience III: Theory of Neural Systems Lecture Dr. Laurenz Wiskott Dr. Martin Stemmler Exercises Dr. Martin Stemmler Henning Sprekeler Computer Course Mathias Franzius Tiziano Zito

Scripts and Links:

• News: The projects for the computer course can be found here .
• Backpropagation Algorithm (script in german)
• For whoever is interested: The sound example of the network who learns to talk (NETtalk) and the
  original publication by Sejnowski & Rosenberg (link)
• Script for the lecture on Hebbian learning (pdf-format with small papersize).

Contents

1. Single Neurons:
   • Weights, threshold and non-linearity
   • McCulloch-Pitts unit
2. Neural Networks: An overview
   • Feedforward vs. recurrent networks
   • Feedforward networks: Function approximators and classifiers
   • Recurrent networks:
     • Synchronous vs. asynchronous update
     • Stable states, limit cycles
     • Attractor bassins
3. Linear Classification (Perceptron)
   • Architecture
   • Linear separability
   • Units without threshold
   • Learning rule
   • Difficult data sets and speed-up of convergence
   • (Convergence proof)
4. Linear regression
   • Learning problem
   • Error function
   • Gradient descent learning rule
   • Incremental learning rule
   • Difficult error landscapes
5. Backpropagation algorithm (script in german)
6. Generalization
   • Bias-Variance Dilemma
   • Bias and Variance
   • Cross Validation
   • Control of Network Complexity
7. Catastrophic Interference
   • Problem
   • Interleaved Training
   • Dual Networks
8. Hebbian Learning:
   • Explicit and Implicit Weight Normalization
   • Linear Stability Analysis
   • Learning Principle Subspaces
   • Principle Components of Natural Images
9. Receptive Fields in the Visual System
10. Coding Schemes:
    • Compact Coding and Principle Component Analysis
    • Sparse Coding
    • Independent Component Analysis (ICA)
11. Slow Feature Analysis
    • The Slowness Principle
    • Nonlinear Expansion
    • The SFA Algorithm
    • Receptive Fields of Complex Cells
12. Competitive Learning
    • Vector Quantization/Clustering
13. Neural Map Formation
    • Cortical Maps
    • Selforganizing Feature Maps
14. Recurrent Networks: Introduction
    • What is computation?
    • Model of the network dnamics
    • Single unit with feedback
15. Auto-associative memories
    • Stability of a single memory pattern
    • Stability of multiple patterns
    • Memory capacity of the network
    • Network dynamics: network flow diagrams
16. Lyapunov functions

Literature

Some of the books listed below can be found at the Zweigbibliothek Physik in the areas ST 152 and ST 300.

• Amit, D. J. (1989). Modeling Brain Function: The world of Attractor Neural Networks. Cambridge University Press.
  (systematic derivation of Hopfield-networks based on statistical physics)
• Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press.
  (systematic derivation of feed-forward networks based Bayesian theory)
• Hertz, J., Krogh, A., and Palmer, R. G. (1991). Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, CA.
  (a good english textbook, especially for physicists)
• Kandel, E. R., Schwartz, J. H., and Jessel, T. M., editors (1995). Neurowissenschaften: Eine Einführung.
  Spektrum Akademischer Verlag, Heidelberg.
  (a good introduction to neurobiology)
• Rojas, R. (1996). Theorie der neuronalen Netze: Eine systematische Einführung. Springer Verlag, Berlin. 4., korrigierter Nachdruck.
  (a good german textbook , especially for computer scientists)
• Stanley, J. and Bak, E. (1991). Neuronale Netze: Computersimulation biologischer Intelligenz. Systhema Verlag, München.
  (a popular scientific introduction)