☆ Save 5. Neural Network Learning Systems and Deep Learning: from Perceptrons to Representation Learning in Deep Networks

10/29/2025

This article surveys neural network learning systems from brain-inspired computation to modern machine learning, deep learning and generative AI. We trace how neural networks have evolved into a general framework for learning·representation·reasoning and generation, rather than just a single algorithm.

Brain-like AI and the concept of neural networks
Early neural networks and self-organizing learning
Machine learning and backpropagation-based learning systems
The rise of deep learning and deep representation learning
Generative AI and the move to multimodal neural networks

Brain-Like AI and the Concept of Neural Networks

Research on neural networks starts from the question: “Can a computer learn and improve from experience like the human brain?” Traditional computers are programmed systems that follow rules and instructions designed by humans. The brain, in contrast, is a learning system: its connections change with experience and it self-organizes its internal structure.

From this viewpoint we talk about brain-like computers, brain-like learning and brain-like intelligence. Intelligence is not a fixed set of rules, but a process that develops through experience.

In this context, Neural Networks are not a single algorithm but a family of models that use neurons, weights and connection patterns to represent and learn knowledge. A neural network is more than a device that computes outputs from inputs: by repeatedly seeing data it discovers patterns and organizes internal representations by itself.

This leads to the idea of brain-like cognitive AI: an agent that perceives the environment·acts·and continuously updates its internal knowledge through interaction. Neural-network-based AI is therefore not just a system with fixed answers, but a learning intelligence whose representations and behaviours grow with experience.

Early Neural Networks and Self-Organizing Learning

From the 1950s to the 1980s, researchers explored how to translate the connectivity and learning mechanisms of the brain into computational models. The goal was to see whether neural networks could store patterns, self-organize their structure and converge to stable states.

Perceptrons
The basic form of a neural network that multiplies inputs by weights, sums them and decides the output. Perceptrons introduced the ideas of linear separability and a simple learning rule for updating weights, becoming the starting point for later work.
Self-organizing Maps
Models that learn structure from data without labels. Similar inputs are mapped to nearby locations, forming a topology that resembles cortical topographic maps in the brain.
Hopfield Networks
Networks that store patterns as energy minima. Even when an input pattern is noisy or partially corrupted, the network converges to the closest stored pattern, acting as an associative memory.
Boltzmann Machines
Networks in which units are activated stochastically, allowing the model to learn probability distributions over patterns. They form a bridge between neural networks and probabilistic graphical models.

These early models encouraged a view of neural networks as self-organizing, memory-capable and pattern-representing cognitive structures, not just function approximators. This perspective laid foundations for later work in machine learning and deep learning.

Machine Learning and Backpropagation-Based Learning Systems

From the 1980s to around 2010, neural networks increasingly became central models in machine learning. The key turning point was the establishment of the Backpropagation algorithm, which made it practical to train networks with several layers.

Multilayer Perceptron (MLP)
Stacks multiple hidden layers between input and output, overcoming the limitations of single-layer perceptrons and enabling more expressive decision boundaries.
Backpropagation networks
Use the error at the output to propagate gradients backward and adjust weights in each layer so as to reduce a loss function. Each layer learns features at a different level of abstraction.
Convolutional Neural Networks (CNN)
Architectures tailored to data with spatial structure, such as images. CNNs learn local patterns and hierarchical features via convolution and pooling, becoming the backbone of modern image recognition.
Deep Belief Networks (DBN)
Layered probabilistic models that stack restricted Boltzmann machines. They highlight unsupervised layer-wise pretraining and introduce the idea of deep representation learning.
Probabilistic Graphical Models and Bayesian Networks
Provide a principled way to encode uncertainty and structure in complex systems. Neural networks and probabilistic graphical models co-evolved, expanding how AI systems represent knowledge.

Through this period, neural networks came to be seen as learning systems that capture data distributions and internal representations, not just tools for computing outputs. This set the stage for the transition to deep learning.

The Rise of Deep Learning and Deep Representation Learning

After 2010, advances in large-scale data and parallel computation allowed neural networks to grow deeper and wider. Deep Learning emerged as a dominant paradigm, where models learn features automatically instead of relying on hand-crafted feature extraction.

This shift connects to the contrast between traditional machine learning and deep learning, to the idea of automatic programming, and to the broader “machine learning revolution” in AI.

AlexNet and GoogLeNet
Delivered dramatic accuracy gains on large-scale image recognition benchmarks, convincing the community that deep convolutional networks work in real-world settings.
DeepFace and DeepSpeech
Pioneered end-to-end learning for visual and speech tasks, mapping raw inputs directly to outputs with a single deep network.
Sequence-to-Sequence learning
Enabled models to map whole input sequences to output sequences, such as in machine translation, and to learn context-aware representations in NLP.

Deep representation learning shows that features can be built layer by layer, with progressively more abstract and structured representations. This phase connects directly to the development of generative models and multimodal AI. Within this trend, Generative Adversarial Networks (GANs) and various residual networks play key roles in scaling deep architectures.

Generative AI and Multimodal Neural Networks

Recent research moves beyond pure prediction toward learning systems that generate new data and representations. Generative models learn the underlying data distribution and can sample new examples that resemble, but are not identical to, the training data. This is the core of modern Generative AI.

Generative Adversarial Networks (GANs)
Use a generator and a discriminator in competition to learn high-quality data representations and synthesize realistic samples.
Multimodal residual networks
Combine different modalities—text·images·audio—within deep residual networks, enabling models to learn joint representations across multiple input types.

In this landscape, neural networks are no longer limited to classification. They are full AI systems that learn representations, organize meaning and generate new content across modalities.

※ This article is an independently organized and restructured summary based on lectures by Professor Byoung-Tak Zhang at Seoul National University.

📚 This article is part of the following hubs

Deep Learning — Neural Networks·CNN·RNN·Transformers·and How They Learn

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 4. Probabilistic Reasoni…|6. Deep Learning and Gen… »

🔖 Tags: Backpropagation · Deep Learning · Neural Networks · Representation Learning