| Artificial neural network is an inspirational technology | | | | power, becoming a simple matrix multiplication |
| of biological neural network. In general a biological | | | | operation from linear algebra. |
| neural network is composed of a group or groups of | | | | Alternative calculation models in neural networks |
| chemically connected or functionally associated | | | | include models with loops, where some kind of time |
| neurons. A single neuron may be connected to many | | | | delay process must be used, and "winner takes all" |
| other neurons and the total number of neurons and | | | | models, where the neuron with the highest value |
| connections in a network may be extensive. | | | | from the calculation fires and takes a value 1, and all |
| Connections, called synapses, are usually formed | | | | other neurons take the value 0. |
| from axons to dendrites, though other kind of | | | | Typically the weights in a neural network are initially |
| connections, also possible. | | | | set to small random values. This represents the |
| An Artificial Neural Network is also called as simulated | | | | network knowing nothing; its output is essentially a |
| neural network of many very simple processors | | | | random function of its input. As the training process |
| ("units"), each possibly having a (small amount of) | | | | proceeds, the connection weights are gradually |
| local memory. The units are connected by | | | | modified according to computational rules specific to |
| unidirectional communication channels ("connections"), | | | | the learning algorithm being used. Ideally the weights |
| which carry numeric (as opposed to symbolic) data. | | | | eventually converge to values allowing them to |
| The units operate only on their local data and on the | | | | perform a useful computation. Thus it can be said |
| inputs they receive via the connections. Artificial | | | | that the neural network commences knowing nothing |
| neural networks are made up of interconnecting | | | | and moves on to gain some real knowledge, though |
| artificial neurons (programming constructs that mimic | | | | the knowledge is sub-symbolic. |
| the properties of biological neurons). Artificial neural | | | | ARCHITECTURE OF NEURAL NETWORKS |
| networks may either be used to gain an | | | | FEED-FORWARD NETWORKS:- |
| understanding of biological neural networks, or for | | | | Feed-forward ANNs allow signals to travel one way |
| solving artificial intelligence problems without | | | | only; from input to output. There is no feedback |
| necessarily creating a model of a real biological | | | | (loops) i.e. the output of any layer does not affect |
| system. The real, biological nervous system is highly | | | | that same layer. Feed-forward ANNs tend to be |
| complex and includes some features that may seem | | | | straight forward networks that associate inputs with |
| superfluous based on an understanding of artificial | | | | outputs. They are extensively used in pattern |
| networks. | | | | recognition. This type of organization is also referred |
| HISTORY OF NEURAL NETWORKS:- | | | | to as bottom-up or top-down. ex., Fault Diagnosis |
| The concept neural networks started in the | | | | systems. |
| late-1800s as an effort to describe how the human | | | | FEEDBACK NETWORKS:- |
| mind performed. These ideas started being applied to | | | | Feedback networks can have signals traveling in both |
| computational models with Turing's B-type machines | | | | directions by introducing loops in the network. |
| and the perceptron. In 1940 Donald Hebb made | | | | Feedback networks are very powerful and can get |
| one of the first hypotheses for a mechanism of | | | | extremely complicated. Feedback networks are |
| neural plasticity (i.e. learning), Hebbian learning. | | | | dynamic; their 'state' is changing continuously until |
| Hebbian learning is considered to be a | | | | they reach an equilibrium point. They remain at the |
| 'typical' unsupervised learning rule. In 1943 McCulloch | | | | equilibrium point until the input changes and a new |
| and Pitts proposed the McCulloch-Pitts neuron model. | | | | equilibrium needs to be found. Feedback architectures |
| In 1948 Alan Tuning, first proposed the concept of | | | | are also referred to as interactive or recurrent, |
| Neural Networks in his paper "Intelligent Machinery". | | | | although the latter term is often used to denote |
| In 1949 Hebb published his book The Organization of | | | | feedback connections in single-layer organizations. Ex., |
| Behavior, in which the Hebbian learning rule was | | | | Radiosity for Virtual Reality Systems (ROVER) |
| proposed. In 1958 Rosenblatt introduced the simple | | | | TRAINING OF ARTIFCIAL NEURAL NETWORKS:- |
| single layer networks now called Perceptrons. In 1969 | | | | A neural network has to be configured such that the |
| Minsky and Papert’s book Perceptrons | | | | application of a set of inputs produces (either 'direct' |
| demonstrated the limitation of single. In layer | | | | or via a relaxation process) the desired set of |
| perceptrons, and almost the whole field went into | | | | outputs. Various methods to set the strengths of the |
| hibernation. In 1982 Hopfield published a series of | | | | connections exist. One way is to set the weights |
| papers on Hopfield networks. In 1982 Kohonen | | | | explicitly, using a priori knowledge. Another way is to |
| developed the Self-Organizing Maps that now bear his | | | | 'train' the neural network by feeding it teaching |
| name. In 1986 The Back-Propagation learning | | | | patterns and letting it change its weights according to |
| algorithm for Multi-Layer Perceptrons was | | | | some learning rule. |
| rediscovered and the whole field took off again. In | | | | We can categorize the learning situations in three |
| 1990s The sub-field of Radial Basis Function Networks | | | | distinct sorts. They are: |
| was developed. In 2000s The power of Ensembles | | | | - Supervised learning or Associative learning in which |
| of Neural Networks and Support Vector Machines | | | | the network is trained by providing it with input and |
| becomes apparent. | | | | matching output patterns. These input-output pairs |
| NEURAL NETWORKS:- | | | | can be provided by an external teacher, or by the |
| Neural networks take a different approach to | | | | system which contains the neural network |
| problem solving than that of conventional computers. | | | | (self-supervised). |
| Conventional computers use an algorithmic approach | | | | |
| i.e. the computer follows a set of instructions in order | | | | This method has two sub varieties called |
| to solve a problem. Unless the specific steps that the | | | | auto-associative and hetero-associative. In |
| computer needs to follow are known the computer | | | | auto-associative learning, the target values are the |
| cannot solve the problem. That restricts the problem | | | | same as the inputs, whereas in hetero-associative |
| solving capability of conventional computers to | | | | learning, the targets are generally different from the |
| problems that we already understand and know how | | | | inputs. |
| to solve. But computers would be so much more | | | | One of the most commonly used supervised NN |
| useful if they could do things that we don't exactly | | | | model is back propagation network that uses back |
| know how to do. | | | | propagation learning algorithm. Back propagation (or |
| Neural networks process information in a similar way | | | | back prop) algorithm is one of the well-known |
| the human brain does. The network is composed of | | | | algorithms in neural networks. |
| a large number of highly interconnected processing | | | | - Unsupervised learning method is not given any |
| elements(neurons) working in parallel to solve a | | | | target value. Unsupervised learning or |
| specific problem. Neural networks learn by example. | | | | Self-organization in which an (output) unit is trained to |
| They cannot be programmed to perform a specific | | | | respond to clusters of pattern within the input. In this |
| task. The examples must be selected carefully | | | | paradigm the system is supposed to discover |
| otherwise useful time is wasted or even worse the | | | | statistically salient features of the input population. |
| network might be functioning incorrectly. The | | | | Unlike the supervised learning paradigm, there is no a |
| disadvantage is that because the network finds out | | | | priori set of categories into which the patterns are to |
| how to solve the problem by itself, its operation can | | | | be classified; rather the system must develop its |
| be unpredictable. | | | | own representation of the input stimuli. A desired |
| On the other hand, conventional computers use a | | | | output of the network is unknown. During training the |
| cognitive approach to problem solving; the way the | | | | network performs some kind of data compression |
| problem is to solved must be known and stated in | | | | such as dimensionality reduction or clustering. The |
| small unambiguous instructions. These instructions are | | | | network learns the distribution of patterns and |
| then converted to a high level language program and | | | | makes a classification of that pattern where, similar |
| then into machine code that the computer can | | | | patterns are assigned to the same output cluster. |
| understand. These machines are totally predictable; if | | | | Kohonen network is the best example of |
| anything goes wrong is due to a software or | | | | unsupervised learning network. According to Sarle |
| hardware fault. | | | | (1997) Kohonen network refers to three types of |
| Neural networks and conventional algorithmic | | | | networks that are Vector Quantization, |
| computers are not in competition but complement | | | | Self-Organizing Map and Learning Vector |
| each other. There are tasks are more suited to an | | | | Quantization. |
| algorithmic approach like arithmetic operations and | | | | - The third major category of learning that will be |
| tasks that are more suited to neural networks. Even | | | | discussed is termed "reinforcement learning". As |
| more, a large number of tasks, require systems that | | | | mentioned earlier, reinforcement learning lies between |
| use a combination of the two approaches (normally a | | | | supervised and unsupervised learning. It is often called |
| conventional computer is used to supervise the neural | | | | learning with a critic rather than learning with a |
| network) in order to perform at maximum efficiency. | | | | teacher as the feedback is evaluative (right or |
| Neural networks, with their remarkable ability to | | | | wrong) rather than instructive (desired output). |
| derive meaning from complicated or imprecise data, | | | | Reinforcement learning in neural networks has its |
| can be used to extract patterns and detect trends | | | | roots in behavioral psychology. Here the learning |
| that are too complex to be noticed by either humans | | | | machine does some action on the environment and |
| or other computer techniques. A trained neural | | | | gets a feedback response from the environment. |
| network can be thought of as an "expert" in the | | | | The learning system grades its action good |
| category of information it has been given to analyze. | | | | (rewarding) or bad (punishable) based on the |
| This expert can then be used to provide projections | | | | environmental response and accordingly adjusts its |
| given new situations of interest and answer "what if" | | | | parameters. Generally, parameter adjustment is |
| questions. | | | | continued until an equilibrium state occurs, following |
| Other advantages include: | | | | which there will be no more changes in its |
| 1. Adaptive learning: An ability to learn how to do | | | | parameters. The self organizing neural learning may |
| tasks based on the data given for training or initial | | | | be categorized under this type of learning. |
| experience. | | | | APPLICATIONS OF NEURAL NETWORK:- |
| 2. Self-Organization: An ANN can create its own | | | | - Signal processing: suppress line noise, with adaptive |
| organization or representation of the information it | | | | echo canceling, blind source separation |
| receives during learning time. | | | | - Control: e.g. backing up a truck: cab position, rear |
| 3. Real Time Operation: ANN computations may be | | | | position, and match with the dock get converted to |
| carried out in parallel, and special hardware devices | | | | steering instructions. Manufacturing plants for |
| are being designed and manufactured which take | | | | controlling automated machines. |
| advantage of this capability. | | | | - Siemens successfully uses neural networks for |
| 4. Fault Tolerance via Redundant Information Coding: | | | | process automation in basic industries, e.g., in rolling |
| Partial destruction of a network leads to the | | | | mill control more than 100 neural networks do their |
| corresponding degradation of performance. However, | | | | job, 24 hours a day |
| some network capabilities may be retained even with | | | | - Robotics - navigation, vision recognition |
| major network damage. | | | | - Pattern recognition, i.e. recognizing handwritten |
| A SIMPLE NEURON:- | | | | characters, e.g. the current version of Apple's |
| An artificial neuron is a device with many inputs and | | | | Newton uses a neural net |
| one output. The neuron has two modes of operation; | | | | - Medicine, i.e. storing medical records based on case |
| the training mode and the using mode. In the training | | | | information |
| mode, the neuron can be trained to fire (or not), for | | | | - Speech production: reading text aloud (NETtalk) |
| particular input patterns. In the using mode, when a | | | | - Speech recognition |
| taught input pattern is detected at the input, its | | | | - Vision: face recognition , edge detection, visual |
| associated output becomes the current output. If the | | | | search engines |
| input pattern does not belong in the taught list of | | | | - Business, eg.. rules for mortgage decisions are |
| input patterns, the firing rule is used to determine | | | | extracted from past decisions made by experienced |
| whether to fire or not. | | | | evaluators, resulting in a network that has a high level |
| STRUCTURE:- | | | | of agreement with human experts. |
| Artificial neural networks are quite different from the | | | | - Financial Applications: time series analysis, stock |
| brain in terms of structure. Like the brain, however, a | | | | market prediction |
| neural net is a massively parallel collection of small | | | | - Data Compression: speech signal, image, e.g. faces |
| and simple processing units where the | | | | - Game Playing: backgammon, chess, ROBOCUP. |
| interconnections form a large part of the network's | | | | Robocup: Robot World Cup:- |
| intelligence; however, in terms of scale, a brain is | | | | The RoboCup Competition pits robots (real and |
| massively larger than a neural network, and the units | | | | virtual) against each other in a simulated soccer |
| used in a neural network are typically far simpler than | | | | tournament. The aim of the RoboCup competition is |
| neurons. Nevertheless, certain functions that seem | | | | to foster an interdisciplinary approach to robotics and |
| exclusive to the brain such as learning, have been | | | | agent-based AI by presenting a domain that requires |
| replicated on a simpler scale, with neural networks. | | | | large-scale cooperation and coordination in a dynamic, |
| MODELS:- | | | | noisy, complex environment. RoboCup has three |
| A typical feed-forward neural network is a set of | | | | different leagues to-date. The Small and Middle-Size |
| nodes. In computing, feed-forward normally refers | | | | Leagues involved physical robots; the Simulation |
| to a multi-layer perceptron network in which the | | | | League is for virtual, synthetic teams. This work |
| outputs from all neurons go to following but not | | | | focus on building softbots for the Simulation League. |
| preceding layers, so there are no feedback loops. | | | | Machine Learning for Robocup involves: |
| Some of these are designated input nodes, | | | | 1.The training of player in the process of making the |
| some output nodes, and in-between are hidden | | | | decision of whether (a) to dribble the ball; (b) to pass |
| nodes. Each connection between neurons has a | | | | it on to another team-mate; (c) to shoot into the |
| numerical weight. When the network is in operation, | | | | net. |
| a value will be applied to each input node -- the | | | | 2. The training of the goalkeeper in process of |
| values being fed in by a human operator, from | | | | intelligent guessing of how the ball is going to be kick |
| environmental sensors, or from some external | | | | by the opponents. Complexities arise when one |
| program. Each node then passes its given value to | | | | opponent decides to pass the ball to another player |
| the connections leading out from it, and on each | | | | instead of attempting a score. |
| connection the value is multiplied by the weight | | | | 3. Evolution of a co-operative and perhaps |
| associated with that connection. Each node in the | | | | unpredictable team. |
| next layer then receives a value which is the sum of | | | | CONCLUSION:- |
| the values produced by the connections leading into | | | | The computing world has a lot to gain from neural |
| it, and in each node a simple computation is | | | | networks. Their ability to learn by example makes |
| performed on the value -- a sigmoid function is | | | | them very flexible and powerful. Furthermore there is |
| typical. This process is then repeated, with the results | | | | no need to device an algorithm in order to perform a |
| being passed through subsequent layers of nodes | | | | specific task; i.e. there is no need to understand the |
| until the output nodes are reached. Early models | | | | internal mechanisms of that task. They are also very |
| (circa 1970) had a fixed number of layers. More | | | | well suited for real time systems because of their |
| recently, genetic algorithms are used to evolve the | | | | fast response and computational times which are due |
| neural structure | | | | to their parallel architecture. |
| CALCULATIONS:- | | | | Neural networks also contribute to other areas of |
| The sigmoid curve is often used as a transfer | | | | research such as neurology and psychology. They are |
| function because it introduces non-linearity into the | | | | regularly used to model parts of living organisms and |
| network's calculations by "squashing" the neuron's | | | | to investigate the internal mechanisms of the brain. |
| activation level into the range [0,1]. The sigmoid | | | | Perhaps the most exciting aspect of neural networks |
| function has the additional benefit of having an | | | | is the possibility that some day 'conscious' networks |
| extremely simple derivative function, as required for | | | | might be produced. There is a number of scientists |
| back-propagating errors through a feed-forward | | | | arguing that consciousness is a 'mechanical' property |
| neural network. Other functions with similar features | | | | and that 'conscious' neural networks are a realistic |
| can be used, most commonly tanh which squashes | | | | possibility. |
| activations into the range of [-1,1] instead, or | | | | Finally, to state that even though neural networks |
| occasionally a piece-wise linear function that simply | | | | have a huge potential we will only get the best of |
| clips the activation rather than squashing it. | | | | them when they are integrated with computing, AI, |
| If no non-linearity is introduced by squashing or | | | | fuzzy logic and related subjects. |
| clipping, the network loses much of its computational | | | | |