Downloaded 54 times
Uploaded byAdri Jovin
Neural Networks
The document is a comprehensive overview of neural networks, detailing various types such as McCulloch-Pitts neurons, perceptrons, adaptive linear neurons, and back-propagation networks. It includes discussions on their architectures, learning algorithms, and practical applications, alongside flowcharts and testing algorithms. Additionally, it covers advanced concepts like Hopfield networks, associative memory networks, and unsupervised learning techniques.
More Related Content
More from Adri Jovin
Neural Networks
- 1. Neural Networks Adri JovinJ J, M.Tech., Ph.D. UITE221- SOFT COMPUTING
- 2. McCulloch-Pitts Neuron • Discoveredin 1943 • Usually called M-P Neuron • Connected by directed weighted paths • Binary • Has both excitatory connections and inhibitory connections 𝑓 𝑦𝑚 = 1 𝑖𝑓 𝑦𝑖𝑛 ≥ 𝜃 0 𝑖𝑓 𝑦𝑖𝑛 < 𝜃 UITE221 SOFT COMPUTING 2 Y x1 x2 xn Xn+1 Xn+m -p -p w w w y
- 3. McCulloch-Pitts Neuron • Forinhibition 𝜃 > 𝑛𝑤 − 𝑝 • For firing 𝑘𝑤 ≥ 𝜃 > 𝑘 − 1 𝑤 • No particular training algorithm • Analysis has to be done to determine the values of the weights and the threshold • Used to make a neuron perform a simple logic function UITE221 SOFT COMPUTING 3
- 4. Linear Separability • ANNdoes not give exact solution for a nonlinear problems rather it provides possible approximate solutions • Linear separability: separation of input space into regions based on whether the network response is positive or not • Decision line (decision-making line or decision-support line or linear separable line) separates the positive and negative responses 𝑦𝑖𝑛 = 𝑏 + 𝑖=1 𝑛 𝑥𝑖𝑤𝑖 UITE221 SOFT COMPUTING 4
- 5. Hebb Network Hebb Rule:𝑤𝑖 𝑛𝑒𝑤 = 𝑤𝑖 𝑜𝑙𝑑 + 𝑥𝑖𝑦 The Algorithm: Step 0 : Initialize the weights. Basically in this network they may be set to zero, i.e., 𝑤𝑖 = 0 𝑓𝑜𝑟 𝑖 𝑡𝑜 𝑛 Step 1 : Steps 2 – 4 have to be performed for each input training vector and the target output pair, s:t Step 2 : Input units activations are set. Generally, the activation function of the input layer us identify function: 𝑥𝑖 = 𝑠𝑖 𝑓𝑜𝑟 𝑖 = 1 𝑡𝑜 𝑛 Step 3 : Output units activations are set: 𝑦 = 𝑡 Step 4 : Weight adjustments and bias adjustments are performed: 𝑤𝑖(𝑛𝑒𝑤) = 𝑤𝑖(𝑜𝑙𝑑) + 𝑥𝑖𝑦 𝑏(𝑛𝑒𝑤) = 𝑏(𝑜𝑙𝑑) + 𝑦 UITE221 SOFT COMPUTING 5 Start Initialize Weights For each 𝑠: 𝑡 Activate input units 𝑥𝑖 = 𝑠𝑖 Activate output units 𝑦 = 𝑡 Weight update 𝑤𝑖(𝑛𝑒𝑤) = 𝑤𝑖(𝑜𝑙𝑑) + 𝑥𝑖𝑦 Bias update 𝑏(𝑛𝑒𝑤) = 𝑏(𝑜𝑙𝑑) + 𝑦 Stop No Yes
- 6. Perceptron Networks • Alsoknown as simple perceptron • Single-layer feed-forward network • Consist of 3 units: sensory unit (input unit), associator unit (hidden unit) and response unit (output unit) • Sensory units are connected to associator units with fixed weights having values 1, 0, -1 (assigned randomly) • Binary activation function is used in sensory and associator unit • Response unit has an activation of 1, 0, -1. The binary step with fixed threshold 𝜃 is used as activation for associator. • Output signals from associator unit to the response unit is binary UITE221 SOFT COMPUTING 6
- 7. Perceptron Networks • Theoutput of the perceptron network is given by 𝑦 = 𝑓(𝑦𝑖𝑛) where 𝑓(𝑦𝑖𝑛) is the activation function which is defined as 𝑓 𝑦𝑖𝑛 = 1 𝑖𝑓 𝑦𝑖𝑛 > 𝜃 0 𝑖𝑓 − 𝜃 ≤ 𝑦𝑖𝑛 ≤ 𝜃 −1 𝑖𝑓 𝑦𝑖𝑛 < −𝜃 • The perceptron learning rule is used in the weight update between the associator unit and the response unit • For each training input the network will calculate the response and it will determine whether or not an error has occurred • Error calculation is based on the comparison of the values of targets with those of the calculated outputs • The weights on the connections from the units that send the nonzero signal will get adjusted suitably UITE221 SOFT COMPUTING 7
- 8. Perceptron Networks The weightswill be adjusted based on the learning rule if an error has occurred for a particular training pattern 𝑤𝑖 𝑛𝑒𝑤 = 𝑤𝑖 𝑜𝑙𝑑 + 𝛼𝑡𝑥𝑖 𝑏 𝑛𝑒𝑤 = 𝑏 𝑜𝑙𝑑 + 𝛼𝑡 • If there is no error, no weight update takes place and hence the training process may be stopped • In the above equation the target value 𝑡 is +1 or −1 and 𝛼 is the learning rate • The associator unit is found to consist if a set of sub-circuits called feature predicates • Feature predicates are hardwired to detect the specific feature of a pattern and are equivalent to the feature detectors • The weights present in the input layers are all fixed, while the weights on the response unit are trainable UITE221 SOFT COMPUTING 8
- 9. Original Perceptron Networks xx x x x UITE221 SOFT COMPUTING 9 𝑥1 𝑥2 𝑥𝑛 . . . 𝑌1 𝑌2 𝑌𝑚 . . . . . . . . . 𝑥1 𝑥2 𝑥𝑛 𝑦1 𝑦2 𝑦𝑚 𝑡1 𝑡2 𝑡𝑛 𝜃1 𝜃2 𝜃𝑛 𝑤11 𝑤12 𝑤1𝑚 𝑤21 𝑤22 𝑤2𝑚 𝑤𝑛1 𝑤𝑛2 𝑤𝑛𝑚 Sensory unit sensor grid representing any pattern Fixed weight value of 1, 0, -1 at random Output 0 or 1 Output 0 or 1 Desired output
- 10. Perceptron Learning Rule Theactivation function applied over the network input is as follows: 𝑓 𝑦𝑖𝑛 = 1 𝑖𝑓 𝑦𝑖𝑛 > 𝜃 0 𝑖𝑓 − 𝜃 ≤ 𝑦𝑖𝑛 ≤ 𝜃 −1 𝑖𝑓 𝑦𝑖𝑛 < −𝜃 The update of weight in case of perceptron is as shown below: If 𝑦 ≠ 𝑡, then 𝑤(𝑛𝑒𝑤) = 𝑤(𝑜𝑙𝑑) + 𝛼𝑡𝑥 else, we have 𝑤(𝑛𝑒𝑤) = 𝑤(𝑜𝑙𝑑) UITE221 SOFT COMPUTING 10
- 11. Architecture UITE221 SOFT COMPUTING11 Y 1 x1 xi Xn wn wi w1 b y x1 xi xn x0
- 12. Flowchart UITE221 SOFT COMPUTING12 Start Initialize weights & bias Set 𝛼(0 𝑡𝑜 1) For each 𝑠: 𝑡 Activate input units xi=si Calculate net input yin Apply activation, obtain y=f(yin) if y!=t wi(new) = wi(old) + αtx b(new) = b(old) + αt wi(new) = wi(old) b(new) = b(old) If weight changes Stop Yes Yes Yes No No No
- 13. Perceptron Network TestingAlgorithm Step 0 : The initial weights to be used here are taken from the training algorithms (the final weights obtained during training). Step 1 : For each input vector X to be classified, perform steps 2-3. Step 2 : Set activations of the input unit. Step 3 : Obtain the response of the output unit 𝑦𝑖𝑛 = 𝑖=1 𝑛 𝑥𝑖𝑤𝑖 𝑦 = 𝑓 𝑦𝑖𝑛 = 1 𝑖𝑓 𝑦𝑖𝑛 > 𝜃 0 𝑖𝑓 − 𝜃 ≤ 𝑦𝑖𝑛 ≤ 𝜃 −1 𝑖𝑓𝑦𝑖𝑛 < −𝜃 UITE221 SOFT COMPUTING 13
- 14. Adaptive Linear Neuron(ADALINE) • Network with single linear unit • The input-output relationship is linear • Uses bipolar activation • Trained using delta rule, also known as Least Mean Square (LMS) rule or Widrow-Hoff rule • Learning rule minimize the mean-squared error between the activation and target value • Delta rule for adjusting the weight of the ith pattern (i = 1 to n) ∆𝑤𝑖 = 𝛼 𝑡 − 𝑦𝑖𝑛 𝑥𝑖 UITE221 SOFT COMPUTING 14 Change of weight rate of learning net input to output unit Vector of activation of input unit
- 15. Architecture UITE221 SOFT COMPUTING15 Σ 1 x1 x2 Xn wn w2 w1 b Y x1 x2 xn x0= 1 f(yin) Adaptive Algorithm Output error generator 𝜃 = 𝑡 − 𝑦𝑖𝑛 𝑦𝑖𝑛 = 𝑥𝑖𝑤𝑖 𝑦𝑖𝑛 t
- 16. Flowchart UITE221 SOFT COMPUTING16 Start Set initial values weights and bias, learning state 𝑤, 𝑏, 𝛼 Input the specified tolerance error 𝐸𝑠 For each 𝑠: 𝑡 Activate input layer units 𝑥𝑖 = 𝑠𝑖(𝑖 = 1 𝑡𝑜 𝑛) Calculate net input 𝑦𝑖𝑛 = 𝑏 + 𝑥𝑖𝑤𝑖 Weight update 𝑤𝑖 𝑛𝑒𝑤 = 𝑤𝑖 𝑜𝑙𝑑 + 𝛼(𝑡 − 𝑦𝑖𝑛)𝑥𝑖 𝑏 𝑛𝑒𝑤 = 𝑏 𝑜𝑙𝑑 + 𝛼(𝑡 − 𝑦𝑖𝑛) Calculate error 𝐸𝑖 = (𝑡 − 𝑌𝑖𝑛)2 If 𝐸𝑖 = 𝐸𝑠 Stop Yes Yes No No
- 17. Training Algorithm UITE221 SOFTCOMPUTING 17
- 18. Testing Algorithm UITE221 SOFTCOMPUTING 18
- 19. Multiple Adaptive LinearNeuron • Multiple adaptive linear neurons (MADLINE) consist of many ADALINE in parallel with a single output unit • Output is based on certain selection rules • Majority vote rule – answer is true or false • AND rule – true if and only if both inputs are true • Weights from Adaline to Madaline are fixed whereas weights between input and Adaline layer are adjusted during the training process UITE221 SOFT COMPUTING 19
- 20. Flow chart UITE221 SOFTCOMPUTING 20
- 21.
- 22.
- 23. Back-Propagation Network Most importantdevelopment in neural networks (Bryson and Ho, 1969; Werbos, 1974; Lecun, 1985; Parker, 1985; Rumelhart, 1986) Applied to multilayer feed-forward networks consisting of processing elements with continuous differentiable activation functions Networks associated with back-propagation learning algorithm are called back-propagation networks (BPNs) Provides procedure for change of weights in BPN to classify patterns correctly Memorization and Generalization UITE221 SOFT COMPUTING 23
- 24.
- 25.
- 26. Training Algorithm UITE221 SOFTCOMPUTING 26
- 27. Training Algorithm (Contd…) UITE221SOFT COMPUTING 27
- 28. Training Algorithm (Contd…) UITE221SOFT COMPUTING 28
- 29. Learning Factors • InitialWeights • Learning Rate • Momentum Factor • Generalization • Number of training data • Number of hidden layers UITE221 SOFT COMPUTING 29
- 30. Testing Algorithm UITE221 SOFTCOMPUTING 30
- 31. Radial Basis FunctionNetwork • Radial Basis Function (RBF) is a classification and functional approximation Neural Network developed by M.J.D. Powell • Uses either sigmoidal or Gaussian kernel functions UITE221 SOFT COMPUTING 31
- 32.
- 33. Training Algorithm UITE221 SOFTCOMPUTING 33
- 34. Training Algorithm (Contd…) UITE221SOFT COMPUTING 34
- 35. Time Delay NeuralNetwork • Respond to sequence of patterns • Produce a particular output sequence in response to particular sequence of inputs UITE221 SOFT COMPUTING 35 Time delay neural network (FIR Filter) TDNN with output feedback (IIR Filter)
- 36. Associative Memory Networks •Can store a set of patterns as memories • Content-addressable memories (CAM) • Associates data to address UITE221 SOFT COMPUTING 36
- 37. Training algorithms forpattern association Hebb Rule UITE221 SOFT COMPUTING 37
- 38. Auto-associative Memory Network •Training input and the target output vectors are the same • Determination of weights of the net is called storing of vectors UITE221 SOFT COMPUTING 38
- 39.
- 40. Training Algorithm UITE221 SOFTCOMPUTING 40
- 41. Testing Algorithm UITE221 SOFTCOMPUTING 41
- 42. Heteroassociative Memory Network •The training input and the target output are different • The weights are determined such that the net can store a set of pattern associations • Determination of weight is done either by Hebb rule or delta rule UITE221 SOFT COMPUTING 42
- 43. Testing Algorithm UITE221 SOFTCOMPUTING 43
- 44. Bidirectional Associative Memory(BAM) • Developed by Kosko in 1988 • Performs forward and backward associative searches for stored stimulus responses • Recurrent pattern-matching network that encodes binary or bipolar patterns using Hebbian rule • Two types: discrete and continuous UITE221 SOFT COMPUTING 44
- 45. Testing algorithm (DiscreteBAM) UITE221 SOFT COMPUTING 45
- 46. Testing algorithm (DiscreteBAM) UITE221 SOFT COMPUTING 46
- 47. Hopfield Networks Developed byJohn J. Hopfield in 1982 Conforms to the asynchronous nature of biological neurons Promoted the design of the first analog VLSI neural chip Two types: discrete and continuous UITE221 SOFT COMPUTING 47
- 48. Architecture of DiscreteHopfield Net UITE221 SOFT COMPUTING 48
- 49. Training algorithm UITE221 SOFTCOMPUTING 49
- 50. Testing Algorithm UITE221 SOFTCOMPUTING 50
- 51. Continuous Hopfield Network UITE221SOFT COMPUTING 51
- 52. Iterative Auto-associative MemoryNetworks • Also known as recurrent auto-associative networks • Developed by James Anderson in 1977 • Brain-in-the-box model, an extension described in 1972 • An activity pattern inside the box receives positive feedback on certain components which has the effect of forcing it outward UITE221 SOFT COMPUTING 52
- 53. Training Algorithm UITE221 SOFTCOMPUTING 53
- 54. Testing Algorithm UITE221 SOFTCOMPUTING 54
- 55. Unsupervised learning networks Kohonenself-organizing feature maps Learning Vector Quantization Counter Propagation networks Adaptive Resonance Theory network
- 56. Kohonen Self-organizing FeatureMaps Feature Mapping – converts patterns of arbitrary dimensionality into a response of one- or two- dimensional neuron (conversion of wide pattern space into a feature space) UITE221 SOFT COMPUTING 56
- 57.
- 58.
- 59. Training Algorithm UITE221 SOFTCOMPUTING 59
- 60. Training Algorithm (Contd…) UITE221SOFT COMPUTING 60
- 61. Kohonen Self-organizing MotorMap UITE221 SOFT COMPUTING 61
- 62. Learning Vector Quantization Processof classifying the patterns where each output unit represents a particular class For each class several units should be used The output weight vector is called the reference vector or code book vector for the class which the unit vector represents Special case of competitive net Minimizes misclassification Used in Optical Character Recognition, speech processing etc. UITE221 SOFT COMPUTING 62
- 63.
- 64.
- 65. Training Algorithm UITE221 SOFTCOMPUTING 65
- 66. Variants • LVQ 2 •LVQ 2.1 • LVQ 3 UITE221 SOFT COMPUTING 66
- 67. Counter Propagation Networks Proposedby Hecht Nielsen in 1987 Based on input, output and clustering layers Used in data compression, function approximation and pattern association Constructed from instar-outstar model UITE221 SOFT COMPUTING 67
- 68. Full Counter PropagationNet UITE221 SOFT COMPUTING 68
- 69. Phase-I of FullCPN UITE221 SOFT COMPUTING 69
- 70. Phase-II of FullCPN UITE221 SOFT COMPUTING 70
- 71.
- 72. Training Algorithm UITE221 SOFTCOMPUTING 72
- 73. Training Algorithm (Contd…) UITE221SOFT COMPUTING 73
- 74. Testing Algorithm UITE221 SOFTCOMPUTING 74
- 75. Forward-only CP Net UITE221SOFT COMPUTING 75
- 76.
- 77. Training Algorithm UITE221 SOFTCOMPUTING 77
- 78. Training Algorithm (Contd…) UITE221SOFT COMPUTING 78
- 79. Testing Algorithm UITE221 SOFTCOMPUTING 79
- 80. Adaptive Resonance TheoryNetwork Developed by Steven Grossberg and Gail Carpenter in 1987 Based on competition Finds categories anonymously and learns new categories if needed Two types: ART 1: designed for clustering binary vectors ART 2: designed to accept continuous-valued vectors UITE221 SOFT COMPUTING 80
- 81. Fundamental Algorithm UITE221 SOFTCOMPUTING 81
- 82. ART 1 Made upof two units: 1. Computational Units 2. Supplemental Units UITE221 SOFT COMPUTING 82 Supplemental part
- 83.
- 84. Training Algorithm UITE221 SOFTCOMPUTING 84
- 85. Training Algorithm UITE221 SOFTCOMPUTING 85
- 86. ART 2 UITE221 SOFTCOMPUTING 86 Supplemental part
- 87.
- 88. Training Algorithm UITE221 SOFTCOMPUTING 88
- 89. Training Algorithm (Contd…) UITE221SOFT COMPUTING 89
- 90. Training Algorithm (Contd…) UITE221SOFT COMPUTING 90
- 91. References Rajasekaran, S., &Pai, G. V. (2017). Neural Networks, Fuzzy Systems and Evolutionary Algorithms: Synthesis and Applications. PHI Learning Pvt. Ltd.. Haykin, S. (2010). Neural Networks and Learning Machines, 3/E. Pearson Education India. Sivanandam, S. N., & Deepa, S. N. (2007). Principles of soft computing. John Wiley & Sons. UITE221 SOFT COMPUTING 91