Neural networks provide a range of powerful new techniques for solving problems in pattern recognition, data analysis, and control. They have several notable features including high processing speeds and the ability to learn the solution to a problem from a set of examples. The majority of practical applications of neural networks currently make use of two basic network models. We describe these models in detail and explain the various techniques used to train them. Next we discuss a number of key issues which must be addressed when applying neural networks to practical problems, and highlight several potential pitfalls. Finally, we survey the various classes of problem which may be addressed using neural networks, and we illustrate them with a variety of successful applications drawn from a range of fields. It is intended that this review should be accessible to readers with no previous knowledge of neural networks, and yet also provide new insights for those already making practical use of these techniques.

1.
C. M. Bishop, Neural Networks for Statistical Pattern Recognition (Oxford University Press, Oxford, 1994).
2.
H.
White
,
Neural Comput.
1
,
425
(
1989
).
3.
C. M. Bishop, Proc. IEE, Proceedings: Vision, Image and Speech; Special Issue on Neural Networks (1994).
4.
E. R. Kandel and J. H. Schwartz, Principles of Neuroscience, 2nd ed. (Elsevier, New York, 1985).
5.
Scientific American, special issue on Mind and Brain, September (1992).
6.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing: Explorations in the Microstructure of Cognition (MIT Press, Cambridge, 1986), Vol. 2.
7.
W. S.
McCulloch
and
W.
Pitts
,
Bull. Math. Biophys.
5
,
115
(
1943
).
8.
D. O. Hebb, The Organization of Behaviour (Wiley, New York, 1949).
9.
F.
Rosenblatt
,
Psychol. Rev.
65
,
386
(
1958
).
10.
F. Rosenblatt, Principles of Neurodynamics (Spartan Books, Washington, DC, 1962).
11.
B. Widrow, Self-Organizing Systems, edited by G. T. Yovitts (Spartan Books, Washington, DC, 1962).
12.
B. Widrow and M. E. Hoff, Adaptive Switching Ciruits (IRE WESCON Convention Record, New York, 1960), p. 96.
13.
B.
Widrow
and
M.
Lehr
,
Proc. IEEE
78
,
1415
(
1990
).
14.
M. Minsky and S. Papert, Perceptrons (MIT Press, Cambridge, 1959), also available in an expanded edition (1990).
15.
J. J.
Hopfield
,
Proc. Natl. Acad. Sci.
79
,
2554
(
1982
).
16.
J. J.
Hopfield
,
Proc. Natl. Acad. Sci.
81
,
3088
(
1984
).
17.
D. E.
Rumelhart
,
G. E.
Hinton
, and
R. J.
Williams
,
Nature
323
,
533
(
1986
).
18.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing: Explorations in the Microstructure of Cognition (MIT Press, Cambridge, 1986), Vol. 1.
19.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing: Explorations in the Microstructure of Cognition (MIT Press, Cambridge, 1986), Vol. 3.
20.
Neurocomputing: Foundations of Research, edited by J. A. Anderson and E. Rosenfeld (MIT Press, Cambridge, 1988).
21.
Neurocomputing, edited by J. A. Anderson and E. Rosenfeld (MIT Press, Cambridge, 1990), Vol. 2.
22.
C. M.
Bishop
and
C. M.
Roach
,
Rev. Sci. Instrum.
63
,
4450
(
1992
).
23.
C. M.
Bishop
,
C. M.
Roach
, and
M. G.
von Hellerman
,
Plasma Phys. Control. Fusion
35
,
765
(
1993
).
24.
K.
Funahashi
,
Neural Networks
2
,
183
(
1989
).
25.
G.
Cybenko
,
Math.
Control
,
Signals Syst.
2
,
304
(
1989
).
26.
K.
Hornick
,
M.
Stinchcombe
, and
H.
White
,
Neural Networks
2
,
359
(
1989
).
27.
K.
Hornick
,
Neural Networks
4
,
251
(
1991
).
28.
V. Y.
Kreinovich
,
Neural Networks
4
,
381
(
1991
).
29.
A. R.
Gallant
and
H.
White
,
Neural Networks
5
,
129
(
1992
).
30.
Le
Cun Y
et al.,
Neural Computation
1
,
541
(
1989
).
31.
J. F. Kolen and J. B. Pollack, in Advances in Neural Information Processing Systems (Morgan Kaufmann, San Mateo, CA, 1991), Vol. 3, p. 860.
32.
C. M.
Bishop
,
Neural Computation
4
,
494
(
1992
).
33.
H.
Robbins
and
S.
Monro
,
Annu. Math. Stat.
22
,
400
(
1951
).
34.
J.
Kiefer
and
J.
Wolfowitz
,
Annu. Math. Stat.
23
,
462
(
1952
).
35.
W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes in C: The Art of Scientific Computing, 2nd ed. (Cambridge University Press, Cambridge, 1992).
36.
J. E. Dennis and R. B. Schnabel, Numerical Methods for Unconstrained Optimisation and Non-linear Equations (Prentice-Hall, Englewood Cliffs, NJ, 1983).
37.
E. M.
Johansson
,
F. U.
Dowla
, and
D. M.
Goodman
,
Int. J. Neural Syst.
2
,
291
(
1992
).
38.
D. F.
Shanno
,
Math. Operations Res.
3
,
244
(
1978
).
39.
R.
Battiti
,
Complex Syst.
3
,
331
(
1989
).
40.
D. S.
Broomhead
and
D.
Lowe
,
Complex Syst.
2
,
321
(
1988
).
41.
J.
Moody
and
C. L.
Darken
,
Neural Comput.
1
,
281
(
1989
).
42.
M. J. D. Powell, in Algorithms for Approximations, edited by J. C. Mason and M. G. Cox (Clarendon, Oxford, 1987).
43.
C. A.
Micchelli
,
Constructive Approx.
2
,
11
(
1986
).
44.
E.
Hartman
,
J. D.
Keeler
, and
J.
Kowalski
,
Neural Comput.
2
,
210
(
1990
).
45.
J.
Park
and
I. W.
Sandberg
,
Neural Comput.
3
,
246
(
1991
).
46.
S.
Chen
,
S. F. N.
Cowan
, and
P. M.
Grant
,
IEEE Trans. Neural Networks
2
,
302
(
1991
).
47.
J. MacQueen, in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (University of California Press, Berkeley, CA, 1967), Vol. 1, p. 281.
48.
G. J. McLachlan and K. E. Basford, Mixture Models: Inference and Applications to Clustering (Marcel Dekker, New York, 1988).
49.
A. P.
Dempster
,
M. N.
Laird
, and
D. B.
Rubin
,
J. R. Stat. Soc. B
39
,
1
(
1977
).
50.
G. H.
Golub
and
W.
Kahan
,
SLAM Num. Analysis
2
,
205
(
1965
).
51.
M. D.
Richard
and
R. P.
Lippmann
,
Neural Comput.
3
,
461
(
1991
).
52.
D. W.
Ruck
et al.,
IEEE Trans. Neural Networks
1
,
296
(
1990
).
53.
R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis (Wiley, New York, 1973).
54.
K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed. (Academic, San Diego, CA, 1990).
55.
P. A. Devijer and J. Kittler, Pattern Recognition: A Statistical Approach (Prentice-Hall, Hemel Hempstead, UK, 1982).
56.
D. J. Hand, Discrimination and Classification (Wiley, New York, 1981).
57.
D.
Lowe
and
A. R.
Webb
,
Network
1
,
299
(
1990
).
58.
J. Moody, in Advances in Neural Information Processing Systems (Morgan Kaufmann, San Mateo, CA, 1993).
59.
S.
Geman
,
E.
Bienenstock
, and
R.
Doursat
,
Neural Comput.
4
,
1
(
1992
).
60.
A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-posed Problems (Winston, Washington, DC, 1977).
61.
G.
Wahba
,
Ann. Statist.
13
,
1378
(
1985
).
62.
C. M.
Bishop
,
Neural Comput.
3
,
579
(
1991
).
63.
T.
Poggio
and
F.
Girosi
,
Proc. IEEE
78
,
1481
(
1990
).
64.
T.
Poggio
and
F.
Girosi
,
Science
247
,
978
(
1990
).
65.
C. M.
Bishop
,
IEEE Trans. Neural Networks
4
,
882
(
1993
).
66.
M.
Stone
,
Operationforsh. Statis. Ser. Statist.
9
,
127
(
1978
).
67.
C. M.
Bishop
and
G. D.
James
,
Nucl. Instrum. Methods Phys. Res. A
327
,
580
(
1993
).
68.
R. E. Bellman, Adaptive Control Processes (Princeton University Press, Princeton, NJ, 1961).
69.
I. T. Jollife, Principal Component Analysis (Springer, New York, 1986).
70.
C. M. Bishop, P. Cox, P. Haynes, C. M. Roach, M. E. U. Smith, T. N. Todd, and D. L. Trotman, in Neural Network Applications, edited by J. G. Taylor (Springer, London, 1992), p. 114.
71.
C. M. Bishop, P. Cox, P. Haynes, C. M. Roach, M. E. U. Smith, T. N. Todd, and D. L. Trotman, Neural Computation (to be published).
72.
L.
Allen
and
C. M.
Bishop
,
Plasma Phys. Control Fusion
34
,
1291
(
1992
).
73.
C. M.
Bishop
,
I.
Strachan
,
J.
O’Rourke
,
G.
Maddison
, and
P.
Thomas
,
Neural Comput. Appl.
1
,
4
(
1993
).
74.
V. A. Morozov, Methods for Solving Ill-posed Problems (Springer, Berlin, 1984).
75.
J. B.
Lister
and
H.
Schnurrenberger
,
Nucl. Fusion
31
,
1291
(
1991
).
76.
D. G.
Sandler
,
T. K.
Barrett
,
D. A.
Palmer
,
R. Q.
Fugate
, and
W. J.
Wild
,
Nature
351
,
300
(
1991
).
77.
K. J. Åström and B. Wittenmark, Adaptive Control (Addison-Wesley, Redwood City, CA, 1989).
78.
W. T. Miller, R. S. Sutton, and P. J. Werbos, Neural Networks for Control (MIT Press, Cambridge, 1990).
79.
Handbook of Intelligent Control, edited by D. A. White and D. A. Sofge (Van Nostrand Reinhold, New York, 1992).
80.
T. Kohonen, Self-organization and Associative Memory, 3rd ed. (Springer, London, 1989).
81.
R. P.
Lippmann
,
Neural Comput.
1
,
1
(
1989
).
82.
A. Lapedes and R. Farber, in Neural Information Processing Systems, edited by D. Z. Anderson (American Institute of Physics, New York, 1988), p. 442.
83.
D. J. C.
MacKay
,
Neural Comput.
4
,
415
(
1992
).
84.
D. J. C.
MacKay
,
Neural Comput.
4
,
448
(
1992
).
85.
D. J. C.
MacKay
,
Neural Comput.
4
,
720
(
1992
).
86.
W. L.
Buntine
and
A. S.
Weigend
,
Complex Syst.
5
,
603
(
1991
).
87.
J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation (Addison-Wesley, Redwood City, CA, 1991).
88.
R. Hecht-Nielsen, Neurocomputing (Addison-Wesley, Redwood City, CA, 1990).
This content is only available via PDF.
You do not currently have access to this content.