## Reid Prize Honors a Career in Stochastic Control

**October 31, 2003**

Harold Kushner (left), recipient of the 2003 Reid Prize, with SIAM president Mac Hyman.

*On June 17, at the SIAM Annual Meeting in Montreal, Harold Kushner of Brown University received the W.T. and Idalia Reid Prize. He gave the prize lecture, "Modeling, Control, and Numerical Methods for Jump-Diffusions with Controlled Jumps," the following afternoon. For readers of SIAM News, George Yin of Wayne State University offers the following glimpse of contributions for which Kushner was honored.*

Underlying the concise citation of the Reid Prize---honoring Kushner's "fundamental and lasting contributions to stochastic control theory"---is a long list of seminal results covering a substantial part of stochastic control theory: stochastic stability for Markov and non-Markov systems, nonlinear filtering, stochastic distributed and delay systems, stochastic variational methods and the maximum principle, stochastic approximations and recursive algorithms, efficient numerical methods for Markov chain models (the numerical methods of choice for general continuous-time systems), singularly perturbed stochastic systems, stochastic networks, heavy traffic analysis of queueing/communications systems, systems driven by wide-band noise, large-deviation methods for problems with small noise effects, and nearly optimal control and filtering for non-Markovian systems.

In the mid-1960s, Kushner established much of the basic theory of stochastic stability, based on the concept of supermartingales as Lyapunov functions. Extension to non-Markovian and infinite-dimensional systems followed, and he developed a full stochastic analog of the LaSalle invariance theorem, which is essential for application to distributed and delay systems. Such results, which are analogous to the theory for deterministic systems, are essential for the analysis of the long-term behavior of complex stochastic systems arising in control, economics, recursive algorithms, and elsewhere. Because proof of stability is often required before any further analysis can be done, these have been fundamental tools for the analysis of stochastic systems ever since.

It was Kushner who, in the mid-1960s, provided the first rigorous development of nonlinear filters for diffusion-type processes with white observation noise. This is the analog of Kalman filtering for nonlinear systems, and concerns the tracking of systems with nonlinear dynamics or observations. He also developed many practical algorithms for approximating optimal filters, adaptations of the theory for dealing with robustness and for systems that are only "approximately" Markovian or have only wide-bandwidth noise, as well as extensions to distributed systems.

Numerical methods for problems arising in stochastic control have always been a challenge. The optimal value function formally solves the Bellman-Hamilton-Jacobi equation. Even when the derivation is formally justified, the equation can be a highly nonlinear (even in the highest-order terms) elliptic or parabolic partial differential equation, an integral-differential equation, or a variational inequality. The equations tend to be degenerate and to have serious singularities; moreover, the reflections on the boundaries are often not continuous. The solutions, when they are known to exist, might not be differentiable or even continuous. Generally, there is little theory concerning regularity, or even existence of solutions, and the use of classical numerical methods can be problematic. Kushner's Markov chain approximation method is the current approach of choice for such problems. The algorithms are robust; they are intuitively reasonable and have physical meaning because the approximating Markov chains represent systems similar to the one being approximated. The convergence theory is purely probabilistic, using methods of stochastic control, so that the analytical difficulties are avoided.

Kushner is the author of nine books and more than two hundred papers. In his 1984 book he developed a comprehensive approach to approximation and weak convergence methods for random processes, with emphasis on problems that arise in control and communications, e.g., systems driven by wide-band noise, perhaps appearing nonlinearly in the dynamics. The original physical models are often not Markovian; for purposes of analysis or numerical approximation, approximation by a Markovian model is important. The book presents powerful methods for obtaining such approximations, as well as for proving stability of the original physical systems. The methods are widely used in the analysis of stochastic recursive algorithms, for the approximation of stochastic networks under general conditions, and for obtaining "nearly optimal" controls for systems driven by wide-bandwidth noise; recently, they have been applied to the development of scheduling algorithms for mobile communications with rapidly varying connecting channels.

His 1990 book sets out a complete theory of singularly perturbed stochastic control systems and nonlinear filters, with multiple time scales and white or wide-band noise processes. Most recently, his 2001 book presents a thorough development of the theory of heavy traffic analysis of both controlled and uncontrolled queueing and communications systems.

With his work in stochastic approximation and recursive stochastic algorithms, Kushner put in place much of the modern framework. The theory concerns the analysis of the asymptotic properties of the paths of a large class of stochastic difference equations. Such equations model a large number of adaptive processes in control and communications, learning in neural networks, market-adjustment algorithms in economics, and processes in other settings. They are ubiquitous in current applications. Kushner established powerful methods (many of them based on asymptotic approximations of the paths by "mean" ordinary differential equations---the so-called ODE method) for the analysis of convergence and rate of convergence under very weak conditions on the noise and dynamics.

*The next presentation of the W.T. and Idalia Reid Prize will take place at the 2004 SIAM Annual Meeting in Portland, Oregon. The annual prize recognizes outstanding work in differential equations and control theory, whether a single worthy achievement or, as in the case of Harold Kushner, a collection of such achievements. Readers are encouraged to submit nominations for the prize, keeping in mind that eligible candidates can be working in any area of differential equations, analytical or numerical, and in control theory. The award consists of an engraved medal and $10,000.*

*Letters of nomination should be addressed to John A. Burns, chair of the prize committee (c/o Joanna Littleton, SIAM, 3600 University City Science Center, Philadelphia, PA 19104-2688; fax: (215) 386-7999; littleton@siam.org). Nominations must be received in the SIAM office no later than January 30, 2004.*