Cascade Networks, Generalized Neural Networks, and Approximation of Functions

Abstract

A neural network is a supervised machine learning model based on how the brain acquires and stores knowledge. Due to the availability of large amounts of training data and improvements in computing power, neural networks are increasingly used in a wide range of machine learning problems. While neural networks have produced an abundance of successes in practical applications, the basis of these successes lacks rigorous mathematical analysis. In its most general form, a neural network is a function given by repeatedly applying a fixed function, in general nonlinear, to an affine operator. In this thesis, close analogs to neural networks using the rectified linear unit (ReLU) activation function are introduced. These analogs, called cascade networks, are also functions given by repeatedly applying a fixed function to an affine operator. Cascade networks have a close connection with algorithms used in computer aided geometric design and multiresolution analysis. In particular, the connection between cascade networks, subdivision algorithms, and the cascade algorithm is discussed. Using cascade networks to approximate polynomials and smooth functions, similar results were obtained when compared to the results for ReLU neural networks.