Gradient descent methods with step-size defined by multiple parameters.

What is it about?

Two transformations of gradient descent iterative methods for solving unconstrained optimization are proposed. The first transformation is called modification and it is defined using a small enlargement of the step size in various gradient descent methods. The second transformation is termed as hybridization and it is defined as a composition of gradient descent methods with the Picard-Mann hybrid iterative process. As a result, several accelerated gradient descent methods for solving unconstrained optimization problems are presented, investigated theoretically and numerically compared. The proposed methods are globally convergent for uniformly convex functions satisfying certain condition under the assumption that the step size is determined by the backtracking line search. In addition, the convergence on strictly convex quadratic functions is discussed. Numerical comparisons show better behaviour of the proposed methods with respect to some existing methods in view of the Dolan and Mor\'{e}'s performance profile with respect to all analyzed characteristics: number of iterations, the CPU time, and the number of function evaluations.

Featured Image

Why is it important?

Main highlights of the present paper can be emphasized as follows. (1) Transformation of gradient descent methods, called {\em modification}, is proposed and investigated theoretically and numerically. The modification is defined by replacing the basic step size $t_k$ in $GD$ and $AGD$ methods as well as in the $IGD$ iterative class by the step size $t_{k}+t_{k}^2-t_{k}^3$. The resulting iterations will be termed as $MGD$, $MAGD$ and $MIGD$, respectively. (2) A {\em hybridization} of gradient descent methods is defined as a composition of all considered $GD$ methods and modified $GD$ methods with the Picard-Mann hybrid iterative process. (3) Convergence properties of defined methods on the class of uniformly convex as well as on strictly convex quadratic functions are investigated. (4) Numerical experiments analyze three main characteristics of iterative methods: number of iterations, the CPU time, and the number of function evaluations.

Perspectives