深度学习中的优化算法之Adadelta

      之前在https://blog.csdn.net/fengbingchun/article/details/124766283 介绍过深度学习中的优化算法AdaGrad,这里介绍下深度学习的另一种优化算法Adadelta。论文名字为:《ADADELTA: AN ADAPTIVE LEARNING RATE METHOD》,论文地址:https://arxiv.org/pdf/1212.5701.pdf

      Adadelta一种自适应学习率方法,是AdaGrad的扩展,建立在AdaGrad的基础上,旨在减少其过激的、单调递减的学习率。Adadelta不是积累所有过去的平方梯度,而是将积累的过去梯度的窗口限制为某个固定大小。如下图所示,截图来自:https://arxiv.org/pdf/1609.04747.pdf

     深度学习中的优化算法之Adadelta_第1张图片

       使用Adadelta,我们甚至不需要设置默认学习率这一超参数,因为它已从更新规则中消除,它使用参数本身的变化率来调整学习率。

      Adadelta可被认为是梯度下降的进一步扩展,它建立在AdaGrad和RMSProp的基础上,并改变了自定义步长的计算(changes the calculation of the custom step size),进而不再需要初始学习率超参数

      Adadelta旨在加速优化过程,例如减少达到最优值所需的迭代次数,或提高优化算法的能力,例如获得更好的最终结果。

      最好将Adadelta理解为AdaGrad和RMSProp算法的扩展。Adadelta是RMSProp的进一步扩展,旨在提高算法的收敛性并消除对手动指定初始学习率的需要。

      与RMSProp一样,Adadelta为每个参数计算平方偏导数的衰减移动平均值。关键区别在于使用delta的衰减平均值或参数变化来计算参数的步长。(The decaying moving average of the squared partial derivative is calculated for each parameter, as with RMSProp. The key difference is in the calculation of the step size for a parameter that uses the decaying average of the delta or change in parameter.)

      以上内容主要参考:https://machinelearningmastery.com

      以下是与AdaGrad不同的代码片段:

      1.在原有枚举类Optimizaiton的基础上新增Adadelta:

enum class Optimization {
	BGD, // Batch Gradient Descent
	SGD, // Stochastic Gradient Descent
	MBGD, // Mini-batch Gradient Descent
	SGD_Momentum, // SGD with Momentum
	AdaGrad, // Adaptive Gradient
	RMSProp, // Root Mean Square Propagation
	Adadelta // an adaptive learning rate method
};

      2.calculate_gradient_descent函数:

void LogisticRegression2::calculate_gradient_descent(int start, int end)
{
	switch (optim_) {
		case Optimization::Adadelta: {
			int len = end - start;
			std::vector g(feature_length_, 0.), p(feature_length_, 0.);
			std::vector z(len, 0.), dz(len, 0.);
			for (int i = start, x = 0; i < end; ++i, ++x) {
				z[x] = calculate_z(data_->samples[random_shuffle_[i]]);
				dz[x] = calculate_loss_function_derivative(calculate_activation_function(z[x]), data_->labels[random_shuffle_[i]]);

				for (int j = 0; j < feature_length_; ++j) {
					float dw = data_->samples[random_shuffle_[i]][j] * dz[x];
					g[j] = mu_ * g[j] + (1. - mu_) * (dw * dw); // formula 10

					float alpha = (eps_ + std::sqrt(p[j])) / (eps_ + std::sqrt(g[j]));
					float change = alpha * dw;
					p[j] = mu_ * p[j] +  (1. - mu_) * (change * change); // formula 15

					w_[j] = w_[j] - change;
				}

				b_ -= (eps_ * dz[x]);
			}
		}
			break;
		case Optimization::RMSProp: {
			int len = end - start;
			std::vector g(feature_length_, 0.);
			std::vector z(len, 0), dz(len, 0);
			for (int i = start, x = 0; i < end; ++i, ++x) {
				z[x] = calculate_z(data_->samples[random_shuffle_[i]]);
				dz[x] = calculate_loss_function_derivative(calculate_activation_function(z[x]), data_->labels[random_shuffle_[i]]);

				for (int j = 0; j < feature_length_; ++j) {
					float dw = data_->samples[random_shuffle_[i]][j] * dz[x];
					g[j] = mu_ * g[j] + (1. - mu_) * (dw * dw); // formula 18
					w_[j] = w_[j] - alpha_ * dw / (std::sqrt(g[j]) + eps_);
				}

				b_ -= (alpha_ * dz[x]);
			}
		}
			break;
		case Optimization::AdaGrad: {
			int len = end - start;
			std::vector g(feature_length_, 0.);
			std::vector z(len, 0), dz(len, 0);
			for (int i = start, x = 0; i < end; ++i, ++x) {
				z[x] = calculate_z(data_->samples[random_shuffle_[i]]);
				dz[x] = calculate_loss_function_derivative(calculate_activation_function(z[x]), data_->labels[random_shuffle_[i]]);

				for (int j = 0; j < feature_length_; ++j) {
					float dw = data_->samples[random_shuffle_[i]][j] * dz[x];
					g[j] += dw * dw;
					w_[j] = w_[j] - alpha_ * dw / (std::sqrt(g[j]) + eps_);
				}

				b_ -= (alpha_ * dz[x]);
			}
		}
			break;
		case Optimization::SGD_Momentum: {
			int len = end - start;
			std::vector change(feature_length_, 0.);
			std::vector z(len, 0), dz(len, 0);
			for (int i = start, x = 0; i < end; ++i, ++x) {
				z[x] = calculate_z(data_->samples[random_shuffle_[i]]);
				dz[x] = calculate_loss_function_derivative(calculate_activation_function(z[x]), data_->labels[random_shuffle_[i]]);

				for (int j = 0; j < feature_length_; ++j) {
					float new_change = mu_ * change[j] - alpha_ * (data_->samples[random_shuffle_[i]][j] * dz[x]);
					w_[j] += new_change;
					change[j] = new_change;
				}

				b_ -= (alpha_ * dz[x]);
			}
		}
			break;
		case Optimization::SGD:
		case Optimization::MBGD: {
			int len = end - start;
			std::vector z(len, 0), dz(len, 0);
			for (int i = start, x = 0; i < end; ++i, ++x) {
				z[x] = calculate_z(data_->samples[random_shuffle_[i]]);
				dz[x] = calculate_loss_function_derivative(calculate_activation_function(z[x]), data_->labels[random_shuffle_[i]]);

				for (int j = 0; j < feature_length_; ++j) {
					w_[j] = w_[j] - alpha_ * (data_->samples[random_shuffle_[i]][j] * dz[x]);
				}

				b_ -= (alpha_ * dz[x]);
			}
		}
			break;
		case Optimization::BGD:
		default: // BGD
			std::vector z(m_, 0), dz(m_, 0);
			float db = 0.;
			std::vector dw(feature_length_, 0.);
			for (int i = 0; i < m_; ++i) {
				z[i] = calculate_z(data_->samples[i]);
				o_[i] = calculate_activation_function(z[i]);
				dz[i] = calculate_loss_function_derivative(o_[i], data_->labels[i]);

				for (int j = 0; j < feature_length_; ++j) {
					dw[j] += data_->samples[i][j] * dz[i]; // dw(i)+=x(i)(j)*dz(i)
				}
				db += dz[i]; // db+=dz(i)
			}

			for (int j = 0; j < feature_length_; ++j) {
				dw[j] /= m_;
				w_[j] -= alpha_ * dw[j];
			}

			b_ -= alpha_*(db/m_);
	}
}

      执行结果如下图所示:测试函数为test_logistic_regression2_gradient_descent,多次执行每种配置,最终结果都相同。图像集使用MNIST,其中训练图像总共10000张,0和1各5000张,均来自于训练集;预测图像总共1800张,0和1各900张,均来自于测试集。在AdaGrad中设置学习率为0.01,eps为1e-8及其它配置参数相同的情况下,AdaGrad耗时为17秒;在Adadelta中设置eps为1e-3时,Adadelta耗时为26秒;它们的识别率均为100%。

深度学习中的优化算法之Adadelta_第2张图片

      GitHub: https://github.com/fengbingchun/NN_Test

你可能感兴趣的