Almaarefa University
The authoritative source for research information and outputs is Elsevier Scopus, with researcher profile verification from the Deanship of Scientific Research.
ArticleOpen access20234 citation

Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning

International Journal of Advanced Computer Science and Applications · vol. 14 (2) · 2023

Abstract

A convolutional neural network (CNN) is a subset of machine learning as well as one of the different types of artificial neural networks that are used for different applications and data types. Activation functions (AFs) are used in this type of network to determine whether or not its neurons are activated. One non-linear AF named as Rectified Linear Units (ReLU) which involves a simple mathematical operations and it gives better performance. It avoids rectifying vanishing gradient problem that inherents older AFs like tanh and sigmoid. Additionally, it has less computational cost. Despite these advantages, it suffers from a problem called Dying problem. Several modifications have been appeared to address this problem, for example; Leaky ReLU (LReLU). The main concept of our algorithm is to improve the current LReLU activation functions in mitigating the dying problem on deep learning by using the readjustment of values (changing and decreasing value) of the loss function or cost function while number of epochs are increased. The model was trained on the MNIST dataset with 20 epochs and achieved lowest misclassification rate by 1.2%. While optimizing our proposed methods, we received comparatively better results in terms of simplicity, low computational cost, and with no hyperparameters.

Details

Journal
International Journal of Advanced Computer Science and Applications
Publisher
Science and Information Organization
Volume
14
Issue
2
Pages
—
ISSN
2156-5570
Year
2023
Language
en
License
cc-by
Open access
diamond
Citations
4
Source
scopus

Topics

MNIST databaseComputer scienceSigmoid functionActivation functionConvolutional neural networkHyperparameter

Neural Networks and Applications (Computer Science) · Advanced Neural Network Applications (Computer Science) · Stochastic Gradient Optimization Techniques (Computer Science)

Cite this

APA
Atoum, I. A. (2023). Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning. International Journal of Advanced Computer Science and Applications, 14(2). https://doi.org/10.14569/ijacsa.2023.0140212
Harvard
Atoum, I. A. 2023, 'Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning', International Journal of Advanced Computer Science and Applications, vol. 14(2). Available at: https://doi.org/10.14569/ijacsa.2023.0140212
Vancouver
Atoum IA. Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning. International Journal of Advanced Computer Science and Applications. 2023;14(2).
BibTeX
@article{atoum20232089,
  title = {Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning},
  author = {Ibrahim A. Atoum},
  year = {2023},
  journal = {International Journal of Advanced Computer Science and Applications},
  volume = {14},
  number = {2},
  publisher = {Science and Information Organization},
  doi = {10.14569/ijacsa.2023.0140212},
  url = {http://dx.doi.org/10.14569/ijacsa.2023.0140212}
}
RIS
TY  - JOUR
AU  - Ibrahim A. Atoum
TI  - Adaptive Rectified Linear Unit (Arelu) for Classification Problems to Solve Dying Problem in Deep Learning
PY  - 2023
JO  - International Journal of Advanced Computer Science and Applications
VL  - 14
IS  - 2
DO  - 10.14569/ijacsa.2023.0140212
UR  - http://dx.doi.org/10.14569/ijacsa.2023.0140212
ER  - 

Related outputs