Neural and Evolutionary Computing
See recent articles
- [1] arXiv:2408.05098 [pdf, html, other]
-
Title: Overcoming the Limitations of Layer Synchronization in Spiking Neural NetworksSubjects: Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI)
Currently, neural-network processing in machine learning applications relies on layer synchronization, whereby neurons in a layer aggregate incoming currents from all neurons in the preceding layer, before evaluating their activation function. This is practiced even in artificial Spiking Neural Networks (SNNs), which are touted as consistent with neurobiology, in spite of processing in the brain being, in fact asynchronous. A truly asynchronous system however would allow all neurons to evaluate concurrently their threshold and emit spikes upon receiving any presynaptic current. Omitting layer synchronization is potentially beneficial, for latency and energy efficiency, but asynchronous execution of models previously trained with layer synchronization may entail a mismatch in network dynamics and performance. We present a study that documents and quantifies this problem in three datasets on our simulation environment that implements network asynchrony, and we show that models trained with layer synchronization either perform sub-optimally in absence of the synchronization, or they will fail to benefit from any energy and latency reduction, when such a mechanism is in place. We then "make ends meet" and address the problem with unlayered backprop, a novel backpropagation-based training method, for learning models suitable for asynchronous processing. We train with it models that use different neuron execution scheduling strategies, and we show that although their neurons are more reactive, these models consistently exhibit lower overall spike density (up to 50%), reach a correct decision faster (up to 2x) without integrating all spikes, and achieve superior accuracy (up to 10% higher). Our findings suggest that asynchronous event-based (neuromorphic) AI computing is indeed more efficient, but we need to seriously rethink how we train our SNN models, to benefit from it.
- [2] arXiv:2408.05156 [pdf, html, other]
-
Title: Neuromorphic Keyword Spotting with Pulse Density Modulation MEMS MicrophonesComments: Accepted at INTERSPEECH 2024Subjects: Neural and Evolutionary Computing (cs.NE)
The Keyword Spotting (KWS) task involves continuous audio stream monitoring to detect predefined words, requiring low energy devices for continuous processing. Neuromorphic devices effectively address this energy challenge. However, the general neuromorphic KWS pipeline, from microphone to Spiking Neural Network (SNN), entails multiple processing stages. Leveraging the popularity of Pulse Density Modulation (PDM) microphones in modern devices and their similarity to spiking neurons, we propose a direct microphone-to-SNN connection. This approach eliminates intermediate stages, notably reducing computational costs. The system achieved an accuracy of 91.54\% on the Google Speech Command (GSC) dataset, surpassing the state-of-the-art for the Spiking Speech Command (SSC) dataset which is a bio-inspired encoded GSC. Furthermore, the observed sparsity in network activity and connectivity indicates potential for remarkably low energy consumption in a neuromorphic device implementation.
New submissions for Monday, 12 August 2024 (showing 2 of 2 entries )
- [3] arXiv:2408.05110 (cross-list from cs.AI) [pdf, other]
-
Title: Application of Unsupervised Artificial Neural Network (ANN) Self_Organizing Map (SOM) in Identifying Main Car Sales FactorsSubjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
Factors which attract customers and persuade them to buy new car are various regarding different consumer tastes. There are some methods to extract pattern form mass data. In this case we firstly asked passenger car marketing experts to rank more important factors which affect customer decision making behavior using fuzzy Delphi technique, then we provided a sample set from questionnaires and tried to apply a useful artificial neural network method called self_organizing map SOM to find out which factors have more effect on Iranian customer's buying decision making. Fuzzy tools were applied to adjust the study to be more real. MATLAB software was used for developing and training network. Results report four factors are more important rather than the others. Results are rather different from marketing expert rankings. Such results would help manufacturers to focus on more important factors and increase company sales level.
Cross submissions for Monday, 12 August 2024 (showing 1 of 1 entries )
- [4] arXiv:2406.16506 (replaced) [pdf, html, other]
-
Title: Natural Gradient Interpretation of Rank-One Update in CMA-ESComments: This paper has been accepted for presentation at PPSN2024. Updated experimental results and related sentences in Sections 6 and 7Subjects: Neural and Evolutionary Computing (cs.NE)
The covariance matrix adaptation evolution strategy (CMA-ES) is a stochastic search algorithm using a multivariate normal distribution for continuous black-box optimization. In addition to strong empirical results, part of the CMA-ES can be described by a stochastic natural gradient method and can be derived from information geometric optimization (IGO) framework. However, there are some components of the CMA-ES, such as the rank-one update, for which the theoretical understanding is limited. While the rank-one update makes the covariance matrix to increase the likelihood of generating a solution in the direction of the evolution path, this idea has been difficult to formulate and interpret as a natural gradient method unlike the rank-$\mu$ update. In this work, we provide a new interpretation of the rank-one update in the CMA-ES from the perspective of the natural gradient with prior distribution. First, we propose maximum a posteriori IGO (MAP-IGO), which is the IGO framework extended to incorporate a prior distribution. Then, we derive the rank-one update from the MAP-IGO by setting the prior distribution based on the idea that the promising mean vector should exist in the direction of the evolution path. Moreover, the newly derived rank-one update is extensible, where an additional term appears in the update for the mean vector. We empirically investigate the properties of the additional term using various benchmark functions.
- [5] arXiv:2103.02238 (replaced) [pdf, html, other]
-
Title: EmoWrite: A Sentiment Analysis-Based Thought to Text Conversion -- A Validation StudyImran Raza (COMSATS University Islamabad, Lahore Campus), Syed Asad Hussain (COMSATS University Islamabad, Lahore Campus), Muhammad Hasan Jamal (COMSATS University Islamabad, Lahore Campus), Isabel de la Torre Diez (University of Valladolid), Carmen Lili Rodriguez Velasco (Universidad Europea del Atlantico, Universidad Internacional Iberoamericana Campeche, Fundacion Universitaria Internacional de Colombia Bogota), Jose Manuel Brenosa (Universidad Europea del Atlantico, Universidad Internacional Iberoamericana Arecibo, Puerto Rico, Universidade Internacional do Cuanza. Cuito, Bie, Angola), Imran Ashraf (Yeungnam University)Subjects: Human-Computer Interaction (cs.HC); Neural and Evolutionary Computing (cs.NE)
Objective- The objective of this study is to introduce EmoWrite, a novel brain-computer interface (BCI) system aimed at addressing the limitations of existing BCI-based systems. Specifically, the objective includes improving typing speed, accuracy, user convenience, emotional state capturing, and sentiment analysis within the context of BCI technology. Method- The method involves the development and implementation of EmoWrite, utilizing a user-centric Recurrent Neural Network (RNN) for thought-to-text conversion. The system incorporates visual feedback and introduces a dynamic keyboard with a contextually adaptive character appearance. Comprehensive evaluation and comparison against existing approaches are conducted, considering various metrics such as accuracy, typing speed, sentiment analysis, emotional state capturing, and user interface latency. The data required for this experiment was obtained from a total of 72 volunteers (40 male and 32 female) aged between 18 and 40 Results- EmoWrite achieves notable results, including a typing speed of 6.6 Words Per Minute (WPM) and 31.9 Characters Per Minute (CPM) with a high accuracy rate of 90.36%. It excels in capturing emotional states, with an Information Transfer Rate (ITR) of 87.55 bits/min for commands and 72.52 bits/min for letters, surpassing other systems. Additionally, it offers an intuitive user interface with low latency of 2.685 seconds. Conclusion- The introduction of EmoWrite represents a significant stride towards enhancing BCI usability and emotional integration. The findings suggest that EmoWrite holds promising potential for revolutionizing communication aids for individuals with motor disabilities.
- [6] arXiv:2308.06422 (replaced) [pdf, html, other]
-
Title: Sensitivity-Aware Mixed-Precision Quantization and Width Optimization of Deep Neural Networks Through Cluster-Based Tree-Structured Parzen EstimationSubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
As the complexity and computational demands of deep learning models rise, the need for effective optimization methods for neural network designs becomes paramount. This work introduces an innovative search mechanism for automatically selecting the best bit-width and layer-width for individual neural network layers. This leads to a marked enhancement in deep neural network efficiency. The search domain is strategically reduced by leveraging Hessian-based pruning, ensuring the removal of non-crucial parameters. Subsequently, we detail the development of surrogate models for favorable and unfavorable outcomes by employing a cluster-based tree-structured Parzen estimator. This strategy allows for a streamlined exploration of architectural possibilities and swift pinpointing of top-performing designs. Through rigorous testing on well-known datasets, our method proves its distinct advantage over existing methods. Compared to leading compression strategies, our approach records an impressive 20% decrease in model size without compromising accuracy. Additionally, our method boasts a 12x reduction in search time relative to the best search-focused strategies currently available. As a result, our proposed method represents a leap forward in neural network design optimization, paving the way for quick model design and implementation in settings with limited resources, thereby propelling the potential of scalable deep learning solutions.
- [7] arXiv:2404.02785 (replaced) [pdf, html, other]
-
Title: Domain Generalization through Meta-Learning: A SurveySubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
Deep neural networks (DNNs) have revolutionized artificial intelligence but often lack performance when faced with out-of-distribution (OOD) data, a common scenario due to the inevitable domain shifts in real-world applications. This limitation stems from the common assumption that training and testing data share the same distribution--an assumption frequently violated in practice. Despite their effectiveness with large amounts of data and computational power, DNNs struggle with distributional shifts and limited labeled data, leading to overfitting and poor generalization across various tasks and domains. Meta-learning presents a promising approach by employing algorithms that acquire transferable knowledge across various tasks for fast adaptation, eliminating the need to learn each task from scratch. This survey paper delves into the realm of meta-learning with a focus on its contribution to domain generalization. We first clarify the concept of meta-learning for domain generalization and introduce a novel taxonomy based on the feature extraction strategy and the classifier learning methodology, offering a granular view of methodologies. Additionally, we present a decision graph to assist readers in navigating the taxonomy based on data availability and domain shifts, enabling them to select and develop a proper model tailored to their specific problem requirements. Through an exhaustive review of existing methods and underlying theories, we map out the fundamentals of the field. Our survey provides practical insights and an informed discussion on promising research directions.