With the explosion of artificial intelligence in recent years, the intellectual home speech recognition technique has become increasingly visible, awakening the intelligence of the home, putting “ears”, which is easier and more natural than traditional key control, and has greatly helped people to control and use household functions and to throw out cumbersome buttons...
With the explosion of artificial intelligence in recent years, there has been a growing interest in intellectual home-based speech recognition techniques that awaken the intelligence of the home and place “ears” on it, which are easier and more natural than traditional key-button controls, which have greatly helped people to control and use household functions, to throw out complex buttons and to bring better interactive home-based use experiences to people. Although technology is still inadequate, research by researchers has continued, one day speech recognition will reach thousands of households, and then speech recognition will change everyone ' s life and text input habits.
The research on voice recognition also contributes to our in-depth understanding of speech recognition systems, and speech recognition is a major thrust of subsequent high-level technological developments, and through research on voice recognition, we are also better aware that the current state and trends of state-of-the-art technology are conducive to future learning and development and have good prospects for development.
Requirements analysis
The speech recognition system, designed here, is intended to be applied in a family's smart home. As opposed to the traditional key control electronics, voice recognition controls will bring easier, simpler, easier control experiences. So smart home-based speech recognition will become a technology that people need so much later.
The technical need for voice recognition, which applies to electronics in the home, is the accurate recognition of the voice and the control of its ability to perform the function. We have to deal with the diversity and complexity of voice messages. Because, in the home, each family has a different voice, tone, language, etc., we need to have good stability in our identification system, which can be identified in the face of different voices. On the other hand, the accuracy of identification, because if we say that different voices can be identified as textual information, functional control becomes complex for the home, and if we can determine the fixed identification of several functional commands, one information corresponds to one function, and control becomes easier. For the voice recognition system, we need to satisfy the recording function, which can be identified and controlled only if everyone’s voice is taken into account. Having audio, we also need to pre-process audio, because the recorded sound often contains a lot of groceries and interferences, so pre-processed audio can be used for model training and recognition. The neuronet model is the focus of these functions, and it determines the effects and performance of voice recognition, and ultimately calls on the neural network model to output the results of the analysis and, on that basis, achieve the corresponding functional control.
Feasibility analysis
At the technical level, we need to rely on the Python language for the preparation of functional codes, the use of the Tensorflow framework and the Keras in-depth learning function library to model artificial neuronets, based on the neuronet model, the development of corresponding audio functions, pre-processing functions and corresponding recognition functions, etc. It is technically feasible.
On the equipment level, we just need a computer with Windows systems, and the recording function calls on the computer's own microphone. The rest of the functionality is enough for Python's corresponding functional library to support our development. The equipment needs are low and easy to meet.
Overview of related content of speech recognition technology
Development environment overview
The development environment used for this project is based on the Windows system Anaconda3 and the Visual Studio Code. Using the version of Python 3.7.6 brought by Anaconda3, the functionality development of the system can be completed without additional downloading of Python. The corresponding version of the Keras Function Library is 2.2.4, and the two key frames and function libraries must match the version, otherwise the function will not be possible due to an anomaly, as the corresponding version number can be found on the web-based 1 search.
Overview of artificial neural networks
The artificial neural network is inspired by the human brain, by the structure and function of the neural network imitating the human brain, which is used to model the calculation of approximate functional functions. It is calculated by connecting a large number of computing units to simulate the human brain neurons. Usually, the artificial neural network can modify its own parameters with different data, a self-adapting system. Through statistical analysis and probabilistic predictions, artificial neural networks can have the same simple decision-making and judgement skills as humans.
Introduction to some types and principles of artificial neural networks
Now, the neural network that can be used for voice recognition is also evolving, with very sophisticated modelling systems such as the Gaussian Mixed Model (GMM) and the Invisible Markov Model (HMM) being in place before the rise of in-depth learning, with many deeper neural networks such as the circular neural network (RNN), the curly nervous network (CNN), the long-term memory network (LSTM), the code-decoding framework and so forth, as follows: The method reduces the recognition error rate of traditional acoustic models, so that in-depth learning techniques will become the core technology for speech recognition.
Voice recognition has evolved to this point, and it is a question of serial recognition, because acoustic waves are signals, so we often call them audio signals. Original audio signals are usually audio-messages, such as silent clips, stacks, noise and high-intensity noises, which include interferences that affect the quality of voice signals to some extent, before formal acoustic models are used for voice recognition. Pre-processing typically involves the re-sampling of audio signals, the reduction of audio data volume and the reduction of interference in signals, followed by feature extraction, which is the process of selecting effective audio displays from audio signals, with two commonly used extract features: a linear projection reverse coefficient (LPCC) and a Mel frequency reverse coefficient (MFCC). The LPCC feature is a feature based on the acoustic model and is a characteristic of a channel response. The MFCC feature is a characteristic of human hearing and is used by most people. As the mfcc feature called by the librosa function is used directly in this paper, it is not necessary to construct the mfcc method.
The complete MFCC algorithm includes the following steps:
(1) Fast Fourier Transform (FFT).
(2) Mel frequency conversion.
(3) Configure the triangulation filter group and calculate the output of each trigulation filter after filtering the signal range spectrum.
(4) Perform logarithmic calculations for all filter outputs and then perform discrete cosine transformation (DTC) to obtain MFCCs.
Past developments and research on voice recognition have proven that MFCC is an effective feature extraction method.
The earliest neural network for voice recognition modelling is the most common deep nervous network (DNN). Traditional acoustic models (e.g. GMM) are inefficient in displaying audio signals, but DNN can address this in some ways. However, in practical modelling, DNN needs a fixed-sized input because it is a time-series continuous signal.
In addition to DNN, CNN is widely used for image identification, but it can also be used to build voice recognition models. Of course, CNN is usually used better in conjunction with other models. These include the TDNN, CNN-DNN framework, DFCNN, CNN-LSTM-DNN framework, CNN-DNM framework, and CNN-DNN-LSTM (CDL) framework, many of which are based on CNN have produced many results in acoustic models.
After long research and practice, speech recognition models based on in-depth learning have proved to be better able to handle voice recognition tasks than traditional models based on subsurface models. Voice recognition applications are often more complex. HMM is capable of processing variable-long voice signals, CNN is good at processing variable channel information, and RNN/CNN is capable of processing variable context information. In voice recognition modelling, hybrid models are currently the most dominant modelling method and become the dominant model trend over time by combining the advantages of various models.
Artificial neural network screening
The topic, which is based on voice recognition control for intelligent homes, selects the CNN model. To improve the recognition of correctness and to try to overcome the complex diversity of voice signals, including the diversity of the talking person (the talking person himself and the speaking person), the diversity of the environment, etc. Volta nervous networks provide flat volume over time and in space. They can use volume fixation to overcome the diversity of voice signals themselves. In this way, it can be argued that the time spectrum obtained by analysing the entire voice signal is considered to be imaged and widely used in the image's CNN network. From a practical point of view, CNN is also relatively easy to achieve large-scale parallel operations. Although CNN volume calculations involve many small arrays, they work very slowly. But CNN's accelerated calculations are relatively mature. Researchers have proposed a technology that can convert all these small matrices into a product of a large matrix. Some learning frameworks (e.g. Tensorflow) also offer CNN as a means of accelerating processing through GPU, which allows CNN to use for voice recognition.
Overview of speech recognition
Voice recognition, the goal of which is to allow computers to automatically convert human languages into word information, allows machines to achieve more functionality based on voice only if natural languages are translated into word information.
Basic principles and structure of speech recognition
The voice recognition system is essentially a serial recognition problem, including three basic components, such as sound characterization extraction, model matching, model library with reference to a trained model, the basic structure of which is shown in the chart below:
The microphone converts an unknown sound to a telecommunications signal and adds it to the input of the identification system. First, it preprocesss it, then builds a voice model based on the characteristics of the human voice, training the model by analysing the voice signals entered and extracting the signature vector. During the identification process, the computer compares the characteristics of the voice template stored in the computer with those of the voice signal entered, based on a trained neural network model, and finds the best match for the voice signal based on a certain search and matching strategy. Clearly, the choice of the optimal result and function is determined by the good or bad of the voice model and the accuracy of the template data.
The speech recognition system usually consists of two components: training and recognition. The training is usually carried out offline. The extensive pre-collected voice and language data sets are processed and extracted to obtain the neural network model required for the voice recognition system. The identification process is usually done online. Users enter the voice and recognize it through model analysis. The identification process is usually divided into "front end" and "back-end": the main function of the "front end" module is to perform voice loading, pre-processing, etc.; The function of the "back-end" module is to identify the characterization vectors of a user's voice in order to obtain the text information contained in it, using a neural network model.
Basic process of speech recognition
See Figure 3-2 for the speech recognition workflow diagram.
Speech recognition system module design
Recording module
Description of the main function parameters of recording
Sounddevice.rec()
1. Samplerate indicates the sampling rate of the recording.
2. DURATION indicates the time of recording.
3. Channels indicate the number of audio entries.
4. Blocking indicates that the recording will be completed and returned immediately.
Functions used for recording
Function
Function
Sounddevice.rec()
Recording audio
Sounddevice.wait()
Check if the recording is finished, return if finished, and wait if not
Soundfile.write()
Write data file
Basic recording process
Set the audio parameters for sound recordings, set them up and start with a computer microphone using the function dounddevice.rec() and then save the recording data using the soundfile.write function. The recording process ends as illustrated.
Data preprocessing module
Preprocessing main function parameter description
Wav2mfcc()
1. Wave represents a time series of audio.
2. sr represents the audio sampling rate.
3. mono means processing in mono.
Functions used in preprocessing
Function
Function
librosa.load()
Read audio files
librosa.feature.mfcc()
Extract mfcc features of sound
np.pad()
Fill the feature array
Basic process of pretreatment
The audio tags are extracted from the data path and the labels are coded thermally. In the imported audio, sampling of the audio lowers the sampling rate and extracts the mfcc feature. The processed audio tags and audio features are stored in arrays, and the final split data sets are used for training, part for testing, and then the pre-processing is completed.
Data training module
Description of main function parameters of data training
model.fit()
1. X train indicates the Numpy array of training data.
2. y train hot indicates Numpy arrays of training data labels.
3.Batch size indicates the number of samples per batch of training.
4. epochs represents the number of training times.
5. Verbose indicates the log display mode and enters the progress bar at 01.
6. Validation data indicates that the test data are used to evaluate the model.
Functions used for data training
Function
Function
X_train.reshape()
Change the Nummy array of training data from 1D to 2D
to_categorical()
Hot encoding audio tags
model.fit()
Train the model according to the set parameters
Data training workflow
Parameter models for training models are developed, training data and labels are included, data and labels are tested, and model training is conducted according to parameters.
Neural Network Model Module
Description of the main function parameters of the neural network model
get_model()
1. Sequential indicates that the model is a sequential model, linearly superimposed over multiple network layers and executed sequentially.
2. Conv2D is a 2-dimensional volume layer.
Kernel size indicates the width and height of a 2D volume window.
4. Activation indicates the active function to be used.
5. Input shape indicates the size of the model it has built.
Max Popling2D represents the pool layer, further extracts the characteristic values of the filter and retains the maximum feature values.
7. Dropout indicates the disposal of some neurons to prevent model alignment.
Flatten indicates that the model is “pressed”, i.e., converted from multidimensional to one dimension.
Dense indicates a complete interconnection layer, a non-linear combination of characterization vectors, and a characterization classification based on the training set.
Functions used in neural network models
Function
Function
Sequential()
Sequenced model, model executes each layer of model layer in sequence
model.add(Conv2D())
Add 2D convolution layer to extract information features
model.add(MaxPooling2D())
Add a pool layer to further extract the filter features and retain the maximum feature as
model.add(Dropout())
Add a disconnected neural layer to prevent model alignment
model.add(Flatten())
Adds a layer to make multi-dimensional data into one-dimensional data.
model.add(Dense())
Add a full-link layer to provide a non-linear combination of features, and the last layer will also classify features according to the number of training sets.
Workflow of neural network model
The process of establishing a neural network is as shown in figures 4-5. First, the type of model is selected as the sequential model, which means that each layer of the model is executed sequentially, without jumping or backwards, and then the model is selected so that the model layer can be added, and the functions of each layer vary, as shown by the functions of the above function, and lastly, according to the incentive function, the maximum probability is predicted as the result of the label as the result of the identification.
Identification module
Description of main function parameters of voice recognition
Predict()
1. Filepath indicates the audio file path.
2.Model indicates the model used for the identification.
3.np.argmax indicates the series of labels where the extraction model predicts the maximum probability.
Identify the function used
Function
Function
predict()
Extract the mfcc feature of the audio, insert it in the model and find the maximum probability label array based on the model
load_model()
Import the model used to determine mfcc characteristics
Basic process of identification
The identification process is as shown in Charts 4-6. First defined identification functions, trained neural network models, then processed audio is inserted into the model, identified according to the model, and the results are exported to complete the identification process.
Key issues encountered in speech recognition technology and solutions
Figure 5-4 Sound frequency
Preprocessing part of the problems and solutions
The main issues in the pre-processing section are the coding of audio labels and the extraction of audio features.
I first defined a function that captures audio labels, Get labels, to extract audio labels from a data set that extracts the name of the folder as a label and encodes the labels with the to categorical function, with the following effect:
#Define a vector with 10 types:
b = [0,1,2,3,4,5,6,7,8,9]
# Process class vectorb through the to categorical function:
b = to_categorical(b, 10)
#displayb
print(b)
Then you define a function wav2mfcc that extracts an audio feature, which reads audio through librosa and uses the results of sampling the audio signal at the given sampling rate as the first part of the return value, which is a one-dimensional Numpy array for subsequent analysis. The second part of the return value is the sampling rate. The result of the sample that returns the value in the first part is then taken down to sample, reduce the sample value and facilitate subsequent model calculations. If the second dimension vector of CNN ' s characteristic is greater than mfcc ' s length, the length of the second dimension vector of CNN ' s characteristic is filled in with 0 and, if smaller, the remainder is removed, so that mfcc can be placed in the corresponding CNN model.
Defines a function save data to array, which is used to place audio and label data into arrays, and to deposit mfcc of each label and corresponding audio vector into arrays, and to store them as .npy files for subsequent model training.
The final definition of the Get train test function, which is used to automatically split data from data centralized, uses most as training data and a few as testing data to verify the validity of the model.
The pre-treatment was thus completed, and subsequent model training and audio recognition could be deployed.
If there is no concept of feature parameters, the librosa function can be used to view the mfcc image.
Figure 5-5 mfcc
An array that also shows the mfcc characteristic vector
Use activation function softmax.
Model structure can be viewed by model.summary() after modeling is completed
Down
When the CNN model was created, the correctness of the training data was close to 1, and the correctness of the test data was more than 90 per cent. On this basis, it can be judged that the volume neural network model has learned the sound recognition of these materials, has been successful in predicting correspondingly unknown voice information, and has been tested several times, and has been successful in identifying audio information, although the recognition rate has been limited to the extent that this recognition has been successful, but has largely met the objectives and requirements of the experiment. It can be explained that the machine has successfully learned the content of voice recognition and has been able to control the electronic devices by voice.