Entropy Approach in Methods of Electroencephalogram Automatic Analysis

Shannon entropy

Information theory gives us the following definition of entropy, as a measure of system uncertainty proposed by Shannon (Shannon & Weaver, 1948):

$S\sim -{\sum }_i{P}_i\mathit{ln}{P}_i$, (1)

where {Pi} – probabilities for the system to be in states {i}.

Kolmogorov entropy

In the theory of dynamical systems, Kolmogorov generalized the concept of entropy to ergodic random processes through the limiting probability distribution having density f(x) (Chumak, 2011). Ergodic theory makes an attempt to explain the macroscopic characteristics of physical systems in terms of the behavior of the microscopic structure of the system (Martin & England, 1984). The Kolmogorov entropy is the most important characteristic of a chaotic system in a phase space of arbitrary dimension, showing the degree of chaoticity of a dynamic system. It can be defined as follows (Chumak, 2011; Schuster, 1988): let there be some stationary random process X(t) = [x₁(t), x₂(t), … x_d(t)] on a strange attractor in a d-dimensional phase space. Consider some implementation of a random process, which is a temporal sequence of system states. We also define some finite sample (trajectory) i of the dynamical system from the given implementation. Let us divide the d-dimensional phase space of the system into cells of l_d size. We will determine the state of the system at equal time intervals τ. Let us denote the $P_{i_{\mathrm{0}}}\dots P_{i_n}$joint probability so that X(t=0) is in i₀ cell, X(t=τ) is in i₁ cell , … , X(t=nτ) is in i_n cell. Then, according to Shannon, the quantity

$K_n=-{\sum }_{i_{\mathrm{0}}\dots i_n}{P}_{i_{\mathrm{0}}\dots i_n}\mathit{ln}{P}_{i_{\mathrm{0}}\dots i_n}$, (2)

which is a measure of the a priori uncertainty of the position of the system is proportional to the information required to obtain the position of the system on a given $i_{\mathrm{0}}^{\mathrm{*}},\dots i_n^{\mathrm{*}}$trajectory with an accuracy of l, if only $P_{i_0}\dots P_{i_n}$ is known a priori. Therefore, K_n+1-K_n is the amount of loss of information about the system in the time interval from n to n+1. Thus, the Kolmogorov entropy is the average rate of information loss over time:

$K=\underset{\mathrm{\tau }\to \mathrm{0}}{\lim }\underset{\mathrm{l}\to \mathrm{0}}{\mathrm{l}\mathrm{i}\mathrm{m}}\underset{\mathrm{N}\to \mathrm{\infty }}{\mathrm{}\mathrm{l}\mathrm{i}\mathrm{m}}\frac{1}{N\mathrm{\tau }}\underset{n=0}{\overset{N-1}{\sum }}(K_{n+1}-K_n)=-\underset{\mathrm{\tau }\to \mathrm{0}}{\lim }\underset{\mathrm{l}\to \mathrm{0}}{\mathrm{l}\mathrm{i}\mathrm{m}}\underset{\mathrm{N}\to \mathrm{\infty }}{\mathrm{}\mathrm{l}\mathrm{i}\mathrm{m}}\frac{1}{N\mathrm{\tau }}{\sum }_{i_{\mathrm{0}}\dots i_N}{P}_{i_{\mathrm{0}}\dots i_N}\mathit{ln}{P}_{i_{\mathrm{0}}\dots i_N}$. (3)

For regular motion K=0, for random systems K=∞, for systems with deterministic chaos K is positive and constant (Schuster, 1988).

Algorithms for calculating sample and approximate entropy for physiological data

Equation (3) implies limiting values for temporal and spatial partitions and an infinite length of the time series under study. This limits the use of the Kolmogorov entropy by analytical systems and makes it difficult to use it for short and noisy time series of real measurements. To calculate the entropy of such time series, special robust methods were developed, based, for example, on the concept of approximate entropy and elementary sample entropy (sample entropy). These methods can significantly reduce the dependence of the calculation result on the sample length (Pincus, 1991; Richman & Moorman, 2000).

Approximate entropy and sample entropy are two statistics that make it possible to evaluate the randomness of a time series without having any prior knowledge of the data source (Delgado-Bonal & Marshak, 2019). Sample entropy (sampEn) is a method for estimating the entropy of a system that can be applied to short and noisy time series. The sample entropy gives a more accurate entropy estimate (Richman & Moorman, 2000) than the approximate entropy (Approximate entropy, apEn) introduced by Pincus (Pincus, 1991; Pincus & Goldberger, 1994).