Information Theory with Applications, Math Texts: A. I. Khinchin, Mathematical Foundations of Information Theory, Dover, , reprint. mathematical foundations have been put down for information theory Khinchin noticed that except for some simple classes of channel (e.g. The first comprehensive introduction to information theory, this text explores by Shannon and continued by McMillan, Feinstein, and Khinchin.
|Language:||English, Spanish, Dutch|
|ePub File Size:||22.77 MB|
|PDF File Size:||13.22 MB|
|Distribution:||Free* [*Register to download]|
Mathematical foundations of information theory External-identifier: urn:acs6: mathematicalfoun00ayak:pdfac05f-b33e-4baa The first comprehensive introduction to information theory, this text explores the work begun by Shannon and continued by McMillan, Feinstein, and Khinchin. principle of information theory) is provable by the formula for the measure of information. the mathematical foundations for the model of an interdisciplinary information (echecs16.info pdf).
J L Doob wrote:- [Khinchin] shows how to make classical statistical mechanics a respectable rigorous discipline, with a consistent mathematical content. G Gamow translated the book into English and this version of the work was published by Dover Publications in After the molecular theory of the structure of matter attained a predominant role in physics, the appearance of new statistical or probabilistic methods of investigation in physical theories became unavoidable. From this new point of view each portion of matter solid, liquid, or gaseous was considered as a collection of a large number of very small particles. Very little was known about the nature of these particles except that their number was extremely large, that in a homogeneous material these particles had the same properties, and that these particles were in a certain kind of interaction. The dimensions and structure of the particles, as well as the laws of the interaction could be determined only hypothetically.
This tendency is clearly present in many modem investigations. According to a historically accepted terminology, such investigations are considered to belong to the kinetic theory of matter, as distinct from the statistical mechanics which tries to reduce such hypotheses to a minimum by using statistical methods as much as possible.
Each of these two tendencies has its own advantages. For instance, the kinetic theory is indispensable when dealing with problems concerning the motion of separate particles number of collisions, problems concerning the study of systems of special kinds, mono-atomic ideal gas ; the methods of the kinetic theory are also often preferable, because they give a treatment of the phenomena which is simpler mathematically and more detailed.
But in questions concerning the theoretical foundation of general laws valid for a great variety of systems, the kinetic theory naturally becomes sometimes powerless and has to be replaced by a theory which makes as few special hypotheses as possible concerning the nature of the particles.
In particular, it was precisely the necessity of a statistical foundation for the general laws of thermodynamics that produced trends which found their expression in the construction of statistical mechanics. To avoid making any special hypotheses about the nature of the particles it became necessary in establishing a statistical foundation to develop laws which had to be valid no matter what was the nature of these particles within quite wide limitations.
The first systematic exposition of the foundations of statistical mechanics, with fairly far developed applications to thermodynamics and some other physical theories, was given in Gibbs' well-known book [W Gibbs, Elementary principles of statistical mechanics, Yale University Press, ]. Besides the above mentioned tendency not to make any hypotheses about the nature of particles the following are characteristic of the exposition of Gibbs.
At the time of publication of Gibbs' book, the fundamental problems raised in mathematical science in connection with the foundation of statistical mechanics became more or less clear.
If we disregard some isolated small problems, we have here two fundamental groups of problems representing a broad, deep, interesting and difficult field of research in mathematics which is far from being exhausted even at present.
The first of these groups is centred around the so-called ergodic problem, that is, the problem of the logical foundation for the interpretation of physical quantities by averages of their corresponding functions, averages taken over the phase-space or a suitably selected part of it.
This problem, originated by Boltzmann, apparently is far from its complete solution even at the present time. This group of problems was neglected by the investigators for a long time after some unsuccessful attempts, based either on some inappropriate hypotheses introduced ad hoc, or on erroneous logical and mathematical arguments which, unfortunately, have been repeated without any criticism in later handbooks.
In the book of Gibbs these problems naturally are not considered because of the tendency to construct models. Only recently , the remarkable work of G D Birkhoff again attracted the attention of many investigators to these problems, and since then this group of problems has never ceased to interest mathematicians, who devote more and more effort to it every year.
We will discuss this group of problems in more detail [later]. The second group of problems is connected with the methods of computation of the phase-averages.
In the majority of cases, these averages cannot be calculated precisely. The formulas which are derived for them in the general theory that is, without specification of the mechanical system under discussion are complicated, not easy to survey, and as a rule, not suited for mathematical treatment. It is quite natural, therefore, to try to find simpler and more convenient approximations for these averages. This problem is always formulated as a problem of deriving asymptotic formulas which approach the precise formulas when the number of particles constituting the given system increases beyond any limit.
These asymptotic formulas have been found long ago by a semi-heuristic method by means of an unproved extrapolation, starting from some of the simplest examples and were without rigorous mathematical justification until fairly recent years.
A decided change in this direction was brought about by the papers of Darwin and Fowler about twenty years ago. Strictly speaking these authors were the first to give a systematic computation of the average values; up to that time, such a computation was in most cases replaced by a more or less convincing determination of "most probable" values which without rigorous justification were assumed to be approximately equal to the corresponding average values.
Darwin and Fowler also created a simple, convenient, and mathematically rigorous apparatus for the computation of asymptotic formulas. The only defect of their theory lies in an extreme abstruseness of the justification of their mathematical method.
To a considerable extent this abstruseness was due to the fact that the authors did not use the limit theorems of the theory of probability sufficiently developed by that time , but created anew the necessary analytical apparatus. In any case, the course in statistical mechanics published by Fowler on the basis of this method, represents up to now the only book on the subject, which is on a satisfactory mathematical level.
In closing this brief sketch we should mention that the development of atomic mechanics during the last decades has changed the face of physical statistics to such a degree that, naturally, statistical mechanics had to extend its mathematical apparatus in order to include also quantum phenomena. Moreover, from the modem point of view, we should consider quantized systems as a general type of which the classical systems are a limiting case. Fowler's course is arranged according to precisely this point of view: the new method of constructing asymptotic formulas for phase-averages is established and developed for the quantized systems, and the formulas which correspond to the classical systems are obtained from these by a limiting process.
The full implication of Eq. The statistical properties of a physical system—for instance, a superstatistical system as discussed in Hanel et al. This unique duality allows us to derive a complete theory of generalized logarithms naturally arising as a consequence of the fourth SK axiom being violated.
We present a full understanding of how the TS and HT approaches are interrelated and derive the most general form of families of generalized logarithms that are compatible with a maximum entropy principle and the first three SK axioms. Finally, we demonstrate how these logarithms can be classified according to their asymptotic scaling properties, following the results presented in Hanel and Thurner 6.
In the following, we call the elements g of this automorphism group scale transformations. Scale transformations leave the image of a generalized logarithm invariant, which allows us to parameterize classes in the following way.
Next, we define sets of proper generalized logarithms, Members of all have the same maximum and minimum values.
The space of generalized logarithms can be written as the union of these sets,. With these definitions we now analyze the relation between the HT and TS approaches. Functional Form of the Generalized Logarithms Eq. The same holds for , which allows us to construct with the properties into Eq. A particular solution of Eq. The above argument means that we can generate a specific family of logarithms , following Eq.
In particular, some family with the property can be reached by a family of scale transformations , where also follows Eq. The family of dual logarithms discussed in Hanel et al. These classes correspond to logarithms that are unbounded either from below or from above, whereas the duality maps.
Moreover, in Hanel et al. We are now in a position to understand all observable distribution functions emerging from the two approaches in terms of a single two-parameter family of generalized logarithms and a scale transformation. This result now raises the question of how is related to the two-parameter logarithms associated with the c, d entropies in Eq.
Using Eq. From Eq. The dual is , and we recover the well-known duality for q-logarithms. An example of a generalized logarithm that is not a power is obtained by taking in Eq. One obtains , with the dual Example 3.
Scale transformations: Any proper generalized logarithm can be written as a composition of a representative logarithm from Eq. Example 4. Then can only differ by a scale transformation with. For the particular functions hA and hB we have chosen, we get. Discussion By studying the two types of entropies that are related to the two possible ways to formulate a maximum entropy principle for systems that explicitly violate the fourth SK axiom, we find that there exists a unique duality that relates the two entropies.
Consequently, thermodynamic properties derived from those two entropies will also be related through the duality.
The existence of this duality opens the way to characterizing all possible generalized logarithms as compositions of a specific functional form and scale transformations g.
We derive the explicit form of and show that these logarithms are one-to-one related to two asymptotic scaling exponents c, d that allow one to characterize strongly nonergodic or non-Markovian systems in their thermodynamic limit 6.
In summary, we provide a complete theory of all generalized logarithms that can arise as a consequence of the violation of the fourth SK axiom. Supplementary Material Click here to view. Acknowledgments R. Zwan Foundation M.
Footnotes The authors declare no conflict of interest.