Throughout the telecommunication increase, Claude Shannon, in his seminal 1948 paper¹, posed a query that will revolutionise expertise:
How can we quantify communication?
Shannon’s findings stay elementary to expressing info quantification, storage, and communication. These insights made main contributions to the creation of applied sciences starting from sign processing, information compression (e.g., Zip recordsdata and compact discs) to the Web and synthetic intelligence. Extra broadly, his work has considerably impacted numerous fields equivalent to neurobiology, statistical physics and pc science (e.g, cybersecurity, cloud computing, and machine studying).
[Shannon’s paper is the]
Magna Carta of the Info Age
That is the primary article in a collection that explores info quantification – a vital device for information scientists. Its functions vary from enhancing statistical analyses to serving as a go-to choice heuristic in cutting-edge machine studying algorithms.
Broadly talking, quantifying info is assessing uncertainty, which can be phrased as: “how shocking is an consequence?”.
This text concept rapidly grew right into a collection since I discovered this matter each fascinating and numerous. Most researchers, at one stage or one other, come throughout generally used metrics equivalent to entropy, cross-entropy/KL-divergence and mutual-information. Diving into this matter I discovered that in an effort to absolutely admire these one must be taught a bit in regards to the fundamentals which we cowl on this first article.
By studying this collection you’ll achieve an instinct and instruments to quantify:
- Bits/Nats – Unit measures of knowledge.
- Self-Info – **** The quantity of knowledge in a particular occasion.
- Pointwise Mutual Info – The quantity of knowledge shared between two particular occasions.
- Entropy – The common quantity of knowledge of a variable’s consequence.
- Cross-entropy – The misalignment between two chance distributions (additionally expressed by its spinoff KL-Divergence – a distance measure).
- Mutual Info – The co-dependency of two variables by their conditional chance distributions. It expresses the knowledge achieve of 1 variable given one other.
No prior information is required – only a primary understanding of chances.
I reveal utilizing widespread statistics equivalent to coin and cube 🎲 tosses in addition to machine studying functions equivalent to in supervised classification, function choice, mannequin monitoring and clustering evaluation. As for actual world functions I’ll talk about a case examine of quantifying DNA range 🧬. Lastly, for enjoyable, I additionally apply to the favored mind tornado generally often known as the Monty Corridor drawback 🚪🚪 🐐 .
All through I present python code 🐍 , and attempt to maintain formulation as intuitive as attainable. In case you have entry to an built-in improvement atmosphere (IDE) 🖥 you would possibly need to plug 🔌 and play 🕹 round with the numbers to achieve a greater instinct.
This collection is split into 4 articles, every exploring a key facet of Information Theory:
-
😲 Quantifying Shock: 👈 👈 👈 YOU ARE HERE On this opening article, you’ll learn to quantify the “shock” of an occasion utilizing _self-informatio_n and perceive its models of measurement, equivalent to _bit_s and _nat_s. Mastering self-information is crucial for constructing instinct in regards to the subsequent ideas, as all later heuristics are derived from it.
- 🤷 Quantifying Uncertainty: Constructing on self-information, this text shifts focus to the uncertainty – or “common shock” – related to a variable, often known as entropy. We’ll dive into entropy’s wide-ranging functions, from Machine Learning and information evaluation to fixing enjoyable puzzles, showcasing its adaptability.
- 📏 Quantifying Misalignment: Right here, we’ll discover how you can measure the gap between two chance distributions utilizing entropy-based metrics like cross-entropy and KL-divergence. These measures are significantly helpful for duties like evaluating predicted versus true distributions, as in classification loss features and different alignment-critical situations.
- 💸 Quantifying Achieve: Increasing from single-variable measures, this text investigates the relationships between two. You’ll uncover how you can quantify the knowledge gained about one variable (e.g, goal Y) by understanding one other (e.g., predictor X). Functions embrace assessing variable associations, function choice, and evaluating clustering efficiency.
Every article is crafted to face alone whereas providing cross-references for deeper exploration. Collectively, they supply a sensible, data-driven introduction to info concept, tailor-made for information scientists, analysts and machine studying practitioners.
Disclaimer: Except in any other case talked about the formulation analysed are for categorical variables with c≥2 courses (2 that means binary). Steady variables can be addressed in a separate article.
🚧 Articles (3) and (4) are at the moment underneath development. I’ll share hyperlinks as soon as out there. Follow me to be notified 🚧
Quantifying Shock with Self-Info
Self-information is taken into account the constructing block of knowledge quantification.
It’s a means of quantifying the quantity of “shock” of a particular consequence.
Formally self-information, or additionally known as Shannon Info or info content material, quantifies the shock of an occasion x occurring based mostly on its chance, p(x). Right here we denote it as hₓ:
The models of measure are referred to as bits. One bit (binary digit) is the quantity of knowledge for an occasion x that has chance of p(x)=½. Let’s plug in to confirm: hₓ=-log₂(½)= log₂(2)=1 bit.
This heuristic serves as an alternative choice to chances, odds and log-odds, with sure mathematical properties that are advantageous for info concept. We talk about these under when studying about Shannon’s axioms behind this selection.
It’s all the time informative to discover how an equation behaves with a graph:
To deepen our understanding of self-information, we’ll use this graph to discover the stated axioms that justify its logarithmic formulation. Alongside the best way, we’ll additionally construct instinct about key options of this heuristic.
To emphasize the logarithmic nature of self-information, I’ve highlighted three factors of curiosity on the graph:
- At p=1 an occasion is assured, yielding no shock and therefore zero bits of knowledge (zero bits). A helpful analogy is a trick coin (the place either side present HEAD).
- Decreasing the chance by an element of two (p=½) will increase the knowledge to _hₓ=_1 bit. This, after all, is the case of a good coin.
- Additional decreasing it by an element of 4 leads to hₓ(p=⅛)=3 bits.
In case you are all for coding the graph here’s a python script:
To summarise this part:
Self-Info hₓ=-log₂(p(x)) quantifies the quantity of “shock” of a particular consequence x.
Three Axioms
Referencing prior work by Ralph Hartley, Shannon selected -log₂(p) as a way to satisfy three axioms. We’ll use the equation and graph to look at how these are manifested:
-
An occasion with chance 100% isn’t a surprise and therefore doesn’t yield any info. Within the trick coin case that is evident by p(x)=1 yielding hₓ=0.
-
Much less possible occasions are extra shocking and supply extra info. That is obvious by self-information reducing monotonically with growing chance.
- The property of Additivity – the overall self-information of two unbiased occasions equals the sum of particular person contributions. This can be explored additional within the upcoming fourth article on Mutual Info.
There are mathematical proofs (that are past the scope of this collection) that present that solely the log operate adheres to all three².
The applying of those axioms reveals a number of intriguing and sensible properties of self-information:
Necessary properties :
- Minimal sure: The primary axiom hₓ(p=1)=0 establishes that self-information is non-negative, with zero as its decrease sure. That is extremely sensible for a lot of functions.
- Monotonically reducing: The second axiom ensures that self-information decreases monotonically with growing chance.
- No Most sure: On the excessive the place _p→_0, monotonicity results in self-information rising with out sure hₓ(_p→0) →_ ∞, a function that requires cautious consideration in some contexts. Nonetheless, when averaging self-information – as we are going to later see within the calculation of entropy – chances act as weights, successfully limiting the contribution of extremely unbelievable occasions to the general common. This relationship will turn into clearer after we discover entropy intimately.
It’s helpful to know the shut relationship to log-odds. To take action we outline p(x) because the chance of occasion x to occur and p(¬x)=1-p(x) of it to not occur. log-odds(x) = log₂(p(x)/p(¬x))= h(¬x) – h(x).
The principle takeaways from this part are
Axiom 1: An occasion with chance 100% isn’t a surprise
Axiom 2: Much less possible occasions are extra shocking and, after they happen, present extra info.
Self info (1) monotonically decreases (2) with a minimal sure of zero and (3) no higher sure.
Within the subsequent two sections we additional talk about models of measure and selection of normalisation.
Info Models of Measure
Bits or Shannons?
A bit, as talked about, represents the quantity of knowledge related to an occasion that has a 50% chance of occurring.
The time period can be typically known as a Shannon, a naming conference proposed by mathematician and physicist David MacKay to keep away from confusion with the time period ‘bit’ within the context of digital processing and storage.
After some deliberation, I made a decision to make use of ‘bit’ all through this collection for a number of causes:
- This collection focuses on quantifying info, not on digital processing or storage, so ambiguity is minimal.
- Shannon himself, inspired by mathematician and statistician John Tukey, used the time period ‘bit’ in his landmark paper.
- ‘Bit’ is the usual time period in a lot of the literature on info concept.
- For comfort – it’s extra concise
Normalisation: Log Base 2 vs. Pure
All through this collection we use base 2 for logarithms, reflecting the intuitive notion of a 50% likelihood of an occasion as a elementary unit of knowledge.
An alternate generally utilized in machine studying is the pure logarithm, which introduces a distinct unit of measure referred to as nats (brief for natural models of knowledge). One nat corresponds to the knowledge gained from an occasion occurring with a chance of 1/e the place e is Euler’s quantity (≈2.71828). In different phrases, 1 nat = -ln(p=(1/e)).
The connection between bits (base 2) and nats (pure log) is as follows:
1 bit = ln(2) nats ≈ 0.693 nats.
Consider it as much like a financial present change or changing centimeters to inches.
In his seminal publication Shanon defined that the optimum selection of base relies on the precise system being analysed (paraphrased barely from his unique work):
- “A tool with two secure positions […] can retailer one bit of knowledge” (bit as in binary digit).
- “A digit wheel on a desk computing machine that has ten secure positions […] has a storage capability of 1 decimal digit.”³
- “In analytical work the place integration and differentiation are concerned the bottom e is typically helpful. The ensuing models of knowledge can be referred to as pure models.“
Key facets of machine studying, equivalent to in style loss features, typically depend on integrals and derivatives. The pure logarithm is a sensible selection in these contexts as a result of it may be derived and built-in with out introducing further constants. This probably explains why the machine studying neighborhood regularly makes use of nats because the unit of knowledge – it simplifies the arithmetic by avoiding the necessity to account for elements like ln(2).
As proven earlier, I personally discover base 2 extra intuitive for interpretation. In circumstances the place normalisation to a different base is extra handy, I’ll make an effort to clarify the reasoning behind the selection.
To summarise this part of models of measure:
bit = quantity of knowledge to tell apart between two equally probably outcomes.
Now that we’re accustomed to self-information and its unit of measure let’s look at a number of use circumstances.
Quantifying Occasion Info with Cash and Cube
On this part, we’ll discover examples to assist internalise the self-information axioms and key options demonstrated within the graph. Gaining a strong understanding of self-information is crucial for greedy its derivatives, equivalent to entropy, cross-entropy (or KL divergence), and mutual info – all of that are averages over self-information.
The examples are designed to be easy, approachable, and lighthearted, accompanied by sensible Python code that will help you experiment and construct instinct.
Be aware: For those who really feel snug with self-information, be happy to skip these examples and go straight to the Quantifying Uncertainty article.
To additional discover the self-information and bits, I discover analogies like coin flips and cube rolls significantly efficient, as they’re typically helpful analogies for real-world phenomena. Formally, these will be described as multinomial trials with n=1 trial. Particularly:
- A coin flip is a Bernoulli trial, the place there are c=2 attainable outcomes (e.g., heads or tails).
- Rolling a die represents a categorical trial, the place c≥3 outcomes are attainable (e.g., rolling a six-sided or eight-sided die).
As a use case we’ll use simplistic climate studies restricted to that includes solar 🌞 , rain 🌧 , and snow ⛄️.
Now, let’s flip some digital cash 👍 and roll some funky-looking cube 🎲 …
Truthful Cash and Cube
We’ll begin with the only case of a good coin (i.e, 50% likelihood for achievement/Heads or failure/Tails).
Think about an space for which at any given day there’s a 50:50 likelihood for solar or rain. We are able to write the chance of every occasion be: p(🌞 )=p(🌧 )=½.
As seen above, in accordance the the self-information formulation, when 🌞 or 🌧 is reported we’re supplied with h(🌞 __ )=h(🌧 )=-log₂(½)=1 bit of knowledge.
We are going to proceed to construct on this analogy afterward, however for now let’s flip to a variable that has greater than two outcomes (c≥3).
Earlier than we tackle the usual six sided die, to simplify the maths and instinct, let’s assume an 8 sided one (_c=_8) as in Dungeons Dragons and different tabletop video games. On this case every occasion (i.e, touchdown on either side) has a chance of p(🔲 ) = ⅛.
When a die lands on one aspect dealing with up, e.g, worth 7️⃣, we’re supplied with h(🔲 =7️⃣)=-log₂(⅛)=3 bits of knowledge.
For the standard six sided truthful die: p(🔲 ) = ⅙ → an occasion yields __ h(🔲 )=-log₂(⅙)=2.58 bits.
Evaluating the quantity of knowledge from the truthful coin (1 bit), 6 sided die (2.58 bits) and eight sided (3 bits) we establish the second axiom: The much less possible an occasion is, the extra shocking it’s and the extra info it yields.
Self info turns into much more fascinating when chances are skewed to desire sure occasions.
Loaded Cash and Cube
Let’s assume a area the place p(🌞 ) = ¾ and p(🌧 )= ¼.
When rain is reported the quantity of knowledge conveyed will not be 1 bit however somewhat h(🌧 )=-log₂(¼)=2 bits.
When solar is reported much less info is conveyed: h(🌞 )=-log₂(¾)=0.41 bits.
As per the second axiom— a rarer occasion, like p(🌧 )=¼, reveals extra info than a extra probably one, like p(🌞 )=¾ – and vice versa.
To additional drive this level let’s now assume a desert area the place p(🌞 ) =99% and p(🌧 )= 1%.
If sunshine is reported – that’s type of anticipated – so nothing a lot is learnt (“nothing new underneath the solar” 🥁) and that is quantified as h(🌞 )=0.01 bits. If rain is reported, nevertheless, you possibly can think about being fairly shocked. That is quantified as h(🌧 )=6.64 bits.
Within the following python scripts you possibly can look at all of the above examples, and I encourage you to play with your individual to get a sense.
First let’s outline the calculation and printout operate:
import numpy as np
def print_events_self_information(probs):
for ps in probs:
print(f"Given distribution {ps}")
for occasion in ps:
if ps[event] != 0:
self_information = -np.log2(ps[event]) #identical as: -np.log(ps[event])/np.log(2)
text_ = f'When `{occasion}` happens {self_information:0.2f} bits of knowledge is communicated'
print(text_)
else:
print(f'a `{occasion}` occasion can't occur p=0 ')
print("=" * 20)Subsequent we’ll set a number of instance distributions of climate frequencies
# Setting a number of chance distributions (every sums to 100%)
# Enjoyable truth - 🐍 💚 Emojis!
probs = [{'🌞 ': 0.5, '🌧 ': 0.5}, # half-half
{'🌞 ': 0.75, '🌧 ': 0.25}, # more sun than rain
{'🌞 ': 0.99, '🌧 ': 0.01} , # mostly sunshine
]
print_events_self_information(probs)This yields printout
Given distribution {'🌞 ': 0.5, '🌧 ': 0.5}
When `🌞 ` happens 1.00 bits of knowledge is communicated
When `🌧 ` happens 1.00 bits of knowledge is communicated
====================
Given distribution {'🌞 ': 0.75, '🌧 ': 0.25}
When `🌞 ` happens 0.42 bits of knowledge is communicated
When `🌧 ` happens 2.00 bits of knowledge is communicated
====================
Given distribution {'🌞 ': 0.99, '🌧 ': 0.01}
When `🌞 ` happens 0.01 bits of knowledge is communicated
When `🌧 ` happens 6.64 bits of knowledge is communicated Let’s look at a case of a loaded three sided die. E.g, info of a climate in an space that studies solar, rain and snow at uneven chances: p(🌞 ) = 0.2, p(🌧 )=0.7, p(⛄️)=0.1.
Operating the next
print_events_self_information([{'🌞 ': 0.2, '🌧 ': 0.7, '⛄️': 0.1}])yields
Given distribution {'🌞 ': 0.2, '🌧 ': 0.7, '⛄️': 0.1}
When `🌞 ` happens 2.32 bits of knowledge is communicated
When `🌧 ` happens 0.51 bits of knowledge is communicated
When `⛄️` happens 3.32 bits of knowledge is communicated What we noticed for the binary case applies to greater dimensions.
To summarise – we clearly see the implications of the second axiom:
- When a extremely anticipated occasion happens – we don’t be taught a lot, the bit rely is low.
- When an sudden occasion happens – we be taught quite a bit, the bit rely is excessive.
Occasion Info Abstract
On this article we launched into a journey into the foundational ideas of knowledge concept, defining how you can measure the shock of an occasion. Notions launched function the bedrock of many instruments in info concept, from assessing information distributions to unraveling the inside workings of machine studying algorithms.
By means of easy but insightful examples like coin flips and cube rolls, we explored how self-information quantifies the unpredictability of particular outcomes. Expressed in bits, this measure encapsulates Shannon’s second axiom: rarer occasions convey extra info.
Whereas we’ve centered on the knowledge content material of particular occasions, this naturally results in a broader query: what’s the common quantity of knowledge related to all attainable outcomes of a variable?
Within the subsequent article, Quantifying Uncertainty, we construct on the inspiration of self-information and bits to discover entropy – the measure of common uncertainty. Removed from being only a lovely theoretical assemble, it has sensible functions in information evaluation and machine studying, powering duties like choice tree optimisation, estimating range and extra.
Liked this submit? ❤️🍕
💌 Comply with me right here, be part of me on LinkedIn or 🍕 buy me a pizza slice!
About This Collection
Although I’ve twenty years of expertise in information evaluation and predictive modelling I all the time felt fairly uneasy about utilizing ideas in info concept with out really understanding them.
The aim of this collection was to place me extra relaxed with ideas of knowledge concept and hopefully present for others the reasons I wanted.
Try my different articles which I wrote to higher perceive Causality and Bayesian Statistics:
Footnotes
¹ A Mathematical Principle of Communication, Claude E. Shannon, Bell System Technical Journal 1948.
It was later renamed to a e book The Mathematical Principle of Communication in 1949.
[Shannon’s “A Mathematical Theory of Communication”] the blueprint for the digital period – Historian James Gleick
² See Wikipedia web page on Information Content (i.e, self-information) for an in depth derivation that solely the log operate meets all three axioms.
³ The decimal-digit was later renamed to a hartley (image Hart), a ban or a dit. See Hartley (unit) Wikipedia web page.
Credit
Except in any other case famous, all photos have been created by the writer.
Many due to Will Reynolds and Pascal Bugnion for his or her helpful feedback.
Source link
