I was reviewing self information and then came to realize that there is one idea that I have that I believe should be wrong but don’t know why. Let self-information associated with a random variable realization be: $$I_{X}(x) = \log{\frac{1}{P_{X}(x)}}$$ The main idea is that realizing rare events contains more information that realizing event that […]

I’m reading the tutorial of Information Gain, and I see the following page: I know in the example above, I can encode this way: A 0 B 10 C 11 and then this will need $1/3*1+1/3*2+1/3*2=1.6667$ bits per symbol. But how can I code to achieve the 1.58 goal in theory? Is that possible in […]

A person is thinking of a number between 1 and 1000. What is the least number of yes/no questions that we can ask and know what that person’s number is given that the person is allowed to lie on at most one of her answers.

consider this a soft-question. Information Theory is fairly young branch of mathematics (60 years). I am interested in question, whether there have been any information theoretic results that had impact on other seemingly ‘unrelated’ branches of mathematics? Looking forward to hearing you thoughts.

We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon’s source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$. Also $H(p)=\log \frac{1}{p}$ is additive in the following sense. If $E$ and $F$ are two independent events with […]

I encountered the following claim $$\frac{1}{n+1}2^{nH_2(k/n)} \le \binom{n}{k} \le 2^{nH_2(k/n)}$$ where $H_2$ is the binary entropy function. The upper bound is rather well known but how does one show the lower bound?

mutual information of tow variables is $\displaystyle\sum\sum p(x,y)\ln\frac{p(x,y)}{p(x)p(y)}$ what is the mutual information of three variables? is it $\displaystyle\sum\sum\sum p(x,y,z)\ln\frac{p(x,y,z)}{p(x)p(y)p(z)}$?

This is the Guess-The-Number game with a twist! Variant 1 Take any positive integer $n$. The game-master chooses an $n$-bit integer $x$. The player makes queries one by one, each of the form “Is $x$ (strictly) less than $k$?”. The game-master answers each query immediately, always truthfully except at most once. At the end the […]

I am doing Information Retrieval using Cosine Similarity. My data is binary vector. Since most of all reference I read is using non-binary vector (non-binary matrix) data, I am wondering if it is wrong to use binary vector data in cosine similarity function.

Lets assume that two extreme intelligent species in the universe can exchange morse code messages for the first time. A can send messages to B and B to A, both have unlimited time, but they can not meet. Is it possible to transfer information between both species? What would be the mathematical precondition for information […]

Intereting Posts

Probability of picking a specific value from a countably infinite set
Finding $\lim\limits_{x \to \infty} (\sqrt{9x^2+x} – 3x)$
How to compute eigenvalues of big $5×5$ matrix (symmetric matrix) .
Truth Table for If P then Q
Proving that one of $a(1-b), b(1-c), c(1-a) \le \frac{1}{4}$
Distance to a closed set
Sample variance converge almost surely
Examples of nonlinear ordinary differential equations with elementary solutions.
Sieving integers
Evaluate a limit involving definite integral
On the existence of closed form solutions to finite combinatorial problems
Cube stack problem
Compute: $\int_{0}^{1}\frac{x^4+1}{x^6+1} dx$
Number of rooted subtrees of given size in infinite d-regular tree
Is there any formula for number of divisors of $a \times b$?