Over four outcomes {x1,x2,x3,x4}\{x_1, x_2, x_3, x_4\}{x1,x2,x3,x4} consider
P=[ 0.5, 0.5, 0, 0 ]Q=[ 0.25, 0.25, 0.25, 0.25 ]P = [\,0.5,\ 0.5,\ 0,\ 0\,] \qquad Q = [\,0.25,\ 0.25,\ 0.25,\ 0.25\,]P=[0.5, 0.5, 0, 0]Q=[0.25, 0.25, 0.25, 0.25]
Evaluate the KL divergence in both directions, in bits, using the convention 0log0=00 \log 0 = 00log0=0.
Which statement is correct?
Select all that apply.
DKL(P∥Q)=2D_{KL}(P \parallel Q) = 2DKL(P∥Q)=2 bits, and DKL(Q∥P)=2D_{KL}(Q \parallel P) = 2DKL(Q∥P)=2 bits
Both directions equal 111 bit, because QQQ is the uniform distribution
DKL(P∥Q)=1D_{KL}(P \parallel Q) = 1DKL(P∥Q)=1 bit, and DKL(Q∥P)=−0.5D_{KL}(Q \parallel P) = -0.5DKL(Q∥P)=−0.5 bits
DKL(P∥Q)=1D_{KL}(P \parallel Q) = 1DKL(P∥Q)=1 bit, and DKL(Q∥P)D_{KL}(Q \parallel P)DKL(Q∥P) is infinite