Hcfyk

Checking @@ROWCOUNT failing

How do you justify more code being written by following clean code practices?

What 1968 Moog synthesizer was used in the Movie Apollo 11?

Is there any common country to visit for persons holding UK and Schengen visas?

Toggle window scroll bar

Friend wants my recommendation but I don't want to give it to him

How can a new country break out from a developed country without war?

Sort with assumptions

Strange behavior in TikZ draw command

Reasons for having MCU pin-states default to pull-up/down out of reset

PTIJ: Which Dr. Seuss books should one obtain?

Are hand made posters acceptable in Academia?

1 John in Luther’s Bibel

How are passwords stolen from companies if they only store hashes?

Can creatures abilities target that creature itself?

How to test the sharpness of a knife?

is this saw blade faulty?

Travelling in US for more than 90 days

How to evaluate the research level of a paper before any publication?

What is the purpose of using a decision tree?

Output visual diagram of picture

What is the meaning of "You've never met a graph you didn't like?"

C++ lambda syntax

Why is implicit conversion not ambiguous for non-primitive types?

Unnormalized Log Probability - RNN

Is There any RNN method used for Object detectionHow is the LSTM RNN forget gate calculated?How to train the same RNN over multiple series?Find most important inputs of LSTM-RNN for multivariate time series modelingLoss function for an RNN used for binary classificationWhy the RNN has input shape error?Input and output Dimension of LSTM RNNWhat is the advantage of using RNN with fixed timestep length over Neural Network?Need to make an multivariate RNN, confused about input shape?RNN for prediciting Development over time

2

$begingroup$

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^(t)$ above is considered as unnormalized log probability. But if this is true, then the value of $O^(t)$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $Pin[0,1]$, where brackets denote closed interval. And $log(P) le 0$ on this interval. But in the equations above, nowhere this condition that $O^(t) le 0$ is explicitly enforced.

What am I missing!

deep-learning lstm recurrent-neural-net

share|improve this question

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

$endgroup$

add a comment | 

2

$begingroup$

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^(t)$ above is considered as unnormalized log probability. But if this is true, then the value of $O^(t)$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $Pin[0,1]$, where brackets denote closed interval. And $log(P) le 0$ on this interval. But in the equations above, nowhere this condition that $O^(t) le 0$ is explicitly enforced.

What am I missing!

deep-learning lstm recurrent-neural-net

share|improve this question

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

$endgroup$

add a comment | 

$begingroup$

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^(t)$ above is considered as unnormalized log probability. But if this is true, then the value of $O^(t)$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $Pin[0,1]$, where brackets denote closed interval. And $log(P) le 0$ on this interval. But in the equations above, nowhere this condition that $O^(t) le 0$ is explicitly enforced.

What am I missing!

deep-learning lstm recurrent-neural-net

share|improve this question

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

$endgroup$

share|improve this question

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

share|improve this question

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

edited 2 days ago

Siong Thye Goh

1,332419

edited 2 days ago

Siong Thye Goh

1,332419

asked 2 days ago

user3001408user3001408

360146

asked 2 days ago

user3001408user3001408

360146

2 Answers
2

active

oldest

votes

3

$begingroup$

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $textlog(0.5) < 0$ and $textlog(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $P(i) = e^o_i/sum_k=1^Ke^o_k$ (using softmax as mentioned in Figure 10.3 caption, and assuming $mathbfo=(o_1,..,o_K)$ is the output of layer before softmax),




2. Unnormalized probability: $tildeP(i) = e^o_i$, which can be larger than 1,




3. Log of unnormalized probability: $textlogtildeP(i) = o_i$, which can be positive or negative.

share|improve this answer

edited 2 days ago

answered 2 days ago

EsmailianEsmailian

1,511113

$endgroup$

add a comment | 

2

$begingroup$

You are right, nothing stop $o_k^(t)$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^(t)=ln q_k^(t)$

$$haty^(t)_k= fracexp(o^(t)_k)sum_k=1^K exp(o^(t)_k) = fracq_k^(t)sum_k=1^K q_k^(t)$$

Here $q_k^(t)$ can be any positive number, they will be normalized to be sum to $1$.

share|improve this answer

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

$endgroup$

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47460%2funnormalized-log-probability-rnn%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $textlog(0.5) < 0$ and $textlog(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $P(i) = e^o_i/sum_k=1^Ke^o_k$ (using softmax as mentioned in Figure 10.3 caption, and assuming $mathbfo=(o_1,..,o_K)$ is the output of layer before softmax),




2. Unnormalized probability: $tildeP(i) = e^o_i$, which can be larger than 1,




3. Log of unnormalized probability: $textlogtildeP(i) = o_i$, which can be positive or negative.

share|improve this answer

edited 2 days ago

answered 2 days ago

EsmailianEsmailian

1,511113

$endgroup$

add a comment | 

3

$begingroup$

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $textlog(0.5) < 0$ and $textlog(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $P(i) = e^o_i/sum_k=1^Ke^o_k$ (using softmax as mentioned in Figure 10.3 caption, and assuming $mathbfo=(o_1,..,o_K)$ is the output of layer before softmax),




2. Unnormalized probability: $tildeP(i) = e^o_i$, which can be larger than 1,




3. Log of unnormalized probability: $textlogtildeP(i) = o_i$, which can be positive or negative.

share|improve this answer

edited 2 days ago

answered 2 days ago

EsmailianEsmailian

1,511113

$endgroup$

add a comment | 

$begingroup$

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $textlog(0.5) < 0$ and $textlog(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $P(i) = e^o_i/sum_k=1^Ke^o_k$ (using softmax as mentioned in Figure 10.3 caption, and assuming $mathbfo=(o_1,..,o_K)$ is the output of layer before softmax),




2. Unnormalized probability: $tildeP(i) = e^o_i$, which can be larger than 1,




3. Log of unnormalized probability: $textlogtildeP(i) = o_i$, which can be positive or negative.

share|improve this answer

edited 2 days ago

answered 2 days ago

EsmailianEsmailian

1,511113

$endgroup$

share|improve this answer

edited 2 days ago

answered 2 days ago

EsmailianEsmailian

1,511113

answered 2 days ago

EsmailianEsmailian

1,511113

answered 2 days ago

EsmailianEsmailian

1,511113

2

$begingroup$

You are right, nothing stop $o_k^(t)$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^(t)=ln q_k^(t)$

$$haty^(t)_k= fracexp(o^(t)_k)sum_k=1^K exp(o^(t)_k) = fracq_k^(t)sum_k=1^K q_k^(t)$$

Here $q_k^(t)$ can be any positive number, they will be normalized to be sum to $1$.

share|improve this answer

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

$endgroup$

add a comment | 

2

$begingroup$

You are right, nothing stop $o_k^(t)$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^(t)=ln q_k^(t)$

$$haty^(t)_k= fracexp(o^(t)_k)sum_k=1^K exp(o^(t)_k) = fracq_k^(t)sum_k=1^K q_k^(t)$$

Here $q_k^(t)$ can be any positive number, they will be normalized to be sum to $1$.

share|improve this answer

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

$endgroup$

add a comment | 

$begingroup$

You are right, nothing stop $o_k^(t)$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^(t)=ln q_k^(t)$

$$haty^(t)_k= fracexp(o^(t)_k)sum_k=1^K exp(o^(t)_k) = fracq_k^(t)sum_k=1^K q_k^(t)$$

Here $q_k^(t)$ can be any positive number, they will be normalized to be sum to $1$.

share|improve this answer

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

$endgroup$

share|improve this answer

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419

answered 2 days ago

Siong Thye GohSiong Thye Goh

1,332419