Can we assume that a hash function with high collision resistance also means a highly uniform distribution?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
$begingroup$
I want to use a hash function to generate a random sequence from number 0-n. And so I would like to find a good function that results in values that are seemingly random (does not need to be secure), but gives a sequence that is uniformly distributed.
Can I look at a function that has the property of high collision resistance and expect that it would also have a highly uniform distribution?
hash random-number-generator collision-resistance
$endgroup$
add a comment |
$begingroup$
I want to use a hash function to generate a random sequence from number 0-n. And so I would like to find a good function that results in values that are seemingly random (does not need to be secure), but gives a sequence that is uniformly distributed.
Can I look at a function that has the property of high collision resistance and expect that it would also have a highly uniform distribution?
hash random-number-generator collision-resistance
$endgroup$
1
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
1
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
1
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
1
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
1
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09
add a comment |
$begingroup$
I want to use a hash function to generate a random sequence from number 0-n. And so I would like to find a good function that results in values that are seemingly random (does not need to be secure), but gives a sequence that is uniformly distributed.
Can I look at a function that has the property of high collision resistance and expect that it would also have a highly uniform distribution?
hash random-number-generator collision-resistance
$endgroup$
I want to use a hash function to generate a random sequence from number 0-n. And so I would like to find a good function that results in values that are seemingly random (does not need to be secure), but gives a sequence that is uniformly distributed.
Can I look at a function that has the property of high collision resistance and expect that it would also have a highly uniform distribution?
hash random-number-generator collision-resistance
hash random-number-generator collision-resistance
edited May 23 at 9:31
Patriot
1,0211 gold badge6 silver badges25 bronze badges
1,0211 gold badge6 silver badges25 bronze badges
asked May 21 at 5:08
ThainaThaina
1881 silver badge5 bronze badges
1881 silver badge5 bronze badges
1
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
1
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
1
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
1
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
1
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09
add a comment |
1
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
1
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
1
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
1
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
1
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09
1
1
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
1
1
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
1
1
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
1
1
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
1
1
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09
add a comment |
3 Answers
3
active
oldest
votes
$begingroup$
Define $H(x) = operatorname{SHA-256}(x) mathbin| 1$; that is, append a single 1 bit to SHA-256. Can you find a collision under $H$? Does $H$ have anything resembling uniform distribution?
This counterexample is not merely pathological; designs like Rumba20 and VSH provide collision resistance but neither preimage resistance nor uniformity.
That said, typical ‘cryptographic hash functions’ like SHA-256, BLAKE2b, and SHAKE128 are designed for collision resistance and preimage resistance, and more broadly for random oracle modeling (barring length extension attacks on SHA-256), meaning that the outputs on distinct inputs can reasonably be modeled as independently uniformly distributed.
In the dark ages of the early '90s, when the United States still banned the export of cryptography as a munition, the ban covered encryption, per se, like DES—but had an explicit exception for authentication (22 CFR §121.1(XIII)(b)(1)(vi), since rescinded), and so the hash function Snefru was allowed to be published and exported.
A grad student named Dan pointed out in 1992 that you could use Snefru as a subroutine in an otherwise cryptography-free program to encrypt messages. When he informed the United States Department of State of his remarkable discovery, and asked them to confirm his understanding that publishing his cryptography-free program together with the exempted Snefru would not fall afoul of the export controls, they were not amused.
The State Department's lack of humor led to a nearly decade-long court battle, Bernstein v. United States, about whether the regulations in 22 CFR §§120–130 and 15 CFR §§730–744 prohibiting the export of encryption software constitute prior restraint in violation of the First Amendment to the United States Constitution. Eventually, the United States federal government, backed into a corner by an annoying grad student, relaxed the regulations and the case was dismissed.
Today, a newer incarnation of the same idea—using a hash function, ChaCha, together with a method inspired by the advanced technology of the one-time pad (also known in some circles as ‘xor’), to encrypt messages—protects the confidentiality of probably petabytes of data daily on the internet, in the form of the ChaCha/Poly1305 cipher suites in TLS.
But collision resistance is neither necessary—indeed, it is well-known that ChaCha is not collision-resistant—nor sufficient—as Rumba20 and VSH show—for indistinguishability from uniform random, which is what one needs for, e.g., a one-time pad to get any security.
P.S. If you do use a hash function, e.g. generating the sequence of bits $H(k mathbin| 0)$, $H(k mathbin| 1)$, etc., and want to use that sequence of bits to choose an integer $x$ with $0 leq x < n$ uniformly at random, make sure to do rejection sampling to avoid modulo bias if $n$ is not a power of two: if $H$ returns a string of $t$ bits interpreted as a $t$-bit integer, and $H(k mathbin| i)$ is below $2^t bmod n$, reject it and try $i + 1$; otherwise return $H(k mathbin| i) bmod n$
$endgroup$
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
add a comment |
$begingroup$
No, but high collision resistance per bit has an influence.
Non-uniformity -> less entropy -> weakned collision resistance.
As keysize is significant factor: most cryptographic hash functions have uniform output given entropic input. Using a hash (or encryption) routine to make a stream of random numbers from a single block of entropy is an established practice. There are lists of recommended primatives and pitfalls etc.
From wikipedia:
"A cryptographically secure hash of a counter might also act as a good CSPRNG in some cases. In this case, it is also necessary that the initial value of this counter is random and secret. However, there has been little study of these algorithms for use in this manner, and at least some authors warn against this use."
$endgroup$
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
add a comment |
$begingroup$
Only my 2 cents, you don't necessarily need an (heavy) hash function to generate a sequence or a series of pseudo-random numbers.
A practical, low-level way, to generate an uniform distribution of values, in a specific range, could be to use a simple acceptance/rejection method, combined with pseudo-random generation of numbers:
if we want $k$ different output values, then our range will be $[0, k-1] $
use a dumb 0/1- generator, like Math.random for js, a refined quantum random input source, or simply flip a coin $r$ times, for generating a pseudo-random sequence of $r$ bits, where:
- $r = roundup( log_{2}(k) )$
the resulting value $n$ is in the interval $ [0, 2^{r}-1]$
if $( n < k )$, pick and output the number (no modulo operation)
otherwise, discard the number and regenerate a new pseudo-random sequence/number.
Obviously, we will consume more random input data from our source, when $k<<2^{r}$, because we expect much more rejections, but using the modulo is not a good solution in this case, it will not give us a uniform distribution.
The best scenario occurs when $k$ is a power of $2$:
$k = 2^{i}$, for a generic natural $i$
- we consume precisely $1$-bit of input data to produce $1$-bit of output
- no waste of time
The worst scenario for rejections occurs when:
- $k = (2^{r-1}+1)$
$(k - 1)= 2^{r-1}$- $r = 1 + log_{2}(k - 1)$
- we'll consume ~ $2$-bit of input data to produce $1$-bit of output
- we'll spend much more time in the generation process
For example, if we need, for a chosen interval $[0, k-1]$:
$r = 10$ bits to generate at least $k$ different values, then:- when $k = ( 2^{10-1} + 1 ) = 513$
- we expect to sacrifice, on average, about ~$50%$ of our pseudo-random input data, discarding all output numbers $> 513$, when they occur.
- a total of $(2^{10} - 513 ) = 511$ values
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "281"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcrypto.stackexchange.com%2fquestions%2f70707%2fcan-we-assume-that-a-hash-function-with-high-collision-resistance-also-means-a-h%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Define $H(x) = operatorname{SHA-256}(x) mathbin| 1$; that is, append a single 1 bit to SHA-256. Can you find a collision under $H$? Does $H$ have anything resembling uniform distribution?
This counterexample is not merely pathological; designs like Rumba20 and VSH provide collision resistance but neither preimage resistance nor uniformity.
That said, typical ‘cryptographic hash functions’ like SHA-256, BLAKE2b, and SHAKE128 are designed for collision resistance and preimage resistance, and more broadly for random oracle modeling (barring length extension attacks on SHA-256), meaning that the outputs on distinct inputs can reasonably be modeled as independently uniformly distributed.
In the dark ages of the early '90s, when the United States still banned the export of cryptography as a munition, the ban covered encryption, per se, like DES—but had an explicit exception for authentication (22 CFR §121.1(XIII)(b)(1)(vi), since rescinded), and so the hash function Snefru was allowed to be published and exported.
A grad student named Dan pointed out in 1992 that you could use Snefru as a subroutine in an otherwise cryptography-free program to encrypt messages. When he informed the United States Department of State of his remarkable discovery, and asked them to confirm his understanding that publishing his cryptography-free program together with the exempted Snefru would not fall afoul of the export controls, they were not amused.
The State Department's lack of humor led to a nearly decade-long court battle, Bernstein v. United States, about whether the regulations in 22 CFR §§120–130 and 15 CFR §§730–744 prohibiting the export of encryption software constitute prior restraint in violation of the First Amendment to the United States Constitution. Eventually, the United States federal government, backed into a corner by an annoying grad student, relaxed the regulations and the case was dismissed.
Today, a newer incarnation of the same idea—using a hash function, ChaCha, together with a method inspired by the advanced technology of the one-time pad (also known in some circles as ‘xor’), to encrypt messages—protects the confidentiality of probably petabytes of data daily on the internet, in the form of the ChaCha/Poly1305 cipher suites in TLS.
But collision resistance is neither necessary—indeed, it is well-known that ChaCha is not collision-resistant—nor sufficient—as Rumba20 and VSH show—for indistinguishability from uniform random, which is what one needs for, e.g., a one-time pad to get any security.
P.S. If you do use a hash function, e.g. generating the sequence of bits $H(k mathbin| 0)$, $H(k mathbin| 1)$, etc., and want to use that sequence of bits to choose an integer $x$ with $0 leq x < n$ uniformly at random, make sure to do rejection sampling to avoid modulo bias if $n$ is not a power of two: if $H$ returns a string of $t$ bits interpreted as a $t$-bit integer, and $H(k mathbin| i)$ is below $2^t bmod n$, reject it and try $i + 1$; otherwise return $H(k mathbin| i) bmod n$
$endgroup$
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
add a comment |
$begingroup$
Define $H(x) = operatorname{SHA-256}(x) mathbin| 1$; that is, append a single 1 bit to SHA-256. Can you find a collision under $H$? Does $H$ have anything resembling uniform distribution?
This counterexample is not merely pathological; designs like Rumba20 and VSH provide collision resistance but neither preimage resistance nor uniformity.
That said, typical ‘cryptographic hash functions’ like SHA-256, BLAKE2b, and SHAKE128 are designed for collision resistance and preimage resistance, and more broadly for random oracle modeling (barring length extension attacks on SHA-256), meaning that the outputs on distinct inputs can reasonably be modeled as independently uniformly distributed.
In the dark ages of the early '90s, when the United States still banned the export of cryptography as a munition, the ban covered encryption, per se, like DES—but had an explicit exception for authentication (22 CFR §121.1(XIII)(b)(1)(vi), since rescinded), and so the hash function Snefru was allowed to be published and exported.
A grad student named Dan pointed out in 1992 that you could use Snefru as a subroutine in an otherwise cryptography-free program to encrypt messages. When he informed the United States Department of State of his remarkable discovery, and asked them to confirm his understanding that publishing his cryptography-free program together with the exempted Snefru would not fall afoul of the export controls, they were not amused.
The State Department's lack of humor led to a nearly decade-long court battle, Bernstein v. United States, about whether the regulations in 22 CFR §§120–130 and 15 CFR §§730–744 prohibiting the export of encryption software constitute prior restraint in violation of the First Amendment to the United States Constitution. Eventually, the United States federal government, backed into a corner by an annoying grad student, relaxed the regulations and the case was dismissed.
Today, a newer incarnation of the same idea—using a hash function, ChaCha, together with a method inspired by the advanced technology of the one-time pad (also known in some circles as ‘xor’), to encrypt messages—protects the confidentiality of probably petabytes of data daily on the internet, in the form of the ChaCha/Poly1305 cipher suites in TLS.
But collision resistance is neither necessary—indeed, it is well-known that ChaCha is not collision-resistant—nor sufficient—as Rumba20 and VSH show—for indistinguishability from uniform random, which is what one needs for, e.g., a one-time pad to get any security.
P.S. If you do use a hash function, e.g. generating the sequence of bits $H(k mathbin| 0)$, $H(k mathbin| 1)$, etc., and want to use that sequence of bits to choose an integer $x$ with $0 leq x < n$ uniformly at random, make sure to do rejection sampling to avoid modulo bias if $n$ is not a power of two: if $H$ returns a string of $t$ bits interpreted as a $t$-bit integer, and $H(k mathbin| i)$ is below $2^t bmod n$, reject it and try $i + 1$; otherwise return $H(k mathbin| i) bmod n$
$endgroup$
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
add a comment |
$begingroup$
Define $H(x) = operatorname{SHA-256}(x) mathbin| 1$; that is, append a single 1 bit to SHA-256. Can you find a collision under $H$? Does $H$ have anything resembling uniform distribution?
This counterexample is not merely pathological; designs like Rumba20 and VSH provide collision resistance but neither preimage resistance nor uniformity.
That said, typical ‘cryptographic hash functions’ like SHA-256, BLAKE2b, and SHAKE128 are designed for collision resistance and preimage resistance, and more broadly for random oracle modeling (barring length extension attacks on SHA-256), meaning that the outputs on distinct inputs can reasonably be modeled as independently uniformly distributed.
In the dark ages of the early '90s, when the United States still banned the export of cryptography as a munition, the ban covered encryption, per se, like DES—but had an explicit exception for authentication (22 CFR §121.1(XIII)(b)(1)(vi), since rescinded), and so the hash function Snefru was allowed to be published and exported.
A grad student named Dan pointed out in 1992 that you could use Snefru as a subroutine in an otherwise cryptography-free program to encrypt messages. When he informed the United States Department of State of his remarkable discovery, and asked them to confirm his understanding that publishing his cryptography-free program together with the exempted Snefru would not fall afoul of the export controls, they were not amused.
The State Department's lack of humor led to a nearly decade-long court battle, Bernstein v. United States, about whether the regulations in 22 CFR §§120–130 and 15 CFR §§730–744 prohibiting the export of encryption software constitute prior restraint in violation of the First Amendment to the United States Constitution. Eventually, the United States federal government, backed into a corner by an annoying grad student, relaxed the regulations and the case was dismissed.
Today, a newer incarnation of the same idea—using a hash function, ChaCha, together with a method inspired by the advanced technology of the one-time pad (also known in some circles as ‘xor’), to encrypt messages—protects the confidentiality of probably petabytes of data daily on the internet, in the form of the ChaCha/Poly1305 cipher suites in TLS.
But collision resistance is neither necessary—indeed, it is well-known that ChaCha is not collision-resistant—nor sufficient—as Rumba20 and VSH show—for indistinguishability from uniform random, which is what one needs for, e.g., a one-time pad to get any security.
P.S. If you do use a hash function, e.g. generating the sequence of bits $H(k mathbin| 0)$, $H(k mathbin| 1)$, etc., and want to use that sequence of bits to choose an integer $x$ with $0 leq x < n$ uniformly at random, make sure to do rejection sampling to avoid modulo bias if $n$ is not a power of two: if $H$ returns a string of $t$ bits interpreted as a $t$-bit integer, and $H(k mathbin| i)$ is below $2^t bmod n$, reject it and try $i + 1$; otherwise return $H(k mathbin| i) bmod n$
$endgroup$
Define $H(x) = operatorname{SHA-256}(x) mathbin| 1$; that is, append a single 1 bit to SHA-256. Can you find a collision under $H$? Does $H$ have anything resembling uniform distribution?
This counterexample is not merely pathological; designs like Rumba20 and VSH provide collision resistance but neither preimage resistance nor uniformity.
That said, typical ‘cryptographic hash functions’ like SHA-256, BLAKE2b, and SHAKE128 are designed for collision resistance and preimage resistance, and more broadly for random oracle modeling (barring length extension attacks on SHA-256), meaning that the outputs on distinct inputs can reasonably be modeled as independently uniformly distributed.
In the dark ages of the early '90s, when the United States still banned the export of cryptography as a munition, the ban covered encryption, per se, like DES—but had an explicit exception for authentication (22 CFR §121.1(XIII)(b)(1)(vi), since rescinded), and so the hash function Snefru was allowed to be published and exported.
A grad student named Dan pointed out in 1992 that you could use Snefru as a subroutine in an otherwise cryptography-free program to encrypt messages. When he informed the United States Department of State of his remarkable discovery, and asked them to confirm his understanding that publishing his cryptography-free program together with the exempted Snefru would not fall afoul of the export controls, they were not amused.
The State Department's lack of humor led to a nearly decade-long court battle, Bernstein v. United States, about whether the regulations in 22 CFR §§120–130 and 15 CFR §§730–744 prohibiting the export of encryption software constitute prior restraint in violation of the First Amendment to the United States Constitution. Eventually, the United States federal government, backed into a corner by an annoying grad student, relaxed the regulations and the case was dismissed.
Today, a newer incarnation of the same idea—using a hash function, ChaCha, together with a method inspired by the advanced technology of the one-time pad (also known in some circles as ‘xor’), to encrypt messages—protects the confidentiality of probably petabytes of data daily on the internet, in the form of the ChaCha/Poly1305 cipher suites in TLS.
But collision resistance is neither necessary—indeed, it is well-known that ChaCha is not collision-resistant—nor sufficient—as Rumba20 and VSH show—for indistinguishability from uniform random, which is what one needs for, e.g., a one-time pad to get any security.
P.S. If you do use a hash function, e.g. generating the sequence of bits $H(k mathbin| 0)$, $H(k mathbin| 1)$, etc., and want to use that sequence of bits to choose an integer $x$ with $0 leq x < n$ uniformly at random, make sure to do rejection sampling to avoid modulo bias if $n$ is not a power of two: if $H$ returns a string of $t$ bits interpreted as a $t$-bit integer, and $H(k mathbin| i)$ is below $2^t bmod n$, reject it and try $i + 1$; otherwise return $H(k mathbin| i) bmod n$
edited May 23 at 22:16
answered May 21 at 5:33
Squeamish OssifrageSqueamish Ossifrage
30.4k1 gold badge51 silver badges131 bronze badges
30.4k1 gold badge51 silver badges131 bronze badges
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
add a comment |
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
$begingroup$
Indeed. The property people often want when they reach for a cryptographic hash function is indifferentiability from a random oracle, which covers a lot of bases.
$endgroup$
– Bristol
May 22 at 19:28
add a comment |
$begingroup$
No, but high collision resistance per bit has an influence.
Non-uniformity -> less entropy -> weakned collision resistance.
As keysize is significant factor: most cryptographic hash functions have uniform output given entropic input. Using a hash (or encryption) routine to make a stream of random numbers from a single block of entropy is an established practice. There are lists of recommended primatives and pitfalls etc.
From wikipedia:
"A cryptographically secure hash of a counter might also act as a good CSPRNG in some cases. In this case, it is also necessary that the initial value of this counter is random and secret. However, there has been little study of these algorithms for use in this manner, and at least some authors warn against this use."
$endgroup$
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
add a comment |
$begingroup$
No, but high collision resistance per bit has an influence.
Non-uniformity -> less entropy -> weakned collision resistance.
As keysize is significant factor: most cryptographic hash functions have uniform output given entropic input. Using a hash (or encryption) routine to make a stream of random numbers from a single block of entropy is an established practice. There are lists of recommended primatives and pitfalls etc.
From wikipedia:
"A cryptographically secure hash of a counter might also act as a good CSPRNG in some cases. In this case, it is also necessary that the initial value of this counter is random and secret. However, there has been little study of these algorithms for use in this manner, and at least some authors warn against this use."
$endgroup$
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
add a comment |
$begingroup$
No, but high collision resistance per bit has an influence.
Non-uniformity -> less entropy -> weakned collision resistance.
As keysize is significant factor: most cryptographic hash functions have uniform output given entropic input. Using a hash (or encryption) routine to make a stream of random numbers from a single block of entropy is an established practice. There are lists of recommended primatives and pitfalls etc.
From wikipedia:
"A cryptographically secure hash of a counter might also act as a good CSPRNG in some cases. In this case, it is also necessary that the initial value of this counter is random and secret. However, there has been little study of these algorithms for use in this manner, and at least some authors warn against this use."
$endgroup$
No, but high collision resistance per bit has an influence.
Non-uniformity -> less entropy -> weakned collision resistance.
As keysize is significant factor: most cryptographic hash functions have uniform output given entropic input. Using a hash (or encryption) routine to make a stream of random numbers from a single block of entropy is an established practice. There are lists of recommended primatives and pitfalls etc.
From wikipedia:
"A cryptographically secure hash of a counter might also act as a good CSPRNG in some cases. In this case, it is also necessary that the initial value of this counter is random and secret. However, there has been little study of these algorithms for use in this manner, and at least some authors warn against this use."
edited May 23 at 8:52
answered May 21 at 14:55
n0n3n0n3
392 bronze badges
392 bronze badges
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
add a comment |
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
$begingroup$
How do you draw these implications? The most prevalent ciphers today—AES-GCM and ChaCha/Poly1305—use a ‘hash function’ of a counter, inspired in part because the United States export controls on cryptography as a munition did not cover the hash function Snefru. But neither AES nor ChaCha provide collision resistance. Conversely, Rumba20 and VSH, designed for collision resistance, do not provide preimage resistance or uniformity. GHASH has uniformity and low collision probability, but no collision resistance. ‘Hash function’ means many things; the question was about specific properties.
$endgroup$
– Squeamish Ossifrage
May 21 at 15:47
add a comment |
$begingroup$
Only my 2 cents, you don't necessarily need an (heavy) hash function to generate a sequence or a series of pseudo-random numbers.
A practical, low-level way, to generate an uniform distribution of values, in a specific range, could be to use a simple acceptance/rejection method, combined with pseudo-random generation of numbers:
if we want $k$ different output values, then our range will be $[0, k-1] $
use a dumb 0/1- generator, like Math.random for js, a refined quantum random input source, or simply flip a coin $r$ times, for generating a pseudo-random sequence of $r$ bits, where:
- $r = roundup( log_{2}(k) )$
the resulting value $n$ is in the interval $ [0, 2^{r}-1]$
if $( n < k )$, pick and output the number (no modulo operation)
otherwise, discard the number and regenerate a new pseudo-random sequence/number.
Obviously, we will consume more random input data from our source, when $k<<2^{r}$, because we expect much more rejections, but using the modulo is not a good solution in this case, it will not give us a uniform distribution.
The best scenario occurs when $k$ is a power of $2$:
$k = 2^{i}$, for a generic natural $i$
- we consume precisely $1$-bit of input data to produce $1$-bit of output
- no waste of time
The worst scenario for rejections occurs when:
- $k = (2^{r-1}+1)$
$(k - 1)= 2^{r-1}$- $r = 1 + log_{2}(k - 1)$
- we'll consume ~ $2$-bit of input data to produce $1$-bit of output
- we'll spend much more time in the generation process
For example, if we need, for a chosen interval $[0, k-1]$:
$r = 10$ bits to generate at least $k$ different values, then:- when $k = ( 2^{10-1} + 1 ) = 513$
- we expect to sacrifice, on average, about ~$50%$ of our pseudo-random input data, discarding all output numbers $> 513$, when they occur.
- a total of $(2^{10} - 513 ) = 511$ values
$endgroup$
add a comment |
$begingroup$
Only my 2 cents, you don't necessarily need an (heavy) hash function to generate a sequence or a series of pseudo-random numbers.
A practical, low-level way, to generate an uniform distribution of values, in a specific range, could be to use a simple acceptance/rejection method, combined with pseudo-random generation of numbers:
if we want $k$ different output values, then our range will be $[0, k-1] $
use a dumb 0/1- generator, like Math.random for js, a refined quantum random input source, or simply flip a coin $r$ times, for generating a pseudo-random sequence of $r$ bits, where:
- $r = roundup( log_{2}(k) )$
the resulting value $n$ is in the interval $ [0, 2^{r}-1]$
if $( n < k )$, pick and output the number (no modulo operation)
otherwise, discard the number and regenerate a new pseudo-random sequence/number.
Obviously, we will consume more random input data from our source, when $k<<2^{r}$, because we expect much more rejections, but using the modulo is not a good solution in this case, it will not give us a uniform distribution.
The best scenario occurs when $k$ is a power of $2$:
$k = 2^{i}$, for a generic natural $i$
- we consume precisely $1$-bit of input data to produce $1$-bit of output
- no waste of time
The worst scenario for rejections occurs when:
- $k = (2^{r-1}+1)$
$(k - 1)= 2^{r-1}$- $r = 1 + log_{2}(k - 1)$
- we'll consume ~ $2$-bit of input data to produce $1$-bit of output
- we'll spend much more time in the generation process
For example, if we need, for a chosen interval $[0, k-1]$:
$r = 10$ bits to generate at least $k$ different values, then:- when $k = ( 2^{10-1} + 1 ) = 513$
- we expect to sacrifice, on average, about ~$50%$ of our pseudo-random input data, discarding all output numbers $> 513$, when they occur.
- a total of $(2^{10} - 513 ) = 511$ values
$endgroup$
add a comment |
$begingroup$
Only my 2 cents, you don't necessarily need an (heavy) hash function to generate a sequence or a series of pseudo-random numbers.
A practical, low-level way, to generate an uniform distribution of values, in a specific range, could be to use a simple acceptance/rejection method, combined with pseudo-random generation of numbers:
if we want $k$ different output values, then our range will be $[0, k-1] $
use a dumb 0/1- generator, like Math.random for js, a refined quantum random input source, or simply flip a coin $r$ times, for generating a pseudo-random sequence of $r$ bits, where:
- $r = roundup( log_{2}(k) )$
the resulting value $n$ is in the interval $ [0, 2^{r}-1]$
if $( n < k )$, pick and output the number (no modulo operation)
otherwise, discard the number and regenerate a new pseudo-random sequence/number.
Obviously, we will consume more random input data from our source, when $k<<2^{r}$, because we expect much more rejections, but using the modulo is not a good solution in this case, it will not give us a uniform distribution.
The best scenario occurs when $k$ is a power of $2$:
$k = 2^{i}$, for a generic natural $i$
- we consume precisely $1$-bit of input data to produce $1$-bit of output
- no waste of time
The worst scenario for rejections occurs when:
- $k = (2^{r-1}+1)$
$(k - 1)= 2^{r-1}$- $r = 1 + log_{2}(k - 1)$
- we'll consume ~ $2$-bit of input data to produce $1$-bit of output
- we'll spend much more time in the generation process
For example, if we need, for a chosen interval $[0, k-1]$:
$r = 10$ bits to generate at least $k$ different values, then:- when $k = ( 2^{10-1} + 1 ) = 513$
- we expect to sacrifice, on average, about ~$50%$ of our pseudo-random input data, discarding all output numbers $> 513$, when they occur.
- a total of $(2^{10} - 513 ) = 511$ values
$endgroup$
Only my 2 cents, you don't necessarily need an (heavy) hash function to generate a sequence or a series of pseudo-random numbers.
A practical, low-level way, to generate an uniform distribution of values, in a specific range, could be to use a simple acceptance/rejection method, combined with pseudo-random generation of numbers:
if we want $k$ different output values, then our range will be $[0, k-1] $
use a dumb 0/1- generator, like Math.random for js, a refined quantum random input source, or simply flip a coin $r$ times, for generating a pseudo-random sequence of $r$ bits, where:
- $r = roundup( log_{2}(k) )$
the resulting value $n$ is in the interval $ [0, 2^{r}-1]$
if $( n < k )$, pick and output the number (no modulo operation)
otherwise, discard the number and regenerate a new pseudo-random sequence/number.
Obviously, we will consume more random input data from our source, when $k<<2^{r}$, because we expect much more rejections, but using the modulo is not a good solution in this case, it will not give us a uniform distribution.
The best scenario occurs when $k$ is a power of $2$:
$k = 2^{i}$, for a generic natural $i$
- we consume precisely $1$-bit of input data to produce $1$-bit of output
- no waste of time
The worst scenario for rejections occurs when:
- $k = (2^{r-1}+1)$
$(k - 1)= 2^{r-1}$- $r = 1 + log_{2}(k - 1)$
- we'll consume ~ $2$-bit of input data to produce $1$-bit of output
- we'll spend much more time in the generation process
For example, if we need, for a chosen interval $[0, k-1]$:
$r = 10$ bits to generate at least $k$ different values, then:- when $k = ( 2^{10-1} + 1 ) = 513$
- we expect to sacrifice, on average, about ~$50%$ of our pseudo-random input data, discarding all output numbers $> 513$, when they occur.
- a total of $(2^{10} - 513 ) = 511$ values
edited Jun 7 at 12:21
answered Jun 3 at 23:18
assobucoassobuco
415 bronze badges
415 bronze badges
add a comment |
add a comment |
Thanks for contributing an answer to Cryptography Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcrypto.stackexchange.com%2fquestions%2f70707%2fcan-we-assume-that-a-hash-function-with-high-collision-resistance-also-means-a-h%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
$begingroup$
Why cannot you use a stream cipher? That would be more efficient: stream ciphers are faster than hash functions. They also produce an output indistinguishable from truly random data.
$endgroup$
– A. Hersean
May 21 at 15:09
1
$begingroup$
If you have a billion possibilities and are generating a million values, if those values are in the first million slots, is that uniformly distributed? Even if the likelihood of collision for the next generated value is not likely to use one of the previously used values, I’d argue that it is low risk of collision, but not uniform in disparity.
$endgroup$
– vol7ron
May 21 at 22:00
1
$begingroup$
Not an answer (@Squeamish Ossifrage has already answered your headline question), but to address your first paragraph: yes, concrete cryptographic hash functions can be and are used to generate streams of random data (which can be for cryptographic purposes when seeded appropriately, so should meet your "seemingly random" requirement). See, for example, Hash_DRBG in the NIST recommendations. But look for a good implementation - don't go re-inventing the wheel!
$endgroup$
– user7761803
May 22 at 7:38
1
$begingroup$
This seems like an X–Y problem: If you need to produce uniformly distributed random numbers, don’t use a hash function, use a (CS)PRNG, possibly in conjunction with a uniform distribution derivation function.
$endgroup$
– Konrad Rudolph
May 22 at 8:53
1
$begingroup$
If you don't care about the cryptographic properties, then maybe take a look at the Halton sequence. This is useful for numeric sampling algorithms (not super highly-dimensional ones tho!) as a pseudo-random number sequence that very uniformly covers the sampling space.
$endgroup$
– Elaskanator
May 22 at 19:09