Project Wycheproof

This page describes the goals and strategies of project Wycheproof. See
README for an introduction to the project.

Defense in depth

There are a number of tests where we check for expected behaviour
rather than exploitability. Examples:

• default values: we expect that default values are reasonable and correspond
  to recommendations by current standards. Concretely, in 2016 it is not OK
  if an RSA key generation uses 1024 bits as default or digital signatures
  use SHA-1 as default.
• timing attacks: any timing that relation between keys (or other sensitive)
  data and the measured time fails the test. However tests are set up
  such that too much noise during the test can prevent that a relation
  is detected.
• wrong exceptions: The JCE interface often specifies the exceptions that
  should be thrown when the input is invalid. We expect the specified
  exceptions in the tests.
• leaking information through exceptions: While it is a good practice to not
  return detailed logs to a sender, we consider text in exceptions as
  information that a potential attacker can learn. For example padding
  failures during decryption should not contain information about the
  reason why a decryption failed.
• RSA PKCS #1 signatures: If a signature verification allows signatures
  with lots of modifications, then RSA signatures can be forged for small
  public exponents. Tests do not measure how many bytes can be modified.
  Any accepted modification of the PKCS #1 padding fails the test.

Compatibility between providers

One of the goals of Wycheproof is to test for compatibility issues.
Switching JCE providers should not introduce vulnerabilities simply because
the solution was developed by another provider.

An example for this was the following observation: When using AES-GCM then
javax.crypto.CipherInputStream worked sort of with JCE and
org.bouncycastle.jcajce.io.CipherInputStream.java worked with BouncyCastle.
However, authentication was skipped in some cases when
javax.crypto.CipherInputStream was used with BouncyCastle.

Comparing cryptographic libraries is not a primary goal

Because of the strategies mentioned above we expect that a comparison of
cryptographic libraries based on the bugs found would be biased:

• Libraries used internally in Google get more attention.
  Serious vulnerabilities in these libraries should be fixed at the time the
  tests are added to Wycheproof. On the other hand it is also likely that
  tests find a larger number of bugs in these libraries when old versions are
  tested.
• Tests often check for expected behaviour and compatibility.
  Expected behaviour is often defined by a prominent library.
  Pointing out such problems can therefore penalize smaller third party
  libraries.
• We are working toward covering as many potential vulnerabilities as possible
  with test vectors, because this simplifies porting the tests to other
  languages or interfaces. Thus a single test case can cover multiple
  vulnerabilities.

We are not trying to remove this bias when this interferes with more important
goals such as early reporting.
Hence we are reluctant to publish comparisons.

Thoughts on the design of cryptographic libraries

We should promote robust interfaces with the goal to simplify
the use of the library, code reviews of applications using the
library and testing the library.

• When cryptographic primitives require randomness then the random
  numbers should be chosen by the library. It shouldn't be possible
  for a user to provide randomness. If the library itself chooses the
  randomness then it is possible (at least to some degree) to check
  that the random number generation is appropriate for the primitive.
  If the user can provide the randomness then it is not possible to
  catch this in our tests.