keccak - ritark/winter2012/482/team/u5/report.pdf · description of the algorithm keccak is the...

KeccakFinal PaperTeam Bletchley

Chris BentivengaFrederick Christie

Michael Kitson

Description of the AlgorithmKeccak is the winner of NIST hash function competition, and is the accepted standard for SHA-3. Designed by Guido Bertoni, Joan Daemen, Michaël Peeteres and Gilles Van Assche, it is based on the sponge construction. Keccak is particularly good for hardware implementations. Of all the NIST finalists, it had the best running time in hardware.

Sponge Construction Keccak is a member of the sponge function family. This gives the function many of its attributes, including a good number of its strengths. Perhaps the most notable is that a sponge construction allows for variable length input and specified length output. The construction works in two phases: absorption and squeezing. Throughout the process, state is kept tracked through b=r+c bits, where r is the bitrate and c is the capacity (only modified in the round function). The first phase is absorbing. The construction takes r bits (the bitrate) bits, XORs them into the first r bits of state, and passes them through the f function. This function is specific to each cypher. From there the squeezing phase takes the first r bits of state, applying f again through the process. The number of blocks output from this processes is the value chosen by the user.

1

Sponge StateThe sponge state used is described in terms of lanes. The state is a five by five (5 by 5) array of lanes, where each lane is a specified length (ℓ). The suggested ℓ is 6.

1 http://sponge.noekeon.org/

Round FunctionzThe function f referred to by the sponge construction is the iteration of the Keccak specified round function. Keccak specified five different portions of the round function:

● Theta: The combination of the state columns.● Rho: Bitwise circular rotation of each lane.● Pi: Reordering of the lanes● Chi: Combination of state rows● Iota: LFSR-based round constant xored into state[0][0]

Strengths and WeaknesesKeccak is a very secure function. Its has been proved secure against generic attacks. It also has a cascading effect, where simple changes to the input message result in huge variances in the hash. Keccak has excellent performance in hardware implementations, and has very strong support for parallel implementations. While the slowest of the NIST finalists in software, it still boasts one of the smallest profiles, having a small implementation, and more than adequate performance in real world scenarios.

Example I/O$ ./HashSHA3 [digestSize] <message> $ ./HashSHA3 512 'The quick brown fox jumps over the lazy dog'Performing SHA3-512 on: 'The quick brown fox jumps over the lazy dog'd135bb84d0439dbac432247ee573a23ea7d3c9deb2a968eb31d47c4fb45f1ef4422d6c531b5b9bd6f449ebcc449ea94d0a8f05f62130fda612da53c79659f609

Original

Source Code#include <cassert>#include <cstring>#include <iostream>#include "SHA3.h"

// Circular rotate left

#define ROT_L( X, Y ) (( X << Y ) | ( X >> (64 - Y) ))#define ROUNDS 24

/// For converting binary output to hexidecimal for printing

const char *hexLookup = "0123456789abcdef";

const keccakLane_t roundConstants[] = { 0x0000000000000001, 0x0000000000008082, 0x800000000000808A, 0x8000000080008000, 0x000000000000808B, 0x0000000080000001, 0x8000000080008081, 0x8000000000008009, 0x000000000000008A, 0x0000000000000088, 0x0000000080008009, 0x000000008000000A, 0x000000008000808B, 0x800000000000008B, 0x8000000000008089, 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, 0x000000000000800A, 0x800000008000000A, 0x8000000080008081, 0x8000000000008080, 0x0000000080000001, 0x8000000080008008};

int rotationOffsets[5][5] = { { 0, 36, 3, 41, 18}, { 1, 44, 10, 45, 2}, {62, 6, 43, 15, 61}, {28, 55, 25, 21, 56}, {27, 20, 39, 8, 14}};

SHA3::SHA3( int digestSize ) : _digestSize( digestSize ){ // zero the state _spongeCapacity = 2 * 8 * _digestSize; _spongeRate = 1600 - _spongeCapacity; _messageBuffer = new unsigned char[_spongeRate]; _reset();}

////////// Accessors //////////

int SHA3::digestSize(){ return _digestSize;}

////////// Ingesting Data //////////

void SHA3::hash( const int b ){ _bufferLocation[0] = (unsigned char)b; _bufferLocation++; if( _bufferLocation == &_messageBuffer[_spongeRate/8] ){ _bufferLocation = _messageBuffer; _absorbBuffer(); }}

void SHA3::hashString( const char *str ){ int byte = 0; while( str[byte] != '\0' ){ hash( (int)( (unsigned char) str[byte] ) ); byte++; }}

void SHA3::hashHexString( const char *str ){ int byte = 0; while( str[byte] != '\0' ){ int f = str[byte];

int s = str[byte+1]; if( f >= 97 ) f -= 87; // lowercase else if( f >= 65 ) f -= 55; // uppercase else f -= 48; // numeric

if( s >= 97 ) s -= 87; // lowercase else if( s >= 65 ) s -= 55; // uppercase else s -= 48; // numeric

hash( (f << 4) | s ); byte+=2; }}

////////// Expelling Data //////////

void SHA3::digest( unsigned char d[] ){ // Pad with 10*1 padding _bufferLocation[0] = 1; _bufferLocation++; while( _bufferLocation != &_messageBuffer[_spongeRate/8] ){ _bufferLocation[0] = 0; _bufferLocation++; } _messageBuffer[_spongeRate/8 - 1] |= 0x80; _absorbBuffer();

// Squeeze memcpy( d, _state, digestSize() ); _reset(); // Ready the function to hash another message}

char *SHA3::digestInHex(){ unsigned char *bytes = new unsigned char[ digestSize() ]; char *hex = new char[ 2 * digestSize() + 1 ]; hex[2*digestSize()] = '\0'; digest( bytes );

for( int byte = 0; byte < digestSize(); byte++ ){ hex[2*byte] = hexLookup[bytes[byte] >> 4]; hex[2*byte+1] = hexLookup[bytes[byte] & 15]; } delete( bytes ); assert( hex[2*digestSize()] == '\0' ); return hex;

}

////////// Internals //////////

void SHA3::_reset(){ for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ _state[x][y] = 0; } } _bufferLocation = _messageBuffer;}

void SHA3::_absorbBuffer(){ keccakLane_t *x = (keccakLane_t *)_messageBuffer; for( int i = 0; i*64 < _spongeRate; i++ ){ _state[i/5][i%5] |= x[i]; } _performRounds( ROUNDS );}

void SHA3::_performRounds( int rounds ){ keccakLane_t b[5][5]; keccakLane_t c[5]; keccakLane_t d[5];

for( int i = 0; i < rounds; i++ ){ // Theta step for( int x = 0; x < 5; x++ ){ c[x] = _state[0][x] ^ _state[1][x] ^ _state[2][x] ^ _state[3][x] ^ _state[4][x]; } for( int x = 0; x < 5; x++ ){ d[x] = c[(x+4) % 5] ^ ROT_L( c[(x+1) % 5], 1); } for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ _state[x][y] ^= d[y]; } } // Rho and Pi steps for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ b[y][(2*x + 3*y) % 5] = ROT_L( _state[y][x], rotationOffsets[x][y] );

} } // Chi step for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ _state[y][x] = b[x][y] ^ ((~b[(x+1) % 5][y]) & b[(x+2) % 5][y]); } } // Iota step _state[0][0] ^= roundConstants[i]; }}

////////// Debugging Functions //////////

void SHA3::_printMessageBuffer(){ std::cout << "mb = [ "; for( int i = 0; i < _spongeRate/8; i++ ){ std::cout << (int)_messageBuffer[i] << " "; } std::cout << "]" << std::endl;}

void SHA3::_printSponge(){ std::cout << "s = [ " << std::hex; for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ std::cout << _state[x][y] << " "; } } std::cout << std::dec << "]" << std::endl;}

Running Time

Intel i7-2677 Xeon X5560 (glados.cs.rit.edu)

2.1GB of zeros 9m45.061s 13m41.416s

2.1GB of zeros with -O3 2m15.476s 3m40.460s

3500000 8K messages (28 GB) of pseudorandom data 17m45.757s 24m37.451s

3500000 8K messages (28 GB) of pseudorandom data with - 3m59.650s 6m40.771s

O3

Analysis of MeasurementsAt the most basic level, the unoptimized running time shows us that optimizations need to be performed to make the implementation production ready. Because our implementation is created in C++ and compiled with GCC, one of the simplest performance increases that can be performed is compiler optimization flags. The simple tradeoff with compiler optimization is decreased running time at the expense of compilation and loss of debugging abilities. Another standard C++ optimization is reducing the call stack depth. This is most commonly done by replacing the use of loops with the equivalent, non-looped statements. In the unoptimized implementation, there are quite a few loops with predefined sizes. While these are much easier to read and describe pseudo code (the representation used in the specification), they add to the load the computer in a way a programer can optimize for. The call stack depth can also be reduced by use of the inline keyword. Inline instructs the compiler to insert a complete copy of the function body wherever the function is used in code. This means that the call stack is not increased, because in execution the code is executed at the same stack level. However, inlining does increase the function body size. If the function bloats too much because of this, it is possible that the function may not fit into cache. This would cause cache misses and calling out to other memory, decreasing performances. It also

increases the size of the binary. Other small, standard C++ issues can be checked. For example, removal of memory leaks obviously reduces the load. For additional analysis, Xcode’s Instruments2 and the OS X program sample3 were used to analyze the impact of individual functions on the runtime.

2 https://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html3 https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/dscl.1.html

Revised

Source Code#include <cstring>#include <iostream>#include "SHA3.h"

// Circular rotate left

#define ROT_L( X, Y ) (( X << Y ) | ( X >> (64 - Y) ))#define ROUNDS 24

/// For converting binary output to hexidecimal for printing

const char *hexLookup = "0123456789abcdef";

const keccakLane_t roundConstants[] = { 0x0000000000000001, 0x0000000000008082, 0x800000000000808A, 0x8000000080008000, 0x000000000000808B, 0x0000000080000001, 0x8000000080008081, 0x8000000000008009, 0x000000000000008A, 0x0000000000000088, 0x0000000080008009, 0x000000008000000A, 0x000000008000808B, 0x800000000000008B, 0x8000000000008089, 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, 0x000000000000800A, 0x800000008000000A, 0x8000000080008081, 0x8000000000008080, 0x0000000080000001, 0x8000000080008008};

SHA3::SHA3( int digestSize ) : _digestSize( digestSize ){

// zero the state // CHANGE: Now uses bit shifting instead of multiplication _spongeCapacity = _digestSize << 4; _spongeRate = 1600 - _spongeCapacity; _messageBuffer = new unsigned char[_spongeRate]; _reset();}

SHA3::~SHA3(){ // CHANGE: Deconstructor included delete( _messageBuffer );}////////// Accessors //////////

int SHA3::digestSize(){ return _digestSize;}

////////// Ingesting Data //////////

void SHA3::hash( const int b ){ _bufferLocation[0] = (unsigned char)b; _bufferLocation++; if( _bufferLocation == &_messageBuffer[_spongeRate>>3] ){ _bufferLocation = _messageBuffer; _absorbBuffer(); }}

void SHA3::hashString( const char *str ){ int byte = 0; while( str[byte] != '\0' ){ hash( (int)( (unsigned char) str[byte] ) ); byte++; }}

void SHA3::hashHexString( const char *str ){ int byte = 0; while( str[byte] != '\0' ){ int f = str[byte]; int s = str[byte+1]; if( f >= 97 ) f -= 87; // lowercase else if( f >= 65 ) f -= 55; // uppercase else f -= 48; // numeric

if( s >= 97 ) s -= 87; // lowercase else if( s >= 65 ) s -= 55; // uppercase else s -= 48; // numeric

hash( (f << 4) | s ); byte+=2; }}

////////// Expelling Data //////////

void SHA3::digest( unsigned char d[] ){ // Pad with 10*1 padding _bufferLocation[0] = 1; _bufferLocation++; // CHANGE: Uses system bzero function instead of while loop to initilize

bzero( _bufferLocation, &_messageBuffer[_spongeRate>>3] - _bufferLocation ); _messageBuffer[(_spongeRate >> 3) - 1] |= 0x80; _absorbBuffer();

// Squeeze memcpy( d, _state, digestSize() ); _reset(); // Ready the function to hash another message}

char *SHA3::digestInHex(){ unsigned char *bytes = new unsigned char[ digestSize() ]; char *hex = new char[ (digestSize() << 1) + 1 ];

// CHANGE: Uses bitshifting instead of multiplication hex[digestSize() << 1] = '\0'; digest( bytes );

for( int byte = 0; byte < digestSize(); byte++ ){ // CHANGE: Uses bitshifting instead of multiplication hex[byte << 1] = hexLookup[bytes[byte] >> 4]; hex[(byte << 1)+1] = hexLookup[bytes[byte] & 15]; } delete( bytes ); return hex;}

////////// Internals //////////

inline void SHA3::_reset(){ // CHANGE: Uses system bzero function instead of while loop to initilize

bzero( _state, 200 ); //25 64-byte lanes _bufferLocation = _messageBuffer;}

void SHA3::_absorbBuffer(){ keccakLane_t *x = (keccakLane_t *)_messageBuffer; for( int i = 0; i*64 < _spongeRate; i++ ){ _state[i/5][i%5] |= x[i]; // TODO: unroll } _performRounds( ROUNDS );}

// CHANGE: Function changed to inline

inline void SHA3::_performRounds( int rounds ){ keccakLane_t b[5][5]; keccakLane_t c[5]; keccakLane_t d[5];

for( int i = 0; i < rounds; i++ ){

//CHANGE: For loops change to pre-determined steps, reduces call stack

// Theta step c[0] = _state[0][0] ^ _state[1][0] ^ _state[2][0] ^ _state[3][0] ^ _state[4][0]; c[1] = _state[0][1] ^ _state[1][1] ^ _state[2][1] ^ _state[3][1] ^ _state[4][1]; c[2] = _state[0][2] ^ _state[1][2] ^ _state[2][2] ^ _state[3][2] ^ _state[4][2]; c[3] = _state[0][3] ^ _state[1][3] ^ _state[2][3] ^ _state[3][3] ^ _state[4][3]; c[4] = _state[0][4] ^ _state[1][4] ^ _state[2][4] ^ _state[3][4] ^ _state[4][4];

d[0] = c[4] ^ ROT_L( c[1], 1 ); d[1] = c[0] ^ ROT_L( c[2], 1 ); d[2] = c[1] ^ ROT_L( c[3], 1 ); d[3] = c[2] ^ ROT_L( c[4], 1 ); d[4] = c[3] ^ ROT_L( c[0], 1 );

_state[0][0] ^= d[0]; _state[0][1] ^= d[1]; _state[0][2] ^= d[2]; _state[0][3] ^= d[3]; _state[0][4] ^= d[4]; _state[1][0] ^= d[0]; _state[1][1] ^= d[1]; _state[1][2] ^= d[2]; _state[1][3] ^= d[3]; _state[1][4] ^= d[4]; _state[2][0] ^= d[0]; _state[2][1] ^= d[1]; _state[2][2] ^= d[2]; _state[2][3] ^= d[3]; _state[2][4] ^= d[4]; _state[3][0] ^= d[0]; _state[3][1] ^= d[1]; _state[3][2] ^= d[2]; _state[3][3] ^= d[3]; _state[3][4] ^= d[4]; _state[4][0] ^= d[0]; _state[4][1] ^= d[1]; _state[4][2] ^= d[2]; _state[4][3] ^= d[3]; _state[4][4] ^= d[4];

// Rho and Pi steps b[0][0] = _state[0][0]; // rotate left by 0 bits b[1][3] = ROT_L( _state[1][0], 36 ); b[2][1] = ROT_L( _state[2][0], 3 ); b[3][4] = ROT_L( _state[3][0], 41 ); b[4][2] = ROT_L( _state[4][0], 18 );

b[0][2] = ROT_L( _state[0][1], 1 ); b[1][0] = ROT_L( _state[1][1], 44 ); b[2][3] = ROT_L( _state[2][1], 10 ); b[3][1] = ROT_L( _state[3][1], 45 ); b[4][4] = ROT_L( _state[4][1], 2 );




// Chi step _state[0][0] = b[0][0] ^ ((~b[1][0]) & b[2][0]); _state[1][0] = b[0][1] ^ ((~b[1][1]) & b[2][1]); _state[2][0] = b[0][2] ^ ((~b[1][2]) & b[2][2]); _state[3][0] = b[0][3] ^ ((~b[1][3]) & b[2][3]); _state[4][0] = b[0][4] ^ ((~b[1][4]) & b[2][4]);

_state[0][1] = b[1][0] ^ ((~b[2][0]) & b[3][0]); _state[1][1] = b[1][1] ^ ((~b[2][1]) & b[3][1]); _state[2][1] = b[1][2] ^ ((~b[2][2]) & b[3][2]); _state[3][1] = b[1][3] ^ ((~b[2][3]) & b[3][3]); _state[4][1] = b[1][4] ^ ((~b[2][4]) & b[3][4]);




// Iota step

_state[0][0] ^= roundConstants[i]; }}

////////// Debugging Functions //////////

void SHA3::_printMessageBuffer(){ std::cout << "mb = [ "; for( int i = 0; i < _spongeRate/8; i++ ){ std::cout << (int)_messageBuffer[i] << " "; } std::cout << "]" << std::endl;}

void SHA3::_printSponge(){ std::cout << "s = [ " << std::hex; for( int x = 0; x < 5; x++ ){ for( int y = 0; y < 5; y++ ){ std::cout << _state[x][y] << " "; } } std::cout << std::dec << "]" << std::endl;}

Running Time

Intel i7-2677 Xeon X5560 (glados.cs.rit.edu)

2.1GB of zeros 1m26.483s 4m31.554s

2.1GB of zeros with -O3 0m59.322s 2m19.864s

3500000 8K messages (28 GB) of pseudorandom data 2m37.313s 8m11.920s

3500000 8K messages (28 GB) of pseudorandom data with -O3

1m36.851s 4m15.848s

Analysis of MeasurementsThe analysis of the measurements show very clearly that the changes drastically improve performance on the project. In the first case alone, the final runtime was 14.778% of the original. Across every field there was improvement. This shows two major points. Firstly, that the -O3 compiler flag has large impact on performance. Secondly, that with or without -03, the code changes still have a huge impact on performance.

Developer’s ManualAll compilation is handled by make. $ cd <directory of unzipped final submission>/src-optimized$ make all$ ./HashSHA3 <size> <message> Compilation has been tested and verified with the gcc compiler available on the RIT CS Department machines (gcc version 4.6.3). The all target will also compile the test programs.

User’s ManualHashing a stringThe program is delivered as a command line interface. All operations on the program are controlled through the two parameters. $ ./HashSHA3 [digestSize] <message> Digest Size: The final digest of the hash function (in bits). Shorter digest sizes result in a faster run time, but a lower probability of a unique hash. Larger digest sizes result in slower run time, but higher probability of a unique hash.Message: The message to hash. Hashing a fileThe program is delivered as a command line interface. The operations are designed to reflect some aspects of the standard shasum program. $ ./sha3sum [-a <digestSize>] <file> Digest Size: The final digest of the hash function (in bits). Shorter digest sizes result in a faster run time, but a lower probability of a unique hash. Larger digest sizes result in slower run time, but higher probability of a unique hash.File: The file to hash.

Discussion

LearnedWriting code for cryptography is one of the hardest challenges within programing. It makes use of every programing skill a developer has. Within cryptography, the team learned how to read a full cryptographic specification. Implementing a cypher requires a very deep level of understanding of its functionality.

Apart from what the team learned from the cryptographic domain, the practice of optimizing a C++ program was a hugely educational experience. The implementation model used was to first create a working implementation, then to optimize it. This was especially important with the nature of cryptographic implementations. The functions are far more complicated than usual code and very susceptible to regression bugs, and it was extremely important to test after every change. If the team had attempted to make the most performant solution first, it would have been a nightmare to test and verify the solution. Analysis of implemented code requires a critical eye for details and developing a strong set of tools. Both Xcode’s Instruments and OS X’s sample were crucial for the optimization. They made getting down to a function level analysis of performance simple, and let us understand our code better than any other technique.

Future WorkWe are confident in the accuracy of the implementation to Keccak specification. The most important thing to do to further the code base would be to increase the performance. To increase the performance at this stage of the project, it would require a more indepth analysis of performance in C++ applications. Another area to improve the implementation would be to make it support parallelism. The other large concern would be security, as with any other program.

Individual ContributionsChris Bentivenga: Algorithm research, optimization and analysis, and documentation of findings and program usage.Frederick Christie: Algorithm research and presentations.Michael Kitson: Initial keccak implementation, profiling and timing, optimizations.

ReferencesAbout Instruments [Online]. Available: https://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html Examples of SHA-3 (Keccak) variants [Online]. Available: https://en.wikipedia.org/wiki/Sha3#Examples_of_SHA-3_.28Keccak.29_variants G. Bertoni and J. Daemen and M. Peeters and G. Assche. The Keccak Reference [Online]. Jan 14. 2011. Available: http://keccak.noekeon.org/Keccak-reference-3.0.pdf Options That Control Optimization [Online]. Available: http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html The Keecak sponge function family: Specification Summary [Online] . Jan 1. 20011Available: http://keccak.noekeon.org/specs_summary.html

https://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html


























https://en.wikipedia.org/wiki/Sha3#Examples_of_SHA-3_.28Keccak.29_variants






















http://keccak.noekeon.org/Keccak-reference-3.0.pdf













http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html


















http://keccak.noekeon.org/specs_summary.html













keccak - ritark/winter2012/482/team/u5/report.pdf · description of the algorithm keccak is the...

Documents