antirez/word2vec-simple-parsing

Word2Vec in a non overengineered file format

This is the Word2Vec words embedding collection by Google(TM) converted in a sane binary format that you can read with a trivial program, without resorting to complicated libraries.

This is the format:

1. All the integers are unsigned, in little endian format.
2. Floating point numbers of the embedding vectors are stored as single precision floating point numbers, float32.

The file starts with a header of 8 bytes:

uint32_t num_entries    // Number of entries: 3000000 in this file
uint32_t dimension      // Embedding dimension: 300 in this file

Then each entry is stored as:

uint16_t word_len       // Length of the UTF-8 encoded word.
word_len bytes          // UTF-8 encoded word.
float[300]              // 300 float32 numbers, 1200 bytes in total.

The license is the same as Google's original Word2Vec, Apache 2.0.