|
---
|
|
license: mit
|
|
---
|
|
|
|
# Static Embeddings using SQLiteVec
|
|
|
|
- This repo contains [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1)
|
|
model converted to [sqlite-vec](https://github.com/asg017/sqlite-vec) database.
|
|
|
|
- This repo contains a Swift package that demonstrates how to use the [SQLiteVec](https://github.com/jkrukowski/SQLiteVec) to compute static embeddings.
|
|
|
|
- This repo contains a benchmark that compares the performance of computing embeddings using [swift-embeddings](https://github.com/jkrukowski/swift-embeddings) and [SQLiteVec](https://github.com/jkrukowski/SQLiteVec).
|
|
|
|
The sqlite-vec database was created using the following command:
|
|
|
|
```sql
|
|
CREATE VIRTUAL TABLE embeddings USING vec0(embedding float[1024])
|
|
```
|
|
|
|
and then the embeddings were inserted using the following command:
|
|
|
|
```sql
|
|
INSERT INTO embeddings(rowid, embedding) VALUES (?, ?)
|
|
```
|
|
|
|
## Benchmark
|
|
|
|
To run the benchmark, execute the following command:
|
|
|
|
```bash
|
|
cd EmbeddingsBenchmark
|
|
swift package --disable-sandbox benchmark
|
|
```
|
|
|
|
### Results
|
|
|
|
```
|
|
|
|
CoreML benchmark (count: 10)
|
|
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
|
|
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
|
|
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
|
|
β Instructions (K) * β 1712 β 1779 β 1779 β 1791 β 1792 β 1805 β 1899 β 5136 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Malloc (total) * β 1131 β 1132 β 1132 β 1132 β 1132 β 1132 β 1139 β 5136 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Memory (resident peak) (M) β 156 β 158 β 158 β 159 β 159 β 159 β 159 β 5136 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Throughput (# / s) (#) β 11121 β 9471 β 9287 β 8919 β 8607 β 7747 β 5810 β 5136 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (total CPU) (ΞΌs) * β 92 β 107 β 109 β 114 β 118 β 131 β 174 β 5136 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (wall clock) (ΞΌs) * β 90 β 106 β 108 β 112 β 116 β 129 β 172 β 5136 β
|
|
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
|
|
|
|
CoreML benchmark (count: 1000)
|
|
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
|
|
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
|
|
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
|
|
β Instructions (K) * β 3320 β 3422 β 3467 β 3516 β 3545 β 3566 β 3640 β 2698 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Malloc (total) * β 1131 β 1132 β 1132 β 1132 β 1132 β 1132 β 1140 β 2698 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Memory (resident peak) (M) β 159 β 162 β 162 β 162 β 162 β 162 β 162 β 2698 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Throughput (# / s) (#) β 4418 β 4299 β 4231 β 4139 β 3999 β 3515 β 2093 β 2698 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (total CPU) (ΞΌs) * β 228 β 234 β 238 β 243 β 252 β 287 β 480 β 2698 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (wall clock) (ΞΌs) * β 226 β 233 β 237 β 242 β 250 β 285 β 478 β 2698 β
|
|
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
|
|
|
|
CoreML benchmark (count: 100000)
|
|
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
|
|
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
|
|
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
|
|
β Instructions (M) * β 163 β 163 β 163 β 163 β 163 β 168 β 168 β 20 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Malloc (total) * β 1131 β 1131 β 1131 β 1132 β 1133 β 1139 β 1139 β 20 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Memory (resident peak) (M) β 566 β 567 β 567 β 568 β 568 β 568 β 568 β 20 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Throughput (# / s) (#) β 21 β 21 β 21 β 21 β 21 β 20 β 20 β 20 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (total CPU) (ms) * β 47 β 48 β 48 β 48 β 48 β 50 β 50 β 20 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (wall clock) (ms) * β 47 β 48 β 48 β 48 β 48 β 50 β 50 β 20 β
|
|
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
|
|
|
|
SQLite benchmark (count: 10)
|
|
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
|
|
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
|
|
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
|
|
β Instructions (M) * β 103 β 117 β 122 β 128 β 132 β 136 β 140 β 144 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Malloc (total) * β 99 β 106 β 106 β 106 β 106 β 107 β 107 β 144 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Memory (resident peak) (M) β 15 β 17 β 19 β 19 β 19 β 19 β 19 β 144 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Throughput (# / s) (#) β 186 β 155 β 147 β 139 β 132 β 126 β 126 β 144 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (total CPU) (ΞΌs) * β 5382 β 6451 β 6779 β 7180 β 7582 β 7922 β 7932 β 144 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (wall clock) (ΞΌs) * β 5380 β 6443 β 6779 β 7180 β 7574 β 7918 β 7930 β 144 β
|
|
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
|
|
|
|
SQLite benchmark (count: 1000)
|
|
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
|
|
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
|
|
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
|
|
β Instructions (M) * β 1517 β 1572 β 1599 β 1611 β 1618 β 1618 β 1618 β 9 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Malloc (total) * β 6330 β 6339 β 6367 β 6379 β 6414 β 6414 β 6414 β 9 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Memory (resident peak) (M) β 22 β 23 β 23 β 23 β 23 β 23 β 23 β 9 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Throughput (# / s) (#) β 9 β 8 β 8 β 8 β 8 β 8 β 8 β 9 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (total CPU) (ms) * β 114 β 118 β 120 β 121 β 121 β 121 β 121 β 9 β
|
|
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
|
|
β Time (wall clock) (ms) * β 114 β 118 β 120 β 121 β 121 β 121 β 121 β 9 β
|
|
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
|
|
```
|
|
|
