File size: 19,681 Bytes
087dd55 902bb3d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
---
license: mit
---
# Static Embeddings using SQLiteVec
- This repo contains [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1)
model converted to [sqlite-vec](https://github.com/asg017/sqlite-vec) database.
- This repo contains a Swift package that demonstrates how to use the [SQLiteVec](https://github.com/jkrukowski/SQLiteVec) to compute static embeddings.
- This repo contains a benchmark that compares the performance of computing embeddings using [swift-embeddings](https://github.com/jkrukowski/swift-embeddings) and [SQLiteVec](https://github.com/jkrukowski/SQLiteVec).
The sqlite-vec database was created using the following command:
```sql
CREATE VIRTUAL TABLE embeddings USING vec0(embedding float[1024])
```
and then the embeddings were inserted using the following command:
```sql
INSERT INTO embeddings(rowid, embedding) VALUES (?, ?)
```
## Benchmark
To run the benchmark, execute the following command:
```bash
cd EmbeddingsBenchmark
swift package --disable-sandbox benchmark
```
### Results
```
CoreML benchmark (count: 10)
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
β Instructions (K) * β 1712 β 1779 β 1779 β 1791 β 1792 β 1805 β 1899 β 5136 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Malloc (total) * β 1131 β 1132 β 1132 β 1132 β 1132 β 1132 β 1139 β 5136 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Memory (resident peak) (M) β 156 β 158 β 158 β 159 β 159 β 159 β 159 β 5136 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Throughput (# / s) (#) β 11121 β 9471 β 9287 β 8919 β 8607 β 7747 β 5810 β 5136 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (total CPU) (ΞΌs) * β 92 β 107 β 109 β 114 β 118 β 131 β 174 β 5136 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (wall clock) (ΞΌs) * β 90 β 106 β 108 β 112 β 116 β 129 β 172 β 5136 β
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
CoreML benchmark (count: 1000)
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
β Instructions (K) * β 3320 β 3422 β 3467 β 3516 β 3545 β 3566 β 3640 β 2698 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Malloc (total) * β 1131 β 1132 β 1132 β 1132 β 1132 β 1132 β 1140 β 2698 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Memory (resident peak) (M) β 159 β 162 β 162 β 162 β 162 β 162 β 162 β 2698 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Throughput (# / s) (#) β 4418 β 4299 β 4231 β 4139 β 3999 β 3515 β 2093 β 2698 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (total CPU) (ΞΌs) * β 228 β 234 β 238 β 243 β 252 β 287 β 480 β 2698 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (wall clock) (ΞΌs) * β 226 β 233 β 237 β 242 β 250 β 285 β 478 β 2698 β
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
CoreML benchmark (count: 100000)
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
β Instructions (M) * β 163 β 163 β 163 β 163 β 163 β 168 β 168 β 20 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Malloc (total) * β 1131 β 1131 β 1131 β 1132 β 1133 β 1139 β 1139 β 20 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Memory (resident peak) (M) β 566 β 567 β 567 β 568 β 568 β 568 β 568 β 20 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Throughput (# / s) (#) β 21 β 21 β 21 β 21 β 21 β 20 β 20 β 20 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (total CPU) (ms) * β 47 β 48 β 48 β 48 β 48 β 50 β 50 β 20 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (wall clock) (ms) * β 47 β 48 β 48 β 48 β 48 β 50 β 50 β 20 β
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
SQLite benchmark (count: 10)
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
β Instructions (M) * β 103 β 117 β 122 β 128 β 132 β 136 β 140 β 144 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Malloc (total) * β 99 β 106 β 106 β 106 β 106 β 107 β 107 β 144 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Memory (resident peak) (M) β 15 β 17 β 19 β 19 β 19 β 19 β 19 β 144 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Throughput (# / s) (#) β 186 β 155 β 147 β 139 β 132 β 126 β 126 β 144 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (total CPU) (ΞΌs) * β 5382 β 6451 β 6779 β 7180 β 7582 β 7922 β 7932 β 144 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (wall clock) (ΞΌs) * β 5380 β 6443 β 6779 β 7180 β 7574 β 7918 β 7930 β 144 β
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
SQLite benchmark (count: 1000)
βββββββββββββββββββββββββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ€ββββββββββ
β Metric β p0 β p25 β p50 β p75 β p90 β p99 β p100 β Samples β
βββββββββββββββββββββββββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββͺββββββββββ‘
β Instructions (M) * β 1517 β 1572 β 1599 β 1611 β 1618 β 1618 β 1618 β 9 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Malloc (total) * β 6330 β 6339 β 6367 β 6379 β 6414 β 6414 β 6414 β 9 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Memory (resident peak) (M) β 22 β 23 β 23 β 23 β 23 β 23 β 23 β 9 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Throughput (# / s) (#) β 9 β 8 β 8 β 8 β 8 β 8 β 8 β 9 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (total CPU) (ms) * β 114 β 118 β 120 β 121 β 121 β 121 β 121 β 9 β
βββββββββββββββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββΌββββββββββ€
β Time (wall clock) (ms) * β 114 β 118 β 120 β 121 β 121 β 121 β 121 β 9 β
βββββββββββββββββββββββββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ§ββββββββββ
```
|