This is actually where LanceDB really shines because of 2 major reason:

1. LanceDB is based on a columnar storage format specialized for unstructured data.

2. LanceDB's index is a custom disk-based index written in Rust.

Largest known self-hosted production installation is ~O(1B) vectors of about 1500 dimensions that uses a single GCP node.