Qdrant 嵌入模型迁移
qdrant-model-migration
指导在 Qdrant 中不停机迁移嵌入模型。当有人询问“如何切换嵌入模型”“如何迁移向量”“如何更新到新模型”“零停机模型切换”“如何重新嵌入数据”,或“能否同时使用两个模型”时使用。也适用于升级模型维度、切换提供商或开展模型 A/B 测试。
介绍来源: author · github · b6483db44fb602654bca6a097185fcaddb305f97 参考来源
作者原文
Guides embedding model migration in Qdrant without downtime. Use when someone asks 'how to switch embedding models', 'how to migrate vectors', 'how to update to a new model', 'zero-downtime model change', 'how to re-embed my data', or 'can I use two models at once'. Also use when upgrading model dimensions, switching providers, or A/B testing models.
SkillPi 解读
- 能完成的任务
- Qdrant 嵌入模型迁移
- 适合谁
- 请按作者原文中的适用场景判断。
- 输入和输出
- 以作者完整文档为准;本站未运行验证产物。
- 软件、账号及付费要求
- 使用前按完整文档确认软件、账号、费用及权限;由宿主核对并授权实际文件读写和外部服务。文档引用域名(非运行白名单):search.qdrant.tech
- 限制与区别
- 原文中的功能、版本状态和效果描述属于作者陈述;中文为对应翻译。仅核对固定版本及静态附件证据,不保证运行效果、外部服务或兼容性。
- 实测状态
- 未测试
权限与使用前检查
根据当前主文档自动整理,命中内容可能是示例或防护说明;不是实际权限声明,也未覆盖附件、外部依赖或运行行为。
未从主文档识别到上述常见操作,不能据此判断无需权限或没有风险。
文档引用的域名或地址
这些地址可能用于文档链接或示例,不是已批准的联网清单。使用前核对实际目的地及发送的数据。
- search.qdrant.tech
安全评级与访问范围
caution · 静态证据核验,不代表运行安全保证。
使用前按完整文档确认软件、账号、费用及权限;由宿主核对并授权实际文件读写和外部服务。文档引用域名(非运行白名单):search.qdrant.tech
使用前请由宿主核对并授权实际文件读写、外部服务和联网目的地;本站的核验不会授予任何运行权限。
原始 SKILL.md
--- name: qdrant-model-migration description: "Guides embedding model migration in Qdrant without downtime. Use when someone asks 'how to switch embedding models', 'how to migrate vectors', 'how to update to a new model', 'zero-downtime model change', 'how to re-embed my data', or 'can I use two models at once'. Also use when upgrading model dimensions, switching providers, or A/B testing models." --- # What to Do When Changing Embedding Models Vectors from different models are incompatible. You cannot mix old and new embeddings in the same vector space. You also cannot add new named vector fields to an existing collection. All named vectors must be defined at collection creation time. Both migration strategies below require creating a new collection. - Understand collection aliases before choosing a strategy [Collection aliases](https://search.qdrant.tech/md/documentation/manage-data/collections/?s=collection-aliases) ## Can I Avoid Re-embedding? Use when: looking for shortcuts before committing to full migration. You MUST re-embed if: changing model provider (OpenAI to Cohere), changing architecture (CLIP to BGE), incompatible dimension counts across different models, or adding sparse vectors to dense-only collection. You CAN avoid re-embedding if: using Matryoshka models (use `dimensions` parameter to output lower-dimensional embeddings, learn linear transformation from sample data, some recall loss, good for 100M+ datasets). Or changing quantization (binary to scalar): Qdrant re-quantizes automatically. [Quantization](https://search.qdrant.tech/md/documentation/manage-data/quantization/) ## Need Zero Downtime (Alias Swap) Use when: production must stay available. Recommended for model replacement at scale. - Create a new collection with the new model's dimensions and distance metric - Re-embed all data into the new collection in the background - Point your application at a collection alias instead of a direct collection name - Atomically swap the alias to the new collection [Switch collection](https://search.qdrant.tech/md/documentation/manage-data/collections/?s=switch-collection) - Verify search quality, then delete the old collection Careful, the alias swap only redirects queries. Payloads must be re-uploaded separately. ## Need Both Models Live (Side-by-Side) Use when: A/B testing models, multi-modal (dense + sparse), or evaluating a new model before committing. You cannot add a named vector to an existing collection. Create a new collection with both vector fields defined upfront: - Create new collection with old and new named vectors both defined [Collection with multiple vectors](https://search.qdrant.tech/md/documentation/manage-data/collections/?s=collection-with-multiple-vectors) - Migrate data from old collection, preserving existing vectors in the old named field - Backfill new model embeddings incrementally using `UpdateVectors` [Update vectors](https://search.qdrant.tech/md/documentation/manage-data/points/?s=update-vectors) - Compare quality by querying with `using: "old_model"` vs `using: "new_model"` - Swap alias to new collection once satisfied Co-locating large multi-vectors (especially ColBERT) with dense vectors degrades ALL queries, even those only using dense. At millions of points, users report 13s latency dropping to 2s after removing ColBERT. Put large vectors on disk during side-by-side migration. If you anticipate future model migrations, define both vector fields upfront at collection creation. ## Dense to Hybrid Search Migration Use when: adding sparse/BM25 vectors to an existing dense-only collection. Most common migration pattern. You cannot add sparse vectors to an existing dense-only collection. Must recreate: - Create new collection with both dense and sparse vector configs defined - Re-embed all data with both dense and sparse models - Migrate payloads, swap alias Sparse vectors at chunk level have different TF-IDF characteristics than document level. Test retrieval quality after migration, especially for non-English text without stop-word removal. ## Re-embedding Is Too Slow Use when: dataset is large and re-embedding is the bottleneck. - Use `update_mode: insert` (v1.17+) for safe idempotent migration [Update mode](https://search.qdrant.tech/md/documentation/manage-data/points/?s=update-mode) - Scroll the old collection with `with_vectors=False`, re-embed in batches, upsert into new collection - Upload in parallel batches (64-256 points per request, 2-4 parallel streams) [Bulk upload](https://search.qdrant.tech/md/documentation/tutorials-develop/bulk-upload/) - Disable HNSW during bulk load (set `indexing_threshold_kb` very high, restore after) - For Qdrant Cloud inference, switching models is a config change, not a pipeline change [Inference docs](https://search.qdrant.tech/md/documentation/inference/) For 400GB+ datasets, expect days. For small datasets (<25MB), re-indexing from source is faster than using the migration tool. ## What NOT to Do - Assume you can add named vectors to an existing collection (must be defined at creation time) - Delete the old collection before verifying the new one - Forget to update the query embedding model in your application code - Skip payload migration when using alias swap (aliases redirect queries, they do not copy data) - Keep ColBERT vectors co-located with dense vectors during a long migration (I/O cost degrades all queries) - Migrate to hybrid search without testing BM25 quality at chunk level
版权与许可声明
MIT License Copyright GitHub, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
用户讨论