Inverting Multi-Vector Visual Document Indices

The paper explores the vulnerability of multi-vector visual document retrievers, where stored page indices can be inverted to reconstruct document content. Using the ViDoRe v3 benchmark, it shows that inverted pages recover 47% of words and 45% of sensitive tokens, with 98.4% of inverted queries ranking the original page first. Simple protections like shuffling and pooling reduce effectiveness, but order restoration significantly improves results. The study highlights the need to treat stored indices as sensitive as the documents themselves.