Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
arxiv(2023)
摘要
Diffusion models have been shown to be capable of generating high-quality
images, suggesting that they could contain meaningful internal representations.
Unfortunately, the feature maps that encode a diffusion model's internal
information are spread not only over layers of the network, but also over
diffusion timesteps, making it challenging to extract useful descriptors. We
propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and
multi-timestep feature maps into per-pixel feature descriptors that can be used
for downstream tasks. These descriptors can be extracted for both synthetic and
real images using the generation and inversion processes. We evaluate the
utility of our Diffusion Hyperfeatures on the task of semantic keypoint
correspondence: our method achieves superior performance on the SPair-71k real
image benchmark. We also demonstrate that our method is flexible and
transferable: our feature aggregation network trained on the inversion features
of real image pairs can be used on the generation features of synthetic image
pairs with unseen objects and compositions. Our code is available at
https://diffusion-hyperfeatures.github.io.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要