Stability AI, in partnership with its AI research lab DeepFloyd, has recently released the research version of its latest technology, DeepFloyd IF. This text-to-image cascaded pixel diffusion model generates high-quality images from text inputs, making it a breakthrough in the field of artificial intelligence.
The DeepFloyd IF model is available on a non-commercial, research-permissible license, allowing research labs to explore and experiment with advanced text-to-image generation methods.
Features of DeepFloyd IF
The DeepFloyd IF model boasts several impressive features. It uses the T5-XXL-1.1 language model as a text encoder to aid in understanding text prompts. The model also employs cross-attention layers to better align the text prompt and the generated image.
One of the standout features of the DeepFloyd IF model is its ability to accurately apply text descriptions to generate images with various objects appearing in different spatial relations. This has been a challenging task for other text-to-image models. The model also produces photorealistic images, reflected in its impressive zero-shot FID score of 6.66 on the COCO dataset. Moreover, the DeepFloyd IF model can generate images with non-standard aspect ratios, including vertical or horizontal orientations and the standard square aspect.
How DeepFloyd IF Works
The DeepFloyd IF model works in three stages to generate high-quality images from text prompts. A frozen T5-XXL language model converts the text prompt into a qualitative representation in the first stage. In the second stage, a base diffusion model is applied to transform the qualitative text into a 64×64 image, which is then upscaled to 256×256 using two text-conditional super-resolution models. During the third stage of the process, a final model is used to enhance the image to a clear and high-quality 1024×1024 resolution.
The IF model includes different versions of the base and super-resolution models, which have other parameters. Although the third-stage model has yet to be available, alternative upscale models like the Stable Diffusion x4 Upscaler can be utilized.
Dataset and License
The DeepFloyd IF model was trained on a high-quality custom dataset called LAION-A, which contains 1 billion (image, text) pairs. The dataset is an aesthetic subset of the English part of the LAION-5B dataset, and the data were filtered using custom filters to remove inappropriate content.
Access to the model’s weights is available on Deep Floyd’s Hugging Face (https://huggingface.co/DeepFloyd) space, and the model card and code are also available on GitHub (https://github.com/deep-floyd/IF ). A Gradio demo is provided for everyone, and the creators invite people to join public discussions.
