Advancements in artificial intelligence and virtual reality technology have opened up a world of possibilities for creating immersive experiences.
At Akvelon, we have combined the power of AI and virtual reality technology to create an immersive AI-powered VR application, an experience that transports users to breathtaking environments infused with different artistic styles.
Key takeaways:
- Akvelon combines AI and VR technology to create captivating immersive experiences
- Stylizing panoramic photos in various artistic genres with the Stable Diffusion neural network showcases the potential of Virtual reality and artificial intelligence in the creative industry.
- Splitting a single 2D image into normal-projected images with Blender software while using MiDaS as a guide for stylization
- Stitching scaled, stylized images together for a seamless experience with pre-calculated anchor points and masks
- Demonstrating AI and VR's limitless potential in various fields with potential use cases in virtual art therapy, education, and collaborative art/design projects.
Our journey towards creating this application began with a desire to push the boundaries of what was possible with panoramic photography. While traditional panoramic photos capture breathtaking landscapes or cityscapes, we wanted to take it a step further by infusing them with artistic styles that could transport users to new and exciting worlds. We began exploring the use of AI to achieve this goal, eventually settling on the Stable Diffusion neural network as the best tool for the job.
With this technology in place, we set out to create an application that would allow users to experience these panoramic environments in an immersive way. We knew that virtual reality technology would be the perfect tool for this task, allowing users to step into the worlds we had created and explore them in a way that was impossible with traditional photography. And thus, our AI ArtDive VR application was born.
Virtual reality and artificial intelligence together have created endless possibilities for the future of immersive experiences. Our hope is that by combining the power of AI and VR technology, we can create an experience that not only entertains but also inspires and educates users about the world of art and its impact on our lives. We believe that the possibilities are endless, and we are excited to see where this technology takes us next.
Technology Behind the Scenes
Cutting Panoramas
Images for VR panoramas are stored in something known as equirectangular projection – this is a useful method to store and treat 3D panoramas as 2D images with corresponding transformations.
Below is an example of the equirectangular projection of a 3D image:
Our main goal is to apply styles to a source 3D panorama photo in order to get a styled panorama for our application. ANN can apply style to a 2D image, but in this process, the resulting styled 2D image loses correct equirectangular proportions – thus resulting in a styled 3D panorama that will have conspicuous artifacts.
To avoid this, at first we split the source 2D equirectangular – a projected image by Blender software to a set of normal-projected images, like we’re watching through a camera with a fixed field of view to the mapped 3D spherical panorama - with overlapped areas:
Then, we applied style to the set of these images.
Stylizing 3D Panoramas
After splitting the source 2D equirectangular image by Blender software, we apply the Stable Diffusion model to change the style of each projected image. The Stable Diffusion is a deep learning model released in 2022 that is primarily used to generate detailed images conditioned on text descriptions, inpainting, outpainting, and generating image-to-image translations guided by a text prompt. This is the exact image-to-image translation feature used for image stylization.
The Stable Diffusion model was selected for this project because of its ability to perform image-to-image translations for stylization purposes. By conditioning the model on the source image and a desired style, we can generate a new image with the same content as the original, but with a different artistic style. This allows us to easily experiment with different styles and create a diverse set of stylized images. Additionally, the model's ability to generate detailed images and handle complex text descriptions makes it a powerful tool for this project.
However, using only the Stable Diffusion model can result in images with many artifacts, which deforms the form of objects, making them unrecognizable. To solve this problem, we use a guide to control the stylization process. We convert the projected images into maps of depth with another neural network called MiDaS. During the stylization of the images, we use loss based on the difference in the map of the stylized image’s depth and the depth of the original one. This helps us to save the depth on a stylized image at the same level as on the original image, which significantly improves the final result. This method allows us to change the style of each image while preserving the original form of objects.
To illustrate, we use the following settings to generate a stylized image:
- Prompt: "an oil-on-canvas painting in the style of Starry Night, constellations, incredible detail, trending on artstation"
- Negative prompt: "ugly, tiling, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, extra limbs, disfigured, deformed, body out of frame, bad anatomy, watermark, signature, cut off, low contrast, underexposed, overexposed, bad art, beginner, amateur, distorted face"
- Steps: 50
- Sampler: DDIM
- CFG scale: 7
- Size: 1024x1024
- Denoising strength: 0.5
After stylizing the images, the next step is to increase their resolution. Since processing high resolutions is resource-consuming and time-consuming, we process images with a resolution of 1024x1024 and then scale them with another neural network called R-ESRGAN 4x+. This network allows us to upscale images up to 4096x4096 without losing image quality.
Finally, the scaled stylized images are sent to the stitching process. With the pre-calculated anchor points and masks, we can stitch all the images back and save them as an equirectangular resulting 2D image. This final step allows us to integrate the resulting image into our VR application, providing a seamless and immersive experience for users.
Stitching Panoramas
At this step, we were faced with this next problem to solve: ANN applies style to each image of the set individually, thus common anchor points for stitching images are lost in the process. To fix this, we first stitched an un-styled set of images to calculate and fix anchor points. In this process, we calculated anchor points and masks to stitch our image set. This was done with Hugin software.
Then, we were ready to apply style to our images in set:
And, finally, with pre-calculated anchor points and masks, we stitched all back together and saved them as equirectangular, resulting in a 2D image to integrate to our VR application:
Use Cases
The creation of AI-generated immersive artistic environments has opened up many possibilities in several industries such as healthcare, education, and art. We have collected some potential use cases below.
Use Case 1: Virtual Art Therapy
The AI-generated immersive artistic environments, powered by VR technology, can enhance the way healthcare professionals approach various treatments, including pain management, mental health, and rehabilitation. AI in VR games and VR analytics and AI can be used to create Intelligent VR environments that are tailored to the specific needs of patients. By creating virtual environments that are tailored to the specific needs of patients, healthcare providers can help them achieve a more profound level of healing and well-being.
One of the most promising applications of this technology is virtual art therapy is VR training with AI for virtual art therapy. By creating immersive environments that stimulate the senses, patients can engage in creative expression that promotes healing and reduces stress. The ability to control their environment and adjust it based on the patient's needs can lead to significant breakthroughs in pain management and mental health treatment.
This technology can be used to create a virtual art therapy platform for patients dealing with stress, anxiety, depression, or other mental health issues. In this use case, healthcare providers can curate a collection of soothing and inspiring VR experiences, allowing patients to explore and immerse themselves in various artistic styles and landscapes.
Benefits:
- Provides a calming and immersive experience for patients
- Encourages patients to explore new artistic styles and environments
- Complements traditional art therapy methods
- Helps patients develop mindfulness and coping strategies
Use Case 2: Education and Cultural Appreciation
AI-generated immersive artistic environments can be used in educational settings for teaching students about art history, different artistic styles, and cultural appreciation. Teachers can curate a collection of virtual experiences that showcase various art movements and styles, allowing students to explore and appreciate art in an engaging and interactive way.
Benefits:
- Provides a unique and immersive learning experience, demonstrating the potential of AI-powered VR for education.
- Encourages students to explore various art movements and styles, utilizing VR training with AI for an optimized learning experience.
- Enhances understanding and appreciation of art and culture, promoting Virtual reality and artificial intelligence for cultural appreciation.
- Supports a diverse and inclusive curriculum, showcasing different perspectives and styles.
Use Case 3: Collaborative Art and Design Projects
The AI-generated immersive environments can be used as a platform for collaborative art and design projects, allowing artists, designers, and other creative professionals to work together in a virtual space. This can facilitate brainstorming, idea generation, and the development of unique artistic concepts, bridging the gap between physical and digital creative processes.
Benefits:
- Enhances collaboration among artists and designers
- Provides a unique and engaging platform for creative projects
- Fosters innovation and creativity in art and design
- Supports the development of new artistic styles and techniques
Conclusion:
Akvelon's innovative use of AI and VR technology has resulted in the creation of AI ArtDive VR, an immersive application that allows users to experience panoramic environments in different artistic styles. By using the Stable Diffusion neural network and a guide called MiDaS, Akvelon has been able to preserve the original form of objects while applying different artistic styles to the images. This application not only entertains, but also inspires and educates users about the world of art and its impact on our lives.
As this technology continues to evolve, its potential to enhance different human experiences in many areas is only expected to grow. We are excited to see what the combination of AI and VR technology can offer for better health, education, and wellness for all.
If you are interested in experiencing the power of AI-generated immersive artistic environments, reach out to us at hello@akvelon.com to learn more about our application.
This article was written by
Aleksei Korobkin
Software Development Engineer
Iuliia Manina
Project Manager
Ilya Polishchuk
Director of Engineering
