Video understanding

a method for transferring the style from a set of images to a 3D object. The texture appearance of an asset is optimized with a differentiable renderer in a pipeline based on losses using pretrained deep neural networks. More specifically, we utilize a nearest-neighbor feature matching loss with CLIP-ResNet50 to extract the style from images. We show that a CLIP-based style loss provides a different appearance over a VGG-based loss by focusing more on texture over geometric shapes. Additionally, we extend the loss to support multiple images and enable loss-based control over the color palette combined with automatic color palette extraction from style images.

Text-driven framework for 3D avatar generation and animation. Unlike professional software that requires expert knowledge, AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages.

Changing the style of a given image based on text input

Combining neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions.

Generating pixelated animation and images based on given text

Generate modern paintings from text descriptions.

Analysing videos and creating caption for them

Converting an image of an object to a sketch, allowing for varying levels of abstraction.

Create a description of an image on Russian