Video production using Generative A.I.

The experiments in this page are the synergy between technology and artistic expression. The videos and images have been meticulously crafted through the integration of creative coding and AI. Throughout my exploration, I've designed diverse workflows to align with my specific requirements, keeping a keen eye on the significant influence of human emotions in shaping expressions powered by AI.

Press play on the header of this page to watch the complete video.

Project development

01 - Lyrics to json

I created a program to convert the lyrics from text into a JSON format that suits my needs perfectly. As you can see in the image, the original and transformed formats look similar, however with my custom software, I can achieve this conversion in less than a second, effortlessly transferring all lyric lines into a designated "lyric" object within a JSON file. This format enables seamless integration across various stages of production, facilitating use in AI algorithms, After Effects integration, and more. Additionally, this approach ensures a robust and easily reproducible process for future song projects.

click on the image to see it on full screen

02 - Automation setup

In my work with After Effects, I've built a smart way to handle lyrics animation using coding. Instead of manually adjusting each lyric line, I import a JSON file directly into the software (from the previous step). This means I don't have to spend time setting up keyframes for every line, saving loads of time, especially when working on bigger projects. It's a real time-saver ensuring everything lines up perfectly with the song's timing. This approach not only speeds up my workflow but also makes it easier to work on future projects. By avoiding manual adjustments and letting the software handle the heavy lifting, I can focus more on the creative side of things and less on the technical details.

click on the image to see it on full screen

03 - Extracting and engineering A.I. prompts

The same JSON went through another Node.js application I designed, extracting only the frame position and the lyrics. Each line of the lyric was programmatically added to a style description, which together was fed into the AI. Effectively each line of the music now is transformed into a prompt, resulting in an animation frame.

click on the image to see it on full screen

04 - Generating frames & finishing

After setting up the prompts, I loaded them into the code, so the AI started to generate the frames of the animation. The prompts served as keyframes, whilst the A.I. calculated the interpolation with the parameters I created. These resulted in the first iteration of the animation, which was loaded in After Effects for post-production. This last step has also utilised the previous automation, rendering the video with the lyrics on top.

click on the image to see it on full screen

Further experiments

This collaborative venture emerged from the creative synergy between myself and the talented musician, Henry Dearden. Henry composed a series of songs with a vision to release them on the week before Christmas 2023. The two songs, re-imagined from distinct points of view, delved into apocalyptic futures, black comedy, and moments of profound sadness.

Jingle Bells: Dark Comedy

I visualised the different instruments and layers in the music as distinctive across elements in the dark comedy. The visuals resemble Tim Burton's as the song suggests, but I also delve into The 1920s expressionism in Germany, reflected on the assets and treatments.

watch the video

Jingle Bells: Dead Planet

This song is about humanity in 2200, after and apocalyptic event that consumed the planet. A few survivors now are living in bunkers, where they may never find a glimpse of hope. For this video, I tried to imagine a MadMax in snow, perhaps with a few cyberpunk elements.

watch the video