Before we get started, our AI Flying ad kinda went viral so if you came to this newsletter from that video, here’s the full walkthrough on how to make your own flying video.
For years animation has been a time intensive task executed by individuals with a highly specialized skillset, but with rise of AI video generation this once hard to create style is being generated for the masses with the click of a button, and now with Runway Act-Two, you can voice and direct your own characters with your very own body.
So let’s get into the nitty gritty, here’s what you need to make your own Pixar movie scenes: phone/camera, image generator (options below), and Runway Act-Two. If you’re wanting to create a back and forth dialogue then you’ll also need a video editor like Final Cut, CapCut, or Adobe Premiere.
The basic work flow here is that you will be filming yourself doing the actions and saying the lines - this will be your “performance clip” and then you will create an image to be the animated look you want your performance to be applied too. In our examples, we filmed on a green screen but that isn’t required - although solid backgrounds seem to provide better results.
Now let’s get started:
1- Setup your phone on a tripod (no shaky footage now!) and film yourself from the waste up in landscape. Make sure you’re well lit and film different clips for each of your scenes.
2- Once you’ve filmed your “performance clips” it’s time to switch over to your photo generator, our favorite for this was Whisk (Googles photo generation tool) but others could work like; ChatGPT, Runway’s image generator, or Midjourney. We weren’t being very picky for our character generation prompts but here is a more detailed prompt structure you can use to generate your characters.
A 3D-rendered, Pixar-style headshot of a 70-year-old man with East Asian features, white wispy hair combed back, gentle crow’s feet around his eyes, and round spectacles. He wears a patchwork vest over a turtleneck in a cozy wood-paneled workshop. The character is content, smiling warmly, facing slightly left with hands folded. The scene is lit by warm evening lamp light, shot close-up with cinematic shallow focus. Pixar style, 4K resolution.
3- Next you’ll need access to Runway Act-Two, this is where you’ll construct your video using your “performance clip” and your character image. There is no prompting here as your performance is the guide. Just upload both elements and let Runway work.
So now you have generated one scene of a character talking or moving - but what about the person or thing they are talking too? Well let’s go back to Whisk, where we can use our image of the knight to direct the scene and then prompt a character for him to talk too.
To do this, you’ll select “add image” in the prompt box and a menu on the left hand side will pop out. Here you want to add the image you want to create a cohesive scene with as the “scene” - then write out the description of your new character.
Now we have a young lady that our knight can talk too, next repeat the previous steps to create the second characters dialogue and then assemble the back-and-forth in your preferred editing software and wah la!
-
This workflow feels like it has ton of potential for children’s entertainment, animated shorts, comedy series, and mascots for brands.
If you’re looking for some help tapping into the power of AI content then let’s talk!
🛠️ Tools we used this week
Runway (you’ll need the paid plan, $15 for 30 days - you can cancel whenever)