The topic of AI-generated images has come up here several times already, and I personally find it quite interesting how it can expand our hobby.
So far, the discussion has always focused on text-to-image or image-to-image transformation. However, there's another option: image-to-video, the creation of a video clip based on an image.
I've now tried it out and experimented a bit with images from my second-to-last diorama (the Villa by the Sea). I selected photos that suggested some kind of movement or sequence, so that something meaningful could actually happen in the video.
As I'll show you below, you get some quite interesting results. You can see what's already possible these days, but also some limitations. It's important to remember that, in my estimation, the underlying AI models were essentially trained on real photos – the images of my diorama, despite my best efforts at realism, are obviously miles away from images with real people, etc. Considering that, I find what the AI has produced quite impressive.
(Side note: Unfortunately, the forum's BB code does not allow videos from any server to be embedded directly; you either have to go via YouTube or, as I did, first convert the videos into animated GIFs).
General
Bringing a Diorama to Life
10 posts
• Page 1 of 1
-
Santi Pérez likes this post. •
- Reply
-
sberry
- Posts: 1174
- Member since:
12 Mar 2010, 20:37
Experiment 1
Photo:
Text Prompt:
The rowers in an ancient rowing boat struggle against the swell to make headway. A dolphin plays in the waves and leaps out of the water.
Result:
Commentary:
I figured that a rowboat in full motion combined with a dolphin leaping out of the water should be enough action for the AI to work with. Therefore this was the very first image I ever used to conduct a video experiment.
My initial reaction upon seeing the result was, "Wow!" While the AI evidently didn't quite know what to do with the dolphin, the movement of the rowboat is actually rendered quite well. Unfortunately, however, the AI also detected – with ruthless precision – the exact boundary where the diorama's water surface meets the poster background. The way these two planes shift against one another looks bizarre and ridiculous.
Photo:
Text Prompt:
The rowers in an ancient rowing boat struggle against the swell to make headway. A dolphin plays in the waves and leaps out of the water.
Result:
Commentary:
I figured that a rowboat in full motion combined with a dolphin leaping out of the water should be enough action for the AI to work with. Therefore this was the very first image I ever used to conduct a video experiment.
My initial reaction upon seeing the result was, "Wow!" While the AI evidently didn't quite know what to do with the dolphin, the movement of the rowboat is actually rendered quite well. Unfortunately, however, the AI also detected – with ruthless precision – the exact boundary where the diorama's water surface meets the poster background. The way these two planes shift against one another looks bizarre and ridiculous.
-
sberry
- Posts: 1174
- Member since:
12 Mar 2010, 20:37
Experiment 2
Photo:
Text Prompt:
A man and a woman in ancient Roman attire stand in the hall of a seaside villa. They are examining a golden equestrian statue.
Result:
Commentary:
I think the animation of the two figures in the center of the image is really fantastic. Furthermore, the AI animated the construction worker in the background on the left – even though I hadn't mentioned him in the text prompt – and went right ahead and invented a colleague for him, who appears on the right side. On one hand, this demonstrates just how powerful this tool is; on the other hand, however, these invented details make it difficult to precisely control the specific result you will receive.
A particularly curious detail is the dark line visible in the original image within the upper right-hand archway: here, the poster background was poorly positioned, leaving the edge of the poster visible in my photograph. For this reason, I had deemed the photo unusable and used it solely for this AI experiment – yet the AI has now detected this very flaw with uncanny precision and reproduced, even inflated it in the video. Absurd!
Photo:
Text Prompt:
A man and a woman in ancient Roman attire stand in the hall of a seaside villa. They are examining a golden equestrian statue.
Result:
Commentary:
I think the animation of the two figures in the center of the image is really fantastic. Furthermore, the AI animated the construction worker in the background on the left – even though I hadn't mentioned him in the text prompt – and went right ahead and invented a colleague for him, who appears on the right side. On one hand, this demonstrates just how powerful this tool is; on the other hand, however, these invented details make it difficult to precisely control the specific result you will receive.
A particularly curious detail is the dark line visible in the original image within the upper right-hand archway: here, the poster background was poorly positioned, leaving the edge of the poster visible in my photograph. For this reason, I had deemed the photo unusable and used it solely for this AI experiment – yet the AI has now detected this very flaw with uncanny precision and reproduced, even inflated it in the video. Absurd!
-
sberry
- Posts: 1174
- Member since:
12 Mar 2010, 20:37
Experiment 3
Photo:
Text Prompt:
Construction workers in ancient Rome load a large beam onto a cart.
Result:
Comment:
Of the three results shown, this one is somehow the least spectacular – yet actually the best: In this instance, the AI adhered very precisely to the text prompt and didn't tinker with other parts of the image.
Photo:
Text Prompt:
Construction workers in ancient Rome load a large beam onto a cart.
Result:
Comment:
Of the three results shown, this one is somehow the least spectacular – yet actually the best: In this instance, the AI adhered very precisely to the text prompt and didn't tinker with other parts of the image.
-
sberry
- Posts: 1174
- Member since:
12 Mar 2010, 20:37
impressive performance, I can hardly imagine what will be the AI possibility in few years from now !!
-
Iceman1964
- Posts: 512
- Member since:
26 Dec 2020, 17:43
Very interesting experiments. I agree, with you concerning example 3. But the rowing men in particular are also convincing.
-
MABO
Supporting Member (Gold) 
- Posts: 9617
- Member since:
12 May 2008, 18:01
Help keep the forum online!
or become a supporting member
Iceman1964 wrote:impressive performance, I can hardly imagine what will be the AI possibility in few years from now !!
Yes, that will be cooler and also more dangerous in combination with social media, than it is already now.
-
MABO
- Supporting Member (Gold)
- Posts: 9617
- Member since:
12 May 2008, 18:01
Today I have read that the Google search as we know it, providing a list of search results, will be terminated. In the future, they want you to engage in a "dialogue" with their AI Gemini. Oh my.
-
Santi Pérez likes this post. •
- Reply
-
sberry
- Posts: 1174
- Member since:
12 Mar 2010, 20:37
-
elegantmess
- Posts: 158
- Member since:
05 Oct 2022, 08:13
Well, I’ve just been praising Steve (steve_pickstock) in another section for his ChatGPT creations, and now I’ve come across these of yours, Stephan. 
Although all three ‘minifilms’ are amazing, my favourite is the one in the middle, with the figures gazing at the equestrian statue and the two workmen labouring outside.
Great works, my friend!
Santi.
Although all three ‘minifilms’ are amazing, my favourite is the one in the middle, with the figures gazing at the equestrian statue and the two workmen labouring outside.
Great works, my friend!
Santi.
-
Santi Pérez
- Posts: 2739
- Member since:
28 Aug 2016, 19:42
10 posts
• Page 1 of 1