Sonic Flowers – TouchDesigner x StableDiffusion Tutorial 1

In this tutorial we’re having a first look at how to integrate the image-generation AI tool #stablediffusion into TouchDesigner. We’re creating an audio-reactive texture in an independant component to be able to do frame by frame animation without worrying about losing frames. TD is running in real-time, but the process itself is not (yet) real-time, as one frame takes 5-10 seconds to be created.

Please mind that this is very experimental and just one way (my current approach). I am very open to hear about ways of improving this or ideas on how to expand it. See this more of a starting point and inspiration than a perfectly refined technique.

Make sure that you download the automatic1111 WebUI (below) and some models to work with. The models should be placed inside “stable-diffusion-webuimodelsStable-diffusion”.
Only tested on Windows, running on an Nvidia RTX 3070 Notebook version.

dotsimulate’s SD_API: https://www.patreon.com/posts/sd-api-1-22-85238082
automatic1111: https://github.com/AUTOMATIC1111/stable-diffusion-webui
Models: https://civitai.com/
Parameters explained: https://blog.openart.ai/2023/02/13/the-most-complete-guide-to-stable-diffusion-parameters/
Create your own API: https://www.youtube.com/watch?v=4khcLvGjoX8
IG Post: https://www.instagram.com/reel/Ctue_S_o9eF/?igshid=NTc4MTIwNjQ2YQ==

The prompt used for this example:
ultrarealistic surreal flowers, ultra detailed, texture, generative art, focus, wes anderson, kodak, light and shadow

When you’re working with animation in the independant comp, instead of e.g. absTime.seconds, it makes sense to use me.time.seconds.

—–
00:00 Intro & Overview
02:40 Independant Comp & Audio Setup
07:50 Audioreactive Circles
16:00 Circles Processing
18:30 SD API Component
27:54 Feedback Loop
32:58 Frame by Frame
36:52 Exporting
40:34 Additional Info
43:16 Outro
—–

Files, exclusive content and more:
https://www.patreon.com/elekktronaut

The special thanks is integrated at the end of the video ❤
If you have any questions, feel free to ask.

IG: @elekktronaut
https://www.instagram.com/elekktronaut/