I've never ventured into using Stable Diffusion locally, but the idea of trying it out with Houdini caught my attention. When combining Stable Diffusion with ControlNet in Houdini, the results are quite intriguing. It's a powerful technique that, surprisingly, isn't overly complicated to set up.
In order to get this right it requires a fair bit of patience , since there's nothing like a single click solution, it requires a lot of playing with installs and settings . But its pretty impressive once you get it right .
Installation
Need to instal stable-diffusion-webui
Install Automatic1111 ( follow this tutorial in pararell )
Step 1: Install python > https://apps.microsoft.com/store/detail/python-310/9PJPW5LDXLZ5
Step 2: Install git > https://git-scm.com/download/win
Step 3: Clone web-ui > git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
Step 4: Download a model file https://stable-diffusion-art.com/models/#Stable_diffusion_v15
Step 5: Run webui to see if it works C:\Users\YOURuSER\stable-diffusion-webui\webui-user.bat
Step 6: Add --api command line key to the webui-user.bat
Step 6: Download control Net https://github.com/Mikubill/sd-webui-controlnet
Step 7: Download from Huging Face the ControlNet models
terminal: git lfs clone https://huggingface.co/lllyasviel/ControlNet-v1-1
Keep an eye on terminal, since most of the background work is happening there
Install asset libraries from /hda/ and use assets inside Houdini or use the provide hip file
The Hip provide file and the hda are great starting points and do some of the heavy lifting for you
This video explains it well although a bit lengthy