I think Kohya_ss really hates non Nvidia GPUs. Tried with a lot of different configurations, most of them resulted in out of memory error. (I am on a 16GB AMD Radeon GPU!). Even with successful runs (extremely slow and loud from coil whines) results were undesirable. I really wish there could be a Kohya_ss alternative software that doesn’t sucks and ACTUALLY WORKS.
Speculations
The problem of training on non Nvidia GPU is that there are just not enough demand for it. The guide on the Kohya_SS is out dated and nobody mentions it in the issue thread. To make matter worst. It seems like Bitsandbytes(a library for optimising the training process to make it not out of memory so easily) is backed by Nvidia and I had a weak feeling that Nvidia is trying to prevent the author from making it support AMD and Intel GPU hardware.
There’s this alternative fork made by the AMD ROCm official account. But there’s very little support and no guide whatsoever on using it with Kohya_SS.
https://github.com/ROCm/bitsandbytes
Model Merging
As an alternative effort, I decided to experiment with model merging – It’s done in CPU and RAM so no graphics card issue might happen. I merged two of the best performing models of my collection with two of my favourite art style Loras. I wanted to add some of my own SD 1.5 Loras but they are not compatible with SDXL.
Results
I got absolutely burnt out dealing with Kohya_SS. But at last at least I got a pretty cute checkpoint. I personally think it’s a pretty high performance checkpoint.
For my personal safety, I will not be showing the name of two artists that I merged.
Verdict
I should buy an RTX5090 once it’s out. I am enough dealing with AMD and Intel non-senses. Nvidia made their GPU more open under Linux anyway so why not try it out(by moving most of the closed-source part into their GSP, still closed-source).
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Only if Nvidia can sell their cards cheaper…
Leave a Reply