So the fix here was to run a backup of my old Joomla3.9 web and tell the J4 script to use the backup's DB and folder locations as source.
Yes, this is also possible. Its not how it was ment to be but it will work. But you still have to make sure that at the source folder location a joomgallery.xml is available.
You dont need a full joomgallery 3 to be installed there. Just copy the joomgallery.xml to the location where it would have to be.
administrator/components/com_joomgallery/joomgallery.xml