I recommend that you first generate the desired image, and then change the face using LORA in the inpaint tab:
Inpaint settings:
Resize mode - Just resize
Mask mode - Inpaint masked
Masked content - original
Inpaint area - Only masked
Width and Height - 512x512 or higher
Don't forget to switch to a random seed
Experiment with Denoising strength (0.3 - 0.9) and Only masked padding, pixels (10-100)