Render2Photo Converter Tool

This project transforms 3D renders of office interiors into photorealistic images using LoRA-Adapter.

Frontend view:

Demo:

Example transformations:

Input Photo	Output Photo

Images by PerOla Hammar and Gia Tu Tran, posted on Unsplash.

Getting Started

1. Create and activate a virtual environment:

   # Create and activate virtual environment
   python -m venv venv
      
   # For Mac/Linux:
   source venv/bin/activate
   # For Windows:
   venv\Scripts\activate
   
   # Install requirements
   pip install -r requirements.txt
   

2. You can work with the script using three methods:

   Option A: Batch Processing Script

   • Place images in the task-images folder (images in subfolders will not be processed)
   • Run the script:

     python scripts/enhance-image-lora_sdxl_render2photo_enhanced.py
     

   • Results will be saved to the processed-images folder
   • A settings.json file will be generated for reproducibility
   • You can specify custom input and output directories using settings in the script

   Option B: API

   • Start the API server:

     python api.py
     

   • Open Postman and import the collection: enhance_image.postman_collection.json (link)
   • Run the 'Process image' request (adjust the URL if necessary)

   Option C: Web-Interface

   • Start the API server:

     python api.py
     

   • Change to the web-interface directory, install the requirements and run the server:

     cd website/render2image-processor/ 
     npm i
     npm run dev -- --host 0.0.0.0
     

   • Open the URL in your browser that is displayed after you run the server

3. Optional: After you finished working, clear cache to free up disk space:

   rm -rf ~/cache/huggingface/hub/models--stabilityai--sdxl-vae
   rm -rf ~/cache/huggingface/hub/models--stabilityai--stable-diffusion-xl-base-1.0
   

   (This removes downloaded Stable Diffusion models that may occupy significant space)

Training on your own data

To train your own LoRA-Adapter, you need to do the following:

1. Collect the image pairs - 3d-renders and photos and post them on Hugging Face using the following structure:

   • photos
     • 0.png
     • 1.png
     • ...
   • renders
     • 0.png
     • 1.png
     • ...
   • metadata.json

     {
       {
         id: 0,
         photo_path: "path",
         render_path: "path",
         ...
       },
       ...
     }
     

2. Add your Hugging Face token with this dataset to the .env file under HUGGING_FACE_TOKEN variable

3. Run the script scripts/final-fine-tune-lora-with-descaled-aspect-ratio.py

4. This will produce a LoRA-adapter, that you can use for the inference script

Parameters

The following parameters can be customized: