processing package

The processing package provides the execution layer of the VISWIR pipeline. It manages batch runs, task orchestration, SQL logging, and interruption handling. This package is responsible for coordinating the fusion tasks and ensuring results are stored and retrievable.

Submodules

processing.batch_runner module

Main batch runner for executing multiple fusion tasks in sequence.

Batch processing orchestration for running VISWIR on directories.

processing.batch_runner.process_folder(visible_folder: str, swir_folder: str, output_dir: str, batch_size: int = 20, ref_image_path: str | None = None, params: dict | None = None, run_detection: bool = False, ground_truth_path: str | None = None, save_output: bool = True)[source]

Orchestrate batch processing of image fusion tasks and save results to CSV.

This function performs the following steps: 1. Discover visible and SWIR image files in the provided folders. 2. Validate the number of images and optionally load a reference image. 3. Generate fusion tasks with the given parameters. 4. Execute tasks in parallel batches with progress tracking. 5. Save computed metrics to a CSV file.

Parameters:

  • visible_folder (str) – Path to the folder containing visible images.

  • swir_folder (str) – Path to the folder containing SWIR images.

  • output_dir (str) – Directory where results (CSV and optional outputs) will be saved.

  • batch_size (int, default=20) – Number of tasks to process in parallel per batch.

  • ref_image_path (str or None, optional) – Path to the reference image (optional).

  • params (dict or None, optional) – Fusion parameters to apply. If None, defaults are used.

  • run_detection (bool, default=False) – Whether to run YOLO detection in addition to metric computation.

  • ground_truth_path (str or None, optional) – Path to ground truth annotations (used if detection is enabled).

  • save_output (bool, default=True) – Whether to save intermediate outputs (images, annotations).

Returns:

Results are written to a CSV file in the output directory.

Return type:

None

Raises:

  • ValueError – If the number of visible and SWIR images does not match.

  • FileNotFoundError – If the reference image path is provided but cannot be loaded.

Notes

  • The CSV file is saved as combinations.csv in the output directory.

  • Metrics include both no-reference and full-reference metrics.

  • Progress is displayed in the console with a live progress bar.

processing.batch_runner.process_image_wrapper(task: FusionTask) → ProcessResult[source]

Execute the fusion of a pair of images from a FusionTask, compute metrics, and optionally run detection. Returns a ProcessResult object.

This function performs the following steps:

  1. Fusion of visible and SWIR images using the provided parameters.

  2. Loading of the reference image (if available).

  3. Computation of quality metrics (fusion, visible, SWIR).

  4. Optional YOLOv8 detection and F1-score computation.

  5. Return of results in a ProcessResult object.

Parameters:

task (FusionTask) –

Task object containing:

  • visible_path (str) - Path to the visible image.

  • swir_path (str) - Path to the SWIR image.

  • ref_image_path (str or None) - Path to the reference image (optional).

  • ground_truth_path (str or None) - Path to ground truth annotations (optional).

  • params (dict) - Fusion parameters (facteur_swir, beta, level, apply_gamma, gamma_value).

  • save_output (bool) - Whether to save intermediate results.

  • run_detection (bool) - Whether to run YOLO detection.

  • output_dir (str or Path) - Directory for saving outputs.

Returns:

Object containing:

  • visible_path (str)

  • swir_path (str)

  • ground_truth_path (str or None)

  • params (dict) - Fusion parameters used.

  • metrics_fusion (dict) - Metrics computed on the fused image.

  • metrics_visible (dict) - Metrics computed on the visible image.

  • metrics_swir (dict) - Metrics computed on the SWIR image.

  • error (str or None) - Error message if the process failed.

Return type:

ProcessResult

Notes

  • If fusion fails, returns a ProcessResult with the error message.

  • If detection is enabled, YOLOv8 is run on fused, visible, and SWIR images.

  • Memory cleanup is performed at the end to avoid leaks.

processing.interruption module

Utilities for handling interruptions and safely stopping long-running jobs.

Interruption handling and signal management for long-running batch processes.

processing.interruption.save_last_params()[source]

Save the last combination of parameters before program termination.

This function writes the current content of the global last_params dictionary into a JSON file located at config/last_params.json.

Notes

  • The file is overwritten each time this function is called.

  • Called automatically when a keyboard interruption (Ctrl+C) is detected.

processing.interruption.signal_handler(sig, frame)[source]

Handle SIGINT (Ctrl+C) signals by saving the last parameters and exiting.

Parameters:

  • sig (int) – Signal number (e.g., signal.SIGINT).

  • frame (frame object) – Current stack frame (unused).

Notes

  • Logs a warning message before saving.

  • Calls save_last_params() to persist the last parameters.

  • Exits the program with status code 0.

processing.sql_runner module

SQL interface for logging and retrieving results from the database.

SQL database orchestration for batch runs and logging.

processing.sql_runner.process_folder_sql(visible_folder: Path, swir_folder: Path, output_dir: Path, batch_size: int = 10, ref_image_path: Path | None = None, save_output: bool = False, run_detection: bool = True, ground_truth_path: Path | None = None, params: dict | None = None, workers: int = 1) → None[source]

Process a folder of visible and SWIR images for SQL-based analysis with resume capability. Handles matching between images (e.g. 0000_rgb.jpg) and GT (e.g. 0000.xml).

This function orchestrates the batch processing of image fusion tasks, computes metrics, and stores results directly into a SQLite database (results.db). It includes a robust “resume” feature that checks the database for previously processed images to avoid redundant calculations.

Steps

  1. Create output directory and initialize SQLite database session.

  2. Query the database to identify images that have already been processed.

  3. Discover visible and SWIR image files in the provided folders.

  4. Filter out files that are already present in the database (Skip logic).

  5. Match remaining images with their corresponding Ground Truth files (if any).

  6. Generate tasks in memory for the remaining matched pairs.

  7. Execute tasks in batches.

  8. Save results into the database.

param visible_folder:

Path to the folder containing visible images.

type visible_folder:

Path

param swir_folder:

Path to the folder containing SWIR images.

type swir_folder:

Path

param output_dir:

Directory where the SQLite database and logs will be saved.

type output_dir:

Path

param batch_size:

Number of tasks to process per batch. A smaller size is recommended for stability on large datasets to allow frequent memory clearing and DB commits.

type batch_size:

int, default=10

param ref_image_path:

Path to the reference image (optional).

type ref_image_path:

Path or None, optional

param save_output:

Whether to save intermediate outputs (images, annotations).

type save_output:

bool, default=False

param run_detection:

Whether to run YOLO detection in addition to metric computation.

type run_detection:

bool, default=True

param ground_truth_path:

Path to ground truth annotations (used if detection is enabled).

type ground_truth_path:

Path or None, optional

param params:

Fusion parameters to apply. If None, parameters are loaded from config.

type params:

dict or None, optional

param workers:

Number of parallel processes to use. - If set to 1 (default): Runs in sequential mode (Loop). Most stable for Windows/WSL and large images. - If set to > 1: Runs in parallel using ProcessPoolExecutor. Faster for small datasets.

type workers:

int, default=1

returns:

Results are inserted into the SQLite database.

rtype:

None

raises ValueError:

If the number of visible and SWIR images does not match.

raises FileNotFoundError:

If the reference image path is provided but cannot be loaded.

Notes

  • Results are stored in results.db inside the output directory.

  • The function automatically detects the table name (e.g., fusion_results) to perform the skip check.

  • Progress is displayed in the console with a live progress bar.

processing.task_manager module

Task manager for coordinating individual fusion tasks and their dependencies.

Task manager for batch processing task coordination.

processing.task_manager.batchify_tasks(tasks: List[FusionTask], batch_size: int)[source]

Split a list of tasks into smaller batches of a given size.

Parameters:

  • tasks (list of FusionTask) – List of tasks to split.

  • batch_size (int) – Number of tasks per batch.

Yields:

list of FusionTask – A batch of tasks.

Examples

>>> tasks = [FusionTask(...), FusionTask(...), FusionTask(...)]
>>> for batch in batchify_tasks(tasks, batch_size=2):
...     print(len(batch))
2
1
processing.task_manager.generate_tasks_in_memory(visible_files: List[Path], swir_files: List[Path], ref_image_path: Path | None, params: dict, run_detection: bool, ground_truth_path: Path | None, save_output: bool, output_dir: Path) → List[FusionTask][source]

Generate a list of FusionTask objects in memory (fixed mode or exploration mode).

This function creates tasks for each pair of visible and SWIR images, either with a single fixed set of parameters or by exploring all combinations of parameter ranges.

Parameters:

  • visible_files (list of Path) – List of visible image file paths.

  • swir_files (list of Path) – List of SWIR image file paths.

  • ref_image_path (Path or None) – Path to the reference image (optional).

  • params (dict) – Fusion parameters. Can be: - Fixed mode: contains single values for each parameter. - Exploration mode: contains ranges (min, max, step) or lists of values.

  • run_detection (bool) – Whether to enable YOLO detection.

  • ground_truth_path (Path or None) – Path to ground truth annotations (optional).

  • save_output (bool) – Whether to save intermediate results.

  • output_dir (Path) – Directory where outputs will be saved.

Returns:

List of tasks ready for execution.

Return type:

list of FusionTask

Notes

  • In fixed mode, only one task per image pair is generated.

  • In exploration mode, all parameter combinations are expanded into tasks.

  • Ground truth files are aligned with visible images using prepare_ground_truth_list.

Module contents

The top-level processing module re-exports selected functions and classes from its submodules for convenience.

processing package

Batch processing and pipeline orchestration: - SQL-based processing - Fixed-parameter runs - Task management