BackToBasics

Overview

Training and fine-tuning models is one thing; running them reliably under load is another. This project is a background removal as a service built to learn production ML infrastructure end-to-end: clean architecture, async orchestration, object storage, observability, and horizontal scaling on Kubernetes (Minikube).

Workflow: upload an image → Celery task on Redis → PyTorch inference (DeepLabV3 ResNet50, CPU) → RGBA PNG returned → input and output artifacts stored in S3-compatible MinIO.

Demos

End-to-end pipeline

Load test and horizontal scaling

Current status

The service is working end-to-end on Docker Compose (8 services) and Minikube. Workers autoscale via HPA (1→3 replicas at 75% CPU), with Prometheus scrape and a Grafana demo dashboard (9 panels). Week 8 will add a React frontend (upload → poll → display result).

Architecture

The project follows a hexagonal (ports & adapters) structure across three layers:

• Domain — immutable dataclasses (ImageMetadata, ProcessedImage) and abstract interfaces: ImageProcessor (Strategy pattern contract) and StoragePort (storage abstraction).
• App services — use-case orchestrators: FileService (delegates to StoragePort) and ImageProcessorService (delegates to the active processor strategy).
• Infrastructure — FastAPI router (POST /images/upload, POST /images/process, GET /images/process/{task_id}), Celery process_image_task, S3Storage (boto3 → MinIO), Prometheus metrics (infrastructure/metrics.py + FastAPI instrumentator), Pydantic schemas, PyTorchBackgroundRemover, and logging config.

Dependency injection is handled via FastAPI's Depends() factory functions, keeping the HTTP layer decoupled from business logic. Files live in MinIO as inputs/ and outputs/ object keys; the worker loads the model once at startup (singleton) and exposes custom metrics on port 8000.

Key challenge

Inference is CPU-bound and slow. Without scaling, tasks accumulate and latency explodes. The fix is horizontal scaling of Celery workers via HPA: more requests → more worker pods → stable throughput under load.

Observability & scale

• Metrics — HTTP latency (instrumentator), Celery task duration, ML inference time, mask confidence histogram.
• Compose — Prometheus scrapes API (:5000) and worker (:8000); Grafana provisions backtobasics.json (9 panels: app + K8s/HPA/resources).
• Kubernetes — manifests under k8s/: api, worker + HPA (1→3 @ 75% CPU), redis, minio, flower, prometheus, grafana, kube-state-metrics; cAdvisor/node scrape for pod CPU/RAM; start-stack.ps1 for Minikube on Windows (UI via minikube service --url tunnels).
• Load testing — Locust against the API; used to validate HPA scale-out under charge.

Roadmap

• ✅ Week 1 — Python foundations + project structure: list, tuple, dict, set, comprehension, generators, classes, dataclasses, dunder methods — repo init, FastAPI structure, image upload endpoint, local storage.
• ✅ Week 2 — Advanced OOP & design patterns: factory, strategy, singleton, observer, hexagonal architecture, dependency injection, Git merge/rebase/stash/hooks — ImageProcessorService, logging decorator, basic background removal endpoint.
• ✅ Week 3 — ML pipeline / Computer Vision: PyTorch tensors, dataloaders, preprocessing, batch inference, GPU/CPU, postprocessing — full DeepLabV3 pipeline: load → preprocess → infer → postprocess → RGBA PNG.
• ✅ Week 4 — Async & Orchestration (Docker Transition): asyncio, event loop, Celery + Redis, Docker, docker-compose, container networking — Dockerfile, docker-compose (api + worker + redis + flower), background removal migrated into a Celery task.
• ✅ Week 5 — Code quality & MLOps: Ruff (linting), MLflow (model evaluation), CI/CD basics — Ruff integrated, MLflow evaluation pipeline on ECSSD dataset, GitHub Actions running lint + tests on every push.
• ✅ Week 6 — Object Storage: S3 protocol, MinIO (local S3), boto3 — replace local uploads/ with a MinIO bucket; StoragePort / S3Storage; task I/O entirely via object keys.
• ✅ Week 7 — Scaling & Infrastructure: Prometheus + Grafana (9-panel demo dashboard), custom Celery/ML metrics, Docker Compose extended to 8 services, Kubernetes (Minikube), HPA worker 1→3 @ 75% CPU, kube-state-metrics + cAdvisor scrape, Locust load tests, start-stack.ps1.
• Week 8 — Frontend React: React fundamentals, REST polling — upload form, async status polling, result display from MinIO. Full MVP: Upload → Celery task → Poll → Result (Tailwind UI).

Weekly Log

Week 1 — March 2026

Set up project structure and Poetry environment. Implemented the domain models (ImageMetadata, ProcessedImage) as frozen dataclasses with format validation. Built FileService and wired up the FastAPI app with a working POST /images/upload endpoint that persists bytes and returns file metadata.

Week 2 — March 2026

Introduced the ImageProcessor ABC as a Strategy pattern contract. Implemented DummyProcessor, WhiteProcessor, and CompositeProcessor (Composite pattern for chaining processors). Added ImageProcessorService as the use-case layer. Wired POST /images/process with FastAPI Depends() injection. Added a time_logger decorator for performance tracking, and configured a dual console + file logger.

Week 3 — March–April 2026

Integrated PyTorchBackgroundRemover using DeepLabV3-ResNet50 with pretrained COCO weights. Built a clean three-stage pipeline: preprocess (PIL → tensor + model transforms), inference (torch.no_grad), postprocess (logit upsampling → argmax → RGBA alpha mask → PNG bytes). Moved model instantiation to FastAPI lifespan to avoid reloading weights on every request. Aligned test suite with the real pipeline. Background removal fully working end-to-end.

Week 4 — April 2026

Containerised the full application. Wrote the Dockerfile (Poetry install, PYTHONPATH config) and docker-compose.yml with four services: api, worker, redis, and flower. Migrated the background removal inference into a process_image_task Celery task — the API now returns a task_id immediately and the worker processes asynchronously. Added a shared uploads/ volume between api and worker containers (later replaced by MinIO in Week 6). POST /images/process and GET /images/process/{task_id} polling endpoint fully wired.

Week 5 — April 2026

Focused on code quality and model evaluation tooling. Installed Ruff as a dev dependency (default E + F rules). Rewrote the test suite to match the async API: test_process_enqueues_task mocks process_image_task.delay and verifies the endpoint returns task_id + PENDING; additional tests cover the polling endpoint for PENDING and SUCCESS states. Set up an MLflow evaluation pipeline in project/evaluation/ — one MLflow run per model on the ECSSD salient object dataset (1000 images with ground truth masks), logging mIoU, std IoU, and average inference time as comparable metrics across future model implementations. MLflow is intentionally kept out of the production code path. GitHub Actions CI runs Ruff + Pytest on push/PR to main.

Week 6 — April 2026

Introduced object storage to replace the local uploads/ bind mount. Defined a StoragePort interface in the domain layer and implemented S3Storage in infrastructure using boto3 against a MinIO instance. ImageMetadata.path was renamed to object_key throughout. FileService now delegates all I/O to StoragePort, keeping the app layer storage-agnostic. The Celery task reads the input object, runs inference, and writes the result back — all via S3. docker-compose extended with minio (ports 9000/9001) and minio-init (one-shot bucket creation), bringing the service count to six. Tests updated to mock the storage layer via app.dependency_overrides.

Week 7 — May 2026

Added production-style observability and local orchestration. Defined custom Prometheus histograms/counters in metrics.py (Celery task duration, ML inference, mask confidence) alongside the FastAPI instrumentator on /metrics. Extended Compose with prometheus and grafana (eight services total). Built a Grafana dashboard with nine demo-oriented panels (HTTP/Celery/ML plus K8s pod phase, HPA, top CPU/memory, cluster utilisation). Ported the stack to Kubernetes under k8s/: Deployments/Services, worker HPA (1→3 @ 75% CPU), kube-state-metrics, Prometheus RBAC scrape of cAdvisor and kubelet, Grafana provisioning via ConfigMap. Wrote start-stack.ps1 to bootstrap Minikube on Windows (build images, apply manifests, open tunnels for api/docs, Flower, Grafana, MinIO console). Added Locust load tests to stress the API and observe autoscaling. Removed the last dependency on local bind mounts — all file I/O goes through MinIO.