Introduction
In software engineering, even the simplest ideas can evolve into complex, multi-layered projects when faced with the realities of production environments. This article documents how the idea of "a simple website that cuts clips from YouTube videos" turned into an asynchronous, scalable and resilient system. Our project, yt2clip.live, is the concrete output of this journey.
Chapter 1: Architectural Design – Why Asynchronous?
Video processing is inherently time- and resource-intensive; it is unacceptable for a user to wait minutes for an HTTP response. Therefore, the system is based on an asynchronous architecture that prioritises UX.
The architecture has three fundamental components:
- Frontend (Next.js & Vercel): the UI that interacts with users, initiates requests and displays results in real-time.
- Backend API (Spring Boot & Java): the orchestrator that validates requests, packages them as tasks and immediately delegates them to the queue.
- Worker (Python & yt-dlp/ffmpeg): the heavy-lifter that downloads, cuts and encodes videos.
Communication is handled by Redis and WebSocket:
- Redis: acts as both a task queue (List) and Pub/Sub channel.
- WebSocket: provides a real-time, bidirectional channel to push job status to the client.
Chapter 2: Production Battles & Engineering Solutions
Deployment to a DigitalOcean droplet surfaced seven critical challenges:
1. Security Layer – SSL/TLS Deficiency
Problem: The frontend tried to open a WebSocket over wss:// while Nginx only listened on port 80.
Solution: Installed Certbot, obtained a Let's Encrypt certificate and re-configured Nginx to listen on 443 and proxy WebSockets.
2. Bot Detection – Google's Firewall
Problem: Requests made by yt-dlp were blocked by Google because they originated from a data-centre IP.
Solution: Implemented a dedicated "cookie-renewer" service with Selenium & headless Chrome to refresh cookies.txt.
3. Automation Barriers – Google's Interactive Verification
Problem: The cookie-renewer got stuck on "Verify it's you" challenges.
Solution: Enabled 2FA and used Google App Passwords; persisted the Chrome profile in a Docker volume.
4. Concurrency – "Thundering Herd" Problem
Problem: Multiple workers logging in simultaneously locked the Google account.
Solution: Each worker sleeps a random 0-59 s on startup to spread authentication attempts.
5. File-System Inconsistencies – Volume Sync Delay
Problem: API sometimes couldn't find a freshly created clip and returned 404.
Solution: Workers call os.fsync() after writing to guarantee the data is flushed to disk before emitting the "completed" event.
6. Resource Management – CPU/RAM Explosions
Problem: ffmpeg could exhaust CPU & RAM, freezing the server.
Solution: Added deploy.resources.limits in docker-compose.yml to cap container resources.
7. Quality Control – yt-dlp Format Selection
Problem: The service downloaded the best progressive format (360/720 p) instead of the requested 1080 p.
Solution: Used the rule 'bestvideo[height<=1080]+bestaudio/best' to fetch separate video & audio streams and merge them.
Conclusion
yt2clip.live proved that delivering a modern web service is far more than writing code; infrastructure, security, concurrency and error management are equally critical. Each obstacle led to an engineering workaround that made the system more resilient and efficient.