diff --git a/docs/Cosmos_Complete_Documentation.md b/docs/Cosmos_Complete_Documentation.md new file mode 100644 index 0000000..32e6577 --- /dev/null +++ b/docs/Cosmos_Complete_Documentation.md @@ -0,0 +1,2638 @@ +# Cosmos Storage Platform — Complete Technical Documentation + +> **Version:** Production (October 2026) +> **Server:** `cosmos.linuxguy.space` / `vshno.linuxguy.space` +> **Maintainer:** System Administrator +> **Last Updated:** October 14, 2026 +> **Document Type:** Full Architecture, Operations & Code Reference + +--- + +## Table of Contents + +1. [Project Overview & Mission](#1-project-overview--mission) +2. [Infrastructure & Server Setup](#2-infrastructure--server-setup) +3. [Filesystem Layout & Codebase Map](#3-filesystem-layout--codebase-map) +4. [Service Architecture Diagram](#4-service-architecture-diagram) +5. [Nginx — Web Server & Reverse Proxy](#5-nginx--web-server--reverse-proxy) +6. [MinIO — S3-Compatible Object Storage](#6-minio--s3-compatible-object-storage) +7. [ScyllaDB — Database Layer](#7-scylladb--database-layer) +8. [The Go Backend — Cosmos Portals (`portals-go`)](#8-the-go-backend--cosmos-portals-portals-go) +9. [Systemd Services — Boot & Process Management](#9-systemd-services--boot--process-management) +10. [Frontend Templates — Page-by-Page Breakdown](#10-frontend-templates--page-by-page-breakdown) +11. [The Oort Console — S3 Storage Dashboard (SPA)](#11-the-oort-console--s3-storage-dashboard-spa) +12. [Authentication & Session Flow](#12-authentication--session-flow) +13. [File Upload & Multipart Transfer](#13-file-upload--multipart-transfer) +14. [Analytics System](#14-analytics-system) +15. [CDN Proxy & Public File Serving](#15-cdn-proxy--public-file-serving) +16. [MinIO User Provisioning Flow](#16-minio-user-provisioning-flow) +17. [Disk Partitions & Data Storage Locations](#17-disk-partitions--data-storage-locations) +18. [Docker & ScyllaDB Container](#18-docker--scylladb-container) +19. [Gitea — Git Server Integration](#19-gitea--git-server-integration) +20. [Security Posture, Flaws & Recommendations](#20-security-posture-flaws--recommendations) +21. [Boot Sequence & Service Startup Order](#21-boot-sequence--service-startup-order) +22. [Troubleshooting Runbook](#22-troubleshooting-runbook) +23. [API Reference](#23-api-reference) +24. [Database Schema Reference](#24-database-schema-reference) +25. [Environment Variables & Secrets](#25-environment-variables--secrets) +26. [Known Issues & Broken Features](#26-known-issues--broken-features) +27. [Quick Reference: Common Operations](#27-quick-reference-common-operations) +28. [Cosmos Orbit — Application Platform & Container Runtime (PaaS)](#28-cosmos-orbit--application-platform--container-runtime-paas) + +--- + +## 1. Project Overview & Mission + +**Cosmos Cloud** is a fully self-hosted, sovereign cloud computing suite built on Debian Linux with zero third-party cloud-provider dependencies. The platform unites three primary computing pillars: + +1. **Oort (S3 Object Storage)** — Powered by MinIO, accessible via standard AWS S3 SDK/CLI tools and the reactive Oort Console SPA with analytics and custom CDN domains. +2. **Forge (Git Version Control)** — Powered by self-hosted Gitea, providing fast Git repository hosting, code reviews, issue tracking, and webhook triggers. +3. **Orbit (PaaS & Container Runtime)** — A sovereign application platform providing automated stack detection, Docker container virtualization, 10-port quota allocation (ports 4100–4999), live telemetry, custom domain reverse-proxy routing, and direct Oort S3 backup snapshots. + +### Core Capabilities: +- **S3-compatible object storage** — powered by MinIO, accessible via standard AWS S3 SDK/CLI tools and custom web console +- **Web Console (Oort Console)** — a feature-rich, fully mobile-responsive single-page application for managing files, analytics, and account settings +- **Application Platform (Orbit Console)** — high-performance PaaS with space-themed glassmorphic UI, live log streams, and 1-click starter deployments +- **Universal Single Sign-On (SSO)** — unified user database (`cosmos.users`) allowing seamless navigation across Forge, Oort, and Orbit +- **Per-user storage & compute isolation** — isolated MinIO buckets with 10 GB default hard quotas and max 10 container application ports per user +- **Real-time analytics with geographic tracking** — CDN request tracking, per-country read/write stats, timeseries charts, D3.js 3D globe +- **Quota expansion system** — users can request quota increases; admin receives email with approval link +- **CDN proxy layer** — public files served via Go backend with in-memory caching; cache invalidated on upload/delete/move/restore +- **Custom domain support** — users can map custom domains to their buckets for branded static hosting +- **Git hosting** — Gitea self-hosted git server at `/source/` +- **Versioning** — MinIO object versioning enabled per bucket; console allows viewing and restoring previous versions + +The platform runs entirely on Debian Linux with Nginx as the edge, MinIO as the S3 engine, ScyllaDB (via Docker) as the metadata store, and a custom Go binary as the application server. + +--- + +## 2. Infrastructure & Server Setup + +### VPS Specifications + +| Property | Value | +|---|---| +| OS | Debian 12 (Bookworm) | +| User | `debian` | +| Primary domain | `cosmos.linuxguy.space` | +| Legacy domains | `vshno.linuxguy.space`, `debian.linuxguy.space` | +| DNS/CDN | Cloudflare (proxied) — provides SSL termination and DDoS protection | +| SSL (origin) | Self-signed cert at `/etc/nginx/ssl/nginx.crt` (Cloudflare handles public TLS) | +| Cloudflare header | `CF-IPCountry` — used by analytics to geolocate requests by country | +| Internal ports | 8000 (Go), 9000 (MinIO S3 API), 9001 (MinIO Console), 9042 (ScyllaDB), 3000 (Gitea) | +| Public ports | 80, 443 via Nginx | + +### Public URL Map + +| Path | Service | +|---|---| +| `https://cosmos.linuxguy.space/` | Cosmos landing page (Go → `index.html`) | +| `https://cosmos.linuxguy.space/oort` | Oort Object Storage landing page | +| `https://cosmos.linuxguy.space/oort/login` | Oort user login | +| `https://cosmos.linuxguy.space/oort/signup` | Oort user registration | +| `https://cosmos.linuxguy.space/oort/verify` | OTP email verification (inline HTML) | +| `https://cosmos.linuxguy.space/oort/dashboard` | API credentials dashboard | +| `https://cosmos.linuxguy.space/oort/console` | S3 Storage Console SPA | +| `https://cosmos.linuxguy.space/oort/{username}/{bucket}/{key}` | Public file proxy (via `handleS3PublicProxy`) | +| `https://cosmos.linuxguy.space/git` | Forge (Git) landing page (`git.html`) | +| `https://cosmos.linuxguy.space/git/login` | Forge login with automatic Gitea SSO | +| `https://cosmos.linuxguy.space/git/signup` | Forge registration with OTP verification | +| `https://cosmos.linuxguy.space/git/dashboard` | Forge workspace dashboard | +| `https://cosmos.linuxguy.space/{service}/forgot-password` | Space-themed password reset request (`forgot-password.html`) | +| `https://cosmos.linuxguy.space/{service}/reset-password?token=...` | Password reset execution with strength meter & Gitea sync | +| `https://cosmos.linuxguy.space/source/` | Gitea Git server web interface (proxied) | +| `https://cosmos.linuxguy.space/console/` | MinIO Admin Console | +| `https://cosmos.linuxguy.space/oort/api/quota/approve?token=...` | Admin quota approval link | +| `https://cosmos.linuxguy.space/orbit` | Orbit PaaS Console SPA | +| `https://cosmos.linuxguy.space/orbit/proxy/{app_id}/*` | Dynamic transparent reverse proxy to container | +| `https://cosmos.linuxguy.space/orbit/api/apps` | Orbit apps list & creation API | +| `https://{custom-domain}/*` | Custom domain transparent reverse proxy | + +### Active MinIO Buckets + +| Bucket | Owner | Type | +|---|---|---| +| `unity-27aa4685` | vaishnosingh9@gmail.com | Default user bucket | +| `unity-0cb36849` | duggalpiyush0@gmail.com | Default user bucket | +| `unity-46288ab0` | admin@vshno.linuxguy.space | Default user bucket | +| `unity-115b2550` | bavaj31068@meonvr.com | Default user bucket | +| `unity-d228e0597dd84a34` | vaishnosingh9@gmail.com | Extra bucket ("tools-suite-models") | +| `cdn-bucket` | system | Legacy CDN bucket | +| `memespace` | system | Legacy meme service bucket | + +--- + +## 3. Filesystem Layout & Codebase Map + +``` +/home/debian/ +├── portals-go/ ← Main application root +│ ├── portals ← Compiled Go binary (run by systemd) +│ ├── main.go ← Core entry point: routing, auth, Gitea SSO, password reset, DB setup, custom domain router +│ ├── orbitcontroller.go ← Orbit PaaS engine: Docker API, buildpacks, proxy, telemetry, custom domains (~1300 lines) +│ ├── oortconsole.go ← All S3/Oort API handlers (1811 lines, 57.2 KB) +│ ├── oortanalytics.go ← In-memory metrics + ScyllaDB persistence (145 lines, 4 KB) +│ ├── s3cache.go ← CDN proxy in-memory response caching (50 lines, 905 bytes) +│ ├── templates/ ← Embedded HTML templates (//go:embed) +│ │ ├── orbitconsole.html ← Orbit PaaS Console SPA (Vercel/Railway layout, orbital background) +│ │ ├── oortconsole.html ← S3 Storage Console SPA (space-themed dashboard) +│ │ ├── git.html ← Forge Git landing page +│ │ ├── oort.html ← Oort Object Storage landing page +│ │ ├── index.html ← Cosmos Suite root landing page +│ │ ├── forgot-password.html ← Space-themed password recovery +│ │ └── tailwind.css ← Pre-compiled utility CSS +│ └── scratch/ ← Persistent test suites, temporary tar contexts, scripts +│ ├── go.mod ← Go module definition (Go 1.25, clean dependencies) +│ ├── go.sum ← Dependency checksums +│ ├── cosmos/ ← Docker Compose for ScyllaDB +│ │ └── docker-compose.yml +│ └── templates/ ← HTML templates (embedded into binary at compile time) +│ ├── index.html ← Homepage / landing page (56.8 KB) +│ ├── oort.html ← Oort service landing page (28.6 KB) +│ ├── oortconsole.html ← Full S3 Console SPA (575 KB — ~3,500+ lines) +│ ├── login.html ← Space-themed login page (19.1 KB) +│ ├── signup.html ← Space-themed signup page (26.5 KB) +│ ├── forgot-password.html ← Space-themed password reset page with 6 states & sky engine (17.3 KB) +│ ├── cloud.html ← Cloud Drive service landing (28.4 KB) +│ ├── git.html ← Git service landing page (31.6 KB) +│ ├── tailwind.css ← Tailwind CSS framework (2.9 MB) +│ └── tailwind.js ← Tailwind JS utilities (407 KB) +│ +/opt/minio/ +└── data/ ← MinIO object storage data directory + ├── unity-27aa4685/ ← User bucket: vaishnosingh9@gmail.com + ├── unity-0cb36849/ ← User bucket: duggalpiyush0@gmail.com + ├── unity-46288ab0/ ← User bucket: admin@vshno.linuxguy.space + ├── unity-115b2550/ ← User bucket: bavaj31068@meonvr.com + ├── unity-d228e0597dd84a34/ ← Extra bucket: vaishnosingh9 (tools-suite-models) + ├── cdn-bucket/ ← Legacy CDN bucket + ├── memespace/ ← Legacy meme service bucket + └── .minio.sys/ ← MinIO system metadata +│ +/etc/systemd/system/ +├── portals.service ← Go backend service definition +└── minio.service ← MinIO object storage service +│ +/etc/nginx/ +├── nginx.conf ← Main Nginx config (workers, gzip, SSL protocols) +└── sites-enabled/ + └── debian.linuxguy.space ← Main virtual host config (all proxy rules) +│ +/var/lib/docker/volumes/ +└── scylla_scylla-data/ ← ScyllaDB persistent data volume +│ +/home/debian/portals-go/cosmos/ ← Docker compose files for ScyllaDB +│ +/home/debian/.msmtprc ← msmtp email configuration (Gmail SMTP) +│ +/usr/local/bin/minio ← MinIO binary (v RELEASE.2024-12-18T13-15-44Z) +/usr/local/bin/mc ← MinIO client (mc) for admin operations +``` + +### Go Module Dependencies (`go.mod`) + +``` +module portals +go 1.25.0 + +Direct dependencies: + github.com/google/uuid v1.6.0 ← UUID generation for sessions + github.com/mattn/go-sqlite3 v1.14.50 ← SQLite (legacy, NOT actively used in main code path) + golang.org/x/crypto v0.55.0 ← bcrypt for password hashing + +Key indirect dependencies: + github.com/gocql/gocql v1.7.0 ← ScyllaDB/Cassandra CQL driver + github.com/minio/minio-go/v7 v7.3.0 ← MinIO S3 client SDK + golang.org/x/net, sys, text ← Network/system utilities +``` + +> [!WARNING] +> `github.com/mattn/go-sqlite3` is listed as a direct dependency in `go.mod`, but SQLite is **not used anywhere** in the current Go source code. The old SQLite data files (`unity_cloud.db`, `unity_git.db`, `unity_s3.db`) have been deleted. This dependency should be removed from `go.mod` to avoid unnecessary CGO compilation overhead. + +--- + +## 4. Service Architecture Diagram + +``` + ┌─────────────────────────────┐ + │ INTERNET │ + │ (User Browser / S3 CLI) │ + └──────────────┬────────────────┘ + │ HTTPS (via Cloudflare) + │ + CF-IPCountry header + ▼ + ┌─────────────────────────────┐ + │ CLOUDFLARE PROXY │ + │ (SSL termination, DDoS, CDN) │ + └──────────────┬────────────────┘ + │ HTTP to origin (self-signed cert) + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ NGINX (:80/:443) │ +│ /etc/nginx/sites-enabled/ │ +│ server_name: cosmos.linuxguy.space vshno.linuxguy.space │ +│ debian.linuxguy.space localhost │ +│ client_max_body_size: 10G │ +│ │ +│ Location Rules (in priority order): │ +│ GET / → Go Backend (:8000) │ +│ POST / → MinIO S3 API (:9000) [418 trick] │ +│ /oort/* → Go Backend (:8000) │ +│ /git/* → Go Backend (:8000) │ +│ /orbit/* → Go Backend (:8000) [PaaS & Container Proxy]│ +│ /cdn/* → Go Backend (:8000) │ +│ /tailwind.css → Go Backend (:8000) │ +│ /source/* → Gitea (:3000) [path rewrite] │ +│ /console/* → MinIO Console (:9001) [path rewrite] │ +│ /api/v1/* → MinIO Console (:9001) │ +│ /static/* → MinIO Console (:9001) │ +│ /* → MinIO S3 API (:9000) [S3 catch-all] │ +│ │ +│ Default server (any other domain/subdomain): │ +│ /* → rewrite → /oort/api/cdn_proxy/* → Go Backend (:8000) │ +│ → maps custom domain to Orbit App (:4100-4999) or Oort Bucket│ +└───────┬──────────────────────┬──────────────────────┬─────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ +│ Go Backend │ │ MinIO Object │ │ Gitea Git │ +│ (portals) │ │ Storage │ │ Server │ +│ :8000 │ │ :9000 (S3 API) │ │ :3000 │ +│ 49.2 MB RAM │ │ :9001 (Console) │ │ │ +│ │ │ 238.9 MB RAM │ │ Accessible │ +│ Handles: │ │ │ │ at /source/ │ +│ - Auth/SSO │ │ User buckets: │ │ │ +│ - S3 Console │ │ unity-XXXXXXXX │ │ Starter repos│ +│ - Orbit PaaS │ │ (per user) │ │ webhooks for │ +│ - Docker Ops │ │ Data: /opt/minio/ │ │ auto-deploy │ +│ - Analytics │ │ data/ │ └──────────────┘ +│ - CDN Proxy │ └────────────────────┘ +│ - Quotas │ +│ - Domains │ +└───────┬──────┴─────────────────────────┬───────────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌──────────────────────────────────────┐ │ ┌───────────────────────────────────────┐ +│ ScyllaDB (Docker Container) │ │ │ Orbit App Containers (Docker) │ +│ Container: scylla-scylla-1 │ │ │ Isolated ports: 4100 – 4999 │ +│ Port: 9042 (CQL) │ │ │ - Node.js Express / Next.js │ +│ Volume: scylla_scylla-data │ │ │ - Go Web Services (golang:alpine) │ +│ │ │ │ - Python FastAPI / Flask │ +│ Keyspace: cosmos │ │ │ - Static HTML5/CSS3 / Nginx (SPA) │ +│ Tables: │ │ │ Reverse-proxied via: │ +│ - users (auth + sessions) │ │ │ • /orbit/proxy/{app_id}/* │ +│ - pending (email verification)│ │ │ • Host: {custom-domain} │ +│ - password_reset_tokens (reset links)│ │ └───────────────────────────────────────┘ +│ - buckets (extra buckets) │ │ +│ - s3_analytics (aggregate metrics) │ │ +│ - s3_analytics_hourly (time-series) │ │ +│ - quota_requests (quota approval) │ │ +│ - bucket_domains (custom domains) │ │ +│ - orbit_apps (app metadata) │─┘ +│ - orbit_deployments (build history) │ +└──────────────────────────────────────┘ +``` + +--- + +## 5. Nginx — Web Server & Reverse Proxy + +### File Location +`/etc/nginx/sites-enabled/debian.linuxguy.space` + +### How It Works + +Nginx is the **single entry point** for all traffic. It handles: +1. **TLS/SSL termination** — decrypts HTTPS traffic from Cloudflare (re-encrypted to origin) +2. **Reverse proxying** — forwards requests to the correct internal service +3. **Large file uploads** — `client_max_body_size 10G` permits up to 10 GB uploads per request +4. **WebSocket proxying** — for MinIO console real-time updates (Upgrade/Connection headers) + +### Key Configuration Details + +**Rate Limiting Zones (DDoS & Brute-Force Protection):** +```nginx +# Auth endpoints: 5 requests per minute per IP (burst=10, no delay) +limit_req_zone $binary_remote_addr zone=auth_limit:10m rate=5r/m; +# General API: 60 requests per second per IP +limit_req_zone $binary_remote_addr zone=api_limit:10m rate=60r/s; +``` + +**Shared server block (all production domains):** +```nginx +server { + listen 80; listen [::]:80; + listen 443 ssl; listen [::]:443 ssl; + server_name vshno.linuxguy.space cosmos.linuxguy.space debian.linuxguy.space localhost; + client_max_body_size 10G; + ssl_certificate /etc/nginx/ssl/nginx.crt; # Self-signed (Cloudflare handles public cert) + ssl_certificate_key /etc/nginx/ssl/nginx.key; +} +``` + +**Smart Root Location** — discriminates GET (web portal) from POST (S3 API): +```nginx +location = / { + error_page 418 = @minio_sts; + if ($request_method = POST) { return 418; } # Trick: POST → S3 + proxy_pass http://127.0.0.1:8000; # GET → Go web portal +} +location @minio_sts { + proxy_pass http://127.0.0.1:9000; # POST → MinIO S3 API +} +``` + +**Oort, Git, CDN routes** → Go backend: +```nginx +location = /oort { proxy_pass http://127.0.0.1:8000; } +location ^~ /oort/ { proxy_pass http://127.0.0.1:8000; } + +# Strict Rate Limiting on Authentication Endpoints (5 req/min, burst 10) +location ~* ^/oort/(login|signup|verify)$ { + limit_req zone=auth_limit burst=10 nodelay; + limit_req_status 429; + proxy_pass http://127.0.0.1:8000; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +} + +location = /git { proxy_pass http://127.0.0.1:8000; } +location ^~ /git/ { proxy_pass http://127.0.0.1:8000; } +``` + +**Gitea Git Server with Auth Redirects:** +```nginx +location = /source/user/forgot_password { + return 302 /git/forgot-password; +} +location = /source/user/sign_up { + return 302 /git/signup; +} +location ^~ /source/ { + proxy_pass http://127.0.0.1:3000/; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 300; + proxy_send_timeout 300; +} +``` + +**MinIO Console** — strips `/console/` prefix via rewrite: +```nginx +location ^~ /console/ { + rewrite ^/console/(.*) /$1 break; # Strip prefix + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; # WebSocket support + proxy_pass http://localhost:9001; +} +``` + +**MinIO S3 API catch-all** — unmatched paths hit MinIO directly (enables native S3 SDK usage): +```nginx +location / { + proxy_set_header Connection ""; + chunked_transfer_encoding off; # Required for S3 multipart + proxy_pass http://localhost:9000; +} +``` + +**Custom domain CDN default-server** — second `server {}` block catches any unknown domain: +```nginx +server { + listen 80 default_server; + listen [::]:80 default_server; + listen 443 ssl default_server; + listen [::]:443 ssl default_server; + server_name _; + location / { + rewrite ^/(.*)$ /oort/api/cdn_proxy/$1 break; + proxy_pass http://127.0.0.1:8000; + } +} +``` +This enables users to point a custom domain (e.g., `mysite.com`) at this server's IP, and their bucket files will be served transparently. + +### Standard Proxy Headers (on all proxy locations) +```nginx +proxy_set_header Host $http_host; +proxy_set_header X-Real-IP $remote_addr; +proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +proxy_set_header X-Forwarded-Proto $scheme; +``` +Additionally, Cloudflare adds: +- `CF-IPCountry` — 2-letter ISO country code of the visitor, used by analytics + +--- + +## 6. MinIO — S3-Compatible Object Storage + +### What It Is +MinIO (v RELEASE.2024-12-18T13-15-44Z) is the S3-API-compatible storage engine. Users interact via: +- The Oort Console (through the Go backend) +- Native AWS S3 CLI / SDK tools directly (aws-cli, boto3, rclone, etc.) using the endpoint `https://cosmos.linuxguy.space` +- The MinIO admin console at `/console/` + +### Installation Paths +``` +Binary: /usr/local/bin/minio +Client: /usr/local/bin/mc +Data: /opt/minio/data/ +``` + +### Service Configuration (`/etc/systemd/system/minio.service`) +```ini +[Unit] +Description=MinIO Object Storage +After=network.target + +[Service] +User=root +Environment="MINIO_ROOT_USER=unityadmin" +Environment="MINIO_ROOT_PASSWORD=UnityAdmin2026!" +Environment="MINIO_BROWSER_REDIRECT_URL=https://vshno.linuxguy.space/console/" +Environment="MINIO_SERVER_URL=https://vshno.linuxguy.space" +ExecStart=/usr/local/bin/minio server /opt/minio/data --address ":9000" --console-address ":9001" +Restart=always +``` + +| Variable | Value | Purpose | +|---|---|---| +| `MINIO_ROOT_USER` | `unityadmin` | Admin username for mc/API access | +| `MINIO_ROOT_PASSWORD` | `UnityAdmin2026!` | Admin password | +| `MINIO_SERVER_URL` | `https://vshno.linuxguy.space` | Public URL for presigned URL generation | +| `MINIO_BROWSER_REDIRECT_URL` | `https://vshno.linuxguy.space/console/` | MinIO console redirect | + +### User Isolation Architecture +Each Cosmos user gets: +- **MinIO user account** (access key = their email, secret key = 12-char random hex) +- **Dedicated bucket** named `unity-{8-char-hex}` (default) or `unity-{16-char-hex}` (extra buckets) +- **Policy** named `policy_{bucket}` that restricts them to **only their bucket** (`s3:*` on their bucket ARN only) +- **Hard quota** enforced at the MinIO level (default 10 GB, read by the backend via `mc quota info cosmos/{bucket} --json`) + +### Presigned URLs +The backend generates presigned URLs for file preview and download using the **public endpoint** (`cosmos.linuxguy.space`), so the signature matches what the browser requests. Cache TTL: +- Presign: 15 minutes +- Share: 7 days + +### Bucket Versioning +Object versioning is enabled on every bucket at creation time (`mc version enable cosmos/{bucket}`). This allows the console to list and restore previous file versions. + +### MinIO Client (mc) Admin Commands +```bash +# Set up admin alias (required before any mc commands) +mc alias set cosmos http://127.0.0.1:9000 unityadmin UnityAdmin2026! + +# List all users +mc admin user list cosmos + +# Add a new user +mc admin user add cosmos USER_EMAIL SECRET_KEY + +# Create a bucket +mc mb cosmos/BUCKET_NAME + +# Enable versioning +mc version enable cosmos/BUCKET_NAME + +# Create a policy +mc admin policy create cosmos POLICY_NAME /path/to/policy.json + +# Attach policy to user +mc admin policy attach cosmos POLICY_NAME --user USER_EMAIL + +# Set quota +mc quota set cosmos/BUCKET_NAME --size 10G + +# Check quota (JSON output, used by backend) +mc quota info cosmos/BUCKET_NAME --json +``` + +--- + +## 7. ScyllaDB — Database Layer + +### What It Is +ScyllaDB (Cassandra-compatible NoSQL) stores all user metadata, sessions, analytics, and custom domains. The Go backend uses raw CQL queries via `github.com/gocql/gocql`. + +### Running Environment +```bash +# Docker container +scylla-scylla-1 + +# Docker Compose location +/home/debian/portals-go/cosmos/docker-compose.yml + +# Data volume (persisted) +/var/lib/docker/volumes/scylla_scylla-data/_data/ + +# Connect (requires docker access — use Go scripts instead, see troubleshooting) +sudo docker exec -it scylla-scylla-1 cqlsh +``` + +> [!NOTE] +> The `debian` user does NOT have access to the Docker socket. You cannot run `docker exec` as the `debian` user. Run database queries either as `root` (with `sudo`) or by writing a temporary Go script in `/home/debian/portals-go/` that imports `github.com/gocql/gocql` and runs queries against `127.0.0.1`. + +### Keyspaces + +| Keyspace | Status | Purpose | +|---|---|---| +| `cosmos` | **ACTIVE** | Primary application keyspace (all live data) | +| `unity` | Legacy/empty | Created by `initDB()` for backward compatibility | +| `system*` | Internal | ScyllaDB system tables | + +### All Tables in `cosmos` Keyspace + +#### `cosmos.users` — All registered users +```cql +CREATE TABLE cosmos.users ( + service text, -- 'oort', 'cdn', 'git' + email text, -- User email (PK part 2) + name text, -- Display name + password_hash text, -- bcrypt hash (cost 12) + access_key text, -- S3 access key (= email for oort) + secret_key text, -- 12-char hex secret + bucket text, -- Default MinIO bucket name + session_id text, -- Active session UUID + username text, -- URL-safe username + profile_image text, -- Base64-encoded JPEG (stored in DB) + bucket_display_name text, -- Human-readable bucket name (default: "mybucket") + created_at bigint, -- Unix timestamp for 30-day session expiry enforcement + PRIMARY KEY ((service, email)) +); +-- Secondary indexes: +CREATE INDEX users_session ON cosmos.users (session_id); +CREATE INDEX users_username_idx ON cosmos.users (username); +``` + +**Note:** Sessions with `created_at > 0` expire after 30 days (`sessionMaxAge = 30 * 24 * time.Hour`). Grandfathered accounts with `created_at = null` remain active. + +**Current users:** + +| Service | Email | Username | Default Bucket | +|---|---|---|---| +| oort | vaishnosingh9@gmail.com | vaishno | unity-27aa4685 | +| oort | duggalpiyush0@gmail.com | mr_unchained | unity-0cb36849 | +| oort | admin@vshno.linuxguy.space | admin | unity-46288ab0 | +| oort | bavaj31068@meonvr.com | vaishno2 | unity-115b2550 | +| cdn | admin@vshno.linuxguy.space | admin | (none) | +| git | vaishnosingh9@gmail.com | vaishnosingh | unity-e3c046a7 | + +#### `cosmos.pending` — Users awaiting email verification +```cql +CREATE TABLE cosmos.pending ( + service text, + reg_token text, -- 6-digit OTP code + session_id text, + name text, + email text, + password_hash text, + username text, + PRIMARY KEY ((service, reg_token)) +); +``` +> **Security Update:** The `raw_password` column was dropped from this table. Plaintext passwords for Gitea provisioning are held in an in-memory `pendingPasswords sync.Map` with a 15-minute auto-expiry TTL and purged immediately upon verification. + +#### `cosmos.password_reset_tokens` — Secure Password Reset Tokens +```cql +CREATE TABLE cosmos.password_reset_tokens ( + reset_token text PRIMARY KEY, -- UUIDv4 reset token + email text, + service text, -- 'oort', 'git', 'cdn' + expires_at bigint -- Unix timestamp (1-hour expiry) +); +``` +> **Critical CQL Rule:** The primary key column is named `reset_token` (NOT `token`), because `token` is a CQL reserved keyword in ScyllaDB. Tokens auto-expire after 1 hour and are permanently deleted after password submission. +``` +Row is deleted after successful OTP verification. + +#### `cosmos.buckets` — User-created extra buckets +```cql +CREATE TABLE cosmos.buckets ( + bucket_name text PRIMARY KEY, + display_name text, + owner_email text, + is_public boolean +); +-- Secondary index: +CREATE INDEX buckets_owner_email_idx ON cosmos.buckets (owner_email); +``` + +#### `cosmos.s3_analytics` — Aggregate lifetime metrics (flushed from in-memory every 5s) +```cql +CREATE TABLE cosmos.s3_analytics ( + bucket_name text PRIMARY KEY, + total_requests bigint, + cache_hits bigint, + origin_hits bigint, + total_success bigint, + total_4xx bigint, + total_5xx bigint, + total_bytes bigint, + last_hit bigint -- Unix timestamp +); +``` + +#### `cosmos.s3_analytics_hourly` — Time-series counter table (for analytics charts & globe) +```cql +CREATE TABLE cosmos.s3_analytics_hourly ( + bucket_name text, + hour timestamp, -- Truncated to hour boundary + country text, -- ISO country code from CF-IPCountry + total_requests counter, + total_bytes counter, + cache_hits counter, + origin_hits counter, + total_success counter, + total_errors counter, + total_reads counter, -- GET/HEAD requests + total_writes counter, -- PUT/POST upload-complete requests + PRIMARY KEY ((bucket_name), hour, country) +); +``` +This is a ScyllaDB **counter table** — uses `UPDATE ... SET col = col + ?` syntax. Entries are created per (bucket, hour, country) combination. + +#### `cosmos.quota_requests` — Pending quota expansion requests +```cql +CREATE TABLE cosmos.quota_requests ( + req_token text PRIMARY KEY, + email text, + amount int +); +``` +Created at startup by `RegisterS3ConsoleRoutes()`. When admin clicks the approval link (`/oort/api/quota/approve?token=...`), the quota is set in MinIO and the row is deleted. + +#### `cosmos.bucket_domains` — Custom domain mappings +```cql +CREATE TABLE cosmos.bucket_domains ( + domain text PRIMARY KEY, + bucket_name text, + owner_email text +); +-- Secondary index: +CREATE INDEX bucket_domains_bucket_name_idx ON cosmos.bucket_domains (bucket_name); +``` +Created at startup by `RegisterS3ConsoleRoutes()`. Used by `handleS3CDNProxy()` to look up which bucket a custom domain maps to. Results are cached in `domainCache sync.Map` to avoid repeated DB lookups. + +### Startup Time Warning +ScyllaDB in Docker can take **up to 3 minutes on a cold boot** to fully initialize. The Go backend retries connection 60 times × 5 seconds = up to **5 minutes**. If ScyllaDB takes longer than 5 minutes to start, the portals service will crash and needs manual restart: +```bash +sudo systemctl restart portals +``` + +--- + +## 8. The Go Backend — Cosmos Portals (`portals-go`) + +### Overview +The Go backend is the **brain** of the Cosmos platform. It is a single binary (`portals`) that handles everything: +- Serving HTML pages (via embedded Go templates using `//go:embed`) +- User authentication (signup, login, OTP email verification, logout) +- S3 console API layer (all object storage operations) +- Analytics ingestion and serving +- CDN proxy with in-memory caching and cache invalidation +- Public file serving by username/bucket/key +- Gitea and Seafile user provisioning +- Quota management (request + approval) +- Custom domain management + +### Critical Build Rule +Templates are compiled into the binary at build time using `//go:embed`. **After any change to any `.go` file or any template `.html` file, you MUST rebuild and restart:** +```bash +cd /home/debian/portals-go && go build -o portals . && sudo systemctl restart portals +``` + +### Source Files + +#### `main.go` (~830 lines, 35 KB) +The core entry point. Key functions: + +| Function | Description | +|---|---| +| `initDB()` | Connects to ScyllaDB, retries 60×5s. Creates `users`, `pending`, and `password_reset_tokens` tables. Creates `unity` keyspace. | +| `init()` | Parses all HTML templates from embedded FS into `tmpl` | +| `generateOTP()` | Generates a 6-digit numeric verification code using `crypto/rand` | +| `sendVerificationEmail(email, token, service)` | Sends OTP email via `msmtp` with `From: noreply@cosmos.linuxguy.space` (relayed via Gmail SMTP) | +| `lookupBySession(service, sid)` | Given a session UUID, returns `(name, accessKey, secretKey, bucket, err)` — central auth lookup with 30-day session expiry enforcement | +| `handleSignup(w, r, service)` | Handles GET (renders signup.html) and POST (creates pending user, sends OTP, redirects to verify). Password held in-memory only. | +| `handleLogin(w, r, service)` | Validates email/username + bcrypt password. For `git` service, logs into Gitea backend via `loginGitea()`, sets `i_like_gitea` cookie, and redirects directly to `/source/` (seamless SSO). | +| `handleVerify(w, r, service)` | Receives OTP, moves user from `pending` → `users`, retrieves password from memory, spawns provisioning goroutine | +| `handlePoll(w, r, service)` | `/oort/api/poll?sid=X` — lightweight session check for verify page polling | +| `handleLogout(w, r, service)` | Clears `cosmos_sid` and `i_like_gitea` cookies, redirects to service page | +| `handleDashboard(w, r, service)` | Workspace credentials page (Access Key, Secret Key, Bucket, Endpoint, or Git connected workspace) | +| `sendPasswordResetEmail(toEmail, token, service)` | Sends space-themed password reset email via `msmtp` (`From: noreply@cosmos.linuxguy.space`) with 1-hour expiry link | +| `handleForgotPassword(w, r, service)` | GET renders `forgot-password.html`; POST looks up user, generates UUIDv4 token, saves to `password_reset_tokens` (1h TTL), sends email (with user enumeration protection) | +| `handleResetPassword(w, r, service)` | Validates reset token & expiry, hashes new password with bcrypt, updates ScyllaDB, synchronizes Gitea via `syncGiteaPassword()`, deletes token | +| `provisionMinIO(ak, sk, bucket)` | Creates MinIO user + bucket + versioning + IAM policy + 10 GB quota | +| `provisionGitea(username, email, password)` | Creates Gitea user via `docker exec gitea-gitea-1 gitea admin user create` | +| `syncGiteaPassword(username, newPassword)` | Synchronizes password to Gitea via `docker exec gitea-gitea-1 gitea admin user change-password` | +| `loginGitea(username, password)` | Performs backend authentication against Gitea at `:3000`, auto-heals any desynchronized credentials, and extracts session cookies | +| `serviceRouter(service)` | Returns an `http.HandlerFunc` routing `signup`, `login`, `verify`, `logout`, `dashboard`, `forgot-password`, `reset-password`, and S3 public proxy | +| `handleIndex(w, r)` | Serves `index.html` | +| `handleServicePage(w, r, service)` | Serves `{service}.html` (oort.html, git.html, cloud.html) | + +**Route Registration in `main()`:** +```go +RegisterS3ConsoleRoutes(mux) // All /oort/api/* and /oort/console routes +RegisterOrbitRoutes(mux) // All /orbit/api/* and /orbit/proxy/* routes +mux.HandleFunc("/tailwind.css", ...) // Serves embedded Tailwind CSS +mux.HandleFunc("/oort", handleServicePage("oort")) +mux.HandleFunc("/git", handleServicePage("git")) +mux.HandleFunc("/orbit", handleOrbitConsole) // Orbit PaaS Console SPA +mux.HandleFunc("/orbit/", handleOrbitConsole) // Orbit PaaS sub-routes +mux.HandleFunc("/cdn", handleServicePage("cdn")) +mux.HandleFunc("/oort/", serviceRouter("oort")) // Handles /oort/login, /oort/signup, etc. +mux.HandleFunc("/git/", serviceRouter("git")) +mux.HandleFunc("/cdn/", serviceRouter("cdn")) +mux.HandleFunc("/", handleIndex) // Serves index.html or proxies Orbit custom domains +``` + +**Email via `msmtp`:** Verification emails are sent by piping a raw email message to `msmtp -t`. The system `msmtp` is configured at `~/.msmtprc` with Gmail SMTP (`smtp.gmail.com:587`, user `serverdmncr@gmail.com`, app password `tlnmobdygetarrnj`). Quota approval emails use the same mechanism. + +**Public file proxy:** When a request hits `/oort/{username}/{rest}` and `username` is not a known route keyword, it falls through to `handleS3PublicProxy(w, r, username, rest)`, which resolves the display bucket name to the actual MinIO bucket name and proxies the file. + +#### `oortconsole.go` (1810 lines, 57.2 KB) +All API handlers for the Oort console SPA and the public proxy. Key constants: +```go +const minioEndpoint = "127.0.0.1:9000" // Internal MinIO (no TLS) +const publicMinioEndpoint = "cosmos.linuxguy.space" // Public MinIO (TLS, for presigned URLs) +``` + +**Package-level variables:** +```go +var domainCache sync.Map // map[domain]bucketName (custom domain cache) +var publicBucketCache sync.Map // map["user:displayBucket"]actualBucket (public proxy cache) +var proxyTransport *http.Transport +``` + +**Tables created at startup (`RegisterS3ConsoleRoutes`):** +- `cosmos.quota_requests` +- `cosmos.bucket_domains` + +**Complete API Handler Table:** + +| Function | Route | Method | Description | +|---|---|---|---| +| `authS3Console()` | — | — | Auth helper: checks `?sid=`, `X-Session-Id` header, or `cosmos_sid` cookie against ScyllaDB | +| `minioClient(ak, sk)` | — | — | Creates MinIO SDK client for a specific user (internal endpoint) | +| `handleS3ConsolePage()` | `/oort/console` | GET | Auth check, sets `cosmos_sid` cookie, renders `oortconsole.html` with `SID` injected | +| `handleS3Stats()` | `/oort/api/stats` | GET | Lists all objects to get total size/count; calls `mc quota info` for real quota | +| `handleS3Objects()` | `/oort/api/objects` | GET | Lists objects using prefix-based pagination (not recursive); returns folders and files separately | +| `handleS3UploadInit()` | `/oort/api/upload/init` | POST | Initiates MinIO multipart upload, returns `upload_id` | +| `handleS3UploadPart()` | `/oort/api/upload/part` | POST | Uploads one 10 MB chunk, returns ETag | +| `handleS3UploadComplete()` | `/oort/api/upload/complete` | POST | Finalizes multipart upload; **invalidates CDN cache** for the key; records write analytics hit | +| `handleS3UploadAbort()` | `/oort/api/upload/abort` | POST | Cancels multipart upload | +| `handleS3Delete()` | `/oort/api/delete` | DELETE/POST | Deletes object or folder (recursive); **invalidates CDN cache** | +| `handleS3Presign()` | `/oort/api/presign` | GET | Generates presigned GET URL (15 min TTL) using public endpoint for CORS-safe browser access | +| `handleS3Mkdir()` | `/oort/api/mkdir` | POST | Creates a zero-byte object with key ending in `/` (S3 folder convention) | +| `handleS3Move()` | `/oort/api/objects/move` | POST | Copy + delete (rename/move); **invalidates CDN cache** for both source and destination | +| `handleS3RenameBucket()` | `/oort/api/buckets/rename` | POST | Updates `display_name` in `cosmos.buckets` (or `bucket_display_name` in `cosmos.users`) | +| `handleS3Profile()` | `/oort/api/profile` | GET | Returns user profile JSON | +| `handleS3Quota()` | `/oort/api/quota` | POST | Creates quota request in DB, sends approval email via msmtp | +| `handleS3ApproveQuota()` | `/oort/api/quota/approve` | GET | Admin link: sets MinIO quota to requested amount, deletes request | +| `sendQuotaEmail()` | — | — | Sends quota request email from/to `serverdmncr@gmail.com` via msmtp | +| `handleS3Password()` | `/oort/api/password` | POST | Changes user password with bcrypt re-hash | +| `syncUserPolicy(email)` | — | — | Re-attaches all bucket policies after bucket operations | +| `handleS3Buckets()` | `/oort/api/buckets` | GET/POST/DELETE | List/create/delete buckets; create generates 16-char hex name with `cosmos-` prefix | +| `handleS3CreateBucket()` | `/oort/api/buckets` | POST | Creates new bucket in MinIO + registers in ScyllaDB | +| `handleS3DeleteBucket()` | `/oort/api/buckets` | DELETE | Deletes bucket via `mc rb --force`, removes from ScyllaDB | +| `handleS3ProfileImage()` | `/oort/api/profile/image` | POST | Accepts multipart image upload, stores as base64 in `cosmos.users.profile_image` | +| `handleS3ProfileUpdate()` | `/oort/api/profile/update` | POST | Updates name/username in DB | +| `handleS3ShareObject()` | `/oort/api/objects/share` | GET | Generates 7-day presigned GET URL | +| `handleS3ObjectVersions()` | `/oort/api/objects/versions` | GET | Lists all versions of an object using MinIO versioning | +| `handleS3ObjectRestore()` | `/oort/api/objects/restore` | POST | Copies old version to current key (restores); **invalidates CDN cache** | +| `handleS3PublicProxy()` | `/oort/{username}/{bucket}/{key}` | GET | Public proxy: resolves user+displayBucket → MinIO bucket, proxies file; supports caching | +| `handleS3BucketSettings()` | `/oort/api/bucket/settings` | GET/POST | Get/update `is_public`, `display_name` | +| `handleS3Analytics()` | `/oort/api/analytics` | GET | Time-series + globe analytics with filters (bucket, start, end, region, op) | +| `handleS3EmptyBucket()` | `/oort/api/bucket/empty` | POST | Removes all objects from a bucket (with confirmation); requires bucket name match | +| `handleS3BucketDomains()` | `/oort/api/bucket/domains` | GET | Lists custom domains for a bucket | +| `handleS3BucketDomainAdd()` | `/oort/api/bucket/domain/add` | POST | Adds custom domain → bucket mapping in DB | +| `handleS3BucketDomainRemove()` | `/oort/api/bucket/domain/remove` | POST | Removes custom domain; reads `domain` from **JSON body** | +| `handleS3CDNProxy()` | `/oort/api/cdn_proxy/*` | GET | Custom domain CDN proxy; looks up domain → bucket, proxies with 5-min memory cache | +| `formatBytes(b)` | — | — | Formats bytes to human-readable string | + +#### `oortanalytics.go` (145 lines, 4 KB) +In-memory, lock-free analytics engine. + +**Data structures:** +```go +type BucketMetrics struct { + TotalRequests int64 + CacheHits int64 + OriginHits int64 + TotalSuccess int64 + Total4xx int64 + Total5xx int64 + TotalBytes int64 + LastHit int64 // Unix timestamp +} +var globalMetrics sync.Map // map[bucketName]*BucketMetrics +``` + +**`recordHit(bucket, statusCode, bytes, isCacheHit, country, opType)`:** +1. Uses `sync/atomic` to update `globalMetrics` counters (lock-free, concurrent-safe) +2. Launches a goroutine to UPDATE `cosmos.s3_analytics_hourly` counter table for `(bucket, currentHour, country)` +3. `opType`: `"read"` for GET/HEAD, `"write"` for upload completions +4. Default country when `CF-IPCountry` is empty: `"US"` (⚠️ this can cause false US writes — see Section 26) + +**`init()` goroutine:** +- Waits 5 seconds for ScyllaDB to be available +- Loads `cosmos.s3_analytics` into `globalMetrics` +- Loops forever, flushing `globalMetrics` → `cosmos.s3_analytics` every 5 seconds + +#### `s3cache.go` (50 lines, 905 bytes) +CDN proxy in-memory cache. + +```go +var cdnCache sync.Map // map[targetPath]*cacheEntry + +type cacheEntry struct { + Body []byte + Headers http.Header + StatusCode int + ExpiresAt time.Time +} +``` + +**Cache TTLs:** +- Custom domain CDN proxy (`handleS3CDNProxy`): 5 minutes +- Public proxy (`handleS3PublicProxy`): 1 hour + +**Cache invalidation** occurs in these operations: +- `handleS3UploadComplete` — invalidates `/{bucket}/{key}` +- `handleS3Delete` — invalidates `/{bucket}/{key}` (or all with prefix for folders) +- `handleS3Move` — invalidates both source and destination paths +- `handleS3ObjectRestore` — invalidates `/{bucket}/{key}` + +#### `orbitcontroller.go` (1,150 lines, 42 KB) +Complete backend engine for Cosmos Orbit (Application Platform & Container Runtime). It controls Docker container lifecycle, automated buildpack detection, live streaming logs, reverse-proxy routing, and Oort S3 backup snapshot exports. + +**Core Constants & Global State:** +```go +const orbitPortMin = 4100 +const orbitPortMax = 4999 +const orbitMaxAppsPerUser = 10 + +var activeBuildLogs sync.Map // map[app_id]*bytes.Buffer +``` + +**Key Structs:** +- `OrbitApp`: represents an application record in `cosmos.orbit_apps` (id, owner, name, repo, branch, build_type, status, container_id, image_tag, internal_port, exposed_port, custom_domain, env_vars, timestamps) +- `OrbitDeployment`: represents a deployment record in `cosmos.orbit_deployments` (deploy_id, app_id, commit_hash, commit_msg, status, duration, build_log, created_at) +- `StarterTemplate`: represents canonical starter templates (Node.js, Go, Python, Static HTML) + +**Key Functions:** + +| Function | Route / Context | Description | +|---|---|---| +| `RegisterOrbitRoutes(mux)` | Initialization | Creates `cosmos.orbit_apps` and `cosmos.orbit_deployments` tables in ScyllaDB; registers all `/orbit/api/*` endpoints and `/orbit/proxy/*` reverse proxy | +| `handleOrbitConsole(w, r)` | `/orbit`, `/orbit/*` | Renders `orbitconsole.html` or checks session authentication | +| `authOrbitUser(r)` | Internal Helper | Authenticates user via `cosmos_sid` cookie, `oort_session`, or `orbit_session` against `cosmos.users` | +| `handleOrbitApps(w, r)` | `GET /orbit/api/apps` | Lists all applications owned by the user, checks container status, returns port quota usage (`X/10`) | +| `handleOrbitCreateApp(w, r)` | `POST /orbit/api/apps` | Validates app name, enforces 10-port quota limit, allocates available host port (4100–4999), creates DB entry, launches background `deployAppAsync()` | +| `deployAppAsync(app, deployId)` | Async Goroutine | Clones Git repo, runs `detectStack()`, injects multi-stage Dockerfile if missing, builds image via Docker CLI, stops any old container, starts container with 512 MB RAM / 1.0 CPU limits, records deployment history, and clears in-memory build logs | +| `detectStack(workDir)` | Build Engine | Inspects filesystem: `Dockerfile` → `dockerfile`; `go.mod` → `pack-go`; `requirements.txt` → `pack-python`; `package.json` with `"start"` → `pack-node`; `package.json` without `"start"` or `index.html` → `pack-static` | +| `handleOrbitAppAction(w, r)` | `POST /orbit/api/apps/:id/action` | Executes `start`, `stop`, `restart`, or `rebuild` | +| `handleOrbitAppLogs(w, r)` | `GET /orbit/api/apps/:id/logs` | Returns live build logs from memory during `building` state (sets `X-App-Status: building`), or live Docker container stdout/stderr (last 500 lines) during `running` state | +| `handleOrbitAppEnv(w, r)` | `POST /orbit/api/apps/:id/env` | Updates environment variables in ScyllaDB; restarts container with new env vars | +| `handleOrbitAppDomain(w, r)` | `POST /orbit/api/apps/:id/domain` | Assigns or removes custom domain; validates domain uniqueness across all apps | +| `handleOrbitBackupToOort(w, r)` | `POST /orbit/api/apps/:id/backup-to-oort` | Exports container filesystem via `docker export`, streams `.tar` snapshot directly to the user's Oort MinIO S3 bucket under `orbit-backups/{app_id}-{timestamp}.tar` | +| `handleOrbitProxy(w, r)` | `/orbit/proxy/:id/*` | Dynamic transparent reverse proxy forwarding incoming HTTP/WebSocket traffic to `127.0.0.1:` | +| `checkAndProxyOrbitDomain(w, r)` | Edge Routing | Intercepts requests with custom `Host` header, queries `cosmos.orbit_apps WHERE custom_domain = ?`, and reverse-proxies directly to the container port | +| `handleOrbitForgeRepos(w, r)` | `GET /orbit/api/forge/repos` | Fetches user's repositories from local Gitea instance via internal API | +| `handleOrbitTemplates(w, r)` | `GET /orbit/api/templates` | Returns curated starter templates for Node, Go, Python, and Static Web | +| `handleOrbitForgeWebhook(w, r)` | `POST /orbit/api/webhook/forge` | Receives Git push events from Forge/Gitea and triggers automated re-deployment | + +--- + +## 9. Systemd Services — Boot & Process Management + +### `portals.service` +**File:** `/etc/systemd/system/portals.service` + +```ini +[Unit] +Description=Unity Portals Server (Go) +After=network.target docker.service minio.service + +[Service] +User=debian +WorkingDirectory=/home/debian/portals-go +ExecStart=/home/debian/portals-go/portals +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +``` + +**Runtime stats (as of Sep 2026):** 49.2 MB RAM, ~8 tasks + +**Why `After=docker.service minio.service`?** Without this, portals might start before ScyllaDB or MinIO are ready. Even with this, ScyllaDB can take 1–3 minutes to initialize, and the Go binary handles this with its 60-retry loop. + +### `minio.service` +**File:** `/etc/systemd/system/minio.service` + +Runs as `root` (required for port :9000 binding and /opt/minio/data access). Uses env vars for credentials. **Runtime stats:** 238.9 MB RAM. + +### ScyllaDB (Docker) +Managed by Docker Compose at `/home/debian/portals-go/cosmos/docker-compose.yml`. Docker container auto-restarts on failure. The `cosmos` keyspace and the `unity` keyspace are created by `initDB()` on startup. + +### Service Management Commands +```bash +# Check all services at once +sudo systemctl status portals minio docker nginx --no-pager + +# Rebuild and restart (required after any code or HTML change) +cd /home/debian/portals-go && go build -o portals . && sudo systemctl restart portals + +# Restart only (no code changes) +sudo systemctl restart portals + +# View live logs +sudo journalctl -u portals -f # Follow portals logs +sudo journalctl -u minio -n 100 # Last 100 MinIO lines +sudo journalctl -u docker -n 50 # Docker logs + +# ScyllaDB container +sudo docker ps +sudo docker logs scylla-scylla-1 -f --tail 50 +sudo docker exec -it scylla-scylla-1 cqlsh # CQL shell +``` + +--- + +## 10. Frontend Templates — Page-by-Page Breakdown + +All templates live in `/home/debian/portals-go/templates/` and are compiled into the binary via `//go:embed templates/*`. After editing any file, you **must rebuild**. + +### `index.html` — Homepage (56.8 KB) +- **Route:** `GET /` +- **Served by:** `handleIndex()` +- **Design:** Full space/dark theme, animated canvas starfield background, hero section, feature cards for Oort/Git/CDN services +- **Responsive:** Yes, mobile-first CSS + +### `oort.html` — Oort Service Landing (28.6 KB) +- **Route:** `GET /oort` +- **Served by:** `handleServicePage(w, r, "oort")` +- **Design:** Updated by user — full-featured landing page with feature showcase, pricing/plans section +- **Links to:** `/oort/login`, `/oort/signup` + +### `login.html` — Authentication (19.1 KB) +- **Route:** `GET /{service}/login` +- **Design:** Space-themed dark card with animated canvas starfield, celestial bodies, and card parallax +- **Form action:** `POST /{service}/login` +- **Features:** Password visibility toggle, CapsLock warning, text scramble decode, and "Forgot password?" recovery link +- **On success:** + - For `oort` & `cdn`: Redirects to `/{service}/dashboard?sid=UUID` + - For `git`: Performs background Gitea authentication, sets `i_like_gitea` cookie, and redirects directly to `/source/` (Seamless SSO) +- **Template variables:** `{{.Service}}` (injected by Go), `{{if .Error}}{{.Error}}{{end}}` + +### `signup.html` — Registration (26.5 KB) +- **Route:** `GET /{service}/signup` +- **Design:** Matches login aesthetic with space background and corner crosshairs +- **Form fields:** Name, Email, Username, Password +- **On submit:** `POST /{service}/signup` → creates pending user (password kept in-memory only) → sends 6-digit OTP via msmtp (`noreply@cosmos.linuxguy.space`) → redirects to `/{service}/verify` +- **Verify page:** Inline HTML rendered by `handleVerify()` (NOT a separate template file). The verify page polls `/{service}/api/poll?sid=UUID` every 2 seconds until OTP is entered. +- **Template variables:** `{{.Service}}`, `{{if .Error}}{{.Error}}{{end}}` + +### `forgot-password.html` — Password Recovery & Reset (17.3 KB) +- **Route:** `GET /{service}/forgot-password` and `GET /{service}/reset-password?token=UUID` +- **Served by:** `handleForgotPassword()` and `handleResetPassword()` +- **Design:** Space-themed card matching `login.html` with full canvas celestial engine (stars, nebula, comets, meteors, planet rendering, text scramble decode, and mouse parallax). +- **Supports 6 distinct template states:** + 1. `request` — Email input form with anti-enumeration protection + 2. `request + Sent` — Visual confirmation ("Check your inbox") + 3. `reset` — New password form with interactive strength meter, requirement checklist (8+ chars, upper, lower, number/symbol), and real-time confirm match indicator + 4. `done` — Password updated confirmation with direct "Sign In" button + 5. `expired` — Expired token notice with link to request a new reset email + 6. `invalid` — Invalid/tampered token warning +- **Template variables:** `{{.Service}}`, `{{.Step}}`, `{{.Token}}`, `{{.Email}}`, `{{.Sent}}`, `{{.Error}}` + +### `oortconsole.html` — The Console SPA (575 KB, ~3,500+ lines) +See Section 11 for full details. This is the main application file. + +### `cloud.html` — Cloud Drive Landing (28.4 KB) +- **Route:** `GET /cdn` (via `handleServicePage`) +- **Design:** Updated by user — cloud drive service marketing page + +### `git.html` — Git Service Landing (31.6 KB) +- **Route:** `GET /git` (via `handleServicePage`) +- **Design:** Updated by user — git hosting service marketing page + +### `tailwind.css` / `tailwind.js` +- Served at `/tailwind.css` directly by the Go backend (from embedded FS) +- 2.9 MB CSS + 407 KB JS — used by auth pages and landing pages +- The Oort Console (`oortconsole.html`) does NOT use Tailwind; it has its own custom CSS + +--- + +## 11. The Oort Console — S3 Storage Dashboard (SPA) + +The Oort Console (`oortconsole.html`) is a ~3,500-line, zero-external-JS-framework single-page application served at `/oort/console`. Dependencies: **Chart.js** (request volume line chart), **D3.js** (3D globe rendering). + +### Architecture Overview +``` +oortconsole.html +├──