We pick Elixir
A command router on a node
The Flaem router on every node holds the command stream from the core, the job queue and the reports. A slow Proxmox call runs in a separate task, so it does not deafen the rest of the node.
Recovering from failure
An OTP supervisor restarts a process that crashed while the rest of the system keeps running. A closed stream to the core ends in a reconnect, not in a service failure.
A report on what really runs
When the core asks, the node lists the machines that exist in Proxmox and the game servers it hosts. An error while gathering ends in an error report, not in a stopped command stream.
Local node state
The node keeps its servers, assigned addresses, tunnels and SFTP credentials in SQLite through Ecto. A process restart does not lose what the node knows about itself.
A Redis-compatible cache
excache speaks the RESP protocol, so ordinary Redis clients connect to it. The keys are split between supervised processes, and each process handles its own keys in turn, with no locks.
Every connection in its own process
Erlang VM processes are light, so in excache every client connection gets its own. A stuck or dropped client does not block the others.
