Paz Architecture

Back-End Services

The Scheduler and Service Directory are simple Node.js services with REST interfaces. The Orchestrator is like a front-end API that pulls together functionality from the Service Directory as well as Etcd and Fleet. The Orchestrator can also update DNS records for external-facing services.

Architecture of a Paz Cluster

A Paz cluster is simply a CoreOS cluster on which some Paz services have been installed, as Docker containers. Paz leverages CoreOS' Fleet and Etcd to abstract the hosts and schedule services across the cluster as if it were one big host.

As can be seen in this diagram, Paz can be seen as a layer on top of Fleet. Each host run CoreOS, which is clustered by default using Etcd, and Fleet presents an abstraction of systemd across this cluster. Paz sits on top of this and utilises Fleet and Etcd under the hood.

The above stack is what comes "out of the box" from the Vagrant and DigitalOcean cloud-init user-data files in the Paz repository. This is what every host in a Paz cluster looks like.

CoreOS - A minimalistic, auto-updating Linux distribution designed for running Docker containers.
systemd - The init system that comes with CoreOS. systemd is ultimately what will be starting and stopping our services. systemd will run our services based on unit files, text files that describe how to start and stop our services, and these are submitted to systemd by Fleet.
Docker - The Docker LXC daemon. Started by systemd at boot time, and will launch Docker containers for our services when asked by systemd.
Fleet - An abstraction of systemd at a cluster rather than host level. Uses Etcd to store unit files and to coordinate the scheduling of services across the cluster. The Paz scheduler writes unit files for user services and submits them to Fleet, which ultimately means they become a systemd unit on a host in the cluster.
paz-haproxy - An instance of HAProxy that is dynamically configured by confd. Described in detail in the Paz Service Discovery.
paz-dnsmasq - All DNS queries from Docker containers run through this simple service, which simply forwards all requests for other services in the cluster to paz-haproxy, and will forward everything else on to the Internet.
paz-pubkey-watcher - Because paz-scheduler runs in a Docker container and needs to communicate with Fleet via SSH, it needs to have an SSH key. paz-scheduler will generate a keypair on startup and put the public key into Etcd. paz-pubkey-watcher watches this key in Etcd and modifies the host's ~/.ssh/authorized_keys file with that key for the core user, allowing the scheduler to communicate with Fleet.