{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","params":[],"results":{"codes":[]}},"next":{"description":"","pages":[]},"title":"Paz Service Discovery","type":"basic","slug":"paz-service-discovery","excerpt":"How Paz's service discovery and inter-service routing works","body":"[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"The Service Discovery Problem\"\n}\n[/block]\nIn a multi-host cluster of machines running Docker containers, we need a way for services to communicate with each other and receive requests from the outside world, yet do so in such a way that services can move hosts or come and go at any time. This is called *service discovery*.\n\nFor a Docker PaaS such as Paz, this needs to be dynamic. If service A wants to talk to service B, we cannot simply pass the IP address and port of any instance of service B to service A when service A starts, because that binds those two instances together and service A will become unusable if service B goes away. What we instead want is logical routing, whereby service A can talk to any instance of service B, and have the platform take care of the underlying changes as particular instances of service B come and go. We want service A to be able to use the service name of B as a routable address.\n\nPaz's service discovery solution is built around Etcd, Confd and HAProxy; in short it is a dynamically-configured load balancing proxy.\n\nImagine we have a \"web\" service that wants to talk to an \"api\" service, both of which are running in a container; we have multiple instances of each. A logical view of this service discovery problem might look like this:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/7Lfw4PkSryLB0EjIsWZg_Dynamic%20proxy%20-%20logical%20-%20black%20text.png\",\n        \"Dynamic proxy - logical - black text.png\",\n        \"890\",\n        \"233\",\n        \"#0d0e0f\",\n        \"\"\n      ],\n      \"caption\": \"Service discovery - logical view\"\n    }\n  ]\n}\n[/block]\nWe have a dynamically configured proxy that takes care of routing \"web\"s requests to \"api\" by service name.\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"How Paz's Service Discovery Works\"\n}\n[/block]\nPaz's service discovery magic comes about through a combination of HAProxy and Dnsmasq.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/l7wW377sR5aNicXEE3Wh_Paz%20HAProxy%20and%20dnsmasq%20-%20black%20text.png\",\n        \"Paz HAProxy and dnsmasq - black text.png\",\n        \"683\",\n        \"503\",\n        \"#151515\",\n        \"\"\n      ],\n      \"caption\": \"Paz service discovery stack\"\n    }\n  ]\n}\n[/block]\nEach Docker container running on a Paz host is started with `--dns=HOST_IP`. Bound to port `53` on each host is an instance of Dnsmasq, which has been configured to filter DNS requests from the host's containers, returning the address of the local HAProxy instance to those DNS requests for services within the cluster, and forwarding all other DNS requests on to a \"real\" DNS server on the Internet.\n\nThis HAProxy instance is dynamically configured with routing information for all other containers within the cluster. This dynamic configuration is done by a tool called [Confd](https://github.com/kelseyhightower/confd), a tool for writing configuration files (HAProxy config files in this case) from templates populated with data from Etcd.\n\nThe information Confd takes from Etcd is placed there by so-called *side-kick announce units*. For every Paz container there is a corresponding side-kick whose lifetime is bound to the Paz container using the systemd `BindsTo` directive (meaning that when the service starts the announce side-kick will also start, and when it stops so to will the announce sidekick).\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/JDZRF95YQluOjAyGE1zY_Paz%20Etcd%20service%20discovery%20-%20black%20text.png\",\n        \"Paz Etcd service discovery - black text.png\",\n        \"651\",\n        \"383\",\n        \"#0b0c0d\",\n        \"\"\n      ],\n      \"caption\": \"Paz Etcd service announcement\"\n    }\n  ]\n}\n[/block]\nThe announce side-kick's job is to use `docker inspect` to grab the port the container is bound to on the host and write the host IP and container port into Etcd. It writes it with a TTL, then it sleeps for slightly less than that time, and then writes it again. It does this in a loop. This means that as long as the container is running its IP and port will be written in Etcd as per the above diagram.\n\nConfd watches the `/paz/services` directory in Etcd and, on change, takes the values (for all services) and writes an HAProxy config file, then notifies HAProxy to reload its config.\n\nIn this way, HAProxy is always up-to-date with routing information for all services that are running and no services that are not (except for a short window of time before the TTL expires).\n\nBelow is an example excerpt of a Paz HAProxy config file generated by Confd:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"frontend http-in\\n    bind *:80\\n    acl subdom_myservice hdr(host) -i myservice.paz hdr(host) -i myservice.mydomain.com\\n    use_backend backend-myservice if subdom_myservice\\n    acl subdom_paz-web hdr(host) -i paz-web.paz hdr(host) -i paz-web.mydomain.com\\n    use_backend backend-paz-web if subdom_paz-web\\n\\nbackend backend-myservice\\n    balance roundrobin\\n    server myservice-v1-1 10.131.238.241:49177\\n    server myservice-v1-2 10.131.238.242:49165\\n\\nbackend backend-paz-web\\n    server paz-web 10.131.238.233:49164\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]","updates":[],"order":999,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"5522b5af85fa041700623976","version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":false,"is_hidden":false,"is_deprecated":false,"categories":["54fa0050f63cf9210041c3c2","54fa42b3961fea210092068e","54facefd5c41173700ecf296","54fad9d5f63cf9210041c669","54fae9ed5c41173700ecf2ad"],"_id":"54fa0050f63cf9210041c3c1","__v":5,"createdAt":"2015-03-06T19:30:24.258Z","project":"54fa004ff63cf9210041c3be","releaseDate":"2015-03-06T19:30:24.258Z"},"project":"54fa004ff63cf9210041c3be","user":"54fa0018e7a0ba2f00306211","__v":6,"githubsync":"","createdAt":"2015-04-06T16:34:55.649Z","category":{"sync":{"isSync":false,"url":""},"pages":["54fada9b5c41173700ecf29e","54fadb625c41173700ecf2a0","54fae365019b2537006d00bc","5500a5712e8c48170076d163","551a876868da7f0d00f58cbc","5521351e29a3dc0d00fae453","5522b5af85fa041700623976","556d98e442dd610d00a3cfde"],"title":"Getting Started","slug":"getting-started","order":0,"from_sync":false,"reference":false,"_id":"54fa42b3961fea210092068e","version":"54fa0050f63cf9210041c3c1","__v":8,"createdAt":"2015-03-07T00:13:39.405Z","project":"54fa004ff63cf9210041c3be"}}

Getting Started

Integrations

APIs

Videos

Contributing

Paz Service Discovery

How Paz's service discovery and inter-service routing works
[block:api-header] { "type": "basic", "title": "The Service Discovery Problem" } [/block] In a multi-host cluster of machines running Docker containers, we need a way for services to communicate with each other and receive requests from the outside world, yet do so in such a way that services can move hosts or come and go at any time. This is called *service discovery*. For a Docker PaaS such as Paz, this needs to be dynamic. If service A wants to talk to service B, we cannot simply pass the IP address and port of any instance of service B to service A when service A starts, because that binds those two instances together and service A will become unusable if service B goes away. What we instead want is logical routing, whereby service A can talk to any instance of service B, and have the platform take care of the underlying changes as particular instances of service B come and go. We want service A to be able to use the service name of B as a routable address. Paz's service discovery solution is built around Etcd, Confd and HAProxy; in short it is a dynamically-configured load balancing proxy. Imagine we have a "web" service that wants to talk to an "api" service, both of which are running in a container; we have multiple instances of each. A logical view of this service discovery problem might look like this: [block:image] { "images": [ { "image": [ "https://files.readme.io/7Lfw4PkSryLB0EjIsWZg_Dynamic%20proxy%20-%20logical%20-%20black%20text.png", "Dynamic proxy - logical - black text.png", "890", "233", "#0d0e0f", "" ], "caption": "Service discovery - logical view" } ] } [/block] We have a dynamically configured proxy that takes care of routing "web"s requests to "api" by service name. [block:api-header] { "type": "basic", "title": "How Paz's Service Discovery Works" } [/block] Paz's service discovery magic comes about through a combination of HAProxy and Dnsmasq. [block:image] { "images": [ { "image": [ "https://files.readme.io/l7wW377sR5aNicXEE3Wh_Paz%20HAProxy%20and%20dnsmasq%20-%20black%20text.png", "Paz HAProxy and dnsmasq - black text.png", "683", "503", "#151515", "" ], "caption": "Paz service discovery stack" } ] } [/block] Each Docker container running on a Paz host is started with `--dns=HOST_IP`. Bound to port `53` on each host is an instance of Dnsmasq, which has been configured to filter DNS requests from the host's containers, returning the address of the local HAProxy instance to those DNS requests for services within the cluster, and forwarding all other DNS requests on to a "real" DNS server on the Internet. This HAProxy instance is dynamically configured with routing information for all other containers within the cluster. This dynamic configuration is done by a tool called [Confd](https://github.com/kelseyhightower/confd), a tool for writing configuration files (HAProxy config files in this case) from templates populated with data from Etcd. The information Confd takes from Etcd is placed there by so-called *side-kick announce units*. For every Paz container there is a corresponding side-kick whose lifetime is bound to the Paz container using the systemd `BindsTo` directive (meaning that when the service starts the announce side-kick will also start, and when it stops so to will the announce sidekick). [block:image] { "images": [ { "image": [ "https://files.readme.io/JDZRF95YQluOjAyGE1zY_Paz%20Etcd%20service%20discovery%20-%20black%20text.png", "Paz Etcd service discovery - black text.png", "651", "383", "#0b0c0d", "" ], "caption": "Paz Etcd service announcement" } ] } [/block] The announce side-kick's job is to use `docker inspect` to grab the port the container is bound to on the host and write the host IP and container port into Etcd. It writes it with a TTL, then it sleeps for slightly less than that time, and then writes it again. It does this in a loop. This means that as long as the container is running its IP and port will be written in Etcd as per the above diagram. Confd watches the `/paz/services` directory in Etcd and, on change, takes the values (for all services) and writes an HAProxy config file, then notifies HAProxy to reload its config. In this way, HAProxy is always up-to-date with routing information for all services that are running and no services that are not (except for a short window of time before the TTL expires). Below is an example excerpt of a Paz HAProxy config file generated by Confd: [block:code] { "codes": [ { "code": "frontend http-in\n bind *:80\n acl subdom_myservice hdr(host) -i myservice.paz hdr(host) -i myservice.mydomain.com\n use_backend backend-myservice if subdom_myservice\n acl subdom_paz-web hdr(host) -i paz-web.paz hdr(host) -i paz-web.mydomain.com\n use_backend backend-paz-web if subdom_paz-web\n\nbackend backend-myservice\n balance roundrobin\n server myservice-v1-1 10.131.238.241:49177\n server myservice-v1-2 10.131.238.242:49165\n\nbackend backend-paz-web\n server paz-web 10.131.238.233:49164", "language": "text" } ] } [/block]