I have a home server that I run off of a Synology 920+ (a great choice for a home NAS and docker container host). On it, I run a number of docker containers that host many services I use on my home network, things like Flame as a home dashboard, PiHole, Snippet Box, Jupyer Notebooks, the *arr stack, along with a number of other services.
Whilst I have a lot of services, keeping them up to date can be a bit of a nuisance, with manually checking, manually updating, manually restarting. The key here is things are manual. Let me show how I try to do all of this automatically instead.
My main 3 requirements are:
- Reduce container downtime - I want to reduce downtime so I have all my services available on demand, and when a container isn’t working right, fix it automatically.
- Keep containers up to date - I want to be running the latest stable container all the time, without me having to manually complete all the necessary steps to do so.
- Notify me when things change - Keep me updated when containers are updated, so that I know when they happened, but importantly can monitor for any issues, and rollback a container if necessary
The automation magic
I use 3 main containers to help me keep all of my other containers up to date. I have listed them below, and their uses.
- Autoheal - Monitor and restart unhealthy docker containers.
- Watchtower - A process for automating Docker container base image updates.
- Diun - Notifying me via Pushover that my containers have been updated.
Autoheal
I have set Autoheal to watch all containers using an environment variable. If the container has a healthcheck on it, Autoheal will restart the container if it starts to become unhealthy. A container restart will solve ~99% of issues.
Note: Autoheal will not heal any containers that are linked to it, so its best to apply individual healthchecks to all containers.
To set a health status on the container you can add the healthcheck
key to a docker compose service. The one in the example below will check if the service has internet connectivity, and if it does not, will show as unhealthy.
service_name:
image: "containous/whoami"
container_name: whoami
healthcheck:
test: "curl -f api.ipify.org || exit 1"
interval: 60s
timeout: 30s
retries: 3
start_period: 30s
Watchtower
I have Watchtower set to update any container which has a specific label on it. This stops Watchtower updating any random containers I might have running. Watchtower will pull the latest image, as well as restarting the affected container with the new image. Finally if there are any containers depending on the updated container, it will restart those too.
To set a label on the container you can add the label
key to a docker compose service, as per the example below.
service_name:
image: "containous/whoami"
container_name: whoami
labels:
- "com.centurylinklabs.watchtower.enable=true"
Diun
I have set Diun to monitor all my containers every 6 hours and send me a notification via Pushover, when a new one has been created or updated. Diun can be configured in many ways, so its best to read the documentation to get the desired functionality.
Compose File
The following the docker compose file that can be started using docker-compose up -d
. Before starting you will need to update some of the PUSHOVER
variables, and the /data
volume mappings in the duin
service, to match your system/needs. As you can see all these services share the hosts docker socket file with the container, so the service can act on the hosts docker container status updates.
version: '3'
services:
autoheal:
container_name: autoheal
image: willfarrell/autoheal:latest
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- AUTOHEAL_CONTAINER_LABEL=all # Heal all containers that look unhealthy
diun:
container_name: diun
image: crazymax/diun:latest
restart: always
command: serve
volumes:
- ./data:/data
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
# Standard configuration from https://crazymax.dev/diun/usage/basic-example/
- TZ=Europe/London
- LOG_LEVEL=info
- LOG_JSON=false
- DIUN_WATCH_WORKERS=20
- DIUN_WATCH_SCHEDULE=0 */6 * * *
- DIUN_PROVIDERS_DOCKER=true
- DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true
# Pushover specific configuration from https://crazymax.dev/diun/notif/pushover/
- DIUN_NOTIF_PUSHOVER_TOKEN=<<APP API TOKEN>>
- DIUN_NOTIF_PUSHOVER_RECIPIENT=<<USER API TOKEN>>
- DIUN_NOTIF_PUSHOVER_TEMPLATETITLE=DOCKER > {{ .Entry.Image.Path }} was {{ if (eq .Entry.Status "new") }}created{{ else }}updated{{ end }}
- DIUN_NOTIF_PUSHOVER_TEMPLATEBODY=Image **{{ .Entry.Image.Path }}** {{ if (eq .Entry.Status "new") }}was **created**{{ else }}was **updated**{{ end }} at {{ .Entry.Manifest.Created.Format "02 Jan 2006 15:04:05" }}
watchtower:
container_name: watchtower
image: containrrr/watchtower:latest
restart: always
# Command line arguments https://containrrr.dev/watchtower/arguments/
# The following will update only containers with the label on them at 4am every day and remove the old images.
command: --cleanup --schedule "0 0 4 * * *" --label-enable
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Conclusion
At this moment the solution I have documented works for me and my use cases. I am aware there are probably other (probably better) solutions out there, but some of these utilise more advanced services such as Kubernetes, which I don’t understand at the moment. If you do have any other other ideas about how I can improve this further, feel free to email me.