Firefly III on Docker with MariaDB, Redis & LDAP

Firefly III is a self-hosted personal finance manager.

A comprehensive review of Firefly III is coming soon. However for those who have chosen Firefly III as their choice of personal finance manager, this article describes an opinionated way of systematically deploying it and integrating it into an existing ecosystem.

Pre-requisites

Photo by Dan Cristian Pădureț

Containerized deployment of Firefly III on Docker has the following following pre-requisites.

• Docker
• Database service like MariaDB (MySQL) or PostgreSQL

In addition the following optional dependencies are required.

• NGINX, Apache2, Caddy or another reverse proxy - for accessing the instance from outside the local network
• Authelia - for federated authentication
• Redis - for cache
• Adminer - for browsing the database
• Redis Commander - for browsing the Redis cache

Pre-Installation

Photo by Tara Winstead

Directories

This guide assumes that local directories/folders on the Docker host will be used to map volumes into the various containers in the app stack. Here's a summary.

1. Configuration
   1. Main app container configuration
   2. Database container configuration
2. Secrets
   1. Main app secrets files
   2. Database container secrets files
3. Data
   1. Main app data - storage/uploads
   2. Cache (Redis) container data - for persistent cache
   3. Database data

Configuration Folders

Create a configuration space and a protected area for secrets.

Main App Configuration Folder

/usr/local/Ecosystem/etc/FF3App/DirMount/Config
/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/

Database Configuration Folder

/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/
/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/

Configuration Folder Command

sudo mkdir -p /usr/local/Ecosystem/etc/FF3App/DirMount/{DB,}Config/private/

Secrets Files

Create the following secrets files

Main App

The following secrets file are required for the main app container.

App Key

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/APP_KEY

Contents

32-CHARACTER-APP-KEY

Database User

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/DB_USERNAME

Contents

DATABASE_USER

Database Password

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/DB_PASSWORD

Contents

PASSWORD FOR DATABASE_USER

Database Name

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/DB_DATABASE

Contents

DATABASE_NAME

SMTP Password

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/MAIL_PASSWORD

Contents

SMTP PASSWORD

Database

The following secrets files are required for the database container.

Database Root Password

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/MARIADB_ROOT_PASSWORD

Contents

DATABASE ROOT PASSWORD

Database User

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/MARIADB_USER

Contents

DATABASE_USER

Database Password

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/MARIADB_PASSWORD

Contents

PASSWORD FOR DATABASE_USER

Database Name

Location

/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/MARIADB_DATABASE

DATABASE_NAME

Data Directories

The data directories enhance the persistence of otherwise ephemeral containers. These host directories are mapped as volumes inside the container so that the data is retained even when the containers are stopped, updated, replaced or destroyed. This makes the upgrade process very smooth and seamles for the end user.

Main App

The Firefly III Docker container maps a volume to the following location inside the container for user uploads (like receipts)

/var/www/html/storage/upload

Create a directory on the host

/var/local/Apps/FF3App/DirMount/Upload

Persistent Cache

The cache container (Redis) optionally provides the option to make the cache persistent across container restarts & ugprades. It stores this inside the container in the following location:

/data

Create a directory on the host

/var/local/Apps/FF3App/DirMount/CacheData

Database Data

The database (MariaDB) stores all of data (databases, tables, metadata, user data, etc.) in the following location inside the container.

/var/lib/mysql

Create a directory on the host

/var/local/Apps/FF3App/DirMount/DBData

Permissions

Ensure that the secrets files have the appropriate permissions

sudo chmod o= /usr/local/Ecosystem/etc/FF3App/DirMount/Config/private/*
sudo chmod o= /usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig/private/*

Installation

Photo by Rodolfo Quirós

Container Stack Deployment

Photo Kaboompics.com

This guide assumes that the optional components identified in Pre-requisites will be deployed.

Install using Docker ^{[1] [2] [3] [4] [5]}.

Create the Docker Compose file in the following location

/usr/local/Ecosystem/etc/FF3App/docker-compose.yaml

Define the application stack in the Docker Compose file. This guide utilizes the expanded syntax. This is especially useful for homelabs & enthusiasts who may not be spending most of their time maintaining their self-hosted applications. The verbosity helps self-document the application stack & should not be underestimated.

#version: '3' # Version property is deprecated in the latest version of the Docker Compose schema.

name: "ff3app_stack"


services:

  main:

    container_name: "FF3App_Main"
    restart: "unless-stopped"
    depends_on:
      - "cache"
      - "db"

    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "256m"

    networks:
      net_app:
        ipv4_address: "172.24.6.2"
    hostname: "ff3app"
    extra_hosts:
      - "docker.host:172.24.6.1"
    # ports:
    #   - target: 8096
    #     host_ip: "0.0.0.0"
    #     published: 38096
    #     protocol: "tcp"
    #     mode: "host"

    volumes:
      - type: "bind"
        source: "/usr/local/Ecosystem/etc/FF3App/DirMount/Config"
        target: "/etc/FF3App"
        read_only: true
      - type: "bind"
        source: "/var/local/Apps/FF3App/DirMount/Upload"
        target: "/var/www/html/storage/upload"
        read_only: false

    environment:
      TRUSTED_PROXIES: "**"
      APP_DEBUG: "false"
      APP_LOG_LEVEL: "notice"
      LOG_CHANNEL: "stack"
      AUDIT_LOG_LEVEL: "info"
      DEFAULT_LANGUAGE: "en_GB"
      TZ: "Europe/London"
      APP_KEY_FILE: "/etc/FF3App/private/appkey"
      APP_NAME: "FF3App"
      APP_ENV: "local"
      APP_URL: "https://ff3app.ecosystem.tld"
      SITE_OWNER: "[email protected]"
      DB_CONNECTION: "mysql"
      DB_HOST: "ff3app-db"
      DB_PORT: "3306"
      DB_USERNAME_FILE: "/etc/FF3App/private/dbuser"
      DB_PASSWORD_FILE: "/etc/FF3App/private/dbpw"
      DB_DATABASE_FILE: "/etc/FF3App/private/db"
      CACHE_DRIVER: "redis"
      SESSION_DRIVER: "redis"
      REDIS_SCHEME: "tcp"
      REDIS_HOST: "ff3app-cache"
      REDIS_PORT: "6379"
      REDIS_DB: "10"
      REDIS_CACHE_DB: "11"
      AUTHENTICATION_GUARD: "remote_user_guard"
      AUTHENTICATION_GUARD_HEADER: "HTTP_REMOTE_USER"
      AUTHENTICATION_GUARD_EMAIL: "HTTP_REMOTE_EMAIL"
      CUSTOM_LOGOUT_URL: "https://auth.ecosystem.tld/logout/?rd=https%3A%2F%2Fff3app.ecosystem.tld"
      MAIL_MAILER: "smtp"
      MAIL_HOST: "smtp.mail.tld"
      MAIL_PORT: "587"
      MAIL_ENCRYPTION: "tls"
      MAIL_USERNAME: "[email protected]"
      MAIL_PASSWORD_FILE: "/etc/FF3App/private/smtppw"
      MAIL_FROM: "[email protected]"
      SEND_REGISTRATION_MAIL: "true"
      SEND_ERROR_MESSAGE: "true"
      SEND_LOGIN_NEW_IP_WARNING: "true"

    labels:
      info.ecosystem.provider.name: "Firefly III"
      info.ecosystem.provider.variant: "N.A."
      info.ecosystem.provider.version: "6.0.11"
      info.ecosystem.service.name: "FF3App"

    image: "docker.io/fireflyiii/core:version-6.0.11"


  cache:

    container_name: "FF3App_Cache"
    restart: "unless-stopped"

    deploy:
      resources:
        limits:
          cpus: "1"
          memory: "512m"

    healthcheck:
      test: "/usr/local/bin/redis-cli ping"
      interval: "60s"
      timeout: "10s"
      start_period: "20s"
      retries: 3

    networks:
      net_app:
        ipv4_address: "172.24.6.4"
    hostname: "ff3app-cache"
    extra_hosts:
      - "docker.host:172.24.6.1"
      - "ff3app:172.24.6.2"
      - "ff3app-db:172.24.6.6"
      - "ff3app-db-explorer:172.24.6.8"
      - "ff3app-cache-explorer:172.24.6.10"

    volumes:
      - type: "bind"
        source: "/var/local/Apps/FF3App/DirMount/CacheData"
        target: "/data"
        read_only: false

    labels:
      info.ecosystem.provider.name: "Redis"
      info.ecosystem.provider.variant: "Bullseye"
      info.ecosystem.provider.version: "7.0.11"
      info.ecosystem.service.name: "FF3App"

    image: "docker.io/library/redis:7.0.11-bullseye"


  db:

    container_name: "FF3App_DB"
    restart: "unless-stopped"

    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "512m"

    healthcheck:
      test: "/usr/local/bin/healthcheck.sh --connect"
      interval: "60s"
      timeout: "10s"
      start_period: "20s"
      retries: 3

    networks:
      net_app:
        ipv4_address: "172.24.6.6"
    hostname: "ff3app-db"
    extra_hosts:
      - "docker.host:172.24.6.1"
      - "ff3app:172.24.6.2"
      - "ff3app-cache:172.24.6.4"
      - "ff3app-db-explorer:172.24.6.8"
      - "ff3app-cache-explorer:172.24.6.10"
    # ports:
    #   - target: 3306
    #     host_ip: "0.0.0.0"
    #     published: 3306
    #     protocol: "tcp"
    #     mode: "host"

    volumes:
      - type: "bind"
        source: "/usr/local/Ecosystem/etc/FF3App/DirMount/DBConfig"
        target: "/etc/FF3AppDB"
        read_only: false
      - type: "bind"
        source: "/var/local/Apps/FF3App/DirMount/DBData"
        target: "/var/lib/mysql"
        read_only: false
      # - type: "bind"
      #   source: "/var/local/Apps/FF3App/DirMount/InitDB"
      #   target: "/docker-entrypoint-initdb.d"
      #   read_only: true

    environment:
      MARIADB_ROOT_PASSWORD_FILE: "/etc/FF3AppDB/private/dbrootpw"
      MARIADB_USER_FILE: "/etc/FF3AppDB/private/dbuser"
      MARIADB_PASSWORD_FILE: "/etc/FF3AppDB/private/dbpw"
      MARIADB_DATABASE_FILE: "/etc/FF3AppDB/private/db"

    labels:
      info.ecosystem.provider.name: "MariaDB"
      info.ecosystem.provider.variant: "Jammy"
      info.ecosystem.provider.version: "10.11.3"
      info.ecosystem.service.name: "FF3App"

    image: "docker.io/library/mariadb:10.11.3-jammy"


  db-explorer:

    container_name: "FF3App_DB-Explorer"
    restart: "unless-stopped"
    depends_on:
      - "db"

    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "128m"

    networks:
      net_app:
        ipv4_address: "172.24.6.8"
    hostname: "ff3app-db-explorer"
    extra_hosts:
      - "docker.host:172.24.6.1"
      - "ff3app:172.24.6.2"
      - "ff3app-cache:172.24.6.4"
      - "ff3app-db:172.24.6.6"
      - "ff3app-cache-explorer:172.24.6.10"
    # ports:
    #   - target: 8080
    #     host_ip: "0.0.0.0"
    #     published: 8080
    #     protocol: "tcp"
    #     mode: "host"

    environment:
      TZ: 'Europe/London'
      ADMINER_DEFAULT_SERVER: "ff3app-db"

    labels:
      info.ecosystem.provider.name: "Adminer"
      info.ecosystem.provider.variant: "Standalone"
      info.ecosystem.provider.version: "4.8.1"
      info.ecosystem.service.name: "FF3App"

    image: "docker.io/library/adminer:4.8.1-standalone"

  cache-explorer:

    container_name: "FF3App_Cache-Explorer"
    restart: "unless-stopped"

    deploy:
      resources:
        limits:
          cpus: "0.25"
          memory: "128m"

    healthcheck:
      test: "/usr/bin/wget --no-verbose --quiet --tries=1 --spider http://localhost:8081 || exit 1"
      interval: "60s"
      timeout: "10s"
      start_period: "20s"
      retries: 3

    networks:
      net_app:
        ipv4_address: "172.24.6.10"
    hostname: "ff3app-cache-explorer"
    extra_hosts:
      - "docker.host:172.24.6.1"
      - "ff3app:172.24.6.2"
      - "ff3app-cache:172.24.6.4"
      - "ff3app-db:172.24.6.6"
      - "ff3app-db-explorer:172.24.6.8"

    environment:
      TZ: 'Europe/London'
      REDIS_HOSTS: "FF3App-Cache:ff3app-cache"

    labels:
      info.ecosystem.provider.name: "Redis Commander"
      info.ecosystem.provider.variant: "N.A."
      info.ecosystem.provider.version: "0.8.1"
      info.ecosystem.service.name: "FF3App"

    image: "ghcr.io/joeferner/redis-commander:0.8.1"


networks:
  net_app:
    name: "FF3App_net"
    driver: "bridge"
    ipam:
      driver: "default"
      config:
        - subnet: "172.24.6.0/24"

Change the following as per the installation:

• APP_DEBUG
• APP_LOG_LEVEL
• APP_KEY
• APP_ENV
• DB_PASSWORD
• LDAP_PASSWORD
• MAIL_PASSWORD
• Docker image tag.
• Container labels to match the release version.

Configuration & Setup

Photo by Kishan Rahul Jose

Most of the configuration of the application stack is taken care of in the container environment variables in the stack definitions.

Additional configuration is required for associated applications and environment. This is described below.

Authentication Configuration

Photo by Pixabay

This guide assumes that a working & updated instance of Authelia is available. The following guidance focusses on elements of the Authelia configuration that are relevant to the Firefly III app stack being deployed.

Authentication Headers

To ensure that Authelia injects the correct authentication headers in every HTTP request it proxies through to the app, the following code block must be present to ensure that the name of the headers matches what Firefly III expects.

File: /etc/nginx/conf.d/auth-request.auth.authelia (... indicates continuation to & from other existing code in the file)

...
  proxy_set_header Remote-User         $user;
  proxy_set_header Remote-Groups       $groups;
  proxy_set_header Remote-Name         $name;
  proxy_set_header Remote-Email        $email;
...

... indicates continuation to & from other existing code in the file

Reverse Proxy

Accessing the app

Photo by Martin Cheung

This assumes that NGINX will be used as the reverse proxy to access the app. Create a virtual host configuration file in the NGINX area for host configuration files. Typically this is at /etc/nginx/sites-available/. For your installation of NGINX, for your your reverse proxy, this might be different.

  server {
    server_name           ff3app.ecosystem.tld;

    error_log             /var/log/nginx/FF3App_error.log debug;
    access_log            /var/log/nginx/FF3App_access.log combined;

    server_tokens         off;
    client_max_body_size  512M; # Influences maximum file size for uploads.

    # Error pages
    include               /etc/nginx/conf.d/errorblock;

    listen                80;
    if ($scheme = http) {
      # Force redirection to HTTPS.
      return              301 https://$host:443$request_uri;
    }
    listen                443 ssl;
    ssl_certificate       /etc/ssl/certs/FF3App.crt.pem;
    ssl_certificate_key   /etc/ssl/private/FF3App.privkey.pkcs8.pem;

    include               /etc/nginx/conf.d/auth-endpoint.auth.authelia;

    location / {
      # Reverse proxy configuration
      include             /etc/nginx/conf.d/proxy.auth.authelia;

      # Activate Authelia for specified route/location, please ensure you have setup the domain in your configuration.yml
      include             /etc/nginx/conf.d/auth-request.auth.authelia;
      
      # Pass to Docker container on its own isolated network
      proxy_pass          http://172.24.6.2:8080;
    }

  }

Automations

Leveraging FF3's full functionaliy

Photo by Pavel Danilyuk

Firefly III comes with built-in functionality for recurring transactions, rules and regular budget allocation.

Since it is a PHP application, external automation is required for running these operations.

Execute the command by accessing the container from the Docker host ^[1].

Add a daily cron job using curl

/etc/cron.d/daily/Apps

/usr/bin/docker exec --user www-data FF3App_Main /usr/local/bin/php /var/www/html/artisan firefly-iii:cron

Backup & Restore

Managing disaster risk

Photo by Алекс Арцибашев on Unsplash

Irrespective of any underlying backup strategies, an application level backup plan is essential for atomically mitigating any disaster risk without disrupting the wider ecosystem utilizing the underlying system(s).

The following aspects of the application shoudl be considered for backing up.

• Database
• Configuration
• Uploads & other file data

Database

The database is the single most important aspect of all persistence information across the entire application stack.

The approach used in this article is to save the database data in a location on the Docker host as a bind mount into the database container.

1. Filesystem

The filesystem location of the database data on the Docker host can be backed up using several methods like archiving/compression, backup tools (e.g., borg & restic) or filesystem snapshots (e.g., ZFS or BTRFS).

As per the guidance in this article, this is located in...

/var/local/Apps/FF3App/DirMount/DBData

Alternatively the default location of volumes within the Docker directory hierarchy may be used.

2. SQL Data Export

Ultimately, it is the data within the database that is of real importance. So a traditional SQL export of the database can be performed using either the CLI, a database client or the Adminer container designed into the stack.

In this example a command line approach is used, which can then be scheduled as a cron job.

Create the following backup script.

#!/bin/sh
  /usr/bin/mysqldump --defaults-extra-file=/path/to/.my.cnf --host=localhost --user=dbadmin --opt --verbose DATABASE_NAME | /bin/bzip2 -9  > /path/to/backups/Ecosystem/Service/FF3App/Backup/DB/DATABASE_NAME_`date +"%Y-%m-%d-%H-%M-%S"`_MariaDB_FF3App-Ecosystem.sql.bz2

This may require logging into the database container first:

docker exec -it FF3App_DB bash

Configuration

The configuration files for all containers in this guide have been located in a single location

/usr/local/Ecosystem/etc/FF3App/

This also includes the docker-compose.yml file, which is a crucial file for future rebuilding of the application stack (as long as the various pieces of the data are available)

App Data

Finally the following data must also be backed by whatever means are available.

Firefly III Uploads

/var/www/html/storage/upload

Other Tasks

Maintaining & supporting

Photo by Yosafat Herdian on Unsplash

There is a variety of operations for maintaing and supporting an instance of Firefly III. This section will cover some common scenarios encountered by the typical self-hoster.

Exporting Transactions

Firefly III API supports exporting data via the command line ^[1]

Enter the container’s command line.

docker exec -it FF3App_Main bash

Export Transactions For Specific Account(s)

Get the token string for the user whose account data needs to be accessed.

Get the Firefly III internal index of the account to be exported. This can be achieved by navigating to the account on the web UI. It is typically one of the slugs in the URL string. The example below uses the account index 58.

Once logged into the container's CLI, run the following command to export transactions for a specific account.

php artisan firefly-iii:export-data --export-transactions --accounts=58 --token=USER-TOKEN-STRING