Federated Learning

This is a complete end-to-end federated learning example. Proplets train models locally on their private data, and the coordinator aggregates updates using the FedAvg algorithm. The model never leaves the device — only weight updates are shared.

For a comprehensive guide on federated learning architecture, motivation, and the complete training lifecycle, see the Federated Machine Learning documentation.

Prerequisites

Before starting, ensure you have the following installed and configured:

Required Software

Docker and Docker Compose (v2.x or later)
Go 1.25.5 or later (for building WASM)
Python 3 with pip (for provisioning script)
jq (for JSON formatting in terminal)
curl (for API requests)

GitHub Container Registry (GHCR) Setup

You need a GitHub Personal Access Token (PAT) with read:packages and write:packages scopes to push WASM to GHCR:

Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic)
Generate new token with scopes: read:packages, write:packages, delete:packages
Save the token securely — you'll need it later

Port Availability Check

The FL demo requires several ports. Check that these are available:

# Check if port 1883 (MQTT) is in use
sudo lsof -i :1883

If you have mosquitto or another MQTT broker running as a system service, it will conflict with SuperMQ's MQTT adapter. Stop it before proceeding:

sudo systemctl stop mosquitto
sudo systemctl disable mosquitto  # Optional: prevent auto-start

You can verify the port is free:

sudo lsof -i :1883
# Should return nothing

Clone the Repository

git clone https://github.com/absmach/propeller.git
cd propeller

Source Code

The FL client source code is available in the examples/fl-demo/client-wasm directory.

Loading...

Step 1: Clean Environment (Optional but Recommended)

If you've run the demo before, start fresh to avoid stale state:

# Stop and remove all containers, networks, and volumes
docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env down -v

# Remove any orphan containers
docker container prune -f

# Verify nothing is bound to required ports
sudo lsof -i :1883 -i :7070 -i :8080 -i :8083 -i :8084 -i :8085 -i :8086

Step 2: Build Docker Images

The FL demo extends the base SuperMQ + Propeller stack with FL-specific services. You must use both compose files together.

The FL demo compose file (examples/fl-demo/compose.yaml) is an extension to the base compose file (docker/compose.yaml). Running it standalone will fail with:

service "local-data-store" refers to undefined network supermq-base-net

Always use both compose files together.

Build All Required Images

Manager and proplet must be built from source as they include FL endpoints:

cd propeller
docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env build \
  manager proplet proplet-2 proplet-3 coordinator-http

Your output should look like this:

[+] Building 127.3s (52/52) FINISHED                                   docker:default
 => [manager internal] load build definition from Dockerfile                     0.0s
 => => transferring dockerfile: 1.23kB                                           0.0s
 => [proplet internal] load build definition from Dockerfile                     0.0s
 => => transferring dockerfile: 1.45kB                                           0.0s
 => [coordinator-http internal] load build definition from Dockerfile            0.0s
 => => transferring dockerfile: 892B                                             0.0s
 => [manager internal] load metadata for docker.io/library/golang:1.25-alpine    1.2s
 => [manager internal] load .dockerignore                                        0.0s
 => [proplet internal] load .dockerignore                                        0.0s
 => [coordinator-http internal] load .dockerignore                               0.0s
 => [manager builder 1/6] FROM docker.io/library/golang:1.25-alpine@sha256:...   0.0s
 => [manager internal] load build context                                        0.1s
 => => transferring context: 2.34MB                                              0.1s
 => CACHED [manager builder 2/6] WORKDIR /app                                    0.0s
 => CACHED [manager builder 3/6] COPY go.mod go.sum ./                           0.0s
 => CACHED [manager builder 4/6] RUN go mod download                             0.0s
 => [manager builder 5/6] COPY . .                                               0.8s
 => [manager builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -o manager ...  45.2s
 => [proplet builder 5/6] COPY . .                                               0.8s
 => [proplet builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -o proplet ...  52.1s
 => [coordinator-http builder 5/6] COPY . .                                      0.3s
 => [coordinator-http builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build ...    23.4s
 => [manager stage-1 1/3] FROM docker.io/library/alpine:latest                   0.0s
 => [manager stage-1 2/3] RUN apk --no-cache add ca-certificates                 1.2s
 => [manager stage-1 3/3] COPY --from=builder /app/manager /usr/local/bin/       0.1s
 => [proplet stage-1 3/3] COPY --from=builder /app/proplet /usr/local/bin/       0.1s
 => [coordinator-http stage-1 3/3] COPY --from=builder /app/coordinator ...      0.1s
 => exporting to image                                                           0.5s
 => => exporting layers                                                          0.4s
 => => writing image sha256:a1b2c3d4...                                          0.0s
 => => naming to docker.io/library/propeller-manager:latest                      0.0s
 => => naming to docker.io/library/propeller-proplet:latest                      0.0s
 => => naming to docker.io/library/propeller-proplet-2:latest                    0.0s
 => => naming to docker.io/library/propeller-proplet-3:latest                    0.0s
 => => naming to docker.io/library/fl-demo-coordinator-http:latest               0.0s

Step 3: Start All Services

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env up -d

Your output should look like this:

[+] Running 36/36
 ✔ Network supermq-base-net              Created                              0.1s
 ✔ Container supermq-nats                Started                              0.8s
 ✔ Container supermq-rabbitmq            Started                              0.9s
 ✔ Container supermq-spicedb-db          Started                              0.7s
 ✔ Container supermq-auth-db             Started                              0.8s
 ✔ Container supermq-users-db            Started                              0.7s
 ✔ Container supermq-clients-db          Started                              0.8s
 ✔ Container supermq-domains-db          Started                              0.9s
 ✔ Container supermq-channels-db         Started                              0.8s
 ✔ Container supermq-spicedb             Started                              1.2s
 ✔ Container supermq-spicedb-migrate     Started                              1.5s
 ✔ Container supermq-auth                Started                              2.1s
 ✔ Container supermq-users               Started                              2.3s
 ✔ Container supermq-domains             Started                              2.4s
 ✔ Container supermq-clients             Started                              2.5s
 ✔ Container supermq-channels            Started                              2.6s
 ✔ Container supermq-nginx               Started                              2.8s
 ✔ Container supermq-mqtt                Started                              2.9s
 ✔ Container fl-demo-model-registry      Started                              1.1s
 ✔ Container fl-demo-aggregator          Started                              1.2s
 ✔ Container fl-demo-local-data-store    Started                              1.3s
 ✔ Container fl-demo-coordinator         Started                              1.4s
 ✔ Container propeller-manager           Started                              3.1s
 ✔ Container propeller-proplet           Started                              3.2s
 ✔ Container propeller-proplet-2         Started                              3.3s
 ✔ Container propeller-proplet-3         Started                              3.4s
 ✔ Container propeller-proxy             Started                              3.5s
 ✔ Container local-registry              Started                              1.0s

This starts:

SuperMQ stack: Auth, Domains, Clients, Channels, MQTT Adapter, Nginx, SpiceDB
FL services: Model Registry (8084), Aggregator (8085), Local Data Store (8083), Coordinator (8086)
Propeller services: Manager (7070), 3 Proplets, Proxy

Verify Services Are Running

Wait 10-15 seconds for services to initialize, then check:

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}"

Expected output (should show 36 containers, all "Up"):

NAME                          STATUS          PORTS
fl-demo-aggregator            Up 10 seconds   0.0.0.0:8085->8080/tcp
fl-demo-coordinator           Up 10 seconds   0.0.0.0:8086->8080/tcp
fl-demo-local-data-store      Up 10 seconds   0.0.0.0:8083->8080/tcp
fl-demo-model-registry        Up 10 seconds   0.0.0.0:8084->8080/tcp
local-registry                Up 10 seconds   0.0.0.0:5000->5000/tcp
propeller-manager             Up 10 seconds   0.0.0.0:7070->7070/tcp
propeller-proplet             Up 10 seconds
propeller-proplet-2           Up 10 seconds
propeller-proplet-3           Up 10 seconds
propeller-proxy               Up 10 seconds
supermq-auth                  Up 10 seconds   0.0.0.0:8189->8189/tcp
supermq-auth-db               Up 10 seconds   5432/tcp
supermq-channels              Up 10 seconds   0.0.0.0:9005->9005/tcp
supermq-channels-db           Up 10 seconds   5432/tcp
supermq-clients               Up 10 seconds   0.0.0.0:9006->9006/tcp
supermq-clients-db            Up 10 seconds   5432/tcp
supermq-domains               Up 10 seconds   0.0.0.0:9003->9003/tcp
supermq-domains-db            Up 10 seconds   5432/tcp
supermq-mqtt                  Up 10 seconds   0.0.0.0:1883->1883/tcp
supermq-nats                  Up 10 seconds   4222/tcp, 6222/tcp, 8222/tcp
supermq-nginx                 Up 10 seconds   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp
supermq-rabbitmq              Up 10 seconds   5672/tcp, 15672/tcp
supermq-spicedb               Up 10 seconds   50051/tcp
supermq-spicedb-db            Up 10 seconds   5432/tcp
supermq-spicedb-migrate       Up 10 seconds
supermq-users                 Up 10 seconds   0.0.0.0:9002->9002/tcp
supermq-users-db              Up 10 seconds   5432/tcp

You can quickly verify the core services are running:

curl -s http://localhost:7070/health && echo ""
curl -s http://localhost:8086/health && echo ""

Your output should look like this:

OK
{"status":"ok"}

Step 4: Provision SuperMQ Resources

Before the manager and proplets can connect to MQTT, you must provision the necessary SuperMQ resources (domain, channel, and clients).

Provisioning creates persistent data in SuperMQ databases. Only needs to be repeated if you remove Docker volumes (e.g., docker compose down -v).

Install Python Dependencies

pip install -r examples/fl-demo/requirements.txt

Your output should look like this:

Collecting requests
Collecting python-dotenv
Successfully installed requests-2.31.0 python-dotenv-1.0.0

Run the Provisioning Script

(cd examples/fl-demo && python3 provision-smq.py)

Your output should look like this:

============================================================
SuperMQ Provisioning Script for FL Demo
============================================================
Waiting for Users service...
✓ Users service is ready
Waiting for Domains service...
✓ Domains service is ready
Waiting for Clients service...
✓ Clients service is ready
Waiting for Channels service...
✓ Channels service is ready

=== Logging in ===
✓ Login successful

=== Creating Domain ===
✓ Domain created: 4e484e5b-0498-4453-adf8-a6fa6a37454b

=== Creating Clients ===
✓ Client created: manager (ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890)
✓ Client created: proplet-1 (ID: bf7e3581-4dac-472c-bcd9-a83a6ca53197)
✓ Client created: proplet-2 (ID: cf3738c1-07db-41c7-8fc3-234202cf3642)
✓ Client created: proplet-3 (ID: 095d2d54-4017-4da1-b589-0809d64aa517)
✓ Client created: fl-coordinator (ID: e5f6g7h8-1234-5678-abcd-ef9012345678)
✓ Client created: proxy (ID: i9j0k1l2-3456-7890-abcd-ef1234567890)

=== Creating Channel ===
✓ Channel created: c1578c6c-598e-4dbe-9a8b-5d2acd623679

=== Connecting Clients to Channel ===
✓ Connected 6 clients to channel

============================================================
Provisioning Summary
============================================================
Domain ID: 4e484e5b-0498-4453-adf8-a6fa6a37454b
Channel ID: c1578c6c-598e-4dbe-9a8b-5d2acd623679

Clients:
  manager:
    ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
    Key: <secret>
  proplet-1:
    ID: bf7e3581-4dac-472c-bcd9-a83a6ca53197
    Key: <secret>
  proplet-2:
    ID: cf3738c1-07db-41c7-8fc3-234202cf3642
    Key: <secret>
  proplet-3:
    ID: 095d2d54-4017-4da1-b589-0809d64aa517
    Key: <secret>
  fl-coordinator:
    ID: e5f6g7h8-1234-5678-abcd-ef9012345678
    Key: <secret>
  proxy:
    ID: i9j0k1l2-3456-7890-abcd-ef1234567890
    Key: <secret>

✓ Provisioning completed successfully!

✓ Updated docker/.env with new credentials

Note: Recreate services to apply new credentials (use --force-recreate, not restart):
  docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env up -d --force-recreate manager coordinator-http proplet proplet-2 proplet-3 proxy

The proplet-1, proplet-2, and proplet-3 client IDs shown in the summary are SuperMQ-generated UUIDs — created when the script registers each proplet as a client in your SuperMQ domain. These are what you pass as participants when triggering FL experiments. The script automatically writes them to docker/.env.

Verify Environment Variables Were Set

grep -E "PROPLET.*CLIENT_ID|MANAGER_DOMAIN_ID|MANAGER_CHANNEL_ID" docker/.env | grep -v '=$'

Your output should look like this:

MANAGER_DOMAIN_ID=4e484e5b-0498-4453-adf8-a6fa6a37454b
MANAGER_CHANNEL_ID=c1578c6c-598e-4dbe-9a8b-5d2acd623679
PROPLET_CLIENT_ID=bf7e3581-4dac-472c-bcd9-a83a6ca53197
PROPLET_2_CLIENT_ID=cf3738c1-07db-41c7-8fc3-234202cf3642
PROPLET_3_CLIENT_ID=095d2d54-4017-4da1-b589-0809d64aa517

Step 5: Recreate Services with New Credentials

After provisioning, you must recreate (not just restart) the services to pick up the new credentials from docker/.env:

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env up -d \
  --force-recreate manager coordinator-http proplet proplet-2 proplet-3 proxy local-data-store

Your output should look like this:

[+] Running 13/13
 ✔ Container supermq-clients-db      Running                              0.0s
 ✔ Container supermq-users-db        Running                              0.0s
 ✔ Container supermq-nats            Running                              0.0s
 ...
 ✔ Container propeller-manager       Started                              2.3s
 ✔ Container propeller-proplet       Started                              2.1s
 ✔ Container propeller-proplet-2     Started                              2.2s
 ✔ Container propeller-proplet-3     Started                              2.1s
 ✔ Container fl-demo-coordinator     Started                              2.0s
 ✔ Container propeller-proxy         Started                              1.8s
 ✔ Container fl-demo-local-data-store Started                             1.5s

Verify Services Are Healthy After Recreate

Check health endpoints:

curl -s http://localhost:7070/health && echo ""
curl -s http://localhost:8086/health && echo ""

Your output should look like this:

OK
{"status":"ok"}

Check manager logs for successful MQTT connection:

docker logs propeller-manager 2>&1 | grep -E "MQTT|connected|listening" | tail -5

Your output should look like this:

{"time":"...","level":"INFO","msg":"Subscribed to MQTT topics"}
{"time":"...","level":"INFO","msg":"HTTP FL Coordinator enabled","url":"http://coordinator-http:8080"}
{"time":"...","level":"INFO","msg":"Manager service started listening on port 7070"}

Step 6: Verify Initial Model

Check that the model registry has the initial model (version 0):

curl -s http://localhost:8084/models/0 | jq .

Your output should look like this:

{
  "b": 0,
  "version": 0,
  "w": [0, 0, 0]
}

This is the untrained model with zero weights that will be used as the starting point for federated learning.

If no model exists (404 response), create it manually:

curl -X POST http://localhost:8084/models \
  -H "Content-Type: application/json" \
  -d '{
    "version": 0,
    "model": {
      "w": [0.0, 0.0, 0.0],
      "b": 0.0,
      "version": 0
    }
  }'

Step 7: Verify Training Data

The Local Data Store automatically seeds datasets for configured proplet UUIDs on startup. Each proplet gets 64 training samples.

Check Dataset for First Proplet

PROPLET_CLIENT_ID=$(grep '^PROPLET_CLIENT_ID=' docker/.env | grep -v '=""' | tail -1 | cut -d '=' -f2 | tr -d '"')
curl -s "http://localhost:8083/datasets/$PROPLET_CLIENT_ID" | jq '{schema: .schema, proplet_id: .proplet_id, size: .size}'

Your output should look like this:

{
  "schema": "fl-demo-dataset-v1",
  "proplet_id": "bf7e3581-4dac-472c-bcd9-a83a6ca53197",
  "size": 64
}

Inspect Sample Training Data

curl -s "http://localhost:8083/datasets/$PROPLET_CLIENT_ID" | jq '.data[0:3]'

Your output should look like this:

[
  {"x": [0.1, 0.2, 0.3], "y": 1},
  {"x": [0.2, 0.4, 0.6], "y": 0},
  {"x": [0.3, 0.6, 0.9], "y": 1}
]

The dataset consists of feature vectors (x) with 3 dimensions and binary labels (y).

Step 8: Build WASM Client

Build the federated learning client to WebAssembly:

cd examples/fl-demo/client-wasm
GOTOOLCHAIN=go1.25.5 GOOS=wasip2 GOARCH=wasm go build -o fl-client.wasm fl-client.go
cd ../../..

Your output should look like this:

# No output means success

Verify the WASM file was created:

ls -lh examples/fl-demo/client-wasm/fl-client.wasm

Your output should look like this:

-rwxr-xr-x 1 user user 3.1M Mar  1 11:20 examples/fl-demo/client-wasm/fl-client.wasm

The WASM binary should be approximately 3.1MB.

Step 9: Push WASM to GitHub Container Registry (GHCR)

The Proxy service fetches WASM binaries from OCI-compliant container registries and delivers them to proplets. In this example, we use GitHub Container Registry (GHCR) which provides native HTTPS support.

While a local registry is simpler to set up, the proxy service connects via HTTPS. Local registries typically use HTTP, causing connection failures:

failed to fetch container for task: error sending request

Using GHCR avoids this issue. For local registry setup with HTTPS, see the Proxy documentation.

Use your GitHub username and Personal Access Token (PAT):

echo "YOUR_GITHUB_PAT" | docker login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin

Your output should look like this:

Login Succeeded

Push WASM Using ORAS

ORAS (OCI Registry As Storage) pushes arbitrary artifacts to OCI registries:

docker run --rm \
  -v "$(pwd)/examples/fl-demo/client-wasm:/workspace" \
  -w /workspace \
  -v "$HOME/.docker/config.json:/root/.docker/config.json:ro" \
  ghcr.io/oras-project/oras:v1.3.0 \
  push ghcr.io/YOUR_GITHUB_USERNAME/fl-client-wasm:latest \
  fl-client.wasm:application/wasm

Your output should look like this:

✓ Uploaded  fl-client.wasm                                           3.1/3.1 MB 100.00%  6s
  └─ sha256:c9113e117b8a4b839df66ecd076f80b77a55ac5e0ceb18185e406b070b0322e0
✓ Uploaded  application/vnd.oci.empty.v1+json                               2/2  B 100.00%  0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
Pushed [registry] ghcr.io/YOUR_GITHUB_USERNAME/fl-client-wasm:latest
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:c9113e117b8a4b839df66ecd076f80b77a55ac5e0ceb18185e406b070b0322e0

By default, GHCR packages are private. To make it public:

Go to https://github.com/users/YOUR_USERNAME/packages/container/fl-client-wasm/settings
Change visibility to "Public"

Step 10: Configure Proxy for GHCR Authentication

Update docker/.env with GHCR credentials so the Proxy can fetch the WASM binary. For full configuration options, see the Proxy documentation.

Edit Environment Variables

Open docker/.env and update these variables:

# GHCR Authentication
PROXY_AUTHENTICATE=true
PROXY_REGISTRY_URL="ghcr.io"
PROXY_REGISTRY_USERNAME="YOUR_GITHUB_USERNAME"
PROXY_REGISTRY_PASSWORD="YOUR_GITHUB_PAT"

Or use sed to update automatically:

sed -i 's|PROXY_AUTHENTICATE=false|PROXY_AUTHENTICATE=true|' docker/.env
sed -i 's|PROXY_REGISTRY_URL=".*"|PROXY_REGISTRY_URL="ghcr.io"|' docker/.env
sed -i 's|PROXY_REGISTRY_USERNAME=".*"|PROXY_REGISTRY_USERNAME="YOUR_GITHUB_USERNAME"|' docker/.env
sed -i 's|PROXY_REGISTRY_PASSWORD=".*"|PROXY_REGISTRY_PASSWORD="YOUR_GITHUB_PAT"|' docker/.env

Verify the Settings

grep -E "PROXY_(AUTHENTICATE|REGISTRY)" docker/.env

Your output should look like this:

PROXY_AUTHENTICATE=true
PROXY_REGISTRY_TOKEN=""
PROXY_REGISTRY_USERNAME="YOUR_GITHUB_USERNAME"
PROXY_REGISTRY_PASSWORD="ghp_xxxxx..."
PROXY_REGISTRY_URL="ghcr.io"

Recreate Proxy with New Credentials

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env up -d --force-recreate proxy

Your output should look like this:

[+] Running 13/13
 ✔ Container supermq-clients-db      Running                              0.0s
 ...
 ✔ Container propeller-proxy         Started                              1.5s

Step 11: Trigger a Federated Learning Round

Now you're ready to run a federated learning experiment!

Export Proplet Client IDs

First, set environment variables for the proplet UUIDs:

export PROPLET_CLIENT_ID=$(grep '^PROPLET_CLIENT_ID=' docker/.env | cut -d '=' -f2)
export PROPLET_2_CLIENT_ID=$(grep '^PROPLET_2_CLIENT_ID=' docker/.env | cut -d '=' -f2)
export PROPLET_3_CLIENT_ID=$(grep '^PROPLET_3_CLIENT_ID=' docker/.env | cut -d '=' -f2)

Verify they're set correctly:

echo "PROPLET_CLIENT_ID=$PROPLET_CLIENT_ID"
echo "PROPLET_2_CLIENT_ID=$PROPLET_2_CLIENT_ID"
echo "PROPLET_3_CLIENT_ID=$PROPLET_3_CLIENT_ID"

Your output should look like this:

PROPLET_CLIENT_ID=bf7e3581-4dac-472c-bcd9-a83a6ca53197
PROPLET_2_CLIENT_ID=cf3738c1-07db-41c7-8fc3-234202cf3642
PROPLET_3_CLIENT_ID=095d2d54-4017-4da1-b589-0809d64aa517

Submit FL Experiment to Manager

Configure and start the FL experiment via the manager's /fl/experiments endpoint:

curl -s -X POST http://localhost:7070/fl/experiments \
  -H "Content-Type: application/json" \
  -d "{
    \"experiment_id\": \"exp-round-$(date +%s)\",
    \"round_id\": \"r-$(date +%s)\",
    \"model_ref\": \"fl/models/global_model_v0\",
    \"participants\": [\"$PROPLET_CLIENT_ID\", \"$PROPLET_2_CLIENT_ID\", \"$PROPLET_3_CLIENT_ID\"],
    \"hyperparams\": {\"epochs\": 1, \"lr\": 0.01, \"batch_size\": 16},
    \"k_of_n\": 2,
    \"timeout_s\": 120,
    \"task_wasm_image\": \"ghcr.io/YOUR_GITHUB_USERNAME/fl-client-wasm:latest\"
  }" | jq .

Your output should look like this:

{
  "experiment_id": "exp-round-1709309984",
  "round_id": "r-1709309984",
  "status": "configured"
}

Step 12: Monitor the Training Round

Once the experiment is triggered, several things happen automatically:

Manager forwards configuration to coordinator and dispatches tasks to proplets
Proxy fetches WASM from GHCR and sends chunks to proplets via MQTT
Proplets receive WASM, fetch model and dataset, run local training
Coordinator collects weight updates from proplets
Aggregator performs FedAvg aggregation
Model Registry stores the new model version

Check Manager Logs for Task Dispatch

docker logs propeller-manager 2>&1 | grep "launched task" | tail -5

Your output should look like this:

{"time":"...","level":"INFO","msg":"launched task for FL round participant","round_id":"r-1709309984","proplet_id":"bf7e3581-4dac-472c-bcd9-a83a6ca53197","task_id":"4f5916b1-3b1e-4b27-9c78-8a70e02c3c28"}
{"time":"...","level":"INFO","msg":"launched task for FL round participant","round_id":"r-1709309984","proplet_id":"cf3738c1-07db-41c7-8fc3-234202cf3642","task_id":"d5aad2f8-a2ed-4cbe-8c7d-881a10460986"}
{"time":"...","level":"INFO","msg":"launched task for FL round participant","round_id":"r-1709309984","proplet_id":"095d2d54-4017-4da1-b589-0809d64aa517","task_id":"e9f67ce1-ea53-45b6-8d89-eff69974b058"}

Check Proxy Logs for WASM Fetching

docker logs propeller-proxy 2>&1 | tail -15

Your output should look like this:

{"time":"...","level":"INFO","msg":"Chunk 0 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 1 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 2 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 3 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 4 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 5 size: 512000 bytes"}
{"time":"...","level":"INFO","msg":"Chunk 6 size: 167578 bytes"}
{"time":"...","level":"INFO","msg":"successfully fetched container, sending chunks","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","total_chunks":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":0,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":1,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":2,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":3,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":4,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":5,"total":7}
{"time":"...","level":"DEBUG","msg":"sent container chunk to MQTT stream","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","chunk":6,"total":7}
{"time":"...","level":"INFO","msg":"successfully sent all chunks","container":"ghcr.io/YOUR_USERNAME/fl-client-wasm:latest","total_chunks":7}

Check Proplet Logs for Training Execution

docker logs propeller-proplet 2>&1 | grep -E "Fetching|Successfully|Process|completed" | tail -10

Your output should look like this:

... INFO Fetching model from registry: http://model-registry:8081/models/0
... INFO Successfully fetched model v0 and passed to client
... INFO Fetching dataset from Local Data Store: http://local-data-store:8083/datasets/bf7e3581-4dac-472c-bcd9-a83a6ca53197
... INFO Successfully fetched dataset with 64 samples and passed to client
... INFO Starting Host runtime app: task_id=4f5916b1-..., function=fl-round-r-..., daemon=false, wasm_size=3239578
... INFO Process spawned with PID: Some(20)
... INFO Process completed for task: 4f5916b1-3b1e-4b27-9c78-8a70e02c3c28
... INFO Task 4f5916b1-... completed successfully. Result: {"base_model_uri":"fl/models/global_model_v0","metrics":{"loss":0.9335471973452772},"num_samples":64,"proplet_id":"bf7e3581-...","round_id":"r-...","update":{"b":-0.003...,"version":0,"w":[0.012...,-0.003...,0.015...]}}
... INFO Successfully posted FL update to coordinator via HTTP: http://coordinator-http:8080/update

Check Coordinator Logs for Aggregation

docker logs fl-demo-coordinator 2>&1 | tail -10

Your output should look like this:

... INFO Received experiment configuration experiment_id=exp-round-... round_id=r-... model_ref=fl/models/global_model_v0 k_of_n=2
... INFO Loaded initial model from registry version=0
... INFO Experiment configured and round initialized round_id=r-... model_version=0
... INFO Received update round_id=r-... proplet_id=095d2d54-... total_updates=1 k_of_n=2
... INFO Received update round_id=r-... proplet_id=cf3738c1-... total_updates=2 k_of_n=2
... INFO Round complete: received k_of_n updates round_id=r-... updates=2
... INFO Calling aggregator service round_id=r-... num_updates=2
... INFO Aggregated model stored round_id=r-... version=1
... INFO Published round completion notification to MQTT topic=fl/rounds/next round_id=r-... new_model_version=1

Key milestones to look for:

launched task for FL round participant - Manager dispatched tasks
successfully fetched container - Proxy downloaded WASM from GHCR
Successfully fetched model - Proplet loaded initial weights
Successfully fetched dataset - Proplet loaded local training data
Task ... completed successfully - Local training finished
Round complete - Coordinator received k_of_n updates
Aggregated model stored - FedAvg completed, new model saved

Step 13: Verify the Trained Model

After aggregation completes, check the updated model in the registry:

List All Model Versions

curl -s http://localhost:8084/models | jq .

Your output should look like this:

{
  "versions": [0, 1]
}

Fetch the New Aggregated Model (Version 1)

curl -s http://localhost:8084/models/1 | jq .

Your output should look like this:

{
  "b": -0.00026427958266281217,
  "w": [
    0.01638395015724097,
    0.00034361294245900534,
    0.014378351819544267
  ]
}

Compare Initial vs Trained Model

Model	Weights (w)	Bias (b)
Initial (v0)	[0, 0, 0]	0
Trained (v1)	[0.016, 0.0003, 0.014]	-0.00026

The non-zero weights and bias confirm that:

Proplets successfully trained on their local datasets (64 samples each)
Weight updates were aggregated using FedAvg
The global model was updated without raw data leaving the devices

Step 14: Run Additional Training Rounds

To continue training and improve the model, run additional rounds using the new model version:

curl -s -X POST http://localhost:7070/fl/experiments \
  -H "Content-Type: application/json" \
  -d "{
    \"experiment_id\": \"exp-round2-$(date +%s)\",
    \"round_id\": \"r2-$(date +%s)\",
    \"model_ref\": \"fl/models/global_model_v1\",
    \"participants\": [\"$PROPLET_CLIENT_ID\", \"$PROPLET_2_CLIENT_ID\", \"$PROPLET_3_CLIENT_ID\"],
    \"hyperparams\": {\"epochs\": 1, \"lr\": 0.01, \"batch_size\": 16},
    \"k_of_n\": 2,
    \"timeout_s\": 120,
    \"task_wasm_image\": \"ghcr.io/YOUR_GITHUB_USERNAME/fl-client-wasm:latest\"
  }" | jq .

Note the change: model_ref now points to global_model_v1 (the previously trained model).

After completion, check version 2:

curl -s http://localhost:8084/models/2 | jq .

Each round produces a new model version that builds on the previous one's weights.

Cleanup

To stop and remove all containers and volumes:

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env down -v

To only stop containers (preserve data):

docker compose -f docker/compose.yaml -f examples/fl-demo/compose.yaml --env-file docker/.env down

Federated Learning

On this page