Features

Proplet

A Proplet resource represents one worker node. The spec.type field selects between two fundamentally different modes of operation.

Types

k8s — The operator creates and manages a Kubernetes Deployment in the same cluster. The pod runs the proplet image, which connects back to the same SuperMQ broker using the credentials in spec.connectionConfig. The operator reconciles the Deployment whenever the spec changes and reflects the Deployment's readiness in the Proplet status.

external — The proplet is a device or process running outside the cluster (a Raspberry Pi, an ESP32, a Docker container on another host). The operator does not create any Kubernetes resources for it. Instead, it watches the MQTT control/proplet/alive topic and transitions the Proplet phase to Running when heartbeats arrive within the configured threshold, and to Offline when they stop.

Proplet Lifecycle

Figure: Proplet state machine showing transitions between Initializing, Running, and Offline phases. A proplet starts in Initializing, moves to Running when heartbeats are received (external) or when the Deployment is ready (k8s), and transitions to Offline when heartbeats stop. When heartbeats resume, the proplet returns to Running.

• Initializing — default phase on creation; no heartbeats received yet, or k8s Deployment not yet ready.
• Running — heartbeat received within --last-seen-threshold (default 30 s) for external proplets, or all Deployment replicas ready for k8s proplets.
• Offline — heartbeat not received within threshold (external only).

Proplet Conditions

Proplet Spec Reference

Proplet Status Reference

Task

A Task resource defines a single WASM workload. The TaskReconciler drives it through a phase machine, selecting a proplet, dispatching execution, and capturing the result.

Task Lifecycle

Figure: Task state machine showing all valid transitions. Tasks begin in pending and move directly to running when dispatched to a proplet. From running, tasks can complete successfully (completed), fail (failed), or be stopped (interrupted). The operator enforces valid transitions and supports retry/restart flows from terminal states back to pending.

Valid state transitions are enforced by the operator. Task completion and failure arrive via control/proplet/results MQTT messages; the operator validates each proposed transition before applying it.

Execution Paths

Figure: Comparison of task execution paths for external vs k8s proplets. External proplets receive tasks via MQTT and return results through the broker. K8s proplets run as Kubernetes Jobs with ConfigMaps for task configuration and multiple result extraction methods (Job annotations, ConfigMaps, Secrets, or Pod annotations).

External proplet path:

1. TaskReconciler picks up the new Task and resolves the target proplet by spec.propletSelector.propletId.
2. The reconciler looks up the Proplet CR and confirms its type is external.
3. It publishes an MQTT message to control/manager/start containing the full task spec: function name, WASM file or image URL, inputs, environment variables, scheduling parameters, monitoring profile, and the assigned proplet ID.
4. The proplet receives the message, loads the WASM module with its runtime (e.g. Wasmtime), and executes the named function.
5. The proplet publishes the result to control/proplet/results.
6. The TaskReconciler's MQTT result handler receives the message, stores the output in status.results, and sets status.phase = completed.

k8s proplet path:

1. TaskReconciler resolves the proplet and confirms its type is k8s.
2. It creates a ConfigMap containing task environment variables (function name, inputs, monitoring settings, etc.) and an optional wasm_file_provided key if a WASM file is embedded.
3. It creates a Kubernetes Job whose pod runs the image specified in spec.imageUrl.
4. The reconciler is notified of Job changes via Kubernetes owner-reference watches (the Owns(&batchv1.Job{}) watch triggers reconcile on every Job status update). A safety requeue at 5 minutes handles edge cases where the watch event is missed. On Job completion it attempts multi-method result extraction: Job annotations, container terminated message, a result ConfigMap ({jobName}-result), a result Secret ({jobName}-result), or Pod annotations—whichever is populated first.
5. On success it sets status.phase = completed and stores the extracted result in status.results.

Proplet Selection

The spec.propletSelector field controls which proplet receives the task. The scheduler evaluates candidates in two passes:

1. Candidate filter — retains only proplets in Running phase that satisfy all selector criteria:

   • propletId: exact name match
   • matchLabels: all specified labels must be present on the Proplet CR
   • matchDeviceTypes: Proplet's external.deviceType must be in the list
   • matchCapabilities: all listed capabilities must appear in the Proplet's external.capabilities
   • preferredPropletType: filters by k8s or external if set; any accepts both

2. Scoring and selection — the default scheduler is round-robin: the proplet after the last-selected one scores 0.1; all others score 1.0. The proplet with the lowest score is selected.

If no proplet satisfies the filter, the task remains in pending and is requeued.

Task Spec Reference

Task Status Reference

PropellerJob

A PropellerJob groups multiple Tasks into a single managed batch. The PropellerJobReconciler creates a child Task CR for each entry in spec.tasks, sets the jobId field on each to the PropellerJob's UID, and then monitors those tasks to completion. The short name for this resource is pjob (use kubectl get pjob as a shortcut).

PropellerJob Lifecycle

Figure: PropellerJob state machine. Jobs start in Pending, transition to Running when child Tasks are created, and finally move to Completed when all tasks finish successfully or to Failed if any task fails.

PropellerJob Spec Reference

PropellerJob Status Reference

FederatedJob

A FederatedJob coordinates a multi-round federated learning experiment across a set of proplets. The FederatedJobReconciler creates TrainingRound CRs sequentially, one per round, advancing to the next round only when the current one completes aggregation.

FederatedJob Lifecycle

Figure: FederatedJob state machine for multi-round federated learning experiments. Jobs begin in Pending, move to Running as TrainingRounds are created and executed sequentially, and complete when all rounds finish aggregation. Jobs fail if any round times out or encounters an aggregation error.

FederatedJob Spec Reference

FederatedJob Status Reference

TrainingRound

A TrainingRound represents a single round within a FederatedJob. The TrainingRoundReconciler creates one Task per participant, collects their model updates via MQTT result messages, and aggregates them once the k-of-n threshold is reached.

TrainingRound Lifecycle

Figure: TrainingRound state machine showing the four-phase lifecycle. Rounds start in Pending, move to Running when participant Tasks are created, transition to Aggregating once k-of-n updates are received, and finally reach Completed after successful model aggregation. Rounds fail on timeout or aggregation errors.

TrainingRound Spec Reference

TrainingRound Status Reference

Scheduler

The scheduler runs inside the TaskReconciler to select a proplet for each task. It implements a three-phase algorithm:

1. Filter (SelectCandidateProplets) — retains only proplets that are in Running phase and satisfy all criteria in spec.propletSelector.
2. Score (Score) — assigns a numeric score to each candidate (lower = preferred).
3. Pick (Pick) — selects the candidate with the lowest score; ties are broken deterministically.

Figure: Three-phase scheduler algorithm. First, the Filter phase removes proplets that don't meet selector criteria or aren't in Running phase. Then, the Score phase assigns numeric scores to candidates (the default round-robin scorer assigns 0.1 to the next-in-line proplet and 1.0 to others). Finally, the Pick phase selects the lowest-scoring candidate for task assignment.

The default implementation is round-robin: the proplet immediately after the last-selected one receives a score of 0.1; all others receive 1.0. This distributes tasks evenly across available proplets over time.

Errors returned by the scheduler:

Owner References and Garbage Collection

The operator sets Kubernetes owner references on all resources it creates:

When a parent resource is deleted, Kubernetes garbage-collects all owned children automatically. For Proplet and Task resources, the operator also adds a finalizer to ensure it can clean up resources before the object is removed from the API server.