Queue broadcast messages for background sending - #1054

Ideally your PR would include some unit tests - e.g. making sure the various static methods in BroadcastQueue work. on the different database backends.

Thanks, agreed. I'll add focused unit coverage for BroadcastQueue before considering this ready. The tests should cover job creation, busy-domain detection, status transitions, cancellation/reset behavior, and recipient selection, without requiring a real SMTP send. I'll also check the existing test helpers so the SQL paths are validated in the same style as the rest of the project, including the supported database backends where practical.

I added one more commit with a CLI-only broadcast worker mode for controlled testing. The GUI behavior remains unchanged: in normal live mode, queued jobs are processed by the automatic worker and sent normally. For controlled tests, an admin can run php scripts/broadcast-worker.php --mode=dry-run, use the GUI flow normally, and the worker will process queued jobs without SMTP, marking recipients with dry-run. The mode can be restored with php scripts/broadcast-worker.php --mode=live.

I also kept --dry-run as a one-shot worker override for direct CLI processing, but the persistent mode avoids racing the automatic worker when testing the GUI flow.

PostfixAdmin hasn't had any sort of long running (or cron based) background processes before.

I'm curious as to why you need the ability to send (what I assume must be a large volume) of email out; would you be better using some sort of mailing list software if you have that many users?

Thanks, that concern makes sense.

The intended use case is not newsletter-style or high-volume mailing list delivery. For that, I agree that dedicated mailing list software would be the right tool.

This is meant for operational/admin broadcasts to the existing mailbox users of a PostfixAdmin installation, for example maintenance notices, migration notices, outage/security notifications, or other one-off messages related to the hosted mail service.

The reason for queueing is mainly to avoid the current synchronous web request doing all deliveries inline. Even for a moderate number of recipients, that can hit browser/PHP timeouts and gives the admin no useful status, cancellation, or failure reporting.

Also, the worker is not intended to be a permanent daemon or cron requirement. It is a short-lived process started after a job is queued, guarded by a lock so only one worker runs, and it exits once there is no more work. So the PR avoids introducing a continuously running background service.

That said, if this still feels outside PostfixAdmin’s scope, I’m open to adjusting the approach. For example, we could make this explicitly an admin/operational broadcast feature, document that mailing lists are recommended for newsletter/high-volume use, or rework the worker model if there is a preferred project direction.

Thanks, agreed on both points.

BroadcastQueue is already covered by Composer's model classmap through common.php, so I removed the explicit require_once.

For the Linux worker start command, I also redirected stdin from /dev/null so the short-lived background worker does not keep the web request/stdin attached:

php ... --limit=N > /dev/null 2>&1 < /dev/null &

I kept this as the minimal change for now rather than introducing nohup or a daemon/cron requirement. The worker still starts, processes queued work, and exits.

The queue/worker split also leaves room for a future deployment mode where admins could run the worker from CLI/cron/systemd instead of auto-starting it from the web request, if that direction fits the project better.

If this is acceptable for master, would you also consider the same corrective broadcast queue change for the 4.0 branch? I have a 4.0-compatible version prepared, but I can keep it separate unless you think it is useful there too.

Production validation found two issues in the previous head: public/broadcast-message.php used BroadcastQueue without loading the class file, and a partially applied upgrade 1857 could fail on retry with a duplicate idx_broadcast_job_status index while the recorded schema version remained 1856. Commit ddb798a loads the class explicitly and makes index creation restartable on MySQL/MariaDB, PostgreSQL, and SQLite. The regression test covers both the partial state and repeated execution. Recovery was confirmed on the affected production installation: rerunning the upgrade skipped the existing index, completed version 1857, and restored the broadcast page. GitHub CI is green on PHP 8.2-8.5, MySQL, and PostgreSQL.

I'd like to revisit the design question around this PR based on the behaviour of the existing broadcast implementation in real use.

The underlying problem is not really large-volume or mailing-list delivery. The current broadcast feature sends every recipient synchronously inside a single HTTP request, and with a normal number of active mailboxes it can exceed the PHP/webserver request timeout before completing. In practice this makes the existing feature unreliable once the installation is no longer very small.

So the queue in this PR is primarily a way to fix that existing limitation by decoupling delivery from the web request. The short-lived worker was chosen to avoid introducing a permanent daemon or cron dependency, while still allowing the existing broadcast function to complete reliably and report progress/failures.

Would you be willing to re-evaluate the PR from that perspective?

If the current queue/worker implementation is not the preferred architectural fit for PostfixAdmin, I am happy to rework it. The important part is avoiding processing the complete recipient set inside one HTTP request. If you have a simpler model you would prefer for that, I can adapt the implementation around it.