feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <[email protected]>
Follow-up hardening for the audit control plane. The ABBA deadlock
(backlog#961), dispatch failure propagation (backlog#962), and credential
header redaction (backlog#963) already landed on main; this change addresses
the remaining audit-side findings.
- backlog#970: reload/commit now shuts down existing replay workers and closes
old targets *before* activating the replacement set, so old and new workers
never drain the same store concurrently and re-deliver entries. Extracted a
state-neutral `shutdown_runtime_targets` helper (registry-then-cancellers
lock order preserved).
- backlog#978: `start()` claims the `Starting` transition atomically under the
state lock, closing the check-then-act race that could double-activate;
`dispatch()` no longer returns Ok while paused, it surfaces an explicit
`AuditError::Paused` so the audit trail is not silently corrupted.
- backlog#984 (audit part): `dispatch_audit_log` performs a single state read
and interprets not-running/paused as a deliberate skip while surfacing real
delivery failures; `dispatch_batch` records the same observability signals as
single dispatch; documented the unordered cross-target fan-out.
Adds unit tests: paused dispatch returns Err, concurrent start does not hang or
double-activate, commit closes old targets before installing new, and
dispatch/dispatch_batch delivery-outcome coverage (all-fail -> Err, partial ->
Ok, all-success -> Ok).
Relates to rustfs/backlog#970
Relates to rustfs/backlog#978
Relates to rustfs/backlog#984
Co-authored-by: heihutu <[email protected]>
* fix(audit): resolve AB-BA lock-order deadlock between registry and stream_cancellers (backlog#961)
`AuditSystem::runtime_status_snapshot` acquired `stream_cancellers.read()`
before `registry.lock()`, while every other path that holds both locks
(`clear_runtime_targets`, `AuditRuntimeFacade::replace_targets`) acquires
`registry` first. Under concurrency (e.g. admin status polling racing a
config reload) this AB-BA ordering could deadlock the whole audit control
plane with no self-recovery.
Reorder `runtime_status_snapshot` to acquire `registry` before
`stream_cancellers`, unifying the global lock order to
`registry -> stream_cancellers` across all double-lock paths, and add a
multi-threaded regression test that hammers both critical sections with a
timeout guard to fail fast if the ordering regresses.
Refs: https://github.com/rustfs/backlog/issues/961
* docs(audit): reword AB-BA to ABBA to satisfy typos check (backlog#961)
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.
- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets
Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
* improve code for audit
* improve code ecfs.rs
* improve code
* improve code for ecfs.rs
* feat(storage): refactor audit and notification with OperationHelper
This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.
The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.
Key improvements include:
- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.
This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* improve code for audit and notify
* fix
* fix
* fix
