Atomic uploads: why your “rename pattern” matters

Your SFTP client opens a file, writes bytes, closes the channel. On the server, the file appears. Simple.

Except it isn’t. Between the first byte written and the channel close, the file exists on disk in a half-written state. Any process watching that directory — a virus scanner, an ETL pipeline, a downstream polling job — can pick it up mid-write. It reads a truncated CSV, a partial XML document, half of a ZIP archive. The downstream job fails, or worse, it succeeds on bad data and nobody notices until the numbers are wrong.

This is the single most common file-transfer integration bug in the wild, and it has nothing to do with the network. It’s a filesystem visibility problem: the target file is visible before it’s complete.

Here is how we solve it, and why the three configuration knobs that control the solution — the staging directory, the duplicate-handling policy, and the rename pattern — matter more than most people think.

The problem: write-in-place

The default behavior of most SFTP servers is write-in-place. The server opens the target file path, writes bytes as the SSH channel delivers them, and closes the file when the client sends SSH_FXP_CLOSE. The operating system flushes buffers, the file handle goes away, and the transfer is done.

During the entire transfer, the target file is visible in directory listings. Its size grows with each write. Its modification time updates. Any process that stats, reads, or watches the directory can see it. If the transfer takes thirty seconds and a polling job runs every ten seconds, that job will see the file at least twice before it’s finished — and it has no reliable way to know whether the file is complete.

Common workarounds include:

• Polling with a delay. Wait N seconds after the modification time stops changing. This works until it doesn’t (network stalls, large files, slow clients).
• Sentinel files. The sender uploads data.csv first, then uploads data.csv.done as a signal. This requires the sender to cooperate, doubles the number of transfers, and fails if the sender crashes between the two uploads.
• File locking. The SFTP server holds a write lock; the consumer waits for the lock to release. This works for a single consumer but falls apart with multiple readers or non-locking consumers.

All three are fragile, require coordination between sender and receiver, and fail under edge cases that are hard to test for. The real fix is to make the file invisible until it’s complete.

The fix: staging + atomic rename

Stylus SFTP Server never writes to the target file. Every upload goes through a staging lifecycle:

1. Stage. When the SFTP client opens a file for writing, the server creates a temporary file in a centralized .staging/ directory. The staging file’s name includes a UUID to guarantee uniqueness: report.csv.a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp.
2. Write. All bytes go to the staging file. The target directory is untouched. If the client disconnects, the staging file is deleted — no orphan in the user’s home directory.
3. Commit. When the client sends SSH_FXP_CLOSE and the channel closes cleanly, the server calls java.nio.file.Files.move() with StandardCopyOption.ATOMIC_MOVE. The staging file becomes the target file in a single filesystem operation. There is no window where the target exists but is incomplete.

This is what the actual commit code looks like inside the server:

try {
    Files.move(from, to,
            StandardCopyOption.ATOMIC_MOVE,
            StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
    log.warn("ATOMIC_MOVE failed (cross-volume?), falling back: {} → {}", from, to);
    Files.move(from, to, StandardCopyOption.REPLACE_EXISTING);
}

The fallback matters: ATOMIC_MOVE requires both paths to be on the same filesystem volume. If the staging directory and the home directory are on different volumes (a misconfiguration, or a deliberate choice for performance isolation), the rename cannot be atomic. The server logs a warning and falls back to a non-atomic move. The configuration section of the User’s Guide explains how to keep both on the same volume.

The staging directory is centralized

In earlier versions of the server, staging files lived alongside the target file in the user’s home directory. This had two problems: the .tmp files were visible in SFTP directory listings, and a careless downstream consumer might try to read them.

The current design uses a centralized staging directory under the server’s data folder:

${STYLUS_SFTPSERVER_DATA}/.staging/F/homes/john/reports/data.csv.UUID.tmp

The staging path mirrors the full physical path of the target (including the drive letter on Windows). This guarantees uniqueness across users and across mounts without requiring any locking coordination. And because the staging directory is outside the user’s home tree, staging files are completely invisible to SFTP clients, WebDAV clients, the Web Portal, and FTP — every protocol.

On server boot, the entire .staging/ directory is wiped. Everything in it is transient by definition: if the server is starting, no transfers are in progress. Any file that survived in staging is an orphan from a previous crash, and it’s safe to delete.

What happens when the filename already exists

The atomic rename solves the partial-file problem. But what about the case where the target file already exists? A user uploads report.csv today, then uploads report.csv again tomorrow. What should happen?

This is a policy decision that varies by use case. Stylus SFTP Server exposes it as a configuration knob called <on-duplicate>, with three options:

timestamp-suffix (the default)

The existing file is backed up with a timestamp suffix — report_20260707093022.csv — and the new upload takes the canonical name. The new file is always report.csv; the old one moves out of the way. This is the safest default: no data is lost, the latest upload is always findable by its original name, and downstream consumers that poll for report.csv always get the newest version.

The timestamp format is yyyyMMddHHmmss, appended before the extension. In the rare case that two uploads arrive within the same second (same filename, same user, same directory), the server falls back to a nanosecond-resolution suffix to guarantee uniqueness.

overwrite

The new file replaces the existing one. This is standard SFTP behavior — the protocol itself has no built-in “file already exists” guard. Use this when the downstream consumer doesn’t care about history and only wants the latest version.

reject

The server refuses the upload with an SFTP error status. The staging file is cleaned up; the existing file is untouched. Use this when duplicates indicate a sender bug and should be caught immediately, or when regulatory requirements demand that files are never silently overwritten.

The entire dedup decision is synchronized on a global lock to prevent a TOCTOU (time-of-check-to-time-of-use) race. The existence check and the move happen inside the same critical section:

synchronized (RENAME_LOCK) {
    boolean exists = Files.exists(to);
    if (exists) {
        switch (action) {
            case REJECT:
                throw new FileAlreadyExistsException(to.toString());
            case TIMESTAMP_SUFFIX:
                Path backup = dedupPath(to);
                move(to, backup);
                break;
            case OVERWRITE:
                break; // move below replaces in place
        }
    }
    move(from, to);
}

Without the lock, two concurrent uploads targeting the same filename could both check, both see that the file doesn’t exist, and both attempt the rename — one would silently overwrite the other. The lock makes the check-then-move sequence atomic at the application level, on top of the filesystem-level atomicity of the move itself.

The rename pattern

The third configuration knob is <rename-pattern>. This controls the final filename after the atomic move. The default is ${basename} — preserve the original filename exactly as the client sent it.

<upload>
    <rename-pattern>${basename}</rename-pattern>
    <on-duplicate>timestamp-suffix</on-duplicate>
    <orphan-max-age>PT2H</orphan-max-age>
</upload>

The available macros are:

• ${basename} — the original filename including extension (report.csv)
• ${filename} — the filename without extension (report)
• ${ext} — the extension with its leading dot (.csv), or empty string if there is none
• ${username} — the SFTP account name that uploaded the file

This lets you enforce naming conventions at the server level, independent of what the client sends. A pattern like ${username}_${basename} prefixes every uploaded file with the sender’s account name — useful when multiple trading partners deposit files into a shared inbound directory and downstream processing needs to know who sent what without parsing metadata.

The rename pattern, the duplicate policy, and the atomic staging work together. The pattern determines the target filename; the staging lifecycle guarantees the file appears atomically; the duplicate policy handles collisions on the target name. Three knobs, one consistent outcome: the file is either fully present under a deterministic name, or it isn’t there at all.

Why downstream consumers care

If you’re the person who writes the downstream job — the ETL script, the invoice parser, the compliance archiver — atomic uploads change the contract you program against.

Without them, you need defensive code: check file size stability, wait for modification-time quiescence, handle truncated reads, retry on parse failure. You build a state machine around “is this file probably done?” and accept that the answer is sometimes wrong.

With atomic staging, the contract is simpler: if the file exists, it is complete. Your downstream job can read it the instant it appears. No polling delay, no sentinel file, no size-stability check. The filesystem itself is the signal.

This is especially valuable in regulated environments where audit trails matter. The server’s audit system logs the exact moment the atomic rename succeeded, the final path, and the file size. That audit record is the definitive timestamp for “when did this file arrive?” — not the first byte, not the last write, but the atomic commit. It’s a clean, unambiguous event that maps directly to compliance requirements around receipt timestamps.

Configuration

The upload lifecycle is configured in filesystem-config.xml under the <upload> block. All three settings are hot-reloadable — you can change the duplicate policy or the rename pattern without restarting the server. The settings are also exposed through the CLI, the Swing admin GUI, and the Web Admin REST API.

# CLI
sss set-filesystem-config --rename-pattern "\${basename}" \
                          --on-duplicate timestamp-suffix \
                          --orphan-max-age PT2H

The <orphan-max-age> setting controls the startup cleanup threshold. On boot, the server wipes the entire .staging/ directory unconditionally — everything in it is transient. The orphan-max-age applies to the legacy cleanup of .tmp files found inside home directories from older server versions. ISO-8601 duration format: PT2H is two hours, P1D is one day.

The take-home

Most file-transfer integration failures are not network problems. They’re visibility problems — a downstream consumer seeing a file before it’s finished. The fix is not smarter polling or fancier locking. The fix is to make incomplete files invisible.

Staging to a centralized directory, committing with ATOMIC_MOVE, and resolving collisions with a deterministic policy gives downstream consumers a simple guarantee: if the file is there, it’s done. Build your pipeline against that contract and the entire class of partial-file bugs disappears.