From 9559ceb3f0767ab4e2c3f00dfca55d1166fb11f9 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 7 Jul 2026 12:02:37 +0200 Subject: [PATCH 01/10] fuse: switch to direct IO on an inode-invalidation notify storm A FUSE_NOTIFY_INVAL_INODE data invalidation means another (remote) entity is modifying the file. Rather than react to a single notify, keep a per-inode moving average of how fast data invalidations arrive for the whole file: an EWMA of the inter-arrival interval, updated under fi->lock on every notify (fuse_notify_inval_hot()). The inode is latched only once the average spacing drops below an internal threshold (FUSE_NOTIFY_DIO_INTERVAL) while a local writer is open; a lone or occasional notify keeps the average high and does not trip the switch. The heuristic has no external knob -- its parameters (EWMA weight, threshold, seed) are source-level constants. Introduce the forced-direct-IO latch (FUSE_I_FORCE_DIO): - fuse_reverse_inval_inode() folds each data invalidation into the moving average and sets the latch when it trips with a local writer present; - fuse_file_{read,write}_iter() and fuse_cache_write_iter() route to the direct path while latched; fuse_dio_{wr_exclusive_lock,lock,unlock}() use the shared parallel-dio path and bypass the cached/uncached accounting; - fuse_file_io_open() opens new files uncached so they do not re-enter caching mode; - fuse_prepare_release() clears the latch once the last writer is gone and fuse_file_release() drops any clean folios a racing read repopulated; fuse_file_mmap() reverts to caching mode (a mapping needs the page cache). Latching to direct IO is only coherent if no buffered write can deposit dirty folios into the page cache after it has been dropped. Add a per-inode rw_semaphore, wb_inval_rwsem, to serialise the buffered-write page-cache dirtying against the latch transition. The writeback path holds it for read around the dirtying and re-checks the latch under it; fuse_reverse_inval_inode() holds it for write around its invalidate + latch set. The notification may be delivered by the same server thread that still owes a reply to an in-flight write holding the inode lock, so it takes the rwsem with a trylock and never blocks: if the writer has gone it skips the latch and only invalidates the notified range. The writer's read-side section stays free of server round-trips because under fc->dlm the partial-write RMW read is skipped. Signed-off-by: Horst Birthelmer (cherry picked from commit eeec69bb75561f189ab0b68450dcccc64a2f9173) [ahenderson: 6.8 has no iomap buffered-write path, so the writeback selector and the fuse_writeback_write_iter() context are dropped. generic_file_write_iter() is open-coded in fuse_cache_write_iter() so the coherency gate is taken inside i_rwsem, preserving the i_rwsem -> gate order the later gating commits depend on. fuse_inode_uncached_io_start() takes one argument here; there is no FOPEN_PASSTHROUGH arm in fuse_file_io_open(), fuse_file_mmap() or fuse_file_{read,write}_iter(), and no fuse_inode_backing(), so those clauses are dropped] Signed-off-by: Allison Henderson --- fs/fuse/file.c | 161 ++++++++++++++++++++++++++++++++++++++++++++--- fs/fuse/fuse_i.h | 56 +++++++++++++++++ fs/fuse/inode.c | 98 ++++++++++++++++++++++++++++- fs/fuse/iomode.c | 9 +++ 4 files changed, 313 insertions(+), 11 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 55ab5e9cf61715..fbbcec385ff403 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -414,6 +414,18 @@ static void fuse_prepare_release(struct fuse_inode *fi, struct fuse_file *ff, if (likely(fi)) { spin_lock(&fi->lock); list_del(&ff->write_entry); + /* + * Leave forced direct IO mode once the last writer is gone: with + * no local writer left there is no cached-write contention with + * the remote modifier that triggered the switch. Restore + * FUSE_I_CACHE_IO_MODE for any frozen cached opens. + */ + if (test_bit(FUSE_I_FORCE_DIO, &fi->state) && + list_empty(&fi->write_files)) { + clear_bit(FUSE_I_FORCE_DIO, &fi->state); + if (fi->iocachectr > 0) + set_bit(FUSE_I_CACHE_IO_MODE, &fi->state); + } spin_unlock(&fi->lock); } spin_lock(&fc->lock); @@ -450,9 +462,24 @@ void fuse_file_release(struct inode *inode, struct fuse_file *ff, struct fuse_inode *fi = get_fuse_inode(inode); struct fuse_release_args *ra = ff->release_args; int opcode = isdir ? FUSE_RELEASEDIR : FUSE_RELEASE; + bool was_force_dio = test_bit(FUSE_I_FORCE_DIO, &fi->state); fuse_prepare_release(fi, ff, open_flags, opcode, false); + /* + * If this release dropped the last writer, fuse_prepare_release() + * cleared the forced-direct-IO latch (under fi->lock). Drop any clean + * folios a read racing the latch may have repopulated so they cannot be + * served stale once caching mode resumes. No inode lock or + * wb_inval_rwsem: release may run on the fuse server thread (async fput + * from aio completion), where blocking on a contended inode lock could + * stall the connection. Writes were routed direct while latched, so + * only clean folios exist and this invalidate is server-free; the last + * writer is gone, so no forced-dio writer can race the drop. + */ + if (was_force_dio && !test_bit(FUSE_I_FORCE_DIO, &fi->state)) + invalidate_inode_pages2(inode->i_mapping); + if (ra && ff->flock) { ra->inarg.release_flags |= FUSE_RELEASE_FLOCK_UNLOCK; ra->inarg.lock_owner = fuse_lock_owner_id(ff->fm->fc, id); @@ -1400,9 +1427,15 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); + bool force_dio = test_bit(FUSE_I_FORCE_DIO, &fi->state); - /* Server side has to advise that it supports parallel dio writes. */ - if (!(ff->open_flags & FOPEN_PARALLEL_DIRECT_WRITES)) + /* + * Server side has to advise that it supports parallel dio writes. + * When the inode is latched into forced direct IO, parallel writes are + * used unconditionally: the page cache has been flushed and is bypassed + * for this inode. + */ + if (!force_dio && !(ff->open_flags & FOPEN_PARALLEL_DIRECT_WRITES)) return true; /* @@ -1413,7 +1446,7 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from return true; /* shared locks are not allowed with parallel page cache IO */ - if (test_bit(FUSE_I_CACHE_IO_MODE, &fi->state)) + if (!force_dio && test_bit(FUSE_I_CACHE_IO_MODE, &fi->state)) return true; /* Parallel dio beyond EOF is not supported, at least for now. */ @@ -1440,9 +1473,14 @@ static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, * should be performed only after taking shared inode lock. * Previous past eof check was without inode lock and might * have raced, so check it again. + * + * Under the forced-dio latch the cached/uncached accounting is + * bypassed (the latch guarantees the cache is flushed and not + * repopulated), so only re-check the past-eof condition. */ if (fuse_io_past_eof(iocb, from) || - fuse_inode_uncached_io_start(fi) != 0) { + (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && + fuse_inode_uncached_io_start(fi) != 0)) { inode_unlock_shared(inode); inode_lock(inode); *exclusive = true; @@ -1458,12 +1496,19 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) if (exclusive) { inode_unlock(inode); } else { - /* Allow opens in caching mode after last parallel dio end */ - fuse_inode_uncached_io_end(fi); + /* + * Allow opens in caching mode after last parallel dio end. + * Skipped under the forced-dio latch, which never took an + * uncached_io reference in fuse_dio_lock(). + */ + if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) + fuse_inode_uncached_io_end(fi); inode_unlock_shared(inode); } } +static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from); + static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -1472,6 +1517,18 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) struct inode *inode = mapping->host; ssize_t err; struct fuse_conn *fc = get_fuse_conn(inode); + struct fuse_inode *fi = get_fuse_inode(inode); + bool wb_guard = false; + + /* + * The inode may have been latched into forced direct IO -- by a + * NOTIFY_INVAL_INODE arriving while this inode is open for writing here + * -- after this write was routed to the cached path but before it took + * any lock. Re-route to the direct path (before taking a DLM lock) so + * we do not repopulate the page cache the latch just dropped. + */ + if (fuse_inode_force_dio(inode)) + return fuse_direct_write_iter(iocb, from); if (fc->writeback_cache) { /* Update size (EOF optimization) and mode (SUID clearing) */ @@ -1497,12 +1554,70 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) size_t length = iov_iter_count(from); fuse_get_dlm_write_lock(file, pos, length); } - return generic_file_write_iter(iocb, from); + + /* + * Hold wb_inval_rwsem for read across the page-cache dirtying + * so a concurrent NOTIFY_INVAL_INODE -- which latches the inode + * under the write side of this lock via a non-blocking trylock + * -- cannot strand the folios we are about to write. Re-check + * the latch under it (it may have been set while we blocked on + * the inode lock) and re-route to the direct path if set. + * + * generic_file_write_iter() is open-coded here so the gate can + * be taken inside the inode lock. The order matters: every + * gate holder must take i_rwsem first and the gate second, so + * that a writer holding i_rwsem exclusive can wait out gate + * readers without deadlocking. + */ + wb_guard = fc->dlm; + + inode_lock(inode); + if (wb_guard) { + down_read(&fi->wb_inval_rwsem); + if (fuse_inode_force_dio(inode)) { + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + return fuse_direct_write_iter(iocb, from); + } + } + + err = generic_write_checks(iocb, from); + if (err > 0) + err = __generic_file_write_iter(iocb, from); + + if (wb_guard) + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + + if (err > 0) + err = generic_write_sync(iocb, err); + + return err; } writethrough: inode_lock(inode); + /* + * The forced-direct-IO latch feature is active under writeback+dlm; + * hold wb_inval_rwsem for read across the page-cache dirtying so a + * concurrent NOTIFY_INVAL_INODE -- which latches the inode under the + * write side of this lock via a non-blocking trylock -- cannot strand + * the folios we are about to write. Re-check the latch under it (it may + * have been set while we blocked on the inode lock) and re-route to the + * direct path if set. The DLM write lock taken above is harmless as the + * direct path does its own server coordination. + */ + wb_guard = fc->writeback_cache && fc->dlm; + if (wb_guard) { + down_read(&fi->wb_inval_rwsem); + if (fuse_inode_force_dio(inode)) { + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + return fuse_direct_write_iter(iocb, from); + } + } + err = generic_write_checks(iocb, from); if (err <= 0) goto out; @@ -1525,6 +1640,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) written = fuse_perform_write(iocb, from); } out: + if (wb_guard) + up_read(&fi->wb_inval_rwsem); inode_unlock(inode); if (written > 0) written = generic_write_sync(iocb, written); @@ -1785,7 +1902,7 @@ static ssize_t fuse_file_read_iter(struct kiocb *iocb, struct iov_iter *to) if (FUSE_IS_DAX(inode)) return fuse_dax_read_iter(iocb, to); - if (!(ff->open_flags & FOPEN_DIRECT_IO)) + if (!(ff->open_flags & FOPEN_DIRECT_IO) && !fuse_inode_force_dio(inode)) return fuse_cache_read_iter(iocb, to); else return fuse_direct_read_iter(iocb, to); @@ -1803,7 +1920,7 @@ static ssize_t fuse_file_write_iter(struct kiocb *iocb, struct iov_iter *from) if (FUSE_IS_DAX(inode)) return fuse_dax_write_iter(iocb, from); - if (!(ff->open_flags & FOPEN_DIRECT_IO)) + if (!(ff->open_flags & FOPEN_DIRECT_IO) && !fuse_inode_force_dio(inode)) return fuse_cache_write_iter(iocb, from); else return fuse_direct_write_iter(iocb, from); @@ -2469,6 +2586,29 @@ static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma) if (FUSE_IS_DAX(file_inode(file))) return fuse_dax_mmap(file, vma); + /* + * If the inode was latched into forced direct IO after a remote-modify + * notification, a mapping needs the page cache, so revert to caching + * mode. Revert without the inode lock or wb_inval_rwsem: ->mmap runs + * under mmap_lock and the buffered write path holds both across a fault + * on the user buffer (which takes mmap_lock), so taking either here + * would invert lock order (ABBA). Clearing the latch and dropping the + * cache is sufficient -- writers re-check the latch and route to cached + * IO once it is clear, and in-flight parallel dio drains itself. Cached + * opens frozen while latched are still counted in iocachectr, so restore + * FUSE_I_CACHE_IO_MODE for them. + */ + if (fuse_inode_force_dio(file_inode(file))) { + struct fuse_inode *fi = get_fuse_inode(file_inode(file)); + + spin_lock(&fi->lock); + clear_bit(FUSE_I_FORCE_DIO, &fi->state); + if (fi->iocachectr > 0) + set_bit(FUSE_I_CACHE_IO_MODE, &fi->state); + spin_unlock(&fi->lock); + invalidate_inode_pages2(file->f_mapping); + } + /* * FOPEN_DIRECT_IO handling is special compared to O_DIRECT, * as does not allow MAP_SHARED mmap without FUSE_DIRECT_IO_ALLOW_MMAP. @@ -3285,6 +3425,9 @@ void fuse_init_file_inode(struct inode *inode, unsigned int flags) fi->iocachectr = 0; init_waitqueue_head(&fi->page_waitq); init_waitqueue_head(&fi->direct_io_waitq); + init_rwsem(&fi->wb_inval_rwsem); + fi->notify_stamp = jiffies; + fi->notify_interval_ewma = FUSE_NOTIFY_EWMA_SEED << FUSE_NOTIFY_EWMA_SHIFT; if (IS_ENABLED(CONFIG_FUSE_DAX)) fuse_dax_inode_init(inode, flags); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index c8c0b5a1477968..e828606cad3965 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -96,6 +96,20 @@ struct dlm_locked_area size_t size; }; +/* + * Force-DIO switch trigger: an exponentially weighted moving average of the + * interval (in jiffies) between FUSE_NOTIFY_INVAL_INODE data invalidations for + * a file. When the average spacing falls below FUSE_NOTIFY_DIO_INTERVAL -- a + * remote writer streaming invalidations -- and the file is open for writing + * here, it is latched into direct IO. These are the source-level (not + * externally tunable) parameters of the heuristic: EWMA weight 1/2^SHIFT, + * seeded and capped at SEED so it takes a short burst rather than a single + * notify to trip. + */ +#define FUSE_NOTIFY_DIO_INTERVAL max_t(unsigned long, HZ / 10, 1) +#define FUSE_NOTIFY_EWMA_SHIFT 2 +#define FUSE_NOTIFY_EWMA_SEED (2 * FUSE_NOTIFY_DIO_INTERVAL) + /** FUSE inode */ struct fuse_inode { /** Inode data */ @@ -163,6 +177,34 @@ struct fuse_inode { /* dlm locked areas we have sent lock requests for */ struct fuse_dlm_cache dlm_locked_areas; + + /* + * Serializes buffered-write page-cache dirtying against + * the forced-direct-IO latch transition driven by + * NOTIFY_INVAL_INODE (fuse_reverse_inval_inode()), which + * may be delivered by the same server thread that still + * owes a reply to an in-flight write holding the inode + * lock. The buffered writer holds this for read around + * the dirtying and re-checks the latch under it; the + * NOTIFY latch site takes it for write (trylock, never + * blocking) around its page-cache invalidate + latch set. + * Only regular files initialise it -- it shares storage + * with the readdir-cache union arm. + */ + struct rw_semaphore wb_inval_rwsem; + + /* + * Rate of FUSE_NOTIFY_INVAL_INODE data invalidations + * for this whole file: notify_stamp is the jiffies of + * the last one, notify_interval_ewma the EWMA of the + * inter-arrival interval (jiffies, scaled by + * 2^FUSE_NOTIFY_EWMA_SHIFT). A rapid stream (short + * average interval) with a local writer latches the + * inode into direct IO. Protected by fi->lock; regular + * files only (shares the readdir-cache union arm). + */ + unsigned long notify_stamp; + unsigned int notify_interval_ewma; }; /* readdir cache (directory only) */ @@ -224,6 +266,14 @@ enum { FUSE_I_BTIME, /* Wants or already has page cache IO */ FUSE_I_CACHE_IO_MODE, + /* + * Latched into direct IO: a NOTIFY_INVAL_INODE arrived while the file + * was open for writing here, so another (remote) entity is modifying it + * concurrently. Reads and writes are routed direct (shared-lock + * parallel dio) until the last writer closes or the inode is mmapped. + * See fuse_reverse_inval_inode()/fuse_file_io_open(). + */ + FUSE_I_FORCE_DIO, }; struct fuse_conn; @@ -1488,6 +1538,12 @@ void fuse_file_uncached_io_end(struct inode *inode, struct fuse_file *ff); int fuse_file_io_open(struct file *file, struct inode *inode); void fuse_file_io_release(struct fuse_file *ff, struct inode *inode); +/* Inode latched into forced direct IO after a remote-modify notification */ +static inline bool fuse_inode_force_dio(struct inode *inode) +{ + return test_bit(FUSE_I_FORCE_DIO, &get_fuse_inode(inode)->state); +} + /* file.c */ struct fuse_file *fuse_file_open(struct fuse_mount *fm, u64 nodeid, struct inode *inode, diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 82c84a81b6e2d3..3d5d666bf063f1 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -746,6 +746,35 @@ static void fuse_invalidate_inode_entry(struct inode *inode) } } +/* + * Fold one FUSE_NOTIFY_INVAL_INODE data invalidation into the per-inode + * moving average of the notification inter-arrival interval and report whether + * the file is now "hot" -- notifications are arriving fast enough (short + * average interval) that a remote writer is repeatedly invalidating it. The + * average is an EWMA (weight 1/2^FUSE_NOTIFY_EWMA_SHIFT); the sample is clamped + * to FUSE_NOTIFY_EWMA_SEED so a notify after a long idle only cools the average + * and cannot overflow the accumulator. Must be called under fi->lock; called + * for every data invalidation so the average stays current even while no local + * writer is open. + */ +static bool fuse_notify_inval_hot(struct fuse_inode *fi) +{ + unsigned long now = jiffies; + unsigned long sample; + unsigned int avg; + + sample = min_t(unsigned long, now - fi->notify_stamp, + FUSE_NOTIFY_EWMA_SEED); + fi->notify_stamp = now; + + /* E += sample - (E >> SHIFT); avg = E >> SHIFT */ + fi->notify_interval_ewma += sample - + (fi->notify_interval_ewma >> FUSE_NOTIFY_EWMA_SHIFT); + avg = fi->notify_interval_ewma >> FUSE_NOTIFY_EWMA_SHIFT; + + return avg < FUSE_NOTIFY_DIO_INTERVAL; +} + int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { @@ -803,8 +832,73 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pg_end == -1 ? 0 : (offset + len - 1)); - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); + /* + * A data invalidation means another (remote) entity is modifying + * the file. Keep a moving average of how fast these notifications + * arrive for the whole inode; when they come in a rapid stream -- + * a remote writer repeatedly invalidating the file -- and it is + * also open for writing here, latch the inode into direct IO: + * reads and writes are served direct (from the server) until the + * last writer closes or the inode is mmapped. A lone or occasional + * notify keeps the average high and does not trip the switch. Only + * under writeback+dlm, where the buffered write's RMW read is + * skipped so its wb_inval_rwsem read-side section is free of server + * round-trips. + * + * fuse_notify_inval_hot() updates the average under fi->lock and is + * called for every data invalidation so it stays current even while + * no local writer is open. When it trips, take wb_inval_rwsem for + * write so the buffered write path -- which holds it for read across + * its dirtying and re-checks the latch under it -- cannot strand + * dirty folios after the cache is dropped. Use a trylock and never + * block: this may run on the server thread that still owes an + * in-flight write (holding the inode lock) its reply, so blocking on + * the rwsem or the inode lock would deadlock. If the writer has + * gone, skip the latch this round (best effort); the invalidate + * still runs. Only regular files initialise the average and the + * rwsem (they share storage with the readdir-cache union arm), so + * gate on S_ISREG. When latched, drop the whole mapping rather than + * just the notified range, or dirty folios outside it would be + * invisible to the forced direct reads (stale read / lost write). + */ + if (S_ISREG(inode->i_mode) && fc->writeback_cache && fc->dlm && + !FUSE_IS_DAX(inode) && + !mapping_mapped(inode->i_mapping)) { + bool hot, has_writer; + + spin_lock(&fi->lock); + hot = fuse_notify_inval_hot(fi); + has_writer = !list_empty(&fi->write_files); + spin_unlock(&fi->lock); + + if (hot && has_writer && !fuse_inode_force_dio(inode) && + down_write_trylock(&fi->wb_inval_rwsem)) { + bool latched = false; + + spin_lock(&fi->lock); + if (!list_empty(&fi->write_files)) { + set_bit(FUSE_I_FORCE_DIO, &fi->state); + latched = true; + } + spin_unlock(&fi->lock); + + if (latched) { + pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", + nodeid); + invalidate_inode_pages2(inode->i_mapping); + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } + up_write(&fi->wb_inval_rwsem); + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } } iput(inode); return 0; diff --git a/fs/fuse/iomode.c b/fs/fuse/iomode.c index c0496b971da0cb..a42c7ba01307c0 100644 --- a/fs/fuse/iomode.c +++ b/fs/fuse/iomode.c @@ -163,6 +163,15 @@ int fuse_file_io_open(struct file *file, struct inode *inode) if (ff->open_flags & FOPEN_DIRECT_IO) return 0; + /* + * The inode was latched into direct IO after a remote-modify + * notification arrived while it was open for writing here. Open this + * file uncached as well so its IO is routed direct and it does not + * re-enter caching mode. + */ + if (fuse_inode_force_dio(inode)) + return 0; + err = fuse_file_cached_io_start(inode, ff); if (err) goto fail; From 82ef6d62deb9908c8c18c2d1d4dfb604579d7388 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 16 Jul 2026 14:02:56 +0200 Subject: [PATCH 02/10] fuse: balance uncached_io accounting under forced-DIO latch fuse_dio_lock() takes an uncached_io reference (via fuse_inode_uncached_io_start()) only when FUSE_I_FORCE_DIO is clear, while fuse_dio_unlock() decided whether to drop it by re-reading FUSE_I_FORCE_DIO. On this tree the latch is toggled asynchronously by the inode-invalidation notify-storm path, so the bit can differ between the lock and the unlock of a single direct write: - clear at lock (reference taken), set before unlock: the reference is never dropped, leaving fi->iocachectr permanently negative and hanging the next caching-mode open; - set at lock (no reference), cleared before unlock: fuse_inode_uncached_io_end() is called without a matching start, tripping WARN_ON(fi->iocachectr >= 0) and corrupting the counter. Record in fuse_dio_lock() whether a reference was actually taken and have fuse_dio_unlock() drop it based on that captured decision instead of re-testing the racy bit, so the accounting stays balanced regardless of any FORCE_DIO transition mid-write. Signed-off-by: Horst Birthelmer (cherry picked from commit 6bf7ec0a5f17050e002e311959e81e6af201d19b) [ahenderson: 6.8 keeps the past-EOF exclusive-lock trigger, as the parallel-direct-write rework is not ported, so the accounting decision becomes a nested if / else-if rather than upstream's flat form; fuse_inode_uncached_io_start() takes one argument] Signed-off-by: Allison Henderson --- fs/fuse/file.c | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index fbbcec385ff403..f3dce04e84e8e8 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1457,7 +1457,7 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from } static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, - bool *exclusive) + bool *exclusive, bool *uncached) { struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); @@ -1476,19 +1476,28 @@ static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, * * Under the forced-dio latch the cached/uncached accounting is * bypassed (the latch guarantees the cache is flushed and not - * repopulated), so only re-check the past-eof condition. + * repopulated), so no reference is taken. Report back through + * *uncached whether one actually was, so fuse_dio_unlock() can + * drop it based on that decision rather than re-testing the + * latch, which the notify-storm path can toggle mid-write. */ - if (fuse_io_past_eof(iocb, from) || - (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && - fuse_inode_uncached_io_start(fi) != 0)) { + if (fuse_io_past_eof(iocb, from)) { inode_unlock_shared(inode); inode_lock(inode); *exclusive = true; + } else if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) { + if (fuse_inode_uncached_io_start(fi) != 0) { + inode_unlock_shared(inode); + inode_lock(inode); + *exclusive = true; + } else { + *uncached = true; + } } } } -static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) +static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive, bool uncached) { struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); @@ -1496,12 +1505,8 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) if (exclusive) { inode_unlock(inode); } else { - /* - * Allow opens in caching mode after last parallel dio end. - * Skipped under the forced-dio latch, which never took an - * uncached_io reference in fuse_dio_lock(). - */ - if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) + /* Allow opens in caching mode after last parallel dio end */ + if (uncached) fuse_inode_uncached_io_end(fi); inode_unlock_shared(inode); } @@ -1861,10 +1866,11 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) struct fuse_io_priv io = FUSE_IO_PRIV_SYNC(iocb); struct address_space *mapping = inode->i_mapping; loff_t pos = iocb->ki_pos; + bool exclusive = false; + bool uncached = false; ssize_t res; - bool exclusive; - fuse_dio_lock(iocb, from, &exclusive); + fuse_dio_lock(iocb, from, &exclusive, &uncached); res = generic_write_checks(iocb, from); if (res > 0) { if (!is_sync_kiocb(iocb) && iocb->ki_flags & IOCB_DIRECT) { @@ -1885,7 +1891,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) (pos + res - 1) >> PAGE_SHIFT); } } - fuse_dio_unlock(iocb, exclusive); + fuse_dio_unlock(iocb, exclusive, uncached); return res; } From e3570fee3f8019f0acd60a1119fda510402088da Mon Sep 17 00:00:00 2001 From: Hai Zhong Zhou Date: Thu, 16 Jul 2026 09:55:59 +0000 Subject: [PATCH 03/10] fuse: acquire dlm lock for the normal buffer read Acquire the dlm lock from fuse server for the normal buffer read path to ensure the distributed page cache across different nodes can be co-existing and consistency. More importantly, this change will correct the DLM lock and folio locks ordering for the buffer read path, thus can avoid the potential deadlock between the buffer read and page cache invalidation processes. Signed-off-by Hai Zhong Zhou (cherry picked from commit cf349bc7b27864a36a092fbd4318bd8e80796f47) [ahenderson: context only - 6.8 retains the older fc->dlm block in fuse_cache_write_iter() and its 'goto writethrough' structure, so the write-side hunk is re-indented. The change itself is unmodified] Signed-off-by: Allison Henderson --- fs/fuse/file.c | 14 +++++++++++--- fs/fuse/fuse_dlm_cache.c | 22 ++++++++++++---------- fs/fuse/fuse_dlm_cache.h | 6 +++--- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index f3dce04e84e8e8..672ecbb3ca268f 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1124,7 +1124,8 @@ static void fuse_readahead(struct readahead_control *rac) static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { - struct inode *inode = iocb->ki_filp->f_mapping->host; + struct file *file = iocb->ki_filp; + struct inode *inode = file->f_mapping->host; struct fuse_conn *fc = get_fuse_conn(inode); /* @@ -1140,6 +1141,12 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) return err; } + /* if we have dlm support acquire a read lock for the area + * we are reading from. */ + if (fc->writeback_cache && fc->dlm) + fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), FUSE_PAGE_LOCK_READ); + return generic_file_read_iter(iocb, to); } @@ -1557,7 +1564,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * get the performance benefits of 'parallel direct writes'. */ loff_t pos = file->f_flags & O_APPEND ? i_size_read(inode) + iocb->ki_pos : iocb->ki_pos; size_t length = iov_iter_count(from); - fuse_get_dlm_write_lock(file, pos, length); + fuse_get_dlm_lock(file, pos, length, + FUSE_PAGE_LOCK_WRITE); } /* @@ -2482,7 +2490,7 @@ static void fuse_vma_close(struct vm_area_struct *vma) /** * Request a DLM lock from the FUSE server. * - * This routine is similar to fuse_get_dlm_write_lock(), but it + * This routine is similar to fuse_get_dlm_lock(), but it * does not cache the DLM lock in the kernel. */ static int fuse_get_page_mkwrite_lock(struct file *file, loff_t offset, size_t length) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index d765dd8018cc6a..ea296a2e9ec89c 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -487,10 +487,14 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, } /** - * request a dlm lock from the fuse server + * fuse_get_dlm_lock - request a dlm lock from the fuse server + * @file: the file being accessed + * @offset: byte offset into the file (need not be page-aligned) + * @length: length of the region in bytes (need not be page-aligned) + * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE */ -void fuse_get_dlm_write_lock(struct file *file, loff_t offset, - size_t length) +void fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) { struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(file); @@ -500,7 +504,7 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); /* note that the offset and length don't have to be page aligned here - * but since we only get here on writeback caching we will send out + * but since we only get here on writeback caching we will send out * page aligned requests */ offset &= PAGE_MASK; @@ -513,8 +517,7 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care * of any races in lock requests */ - if (fuse_dlm_range_is_locked(fi, offset, - end, FUSE_PAGE_LOCK_WRITE)) + if (fuse_dlm_range_is_locked(fi, offset, end, mode)) return; /* we already have this area locked */ memset(&inarg, 0, sizeof(inarg)); @@ -522,7 +525,8 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, inarg.start = offset; inarg.end = end; - inarg.type = FUSE_DLM_LOCK_WRITE; + inarg.type = (mode == FUSE_PAGE_LOCK_WRITE) ? + FUSE_DLM_LOCK_WRITE : FUSE_DLM_LOCK_READ; args.opcode = FUSE_DLM_WB_LOCK; args.nodeid = get_node_id(inode); @@ -551,8 +555,6 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, return; } else { /* ignore any errors here, there is no way we can react appropriately */ - fuse_dlm_lock_range(fi, outarg.start, - outarg.end, - FUSE_PAGE_LOCK_WRITE); + fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode); } } diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index 438d31d28b666e..5c3deaa3536866 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -43,8 +43,8 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); -/* this is the interface to the filesystem */ -void fuse_get_dlm_write_lock(struct file *file, loff_t offset, - size_t length); +/* This is the interface to the filesystem */ +void fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); #endif /* _FS_FUSE_DLM_CACHE_H */ From b85e8e142ba5211ffe2b85400051df0802833adc Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 16 Jul 2026 14:55:18 +0200 Subject: [PATCH 04/10] fuse: fence cached reads against NOTIFY invalidate with a percpu gate A FUSE_NOTIFY_INVAL_INODE is a coherency event: once the server signals a remote modify, no local read may return a page it has superseded. The invalidate ran unserialized against cache-serving reads, so a buffered read could hand back a stale folio it still held a reference to. Convert the per-inode wb_inval_rwsem to a percpu_rw_semaphore and take its read side around the cache-serving read as well as the existing buffered write. The read side is per-CPU, so it scales on a shared file; the NOTIFY takes the write side blocking, giving the invalidate priority -- it parks new readers, drains in-flight ones, then drops the cache. Every gated invalidate now runs under the write side, not just the storm-latching one. The gate is allocated only for writeback+dlm regular files and is NULL elsewhere (best-effort invalidate, as before). The blocking write side may run on the notify-delivering server thread, so it is safe only under a server that services request replies on other threads; redfs' dlm server provides that contract. Signed-off-by: Horst Birthelmer (cherry picked from commit d6180a20384a2f0d4b0eff4f4f8ebd00a5998c60) [ahenderson: fuse_cache_wr_unlock() comes from the unported shared-i_rwsem commit, so the gate's early exits use plain inode_unlock(); fuse_init_file_inode() gains the struct fuse_conn local that 6.8 lacks; no 'writeback' selector, as 6.8 has no iomap branch] Signed-off-by: Allison Henderson --- fs/fuse/file.c | 93 +++++++++++++++++++++++++--------- fs/fuse/fuse_i.h | 35 ++++++++----- fs/fuse/inode.c | 128 ++++++++++++++++++++++++++++++----------------- 3 files changed, 175 insertions(+), 81 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 672ecbb3ca268f..33c7676811e085 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1122,11 +1122,16 @@ static void fuse_readahead(struct readahead_control *rac) } } +static ssize_t fuse_direct_read_iter(struct kiocb *iocb, struct iov_iter *to); + static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; struct inode *inode = file->f_mapping->host; struct fuse_conn *fc = get_fuse_conn(inode); + struct fuse_inode *fi = get_fuse_inode(inode); + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + ssize_t res; /* * In auto invalidate mode, always update attributes on read. @@ -1147,7 +1152,29 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) fuse_get_dlm_lock(file, iocb->ki_pos, iov_iter_count(to), FUSE_PAGE_LOCK_READ); - return generic_file_read_iter(iocb, to); + /* + * Fence the cache-serving read against a NOTIFY invalidate so we never + * hand back a folio the server has just superseded. The gate read side + * is per-CPU cheap; the NOTIFY holds the write side with priority. + * Re-check the forced-DIO latch under it: if a storm latched us while we + * waited on a pending writer, reroute to direct like the buffered write + * path, so we do not repopulate the cache the latch just dropped. + * wb_sem is NULL on non-writeback+dlm mounts (gate inactive). + */ + if (wb_sem) { + percpu_down_read(wb_sem); + if (fuse_inode_force_dio(inode)) { + percpu_up_read(wb_sem); + return fuse_direct_read_iter(iocb, to); + } + } + + res = generic_file_read_iter(iocb, to); + + if (wb_sem) + percpu_up_read(wb_sem); + + return res; } static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff, @@ -1530,6 +1557,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) ssize_t err; struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; bool wb_guard = false; /* @@ -1569,12 +1597,13 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) } /* - * Hold wb_inval_rwsem for read across the page-cache dirtying - * so a concurrent NOTIFY_INVAL_INODE -- which latches the inode - * under the write side of this lock via a non-blocking trylock - * -- cannot strand the folios we are about to write. Re-check - * the latch under it (it may have been set while we blocked on - * the inode lock) and re-route to the direct path if set. + * Hold the coherency gate (wb_inval_rwsem) for read across the + * page-cache dirtying so a concurrent NOTIFY_INVAL_INODE -- + * which takes the write side (blocking, with priority) around + * its invalidate + latch set -- cannot strand the folios we are + * about to write. Re-check the latch under it (it may have been + * set while we blocked on the inode lock) and re-route to the + * direct path if set. * * generic_file_write_iter() is open-coded here so the gate can * be taken inside the inode lock. The order matters: every @@ -1582,13 +1611,13 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * that a writer holding i_rwsem exclusive can wait out gate * readers without deadlocking. */ - wb_guard = fc->dlm; + wb_guard = !!wb_sem; inode_lock(inode); if (wb_guard) { - down_read(&fi->wb_inval_rwsem); + percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); return fuse_direct_write_iter(iocb, from); } @@ -1599,7 +1628,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) err = __generic_file_write_iter(iocb, from); if (wb_guard) - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); if (err > 0) @@ -1613,19 +1642,20 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) /* * The forced-direct-IO latch feature is active under writeback+dlm; - * hold wb_inval_rwsem for read across the page-cache dirtying so a - * concurrent NOTIFY_INVAL_INODE -- which latches the inode under the - * write side of this lock via a non-blocking trylock -- cannot strand - * the folios we are about to write. Re-check the latch under it (it may - * have been set while we blocked on the inode lock) and re-route to the - * direct path if set. The DLM write lock taken above is harmless as the - * direct path does its own server coordination. + * hold the coherency gate (wb_inval_rwsem) for read across the + * page-cache dirtying so a concurrent NOTIFY_INVAL_INODE -- which takes + * the write side (blocking, with priority) around its invalidate + latch + * set -- cannot strand the folios we are about to write. Re-check the + * latch under it (it may have been set while we blocked on the inode + * lock) and re-route to the direct path if set. The DLM write lock + * taken above is harmless as the direct path does its own server + * coordination. */ - wb_guard = fc->writeback_cache && fc->dlm; + wb_guard = !!wb_sem; if (wb_guard) { - down_read(&fi->wb_inval_rwsem); + percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); return fuse_direct_write_iter(iocb, from); } @@ -1654,7 +1684,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) } out: if (wb_guard) - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); if (written > 0) written = generic_write_sync(iocb, written); @@ -3428,6 +3458,7 @@ static const struct address_space_operations fuse_file_aops = { void fuse_init_file_inode(struct inode *inode, unsigned int flags) { struct fuse_inode *fi = get_fuse_inode(inode); + struct fuse_conn *fc = get_fuse_conn(inode); inode->i_fop = &fuse_file_operations; inode->i_data.a_ops = &fuse_file_aops; @@ -3439,7 +3470,23 @@ void fuse_init_file_inode(struct inode *inode, unsigned int flags) fi->iocachectr = 0; init_waitqueue_head(&fi->page_waitq); init_waitqueue_head(&fi->direct_io_waitq); - init_rwsem(&fi->wb_inval_rwsem); + /* + * Coherency gate for the forced-direct-IO feature; only writeback+dlm + * regular files need it. A percpu_rw_semaphore embeds per-CPU state, + * so allocate it out of line and only when the mount can use it rather + * than paying it on every inode. On failure leave it NULL: the gate + * stays inactive (best-effort invalidate) and the inode is still usable. + */ + fi->wb_inval_rwsem = NULL; + if (fc->writeback_cache && fc->dlm) { + struct percpu_rw_semaphore *sem = kmalloc(sizeof(*sem), GFP_KERNEL); + + if (sem && percpu_init_rwsem(sem)) { + kfree(sem); + sem = NULL; + } + fi->wb_inval_rwsem = sem; + } fi->notify_stamp = jiffies; fi->notify_interval_ewma = FUSE_NOTIFY_EWMA_SEED << FUSE_NOTIFY_EWMA_SHIFT; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index e828606cad3965..7f9611079d3ee0 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -179,19 +180,29 @@ struct fuse_inode { struct fuse_dlm_cache dlm_locked_areas; /* - * Serializes buffered-write page-cache dirtying against - * the forced-direct-IO latch transition driven by - * NOTIFY_INVAL_INODE (fuse_reverse_inval_inode()), which - * may be delivered by the same server thread that still - * owes a reply to an in-flight write holding the inode - * lock. The buffered writer holds this for read around - * the dirtying and re-checks the latch under it; the - * NOTIFY latch site takes it for write (trylock, never - * blocking) around its page-cache invalidate + latch set. - * Only regular files initialise it -- it shares storage - * with the readdir-cache union arm. + * Per-inode read/write coherency gate for the + * forced-direct-IO feature. Cache-serving buffered reads + * and buffered writes hold it for read; being a + * percpu_rw_semaphore the read side is per-CPU cheap and + * scales on a shared file. The NOTIFY invalidate + * (fuse_reverse_inval_inode()) holds it for write, which + * BLOCKS so the coherency notify has priority: it fences + * cache-serving reads (and buffered writes) out for the + * whole invalidate, so no folio a remote modify has + * superseded is ever handed back. + * + * The write side may run on the server thread delivering + * the notify, so a blocking writer is safe only under a + * server that services request replies on threads other + * than the one delivering the notify (see the NOTIFY site). + * + * Allocated out of line only for writeback+dlm regular + * files (it shares storage with the readdir-cache union + * arm); NULL on other mounts and on allocation failure, + * where the gate is inactive and the invalidate falls back + * to best-effort. */ - struct rw_semaphore wb_inval_rwsem; + struct percpu_rw_semaphore *wb_inval_rwsem; /* * Rate of FUSE_NOTIFY_INVAL_INODE data invalidations diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 3d5d666bf063f1..8ffa12d782baff 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -191,6 +191,23 @@ static void fuse_evict_inode(struct inode *inode) WARN_ON(!list_empty(&fi->queued_writes)); fuse_dlm_cache_release_locks(fi); } + + /* + * Free the coherency gate here rather than in ->free_inode: that runs + * from an RCU callback, where percpu_free_rwsem() may sleep in + * rcu_sync_dtor() if the write side has not fully quiesced. No user + * can remain by eviction time: gate readers hold a file reference and + * a concurrent notify holds an inode reference. wb_inval_rwsem lives + * in the regular-file union arm and is only ever allocated for regular + * files, so gate on S_ISREG (but not fuse_is_bad() -- bad-marked + * regular files still own a gate); a directory's overlapping + * readdir-cache fields must not be misread. + */ + if (S_ISREG(inode->i_mode) && fi->wb_inval_rwsem) { + percpu_free_rwsem(fi->wb_inval_rwsem); + kfree(fi->wb_inval_rwsem); + fi->wb_inval_rwsem = NULL; + } } static int fuse_reconfigure(struct fs_context *fsc) @@ -778,6 +795,7 @@ static bool fuse_notify_inval_hot(struct fuse_inode *fi) int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { + struct percpu_rw_semaphore *wb_sem = NULL; struct fuse_inode *fi; struct inode *inode; pgoff_t pg_start; @@ -828,73 +846,91 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * Note that this can lead to some inconsistencies if * the fuse server sends unaligned data */ fuse_dlm_unlock_range(fi, - offset, - pg_end == -1 ? 0 : - (offset + len - 1)); + offset, + pg_end == -1 ? 0 : + (offset + len - 1)); /* * A data invalidation means another (remote) entity is modifying - * the file. Keep a moving average of how fast these notifications - * arrive for the whole inode; when they come in a rapid stream -- - * a remote writer repeatedly invalidating the file -- and it is - * also open for writing here, latch the inode into direct IO: - * reads and writes are served direct (from the server) until the - * last writer closes or the inode is mmapped. A lone or occasional - * notify keeps the average high and does not trip the switch. Only - * under writeback+dlm, where the buffered write's RMW read is - * skipped so its wb_inval_rwsem read-side section is free of server - * round-trips. + * the file. Two things happen here: + * + * 1. Coherency. Drop the affected page-cache range so no local + * read returns a folio the remote modify has superseded. This + * runs under the write side of the per-inode coherency gate + * (wb_inval_rwsem), which fences cache-serving buffered reads + * and buffered writes out for the whole invalidate. Unlike the + * old best-effort trylock this BLOCKS -- the notify has + * priority: percpu_down_write() parks new gate readers, drains + * in-flight ones, then invalidates. A blocking writer here is + * safe only under a server that services request replies on + * threads other than the one delivering this notify: the write + * side waits for gate readers to drain, and a cache-miss read + * holds the read side across its FUSE_READ round-trip. redfs' + * dlm server provides that contract; a server that cannot must + * not enable writeback+dlm. * - * fuse_notify_inval_hot() updates the average under fi->lock and is - * called for every data invalidation so it stays current even while - * no local writer is open. When it trips, take wb_inval_rwsem for - * write so the buffered write path -- which holds it for read across - * its dirtying and re-checks the latch under it -- cannot strand - * dirty folios after the cache is dropped. Use a trylock and never - * block: this may run on the server thread that still owes an - * in-flight write (holding the inode lock) its reply, so blocking on - * the rwsem or the inode lock would deadlock. If the writer has - * gone, skip the latch this round (best effort); the invalidate - * still runs. Only regular files initialise the average and the - * rwsem (they share storage with the readdir-cache union arm), so - * gate on S_ISREG. When latched, drop the whole mapping rather than - * just the notified range, or dirty folios outside it would be - * invisible to the forced direct reads (stale read / lost write). + * 2. Latch. Keep a moving average (fuse_notify_inval_hot(), under + * fi->lock, updated for every data invalidation) of how fast + * these arrive; when they come in a rapid stream -- a remote + * writer repeatedly invalidating -- and the inode is also open + * for writing here, latch it into direct IO until the last + * writer closes or it is mmapped. When latched, drop the whole + * mapping rather than just the notified range, or dirty folios + * outside it would be invisible to the forced direct reads + * (stale read / lost write). + * + * The gate (and the average) exist only for writeback+dlm regular + * files, and not while mmapped; elsewhere wb_sem is NULL and the + * invalidate runs unserialized (best-effort), as before. */ - if (S_ISREG(inode->i_mode) && fc->writeback_cache && fc->dlm && - !FUSE_IS_DAX(inode) && - !mapping_mapped(inode->i_mapping)) { - bool hot, has_writer; + if (S_ISREG(inode->i_mode) && fc->writeback_cache && + fc->dlm && !FUSE_IS_DAX(inode) && + !mapping_mapped(inode->i_mapping)) + wb_sem = fi->wb_inval_rwsem; + + if (wb_sem) { + bool hot, has_writer, latched = false; spin_lock(&fi->lock); hot = fuse_notify_inval_hot(fi); has_writer = !list_empty(&fi->write_files); spin_unlock(&fi->lock); - if (hot && has_writer && !fuse_inode_force_dio(inode) && - down_write_trylock(&fi->wb_inval_rwsem)) { - bool latched = false; + /* + * Priority write side: park new gate readers, + * drain in-flight ones, then invalidate. Blocks + * (unlike the old trylock) -- see the contract in + * the comment above. + */ + percpu_down_write(wb_sem); + if (hot && has_writer && + !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); if (!list_empty(&fi->write_files)) { set_bit(FUSE_I_FORCE_DIO, &fi->state); latched = true; } spin_unlock(&fi->lock); + } - if (latched) { - pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", - nodeid); - invalidate_inode_pages2(inode->i_mapping); - } else { - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); - } - up_write(&fi->wb_inval_rwsem); - } else { + /* + * Latched: drop the whole mapping (dirty folios + * outside the notified range would be invisible to + * the forced direct reads). Otherwise just the + * notified range. + */ + if (fuse_inode_force_dio(inode)) + invalidate_inode_pages2(inode->i_mapping); + else invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); - } + + percpu_up_write(wb_sem); + + if (latched) + pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", + nodeid); } else { invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); From 528eadf46181323964819dec67647cb85a70beab Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 17 Jul 2026 15:51:24 +0200 Subject: [PATCH 05/10] fuse: satisfy DLM read lock requests from a held write lock The buffered read path now acquires a DLM read lock via fuse_get_dlm_lock(..., FUSE_PAGE_LOCK_READ). Before sending the request to the server, fuse_get_dlm_lock() calls fuse_dlm_range_is_locked() to skip regions we already hold. That coverage check compared the held lock mode for exact equality (range->mode != lock_mode), so a range we already hold with an exclusive WRITE lock was reported as not-locked for a READ request. Because fuse_dlm_lock_range() intentionally does not downgrade a WRITE lock on a read, the region stays WRITE-locked and every subsequent read re-requests a DLM read lock from the server. This made read-after-write and re-read workloads flood the server with redundant FUSE_DLM_WB_LOCK requests, never converging. A held WRITE lock (exclusive) subsumes a READ lock. Treat a range as uncovered only when the held mode is strictly weaker than the requested mode (range->mode < lock_mode). READ requests are now satisfied by either a READ or a WRITE lock, while WRITE requests still require an existing WRITE lock (upgrade otherwise), matching the compatibility rules already documented in fuse_dlm_lock_range(). Signed-off-by: Horst Birthelmer (cherry picked from commit 4e3bef093ea1d715452e8f060c37d1c8f6881347) [ahenderson: applies unchanged] Signed-off-by: Allison Henderson --- fs/fuse/fuse_dlm_cache.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index ea296a2e9ec89c..40eda6daf75cae 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -454,9 +454,16 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, /* Check if the entire range is covered */ while (range && current_start <= end) { - /* If we're checking for a specific mode, verify it matches */ - if (lock_mode && range->mode != lock_mode) { - /* Wrong lock mode */ + /* + * The held lock must be at least as strong as the one + * requested. A WRITE lock (exclusive) satisfies a READ + * request, so only treat the range as uncovered when the + * held mode is weaker than what we ask for. This avoids + * re-requesting a READ lock for a range we already hold + * a WRITE lock on (e.g. read-after-write). + */ + if (lock_mode && range->mode < lock_mode) { + /* Held lock is weaker than requested */ up_read(&cache->lock); return false; } From b15a2ad59c6d3145c4ad1ae3905b4ddc68a7b54a Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Sat, 25 Jul 2026 12:45:45 +0200 Subject: [PATCH 06/10] fuse: seed DLM grant merging with an interval-tree lookup fuse_dlm_try_merge() locates the first merge candidate by walking from rb_first_cached() until it reaches the region just granted. The walk runs under the write-held cache rwsem on every fuse_dlm_lock_range() call, and the tree it walks holds every cached grant of the inode. Strided writers (IOR hard-write) accumulate grants that cannot merge with each other, so the tree keeps growing and every new grant pays a scan of all grants below it -- quadratic over the run, with fuse_dlm_range_is_locked() readers blocked behind each scan. Seed the merge with fuse_page_it_iter_first() on the region widened by one unit to each side instead; finding the lowest overlapping range is what the interval tree is there for. This also repairs two edge cases of the linear scan: a region starting at offset 0 made 'start - 1' wrap so the scan degenerated and merging was silently skipped, and a region ending at U64_MAX overflowed 'end + 1' in the loop bound, ending the merge after the first range. Both bounds now saturate. Signed-off-by: Horst Birthelmer (cherry picked from commit 67b2479421a97bd28d59466d445c075c130392c2) [ahenderson: applies unchanged] Signed-off-by: Allison Henderson --- fs/fuse/fuse_dlm_cache.c | 57 ++++++++++++++++++++++++++-------------- 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 40eda6daf75cae..2ec072b86312f4 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -120,26 +120,25 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, uint64_t end) { struct fuse_dlm_range *range, *next; - struct rb_node *node; + uint64_t first = start ? start - 1 : start; + uint64_t last = end < U64_MAX ? end + 1 : end; if (!cache) return; - /* Find the first range that might need merging */ - range = NULL; - node = rb_first_cached(&cache->ranges); - while (node) { - range = rb_entry(node, struct fuse_dlm_range, rb); - if (range->end >= start - 1) - break; - node = rb_next(node); - } - - if (!range || range->start > end + 1) - return; + /* + * Find the first range that might need merging. Directly adjacent + * ranges can merge, hence the region is widened by one unit to each + * side (saturating at the type bounds). This must stay an + * interval-tree lookup: the tree holds every cached grant of the + * inode and strided writers grow it for the lifetime of the file, + * so seeding the merge by walking from the tree minimum would make + * every new grant cost a full scan. + */ + range = fuse_page_it_iter_first(&cache->ranges, first, last); /* Try to merge ranges in and around the specified region */ - while (range && range->start <= end + 1) { + while (range && range->start <= last) { /* Get next range before we potentially modify the tree */ next = NULL; if (rb_next(&range->rb)) { @@ -150,11 +149,11 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, /* Try to merge with next range if adjacent and same mode */ if (next && range->mode == next->mode && range->end + 1 == next->start) { - /* Merge ranges */ - range->end = next->end; - - /* Remove next from tree */ + /* Merge ranges: re-insert so __subtree_end is updated */ fuse_page_it_remove(next, &cache->ranges); + fuse_page_it_remove(range, &cache->ranges); + range->end = next->end; + fuse_page_it_insert(range, &cache->ranges); kfree(next); /* Continue with the same range */ @@ -188,6 +187,7 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, struct fuse_dlm_cache *cache = &inode->dlm_locked_areas; struct fuse_dlm_range *range, *new_range, *next; int lock_mode; + bool covered_to_end = false; int ret = 0; LIST_HEAD(to_lock); LIST_HEAD(to_upgrade); @@ -233,14 +233,17 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, } /* Move current_start past this range */ - current_start = max(current_start, range->end + 1); + if (range->end >= end) + covered_to_end = true; + else + current_start = max(current_start, range->end + 1); /* Move to next range */ range = next; } /* If there's a gap after the last range to the end, extend the range */ - if (current_start <= end) { + if (!covered_to_end && current_start <= end) { new_range = kmalloc(sizeof(*new_range), GFP_KERNEL); if (!new_range) { ret = -ENOMEM; @@ -322,13 +325,17 @@ static int fuse_dlm_punch_hole(struct fuse_dlm_cache *cache, uint64_t start, /* If the hole is at the beginning of the range */ if (start == range->start) { + fuse_page_it_remove(range, &cache->ranges); range->start = end + 1; + fuse_page_it_insert(range, &cache->ranges); goto out; } /* If the hole is at the end of the range */ if (end == range->end) { + fuse_page_it_remove(range, &cache->ranges); range->end = start - 1; + fuse_page_it_insert(range, &cache->ranges); goto out; } @@ -400,10 +407,14 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, break; } else if (start > range->start) { /* Adjust the end of the range */ + fuse_page_it_remove(range, &cache->ranges); range->end = start - 1; + fuse_page_it_insert(range, &cache->ranges); } else if (end < range->end) { /* Adjust the start of the range */ + fuse_page_it_remove(range, &cache->ranges); range->start = end + 1; + fuse_page_it_insert(range, &cache->ranges); } else { /* Complete overlap, remove the range */ fuse_page_it_remove(range, &cache->ranges); @@ -475,6 +486,12 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, return false; } + /* Covered through the end of the requested range? */ + if (range->end >= end) { + up_read(&cache->lock); + return true; + } + /* Move current_start past this range */ current_start = range->end + 1; From ac614ab2cda86282c5999d950a05e968f520c942 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 28 Jul 2026 09:42:53 +0200 Subject: [PATCH 07/10] fuse: fix the DLM revoke range of an inode invalidate The NOTIFY_INVAL_INODE revoke computed fuse_dlm_unlock_range(fi, offset, pg_end == -1 ? 0 : offset + len - 1) which is wrong at both degenerate ends: a to-EOF invalidate (len <= 0, e.g. a remote truncate) with offset > 0 becomes the inverted range [offset, 0] and removes nothing, so the revoked grant stays visible to the re-validating IO paths forever -- cached writes with no server-side lock, zero-filled RMW reads; and an invalidate of byte 0 (offset 0, len 1) becomes [0, 0], the "destroy everything" sentinel, wiping every grant of the inode. Map the range in one helper shared by the gated and the ungated branch: to-EOF revokes through U64_MAX, and the bounds widen to page boundaries to match how grants are recorded -- revoking too much only costs a re-request, too little leaves a stale grant. Drop the in-band (0, 0) sentinel: whole-file invalidates walk the normal removal path, release-all is fuse_dlm_cache_release_locks(), and an inverted range is rejected with -EINVAL instead of silently ignored. Signed-off-by: Horst Birthelmer (cherry picked from commit e6f09f5a2bffa673a449926fe8dda06267a08e0d) [ahenderson: applied ahead of the grant re-validation commit in this series, so this also moves the revoke under the gate write side - work upstream did in that later commit. 'backing' dropped from the ungated-branch comment, as 6.8 has no backing-file support] Signed-off-by: Allison Henderson --- fs/fuse/fuse_dlm_cache.c | 15 +++++++------- fs/fuse/inode.c | 44 +++++++++++++++++++++++++++++++--------- 2 files changed, 41 insertions(+), 18 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 2ec072b86312f4..d5c22de2bebf27 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -369,8 +369,12 @@ static int fuse_dlm_punch_hole(struct fuse_dlm_cache *cache, uint64_t start, * @start: Start page offset * @end: End page offset * - * Release locks on the specified range of pages. - * Note that if start and end are set to zero the cache is destroyed. + * Release locks on the specified range of pages. An inverted range is + * rejected rather than silently removing nothing: the callers revoke + * coverage, and a revoke that quietly keeps the grant alive would let + * the re-validating IO paths trust a lock the server has taken away. + * To drop every grant use fuse_dlm_cache_release_locks() (there is no + * in-band sentinel range for it). * * Return: 0 on success, negative error code on failure */ @@ -381,14 +385,9 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, struct fuse_dlm_range *range, *next; int ret = 0; - if (!cache) + if (!cache || start > end) return -EINVAL; - if (start == 0 && end == 0) { - fuse_dlm_cache_release_locks(inode); - return 0; - } - down_write(&cache->lock); /* Find all ranges that overlap with [start, end] */ diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 8ffa12d782baff..9724ee61cc80e0 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -792,6 +792,25 @@ static bool fuse_notify_inval_hot(struct fuse_inode *fi) return avg < FUSE_NOTIFY_DIO_INTERVAL; } +/* + * Revoke the DLM grants backing an invalidated byte range. Grants are + * recorded page-aligned, so widen the revoke to page boundaries: dropping + * more than the server invalidated only costs a re-request, dropping less + * would leave a stale grant that fuse_dlm_lock_is_held() keeps trusting. + * len <= 0 means "invalidate to EOF" (see fuse_notify_inval_inode()) and + * revokes through U64_MAX -- it must not become an inverted range, which + * fuse_dlm_unlock_range() rejects without removing anything. + */ +static void fuse_dlm_revoke_inval_range(struct fuse_inode *fi, loff_t offset, + loff_t len) +{ + uint64_t start = (uint64_t)offset & PAGE_MASK; + uint64_t end = len <= 0 ? U64_MAX : + (((uint64_t)offset + len - 1) | (PAGE_SIZE - 1)); + + fuse_dlm_unlock_range(fi, start, end); +} + int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { @@ -840,16 +859,6 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, else pg_end = (offset + len - 1) >> PAGE_SHIFT; - if (fc->dlm && fc->writeback_cache) - /* Invalidate the range exactly as the fuse server requested - * except for the case where it sends -1. - * Note that this can lead to some inconsistencies if - * the fuse server sends unaligned data */ - fuse_dlm_unlock_range(fi, - offset, - pg_end == -1 ? 0 : - (offset + len - 1)); - /* * A data invalidation means another (remote) entity is modifying * the file. Two things happen here: @@ -904,6 +913,16 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, */ percpu_down_write(wb_sem); + /* + * Revoke the DLM lock range under the gate write + * side, atomically with the page drop: gate readers + * re-validate their grant right after entering, and + * a grant that passed that check must stay visible + * for their whole gate hold. + */ + if (fc->dlm && fc->writeback_cache) + fuse_dlm_revoke_inval_range(fi, offset, len); + if (hot && has_writer && !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); @@ -932,6 +951,11 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", nodeid); } else { + /* No gate on this inode (mmapped, DAX or + * non-regular): drop the lock range unserialized, + * as before. */ + if (fc->dlm && fc->writeback_cache) + fuse_dlm_revoke_inval_range(fi, offset, len); invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); } From d5649f82c6c4b97edd0f076c4cccc798017a9c4c Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 23 Jul 2026 10:29:12 +0200 Subject: [PATCH 08/10] fuse: re-validate the DLM grant after waiting on the coherency gate Both cached IO paths request their DLM lock first and then go to sleep on things a NOTIFY invalidate can be holding: the read path blocks on the coherency gate (writer priority), the write path additionally sleeps on a contended i_rwsem. A NOTIFY invalidate running in that window revokes exactly the lock just granted (fuse_dlm_unlock_range()), so the task wakes up and populates or dirties the page cache with no DLM coverage. Close the window without ever sending a FUSE_DLM_WB_LOCK request while holding the gate (a grant that had to wait on an invalidate delivered to this same client would deadlock against our own gate hold): - Drop the lock record under the gate write side in fuse_reverse_inval_inode(), so revocation and page drop are one atomic step with respect to the gate. - After entering the gate read side, re-check the grant against the live lock tree; if it was revoked while we waited, drop the gate, re-request, re-enter and check again. With the revoke now gated, passing the check means the lock cannot go away for the whole gate hold: a revoke arriving mid-operation parks until the IO is done. - Keep the write path's lock request ahead of the inode lock: the round trip must not capture the writer-priority i_rwsem for unbounded cluster-grant latency, and the in-gate re-validation already closes the grant-to-use window. Only O_APPEND moves below the lock, because its range is the current EOF -- stable only under the exclusive inode lock. This also fixes the append range itself: generic_write_checks() rewrites ki_pos to i_size for IOCB_APPEND, so the old 'i_size + ki_pos' double-counted (ki_pos is absolute, not relative) and locked a range disjoint from where the data lands. fuse_get_dlm_lock() now reports whether the grant is recorded, and the re-validation never re-requests a grant that failed, so it cannot spin (the read path seeds this from its pre-gate request instead of discarding that result). A grant the server issued but that could not be recorded (small-allocation -ENOMEM) reports FUSE_DLM_GRANT_UNRECORDED: coverage exists cluster-wide, so failing the IO would be wrong -- it proceeds, it just cannot re-validate. Empty ranges are trivially held, so a zero-length IO neither sends a doomed request nor spins in the retry loops. The write path returns a real failure to the caller instead of dirtying the cache without DLM coverage; only -ENOSYS still degrades to a plain cached write, since it means the server has no DLM at all and clears fc->dlm. The read path keeps falling through unlocked and additionally bounds its retry: a reader-only inode has no force-DIO latch to end a revoke storm, so after a few re-requests the read is served unlocked rather than looping in the kernel for the duration of the storm. Signed-off-by: Horst Birthelmer (cherry picked from commit 259ade5f62adf900860f11e0f77d1c143ff16278) [ahenderson: fuse_cache_wr_exclusive_lock() and fuse_cache_wr_unlock() come from the unported shared-i_rwsem commit; the cached write keeps the exclusive inode lock, so only fuse_cache_wr_dlm_lock() is taken from that hunk. The inode.c changes are already present, having come with the revoke-range commit applied earlier here. The in-gate re-validation is omitted from the writethrough path, which never requests a DLM lock on 6.8] Signed-off-by: Allison Henderson --- fs/fuse/file.c | 169 +++++++++++++++++++++++++++++++-------- fs/fuse/fuse_dlm_cache.c | 102 ++++++++++++++++------- fs/fuse/fuse_dlm_cache.h | 17 +++- 3 files changed, 226 insertions(+), 62 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 33c7676811e085..854f08303d6f66 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1124,6 +1124,12 @@ static void fuse_readahead(struct readahead_control *rac) static ssize_t fuse_direct_read_iter(struct kiocb *iocb, struct iov_iter *to); +/* + * Bound on re-requesting a revoked DLM grant before a cached read is + * served unlocked; see fuse_cache_read_iter(). + */ +#define FUSE_DLM_READ_RETRIES 3 + static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; @@ -1132,6 +1138,7 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) struct fuse_inode *fi = get_fuse_inode(inode); struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; ssize_t res; + int lock_err = 0; /* * In auto invalidate mode, always update attributes on read. @@ -1149,8 +1156,9 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) /* if we have dlm support acquire a read lock for the area * we are reading from. */ if (fc->writeback_cache && fc->dlm) - fuse_get_dlm_lock(file, iocb->ki_pos, - iov_iter_count(to), FUSE_PAGE_LOCK_READ); + lock_err = fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ); /* * Fence the cache-serving read against a NOTIFY invalidate so we never @@ -1162,11 +1170,44 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) * wb_sem is NULL on non-writeback+dlm mounts (gate inactive). */ if (wb_sem) { + int tries = FUSE_DLM_READ_RETRIES; + +retry: percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { percpu_up_read(wb_sem); return fuse_direct_read_iter(iocb, to); } + /* + * The DLM lock was requested before entering the gate, and + * the NOTIFY invalidate we may just have waited on revokes + * locks under the gate write side. Re-check the grant here + * and re-request with the gate dropped, so a + * FUSE_DLM_WB_LOCK round trip never parks a pending + * invalidate behind our own gate hold. Once the check + * passes the lock cannot go away for the rest of the gate + * hold. A failed or unrecorded request falls through + * unlocked, as before: the retry is taken even then (the + * latch must be re-checked under the re-entered gate), so + * lock_err has to stay sticky across it -- seeded by the + * pre-gate request above -- or a grant that failed would + * be re-requested forever. The retry is also bounded: a + * remote writer can revoke each successful grant before + * the gate is re-entered, and a reader-only inode has no + * force-DIO latch to end such a storm, so after + * FUSE_DLM_READ_RETRIES re-requests the read is served + * unlocked rather than looping without bound. + */ + if (!lock_err && fc->dlm && tries-- > 0 && + !fuse_dlm_lock_is_held(fi, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ)) { + percpu_up_read(wb_sem); + lock_err = fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ); + goto retry; + } } res = generic_file_read_iter(iocb, to); @@ -1548,6 +1589,26 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive, bool uncached) static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from); +/* + * Request the DLM write lock covering a cached write. -ENOSYS cleared + * fc->dlm: the server has no DLM, proceed as a plain cached write. Any + * other failure means the cache would be dirtied without DLM coverage - + * the caller must fail the write instead. A granted-but-unrecorded + * lock (positive return) is covered cluster-wide; proceed, but flag it + * so the in-gate re-validation skips a check an invisible grant could + * never pass. + */ +static int fuse_cache_wr_dlm_lock(struct file *file, loff_t pos, size_t len, + bool *unrecorded) +{ + int err = fuse_get_dlm_lock(file, pos, len, FUSE_PAGE_LOCK_WRITE); + + if (err < 0 && err != -ENOSYS) + return err; + *unrecorded = err > 0; + return 0; +} + static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -1559,14 +1620,10 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) struct fuse_inode *fi = get_fuse_inode(inode); struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; bool wb_guard = false; + bool dlm_unrecorded = false; + loff_t dlm_pos = 0; + size_t dlm_len = 0; - /* - * The inode may have been latched into forced direct IO -- by a - * NOTIFY_INVAL_INODE arriving while this inode is open for writing here - * -- after this write was routed to the cached path but before it took - * any lock. Re-route to the direct path (before taking a DLM lock) so - * we do not repopulate the page cache the latch just dropped. - */ if (fuse_inode_force_dio(inode)) return fuse_direct_write_iter(iocb, from); @@ -1583,17 +1640,52 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) goto writethrough; } - /* if we have dlm support acquire the lock for the area - * we are writing into */ - if (fc->dlm) { - /* note that a file opened with O_APPEND will have relative values - * in ki_pos. This code is here for convenience and for libfuse overlay test. - * Filesystems should handle O_APPEND with 'direct io' to additionally - * get the performance benefits of 'parallel direct writes'. */ - loff_t pos = file->f_flags & O_APPEND ? i_size_read(inode) + iocb->ki_pos : iocb->ki_pos; - size_t length = iov_iter_count(from); - fuse_get_dlm_lock(file, pos, length, - FUSE_PAGE_LOCK_WRITE); + /* + * Request the DLM write lock before taking i_rwsem: the request + * is an unbounded cluster round trip, and holding the + * writer-priority rwsem across it would park a truncate -- and + * behind it every later writer -- for the duration. The + * grant-to-use window this leaves open is closed by the in-gate + * re-validation below. Only the append case must wait for the + * lock: its range depends on i_size, which is stable only under + * the exclusive inode lock. + */ + if (fc->dlm && !(iocb->ki_flags & IOCB_APPEND)) { + dlm_pos = iocb->ki_pos; + dlm_len = iov_iter_count(from); + + err = fuse_cache_wr_dlm_lock(file, dlm_pos, dlm_len, + &dlm_unrecorded); + if (err) + return err; + } + + /* + * generic_file_write_iter() is open-coded from here so the gate + * can be taken inside the inode lock. The order matters: every + * gate holder must take i_rwsem first and the gate second, so + * that a writer holding i_rwsem exclusive can wait out gate + * readers without deadlocking. + */ + inode_lock(inode); + + /* note that this small code dup will save us a lot of headache later + * when appends are done concurrently without using parallel direct writes */ + if (fc->dlm && (iocb->ki_flags & IOCB_APPEND)) { + /* + * An append write lands at the current EOF no matter + * what ki_pos holds: generic_write_checks() rewrites + * ki_pos to i_size for IOCB_APPEND, and i_size is stable + * here because append writes hold the inode lock + * exclusive. Lock where the data will land. + */ + dlm_pos = i_size_read(inode); + dlm_len = iov_iter_count(from); + + err = fuse_cache_wr_dlm_lock(file, dlm_pos, dlm_len, + &dlm_unrecorded); + if (err) + goto wb_out; } /* @@ -1603,30 +1695,42 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * its invalidate + latch set -- cannot strand the folios we are * about to write. Re-check the latch under it (it may have been * set while we blocked on the inode lock) and re-route to the - * direct path if set. - * - * generic_file_write_iter() is open-coded here so the gate can - * be taken inside the inode lock. The order matters: every - * gate holder must take i_rwsem first and the gate second, so - * that a writer holding i_rwsem exclusive can wait out gate - * readers without deadlocking. + * direct path if set. Then re-validate the DLM grant: the + * request above ran outside the gate, and the invalidate we may + * just have waited on revokes grants under the gate write side. */ wb_guard = !!wb_sem; - - inode_lock(inode); if (wb_guard) { +retry: percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { percpu_up_read(wb_sem); inode_unlock(inode); return fuse_direct_write_iter(iocb, from); } + if (fc->dlm && !dlm_unrecorded && + !fuse_dlm_lock_is_held(fi, dlm_pos, dlm_len, + FUSE_PAGE_LOCK_WRITE)) { + percpu_up_read(wb_sem); + err = fuse_cache_wr_dlm_lock(file, dlm_pos, + dlm_len, + &dlm_unrecorded); + if (err) { + /* The gate is already dropped; funnel + * the failure through the one audited + * exit. */ + wb_guard = false; + goto wb_out; + } + goto retry; + } } err = generic_write_checks(iocb, from); if (err > 0) err = __generic_file_write_iter(iocb, from); +wb_out: if (wb_guard) percpu_up_read(wb_sem); inode_unlock(inode); @@ -1646,10 +1750,9 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * page-cache dirtying so a concurrent NOTIFY_INVAL_INODE -- which takes * the write side (blocking, with priority) around its invalidate + latch * set -- cannot strand the folios we are about to write. Re-check the - * latch under it (it may have been set while we blocked on the inode - * lock) and re-route to the direct path if set. The DLM write lock - * taken above is harmless as the direct path does its own server - * coordination. + * latch under it and re-route to the direct path if set. No DLM + * re-validation here: this path never requested a DLM lock (the + * writeback branch above returns before reaching it). */ wb_guard = !!wb_sem; if (wb_guard) { diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index d5c22de2bebf27..4714d48e6bc9b1 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -509,45 +509,83 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, return true; } +/** + * fuse_dlm_lock_is_held - check that a byte range is covered by a granted lock + * @fi: the fuse inode + * @offset: byte offset into the file (need not be page-aligned) + * @length: length of the region in bytes (need not be page-aligned) + * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE + * + * Re-validation helper for fuse_get_dlm_lock() callers: checks the same + * page-aligned range a fuse_get_dlm_lock() call with these arguments + * requests, against the live lock tree. + */ +bool fuse_dlm_lock_is_held(struct fuse_inode *fi, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) +{ + uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); + + /* + * An empty range needs no coverage. Reporting it held keeps the + * re-validating IO paths from re-requesting a lock the tree can + * never show (the page-aligned end would invert below). + */ + if (!length) + return true; + + return fuse_dlm_range_is_locked(fi, offset & PAGE_MASK, end, mode); +} + /** * fuse_get_dlm_lock - request a dlm lock from the fuse server * @file: the file being accessed * @offset: byte offset into the file (need not be page-aligned) * @length: length of the region in bytes (need not be page-aligned) * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE + * + * Return: 0 when the range is covered by a recorded grant on return, + * FUSE_DLM_GRANT_UNRECORDED when the server granted the lock but + * recording it failed (covered cluster-wide, invisible to + * fuse_dlm_lock_is_held()), a negative error code otherwise. Callers + * re-validating the grant must not re-request on a nonzero return or + * they would spin. */ -void fuse_get_dlm_lock(struct file *file, loff_t offset, - size_t length, enum fuse_page_lock_mode mode) +int fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) { struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(file); struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); struct fuse_mount *fm = ff->fm; - uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); - - /* note that the offset and length don't have to be page aligned here - * but since we only get here on writeback caching we will send out - * page aligned requests */ - offset &= PAGE_MASK; FUSE_ARGS(args); struct fuse_dlm_lock_in inarg; struct fuse_dlm_lock_out outarg; int err; + /* An empty range needs no lock. */ + if (!length) + return 0; + /* note that this can be run from different processes * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care - * of any races in lock requests */ - if (fuse_dlm_range_is_locked(fi, offset, end, mode)) - return; /* we already have this area locked */ + * of any races in lock requests. + * The early exit uses the same helper the callers re-validate + * with, so this check and a later fuse_dlm_lock_is_held() can + * never disagree about what counts as covered. */ + if (fuse_dlm_lock_is_held(fi, offset, length, mode)) + return 0; /* we already have this area locked */ memset(&inarg, 0, sizeof(inarg)); inarg.fh = ff->fh; - inarg.start = offset; - inarg.end = end; + /* note that the offset and length don't have to be page aligned + * here but since we only get here on writeback caching we will + * send out page aligned requests */ + inarg.start = offset & PAGE_MASK; + inarg.end = (offset + length - 1) | (PAGE_SIZE - 1); inarg.type = (mode == FUSE_PAGE_LOCK_WRITE) ? FUSE_DLM_LOCK_WRITE : FUSE_DLM_LOCK_READ; @@ -563,21 +601,31 @@ void fuse_get_dlm_lock(struct file *file, loff_t offset, if (err == -ENOSYS) { /* fuse server does not support dlm, save the info */ fc->dlm = 0; - return; + return err; } if (err) - return; - else - if (inarg.start < outarg.start || - inarg.end > outarg.end) { - /* fuse server is seriously broken */ - pr_warn("fuse: dlm lock request for %llu:%llu returned %llu:%llu bytes\n", - inarg.start, inarg.end, outarg.start, outarg.end); - fuse_abort_conn(fc); - return; - } else { - /* ignore any errors here, there is no way we can react appropriately */ - fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode); - } + return err; + + if (inarg.start < outarg.start || inarg.end > outarg.end) { + /* fuse server is seriously broken */ + pr_warn("fuse: dlm lock request for %llu:%llu returned %llu:%llu bytes\n", + inarg.start, inarg.end, outarg.start, outarg.end); + fuse_abort_conn(fc); + return -EIO; + } + + /* + * The server granted the lock; record it so + * fuse_dlm_lock_is_held() sees it. A failure to record + * (small-allocation -ENOMEM) does not undo the grant: coverage + * exists cluster-wide, only the local bookkeeping is missing. + * Report that as FUSE_DLM_GRANT_UNRECORDED so callers neither + * fail an IO that is actually covered nor keep re-requesting a + * grant that will not become visible. + */ + if (fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode)) + return FUSE_DLM_GRANT_UNRECORDED; + + return 0; } diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index 5c3deaa3536866..b0b16c56e3b0b0 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -17,6 +17,15 @@ struct fuse_inode; /* Lock modes for page ranges */ enum fuse_page_lock_mode { FUSE_PAGE_LOCK_READ, FUSE_PAGE_LOCK_WRITE }; +/* + * fuse_get_dlm_lock() result: the server granted the lock but recording + * it locally failed, leaving the grant invisible to + * fuse_dlm_lock_is_held(). The IO is covered cluster-wide; the caller + * must proceed without re-validating (a re-request would spin) instead + * of failing the IO. + */ +#define FUSE_DLM_GRANT_UNRECORDED 1 + /* Page cache lock manager */ struct fuse_dlm_cache { /* Lock protecting the tree */ @@ -43,8 +52,12 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); +/* Re-validate a fuse_get_dlm_lock() grant against the live lock tree */ +bool fuse_dlm_lock_is_held(struct fuse_inode *inode, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); + /* This is the interface to the filesystem */ -void fuse_get_dlm_lock(struct file *file, loff_t offset, - size_t length, enum fuse_page_lock_mode mode); +int fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); #endif /* _FS_FUSE_DLM_CACHE_H */ From 8f1e69fb5f43344645b30818066bb188708a7a27 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 28 Jul 2026 09:47:47 +0200 Subject: [PATCH 09/10] fuse: order grant recording against concurrent revokes A FUSE_DLM_WB_LOCK reply and a NOTIFY invalidate are serviced on different threads, so a revoke aimed at the grant a reply carries can be processed before fuse_get_dlm_lock() records it: the revoke finds nothing to remove, and the requester then records an already-dead grant that no later NOTIFY will target -- a permanent false positive for the re-validating IO paths. Add a revocation generation to the lock cache, bumped under the cache lock by every revoke path -- unconditionally, because the racing revoke sees an empty overlap precisely when the grant is in flight. fuse_get_dlm_lock() samples it before sending and records through fuse_dlm_lock_range_gen(), which refuses with -EAGAIN once the generation has moved; the grant is then re-requested instead of recorded, bounded so a revoke storm cannot pin the IO here (past the bound the failure reports like any request failure). Signed-off-by: Horst Birthelmer (cherry picked from commit 8983871c58b2dabbb18b87ad5fb731d711c4c392) [ahenderson: applies unchanged] Signed-off-by: Allison Henderson --- fs/fuse/fuse_dlm_cache.c | 110 +++++++++++++++++++++++++++++++++++---- fs/fuse/fuse_dlm_cache.h | 15 ++++++ 2 files changed, 115 insertions(+), 10 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 4714d48e6bc9b1..960d51e7836a3c 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -31,6 +31,12 @@ struct fuse_dlm_range { #define FUSE_PCACHE_LK_READ 1 /* Shared read lock */ #define FUSE_PCACHE_LK_WRITE 2 /* Exclusive write lock */ +/* + * Bound on re-requesting a grant whose recording lost against a + * concurrent revoke; see fuse_get_dlm_lock(). + */ +#define FUSE_DLM_RECORD_TRIES 3 + /* Interval tree definitions for page ranges */ static inline uint64_t fuse_dlm_range_start(struct fuse_dlm_range *range) { @@ -63,6 +69,7 @@ int fuse_dlm_cache_init(struct fuse_inode *inode) init_rwsem(&cache->lock); cache->ranges = RB_ROOT_CACHED; + cache->revoke_gen = 0; return 0; } @@ -84,6 +91,7 @@ void fuse_dlm_cache_release_locks(struct fuse_inode *inode) /* Release all locks */ down_write(&cache->lock); + WRITE_ONCE(cache->revoke_gen, cache->revoke_gen + 1); while ((node = rb_first_cached(&cache->ranges)) != NULL) { range = rb_entry(node, struct fuse_dlm_range, rb); fuse_page_it_remove(range, &cache->ranges); @@ -166,11 +174,13 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, } /** - * fuse_dlm_lock_range - Lock a range of pages + * __fuse_dlm_lock_range - Lock a range of pages * @cache: The page cache * @start: Start page offset * @end: End page offset * @mode: Lock mode (read or write) + * @genp: If non-NULL, the revocation generation sampled before the grant + * was requested; recording fails with -EAGAIN if it has moved * * Add a locked range on the specified range of pages. * If parts of the range are already locked, only add the remaining parts. @@ -181,8 +191,9 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, * * Return: 0 on success, negative error code on failure */ -int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, - uint64_t end, enum fuse_page_lock_mode mode) +static int __fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + const uint64_t *genp) { struct fuse_dlm_cache *cache = &inode->dlm_locked_areas; struct fuse_dlm_range *range, *new_range, *next; @@ -202,6 +213,17 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, down_write(&cache->lock); + /* + * A revoke was processed after @genp was sampled; the grant this + * record carries may be the very one it targeted (a revoke of a + * not-yet-recorded grant removes nothing and would never be + * retried). Refuse, the caller re-requests. + */ + if (genp && cache->revoke_gen != *genp) { + up_write(&cache->lock); + return -EAGAIN; + } + /* Find all ranges that overlap with [start, end] */ range = fuse_page_it_iter_first(&cache->ranges, start, end); while (range) { @@ -297,6 +319,35 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, return ret; } +int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode) +{ + return __fuse_dlm_lock_range(inode, start, end, mode, NULL); +} + +int fuse_dlm_lock_range_gen(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + uint64_t gen) +{ + return __fuse_dlm_lock_range(inode, start, end, mode, &gen); +} + +/** + * fuse_dlm_revoke_gen - sample the revocation generation + * @inode: the fuse inode + * + * Sampled before a FUSE_DLM_WB_LOCK request leaves the client. The + * reply and a NOTIFY revoke can be serviced on different threads, so a + * revoke may be processed between the reply arriving and its grant + * being recorded. fuse_dlm_lock_range_gen() re-checks the generation + * under the cache lock and refuses to record a grant such a revoke may + * have already killed. + */ +uint64_t fuse_dlm_revoke_gen(struct fuse_inode *inode) +{ + return READ_ONCE(inode->dlm_locked_areas.revoke_gen); +} + /** * fuse_dlm_punch_hole - Punch a hole in a locked range * @cache: The page cache @@ -390,6 +441,14 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, down_write(&cache->lock); + /* + * Unconditional, even when nothing overlaps: the revoke racing + * with an in-flight grant finds an empty tree precisely because + * the grant is not recorded yet, and the bump is what makes the + * recording side notice (see fuse_dlm_lock_range_gen()). + */ + WRITE_ONCE(cache->revoke_gen, cache->revoke_gen + 1); + /* Find all ranges that overlap with [start, end] */ range = fuse_page_it_iter_first(&cache->ranges, start, end); while (range) { @@ -562,12 +621,15 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, FUSE_ARGS(args); struct fuse_dlm_lock_in inarg; struct fuse_dlm_lock_out outarg; + uint64_t gen; + int tries = FUSE_DLM_RECORD_TRIES; int err; /* An empty range needs no lock. */ if (!length) return 0; +restart: /* note that this can be run from different processes * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care @@ -578,6 +640,16 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, if (fuse_dlm_lock_is_held(fi, offset, length, mode)) return 0; /* we already have this area locked */ + /* + * Sample the revocation generation before the request leaves. + * The reply and a NOTIFY revoke are serviced on different + * threads, so a revoke aimed at the grant this request returns + * can be processed before the grant is recorded below -- + * recording it anyway would resurrect a dead grant that no later + * NOTIFY will ever remove. + */ + gen = fuse_dlm_revoke_gen(fi); + memset(&inarg, 0, sizeof(inarg)); inarg.fh = ff->fh; @@ -617,14 +689,32 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, /* * The server granted the lock; record it so - * fuse_dlm_lock_is_held() sees it. A failure to record - * (small-allocation -ENOMEM) does not undo the grant: coverage - * exists cluster-wide, only the local bookkeeping is missing. - * Report that as FUSE_DLM_GRANT_UNRECORDED so callers neither - * fail an IO that is actually covered nor keep re-requesting a - * grant that will not become visible. + * fuse_dlm_lock_is_held() sees it. */ - if (fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode)) + err = fuse_dlm_lock_range_gen(fi, outarg.start, outarg.end, mode, gen); + if (err == -EAGAIN) { + /* + * A revoke was processed while the request was in flight; + * the grant may already be dead, so re-request instead of + * recording it. Bounded: a revoke storm must not pin the + * IO here -- past the bound the failure is reported like + * any other request failure (the write path fails the + * write, the read path serves unlocked). + */ + if (--tries) + goto restart; + return -EAGAIN; + } + + /* + * A failure to record (small-allocation -ENOMEM) does not undo + * the grant: coverage exists cluster-wide, only the local + * bookkeeping is missing. Report that as + * FUSE_DLM_GRANT_UNRECORDED so callers neither fail an IO that + * is actually covered nor keep re-requesting a grant that will + * not become visible. + */ + if (err) return FUSE_DLM_GRANT_UNRECORDED; return 0; diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index b0b16c56e3b0b0..647a8c37c36095 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -32,6 +32,13 @@ struct fuse_dlm_cache { struct rw_semaphore lock; /* Interval tree of locked ranges */ struct rb_root_cached ranges; + /* + * Bumped under @lock by every revocation + * (fuse_dlm_unlock_range(), fuse_dlm_cache_release_locks()); + * lets fuse_get_dlm_lock() order recording a reply's grant + * against revokes processed while the reply was in flight. + */ + uint64_t revoke_gen; }; /* Initialize a page cache lock manager */ @@ -44,6 +51,14 @@ void fuse_dlm_cache_release_locks(struct fuse_inode *inode); int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); +/* As above, but refuse (-EAGAIN) if a revoke ran since @gen was sampled */ +int fuse_dlm_lock_range_gen(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + uint64_t gen); + +/* Sample the revocation generation (see fuse_dlm_lock_range_gen()) */ +uint64_t fuse_dlm_revoke_gen(struct fuse_inode *inode); + /* Unlock a range of pages */ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, uint64_t end); From f9bb5f2d72b55e9b6449fa1d81b0fc77e457dc3f Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 28 Jul 2026 09:45:16 +0200 Subject: [PATCH 10/10] fuse: fence mmapped invalidates and local truncates with the coherency gate Two revocation paths bypassed the revoke-under-gate invariant the IO paths re-validate against: - fuse_reverse_inval_inode() skipped the gate once mapping_mapped() turned true, revoking concurrently with gate holders -- but fuse_cache_read_iter()/fuse_cache_write_iter() enter the gate unconditionally, so a single mmap() reopened the race. Keep the gate for mmapped inodes; only the force-DIO latch stays disabled for them (a mapping needs the page cache, and fuse_file_mmap() reverts any latch it races with). - The local truncates in fuse_do_setattr() -- the atomic-O_TRUNC open shortcut and the after-setattr trim -- revoked and dropped the cache with no gate at all, so an already re-validated reader could repopulate the truncated range. Take the gate write side around revoke + drop. This cannot deadlock: both run under exclusive i_rwsem, which no gate holder waits on (the write path takes i_rwsem before the gate, the read path never takes it). Signed-off-by: Horst Birthelmer (cherry picked from commit be0662c3149db03aae7dda5074f20bdd7271a388) [ahenderson: !fuse_inode_backing() dropped from the gate condition and its comment, as 6.8 has no backing-file support; the atomic-O_TRUNC hunk lands without the fi->lock / server_size lines, which come from the unported zero-fill-expanding-writes commit] Signed-off-by: Allison Henderson --- fs/fuse/dir.c | 29 +++++++++++++++++++++++++++++ fs/fuse/inode.c | 19 ++++++++++++------- 2 files changed, 41 insertions(+), 7 deletions(-) diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index bc3e495d619d06..59c11ece2f95e5 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -2074,15 +2074,32 @@ int fuse_do_setattr(struct dentry *dentry, struct iattr *attr, WARN_ON(!(attr->ia_valid & ATTR_SIZE)); WARN_ON(attr->ia_size != 0); if (fc->atomic_o_trunc) { + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + /* * No need to send request to userspace, since actual * truncation has already been done by OPEN. But still * need to truncate page cache. + * + * Revoke and drop under the coherency gate write side, + * like the NOTIFY invalidate path: a gate reader that + * already re-validated its grant must not have the + * lock tree and the cache yanked mid-hold, or it + * would repopulate the truncated range trusting a + * grant that no longer exists. Waiting for gate + * readers here is safe: we hold i_rwsem exclusive, so + * no gate holder can be waiting on it (the write path + * takes i_rwsem before the gate, the read path never + * takes it). */ + if (wb_sem) + percpu_down_write(wb_sem); if (fc->dlm && fc->writeback_cache) fuse_dlm_cache_release_locks(fi); i_size_write(inode, 0); truncate_pagecache(inode, 0); + if (wb_sem) + percpu_up_write(wb_sem); goto out; } file = NULL; @@ -2186,11 +2203,23 @@ int fuse_do_setattr(struct dentry *dentry, struct iattr *attr, */ if ((is_truncate || !is_wb) && S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) { + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + + /* + * Revoke and drop under the coherency gate write side; see + * the atomic-O_TRUNC branch above. i_rwsem is held + * exclusive here as well (setattr), so waiting out gate + * readers cannot deadlock. + */ + if (wb_sem) + percpu_down_write(wb_sem); if (fc->dlm && fc->writeback_cache) fuse_dlm_unlock_range(fi, outarg.attr.size & PAGE_MASK, -1); truncate_pagecache(inode, outarg.attr.size); invalidate_inode_pages2(mapping); + if (wb_sem) + percpu_up_write(wb_sem); } clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 9724ee61cc80e0..f1eb5cdaae4972 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -889,12 +889,16 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * (stale read / lost write). * * The gate (and the average) exist only for writeback+dlm regular - * files, and not while mmapped; elsewhere wb_sem is NULL and the - * invalidate runs unserialized (best-effort), as before. + * files; elsewhere wb_sem is NULL and the invalidate runs + * unserialized (best-effort), as before. An mmapped inode + * keeps the gate -- fuse_cache_read_iter() and + * fuse_cache_write_iter() enter it unconditionally and rely + * on the revoke staying fenced -- but is never latched: + * a mapping needs the page cache, and fuse_file_mmap() + * reverts any latch it races with. */ if (S_ISREG(inode->i_mode) && fc->writeback_cache && - fc->dlm && !FUSE_IS_DAX(inode) && - !mapping_mapped(inode->i_mapping)) + fc->dlm && !FUSE_IS_DAX(inode)) wb_sem = fi->wb_inval_rwsem; if (wb_sem) { @@ -924,6 +928,7 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, fuse_dlm_revoke_inval_range(fi, offset, len); if (hot && has_writer && + !mapping_mapped(inode->i_mapping) && !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); if (!list_empty(&fi->write_files)) { @@ -951,9 +956,9 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", nodeid); } else { - /* No gate on this inode (mmapped, DAX or - * non-regular): drop the lock range unserialized, - * as before. */ + /* No gate on this inode (DAX, non-regular, or the + * gate allocation failed): drop the lock range + * unserialized (best-effort), as before. */ if (fc->dlm && fc->writeback_cache) fuse_dlm_revoke_inval_range(fi, offset, len); invalidate_inode_pages2_range(inode->i_mapping,