trying to steal weird page?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

trying to steal weird page?

Antonio SJ Musumeci

A user reported getting the below errors when *not* using direct_io but atomic_o_trunc, auto_cache, big_writes, default_permissions, splice_move, splice_read, and splice_write are enabled.

Any ideas?

kernel: fuse: trying to steal weird page
kernel: page=ffffea0005320080 index=1120 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0005329d00 index=3932 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea00081a3dc0 index=4160 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea00068b8b00 index=5983 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0005df2a40 index=7264 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd1b40 index=3672 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd20c0 index=3673 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd43c0 index=3679 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)

FUSE library version: 2.9.7
fusermount version: 2.9.7
using FUSE kernel interface version 7.19

Linux 4.10.6 #1 SMP x86_64 GNU/Linux


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Antonio SJ Musumeci
Follow up: https://github.com/trapexit/mergerfs/issues/401#issuecomment-294158972

The problem seems to be with direct_io writes (NFS uses DIO) and splice_read with >=4096 byte writes example:

with splice_read:

FAILS while writing:

dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct oflag=direct bs=4096 count=10
dd: error writing '/storage/path2/testfile': Bad address
1+0 records in
0+0 records out

OK with <4096 blocksize:

dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct oflag=direct bs=4095 count=10
10+0 records in
10+0 records out

without splice_read:

OK:
dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct oflag=direct bs=4096 count=10
10+0 records in
10+0 records out

OK:
dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct oflag=direct bs=4095 count=10
10+0 records in
10+0 records out


On Wed, Apr 12, 2017 at 11:30 AM, Antonio SJ Musumeci <[hidden email]> wrote:

A user reported getting the below errors when *not* using direct_io but atomic_o_trunc, auto_cache, big_writes, default_permissions, splice_move, splice_read, and splice_write are enabled.

Any ideas?

kernel: fuse: trying to steal weird page
kernel: page=ffffea0005320080 index=1120 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0005329d00 index=3932 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea00081a3dc0 index=4160 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea00068b8b00 index=5983 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0005df2a40 index=7264 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd1b40 index=3672 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd20c0 index=3673 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)
kernel: fuse: trying to steal weird page
kernel: page=ffffea0004bd43c0 index=3679 flags=80000000000000ad, count=1, mapcount=0, mapping= (null)

FUSE library version: 2.9.7
fusermount version: 2.9.7
using FUSE kernel interface version 7.19

Linux 4.10.6 #1 SMP x86_64 GNU/Linux



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Miklos Szeredi
In reply to this post by Antonio SJ Musumeci
On Wed, Apr 12, 2017 at 11:30:00AM -0400, Antonio SJ Musumeci wrote:
> A user reported getting the below errors when *not* using direct_io
> but atomic_o_trunc, auto_cache, big_writes, default_permissions,
> splice_move, splice_read, and splice_write are enabled.
>
> Any ideas?

I think this is due to the PageWaiters bit added in v4.10 by

62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit")

That bit is harmless and probably left behind due to a race.  Following patch
should fix the warning.

Thanks,
Miklos

---
 fs/fuse/dev.c |    1 +
 1 file changed, 1 insertion(+)

--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -811,6 +811,7 @@ static int fuse_check_page(struct page *
        1 << PG_uptodate |
        1 << PG_lru |
        1 << PG_active |
+       1 << PG_waiters |
        1 << PG_reclaim))) {
  printk(KERN_WARNING "fuse: trying to steal weird page\n");
  printk(KERN_WARNING "  page=%p index=%li flags=%08lx, count=%i, mapcount=%i, mapping=%p\n", page, page->index, page->flags, page_count(page), page_mapcount(page), page->mapping);

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Miklos Szeredi
In reply to this post by Antonio SJ Musumeci
On Fri, Apr 14, 2017 at 4:37 PM, Antonio SJ Musumeci
<[hidden email]> wrote:

> Follow up:
> https://github.com/trapexit/mergerfs/issues/401#issuecomment-294158972
>
> The problem seems to be with direct_io writes (NFS uses DIO) and splice_read
> with >=4096 byte writes example:
>
> with splice_read:
>
> FAILS while writing:
>
> dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct
> oflag=direct bs=4096 count=10
> dd: error writing '/storage/path2/testfile': Bad address
> 1+0 records in
> 0+0 records out

And this one seems to be a different thing.  The page stealing happens
on the READ path while this happens on the WRITE path.  Would be good
to have a strace of the mergefs process for the failed case.

Thanks,
Miklos

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Antonio SJ Musumeci
https://github.com/trapexit/mergerfs/issues/401#issuecomment-297503800

When dd's buffersize is 4096 you'll see 

splice(6, NULL, 9, [0], 4096, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EFAULT (Bad address)
Which comes from write_buf.
But when it's 4095 you see it use pwrite used instead (falls back to plain read callback?).
The splice manpage doesn't even mention EFAULT.

On Thu, Apr 20, 2017 at 9:56 AM, Miklos Szeredi <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 4:37 PM, Antonio SJ Musumeci
<[hidden email]> wrote:
> Follow up:
> https://github.com/trapexit/mergerfs/issues/401#issuecomment-294158972
>
> The problem seems to be with direct_io writes (NFS uses DIO) and splice_read
> with >=4096 byte writes example:
>
> with splice_read:
>
> FAILS while writing:
>
> dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct
> oflag=direct bs=4096 count=10
> dd: error writing '/storage/path2/testfile': Bad address
> 1+0 records in
> 0+0 records out

And this one seems to be a different thing.  The page stealing happens
on the READ path while this happens on the WRITE path.  Would be good
to have a strace of the mergefs process for the failed case.

Thanks,
Miklos


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Antonio SJ Musumeci
The splice that's returning EFAULT is called from fuse_buf_copy.

On Wed, Apr 26, 2017 at 6:41 PM, Antonio SJ Musumeci <[hidden email]> wrote:
https://github.com/trapexit/mergerfs/issues/401#issuecomment-297503800

When dd's buffersize is 4096 you'll see 

splice(6, NULL, 9, [0], 4096, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EFAULT (Bad address)
Which comes from write_buf.
But when it's 4095 you see it use pwrite used instead (falls back to plain read callback?).
The splice manpage doesn't even mention EFAULT.

On Thu, Apr 20, 2017 at 9:56 AM, Miklos Szeredi <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 4:37 PM, Antonio SJ Musumeci
<[hidden email]> wrote:
> Follow up:
> https://github.com/trapexit/mergerfs/issues/401#issuecomment-294158972
>
> The problem seems to be with direct_io writes (NFS uses DIO) and splice_read
> with >=4096 byte writes example:
>
> with splice_read:
>
> FAILS while writing:
>
> dd if=/storage/path1/testfile of=/storage/path2/testfile iflag=direct
> oflag=direct bs=4096 count=10
> dd: error writing '/storage/path2/testfile': Bad address
> 1+0 records in
> 0+0 records out

And this one seems to be a different thing.  The page stealing happens
on the READ path while this happens on the WRITE path.  Would be good
to have a strace of the mergefs process for the failed case.

Thanks,
Miklos



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Miklos Szeredi
In reply to this post by Antonio SJ Musumeci
On Thu, Apr 27, 2017 at 12:41 AM, Antonio SJ Musumeci
<[hidden email]> wrote:

> https://github.com/trapexit/mergerfs/issues/401#issuecomment-297503800
>
> When dd's buffersize is 4096 you'll see
>
> splice(6, NULL, 9, [0], 4096, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EFAULT
> (Bad address)
>
> Which comes from write_buf.
>
> But when it's 4095 you see it use pwrite used instead (falls back to plain
> read callback?).
>
> The splice manpage doesn't even mention EFAULT.

Which kernel is this?

v4.9 had some splice related changes.  Does it happen with kernels
prior to v4.9?

Thanks,
Miklos

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Antonio SJ Musumeci
4.10.1

Ive asked them to try a pre 4.9 kernel.

On Apr 28, 2017 5:12 AM, "Miklos Szeredi" <[hidden email]> wrote:
On Thu, Apr 27, 2017 at 12:41 AM, Antonio SJ Musumeci
<[hidden email]> wrote:
> https://github.com/trapexit/mergerfs/issues/401#issuecomment-297503800
>
> When dd's buffersize is 4096 you'll see
>
> splice(6, NULL, 9, [0], 4096, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EFAULT
> (Bad address)
>
> Which comes from write_buf.
>
> But when it's 4095 you see it use pwrite used instead (falls back to plain
> read callback?).
>
> The splice manpage doesn't even mention EFAULT.

Which kernel is this?

v4.9 had some splice related changes.  Does it happen with kernels
prior to v4.9?

Thanks,
Miklos

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: trying to steal weird page?

Antonio SJ Musumeci

On Fri, Apr 28, 2017 at 6:54 AM, Antonio SJ Musumeci <[hidden email]> wrote:
4.10.1

Ive asked them to try a pre 4.9 kernel.

On Apr 28, 2017 5:12 AM, "Miklos Szeredi" <[hidden email]> wrote:
On Thu, Apr 27, 2017 at 12:41 AM, Antonio SJ Musumeci
<[hidden email]> wrote:
> https://github.com/trapexit/mergerfs/issues/401#issuecomment-297503800
>
> When dd's buffersize is 4096 you'll see
>
> splice(6, NULL, 9, [0], 4096, SPLICE_F_MOVE|SPLICE_F_NONBLOCK) = -1 EFAULT
> (Bad address)
>
> Which comes from write_buf.
>
> But when it's 4095 you see it use pwrite used instead (falls back to plain
> read callback?).
>
> The splice manpage doesn't even mention EFAULT.

Which kernel is this?

v4.9 had some splice related changes.  Does it happen with kernels
prior to v4.9?

Thanks,
Miklos


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel