fstat returns incorrect values for hard-links after chown

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

fstat returns incorrect values for hard-links after chown

Denis Mikhalkin
Hello,

I'm developing a file system which uses AWS DynamoDB as a storage medium (https://github.com/denismo/dynamo-fuse). It's been ok so far - it passes most of the fstests, however some tests show bizarre failures that I cannot attribute to my implementation. After several days of investigation first in fusepy, then in fuselib, I came to a conclusion that it can only be explained by an error in the fuse kernel driver. As this is way beyond my understanding I hope to get some help in resolving the issue. I'm using Ubuntu 12.10.

Most of the failing tests are the ones which work with hard links, and the nature of failures is similar - either the link count, or some attributes are not matching among several hard links to a single file or fifo.

Here is a sequence of tests that is failing:
expect 0 create ${n0} 0644

>expect regular,0644,1 lstat ${n0} type,mode,nlink
>
>
>expect 0 link ${n0} ${n1}
>expect regular,0644,2 lstat ${n0} type,mode,nlink
>expect regular,0644,2 lstat ${n1} type,mode,nlink
>
>
>expect 0 link ${n1} ${n2}
>#sync
># 8
>expect regular,0644,3 lstat ${n0} type,mode,nlink
>expect regular,0644,3 lstat ${n1} type,mode,nlink
>expect regular,0644,3 lstat ${n2} type,mode,nlink
>
>
>expect 0 chmod ${n1} 0201
>expect 0 chown ${n1} 65534 65533
>
>
># 13
>expect regular,0201,3,65534,65533 lstat ${n0} type,mode,nlink,uid,gid
>expect regular,0201,3,65534,65533 lstat ${n1} type,mode,nlink,uid,gid
>expect regular,0201,3,65534,65533 lstat ${n2} type,mode,nlink,uid,gid
>
>
>expect 0 unlink ${n0}
>
>
The tests start failing at the test #13 and #15 - the stats don't match. The bizarre thing is that if I uncomment the "#sync" line the tests pass. Adding simple "sleep" does not help.

I've enabled the debug log for fuselib (attached) and I can clearly see that the stat calls after "chown" are missing - they don't reach the fuse lib and my module. So that's why I concluded that the error is most likely in the kernel module - perhaps to do with caching? Here is the list of flags I'm using to initialize fuse: foreground=True, nothreads=True, default_permissions=False, debug=True, auto_cache=False, noauto_cache=True, kernel_cache=False, direct_io=True, allow_other=True, use_ino=True. I tried to disable all caching - for this network filesystem the kernel cannot cache anything because the underlying data may change without notifications.

Here is the snippet from the log which shows missing calls:
unique: 23, opcode: SETATTR (4), nodeid: 4, insize: 128, pid: 28865

>getattr /fstest_53d9357cbb50da6582322e9b5c1bbdb6/fstest_1c0bcf88a8fb5f673db14d33a5aaabf4
>   unique: 23, success, outsize: 120
>unique: 24, opcode: GETXATTR (22), nodeid: 4, insize: 68, pid: 28871
>getxattr /fstest_53d9357cbb50da6582322e9b5c1bbdb6/fstest_1c0bcf88a8fb5f673db14d33a5aaabf4 security.capability 0
>   unique: 24, error: -95 (Operation not supported), outsize: 16
>unique: 25, opcode: SETATTR (4), nodeid: 4, insize: 128, pid: 28871
>chown /fstest_53d9357cbb50da6582322e9b5c1bbdb6/fstest_1c0bcf88a8fb5f673db14d33a5aaabf4 65534 65533
>getattr /fstest_53d9357cbb50da6582322e9b5c1bbdb6/fstest_1c0bcf88a8fb5f673db14d33a5aaabf4
>   unique: 25, success, outsize: 120
>unique: 26, opcode: UNLINK (10), nodeid: 2, insize: 80, pid: 28897
>unlink /fstest_53d9357cbb50da6582322e9b5c1bbdb6/fstest_ffb653a2747aab1ee22ad19b3d16335f
As you can see, the second setattr (which corresponds to "chown") is immediately followed by "unlink". Another attachment is manual execution of the same commands - everything works fine then, and you can see a number of LOOKUP requests after "chown".

I hope you can spot a problem. I wonder if there is a simple workaround of some kernel flags that might prevent caching or trigger flush.

Thanks for your time.

Denis
------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel

not_working_link.txt (8K) Download Attachment
working_link.txt (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: fstat returns incorrect values for hard-links after chown

Jean-Pierre André
Denis Mikhalkin wrote:
> Hello,
>
> I'm developing a file system which uses AWS DynamoDB as a storage medium (https://github.com/denismo/dynamo-fuse). It's been ok so far - it passes most of the fstests, however some tests show bizarre failures that I cannot attribute to my implementation. After several days of investigation first in fusepy, then in fuselib, I came to a conclusion that it can only be explained by an error in the fuse kernel driver. As this is way beyond my understanding I hope to get some help in resolving the issue. I'm using Ubuntu 12.10.
>
> Most of the failing tests are the ones which work with hard links, and the nature of failures is similar - either the link count, or some attributes are not matching among several hard links to a single file or fifo.

This an old issue : when it comes to cacheing attributes,
the high-level interface to libfuse considers a file which
has several hard-linked names as several different files.

> I hope you can spot a problem. I wonder if there is a simple workaround of some kernel flags that might prevent caching or trigger flush.

The traditional workaround is either to prevent fuse from
cacheing attributes, by using the option "attr_timeout=0"
or by using the low-level interface. On the low-level
interface a file is designated by its inode number, so
all hard-linked names lead to the same attributes.



------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel