Transmission(bittorrent client) interaction with a FUSE filesystem

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Transmission(bittorrent client) interaction with a FUSE filesystem

Constantin Kulikov
The problem is described here:
https://github.com/transmission/transmission/issues/120

Hello, I'm trying to use transmission to download a torrent onto custom FUSE filesystem(I use fusepy) which actually saves data on a real filesystem but with different path.
It works in general, but I get a high number of 'corrupt' data statistics(a half or more of the downloaded data is 'corrupt')
https://my.mixtape.moe/xmlwyx.jpg

What can I do about that?

I guess that python + fuse is rather slow may it be the cause of this?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Antonio SJ Musumeci
I've never done a comparison with regards to the % corrupt but my passthrough filesystem mergerfs is used by myself and others with Transmission without obvious issues.

What version of Transmission? Could you try using mergerfs and see if it performs similarly? What are your FUSE settings?

On Mon, Dec 19, 2016 at 8:45 AM, Constantin Kulikov <[hidden email]> wrote:
The problem is described here:
https://github.com/transmission/transmission/issues/120

Hello, I'm trying to use transmission to download a torrent onto custom FUSE filesystem(I use fusepy) which actually saves data on a real filesystem but with different path.
It works in general, but I get a high number of 'corrupt' data statistics(a half or more of the downloaded data is 'corrupt')
https://my.mixtape.moe/xmlwyx.jpg

What can I do about that?

I guess that python + fuse is rather slow may it be the cause of this?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Constantin Kulikov
I found that the problem is caused by the bad file handle sometimes passed as input to read/write FUSE callbacks(and maybe to flush, fsync and release too).
Most of the time the file descriptors are vallid, but sometimes it is not. In the python code I worked around it like this:

class MyFS(Operations):
    ...
    def write(self, path, buf, offset, fh):
        with self.rwlock:
            try:
                os.lseek(fh, offset, os.SEEK_SET)
                return os.write(fh, buf)
            except:
                fh=os.open(path, os.O_WRONLY)
                os.lseek(fh, offset, os.SEEK_SET)
                ret=os.write(fh, buf)
                os.close(fh)
                return ret
    ...

And it seem to work, however I'm not sure it is the right way to handle this.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Jean-Pierre André
Constantin Kulikov wrote:
> I found that the problem is caused by the bad file handle sometimes
> passed as input to read/write FUSE callbacks(and maybe to flush, fsync
> and release too).

AFAIK torrent-type transfers are multithreaded, so
couples of lseek()+write() may get intertwined, and
data end up being written at a wrong location.

Maybe you need to use pwrite() or enforce serialization
at the python level.

> Most of the time the file descriptors are vallid, but sometimes it is
> not. In the python code I worked around it like this:
>
>     class MyFS(Operations):
>          ...
>          def write(self, path, buf, offset, fh):
>              with self.rwlock:
>                  try:
>                      os.lseek(fh, offset, os.SEEK_SET)
>                      return os.write(fh, buf)
>                  except:
>                      fh=os.open(path, os.O_WRONLY)
>                      os.lseek(fh, offset, os.SEEK_SET)
>                      ret=os.write(fh, buf)
>                      os.close(fh)
>                      return ret
>          ...
>
>
> And it seem to work, however I'm not sure it is the right way to handle
> this.
>




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Antonio SJ Musumeci
What would matter here is whether or not the FUSE app is multithreaded and if it is then you definitely need to use pread and pwrite. Given the corruption I suspect it's a bunch of race conditions screwing up write locations.

What is it that you're doing exactly with your filesystem?

On Tue, Dec 20, 2016 at 2:37 AM, Jean-Pierre André <[hidden email]> wrote:
Constantin Kulikov wrote:
> I found that the problem is caused by the bad file handle sometimes
> passed as input to read/write FUSE callbacks(and maybe to flush, fsync
> and release too).

AFAIK torrent-type transfers are multithreaded, so
couples of lseek()+write() may get intertwined, and
data end up being written at a wrong location.

Maybe you need to use pwrite() or enforce serialization
at the python level.

> Most of the time the file descriptors are vallid, but sometimes it is
> not. In the python code I worked around it like this:
>
>     class MyFS(Operations):
>          ...
>          def write(self, path, buf, offset, fh):
>              with self.rwlock:
>                  try:
>                      os.lseek(fh, offset, os.SEEK_SET)
>                      return os.write(fh, buf)
>                  except:
>                      fh=os.open(path, os.O_WRONLY)
>                      os.lseek(fh, offset, os.SEEK_SET)
>                      ret=os.write(fh, buf)
>                      os.close(fh)
>                      return ret
>          ...
>
>
> And it seem to work, however I'm not sure it is the right way to handle
> this.
>




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Constantin Kulikov
What is it that you're doing exactly with your filesystem?

Nothing special for now, just call standard functions from python's os module, like os.open, os.read, os.write, etc.

in foreground + debug mode reports that all operations was successful but transmission shows high number of corruption, no matter if launch xmp.py in single thread mode or not.
Adding a lock(from python's threading module) around read and write operations does not change anything.

reports about bad file descriptors in the read operation randomly. And it seems like other operations always get a good file descriptor.
So with fusepy I only need to catch exceptions in read operation and everything seem to work fine:

class MyFS(FUSE.Operations)
    ...
    def read(self, path, length, offset, fh):
        with self.rwlock:
            try:
                os.lseek(fh, offset, os.SEEK_SET)
                ret = os.read(fh, length)
                #print "Good read file descriptor: %s %s" % (fh, path)
                return ret
            except Exception as e:
                #print "Wrong read file descriptor: %s %s %s" % (fh, path, e)
                fh=os.open(path, os.O_RDONLY)
                os.lseek(fh, offset, os.SEEK_SET)
                ret=os.read(fh, length)
                os.close(fh)
                return ret
    ...


Very strange.)
I run all of this on Ubuntu 14.04.5 LTS and fusermount version: 2.9.2.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transmission(bittorrent client) interaction with a FUSE filesystem

Nikolaus Rath
On Dec 21 2016, Constantin Kulikov <[hidden email]> wrote:
>>
>> What is it that you're doing exactly with your filesystem?
>>
>
> Nothing special for now, just call standard functions from python's os
> module, like os.open, os.read, os.write, etc.
>
> The python-fuse example
> https://github.com/libfuse/python-fuse/blob/master/example/xmp.py
[...]
> The fusepy example
> https://github.com/terencehonles/fusepy/blob/master/examples/loopback.py
[...]


If you can, I'd recommend to use python-llfuse instead
(http://pythonhosted.org/llfuse/).

python-fuse has been unmaintained for years, and when I last looked at
fusepy (at that point still maintained by someone else) I did not like
the technical shortcuts it takes.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Loading...