How to tailor FUSE to support big file read/write?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to tailor FUSE to support big file read/write?

Ying LEE

There is a related option, as the following:
" -o big_writes enable larger than 4kB writes\n"

I am developing a fuse application. In my situation, the files are always big and clients prefer performance. So I am thinking to use 1M as a size unit to read/write.

I am working out the way to tailor libfuse options, even source code, to my situation.

But I couldn't find other related options, and I checked the source code, but I had no idea where to change.

Can you show me a direction?

Thank you in advance,
Ying

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------

--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to tailor FUSE to support big file read/write?

Bharath Kumar Reddy Vangoor
Hey Ying,

Following options available in FUSE to increase the size of writes/reads to happen :

1. -o read_ahead=<val> : where <val> is the max. amount to read in single FUSE_READ request, by default this is ON an and value is set to 128 KB. 

2. -o big_writes , -o max_write=<val> : where <val> is the max. amount to write in single FUSE_WRITE request, by default this is OFF and every write happens in 4 KB I/O

3. -o writeback_cache, -o max_write=<val> : This is to allow writes to be happen asynchronously, <val> is same as previous case.

However, Currently there is a limit in FUSE for <val> which is 32 pages i.e, 128 KB. If you want to increase that you need to make some minor changes to kernel code and user library.

Also, you can use ``splice'' (zero copy) option which helps the performance of writes and reads.

Thanks,
Bharath.

On Thu, Sep 22, 2016 at 11:37 PM, Ying LEE <[hidden email]> wrote:

There is a related option, as the following:
" -o big_writes enable larger than 4kB writes\n"

I am developing a fuse application. In my situation, the files are always big and clients prefer performance. So I am thinking to use 1M as a size unit to read/write.

I am working out the way to tailor libfuse options, even source code, to my situation.

But I couldn't find other related options, and I checked the source code, but I had no idea where to change.

Can you show me a direction?

Thank you in advance,
Ying

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------

--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel



------------------------------------------------------------------------------

--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to tailor FUSE to support big file read/write?

Ying LEE

Awesome, thanks!


On Thu, Sep 29, 2016, 07:49 Bharath Kumar Reddy Vangoor <[hidden email]> wrote:
Hey Ying,

Following options available in FUSE to increase the size of writes/reads to happen :

1. -o read_ahead=<val> : where <val> is the max. amount to read in single FUSE_READ request, by default this is ON an and value is set to 128 KB. 

2. -o big_writes , -o max_write=<val> : where <val> is the max. amount to write in single FUSE_WRITE request, by default this is OFF and every write happens in 4 KB I/O

3. -o writeback_cache, -o max_write=<val> : This is to allow writes to be happen asynchronously, <val> is same as previous case.

However, Currently there is a limit in FUSE for <val> which is 32 pages i.e, 128 KB. If you want to increase that you need to make some minor changes to kernel code and user library.

Also, you can use ``splice'' (zero copy) option which helps the performance of writes and reads.

Thanks,
Bharath.

On Thu, Sep 22, 2016 at 11:37 PM, Ying LEE <[hidden email]> wrote:

There is a related option, as the following:
" -o big_writes enable larger than 4kB writes\n"

I am developing a fuse application. In my situation, the files are always big and clients prefer performance. So I am thinking to use 1M as a size unit to read/write.

I am working out the way to tailor libfuse options, even source code, to my situation.

But I couldn't find other related options, and I checked the source code, but I had no idea where to change.

Can you show me a direction?

Thank you in advance,
Ying

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------


--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------

--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to tailor FUSE to support big file read/write?

Ying LEE
After a few tests, I found there is a limit of 65536 bytes for write buffer and a limit of 131072 bytes for read buffer in version 2.9.7.

I tried to modify source code of libfuse, after 2 days struggle I gave up. The logical is complicated.

I wish one day, libfuse will remove the buffer limit or at least make it much bigger.

Thank you in advance,
Ying



On Thu, Sep 29, 2016 at 9:47 AM Ying LEE <[hidden email]> wrote:

Awesome, thanks!


On Thu, Sep 29, 2016, 07:49 Bharath Kumar Reddy Vangoor <[hidden email]> wrote:
Hey Ying,

Following options available in FUSE to increase the size of writes/reads to happen :

1. -o read_ahead=<val> : where <val> is the max. amount to read in single FUSE_READ request, by default this is ON an and value is set to 128 KB. 

2. -o big_writes , -o max_write=<val> : where <val> is the max. amount to write in single FUSE_WRITE request, by default this is OFF and every write happens in 4 KB I/O

3. -o writeback_cache, -o max_write=<val> : This is to allow writes to be happen asynchronously, <val> is same as previous case.

However, Currently there is a limit in FUSE for <val> which is 32 pages i.e, 128 KB. If you want to increase that you need to make some minor changes to kernel code and user library.

Also, you can use ``splice'' (zero copy) option which helps the performance of writes and reads.

Thanks,
Bharath.

On Thu, Sep 22, 2016 at 11:37 PM, Ying LEE <[hidden email]> wrote:

There is a related option, as the following:
" -o big_writes enable larger than 4kB writes\n"

I am developing a fuse application. In my situation, the files are always big and clients prefer performance. So I am thinking to use 1M as a size unit to read/write.

I am working out the way to tailor libfuse options, even source code, to my situation.

But I couldn't find other related options, and I checked the source code, but I had no idea where to change.

Can you show me a direction?

Thank you in advance,
Ying

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------


--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel

--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.
--

Regards,

Mr. Ying LEE
----------------------------------------------------------
Remember. Let go. Move on.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to tailor FUSE to support big file read/write?

Nikolaus Rath
On Oct 10 2016, Ying LEE <[hidden email]> wrote:
> After a few tests, I found there is a limit of 65536 bytes for write buffer
> and a limit of 131072 bytes for read buffer in version 2.9.7.
>
> I tried to modify source code of libfuse, after 2 days struggle I gave up.
> The logical is complicated.
>
> I wish one day, libfuse will remove the buffer limit or at least make it
> much bigger.

As far as I know, the limit is not in libfuse, it's in the kernel. So
it's no wonder that your changes to libfuses didn't work out.

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel