Fwd: ThroughPut Per Process Is Very Less using during the Rewrite and Random Rewrite using the Fuse File System

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: ThroughPut Per Process Is Very Less using during the Rewrite and Random Rewrite using the Fuse File System

Akil Raza
Hi
I am benchmarking the Read Write Performance on the Fuse File System using Fuse 3.0 Library,Since we are using FUSE file system  
FUSE Is  mounted with the Below Option
-o auto_cache,modules=subdir,subdir=/ /mnt/fuse
File System is Initialize using following values
conn->capable |= FUSE_CAP_ASYNC_READ;
conn->want |= FUSE_CAP_WRITEBACK_CACHE;
cfg->nullpath_ok = 1;

Below is the Machine Memory Configuration
[root@labcentos7base ~]# free -mh
              total        used        free      shared  buff/cache   available
Mem:           125G        1.2G        107G         63M         17G        123G
Swap:          4.0G        8.5M        4.0G
And Kernel Is 
[root@labcentos7base ~]# uname -a
Linux labcentos7base.localdomain 3.19.8 #1 SMP Wed Feb 1 01:54:31 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Iozone is used for Performance Testing  and below command is Executed
[root@labcentos7base clearzone]# iozone -i 0 -i 1  -i 2 -r 128k -s 8g -t 1 -b output_clear_8g_1t.xls

During Random Write and Rewrite throughput Observed is Very Less 
Below is the Stack Trace Taken During the Random Write 
[root@labcentos7base ~]# gstack 32703( Fuse Process Id )
Thread 11 (Thread 0x7f8da6ffd700 (LWP 4564)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d98000900, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d980008e0) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f8d827fc700 (LWP 4573)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d8c000960, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d8c000940) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f8da77fe700 (LWP 4586)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d980009c0, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d980009a0) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f8da57fa700 (LWP 4588)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d98001180, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d98001160) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f8da7fff700 (LWP 4590)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d98000ce0, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d98000cc0) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f8d837fe700 (LWP 4591)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d84000960, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d84000940) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f8da67fc700 (LWP 4592)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d78000900, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d780008e0) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f8dbfd6f700 (LWP 4593)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8da0000900, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8da00008e0) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f8dbf56e700 (LWP 4594)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d78000a30, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d78000a10) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f8da4ff9700 (LWP 4595)):
#0  0x00007f8dbfe68043 in splice () from /lib64/libc.so.6
#1  0x00007f8dc097df00 in fuse_session_receive_buf_int (se=0x2151810, buf=buf@entry=0x7f8d78000a90, ch=0x0) at fuse_lowlevel.c:2649
#2  0x00007f8dc09791cf in fuse_do_work (data=0x7f8d78000a70) at fuse_loop_mt.c:123
#3  0x00007f8dc0ba7dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8dbfe6773d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f8dc11e5740 (LWP 32703)):
#0  0x00007f8dc0bad79b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007f8dc0bad82f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f8dc0bad8cb in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f8dc0979418 in fuse_session_loop_mt (se=0x2151810, clone_fd=clone_fd@entry=0) at fuse_loop_mt.c:326
#4  0x00007f8dc097813b in fuse_loop_mt (f=f@entry=0x21511f0, clone_fd=0) at fuse.c:4337
#5  0x00007f8dc09803f7 in fuse_main_real (argc=<optimized out>, argv=<optimized out>, op=<optimized out>, op_size=<optimized out>, user_data=0x0) at helper.c:323
#6  0x000000000044d7f9 in main (argc=4, argv=0x7ffcdfb88548) at /home/workspace/fuse7.0_3.0Lib/code/internal/services/fusefs_server/dfp_fusefs_server.c:2463
[root@labcentos7base ~]#

From Above Trace I could See  the call is Block in Splice in fuse_low_level.c
The ThroughPut goes down Very Drastically. Below is the thoughput when iozone is executed on Fuse File system
        Children see throughput for  1 initial writers  =  164199.11 kB/sec
        Parent sees throughput for  1 initial writers   =  103580.18 kB/sec
        Min throughput per process                      =  164199.11 kB/sec
        Max throughput per process                      =  164199.11 kB/sec
        Avg throughput per process                      =  164199.11 kB/sec
        Min xfer                                        = 8388608.00 kB

        Children see throughput for  1 rewriters        =   33772.88 kB/sec
        Parent sees throughput for  1 rewriters         =   30425.74 kB/sec
        Min throughput per process                      =   33772.88 kB/sec
        Max throughput per process                      =   33772.88 kB/sec
        Avg throughput per process                      =   33772.88 kB/sec
        Min xfer                                        = 8388608.00 kB

        Children see throughput for  1 readers          =  494356.16 kB/sec
        Parent sees throughput for  1 readers           =  494313.07 kB/sec
        Min throughput per process                      =  494356.16 kB/sec
        Max throughput per process                      =  494356.16 kB/sec
        Avg throughput per process                      =  494356.16 kB/sec
        Min xfer                                        = 8388608.00 kB

        Children see throughput for 1 re-readers        = 2453998.25 kB/sec
        Parent sees throughput for 1 re-readers         = 2453210.38 kB/sec
        Min throughput per process                      = 2453998.25 kB/sec
        Max throughput per process                      = 2453998.25 kB/sec
        Avg throughput per process                      = 2453998.25 kB/sec
        Min xfer                                        = 8388608.00 kB

        Children see throughput for 1 random readers    = 2792735.00 kB/sec
        Parent sees throughput for 1 random readers     = 2791633.74 kB/sec
        Min throughput per process                      = 2792735.00 kB/sec
        Max throughput per process                      = 2792735.00 kB/sec
        Avg throughput per process                      = 2792735.00 kB/sec
        Min xfer                                        = 8388608.00 kB
^[[B
        Children see throughput for 1 random writers    =    2064.40 kB/sec
        Parent sees throughput for 1 random writers     =    2048.10 kB/sec
        Min throughput per process                      =    2064.40 kB/sec
        Max throughput per process                      =    2064.40 kB/sec
        Avg throughput per process                      =    2064.40 kB/sec
        Min xfer                                        = 8388608.00 kB
Random Writer throughPut is drastically very low . 
Can anyone please provide some suggestion for Improving the Throughput
Thr ough Put Is much better with the  fuse 2.9 Lib

--
Potentially Secured from threat!!!



--
Potentially Secured from threat!!!

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Loading...