[PATCH] EINVAL when listing large directories

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] EINVAL when listing large directories

Frederik Deweerdt-3
Hi,

I think there's a bug in dev.c that prevents large requests
from being done. I was getting -EINVAL values when issuing
a "ls -1" on a directory hosting 1458 files. This patch
fixed the problem for me. Do you want a patch for -mm btw?

Regards,
Frederik Deweerdt

Index: kernel/dev.c
===================================================================
RCS file: /cvsroot/fuse/fuse/kernel/dev.c,v
retrieving revision 1.76
diff -u -r1.76 dev.c
--- kernel/dev.c        1 Aug 2005 10:44:45 -0000       1.76
+++ kernel/dev.c        2 Aug 2005 17:45:45 -0000
@@ -748,7 +748,7 @@
 
        reqsize += len_args(out->numargs, out->args);
 
-       if (reqsize < nbytes || (reqsize > nbytes && !out->argvar))
+       if (reqsize < nbytes && !out->argvar)
                return -EINVAL;
        else if (reqsize > nbytes) {
                struct fuse_arg *lastarg = &out->args[out->numargs-1];

--
o---------------------------------------------o
| http://open-news.net : l'info alternative   |
| Tech - Sciences - Politique - International |
o---------------------------------------------o


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Miklos Szeredi
> I think there's a bug in dev.c that prevents large requests
> from being done. I was getting -EINVAL values when issuing
> a "ls -1" on a directory hosting 1458 files. This patch
> fixed the problem for me. Do you want a patch for -mm btw?

Well, the code is tricky, but I think is right.

        if (reqsize < nbytes || (reqsize > nbytes && !out->argvar))
                return -EINVAL;

For variable length replies (such as readdir), reqsize means the
_maximum_ size of the request.  So the condition reads:

IF nbytes is more than the maximum request size OR
   (nbytes is less than the maximum request size AND
    last argument is not variable length)  
THEN return -EINVAL

So I think in this case your filesystem or libfuse is doing something
wrong, by returning a larger buffer, than the requested size.

Miklos


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Frederik Deweerdt-3
Le 02/08/05 20:15 +0200, Miklos Szeredi écrivit:
> Well, the code is tricky, but I think is right.
>
Indeed, I misunderstood it, thanks.
>
> So I think in this case your filesystem or libfuse is doing something
> wrong, by returning a larger buffer, than the requested size.
I'm using Sulf (http://arg0.net/users/vgough/sulf/index.html), the problem
may lie in its replacement for libfuse.

Regards,
Frederik

--
o---------------------------------------------o
| http://open-news.net : l'info alternative   |
| Tech - Sciences - Politique - International |
o---------------------------------------------o


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Valient Gough
On Wednesday 03 August 2005 09:23, Frederik Deweerdt wrote:
> Le 02/08/05 20:15 +0200, Miklos Szeredi écrivit:

> > So I think in this case your filesystem or libfuse is doing something
> > wrong, by returning a larger buffer, than the requested size.
>
> I'm using Sulf (http://arg0.net/users/vgough/sulf/index.html), the problem
> may lie in its replacement for libfuse.

Ah ha, that would be my fault then.

The small attached patch may help, although you're still going to have to
modify the ReadDir method of your directory node.  The patch makes the fill
function actually report when it is full, rather then just continuing to
expand.  I still need to make the users (like MapFS, etc) actually stop
writing when the buffer is full and start from the correct offset next time.

When I had tested with large directories and no size limit, it had seemed to
work, but my 'large' directory was only a couple entries larger then a single
PAGE_SIZE buffer that fuse.ko had requested.

I'll take a look at the ReadDir methods this afternoon.

regards,
Valient

sulf-limit-readdir.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Frederik Deweerdt-3
Le 03/08/05 11:00 +0200, Valient Gough écrivit:
> On Wednesday 03 August 2005 09:23, Frederik Deweerdt wrote:
> > Le 02/08/05 20:15 +0200, Miklos Szeredi écrivit:
>
> I'll take a look at the ReadDir methods this afternoon.
OK Thanks, the patch fixes the problem however.

Regards,
Fred


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Valient Gough
On Wednesday 03 August 2005 12:16, Frederik Deweerdt wrote:
> Le 03/08/05 11:00 +0200, Valient Gough écrivit:
> > I'll take a look at the ReadDir methods this afternoon.
>
> OK Thanks, the patch fixes the problem however.

I've committed patches to fix MapFS and PassthroughDirNode to make use of the
offset and return value from the dirent packer.  You can get them from
revision control.  Or download the patches from
http://arg0.net/vgough/devel/darcsweb/

thanks,
Valient


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] EINVAL when listing large directories

Frederik Deweerdt-3
Le 03/08/05 15:47 +0200, Valient Gough écrivit:

> On Wednesday 03 August 2005 12:16, Frederik Deweerdt wrote:
> > Le 03/08/05 11:00 +0200, Valient Gough écrivit:
> > > I'll take a look at the ReadDir methods this afternoon.
> >
> > OK Thanks, the patch fixes the problem however.
>
> I've committed patches to fix MapFS and PassthroughDirNode to make use of the
> offset and return value from the dirent packer.  You can get them from
> revision control.  Or download the patches from
> http://arg0.net/vgough/devel/darcsweb/
>
PassthruDirNode passes now happily thru big directories, cool!
FYI, I've needed these extra steps to compile successfully.

automake -ca
libtoolize --copy --force
chmod +x FuseWrapper/typedef-replace
chmod +x FuseWrapper/autotypemap
chmod +x FuseWrapper/extract-other

Regards,
Frederik
--
o---------------------------------------------o
| http://open-news.net : l'info alternative   |
| Tech - Sciences - Politique - International |
o---------------------------------------------o


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel