Fuse_lowlevel debugging

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Fuse_lowlevel debugging

Joshua J. Berry
Miklos --

I noticed that you added a fuse_lowlevel API to CVS.  Thanks for this!  I've
started playing with an implementation for my project.  I obviously don't
expect anything to work right now, but I want to have my code ready to go by
the time fuse_lowlevel is ready to be released.

Are you interested in any debugging feedback (bug reports and the like), or is
it still too early in fuse_lowlevel's life for that?

Thanks again.

-- Josh

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Miklos Szeredi
> I noticed that you added a fuse_lowlevel API to CVS.  Thanks for
> this!  I've started playing with an implementation for my project.
> I obviously don't expect anything to work right now, but I want to
> have my code ready to go by the time fuse_lowlevel is ready to be
> released.
>
> Are you interested in any debugging feedback (bug reports and the
> like), or is it still too early in fuse_lowlevel's life for that?

Yes, any feedback is most welcome.  If you don't like something in the
API, please let me know.  Now is the time to get it right.

Thanks,
Miklos


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Joshua J. Berry
On Sunday 31 July 2005 01:27, Miklos Szeredi wrote:
> > Are you interested in any debugging feedback (bug reports and the
> > like), or is it still too early in fuse_lowlevel's life for that?
>
> Yes, any feedback is most welcome.  If you don't like something in the
> API, please let me know.  Now is the time to get it right.

The one weak spot/suggestion I have currently for the API is the *dir()
functions.  It was hard to figure out how to handle readdir(), specifically:

- What does "offset" mean? Is offset specified in bytes, or directory entries?

- What format should we use for the buffer?

I eventually got it figured out, but the interface seems very difficult to
use.

Perhaps it would be good to have a fuse_dirlist_t or something, and use
fuse_add_dirent() to append to that list (and handle memory allocation, etc).  
Then make a new fuse_reply_dirlist() for responding.  Also, make "offset" and
"size" be in terms of directory entries, not bytes.  So maybe something like
this:

struct fuse_dirlist_t { ... };

// Pass NULL initially to create a new dirlist
// Returns the dirlist with the item appended
fuse_dirlist_t * fuse_dirlist_add(fuse_dirlist_t * list, char * name, struct
stat * st, ...);
void fuse_dirlist_free(fuse_dirlist_t * list);
void fuse_reply_dirlist(fuse_req_t * req, fuse_dirlist_t * dirents);

Also, I noticed that the libfuse versionscript left off a few functions,
specifically fuse_ll_loop_mt() and fuse_req_ctx().  I had to add those to get
my code to link properly.

Hope this helps.

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Miklos Szeredi
> The one weak spot/suggestion I have currently for the API is the *dir()
> functions.  It was hard to figure out how to handle readdir(), specifically:
>
> - What does "offset" mean? Is offset specified in bytes, or
> - directory entries?

Either.  Or something else entirely.  The important thing is, that the
filesystem can find the next directory entry by what it filled in as
offset.

> - What format should we use for the buffer?

That's an internal detail of the FUSE interface, that the filesystem
should not need to know.

> I eventually got it figured out, but the interface seems very
> difficult to use.

Yes, leaving the memory management to the filesystem makes it a bit
awkward.  However it also let's the filesystem do caching and reusing
the raw directory data (the high level FUSE library uses that
feature).

> Perhaps it would be good to have a fuse_dirlist_t or something, and
> use fuse_add_dirent() to append to that list (and handle memory
> allocation, etc).  Then make a new fuse_reply_dirlist() for
> responding.  Also, make "offset" and "size" be in terms of directory
> entries, not bytes.  So maybe something like this:

Size cannot be the number of entries, since that is not known in
advance (the size of entries varies because of variable length
filenames).

> struct fuse_dirlist_t { ... };
>
> // Pass NULL initially to create a new dirlist
> // Returns the dirlist with the item appended
> fuse_dirlist_t * fuse_dirlist_add(fuse_dirlist_t * list, char * name, struct
> stat * st, ...);
> void fuse_dirlist_free(fuse_dirlist_t * list);
> void fuse_reply_dirlist(fuse_req_t * req, fuse_dirlist_t * dirents);

It would be easy to implement such an interface _on top_ of the
current one.  I'm not averse to that.

Also fuse_dirlist_add() needs to be passed the size parameter, and
return NULL when the buffer is filled.

> Also, I noticed that the libfuse versionscript left off a few
> functions, specifically fuse_ll_loop_mt() and fuse_req_ctx().  I had
> to add those to get my code to link properly.

Good find.  Added them.

> Hope this helps.

Thanks,
Miklos


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Joshua J. Berry
On Monday 01 August 2005 02:01, Miklos Szeredi wrote:

> > The one weak spot/suggestion I have currently for the API is the *dir()
> > functions.  It was hard to figure out how to handle readdir(),
> > specifically:
> >
> > - What does "offset" mean? Is offset specified in bytes, or
> > - directory entries?
>
> Either.  Or something else entirely.  The important thing is, that the
> filesystem can find the next directory entry by what it filled in as
> offset.
But isn't the kernel the one specifying the offset?

> Yes, leaving the memory management to the filesystem makes it a bit
> awkward.  However it also let's the filesystem do caching and reusing
> the raw directory data (the high level FUSE library uses that
> feature).

That makes a certain amount of sense.  I tend to cache things in a name->inode
map, though, because I find the caching also helps for handling lookups.  It
seems like it would be more useful, generally, to cache at a higher level.

> > Perhaps it would be good to have a fuse_dirlist_t or something, and
> > use fuse_add_dirent() to append to that list (and handle memory
> > allocation, etc).  Then make a new fuse_reply_dirlist() for
> > responding.  Also, make "offset" and "size" be in terms of directory
> > entries, not bytes.  So maybe something like this:
>
> Size cannot be the number of entries, since that is not known in
> advance (the size of entries varies because of variable length
> filenames).

I was thinking the kernel could request N entries, and then receive back a
total length (in bytes), because the filesystem knows how long each entry is.  
I guess this would make memory management in the kernel more complicated,
though.

Are there any other implementation barriers in the kernel (aside from the
memory management issues) to making "size" and "off" be in terms of entries
instead of bytes?

> > struct fuse_dirlist_t { ... };
> >
> > // Pass NULL initially to create a new dirlist
> > // Returns the dirlist with the item appended
> > fuse_dirlist_t * fuse_dirlist_add(fuse_dirlist_t * list, char * name,
> > struct stat * st, ...);
> > void fuse_dirlist_free(fuse_dirlist_t * list);
> > void fuse_reply_dirlist(fuse_req_t * req, fuse_dirlist_t * dirents);
>
> It would be easy to implement such an interface _on top_ of the
> current one.  I'm not averse to that.
>
> Also fuse_dirlist_add() needs to be passed the size parameter, and
> return NULL when the buffer is filled.
I was thinking fuse_dirlist would be a linked list whose memory was managed by
the library, thus no need to mess with sizes.  (Of course, having a
fuse_dirlist_t like that implies that the filesystem knows how many entries
to return, as opposed to how big a buffer to fill.)

-- Josh

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Valient Gough

On Monday 01 August 2005 19:21, Joshua J. Berry wrote:
> On Monday 01 August 2005 02:01, Miklos Szeredi wrote:
> > The one weak spot/suggestion I have currently for the
> > Either.  Or something else entirely.  The important thing is, that the
> > filesystem can find the next directory entry by what it filled in as
> > offset.
>
> But isn't the kernel the one specifying the offset?

The kernel is just passing back what was given to it by the filesystem.  I
don't think it interprets the offset value.  

I haven't played with the low-level interface yet, but at least in the
standard libfuse api, the offset is provided by the filesystem.  Whenever you
call the filldir function, you're providing the next offset.  That way if you
weren't able to return the entire directory contents at once (because the
buffer size has been reached), then the kernel can request starting at the
offset provided by the last entry.

> > Size cannot be the number of entries, since that is not known in
> > advance (the size of entries varies because of variable length
> > filenames).
>
> I was thinking the kernel could request N entries, and then receive back a
> total length (in bytes), because the filesystem knows how long each entry
is.  

But that would require two round-trips in order to get even a small listing.  
One to find out the size and the next to get the data.  And what would you
gain from this?

> Are there any other implementation barriers in the kernel (aside from the
> memory management issues) to making "size" and "off" be in terms of entries
> instead of bytes?

I think you misunderstand how offset is used -- it is already much more
flexable then you suspect.  The two are independent, the units for 'size' are
defined by the kernel, and the units for 'offset' are defined by the
filesystem.

For the offset, you can store any value you want - whatever is convenient.  If
you want to store the entry number into an array, that is fine.  For me, I
might want to store the off_t from telldir(3), because my filesystem builds
on top of an existing filesystem.  That way when I get a new request, I can
use seekdir() to seek to the correct location and continue reading entries.  
Or you could even store a pointer to the next element in a linked list.  Very
flexable.

Size is in bytes and is sent from the kernel.  In kernel/dir.c, fuse_readdir
seems to send PAGE_SIZE as the requested size, so one page worth of bytes are
being requested at once.

regards,
Valient



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. <a href="http://ads.osdn.com/?ad_idt77&alloc_id492&op=click">http://ads.osdn.com/?ad_idt77&alloc_id492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Joshua J. Berry
On Monday 01 August 2005 14:52, Valient Gough wrote:

> > But isn't the kernel the one specifying the offset?
>
> The kernel is just passing back what was given to it by the filesystem.  I
> don't think it interprets the offset value.
>
> I haven't played with the low-level interface yet, but at least in the
> standard libfuse api, the offset is provided by the filesystem.  Whenever
> you call the filldir function, you're providing the next offset.  That way
> if you weren't able to return the entire directory contents at once
> (because the buffer size has been reached), then the kernel can request
> starting at the offset provided by the last entry.
In the low-level interface, readdir() gets an offset and a size.  When
generating the response, you create a buffer and fill it using
fuse_add_dirent().  The reply is just the buffer, and the length of the
buffer (in bytes) using fuse_reply_buf().

I would assume that if a listing isn't big enough to fit into a buffer
of /size/ bytes, the kernel will issue another readdir() request starting at
offset(previous)+size(previous).

> > I was thinking the kernel could request N entries, and then receive back
> > a total length (in bytes), because the filesystem knows how long each
> > entry is.  
>
> But that would require two round-trips in order to get even a small
> listing. One to find out the size and the next to get the data.  And what
> would you gain from this?

I don't see how it would.  Userspace already sends back a buffer size.

> I think you misunderstand how offset is used -- it is already much more
> flexable then you suspect.  The two are independent, the units for 'size'
> are defined by the kernel, and the units for 'offset' are defined by the
> filesystem.
>
> For the offset, you can store any value you want - whatever is convenient.
> If you want to store the entry number into an array, that is fine.  For me,
> I might want to store the off_t from telldir(3), because my filesystem
> builds on top of an existing filesystem.  That way when I get a new
> request, I can use seekdir() to seek to the correct location and continue
> reading entries. Or you could even store a pointer to the next element in a
> linked list.  Very flexable.
How do I store an offset, though?  The only way for me to respond to the
kernel is to give it a buffer full of packed dirents, and a size.  Poking
through fuse_lowlevel.h, I don't see any way to give it an offset.

But yes, I think if I could set my own offset (or the "next expected offset",
or something similar), that would work well.

-- Josh

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Valient Gough
On Tuesday 02 August 2005 00:55, Joshua J. Berry wrote:
> How do I store an offset, though?  The only way for me to respond to the
> kernel is to give it a buffer full of packed dirents, and a size.  Poking
> through fuse_lowlevel.h, I don't see any way to give it an offset.

When you add dirent entries, you use fuse_add_dirent().  The last argument is
"off_t off".  That is the offset.

Valient


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Joshua J. Berry
On Monday 01 August 2005 16:36, Valient Gough wrote:
> On Tuesday 02 August 2005 00:55, Joshua J. Berry wrote:
> > How do I store an offset, though?  The only way for me to respond to the
> > kernel is to give it a buffer full of packed dirents, and a size.  Poking
> > through fuse_lowlevel.h, I don't see any way to give it an offset.
>
> When you add dirent entries, you use fuse_add_dirent().  The last argument
> is "off_t off".  That is the offset.

D'oh.  I see it now.  I was also looking at my code, and I just stick the
current buffer size in that parameter, which explains why I was confused.

::adds a long comment talking about offsets to that part of his code::

Thanks for pointing it out to me.

-- Josh

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Joshua J. Berry
In reply to this post by Miklos Szeredi
On Monday 01 August 2005 02:01, Miklos Szeredi wrote:
> > Also, I noticed that the libfuse versionscript left off a few
> > functions, specifically fuse_ll_loop_mt() and fuse_req_ctx().  I had
> > to add those to get my code to link properly.
>
> Good find.  Added them.

Spelling error -- fuse_ll_loop_mNt :)

Thanks.

-- Josh

--
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuse_lowlevel debugging

Miklos Szeredi
> > > Also, I noticed that the libfuse versionscript left off a few
> > > functions, specifically fuse_ll_loop_mt() and fuse_req_ctx().  I had
> > > to add those to get my code to link properly.
> >
> > Good find.  Added them.
>
> Spelling error -- fuse_ll_loop_mNt :)

My fingers seem to think think 'mnt' makes more sense than 'mt' :)

Thanks,
Miklos



-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel