Fuse filesystem - Fragmentation

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fuse filesystem - Fragmentation

Thibaut A
Thank you for the answer !

Yes, I kind of know what you mean but the subject is pretty vague about what we've got to do .
"You have to write a fuse module that will implement a simple filesystem, directly using a raw block device, like a partition such as /dev/sdb1.
Your module must maintain the filesystem structure itself, directly using a block device. Feel free to get inspiration from existing filesystems such as FAT, ext, or even Minix Filesystem.
Your filesystem should try it's best to avoid fragmentation." 

Have you got any ideas about what to do ?

Best regards,

Thibaut 

On May 30 2016, Thibaut A <thibaut.arrigoni-Re5JQEeQqe8AvxtiuMwx3w@...> wrote:
> We’ve got a school’s project about fuse. One of the few things asked is to
> create a fuse module that will implement a simple filesystem. It has to
> support some basics operations like Read/Write/Dele files,
> Create/Delete/rename directories…
> But, there is one point that seems weird and I don’t know at all how to
> proceed :
> The filesystem has to store files in an efficient way that avoids
> fragmentation.
>
> I’ve searched around the web but I didn’t get any interesting results.
> That’s why I’m writing to you.
>
> Could you help us with the problems ? Do you have any idea how to do
> that ?

How/where are the files supposed to be stored? Fragmentation occurs if
you store files at different positions on a continuous storage medium,
and it's only a problem if seeking on that medium is expensive.

If you simply store the files inside another file system, fragmentation
is not a concept that you can apply at your layer.


Best,
Nikolaus


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Nikolaus Rath
On May 31 2016, Thibaut A <[hidden email]> wrote:

> Thank you for the answer !
>
> Yes, I kind of know what you mean but the subject is pretty vague about
> what we've got to do .
> "You have to write a fuse module that will implement a simple filesystem,
> directly using a raw block device, like a partition such as /dev/sdb1.
> Your module must maintain the filesystem structure itself, directly using a
> block device. Feel free to get inspiration from existing filesystems such
> as FAT, ext, or even Minix Filesystem.
> Your filesystem should try it's best to avoid fragmentation."
>
> Have you got any ideas about what to do ?
>

I think the instructions are pretty clear. Which part of them do you not
understand? What have you tried so far? Would you be able to solve this
if it did not mention fragmentation? Then do that first, and later worry
about fragmentation.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Thibaut A
In reply to this post by Thibaut A
So far, I've understood I have to create a fuse Filesystem to "mirror" a raw device like /dev/sdb1.
In order to do that, I got inspired by the fusexmp.c file in the libfuse git repository.
I've created a directory in /tmp where I can execute the fuse executable. After this, I can see all the files and directories from my /dev/sda1 and I can "play" with them.
I can create/delete/move files and create/delete/move directories in this filesystem.
When I'm done, I just have to unmount my recently mounted filesystem.
Am I in the right so far ?
About the defragmentation part, I don't get in which direction that want us to go.

Best regards, 

Thibaut



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Carlos Maiolino
On Wed, Jun 01, 2016 at 12:17:48PM +0200, Thibaut A wrote:

>    So far, I've understood I have to create a fuse Filesystem to "mirror"
>    a raw device like /dev/sdb1.
>    In order to do that, I got inspired by the fusexmp.c file in the
>    libfuse git repository.
>    I've created a directory in /tmp where I can execute the fuse
>    executable. After this, I can see all the files and directories from my
>    /dev/sda1 and I can "play" with them.
>    I can create/delete/move files and create/delete/move directories in
>    this filesystem.
>    When I'm done, I just have to unmount my recently mounted filesystem.
>    Am I in the right so far ?
>    About the defragmentation part, I don't get in which direction that
>    want us to go.
>    Best regards,

Reducing fragmentation is a tricky feature, it really depend on usage patterns,
avoid fragmentation is easy when you have a read most filesystem, since all
files are supposed to be written once, the problem itself begins when you have
a mix of small/large files, and files change their size often. You write file A,
then file B, file A needs to grow, and you have a fragmented file. And of course
it gets worse and worse while the filesystem ages.

Fragmentation is almost impossible to avoid on Read/Write intensive filesystems,
but there are a few techniques that can help reducing this, like delayed
allocation and pre-allocation. But, how and when to use such techniques really
depends on the use case.

If you do not have a specific use case, and you just need to implement a
'generic' filesystem and do your best to avoid fragmentation, delayed and
pre-allocation are quite useful. You can add configuration options so the user
can set these options to their needs, but, I don't know how much time you have
to write that, so, it also depends on how much effort you want to put on these
stuff :)

>    Thibaut

> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e

> --
> fuse-devel mailing list
> To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel


--
Carlos

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Thibaut A
In reply to this post by Thibaut A

Thanks for your precise answer, it's a bit less vague in my mind.

Concerning the delayed and pre-allocation part, correct me if I'm wrong but isn't it what the fallocate function does in the fusexmp.c file ?

If not, how can I implement this in my filesystem ?

I'm sorry if my questions are dumb ones but fuse is totally unknown for me...


Reducing fragmentation is a tricky feature, it really depend on usage patterns,
avoid fragmentation is easy when you have a read most filesystem, since all
files are supposed to be written once, the problem itself begins when you have
a mix of small/large files, and files change their size often. You write file A,
then file B, file A needs to grow, and you have a fragmented file. And of course
it gets worse and worse while the filesystem ages.

Fragmentation is almost impossible to avoid on Read/Write intensive filesystems,
but there are a few techniques that can help reducing this, like delayed
allocation and pre-allocation. But, how and when to use such techniques really
depends on the use case.

If you do not have a specific use case, and you just need to implement a
'generic' filesystem and do your best to avoid fragmentation, delayed and
pre-allocation are quite useful. You can add configuration options so the user
can set these options to their needs, but, I don't know how much time you have
to write that, so, it also depends on how much effort you want to put on these
stuff :)
Carlos

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Nikolaus Rath
In reply to this post by Thibaut A
On Jun 01 2016, Thibaut A <[hidden email]> wrote:
> So far, I've understood I have to create a fuse Filesystem to "mirror" a
> raw device like /dev/sdb1.
> In order to do that, I got inspired by the fusexmp.c file in the libfuse
> git repository.

As I understand what you've said, this is exactly what you're not
supposed to do. fusexmp just "passes through" requests to another file
system, which is then responsible for writing/reading /dev/sdb1.

You're supposed to write a filesystem that accesses /dev/sdb1
directly. fusexmp does not do that.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fuse filesystem - Fragmentation

Thibaut A
In reply to this post by Thibaut A
Really ? Oh, this is not good. And I here I thought I was not far from getting through this.

How can I get on with this ? I think there is nowhere on the web where it's fully explained, right ?

Best, 

Thibaut

As I understand what you've said, this is exactly what you're not
supposed to do. fusexmp just "passes through" requests to another file
system, which is then responsible for writing/reading /dev/sdb1.

You're supposed to write a filesystem that accesses /dev/sdb1
directly. fusexmp does not do that.


Best,
-Nikolaus


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel