Patches for FIEMAP support in fuse and ntfs-3g

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
Hi!

I've written some simple patches for FIEMAP support in fuse and ntfs-3g,  
via a separate FUSE opcode (not a generic ioctl). I wanted this feature to  
be able to play with in-place filesystem conversion, but discovered it  
wasn't there - even current fuse versions don't support it...

The patches are attached, can you review them? There are patches for all  
parts: fuse kernel driver (for 3.12.x branch), fuse library and ntfs-3g  
implementation (including fuse-lite). All fiemap API constants are  
'copied' into fuse. I've done some testing and it seems to work for both  
lowntfs-3g and ntfs-3g, both normal fuse and fuse-lite; 32-bit compat  
ioctl is also passed-through correctly.

What do you think about this feature? :) and what's the process of getting  
it merged in general? This is especially interesting because this requires  
changes to different subsystems.

--
With best regards,
   Vitaliy Filippov
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel

fuse-fiemap-kernel.diff (5K) Download Attachment
fuse-fiemap-lib-master.diff (12K) Download Attachment
fuse-fiemap-ntfs-3g.diff (19K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
> Hi!
>
> I've written some simple patches for FIEMAP support in fuse and ntfs-3g,
> via a separate FUSE opcode (not a generic ioctl). I wanted this feature  
> to
> be able to play with in-place filesystem conversion, but discovered it
> wasn't there - even current fuse versions don't support it...
>
> The patches are attached, can you review them? There are patches for all
> parts: fuse kernel driver (for 3.12.x branch), fuse library and ntfs-3g
> implementation (including fuse-lite). All fiemap API constants are
> 'copied' into fuse. I've done some testing and it seems to work for both
> lowntfs-3g and ntfs-3g, both normal fuse and fuse-lite; 32-bit compat
> ioctl is also passed-through correctly.
>
> What do you think about this feature? :) and what's the process of  
> getting
> it merged in general? This is especially interesting because this  
> requires
> changes to different subsystems.

Anyone?..

--
With best regards,
   Vitaliy Filippov

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Stef Bon-2
I'm sorry, I've read your post, but I can't help you since I'm not
familiar with fiemap.

Stef

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
> I'm sorry, I've read your post, but I can't help you since I'm not
> familiar with fiemap.

fiemap is an ioctl like bmap, but better - it allows to get a map of file  
block locations on the physical media. bmap works on a per-block level, so  
for a 20GB file you'll basically need to call bmap (20GB/filesystem  
cluster size) times which is very slow... Another problem with bmap is  
that it only allows 32-bit block numbers at least on 32-bit platforms.

fiemap on the other hand returns mapped extents, i.e. byte ranges within  
file, and always works in 64bit. You can test it easily with standard  
'filefrag' utility from e2fsprogs (filefrag -v /path/to/file) - it prefers  
fiemap, but also tries to use bmap if fiemap is unavailable (but as  
already said it's very slow); also you can test it using 'hdparm --fibmap  
/path/to/file', it does the same.

Kernel documentation is here:  
https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt

Current versions of fuse and ntfs-3g don't support fiemap... Back in 2008  
there even was a discussion in this mailing list and fiemap was mentioned  
there... https://lkml.org/lkml/2008/10/14/157 - but as I understand, it  
never got implemented... So I would like it very much if someone reviewed  
my patches... :)

--
With best regards,
   Vitaliy Filippo

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Miklos Szeredi
In reply to this post by Vitaliy Filippov
On Sun, Feb 23, 2014 at 12:29 PM, Vitaliy Filippov <[hidden email]> wrote:

> Hi!
>
> I've written some simple patches for FIEMAP support in fuse and ntfs-3g, via
> a separate FUSE opcode (not a generic ioctl). I wanted this feature to be
> able to play with in-place filesystem conversion, but discovered it wasn't
> there - even current fuse versions don't support it...
>
> The patches are attached, can you review them? There are patches for all
> parts: fuse kernel driver (for 3.12.x branch), fuse library and ntfs-3g
> implementation (including fuse-lite). All fiemap API constants are 'copied'
> into fuse. I've done some testing and it seems to work for both lowntfs-3g
> and ntfs-3g, both normal fuse and fuse-lite; 32-bit compat ioctl is also
> passed-through correctly.
>
> What do you think about this feature? :) and what's the process of getting
> it merged in general? This is especially interesting because this requires
> changes to different subsystems.

The implementation looks OK.

The issue I have with the interface is: why reinvent a new API on each
of the interfaces?  Yes, we do that for most requests, but here's a
request that is already in a flat structure format, exactly what the
kernel API needs.  So I suggest that we mirror the fiemap structures
there.  On the library API you can also just pass a "struct fiemap *"
pointer to the fiemap method and be done with that.

Thanks,
Miklos

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
> The issue I have with the interface is: why reinvent a new API on each
> of the interfaces?  Yes, we do that for most requests, but here's a
> request that is already in a flat structure format, exactly what the
> kernel API needs.  So I suggest that we mirror the fiemap structures
> there.  On the library API you can also just pass a "struct fiemap *"
> pointer to the fiemap method and be done with that.

Thanks for answering!

I copied API into fuse because I've seen 2 fiemap APIs in the kernel:
one in include/uapi/linux/fiemap.h and one in include/fs.h... the
'request' structure differs, first API uses platform-independent data
types (u32/u64), and the second uses 'unsigned int'. I think that's what
confused me.

But yes, the individual extent structure is the same... So OK, I'll
change patches to use it instead of a copied structure.

But I have a small question - what about the headers? struct fiemap is
in /usr/include/linux/fiemap.h. What's the correct way - should it be
copied into fuse, or should the system header be required? (what about
non-linux fuse?)

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works.
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Jean-Pierre André
[hidden email] wrote:

>> The issue I have with the interface is: why reinvent a new API on each
>> of the interfaces?  Yes, we do that for most requests, but here's a
>> request that is already in a flat structure format, exactly what the
>> kernel API needs.  So I suggest that we mirror the fiemap structures
>> there.  On the library API you can also just pass a "struct fiemap *"
>> pointer to the fiemap method and be done with that.
> Thanks for answering!
>
> I copied API into fuse because I've seen 2 fiemap APIs in the kernel:
> one in include/uapi/linux/fiemap.h and one in include/fs.h... the
> 'request' structure differs, first API uses platform-independent data
> types (u32/u64), and the second uses 'unsigned int'. I think that's what
> confused me.
>
> But yes, the individual extent structure is the same... So OK, I'll
> change patches to use it instead of a copied structure.
>
> But I have a small question - what about the headers? struct fiemap is
> in /usr/include/linux/fiemap.h. What's the correct way - should it be
> copied into fuse, or should the system header be required? (what about
> non-linux fuse?)

AFAIK struct fiemap is Linux only and cannot be used in
portable code.

This leads to lseek() with modes SEEK_HOLE or SEEK_DATA,
which are supposed to be made Posix compliant some day.
Would not these be a better alternative for portable code ?

Jean-Pierre

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works.
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
> AFAIK struct fiemap is Linux only and cannot be used in
> portable code.

I.e. if you don't copy the interface into FUSE... :)

> This leads to lseek() with modes SEEK_HOLE or SEEK_DATA,
> which are supposed to be made Posix compliant some day.
> Would not these be a better alternative for portable code ?

lseek does a different task - it only allows you to look at holes in
file, while fiemap allows you to determine the location of file data on
the HDD. That's why I started to play with it - I wanted to try in-place
fs conversion.

Also it seems even 'cp' still doesn't use SEEK_HOLE/SEEK_DATA for sparse
file handling, but it uses fiemap :)

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works.
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Miklos Szeredi
In reply to this post by Vitaliy Filippov
On Wed, Mar 5, 2014 at 12:16 PM,  <[hidden email]> wrote:

>> The issue I have with the interface is: why reinvent a new API on each
>> of the interfaces?  Yes, we do that for most requests, but here's a
>> request that is already in a flat structure format, exactly what the
>> kernel API needs.  So I suggest that we mirror the fiemap structures
>> there.  On the library API you can also just pass a "struct fiemap *"
>> pointer to the fiemap method and be done with that.
>
>
> Thanks for answering!
>
> I copied API into fuse because I've seen 2 fiemap APIs in the kernel: one in
> include/uapi/linux/fiemap.h and one in include/fs.h... the 'request'
> structure differs, first API uses platform-independent data types (u32/u64),
> and the second uses 'unsigned int'. I think that's what confused me.
>
> But yes, the individual extent structure is the same... So OK, I'll change
> patches to use it instead of a copied structure.
>
> But I have a small question - what about the headers? struct fiemap is in
> /usr/include/linux/fiemap.h. What's the correct way - should it be copied
> into fuse, or should the system header be required? (what about non-linux
> fuse?)

In <linux/fuse.h> the best is to copy the structures so that it works
on other archs.

On the userspace API we can just forward declare the structure without
having to include anything.   Any portable implementations actually
using FIEMAP will have to check (e.g. with autoconf) whether struct
fiemap is defined and include the header if it is.

Thanks,
Miklos

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works.
Faster operations. Version large binaries.  Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Stef Bon-2
Can you or someone else give me an example of the use of this call.
Let me explain why I ask this. FUSE filesystems are there for some
special backends, like rare filesystems (like exfat), uncommon (like
archives), unsupported by kernel (webdav, sshfs).

In what cases fiemap is supported and/or usefull?

Stef

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Vitaliy Filippov
> Can you or someone else give me an example of the use of this call.
> Let me explain why I ask this. FUSE filesystems are there for some
> special backends, like rare filesystems (like exfat), uncommon (like
> archives),

> unsupported by kernel (webdav, sshfs).

..........ntfs..........

> In what cases fiemap is supported and/or usefull?

Basically in any case when you want to map file blocks to hard drive  
blocks without using FS-specific code! :)

As I've already said, you can test it with standard tool from e2fsprogs:  
'/sbin/filefrag -v /path/to/file' will print out all physical block  
numbers for a given file.

My personal case was to try in-place FS conversion  
(http://tzukanov.narod.ru/convertfs/)... I didn't succeed thus far, but  
I'm planning to make another attempt :)

--
With best regards,
   Vitaliy Filippov

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patches for FIEMAP support in fuse and ntfs-3g

Stef Bon-2
Ha sorry, did not read the subject well.

Stef

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
fuse-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/fuse-devel