Restarting fuse process breaks sub mounts

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Restarting fuse process breaks sub mounts

Ross Boucher
Hi everyone,

I've got a setup that looks something like this:

/shared/fuse  => fuse mounted filesystem
/shared/mount => bind mount of /shared/fuse/foo 

This works well enough, unless the process running the fuse-fs dies. Even once it's brought back online, accessing /shared/mount results in a Transport not connected error. 

Is this expected behavior? Is there any way to work around it? In a normal bind mount, if the original directory went away and came back, the mount would continue to work. 

In case it's not totally clear, I did put together a demo to reproduce:

I should add that I'm actually trying to track down a bug that manifests in the same way, except that the process isn't dying -- all of my existing mount points just become stale for some unknown reason from time to time (and new ones work fine). But I think figuring out the restart case will mitigate this problem, which is why I'm focusing on it.

Thanks,
Ross


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Nikolaus Rath
On Oct 27 2016, Ross Boucher <[hidden email]> wrote:

> Hi everyone,
>
> I've got a setup that looks something like this:
>
> /shared/fuse  => fuse mounted filesystem
> /shared/mount => bind mount of /shared/fuse/foo
>
> This works well enough, unless the process running the fuse-fs dies. Even
> once it's brought back online, accessing /shared/mount results in a
> Transport not connected error.
>
> Is this expected behavior?

Well, I would expect it. The kernel has no way to know that the new file
system is "the same".

> Is there any way to work around it? In a normal
> bind mount, if the original directory went away and came back, the mount
> would continue to work.

While I would actually not have expected this. Interesting. But I don't
think there's a way to get FUSE to do the same without changes to the
kernel interface.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Ross Boucher
That's unfortunate. Do you have any suggestions for ways I might figure out what's causing my existing mounts to "disconnect"? 

On Thu, Oct 27, 2016 at 12:04 PM Nikolaus Rath <[hidden email]> wrote:
On Oct 27 2016, Ross Boucher <[hidden email]> wrote:
> Hi everyone,
>
> I've got a setup that looks something like this:
>
> /shared/fuse  => fuse mounted filesystem
> /shared/mount => bind mount of /shared/fuse/foo
>
> This works well enough, unless the process running the fuse-fs dies. Even
> once it's brought back online, accessing /shared/mount results in a
> Transport not connected error.
>
> Is this expected behavior?

Well, I would expect it. The kernel has no way to know that the new file
system is "the same".

> Is there any way to work around it? In a normal
> bind mount, if the original directory went away and came back, the mount
> would continue to work.

While I would actually not have expected this. Interesting. But I don't
think there's a way to get FUSE to do the same without changes to the
kernel interface.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Nikolaus Rath
Hi Ross,

A: Because it confuses the reader.
Q: Why?
A: No.
Q: Should I write my response above the quoted reply?

..so please quote properly, as I'm doing in the rest of this mail:

On Oct 27 2016, Ross Boucher <[hidden email]> wrote:
> That's unfortunate. Do you have any suggestions for ways I might figure out
> what's causing my existing mounts to "disconnect"?

What error do you get when you try to access the mountpoint?
Are you sure that the filesystem process is still alive? Typically,
you'll get errors only if the filesystem is actively responding with
error codes, or if it has crashed.

Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Ross Boucher
On Thu, Oct 27, 2016 at 6:52 PM Nikolaus Rath <[hidden email]> wrote:

On Oct 27 2016, Ross Boucher <[hidden email]> wrote:
> That's unfortunate. Do you have any suggestions for ways I might figure out
> what's causing my existing mounts to "disconnect"?

What error do you get when you try to access the mountpoint?
Are you sure that the filesystem process is still alive? Typically,
you'll get errors only if the filesystem is actively responding with
error codes, or if it has crashed.

Best,
-Nikolaus

The error is: "Transport endpoint is not connected". I'm reasonably certain the filesystem process is still alive, as its running inside a Docker container and I'd have a log of it exiting/restarting. I can continue to access the filesystem directly, or even through new mounts created into it, but for whatever reason the existing mounts stop working with that error.

Is there a system level log file that might have interesting information in it? 

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Nikolaus Rath
On Oct 28 2016, Ross Boucher <[hidden email]> wrote:

> On Thu, Oct 27, 2016 at 6:52 PM Nikolaus Rath <[hidden email]> wrote:
>
>>
>> On Oct 27 2016, Ross Boucher <
>> [hidden email]> wrote:
>> > That's unfortunate. Do you have any suggestions for ways I might figure
>> out
>> > what's causing my existing mounts to "disconnect"?
>>
>> What error do you get when you try to access the mountpoint?
>> Are you sure that the filesystem process is still alive? Typically,
>> you'll get errors only if the filesystem is actively responding with
>> error codes, or if it has crashed.
>>
> The error is: "Transport endpoint is not connected". I'm reasonably certain
> the filesystem process is still alive,

Check again, I'm more than reasonably sure that you will it is not.

> I can continue to
> access the filesystem directly

I'm not sure what you mean by "directly". The only way to access a FUSE
filesystem is through the mountpoint, which you say does not work.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Ross Boucher


On Fri, Oct 28, 2016 at 3:22 PM Nikolaus Rath <[hidden email]> wrote:
On Oct 28 2016, Ross Boucher <[hidden email]> wrote:
> On Thu, Oct 27, 2016 at 6:52 PM Nikolaus Rath <[hidden email]> wrote:
>
>>
>> On Oct 27 2016, Ross Boucher <
>> [hidden email]> wrote:
>> > That's unfortunate. Do you have any suggestions for ways I might figure
>> out
>> > what's causing my existing mounts to "disconnect"?
>>
>> What error do you get when you try to access the mountpoint?
>> Are you sure that the filesystem process is still alive? Typically,
>> you'll get errors only if the filesystem is actively responding with
>> error codes, or if it has crashed.
>>
> The error is: "Transport endpoint is not connected". I'm reasonably certain
> the filesystem process is still alive,

Check again, I'm more than reasonably sure that you will it is not.

Unfortunately, you're mistaken.
 
> I can continue to
> access the filesystem directly

I'm not sure what you mean by "directly". The only way to access a FUSE
filesystem is through the mountpoint, which you say does not work.

It seems you've missed my original statement. For clarity, my setup is this:

/shared/fuse  => fuse mounted filesystem
/shared/mount => bind mount of /shared/fuse/foo 

By "directly", I mean accessing "/shared/fuse" directly. And when referring to the existing mounts, I'm talking about e.g. "/shared/mount" above (of which I actually have many different instances).

-Ross

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Nikolaus Rath
On Oct 28 2016, Ross Boucher <[hidden email]> wrote:

> On Fri, Oct 28, 2016 at 3:22 PM Nikolaus Rath <[hidden email]> wrote:
>
>> On Oct 28 2016, Ross Boucher <
>> [hidden email]> wrote:
>> > On Thu, Oct 27, 2016 at 6:52 PM Nikolaus Rath <
>> [hidden email]> wrote:
>> >
>> >>
>> >> On Oct 27 2016, Ross Boucher <
>> >> [hidden email]>
>> wrote:
>> >> > That's unfortunate. Do you have any suggestions for ways I might
>> figure
>> >> out
>> >> > what's causing my existing mounts to "disconnect"?
>> >>
>> >> What error do you get when you try to access the mountpoint?
>> >> Are you sure that the filesystem process is still alive? Typically,
>> >> you'll get errors only if the filesystem is actively responding with
>> >> error codes, or if it has crashed.
>> >>
>> > The error is: "Transport endpoint is not connected". I'm reasonably
>> certain
>> > the filesystem process is still alive,
>>
>> Check again, I'm more than reasonably sure that you will it is not.
>>
>
> Unfortunately, you're mistaken.
>
>
>> > I can continue to
>> > access the filesystem directly
>>
>> I'm not sure what you mean by "directly". The only way to access a FUSE
>> filesystem is through the mountpoint, which you say does not work.
>
>
> It seems you've missed my original statement. For clarity, my setup is this:
>
> /shared/fuse  => fuse mounted filesystem
> /shared/mount => bind mount of /shared/fuse/foo
>
> By "directly", I mean accessing "/shared/fuse" directly. And when referring
> to the existing mounts, I'm talking about e.g. "/shared/mount" above (of
> which I actually have many different instances).

Oh, I see. Sorry, I forget about that indeed. So the (full) sequence of
events is as follows?

$ my_fuse_fs /shared/fuse
$ mount --bind /shared/fuse/foo /shared/mount
$ # do some stuff
$ ls /shared/mount/
Transport endpoint is not connected
$ ls /shared/fuse/foo
list of files


That's.. interesting. Do you need to do anything specific in order for
the bind mount to become inaccessible, or is it sufficient to just wait?
Can you reproduce the same problem with one of the example file systems?


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Restarting fuse process breaks sub mounts

Ross Boucher
On Fri, Oct 28, 2016 at 5:15 PM Nikolaus Rath <[hidden email]> wrote:
On Oct 28 2016, Ross Boucher <[hidden email]> wrote:
> On Fri, Oct 28, 2016 at 3:22 PM Nikolaus Rath <[hidden email]> wrote:
>
>> On Oct 28 2016, Ross Boucher <
>> [hidden email]> wrote:
>> > On Thu, Oct 27, 2016 at 6:52 PM Nikolaus Rath <
>> [hidden email]> wrote:
>> >
>> >>
>> >> On Oct 27 2016, Ross Boucher <
>> >> [hidden email]>
>> wrote:
>> >> > That's unfortunate. Do you have any suggestions for ways I might
>> figure
>> >> out
>> >> > what's causing my existing mounts to "disconnect"?
>> >>
>> >> What error do you get when you try to access the mountpoint?
>> >> Are you sure that the filesystem process is still alive? Typically,
>> >> you'll get errors only if the filesystem is actively responding with
>> >> error codes, or if it has crashed.
>> >>
>> > The error is: "Transport endpoint is not connected". I'm reasonably
>> certain
>> > the filesystem process is still alive,
>>
>> Check again, I'm more than reasonably sure that you will it is not.
>>
>
> Unfortunately, you're mistaken.
>
>
>> > I can continue to
>> > access the filesystem directly
>>
>> I'm not sure what you mean by "directly". The only way to access a FUSE
>> filesystem is through the mountpoint, which you say does not work.
>
>
> It seems you've missed my original statement. For clarity, my setup is this:
>
> /shared/fuse  => fuse mounted filesystem
> /shared/mount => bind mount of /shared/fuse/foo
>
> By "directly", I mean accessing "/shared/fuse" directly. And when referring
> to the existing mounts, I'm talking about e.g. "/shared/mount" above (of
> which I actually have many different instances).

Oh, I see. Sorry, I forget about that indeed. So the (full) sequence of
events is as follows?

$ my_fuse_fs /shared/fuse
$ mount --bind /shared/fuse/foo /shared/mount
$ # do some stuff
$ ls /shared/mount/
Transport endpoint is not connected
$ ls /shared/fuse/foo
list of files


That's.. interesting. Do you need to do anything specific in order for
the bind mount to become inaccessible, or is it sufficient to just wait?
Can you reproduce the same problem with one of the example file systems?

Unfortunately I don't have a reliable way to reproduce the problem. It was happening somewhat frequently on an older vm, and seems to be rare on my current vm. That's why I was using the process restart example as a proxy -- I figured if that could be made to work the other problem might go away as well. At this point I'm mostly looking for logs I can check or put in place for the next time it happens.

-Ross
 

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
--
fuse-devel mailing list
To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel
Loading...