Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0

Added by André Werup about 1 year ago

Clean installed NexentaStor Community Edition 3.1.0 and compleated setup. Everything seemed fine, but shortly atfther mounting a CIFS share I lost all network connections to the machine.

ifconfig -a shows the adapter as still beeing up but pinging other machines on the network was a no go

Afther reboot the connection is back up and ping works fine, but afther a short time (10-15 min) network connection is lost again.

  • reboot allways restore network connection, but it's lost again afther a short time
  • nestat -rn shows same route both when network is working and when it's down
  • ifconfig show adapter as up in all situations, taking it down and back up does not reaquire network connection

The same hardware was used earlier with Opens Solaris without any network issues.

Anybody got an ide of how I should procede?


Replies

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Yes, im mounting the CIFS share on Windows 7.

Have tested it a bit more today and even turned of the CIFS share entirely without any change in behavior. The server losses network connectivity within 10-15 min of beeing booted as long as there is network activity against it.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

Can I have you file this as a bug? I actually am seeing this dropping of network connection several places today and it is only with windows 7 clients. I would like this to get into the support queue for a permanent fix.

NMV bottom banner, click on "Found a bug?"

Hopefully we can find a workaround or root cause. If there is anyone reading this and seeing the same problem please report it to support@nexenta.com

thanks.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Just done a complete reinstall and currently testing filetransfer from OS X (Snow Leopard) on a CIFS mount. No issues so far but only got about 30 min of data. Will redo Win 7 tests tomorrow and report bug if i still get the same network issues then.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Tested with both Windows 7 and OS X CIFS client. No issues with OS X, but 5-10 min after Win 7 client starts transfering files the nexenta server looses network and only way i found to restore network is a full reboot.

@Linda: Tried to file this as a bug, but as i´m on a community license i only got "Contact Not Registered - Action Required" as an respons.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

actually i filed this as a bug. thanks for trying :)

i am keeping my eye on this, will advise.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

hey can you run snoop on the server while testing?

snoop -d interfacename windowshostname -o outputfile

we should be able to see what's not working.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Ok, running it now. File is going to be fairly large, but i guess were only interested in the end of it...

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Attached are the last 1000 lines from

snoop -d gani0 -o win-7-cifs.log 10.0.0.4

As you can see there's quite a few packets sent before things go bad

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

nice, i like when i can see the problem. My favorite cifs engineer is out until monday. I would like to run this by him.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

as you are testing, are you reading from the ns and writing to the windows box? or running some io simulator?

also, what does the pool look like on the ns?

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Testing by writing large 500+ MB files from the win box to ns. Below is the output from zpool status

pool: syspool state: ONLINE scan: none requested config:

    NAME        STATE     READ WRITE CKSUM
    syspool     ONLINE       0     0     0
      c1d0s0    ONLINE       0     0     0

errors: No known data errors

pool: tank state: ONLINE scan: none requested config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        c0d0    ONLINE       0     0     0
        c2d0    ONLINE       0     0     0
        c3d0    ONLINE       0     0     0

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Linda Kateley about 1 year ago

can you see the size of the file at the point it stops? there is an adjustment that is made at 500 MB

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Haven´t got excat numbers but judging from the explorer UI i´v had one failure at 600MB and one at 1.3GB

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

And just to clarify, sometimes i can get several 500+MB files across before it fails.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Gordon Ross about 1 year ago

Any chance you can get us a binary snoop of this problem? i.e.

  snoop -o win7stop.snoop host $my_win7_client_name

If you like, you can examine it with wireshark and perhaps save just the last 1000 frames or so to a new file and send that instead of the huge original capture file.

Thanks, Gordon

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Gordon Ross about 1 year ago

I need a binary snoop (the text representation omits too many details) Filtering the the client of interest is helpful, as is saving only the last 1000 frames or so. To do that, open the snoop file in wireshark, select the last 1000 frames, then save as (selected frames)...

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Here's to hoping this will be a case of better late then never...

Attached is a snoop of the last 1 000 packets of a transfer from Win 7 (10.0.0.9) to Nexenta 3.1.1 (10.0.0.6) that killed the Nexenta network connection. The transfer was a single 8GB file.

win7stop-last1000packets.snoop - Last 1000 packets of a CIFS transfer from Win7 to Nexenta 3.1.1 that killed the Nexenta network connection (950 KB)

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by Gordon Ross about 1 year ago

Thanks for providing a snoop capture. I examined it, and the last few frames are SMBwrite responses, followed by some TCP retransmit frames. At that point, the client appears to be sending nothing back. I would look next at what's happening on the client.

RE: Losing network connection shortly afther boot - NexentaStor Community Edition 3.1.0 - Added by André Werup about 1 year ago

Hm, on the client side I see retransmissions both ways then a lot of ARP traffic. The odd bit here is that at the client retains network connectivity after this exchange while the Nexenta server losses all network connectivity and needs to be rebooted to restore it. As said before, doing the same file transfers with my Mac as client works flawlessly.

Attached is both server and client snoop if it's any help.