Bug #98

BNX NIC driver issue (HP DL3x0 G5)

Added by Alex Broad over 2 years ago. Updated about 1 year ago.

Status:New Start:July 26, 2010
Priority:Normal Due date:
Assigned to:Roman Strashkin % Done:

90%
Category:-
Target version:-

Description

Hi, I've recreated this issue on an HP DL360 and HP DL380 G5 - both machines fail to configure networking from the nmc at first boot after installing and again through the web interface.

This bug may be related to bug #97 - i.e. I believe it's a broadcom driver issue.

The error produced is the same however it is done, and can be reproduced in 'expert mode' as well if you attempt to configure the interface's MTU setting with /sbin/ifconfig (which I guess is how you do things behind the GUI anyway):

root@storage-1:/# /sbin/ifconfig bnx0 mtu 1500 ifconfig: setifmtu: SIOCSLIFMTU: bnx0: Invalid argument root@storage-1:/# ifconfig bnx0 bnx0: flags=1000843 mtu 1496 index 2 inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255 ether 0:1c:c4:13:91:2c

The manual workaround is to enable 'expert mode' and configure the interface through files in /etc as you would for Solaris/OpenSolaris.

bnx.conf - /kernel/drv/bnx.conf (16.9 KB) Alex Broad, 07/27/2010 01:45 am

History

Updated by Roman Strashkin over 2 years ago

Did you try to change MTU via NMC ? What error?

Try to change MTU via 'dladm'.

Updated by Alex Broad over 2 years ago

OK, when I set this up through the nmc, I get:

nmc@storage-1:/$ setup
Option ?  network
Option ?  interface
Option ?  bnx0
Option ?  static
bnx0 IP address     : 10.191.124.200
bnx0 netmask        : 255.255.128.0
bnx0 mtu            : 1500
Name Server #1      : 10.191.1.10
Name Server #2      :
Name Server #3      :
Gateway IP address  : 10.191.1.15
OK.
nmc@storage-1:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes

root@storage-1:/volumes# ifconfig bnx0
bnx0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 2
inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255
ether 0:1c:c4:13:91:2c

And when setting up through dladm I get:

root@storage-1:/volumes# dladm set-linkprop -p mtu=1500 bnx0
dladm: warning: cannot set link property 'mtu' on 'bnx0': operation not supported

Updated by Roman Strashkin over 2 years ago

Alex Broad wrote:

OK, when I set this up through the nmc, I get:

nmc@storage-1:/$ setup
Option ?  network
Option ?  interface
Option ?  bnx0
Option ?  static
bnx0 IP address     : 10.191.124.200
bnx0 netmask        : 255.255.128.0
bnx0 mtu            : 1500
Name Server #1      : 10.191.1.10
Name Server #2      :
Name Server #3      :
Gateway IP address  : 10.191.1.15
OK.
nmc@storage-1:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes

root@storage-1:/volumes# ifconfig bnx0
bnx0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 2
inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255
ether 0:1c:c4:13:91:2c

And when setting up through dladm I get:

root@storage-1:/volumes# dladm set-linkprop -p mtu=1500 bnx0
dladm: warning: cannot set link property 'mtu' on 'bnx0': operation not supported

Can you try to set mtu=9000 via NMC?

Updated by Roman Strashkin over 2 years ago

Roman Strashkin wrote:

Alex Broad wrote:

OK, when I set this up through the nmc, I get:

nmc@storage-1:/$ setup
Option ?  network
Option ?  interface
Option ?  bnx0
Option ?  static
bnx0 IP address     : 10.191.124.200
bnx0 netmask        : 255.255.128.0
bnx0 mtu            : 1500
Name Server #1      : 10.191.1.10
Name Server #2      :
Name Server #3      :
Gateway IP address  : 10.191.1.15
OK.
nmc@storage-1:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes

root@storage-1:/volumes# ifconfig bnx0
bnx0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 2
inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255
ether 0:1c:c4:13:91:2c

And when setting up through dladm I get:

root@storage-1:/volumes# dladm set-linkprop -p mtu=1500 bnx0
dladm: warning: cannot set link property 'mtu' on 'bnx0': operation not supported

Can you try to set mtu=9000 via NMC?

and please send '/kernel/drv/bnx.conf' once you try to change the mtu via NMC.

Updated by Alex Broad over 2 years ago

root@storage-1:/volumes# ifconfig bnx1 mtu 9000
ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid argument
root@storage-1:/volumes# ifconfig bnx1
bnx1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 3
inet 172.16.85.1 netmask ffffff00 broadcast 172.16.85.255
ether 0:1c:c4:13:91:28
nmc@storage-1:/$ setup
Option ?  network
Option ?  interface
Option ?  bnx1
Option ?  static
bnx1 IP address     : 172.16.85.1
bnx1 netmask        : 255.255.255.0
bnx1 mtu            : 9000
Warning: changing mtu MAY require driver re-load! Network Interface(s) [bnx0] could be re-initialized.
Change MTU from the current  to 9000 ?  Yes
Name Server #1      :
Name Server #2      :
Name Server #3      :
Gateway IP address  :
Enabling bnx1 as 172.16.85.1/255.255.255.0 mtu 9000 ... Failed.
com.nexenta.nms.SystemCallError: failed to configure bnx1 with ip 172.16.85.1 netmask 255.255.255.0 mtu 9000  broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid argument
root@storage-1:/volumes# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bnx0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 8982 index 6
inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255
ether 0:1c:c4:13:91:2c
bnx1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 8982 index 7
inet 172.16.85.1 netmask ffffff00 broadcast 172.16.85.255
ether 0:1c:c4:13:91:28
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128

Updated by Roman Strashkin over 2 years ago

Alex Broad wrote:

root@storage-1:/volumes# ifconfig bnx1 mtu 9000 ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid argument root@storage-1:/volumes# ifconfig bnx1 bnx1: flags=1000843 mtu 1496 index 3 inet 172.16.85.1 netmask ffffff00 broadcast 172.16.85.255 ether 0:1c:c4:13:91:28 nmc@storage-1:/$ setup Option ? network Option ? interface Option ? bnx1 Option ? static bnx1 IP address : 172.16.85.1 bnx1 netmask : 255.255.255.0 bnx1 mtu : 9000 Warning: changing mtu MAY require driver re-load! Network Interface(s) [bnx0] could be re-initialized. Change MTU from the current to 9000 ? Yes Name Server #1 : Name Server #2 : Name Server #3 : Gateway IP address : Enabling bnx1 as 172.16.85.1/255.255.255.0 mtu 9000 ... Failed. com.nexenta.nms.SystemCallError: failed to configure bnx1 with ip 172.16.85.1 netmask 255.255.255.0 mtu 9000 broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid argument root@storage-1:/volumes# ifconfig -a lo0: flags=2001000849 mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bnx0: flags=1000843 mtu 8982 index 6 inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255 ether 0:1c:c4:13:91:2c bnx1: flags=1000843 mtu 8982 index 7 inet 172.16.85.1 netmask ffffff00 broadcast 172.16.85.255 ether 0:1c:c4:13:91:28 lo0: flags=2002000849 mtu 8252 index 1 inet6 ::1/128

Can you uncomment string (line 278) in bnx.conf and re-try to change MTU via NMC:

mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;

Thanks.

Updated by Alex Broad over 2 years ago

root@storage-1:/volumes# vi /kernel/drv/bnx.conf
root@storage-1:/volumes# exit
exit
Important: To re-sync the appliance's management state information, please
consider running 'setup appliance nms restart' command.
nmc@storage-1:/$ setup appliance nms restart
Trying to gain exclusive access to the appliance.
This operation may take up to 30 seconds to complete. Please wait...
Exclusive access granted.
This operation will make  Management Server temporarily unavailable. Proceed?  (Yes
Restarting NMS. Please wait...
NMS was successfully restarted
nmc@storage-1:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes
root@storage-1:/volumes# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bnx0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 8982 index 6
inet 10.191.124.200 netmask ffff8000 broadcast 10.191.127.255
ether 0:1c:c4:13:91:2c
bnx1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 8982 index 7
inet 172.16.85.1 netmask ffffff00 broadcast 172.16.85.255
ether 0:1c:c4:13:91:28
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
root@storage-1:/volumes# exit
exit
Important: To re-sync the appliance's management state information, please
consider running 'setup appliance nms restart' command.
nmc@storage-1:/$ setup
Option ?  network
Option ?  interface
Option ?  bnx1
Option ?  static
bnx1 IP address     : 172.16.85.1
bnx1 netmask        : 255.255.255.0
bnx1 mtu            : 9000
Warning: changing mtu MAY require driver re-load! Network Interface(s) [bnx0] could be re-initialized.
Change MTU from the current 1500 to 9000 ?  Yes
Name Server #1      :
Name Server #2      :
Name Server #3      :
Gateway IP address  :
Enabling bnx1 as 172.16.85.1/255.255.255.0 mtu 9000 ... Failed.
com.nexenta.nms.SystemCallError: failed to configure bnx1 with ip 172.16.85.1 netmask 255.255.255.0 mtu 9000  broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx1: Invalid argument

Updated by Roman Strashkin over 2 years ago

  • Assigned to set to Roman Strashkin
  • % Done changed from 0 to 90

max MTU for 'bnx' = 8982.

fix will be available in beta version. (updates will be available during 1-2 days). please test and close this ticket if all ok. Thanks.

Updated by Koen Calliauw over 2 years ago

not fixed in the just-released RC5...

Updated by Roman Strashkin over 2 years ago

Koen Calliauw wrote:

not fixed in the just-released RC5...

Can you describe your steps (with logs) ?

Updated by Koen Calliauw over 2 years ago

Roman Strashkin wrote:

Koen Calliauw wrote:

not fixed in the just-released RC5...

Can you describe your steps (with logs) ?

Hi,

Exactly like described in the original issue. When setting up the network on the console the first time, the SIOCSLIFMTU error is displayed (invalid argument) and the bnx0 card is unusable. Also, trying to set the MTU manually doesn't work (same error). Only when adjusting bnx.conf and rebooting, it starts working.

Off topic: is a reboot supposed to take 10-15 minutes on a HP DL360 G5, 2x E5520, 32GB memory MSA2012sa das system attached to Sg80e (LSI I think) SAS HBA.

/K

Updated by Peter Valadez over 2 years ago

This worked for me after changing mtu to 8982 in the nmc and without modifying the bnx.conf file. However, is this going to cause problems if you do not have Jumbo Frames enabled on the switch it is connected to?

I tried modifying the bnx.conf file to mtu=1496; and then set mtu to 1496 in nmc, but this didn't work.

Updated by Oded Maimon about 1 year ago

new user to nexenta, had the same issue. once i've changed it to 8982 and than changed it back to 1500, it worked... i used the "setup network interface bnx0" command to setup the interface.

thanks

Also available in: Atom PDF