Cisco Data Center: Fiber Channel and Cisco Storage Fundations
Share this page:Unified Fabricis a term for all of the equipment that makes LAN and SAN possible. There are two different networks (LAN as Front-end and SAN as Back-end) that we are trying to “converge”, and 10G is an enabler of that convergence.
SAN (Storage Area Network) has 3 high level components:
- Storage Arrays, where the Physical Disks are located. These can be JBOD (Just a Bunch of Disks) or RAID.
- SAN Switches, that connect the Servers (Initiators) to Storage (Target) can do protocol conversions between FC and iSCSI, FC and FCoE, FC and FCIP.
- HBA (Host Bus Adapters) – which is a NIC card for SAN. There are 3 types of HBA:
o Fibre Channel HBAs (SAN traffic only, 1/2/4/8/16 Gbps).
o iSCSI HBA(1/10Gbps LAN plus iSCSI hardware offload).
o FCoE Converged Network Adapter (CNA) (10Gbps LAN plus FCoE hardware offload).
TIP: When you change the mode of the port (Ethernet/FC), you need to reboot the switch. On Nexus 5548UP you can assign the FC ports, but you need to start from the END of the ports in the module. For example, a module of 32 ports, you assign the ports 28-32 as FC, and 1-27 as native Ethernet.
iSCSI: IP based protocol capable of carrying SCSI commands over TCP/IP to and from the Storage. SCSI is the standard that actually defines the communication between the Initiator (Server/Host) and the Target (Disk/Storage). LUN is the 64-bit field that SCSI uses for addressing.
Fibre Channel
HBA (Host Bus Adapter) represents a NIC card for the FC host. Ports have a specific role in the Fibre Channel. Fibre Channel defines the Port-to-Port Flow Control, and also End to End flow control. The HBA will negotiate 1/2/4/8/12/16Gbps with the Switch Port.
Fibre Channel Switch is not like the LAN Switch, because FC is a connection oriented protocol. This means that end-to-end “awareness” must exist between all the Servers and the Disks that will communicate. The ports will adapt to what is connected to it. The most important Fibre Channel port types are:
- N_Port(Node Port) is the Fibre Channel port on the Host (HBA on a Server or a target port on a Storage Array).
- F_Port(Fabric Port) is the Fibre Channel Switch Port where the Host (Servers HBA or Storage Targer port) plugs in. There is also NL and FL mode, which are the Loop modes (they include a type of a loop prevention, but these are not used a lot)
- E_Port(Expansion Port) is the port of the interconnection with another Switch. The connection between two E_Ports forms an Inter-Switch Link (ISL). E_Port needs to be in a DEDICATED RATE MODE, and this MUST be set on the MDS (switchport rate-mode dedicated), while on the Nexus it´s automatical.
- TE_Port(Trunking E_Port), interconnection with another Switch where tagged frames from various VSANs are carried. It is a sort of Enchanced ISL link, and it´s analougous to dot1q trunk, but with VSANs. There is NO “switchport mode te” command
- VN_Port(Virtual N_Port), created on a FCoE node (Server or Storage) to enable FCoE communication with a FCoE switch.
- VF_Port (Virtual F_port) on a switch, created as needed to establish connection with an end-node (N_Port)
- VE_Port (Virtual E_port), created on an FCoE switch to link it with another FCoE switch
- * When you set the FC Mode to AUTO on the interface – it will automatically negotiate the Port Type. If you want to force one type of Type, use the command:
- (config-pc)# switchport mode [F | E | NP …]
WWN (World Wide Names): In the FC, the WORD (4 bytes) entoded in the Frames are exchanged between the INITIATOR and the TARGET. EXCHANGE is the series of WORDS exchanged between INITIATOR (Server) and the TARGET (Storage Array), and it´s like the TCP Session.
FCNS (Fibre Channel Name Server) is used to resolve the WWN.
In order for the Fiber Channel device to talk to the destination device, there has to be a Fabric Registration 3-step process.
Step 1: The host N_port must log in to it´s F port (FLOGI, F Port Loginprocess). Switch identifies the WWN of the HBA and assigned the Fibre Channel ID (FCID).
**Step 2:**The host N_port also has to log in to the final N port (PLOGI – End Port Login process). FCNS is used here to handle all the WWNs, and we will have the list of all the Initiators and all the Disks.
Step 3: Process login (PRLI) process must also occur, which handles the upper layer protocols.
[
](https://www.blogger.com/null)
Configure Fibre Channel on a Nexus 5500UP
IMPORTANT: Using the command “fcping” you can PING the remote FCID or WWN.
You need to start the configuration by following these:
Activate the FC interfaces, from last port of the module going down & Reload the Nexus.
Activate the FCoE feature (even ifyou’re using the native FC only). No need for additional reload.
Check with “show license usage”
If you’re using NPV, you also need to activate that feature.
FC Auto Negotiation in ON, but don’t forget to un-shut the FC ports (interface fc1/20, not ethernet). You should, however, manually configure the ports (Port Type, Trunk Mode if it´s a trunk, Speed). You can verify all this using the “show interface fcx/y brief”.
To configure a port as a FC port, we need to do the following, and then RELOAD the Nexus.
Nexus-5k(conf)# slot 1
Nexus-5k(conf-slot)# port 24-32 type fc <- We need to choose the LAST group of ports on the module
To check the interface configuration you can use the “show interface” or “show tranciever”:
-——————————————————————————
Interface Vsan Admin Admin Status Oper Oper Port-channel
fc2/5 1 E on trunking TE 2 port-channel 2
fc2/6 1 E on trunking TE 2 port-channel 2
fc2/7 1 E on down -- -- --
fc2/8 1 auto on fcotAbsent -- -- --
fc2/9 3 E off up E 2 --
fc2/12 3 E on down -- -- port-channel 4
fc3/14 1 SD -- up SD 1 --
fc9/1 1 auto on fcotAbsent -- -- --
fc9/9 1 auto auto up FL <- Since here we have the FL, the other side is NL
N5k# show interface transceiver
nominal bitrate is 8000 Mbit/sec <- installed SFP is 8Gbps
basic id fields (bytes 0-63)
To verify all the FLOGI processes on the Nexus (just the Local Devices connected to the Switch directly) and check all the Disks/Servers (including the FCID, Port Name or pWWN and Node Name or nWWN) that have done the Login to the Nexus/MDS Switch:
Nexus-5k# show flogi database
INTERFACE VSAN FCID PORT NAME NODE NAME
sup-fc0 2 0xb30100 10:00:00:05:30:00:49:63 20:00:00:05:30:00:49:5e
fc9/13 1 0xb200e2 21:00:00:04:cf:27:25:2c 20:00:00:04:cf:27:25:2c
fc9/13 1 0xb200e1 21:00:00:04:cf:4c:18:61 20:00:00:04:cf:4c:18:61
fc9/13 1 0xb200d1 21:00:00:04:cf:4c:18:64 20:00:00:04:cf:4c:18:64
fc9/13 1 0xb200ce 21:00:00:04:cf:4c:16:fb 20:00:00:04:cf:4c:16:fb
fc9/13 1 0xb200cd 21:00:00:04:cf:4c:18:f7 20:00:00:04:cf:4c:18:f7
Total number of flogi = 6.
To configure the TE (trunk extension) Port, you need to set the port mode to “E”, and set the Trunking ON and define the allowed VSAN. On MDS you also need to set the Rate-Mode to DEDICATED:
(config-if)# switchport mode e
(config-if)# switchport trunk mode on
(config-if)# switchport trunk allowed vsan [add] X
(config-if)# switchport rate-mode dedicated
To configure the Domain ID:
(config)# fcdomain domainid 0x51
NPIV (N_Port ID Virtualization) is used to assign multiple FCIDs to a single N_Port.
Normally, an N_Port would have a single N_Port_ID associated with it; this N_Port_ID is a 24-bit address assigned by the Fibre Channel switch during the FLOGI process. The N_Port_ID is not the same as the World Wide Port Name (WWPN), although there is typically a one-to-one relationship between WWPN and N_Port_ID. Thus, for any given physical N_Port, there would be exactly one WWPN and one N_Port_ID associated with it.
NPIV allows multiple FCIDs to be assigned to a single N port on a host (allows a single physical N_Port to have multiple WWPNs). This means that different VMs using different WWNs can use the same HBA.
The Fibre Channel switch must also support NPIV, as the F_Port on the other end of the link would “see” multiple WWPNs and multiple N_Port_IDs coming from the host and must know how to handle this behavior.
NPV (N _port Virtualization) is the extension of the NPIV, and they define the NP ports (N Proxy) and the Edge switch doesn’t require the separate Domain ID to receive connectivity to the Fabric. The device aggregates the localy connected N ports into the one or more Uplinks to the Core Switch, which should support NPIV.
Well explained NPV and NPIV: http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/
Credit Based Flow Control ensures that the sender keeps track of the free bufferson the receiver side. It´s a very conservative strategy to ensure we don´t drop frames (VERY important in the Storage world). Fiber Channel defines both Flow Controls:
- Port to Port Flow Control, which is Buffer to Buffer Flow Control
- End to End Flow Control
Buffer-to-Buffer (B2B) Credit is negotiated to ensure that the FC flow is lossless. Credit gets decreased with each packet placed on the wire, and gets increased every time “transfer ready” (a sort of an ACK) is received. There is no similarity with this in the Ethernet world.
FSPF (Fabric Shortest Path First) is a FC link-state protocol similar to OSPF, and it does the routing based on the Domain ID.
RSCN (Registered State Change Notification) is a message that is communicated to the End Hosts when something changes. This in most cases means that one path (SAN A for example) is down, and the other SAN should be used.
Fiber Channel Addressing and Routing
To understand how the FC “routing” work, we need to understand the WWN (Hardware identifier, like a MAC address in the NIC) and FCID, which is actually used for the FC “routing”. Fibre Channel routing does not use the flooding, because the FC routing is built on a separate control plane process (that includes all the LOGIN-s).
FC needs to have the WWN (World Wide Names) assigned to every Port and every Node. This is similar to MAC addresses on the Ethernet. There are:
- Node WWN (nWWN) to identify Devices (every Disk Drive, HBA, Switch has a single unique nWWN)
- Port WWN (pWWN) to identify Ports.
This means that HBA with 2 ports will be assigned one N and two P WWNs.
FC address (Identifies the Switch in the Fabric. It can be assigned Automatically (by a principal Switch) or Manualy), or FC ID has 3 bytes (1 byte for each, Domain ID, Area ID and Port ID):
- Domain ID(8 bits), to identify the Switch in the FC Network, and used for the Storage Routing. There are only 239 available (01 - EF) so you can only have 239 Switches in your FC Network. FSPS is a Storage routing protocol that routes between the Domain IDs. The Principal Switch is the one that assigns the Domain IDs (if we don’t do it manyally), and it´s the Switch with the lowest WWN. You can modify the Principal Switch using the command “fcdomain priority 1 vsan X”.
- Area ID(8 bits), to identify Groups of Ports within a Switch.
- Port ID(8 bits), to identify the individual devices on the particular port.
To configure the Domain ID you can use the Preferred per VSAN (you tell the FCNS the ID you´d like, and if it´s not available – you take the one it assignes), or STATIC (you tell the FCNS the ID you´d like, and if it´s not available – you don’t get the Domain ID and you´re unable to communicate):
(config)# fcdomain domain 0x51 preferred vsan 50
VSAN (Virtual SAN) is a technology that separates the FC Services. VSAN Switches are interconnected with the Trunks that carry various VSANs (TE_Ports, or the Trunking Expansion ports).
The MDS Switches use TE (Trunking Expansion) Ports, and we create the EISL (Enhanced Inter-Switch Links), and these can pass the Tagged VSAN traffic. EISL will also carry the control traffic.
VSANs are like VLANs, and they cannot talk to each other. We need to implement the IVR (Inter VSAN routing) to enable the access from one VSAN to another. This can also be done used the ZONING.
To create a new VSAN and assign it to the FC interface:
(config)# vsan database
(config-db)# vsan 10
(config-db)# vsan 10 interface fc1/1
And add the VSAN to a TRUNK link if needed:
(config)# interface fc1/2
(config-if)# switchport trunk allowed vlan add 10
Zoning is defining which SERVER (Source) can access which STORAGE (Destination Disk), like the bidirectional Access Lists. FCNS provides everyone the WWNs of all the Initiators (Servers) and all the Targets (Disks), and if they wanted to talk to each other – they can. This is what we want to avoid, because we don´t want Windows server to mount the Linux partition, because the Data Corruption will occure. Within the Zoning you associate the WWNs and FCIDs and create aliases.
Default zone policy is to deny. This could be change to permit as:
(config)# zone default-zone permit vsan 1
(config)# system default zone
To activate a zone set, you must first create the zone and a zone set.
FULL zone set: Configured, but not activated, so – it´s not to be used.
ACTIVE zone set: Can be used. The active zone set should be the same on all the Fabric Switches.
N5k# show zoneset active
zoneset name ZoneSet1 vsan 1
zone name zone1 vsan 1
fcid 0x080808
fcid 0x090909
fcid 0x0a0a0a
To configure a zone and assign a zone name, follow these steps:
Step 1: Set a Zone NAME and assign it to a VSAN:
N5k(config)# zone name MYZONE vsan 3
Step 2: Configure a member for the specified zone (MYZONE) based on the type (pWWN, fabric pWWN, FC ID, or FC alias) and value specified.
N5k(config-zone)# member pwwn 10:00:00:23:45:67:89:ab
Fabric pWWN example:
N5k(config-zone)# member fwwn 10:01:10:01:10:ab:cd:ef
FC IDexample:
switch(config-zone)# member fcid 0xce00d1
FC alias example (Aliases can be configured using the “fcalias” command in the global config mode):
switch(config-zone)# member fcalias Payroll
Zone-Set is a set of Zones, and it´s used for separating Initiators from the Targets, or separating the sensitive networks. Zones can overlap, but only one Zone Set can be active at a time. Zoning can be enforced in two ways: soft and hard.
- Soft Zoning, as in Software Based, handled by FCNS (Fibre Channel Name Server). If an end device somehow knows the FC ID of a device outside its zone, it can access that device.
- Hard Zoning, or Hardware Based, where the membership is enforced using the ACLs. As frames enter the switch, source-destination IDs are compared with permitted combinations to allow the frame at wirespeed.
- Zone Membership, which can be enforced using the various parameters.
Single Tier Extended Fabric is made of the N7K or N5K as a “Parent”, and N2K as a fabric extender (FEX) with a function of a ToR (Top of the Rack) Switch. VM-FEX (Fabrix Extender) Technology enables the VM traffic flow to be identified within the entire flow form ToR to the Parent switch. VIC Card, such as UCS P8IE helps with VM-FEX.