6 weeks with Ciscos newest wireless controller, the Catalyst 9800

Cisco

With a lot of fanfare, Cisco introduced their newest wireless controller platform, the Catalyst 9800, late last year. They are still commited to AireOS (the software the classic WLCs run - like the 5508/5520 or the giant 8540), but every time you talk to someone from Cisco it is clear that they want the new Catalyst 9800 - based on IOS-XE - to succeed.

The two Catalyst 9800-40 Controllers in my lab

Just to be clear, this is not a converged access controller as the 5760, that also ran on IOS-XE - this was my first thought, but the blog posts from Dave from wifireference.com or Phil from networkphil.com were assuring that this is not the case.

No matter what flavor you like - as VM as the 9800-CL or as hardware appliance as 9800-40 and 9800-80 - Cisco has got you covered.

As my WiSM2s are ageing, get no new software and are about to run out of licenses, I gave this a shot, and ordered a pair of 9800-40s, to run in HA. Sessions and Meet-the-Engineers at Cisco Live! in Barcelona were reassuring. Our Cisco partner arranged a speedy delivery - despite pretty long waiting times at the moment - and also got me a contact directly at Cisco. This has been proven to be very valuable - someone you can direct questions to that arise all the time, little issues you run into, or defects you find. There is already a good amount of documentation online, but it is still lacking in detail.

So, I had the C9800-40s now for about 6 weeks, and learned a LOT in this time:

Webinterface greets me with version 16.10

NOTE: This was a very early look on the C9800, with software version 16.10. A lot has changed over 16.11, 16.12 an on to the 17 train.

  • The quality and features are pretty good. As of software version 16.10.1e, there is not yet feature parity with AireOS, and there are bugs or annoyances to run into. But, as I said - Cisco is very committed to the platform, they are working hard on feature parity (e.g. mDNS, or APs behind NAT) and I had very long WebEx calls with Cisco engineers about defects, with a lot of debugs and packet captures to make sure to find the underlying problem and get it fixed ASAP.

  • My controllers are still in test (with a bunch of APs and clients, including myself), but I expect to take the first buildings on my campus productive on the C9800s in the next few weeks. Maybe there will be a lot more to talk about then ;-)

  • Speaking of debugs - there have been a lot of improvements. The controller is basically always tracing to the internal SSD, and with a few debug commands you can filter through a lot of different log files for the process, MAC or IP you would like to debug. This comes in very handy. The new debug features are documented in this Cisco article.

  • I really like the controller hardware. The 5520 and 8540 are just servers, running the software, and are not cut out for shallow network racks. The C9800-40 and -80 are hardware built for the job, and though their name is “Catalyst”, they are not switches - they are built on a Cisco routing platform (they have a QFP - quantum flow processor, and the first prompt booting up on the console is “Router>"). The C9800-40 is under 50cm deep - compared to the 5520 with over 78cm…

  • One huge deal that makes the transition for me a lot easier: You probably need no VLAN interfaces with IP address on them, like on AireOS. To put a client in a specific VLAN, you only have to make the VLAN known to the controller - and off you go. An SVI is only necessary, if you need to proxy DHCP or use the interface to generate other traffic from (e.g. source RADIUS traffic from there). So I will end up with dozens of VLANs, but only 2 SVIs (one for management, one to source specific RADIUS requests from).

  • The config model has been completely overhauled - it is very different to configure WLANs, VLANs, RF profiles and much more to an AP. To help, the webinterface has two “wireless setup” types - “Basic” and “Advanced”. To be honest: I did not use that at all. It just feels like an easy entry, but I want to name my profiles, tags and so on… and I already have a setup. My recommendation: make yourself familiar with the new config model. I found that this slide from Cisco Live! helps a lot:

Config model of AireOS vs. C9800 compared

Read that, try to understand it, and when configuring, look at it again (because I was just as confused as before) - and then it will make sense. I configured all my WLAN, policy, site and RF tags and profiles by hand. It works. It is unfamiliar at first, but I think I will be better off that way.

  • You can make rules for APs to get specific tags (for example with regex on AP name), and one big thing: AP tags are in config. You can put it into config while the AP is still in the box.
ap 00BE.75xx.yyzz
policy-tag AP-on-campus-policy
rf-tag sparse-campus-density
site-tag HF

When the AP boots up and connects to the controller, tags (and therefore WLANs/VLANs) will be set accordingly.

  • Speaking of site-tags, make use of those. They might not seem that important without flexconnect, but the controller runs multiple wlc-processes, and the APs get distributed on these processes by site tag. So APs that are close together - e.g. same buidling, probably with lots of roaming events, stay in the same process; APs in next building is in different process. These can still communicate cross-process, but grouping them by site tags limits this and distributes AP load over multiple processes.

  • The default policy tag makes no sense at all. I don’t know what Cisco was thinking there. It is not editable, and it binds every SSID (or the first 16) to the one (1) default policy profile. The policy profile is where you set the VLAN. So in the default policy tag, your first 16 SSIDs will be bound to whatever you set in the default policy profile. Which is probably not what you want. So an unconfigured AP will broadcast the first SSIDs, and puts users in all of these SSIDs in the one VLAN you set. You can of course bind your WLAN profiles to a policy profile of your choice in own policy tags, and bind these to APs - but unconfigured, this is what the AP gets. The guides say about that:

Note: You can modify all the default settings except for the Default Policy Tag. The Default Policy Tag automatically links any SSID with a WLAN ID from 1 to 16 to the default policy profile and those links cannot be modified.

In AireOS, an unconfigured AP also broadcasted the first 8 SSIDs - but you set the interface in the WLAN config. So even if you did not want to broadcast all these SSIDs, at least the users were put into a correct VLAN, not all in one. The solution to this is: either have no unconfigured APs, or just deactivate the default policy profile. I found that there is an “off” switch in there, and an unconfigured AP will then just sit there, connected to the controller, doing nothing - so like an “out of box off” setting. This is not mentioned in any guides, so I hope this helps someone.

Switching default policy profile off

  • Mobility to old AireOS controllers: there are versions, that can do IRCM (form mobility tunnels between C9800 and AireOS) on AireOS, but be aware: the WiSM2 is not one of them. 8.8 (where MR1 could do IRCM) does not exist for the WiSM2, and 8.5MR4special (which you need to get from Cisco directly and has IRCM baked into 8.5.140) was not built for WiSM2. Luckily, I have an CT3504 for migration, so it works that way with 8.5MR4special.

  • HA SSO: There is now additionally to the copper HA port an SFP slot, so fiber connections are possible. But be aware, really read the SFP compatibility notes - an GLC-SX-MM is not the same as an GLC-SX-MMD. The latter works, the first one does not. Also, no passive twinax-cables in the 10G SFP+ ports (e.g. SFP-H10GB-CU1M), only the active kind (e.g. SFP-10G-AOC1M), or go with MMF or SMF.

SFP HA Ports connected with MMF

  • Even if you don’t want to automatically migrate your AireOS config to C9800 - the config migration tools are of some help. Some things are just very well hidden, and the migrated config can give you a hint on how to configure stuff. For example: local MAC address filter on an SSID. I probably would have never figured that out on my own. It involves an AAA authorization group, an AAA attribute list and local users that will be bound to that list.
username 000387a491e2 mac aaa attribute list attrlist_WS-internal

  • The webinterface is still a little lacking in information and cross-information. Top WLANs, AP performance, client load top APs - this is not there yet. Why does this only show WLAN ID and not name, why is the username not there, and so on. Of course you can get the info, or even use Prime which does a lot more, but the webinterface is definitely a little behind AireOS. It looks much prettier though ;-)

  • The coloured config diff when saving is godsend :-) I love this.

    Colored diff in action when saving

  • Long missed feature (even meraki can do this since forever) - show the PSK of a WLAN.

    Showing PSK with a single click

  • The CLI is a lot more useable than on AireOS. I never really liked the AireOS CLI, most of the stuff needed multiple lines of config, and these were not connected by context but most of the time by IDs and it got confusing and complicated fast. IOS-XE is much better in that regard, and I use the CLI extensively.

  • If you make extensive use of SNMP, then you are in for… whatever the opposite of a treat is. Since it is no longer AireOS, a lot of OIDs do not work anymore. Also, there is not yet a list of supported MIBs online to look for alternative OIDs. Not ideal. I tried to rewrite a lot of monitoring and checks to the new & fancy netconf-yang to make myself familiar with it - and it works great so far, but also there are some differences between “vanilla” IOS-XE and the WLC IOS-XE, when there was some rewrite necessary (looking at you, radius-stats). Probably on the list to get feature complete in the next versions, but still something good to know.

  • When migrating APs from AireOS to the C9800, a lot of stuff configured on the AP stays. AP name, description, manual set channels and power… all stay. What does not stay is of course the AP group, because there is no analog on the C9800. You can - as written above - configure the tags beforehand though.

  • APs do not have to reload when changing tags. The do disconnect shortly, but there is no lengthy reboot.

So far I really like it. Of course it is not perfect yet and there are some annoyances, but like some of us I really cannot say that I like AireOS and the C9800 is really a breath of fresh air. I still have a few issues open and will work through them with Cisco, but the complete migration will be probably in the next weeks. I will make a new post in case I have new tips or pitfalls… or praises :-)