Cisco

This is a follow up to my Post IPSK on Cisco without ISE but FreeRADIUS. The FreeRADIUS server has already been running in the state since that post, and I will show what has to be done specific to the configuration when working with Cisco Meraki APs.

Meraki’s IPSK comes in two flavors: with RADIUS server and without.

The “without” option lets you configure multiple PSKs that are valid to every client that knows them. Numerous vendors support this system, and the AP basically tries all valid PSKs on client join to find the one that matches. The issue with this approach is that it can’t work with WPA3. With WPA3 (and with that, I mean WPA3-SAE, not 802.1X) the AP has to know the correct secret and has no chance of trying multiple possibilities. One vendor claims to have worked around that, but there have yet to be any technical details.

This leaves us with “Identity PSK with RADIUS” - because we “pre-register” our clients with MAC-address in our RADIUS-Server, the AP can query it for the correct passphrase as soon as it receives a join from a client, which makes this possible with WPA3-SAE. But, as of today, Meraki APs (version MR 30.6) do not support this yet. Ciscos 9800 controller supports this starting from 17.9.2, so we can hope that Meraki will support this soon too. The security issue itself is not that big, because some of the weaknesses of WPA2-PSK are a bit offset by IPSK, but missing WPA3 support also means no IPSK in 6 GHz, as WPA3 is required for the 6 GHz band.

Having said the theoretical parts of this post, let’s get to configuring.

Just a small tidbit: I recently received a request to quickly alert when a specific MAC address joins our 802.1X WLAN and gives its approximate location.

There are a few possible ways to do this - alert on logs on RADIUS server, any MAC table, log concentrators, or SIEM solutions, but getting this quick and easy turned out to be very straightforward and uncomplicated using only out-of-the-box solutions on the Cisco 9800 controller, specifically EEM applets.

To see what we’re after, let’s look at a standard logline when a client joins our 802.1X SSID:

Mar 24 10:18:33.637: %CLIENT_ORCH_LOG-6-CLIENT_ADDED_TO_RUN_STATE: Chassis 1 R0/0: wncd: Username entry (thisisausername) joined with ssid (thisisanssid) for device with MAC: aabb.ccdd.1234

As we are focused on the MAC address, we see that we have it right there after the join at the end of the log line.

EEM is the embedded event manager and can trigger on many things, but one thing is to match on loglines and execute specific commands in the event of a match.

Note: This article is part 2 of a 4-part troubleshooting series, with more in-depth information about a TEN talk at WLPC.
Part 1 - The RADIUS connection
Part 3 - Breaking other Wi-Fi
Part 4 - The suddenly weaker Wi-Fi
Video recording from WLPC Prague

Incoming Ticket: Terrible connection using Zoom on Wi-Fi. Works for a while, then unusable for a minute. Building is almost empty.

This is of course pretty broad, so to get an overview on what I checked:

• The building was new, and got 802.11ac Wave2 Wi-Fi, planned and validated - so it should not be a coverage issue
• Also happens when the building is almost empty, so capacity should not be an issue
• Multiple client typed affected, so probably not a client driver issue
• Log check revealed that there is no unexpected roaming and no channel changes
• It was validated that it worked fine on the wired network, so only wireless was affected
• On the whole campus, only this building had this issue
• It was hard to reproduce, sometimes it took hours, sometimes it was multiple times in an hour

Note: This article is part 1 of a 4-part troubleshooting series, with more in-depth information about a TEN talk at WLPC.
Part 2 - Zoom issues
Part 3 - Breaking other Wi-Fi
Part 4 - The suddenly weaker Wi-Fi
Video recording from WLPC Prague

Incoming Ticket: I’m a student from Institution X in the same town. I am visiting your library and I can’t connect to eduroam here. It works fine on the campus of Institution X.

If you are not familiar with eduroam, it is a worldwide network of Universities, granting each other wireless access.

eduroam is built in a tree-like structure:

What is IPSK?

The concept is not new, other wireless vendors had this or similar features for a while (often named PPSK, DPSK, or MPSK, all with a bit different functionality), but some time ago Cisco released “Identity PSK”, or short, “IPSK”. It has been available on AireOS since version 8.5 and on the 9800 controller since the beginning (16.10) - I did my first experiments with it on AireOS 8.5 and made it into a new service on our campus on 16.10 back then.

As the name suggests, it is a PSK authentication, but not every client on the SSID has to have the same PSK. You can group them by department, or type, or can even give every single device its own PSK. Additionally, with dynamic VLAN assignment (which has been possible forever), this leads to great grouping and security zones. This is why this solution is so great for IOT:

• They can typically not use 802.1X
• You want to separate them, e.g. cameras, sensors, displays, weird printers
• You don’t want to use a separate SSID for every type, because of SSID overhead

IPSK makes it possible to have this on one SSID:

The solution is typically to use a Cisco ISE to configure the auth side of the equation. But it does not have to be ISE - any RADIUS implementation that can send some attributes will work.

This is why I chose to show you how it is done in the great open-source RADIUS server FreeRADIUS.

My two 9800-40’s run in HA. They have been running 16.10.1e for some time now, and I wanted to upgrade to the new suggested 16.11.1c or directly to 16.12.1 to test it out, as long as we are still in the summer holidays and there isn’t that much going on. A newer release is definitely needed in a few weeks, when my first Wi-Fi 6 (or 802.11ax, depending on marketing flavour) APs arrive to go online.

The official Cisco documentation (found here) for upgrading an HA pair is a little lacking:

Yes, this is it. The whole section. Not only is there not much detail, but it is in “old IOS style” upgrade (now called “bundle mode”), setting the .bin file in the boot variable. With this, it does not get copied automatically to the standby-box, there is no rollback, and, if I read correctly, you can’t AP image predownload that way.

NOTE: Documentation has been improved vastly since then. This shows how early I was with the C9800 - I was told one of the first in Europe and the first in Austria.

IOS XE “new style” (officially called “install mode”) works just as the release notes state, and, even if it does not say, it does everything on your standby-box too.

I downloaded the new .bin file from the Cisco download section and copied it to the bootflash on my primary box. Then you use the “install add file” command:

In my last post I wrote about my first findings in trying to implement the C9800 in my wireless network.

As this was a “try&buy”, and things were looking good - not perfect, but very reasonable and with great prospects - I went from “try” to “buy”, and started migrating.

So I want to share some more insight about the process, about issues and about my setup. As of the time I began writing this post, 16.10.1e was the lastest released software. During writing, 16.11.1b was released, that includes some missing features (mDNS, CAPWAP over NAT/PAT,…) and fixes multiple bugs. As I am not yet running this software, this post is about 16.10, and if I discover new insights, I will make a new post about 16.11.

NOTE: This was a very early look on the C9800, with software version 16.10. A lot has changed over 16.11, 16.12 an on to the 17 train.

One of my two 9800-40 (in HA-SSO), mounted into the destination Rack. Connection as Multichassis-Etherchannel (20G for now, option to 40G) and 1G HA link, in different buildings, 2 PSUs on different circuits (one with UPS, one without), one building includes diesel generator.

• I went full IPv6 in the wireless infrastructure. Gone are the days of RFC1918 space, weird routes and NAT. The controller itself does have an IPv4 address in addition to the IPv6, as the RADIUS servers are (not yet) IPv6, and there are some APs outside of the network, that do not have an IPv6 connection. But my management, and the CAPWAP connection between AP and controller is IPv6 only. The APs do not even have an IPv4 address.

With a lot of fanfare, Cisco introduced their newest wireless controller platform, the Catalyst 9800, late last year. They are still commited to AireOS (the software the classic WLCs run - like the 5508/5520 or the giant 8540), but every time you talk to someone from Cisco it is clear that they want the new Catalyst 9800 - based on IOS-XE - to succeed.

Just to be clear, this is not a converged access controller as the 5760, that also ran on IOS-XE - this was my first thought, but the blog posts from Dave from wifireference.com or Phil from networkphil.com were assuring that this is not the case.

No matter what flavor you like - as VM as the 9800-CL or as hardware appliance as 9800-40 and 9800-80 - Cisco has got you covered.

As my WiSM2s are ageing, get no new software and are about to run out of licenses, I gave this a shot, and ordered a pair of 9800-40s, to run in HA. Sessions and Meet-the-Engineers at Cisco Live! in Barcelona were reassuring. Our Cisco partner arranged a speedy delivery - despite pretty long waiting times at the moment - and also got me a contact directly at Cisco. This has been proven to be very valuable - someone you can direct questions to that arise all the time, little issues you run into, or defects you find. There is already a good amount of documentation online, but it is still lacking in detail.

So, I had the C9800-40s now for about 6 weeks, and learned a LOT in this time:

This might be some old news for most, but I only have a few outdoor APs and touch one every few years. But since I just got a new Cisco Aironet 1542I and was surprised again, I’d thought I document this for me. This also applies not only to the 1542, but 1532, 1552 and probably the whole outdoor portfolio, on Cisco wireless controllers.

The simple issue is: the AP is out of the box in “bridge mode”, and the controller wont let it join just that way without authenticating it. To fix, you have two possibilities: authenticate it on the controller, or switch it from bridge mode to local mode:

Over the last years, some nasty bugs have accumulated with Cisco lightweight APs, involving flash corruption. In the first instances, APs have lost configuration (coming up with default name and role), later some refused to boot or were stuck in a boot loop. That occured at any given time the AP reboots - power outage, software upgrade,… The most common I have seen:

*Mar 1 00:00:30.483: %SYS-5-RELOAD: Reload requested by Init. Reload Reason:
Radio FW image download failed ....

There are a number of bugs that resulted in these behaviours, ranging from software version 8.0 to 8.5. Cisco Field Notice: FN - 70330 lists the corresponding bug IDs, software versions and APs (including 1700/2700/3700, which were hit the hardest it seems).

If you are running one of the affected versions, seeing this problems and want to upgrade - Cisco made a Python tool, that tries to fix the flash before you’re upgrading, saving you from stranded APs. You can read how to use it in their document Understanding Various AP-IOS Flash Corruption Issues.

If you already have non-functional APs, they are probably fixable, just not remote. Here’s how you do it: