Over the last years, some nasty bugs have accumulated with Cisco lightweight APs, involving flash corruption. In the first instances, APs have lost configuration (coming up with default name and role), later some refused to boot or were stuck in a boot loop. That occured at any given time the AP reboots - power outage, software upgrade,… The most common I have seen:
*Mar 1 00:00:30.483: %SYS-5-RELOAD: Reload requested by Init. Reload Reason: Radio FW image download failed ....
There are a number of bugs that resulted in these behaviours, ranging from software version 8.0 to 8.5. Cisco Field Notice: FN - 70330 lists the corresponding bug IDs, software versions and APs (including 1700/2700/3700, which were hit the hardest it seems).
If you are running one of the affected versions, seeing this problems and want to upgrade - Cisco made a Python tool, that tries to fix the flash before you’re upgrading, saving you from stranded APs. You can read how to use it in their document Understanding Various AP-IOS Flash Corruption Issues.
If you already have non-functional APs, they are probably fixable, just not remote. Here’s how you do it:
Since the AP probably isn’t booting anymore, you will have to take it down and attach a serial console to it. Set your console to 9600 8N1 and power the AP.
From the output you can see why it wont boot anymore, probably something along the lines with image download fail. To revive the AP and make him a happy little accesspoint again, press and hold the mode button. Keep it pressed, until you see this line:
button is pressed, wait for button to be released...
Then release the button. You will now boot to a “ap:” prompt:
button pressed for 30 seconds process_config_recovery: set IP address and config to default 10.0.0.1 process_config_recovery: image recovery image_recovery: Download default IOS tar image tftp://255.255.255.255/ap3g2-k9w7-tar.default examining image... DPAA Set for Independent Mode tide_boot_speed = 1000 DPAA_INIT = 0x0 %Error opening tftp://255.255.255.255/ap3g2-k9w7-tar.default (connection timed out)ap: ap:
Now, depending on what exactly went wrong with the flash, I had APs working with as little work as booting the recovery image on the AP:
ap: dir flash: Directory of flash:/ 2 -rwx 12312 private-multiple-fs 3 -rwx 56200 event.log 4 -rwx 116551 event.r0 5 -rwx 210 env_vars 38 drwx 576 ap3g2-rcvk9w8-mx 74 drwx 0 configs 75 -rwx 64 sensord_CSPRNG0 76 -rwx 64 sensord_CSPRNG1 77 -rwx 8 sensord_crashWer0 78 -rwx 100 capwap-saved-config-bak 79 -rwx 99 capwap-saved-config 80 drwx 2496 ap3g2-k9w8-mx.153-3.JC15 145 -rwx 137639 event.r1 19586560 bytes available (21572096 bytes used) ap: boot flash:/ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-mx
The AP reboots with recovery image, connects to controller, downloads the new image, and is happy again.
However, I had APs that needed more attention. In that case I formatted the flash completely, and transferred the recovery image via TFTP.
ap: format flash: Are you sure you want to format "flash:" (all data will be lost) (y/n)?y flashfs: 0 files, 1 directories flashfs: 0 orphaned files, 0 orphaned directories flashfs: Total bytes: 41158656 flashfs: Bytes used: 1024 flashfs: Bytes available: 41157632 flashfs: flashfs fsck took 28 seconds. Filesystem "flash:" formatted ap: set IP_ADDR 10.10.6.233 ap: set NETMASK 255.255.255.0 ap: set DEFAULT_ROUTER 10.10.6.1 ap: tftp_init ap: ether_init Initializing ethernet port 0... Ethernet speed is 1000 Mb - FULL Duplex ap: tar -xtract tftp://10.7.7.7/ap3g2-rcvk9w8-tar.153-3.JH.tar flash: extracting info (274 bytes) ap3g2-rcvk9w8-mx/ (directory) 0 (bytes) extracting ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-mx (229940 bytes).................................................. extracting ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-tx (73 bytes) extracting ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-xx (7313476 bytes)......................[output shortened]......................................... extracting ap3g2-rcvk9w8-mx/info (274 bytes) extracting ap3g2-rcvk9w8-mx/file_hashes (438 bytes) extracting ap3g2-rcvk9w8-mx/final_hash (141 bytes) extracting ap3g2-rcvk9w8-mx/final_hash.sig (512 bytes) extracting ap3g2-rcvk9w8-mx/img_sign_rel.cert (1375 bytes) extracting ap3g2-rcvk9w8-mx/img_sign_rel_sha2.cert (1371 bytes) extracting info.ver (274 bytes) ap: set BOOT flash:/ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-mx ap: boot flash:/ap3g2-rcvk9w8-mx/ap3g2-rcvk9w8-mx Rebooting system to reset DPAA...
The AP reboots, boots the recovery image, connects to controller, downloads correct image, and is a happy little access point again.
With the “fixed” versions of controller software (e.g. 188.8.131.52, or 184.108.40.206) I have not had any problem like this so far. Knock on wood for fixed bugs!