AP Joining Issue to WLC Running 8.0.100.0

In this post I will discuss about the issue faced today while joing AP to WLC version 8.0.100.0.

5 Day before I got a new 2602 AP and Today I tried to connect to my switch in right AP VLAN. I saw that AP got IP address from DHCP pool and WLC IP via DHCP Option 43 and AP start updating the Image from WLC.

I was relaxed that it is working so I will test my Important topic like Auto Anchor, Static IP tunneling & Foreign mapping.

After 1-2 minutes I saw that there was some kind of failure which I never seen, here are the logs:

TestAP#
 *Nov 19 13:37:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:38:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.3 peer_port: 5246
 *Nov 19 13:38:29.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x8D69EB4!
 *Nov 19 13:38:59.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.3:5246
 *Nov 19 13:38:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:39:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.1 peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Nov 19 13:39:00.099: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.10.1:5246
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.1:5246
 TestAP#

After googling I got this: APs mfg in September/October 2014 unable to join an AireOS controller CSCur43050

Description

Symptom:
New Aironet APs with factory installed recovery IOS are able to join the controller 8.0.100.0 and download 15.3(3)JA IOS. But after the AP reload, the APs are unable to join the controller. On the AP, logs similar to the following are seen:

*Oct 16 12:39:06.231: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Oct 16 13:14:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: ***.***.***.*** peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Oct 16 13:14:56.127: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to ***.***.***.***:5246
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to ***.***.***.***:5246

Another symptom of this problem is that the AP may be able to join the 8.0.100.0 controller, download the IOS code, boot up and join the controller OK … but when it goes to upgrade to newer 8.x code, it gets stuck in a loop failing the download.

Conditions:
Seen only with APs that were manufactured in September or October, 2014 – all Aironet APs were affected EXCEPT the 700 series. Seen with WLCs running 8.0.100.0 or an 8.0.100.x special.

If the WLC was manufactured in September 2014, or later (i.e. has a SHA2 MIC), then the first symptom is seen, i.e. the AP joins the 8.0.100 WLC, downloads the image, but then fails to rejoin.

If the WLC was manufactured before September 2014 (i.e. does not have a SHA2 MIC), then the second symptom is seen, i.e. the AP can join the 8.0.100 WLC OK, but then will fail download during a subsequent upgrade.

Also seen with new APs trying to join a controller running IOS-XE 3.6.0 (15.3(3)JN k9w8 image.) (Track CSCur50946 for the IOS-XE fix)

Workaround:
Downgrade to AireOS 7.6.130.0, or to IOS-XE 3.3, if the APs are supported in the earlier code.

Further Problem Description:
This problem affects only APs that were manufactured with incorrect SHA2
certificates. APs with only SHA1 certificates are not affected. To determine
whether an AP is affected, use the following AP exec commands (while the AP
has a 15.3(3)/8.0 image installed):

1. Check for the presence of a SHA2 Parameter Block:

ap#test pb display

if the output of this command includes:

SHA2 Parameter Block Doesn’t have any Records

then this AP is not affected. If the output of this command shows

Display of the SHA2 Parameter Block

then

2. See whether a correct SHA2 certificate is present:

ap#show crypto pki trustpoints | include SHA2

if there is no valid SHA2 certificate, then this will show no output.
If there is a valid SHA2 cert, this will show:

cn=Cisco Manufacturing CA SHA2

Only APs which *do* have a SHA2 Parameter Block and which *do not* have
a valid SHA2 certificate are affected by this bug.

The problem symptoms will vary according to whether or not the WLC has a
SHA2 certificate installed. To verify this, use the following command on
the AireOS CLI:

Cisco Controller) >show certificate all
and look for:
Certificate Name: Cisco SHA2 device cert

Then I downgraded my WLC to version 7.6.130.0 and it worked.
So this just a small post, it may help those who is/will get this kinda problem.

Advertisements

Understand Access Point IOS Images

All Cisco Access Points and Bridges are normally shipped with IOS, except Office Extend AP602. Some of the old AP’s don’t run IOS (Example: Aironet 340, 1000 Series AP).

Normally Cisco AP IOS is distributed as .Tar file.

The nomenclature of the image follows as or The IOS image names include the following components:

platformfeatureset-tar.version.tar

Platform: The access point hardware model or family supported by the image

Example:

ap1g1 – 700 series
ap1g2 – 1600 series
ap1g3 – 1530 series
ap3g2 – 3700/3600/2600 series (3700 supported beginning with 15.2(4)JB)
ap3g1 – 3500/1260 series
ap802 – AP embedded in 819, 812, 886VA-W/887VA-W, C88x and C88x routers
ap801 – AP embedded in 861W, most 88xW, and 1911W routers
c1520 – 1550 and 1520 series mesh APs
c1410 – BR1410
c1310 – BR1310
c1250 – 1250 series APs
c1240 – 1240 series APs
c1200 – 1200 series (1200/1210/1220/1230)
c1140 – 1140 and 1040 series APs
c1130 – 1130 series APs
c1100 – 1100 series APs (i.e. the AP1121)
c520 – 521 AP
c350 – 350 series APs

Featureset: The set of software features supported by the image – one of:

k9w7 – autonomous (or “site survey”) IOS
k9w8 – full lightweight IOS (this is what is bundled in the WLC .aes image, and is factory installed on “mesh” APs)
rcvk9w8 – lightweight recovery image – this is factory installed on lightweight APs, unless a “mesh” image is specified; it lacks radio firmware

Version– the IOS version

There is a 1:1 mapping between the lightweight IOS software version (such as 12.4(23c)JA) and the CUWN version (such as 7.0.98.0).

See the Cisco Wireless Solution Software Compatibility Matrix

Example:

c1240k9w7-tar.124-25d.JA1.tar

  • Platform: c1240: 1240 series AP
  • Featureset: k9w7: autonomous IOS
  • Version: 124-25d.JA1: 12.4(25d)JA1

As AP IOS is always distributed as a tar file, the AP cannot directly execute such a file (thus, if you were to copy c1240k9w7-tar.124-25d.JA1.tar directly onto AP flash, and then try to boot it, this could not work.)  The tar file contains, in addition to the IOS image proper, the radio firmware files, the HTML GUI files (if present), and various other files.

The AP IOS tar file must be unbundled into AP flash using the archive exec command (this is done in an automated fashion when a lightweight AP is upgraded after joining a WLC.)

After unbundling, the IOS image itself be in a file called flash:/platformfeatureset-mx.version/platformfeatureset-mx.version for example, flash:/c1240k9w7-mx.124-25d.JA1/c1240k9w7-mx.124-25d.JA1.  The AP is configured to boot this image if the bootloader BOOT environmental variable is set accordingly.