AP Joining Issue to WLC Running 8.0.100.0

In this post I will discuss about the issue faced today while joing AP to WLC version 8.0.100.0.

5 Day before I got a new 2602 AP and Today I tried to connect to my switch in right AP VLAN. I saw that AP got IP address from DHCP pool and WLC IP via DHCP Option 43 and AP start updating the Image from WLC.

I was relaxed that it is working so I will test my Important topic like Auto Anchor, Static IP tunneling & Foreign mapping.

After 1-2 minutes I saw that there was some kind of failure which I never seen, here are the logs:

TestAP#
 *Nov 19 13:37:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:38:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.3 peer_port: 5246
 *Nov 19 13:38:29.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x8D69EB4!
 *Nov 19 13:38:59.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.3:5246
 *Nov 19 13:38:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:39:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.1 peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Nov 19 13:39:00.099: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.10.1:5246
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.1:5246
 TestAP#

After googling I got this: APs mfg in September/October 2014 unable to join an AireOS controller CSCur43050

Description

Symptom:
New Aironet APs with factory installed recovery IOS are able to join the controller 8.0.100.0 and download 15.3(3)JA IOS. But after the AP reload, the APs are unable to join the controller. On the AP, logs similar to the following are seen:

*Oct 16 12:39:06.231: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Oct 16 13:14:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: ***.***.***.*** peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Oct 16 13:14:56.127: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to ***.***.***.***:5246
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to ***.***.***.***:5246

Another symptom of this problem is that the AP may be able to join the 8.0.100.0 controller, download the IOS code, boot up and join the controller OK … but when it goes to upgrade to newer 8.x code, it gets stuck in a loop failing the download.

Conditions:
Seen only with APs that were manufactured in September or October, 2014 – all Aironet APs were affected EXCEPT the 700 series. Seen with WLCs running 8.0.100.0 or an 8.0.100.x special.

If the WLC was manufactured in September 2014, or later (i.e. has a SHA2 MIC), then the first symptom is seen, i.e. the AP joins the 8.0.100 WLC, downloads the image, but then fails to rejoin.

If the WLC was manufactured before September 2014 (i.e. does not have a SHA2 MIC), then the second symptom is seen, i.e. the AP can join the 8.0.100 WLC OK, but then will fail download during a subsequent upgrade.

Also seen with new APs trying to join a controller running IOS-XE 3.6.0 (15.3(3)JN k9w8 image.) (Track CSCur50946 for the IOS-XE fix)

Workaround:
Downgrade to AireOS 7.6.130.0, or to IOS-XE 3.3, if the APs are supported in the earlier code.

Further Problem Description:
This problem affects only APs that were manufactured with incorrect SHA2
certificates. APs with only SHA1 certificates are not affected. To determine
whether an AP is affected, use the following AP exec commands (while the AP
has a 15.3(3)/8.0 image installed):

1. Check for the presence of a SHA2 Parameter Block:

ap#test pb display

if the output of this command includes:

SHA2 Parameter Block Doesn’t have any Records

then this AP is not affected. If the output of this command shows

Display of the SHA2 Parameter Block

then

2. See whether a correct SHA2 certificate is present:

ap#show crypto pki trustpoints | include SHA2

if there is no valid SHA2 certificate, then this will show no output.
If there is a valid SHA2 cert, this will show:

cn=Cisco Manufacturing CA SHA2

Only APs which *do* have a SHA2 Parameter Block and which *do not* have
a valid SHA2 certificate are affected by this bug.

The problem symptoms will vary according to whether or not the WLC has a
SHA2 certificate installed. To verify this, use the following command on
the AireOS CLI:

Cisco Controller) >show certificate all
and look for:
Certificate Name: Cisco SHA2 device cert

Then I downgraded my WLC to version 7.6.130.0 and it worked.
So this just a small post, it may help those who is/will get this kinda problem.