AP Joining Issue to WLC Running 8.0.100.0

In this post I will discuss about the issue faced today while joing AP to WLC version 8.0.100.0.

5 Day before I got a new 2602 AP and Today I tried to connect to my switch in right AP VLAN. I saw that AP got IP address from DHCP pool and WLC IP via DHCP Option 43 and AP start updating the Image from WLC.

I was relaxed that it is working so I will test my Important topic like Auto Anchor, Static IP tunneling & Foreign mapping.

After 1-2 minutes I saw that there was some kind of failure which I never seen, here are the logs:

TestAP#
 *Nov 19 13:37:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:38:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.3 peer_port: 5246
 *Nov 19 13:38:29.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x8D69EB4!
 *Nov 19 13:38:59.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.3:5246
 *Nov 19 13:38:59.999: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Nov 19 13:39:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.10.1 peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Nov 19 13:39:00.099: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.10.1:5246
 *Nov 19 13:39:00.099: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.1:5246
 TestAP#

After googling I got this: APs mfg in September/October 2014 unable to join an AireOS controller CSCur43050

Description

Symptom:
New Aironet APs with factory installed recovery IOS are able to join the controller 8.0.100.0 and download 15.3(3)JA IOS. But after the AP reload, the APs are unable to join the controller. On the AP, logs similar to the following are seen:

*Oct 16 12:39:06.231: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
 *Oct 16 13:14:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: ***.***.***.*** peer_port: 5246Peer certificate verification failed FFFFFFFF
 *Oct 16 13:14:56.127: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to ***.***.***.***:5246
 *Oct 16 13:14:56.127: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to ***.***.***.***:5246

Another symptom of this problem is that the AP may be able to join the 8.0.100.0 controller, download the IOS code, boot up and join the controller OK … but when it goes to upgrade to newer 8.x code, it gets stuck in a loop failing the download.

Conditions:
Seen only with APs that were manufactured in September or October, 2014 – all Aironet APs were affected EXCEPT the 700 series. Seen with WLCs running 8.0.100.0 or an 8.0.100.x special.

If the WLC was manufactured in September 2014, or later (i.e. has a SHA2 MIC), then the first symptom is seen, i.e. the AP joins the 8.0.100 WLC, downloads the image, but then fails to rejoin.

If the WLC was manufactured before September 2014 (i.e. does not have a SHA2 MIC), then the second symptom is seen, i.e. the AP can join the 8.0.100 WLC OK, but then will fail download during a subsequent upgrade.

Also seen with new APs trying to join a controller running IOS-XE 3.6.0 (15.3(3)JN k9w8 image.) (Track CSCur50946 for the IOS-XE fix)

Workaround:
Downgrade to AireOS 7.6.130.0, or to IOS-XE 3.3, if the APs are supported in the earlier code.

Further Problem Description:
This problem affects only APs that were manufactured with incorrect SHA2
certificates. APs with only SHA1 certificates are not affected. To determine
whether an AP is affected, use the following AP exec commands (while the AP
has a 15.3(3)/8.0 image installed):

1. Check for the presence of a SHA2 Parameter Block:

ap#test pb display

if the output of this command includes:

SHA2 Parameter Block Doesn’t have any Records

then this AP is not affected. If the output of this command shows

Display of the SHA2 Parameter Block

then

2. See whether a correct SHA2 certificate is present:

ap#show crypto pki trustpoints | include SHA2

if there is no valid SHA2 certificate, then this will show no output.
If there is a valid SHA2 cert, this will show:

cn=Cisco Manufacturing CA SHA2

Only APs which *do* have a SHA2 Parameter Block and which *do not* have
a valid SHA2 certificate are affected by this bug.

The problem symptoms will vary according to whether or not the WLC has a
SHA2 certificate installed. To verify this, use the following command on
the AireOS CLI:

Cisco Controller) >show certificate all
and look for:
Certificate Name: Cisco SHA2 device cert

Then I downgraded my WLC to version 7.6.130.0 and it worked.
So this just a small post, it may help those who is/will get this kinda problem.

Advertisements

24 Comments

  1. Hey Rasika,

    For unknown reason Cisco does not want to make life “easier” for some people. What I’m trying to say is: How can a user identify if the batch of newly delivered APs are or can be affected by this bug.

    The easiest way is to determine by the serial number of the AP/APs. Serial numbers affected are the following:

    fff1837xxxx up to fff1844xxxx

    APs with serial numbers above are made between September 2014 up to the end of October 2014.

    Hope this helps everyone reading this.

    Best Regards/Leo

    PS: I’m going to put a “fail” at the CBT about this Bug ID.

  2. We bought 2 AIR-CAP1702I-E-K9 access points with a Cisco 2504 Wireless Controller.

    The AIR-CAP1702I-E-K9 needed “IOS 8.0.100.0”. So we could not downgrade to 7.6.130.0.

    I used this command on the controller: config ap dtls-wlc-mic sha1

    After this command, the accesspoint were able to download the config and join the controller.

  3. Hi guys. I´m facing a problem:
    “AP has SHA2 MIC certificate – Using SHA2 MIC certificate for DTLS.”
    My virtual controller has a version: 8.1x.
    Anyone has a suggestion/ facing these problem?

    Thanks a lot!

  4. Happened to me when the AP came pre-configured in bridge mode instead of local mode from the factory (had this with 1552’s and just recently 1532’s).

    The way I fixed it (while running 8.120.0) is manually add the AP’s mac address (located on the sticker on the AP chassis) to the security/mac filtering area of the controller to force the controller to allow it to connect. It will still use SHA2.

  5. hi,
    yes i had mesh image AP so i manually add the AP’s mac address under security -> mac filtering area of the controller to force the controller to allow it to connect and now it works fine.

    Thank’s a lot.
    Best Regards

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s