Wednesday, October 24, 2012

A quick way to drop all your telephones

Changing your switches from Spanning Tree Protocol (STP) to Rapid Spanning Tree Protocol (RSTP) is a Good Thing. Convergence in case of dropped redundant links goes from 50 seconds to 6 seconds.

Here is what I programmed on  my core/distribution switch, a Cisco 3750:

# en
# conf t
# spanning-tree vlan 1-4094 root primary
# spanning-tree mode rapid-pvst
(Complete and utter silence from the switch.
Angry noises from two cubicles over.)

Cold sweat. Fear.

Friends, make the change during a scheduled maintenance period, no matter the pressure. The change will take out the network for about 10s; not enough to disturb users surfing for data, but more than enough time to disconnect VoIP phone calls.

Tuesday, October 23, 2012

Yes, but what do they really want?

One of my favourite personal complaints is that I cannot read minds.  Of course, if I could, I would be tarred, feathered and railed into radioactive slime for a slow 'orrible death for, in this Facebook age, who wants their last barrier violated by the sys admin?

Recently, I got a good chewing out from On High because a network improvement plan I was executing was seemingly making things worse. Whether or not this was true is irrelevant; On High felt it was true and any defense is, well, just being defensive. Cred fail.

The main problem was communications, never a hard-core tech-oriented sysadmin's best talent. Although I was following the overview I had sent to the boss, I had no guarantee he actually read the work breakdown.  Not only that, I had failed to take into consideration what the end-users and their chiefs felt were the priorities. I was pursuing priority a (a wiring closet that had too many trunks to other wiring closets) when the users' most vexing problem was priority b (replacing statically-assigned workstation IP addresses with dynamic DNS; the former was impeding movement over subnets).

The standard method to mitigate this issue is, of course, setting a meeting. The reason for meetings is to find out what the users want, what IT is doing, how it all translates into business terms, how it translates into IT terms, and (hopefully) the beginnings of an improved and better-publicized project plan. A plan that is agreed on by all parties has a high chance of success, everybody on all sides wins, and the admin's cred goes back up.

Meetings should be short. The agenda should also be short and consist of the following items:

  1. What needs to be improved?
  2. Why does it suck and how does it affect the business?
  3. What's the degree of suckiness? (impact)
  4. What is already being done to reduce the suck?
  5. How long can the suckiness be tolerated? (i.e. project close date)
  6. When can we meet again to discuss the plan?
  7. Immediate solutions that come to mind (brainstorm)
The next meeting should be with the boss and your colleagues and held once you have done your research on a solution. The agenda, once again, should be short, and an overview of the strategy to be followed already set up in a document on the projector:
  1. Introduce the issue
  2. Introduce the solutions
  3. Agree to a solution or table solutions for cost analysis
  4. Agree to the next team meeting.
  5. Identify stakeholders: all those affected by the change (hint: the boss, dept heads, team members, testers, suppliers (even those not supplying the project per se), users, possibly general public)
  6. Set up a skeleton communications plan
  7. Set up a skeleton equipment and purchase list
  8. Identify time and team-member constraints (really important since, for example, you don't want an unstable network during a trade show)
  9. Identify other projects affected by this one
  10. Set up next actions
"Set up next actions" is dangerous. It does not mean coming up with the entire work breakdown structure (WBS) set up in an GANTT chart. It means each team-member who has something to do agrees to do at least one task. This may be, in fact, brainstorming a WBS but it may be also be as mundane as writing a draft plan announcement and sending it to the boss for vetting.

Also note that setting the next meeting happens before any brainstorming. In my case, my team has regular meetings anyway so it is not a requirement unless it is busy week for operations.

Tuesday, October 16, 2012

SSSD, Kerberos and RHEL 6

I am not really up on writing succinctly today. However, since I suspect I will run into this issue over and over again, I'm tossing it up for whoever needs it!

The goal is to authorize and authenticate a RHEL host to an Active Directory 2008 infrastructure without resorting to Winbind.  This expands on the FreeBSD guide I wrote one year ago, but it is a lot simpler since this technique uses the Samba suite and the wonderful automatic tools it provides. It also means that all authentication, not just SSH, can go through AD.

The secondary goal is getting NFSv4 to work. It's doable, but the caveat was painful: NFSv4 is kernel-space and RHEL 6's kernel does not use an encryption scheme allowed by Microsoft Server 2008 as a default. Add to this mess a mixed 2003/2008 AD environment and the sysadmin goes home to his kids with a headache every night. The latest Linux kernels do permit strong encryption, but I don't have the infrastructure to send kernel upgrades to all the workstations.

Note that all of this can be made into a script for fast deployment.


Please see the FreeBSD article for credits.


  1. Set up a guest bind user on Active Directory to handle queries.
  2. Configure Kerberos, LDAP and Samba tools (Just the tools, you don't need the service running).
  3. Join the host to the domain.
  4. Set up SSSD on the Linux workstation. SSSD is lovely since it caches usernames/passwords. This permits laptops to operate outside the office and to defend against network dropouts inside. It replaces NSCD.
  5. Alter the host account on Active Directory to permit weak encryption for NFSv4.


Even allowing weak crypto won't work in a mixed 2003 and 2008 environment due to the way the information is passed and stored among the directory servers. Use all 2003 or all 2008. There is a patch that would permit 2003 and 2008 to coexist available at Microsoft (I chose against it since we're using CIFS as an acceptable workaround). 

Set up a guest bind user

When you create the guest user, ensure that it is a guest account with very low privileges. Its password will be the worst-kept secret in your organization. For the purposes of this text, I created the user with the password musicaSacram67.

Install Packages on the Host

  1. samba-common
  2. ntp
  3. sssd
  4. openldap
  5. ca-certificates (if your Active Directory uses LDAPS with a commercially-signed SSL certificate)
  6. oddjob

Set up NTP

If your host clock is not within 10 minutes of the Active Directory, authentication will FAIL. Do yourself a favour and set up the NTP daemon.

Configure Kerberos, LDAP and Samba on the Host


You may not wish to, but I use a softlink in the /etc/openldap directory to point to the certificates in the /etc/pki/tls directories:

ln -s /etc/pki/tls/certs/ca-bundle.crt /etc/openldap/certs/ca-bundle.crt


Remember Kerberos loves capitalization on domain names. Note the weak crypto statement needed for NFS4.

 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 forwardable = yes
 allow_weak_crypto = true
 default_tgs_enctypes = des-cbc-crc aes256-cts-hmac-sha1-96 arcfour-hmac
 default_tkt_enctypes = des-cbc-crc aes256-cts-hmac-sha1-96 arcfour-hmac

  kdc =
  kdc =
  admin_server =

[domain_realm] = EXAMPLE.COM

 pam = {
   debug = false
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false


URI ldaps:// ldaps://
BASE dc=example,dc=com
TLS_CACERT /etc/openldap/certs/ca-bundle.crt
Some comments: I have two AD servers, of course, and since unencrypted ldap can result in group memberships being spoofed, I use secure ldap (ldaps). Set your referrals to no, otherwise the DS servers will pass the authentication buck back and forth. The cert, of course, can be set directly to /etc/pki/tls/certs/ca-bundle.crt rather than what I have set up.


        workgroup = 
        security = ads
        realm = EXAMPLE.COM
        use kerberos keytab = true
        password server =
        log level = 2

Join the host to the domain

Rather than creating the Kerberos principals directly on AD, which takes a while, just use Samba:

net ads join -U administrator_account_name
Enter Administrator 's password: xxxxxxxxxxxxxx

You may get a DNS error, especially if your AD and your DNS systems are divided. Don't worry. You can verify that all went well by seeing if you have a shiny new /etc/krb5.keytab file:

klist -ke


This will be a touch odd. Whenever anyone installs SSSD or makes a change to the authentication system through Red Hat tools, it blows up your /etc/sssd/sssd.conf file horribly. The idea, then, is to install sssd, set up authentication to go through sssd, and then write the sssd.conf you want.

/usr/sbin/authconfig --update --enablesssd --enablesssdauth --enablemkhomedir --disablefingerprint --disablesmartcard

Note I disabled fingerprint and smartcards. I don't need them. Maybe you do. I also ensured that new directories are created for new users.

Now that SSSD is the authentication mechanism, one can set up the SSSD.conf file one wants. Ensure that the file has rw------- permissions or it won't work!

config_file_version = 2
reconnection_retries = 3
sbus_timeout = 30     
services = nss, pam   
domains = LOCAL,default

filter_groups = root
filter_users = root
reconnection_retries = 3
entry_cache_timeout = 300
entry_cache_nowait_percentage = 75

reconnection_retries = 3
offline_credentials_expiration = 0
offline_failed_login_attempts = 0
offline_failed_login_delay = 5   

id_provider = local
min_id = 1       
max_id = 499     
enumerate = False 

description = Kerberos 5 domain with Active Directory servers
id_provider = ldap                                         
auth_provider = krb5                                       
min_id = 500                                               
enumerate = True                                           
timeout = 10                                               
cache_credentials = True                                   
entry_cache_timeout = 300                                   

# General -----------------------
# LDAP                         
ldap_uri = ldaps://,ldaps://
ldap_default_authtok_type = password                                                                       
ldap_default_bind_dn =                                                               
ldap_default_authtok = musicaSacram67                                                                     
ldap_search_base = dc=example,dc=com
ldap_user_search_base = ou=Users,dc=example,dc=com
ldap_group_search_base = dc=example,dc=com
ldap_referrals = False
ldap_schema = rfc2307bis
ldap_search_timeout = 5
ldap_network_timeout = 5

# KRB5
chpass_provider = krb5
ldap_force_upper_case_realm = True
krb5_server =,
krb5_realm = EXAMPLE.COM
krb5_store_password_if_offline = true
krb5_auth_timeout = 15

# Mapping --------------------
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
ldap_user_principal = userPrincipalName
ldap_group_gid_number = gidNumber
ldap_user_gecos = cn
ldap_user_home_directory = unixHomeDirectory
ldap_id_use_start_tls = True
ldap_user_object_class = user
ldap_group_object_class = group
ldap_group_member = member
ldap_group_name = name
ldap_user_name = uid
ldap_user_shell = loginShell

Did you see the LDAP bind user and the badly-kept password? Also note the ldap_schema and ldap_referrals. The user search base and base I will leave to you as an exercise: I put in place holders.The mappings I swiped from other blogs :)

Once this file is in place, put it to work!

service sssd restart

Your host is now ready to welcome your network users.

Set up NFS4

Obviously, if you don't need it, skip this part.

I will give the NFS4 system configuration in a later post. However, you will want to ensure it does work with Active Directory. To do that, you must make two (2) changes to the host account.  I prefer to do it with LDAP since it is a lot more scriptable than installing, then pointing-and-clicking on ADSI edit on the AD server. They are

  1. Set the msDS-SupportedEncryptionTypes to 11 (decimal)
  2. Set UserAccountControl to its current value + 2097152 (decimal). Chances are the total will be 2166784 (69632 + 2907152)
I've done lost my ldapmodify script! I'll get that up in an update.

Anonymous has been very kind. He or she was having trouble with SSH using my guide. They added the following line to the sssd.conf: ldap_tls_reqcert = never to make it work, and let me know. This may be a solution for you. But ...

I haven't experienced the same trouble. Perhaps it may be because of the SSL cert being used on the Active Directory. My company had to install some intermediate certificates on the AD server since our SSL authority had changed its parent in order for the infrastructure to work.