Saturday, June 25, 2011

On First Steps in Systems Administration

Picture some poor accountant of a small consultancy firm who was just told by the Big Boss, "lo, since thou seemest to know thine Windows and thine Quickbooks well, we dub thee System Administrator. Right-o, chop-chop, have at it, and do something about the spam wilt thou?" If she came to me for advice, what would I tell her is the first step?

As for that matter, I ask myself: how much time did I waste as a junior sysadmin in charge of the systems so many years ago, technically proficient but naïve in the reality of business, because I didn't know what needed to be done and I couldn't structure my day? Procrastination through reading net comics didn't help the business and it certainly didn't help my self-esteem. What should I have done?

Here is the executive summary of my answer for the impatient:
  1. Write down a list of what you are responsible for ("service portfolio")
  2. Using the portfolio as a guide, write down out what, if you mess them up, could break the things you are responsible for ("change management system")
  3. Set up your email
  4. Get training / read a book for the most critical service you are responsible for if you don't have it already.
I will explain ...
    Introducing ITIL

    This winter, my company was kind enough to send me on an Information Technology Infrastructure Library (ITIL) course. ITIL, a trademark from OGC, is a compilation of best practices in IT management, researched over three decades.  I believe this is the framework through which I will grow my company's IT without making Athena over into a Hydra. Yes, ITIL has its critics, but for someone who has no framework at all, it can inspire. The ITIL books are expensive, however, and the literature is hard, hard, hard!

    Before taking the course, I had already started going through the second book in the series, "Service Design." English is not the native language of the researchers and, although the information it gives on structuring an IT department is infinitely valuable, the book itself suffers from structure problems. It is a reference maze with traps. This sad fact, along with its CAD$700 price-tag for the series, can make it inaccessible to the junior or intermediate sysadmin.

    The publication did mention quite clearly that one of the toughest things to do is actually get ITIL started. It asserts that existing pre-ITIL services such as email or shared filesystems are already doomed to be en eternal headache for their admins (I disagree). The continual service improvement (CSI) of these services is, by its very nature, circular: no entry point, no exit. The solution that comes to mind quickest -- redesign a IT service in order to design and fire up the ITIL processes -- won't work: those ITIL processes need already to be started and their tools implemented to tie in the service. Holy catch-22, Batman!

    On top of this, my experience with small and medium companies is that the bosses don't have the time or inclination to play ball.

    I needed that course.

    A Good System is Grown, not Imposed

    So where would one start? How does a simple accountant-turned-sysadmin set up what looks like a complicated ITIL framework for a simple office? Does she really need to?

    Unfortunately, since the course was a simple overview, no more than a quick two-day survey, the devil in the details weren't identified although there was some in-depth work on running a help desk. The consultant/instructor, an excellent and experienced lecturer, did say that out of the almost 30 ITIL v3 processes, the one he would start first is "Change Management" (CM), followed by "Financial Management." Change Management is the process by which a business is assured that the components of a system are altered only if the business needs it, everyone who matters is cool with it, and nobody who uses the system gets a nasty surprise.

    How complete should the CM process be made? Not very, to begin with: a company needs to grow in its ITIL maturity. A completed ITIL process, not tied to any other processes, is just bureaucracy without benefit. The ITIL books gave a hint by displaying a diagram of company ITIL maturity levels. Level one is no ITIL set at all. The second is the recognition that ITIL exists at all. The third is basic lip-service to each process, no more.

    So why shouldn't we set up CM completely from the get-go? I state again that the biggest problem for the small- or medium-sized business is that the senior management who need to approve such a complicated project won't have the time or patience to learn about a complicated framework, no matter how much it will smooth the works of their company. Add to that the sysadmin's habitual lack of time for projects and, of course, the lack of funding. A fully created CM processes, with all the tools present, requires more time for interviews, role assignment, tool testing and installation -- to say nothing of training -- than one can reasonably afford.

    So does the new sysadmin want to implement this stuff? The instructor was quite clear: a company needs all the processes to smooth the way for the sysadmin and the people he serves. However, the starting sysadmin wants to start from level zero.

    I disagree with the instructor on what process to start at level zero: setting up CM (other than writing a big black "Don't Touch Anything" sign) won't help if the poor accountant I mentioned above doesn't know what she has inherited.

    How to Break into ITIL

    As the instructor said, set up the processes, but with only the basics in place. The lovely thing about basics is that basics really do not need to have senior management approval or funding (thank you Open Source developers!).  They can be grown by the sysadmin in his spare time, and the benefits will be realized almost at once. Once the basics are set, they can be grown further at leisure.

    So how basic is basic? Where is the line that says "too basic to be useful?" Where is the line that shows "too complicated; you are wasting your time?"

    The First Steps for the New Sysadmin

    Before we start the practical stuff, look to the abstract. Stephen Covey, the famous author of Seven Habits of Effective People, says "start with the end in mind." Develop a mission statement for your job. Of course, the new sysadmin might be too wrapped up in the newness of everything to come up with a credible mission statement (mine is"grow my IT into a reliable service that is predictable, simple, flexible and unnoticed"), but at least some sort of goal might be established in her mind.

    Next, go from the abstract to the concrete: I say that before tackling the processes of system administration, one must set up some basic tools as defined in the books "Service Strategy," "Service Transition" and "Service Operation:"
    1. A service portfolio, just the bare essentials. This doubles as a sysadmin's job description and it is a good one, believe you me. As the service portfolio expands over the years, one can track how one's job has altered without feeling like a mote in the ocean. In the smallest networks, a spreadsheet might suffice.
    2. A configuration management system (CMS), based upon the service portfolio, once again with just the basics. One can use a spreadsheet for a small system.
    3. A service desk application to control the inevitable requests, configuring it based on what one has learned in establishing the service portfolio and the CMS. This may be as simple as setting up folders in one's personal Outlook or Lotus Notes.
    4. Get training. Choose the most critical service, look at its component list (in the service portfolio) and its configuration items (in the CMS). Learn it and experiment with it on an old discarded laptop, even if you have only the budget for books.
    So how does one do all this? That will be the subject of my next posts. Next post, I will discuss setting up a basic service portfolio.

    À bientôt !

    Friday, June 24, 2011

    On FreeBSD and Active Directory Integration: Kerberos and LDAP

    Before proceeding with my solution, I wish to thank the authors of the blogs I consulted first: Scott Lowe of who came up with the solution I used for Windows 2003 Server and Red Hat Linux; and to Jared Barnek of

    The big picture

    Since the company has a great number of Windows, Mac and Linux users, user and computer accounts are controlled by Active Directory on Windows 2003 Server R2, not Kerberos on a *nix host.

    The problem
    1. I wish to tie FreeBSD 8.2 into Active Directory.
    2. The FreeBSD host must be as simple as we can make it: the Samba suite will not be installed.
    3. Kerberos is to be used for authentication to leverage single-sign-on and eventually enable NFSv4 and Kerberos-enabled HTTP.
    4. Only SSHD is tied into Kerberos for now; console access is granted through local accounts only.
    The solution
    1. LDAP is used for authorization (remember Kerberos can only authenticate!)
    2. Simulate kadmin with commands on Windows Server 2003 R2 and manually transferring the keytabs.

    Fortunately, the detailed answers for LDAP were already supplied by Mr Lowe (above) and for Kerberos by Microsoft's own document located at

    Note: I used the 64-bit FreeBSD 8.2. I had segfaults with the 32-bit OpenLDAP.
    Note 2: Mind the word wrap! In the configuration examples, in Courier font, something that looks like a new line may not in fact be a new line!


    On Microsoft Windows Server 2003 R2:

      1. An Active Directory (AD)
      2. Subsystem for UNIX-based Applications (SUA) installed
      3. Windows Server 2003 Support Tools installed
      4. Service Pack 1 applied to the AD server (for a bugfix to ktpass.exe)
      5. An SSL certificate signed by an known Certificate Authority applied to the AD server. (See * and **)
      6. At least one account (to test) has been created on Active Directory with its UNIX fields set (especially the shell!)
      7. WinSCP on Active Directory, used to transfer a Kerberos keytab securely.

    1. On a FreeBSD 8.2 host:
      1. A minimal installation of FreeBSD 8.2 will suffice
      2. At least one local non-root account created for transfers and for troubleshooting. Ensure this account is part of the wheel group since you will need to get to root somehow.

        NB: There is no need to install the advanced Heimdall or MIT Kerberos on FreeBSD; the default Heimdall will do.
    * You want SSL. Although passwords are not sent over the wire, you don't want a rogue LDAP server sending spoofed group membership info.

    ** If you need to use your own CA, be sure to export and merge its public cert in FreeBSD.'s /etc/ssl/cert.pem. That, however, is out of the scope of this little post.

    Part One: OpenLDAP Authorization
    1. In AD, create a non-Domain User account used for "anonymous" ldap lookups with very restricted rights. This account should not be used to log in to any workstation since its password will be an open secret to whomever looks at the ldap configurations on the FreeBSD hosts. For the sample configurations below, I use an account "ldapaccount." Remember to turn on the UNIX fields for ldapaccount.
    2. Turn off nscd if it is running on the FreeBSD host:

      /etc/rc.d/nscd stop

    3. Install root certificates

      cd /usr/ports/security/ca_root_nss
      make install clean
      (When prompted to create a link, say yes)

    4. Install OpenLDAP with Cyrus SASL, pam_ldap, pam_mkhomedir and nss_ldap:

      cd /usr/ports/net/openldap24-client
      make install clean
      (select Cyrus when prompted)

      cd ../nss_ldap
      make install clean
      (defaults are fine)

      cd /usr/ports/security/pam_ldap
      make install clean

      cd ../pam_mkhomedir
      make install clean

    5. Create the /usr/local/etc/openldap/ldap.conf (used for applications)

      base dc=yourdomain,dc=com
      uri ldaps:// ldaps://
      ssl yes
      tls_cacert /etc/ssl/cert.pem

    6. Create /usr/local/etc/ldap.conf (used for nss)

      base dc=yourdomain,dc=com
      uri ldaps:// ldaps://
      scope sub
      ssl yes
      tls_cacert /etc/ssl/cert.pem
      bindpw ldapaccount_passwor

      pam_login_attribute sAMAccountName
      pam_filter          objectClass=User
      pam_password        ad

      nss_base_passwd dc=yourdomain,dc=com?sub&(objectClass=user)(uidNumber=*)
      nss_base_shadow dc=yourdomain,dc=com?sub&(objectClass=user)(uidNumber=*)
      nss_base_group dc=yourdomain,dc=com?sub&(objectClass=group)(gidNumber=*)

      nss_map_objectclass posixAccount    User
      nss_map_objectclass shadowAccount   User

      nss_map_attribute   uid         sAMAccountName
      nss_map_attribute   cn              sAMAccountName
      nss_map_attribute   uniqueMember    member
      nss_map_attribute   userPassword    msSFU30Password
      nss_map_attribute   homeDirectory   unixHomeDirectory
      nss_map_attribute   gecos           name
      nss_map_objectclass posixGroup      Group

      # Active Directory Silliness
        referrals no

      Note the nss_map_attributes. They may change if your version of Windows Server is different from mine. If you run into trouble, execute an ldapsearch on the AD server on your username and make note of the field name that holds usernames, groups, home directories, etc.
    7. Link /usr/local/etc/nss_ldap.conf to /usr/local/ldap.conf

      rm -f /usr/local/etc/nss_ldap.conf
      ln -s /usr/local/etc/ldap.conf /usr/local/etc/nss_ldap.conf
      chmod 644 /usr/local/etc/ldap.conf

    8. If the users on AD have their shells set to binaries located in the /bin directory, for example Fedora or Red Hat users, you will want to link installed port shells on FreeBSD such as bash to the /bin directory to avoid heartache.

      ln -s /usr/local/bin/bash /bin/bash  (etc)

    9. Turn on nscd to keep your AD from holding connections open
      /etc/rc.d/nscd start

    10. If nscd does not start on boot, it is a very good idea to make it start on boot:

      echo 'nscd_enable="YES"' >> /etc/rc.conf
    Part Two: Kerberos Authentication setup on FreeBSD
    1. Ensure that the system time on FreeBSD is within 5 minutes of AD.
    2. Configure Kerberos in /etc/krb5.conf. The configuration below in monospace was taken from a CentOS5 krb5.conf and edited. Check the man pages for explanations. Replace yourdomain with the correct active directory domain. In this case, activedirectory1 is the master AD server. Note the capitalization: it's important!


       # The logging is not really required as this host is not
       # using kadmin. Kept in as it does no harm.
       # Debugging, if required,
      will be set in the
       # /etc/pam.d/ files.

       default = FILE:/var/log/krb5libs.log
       kdc = FILE:/var/log/krb5kdc.log
       admin_server = FILE:/var/log/kadmind.log

       default_realm = YOURDOMAIN.COM
       ticket_lifetime = 24h

       forwardable = yes

        kdc =
        kdc =
        admin_server =

      [domain_realm] = YOURDOMAIN.COM = YOURDOMAIN.COM

       pam = {
         debug = false
         ticket_lifetime = 36000
         renew_lifetime = 36000
         forwardable = true
         krb4_convert = false
    Part Three: Create and transfer the Kerberos Principal

    The naming convention for Kerberos principals (service/ is incompatible with Microsoft naming. Windows requires a Kerberos principal to map to a regular Microsoft account.
    1. Option (strongly suggested): create an Organizational Unit (OU) to hold UNIX accounts
    2. Create a User Account on Active Directory (not a Computer Account).
      1. Example: for the host "," create an account "hostPentheus"
      2. Fill only the first name field with "hostPentheus." Don't fill in the other name fields.
      3. Fill in a complicated password. This password will never be used by humans so go nuts.
    3. Map a Kerberos principal to the user account and generate the keytab
      1. Open a command line interface on AD
      2. Change to a directory used for holding keytabs or to the desktop
      3. Execute the following:
        ktpass princ host/ mapuser userAccount -pass random_complicated_password out new_principal.keytab
    4. Copy the new principal over to FreeBSD from the directory or desktop
        1. Use WinSCP to copy the keytab over the wire to the local non-root account or use the sneaker-net: a USB Key or floppy. FTP is not your friend here and SMB is not in scope.
        2. On FreeBSD: merge the new principal to the (non-existent) /etc/krb5.keytab

          ktutil copy /home/
          local_account/new_principal.keytab /etc/krb5.keytab
          ktutil -k /etc/krb5.keytab list
          chmod 600 /etc/krb5.keytab

        3. For security, delete the keytab from the local account's home directory.
      Part Four:  Configure SSHD to use Kerberos with LDAP as a fallback
      1. Edit /etc/pam.d/sshd
      2. Use the following as a guide. Use man pam_authentication for details :)
        # auth
        auth    sufficient                no_warn
        auth    sufficient /usr/local/lib/ no_warn try_first_pass
        auth    required                no_warn try_first_pass

        # account
        account required
        account required
        account required   /usr/local/lib/ no_warn ignore_authinfo_unavail ignore_unknown_user

        account required

        account required

        # session
        session required
        session required   /usr/local/lib/ umask=0022 skel=/usr/share/skel

        # password
        password sufficient no_warn
        password required no_warn try_first_pass
      Part Five: Set nsswitch and lock down SSH
      1. Edit /etc/nsswitch.conf. The following lines should be set:

        group: files ldap
        passwd: files ldap

      2. Go into /etc/ssh/sshd_config and with either AllowUsers or AllowGroups restrict shell access as required. Restart sshd with /etc/rc.d/sshd restart as needed.

      Part Six: Enable GSSAPI on ssh and sshd
      1. Ensure the following is uncommented and set in /etc/ssh/ssh_config on the FreeBSD host and on the hosts from which you will ssh:

        GSSAPIAuthentication yes

        GSSAPIDelegateCredentials yes

      2. Ensure that the following is uncommented and set in /etc/ssh/sshd_config on the FreeBSD host. Please note that PAM is enabled in sshd by default in FreeBSD 8.2.

        KerberosAuthentication yes
        GSSAPIAuthentication yes
        GSSAPICleanupCredentials yes
      3. Restart the sshd service with /etc/rc.d/sshd restart 

      Part Seven: Test

      On another Kerberized host, ensure you have a TGT by using the klist command. If you don't have a TGT, use kinit. Now go ahead and try ssh'ing in. If SSH demands a password (or worse, a password is demanded by LDAP) there is a problem. Append debug to the entries in /etc/pam.d/sshd, try logging in again and look at the /var/log/debug file for hints.

      If you find any errors or have questions, please comment!


        This blog is about system administration for small to medium-sized businesses. I intend to publish IT-related discoveries from time to time. These discoveries will be either on the hard skills of sysadmin, such as Unix/Microsoft network integration, or on soft skills, such as processes inspired by what I am learning in ITIL(tm), user relations, Service Level Agreements, and keeping sane.

        This is not a blog that is meant to be kept up daily, although of course that might change. It is more of a reference for myself or anyone who Googles for answers to problems I hit before they do.

        I don't pretend to have the discovery skills of the other technical bloggers on the Web, of whom there are dozens if not hundreds. As I suspect most sysadmins do, I rely on these generous bloggers for my successes at work. Perhaps I can furnish some answers too, or at least popularize them to make for easy geek reading.

        I should establish some geek cred, mind.
        • I've remained in system administration - junior, intermediate and now senior - for seventeen years in small and medium-sized businesses, always in tech or academia.
        • My home-built arcade machine is in the basement.
        • My beloved wife and mother of my children complains about the sheer number of gadgets, Macs, Windows-based PCs and Linux boxen in the corners, tables and bedside tables.
        • I am writing this on an Acer Aspire 1430-6436 on a US + French Canadian keyboard even if I haven't ripped out the HDD to replace it with an SSD (those are in my Lenovo at home and my Dell at work). 
        • My gasket-blowing is now limited to a handful a year, thanks to publications by David Allen and Stephen Covey.
        • I claim a certain amount of sysadmin expertise on Linux and Windows, with some Mac on the side.
        • I am slowly converting to the FreeBSD side of the house for its ease of admin.