Danger Will Robinson (Fun with Hyper-V)

In the process of installing Hyper-V for a customer, I made two discoveries. The first discovery was that the only way to manage Hyper-V on a Server 2008 Core system is from another, remote 2008 system or a remote Vista SP1 box. In both cases the Hyper-V Management Tool (MMC) must be installed separately. For Vista SP1, it’s a free download. For Server 2008, it can be installed via the Features. This would of course pose an issue in an XP only environment with a single Server 2008 system. Of course, the customer that I at was XP only; luckily, they did have another Server 2008 system.

Now for the second discovery: instead of installing the Hyper-V tools from features, I installed the Hyper-V role because I didn’t know about the feature. Not a big deal, everything installed fine, rebooted, and everything was still fine. Then I discovered that I could just install the tools via Features so I decided to remove the Hyper-V role. Again, not a big deal. Wrong. In the process of starting up (yes, a reboot was required), the server hung at Installing Updates – 75% for 20+ minutes. Uh-oh. I could connect to the server remotely and look at the logs, but there was nothing significant or out of the ordinary in them. Time to turn to my trusty web search tool of choice and bam: http://support.microsoft.com/default.aspx/kb/950792. Not sure who to blame this one on, but at least I wasn’t the first to experience it.

Server 08 with Hyper-V

I took the plunge and reloaded my laptop, a Dell Latitude D830, with 64-bit Server 2008 Enterprise with Hyper-V. I experienced no problems during actual installation and it even found most of my drivers. I did download all the Vista x64 drivers from the Dell site and installed those just because. They all worked great including the wireless: I’m attributing this to the fact that Server 08 and Vista SP1 share the same code base and presumably the same driver model.

I have experienced a few little “differences” that have caused me to do a little research though.

• The wireless feature is not installed by default, you have to go into the features and add it.
• The latest versions of Live Messenger won’t load. From what I’ve read and seen, it’s not a technical problem, rather an imposed limitation. The work-around to this is to download the Live Messenger 8.5 msi; it installs fine.
• After installing Hyper-V, the power management no longer allows standby and hibernation. From various posts, this appears to be a design choice because very few folks, if any will actually be running a production Hyper-V on a laptop or where these features are needed and so it just wasn’t worth the cost and effort to make it work.
• As with Vista, if you are remote and need to log on with a domain account that is not yet cached, you must log in with another valid account, most likely a local one. You can then connect however you connect to the corporate network and then do a switch user. This will bring you back to the logon screen while still being connected to the VPN.
• Virtualization must be enabled in the BIOS as this was not the default for this particular laptop. Windows will warn you that virtualization must be enabled, but it will not prevent you from actually installing Hyper-V.

I’m sure there will be other little things, but for now, I’m satisfied.

Operations Manager Service Pack 1

Operations Manager Service Pack 1 is finally here. I’ve installed it into production twice already and from my perspective, it is a lot better. The UI is much snappier and the script error noise is sigficantly less. The upgrade was painless in both cases and actually solved a major issue for one of the installations: a particular remote server agent refused to talk to the RMS. I manually upgraded the agent to SP1 and without out any other intervention, the agent starting talking.

I hated giving the answer “wait for SP1″ to customers, but it definatley lived up to the hype even though it was long overdue.

There were two minor gotchas in the upgrade:

1. Increase the Operations Manager DB log size before the upgrade. This is documented in the release notes.
2. Reboot the RMS after the upgrade. The upgrade does not prompt you to do this, but in both cases, the RMS issueing alerts and just acted weird in general.

Another known noise issue is documented and being actively worked in Redmond: http://blogs.technet.com/momteam/archive/2008/03/01/performance-module-could-not-find-a-performance-counter.aspx

Overall, I’m very pleased with this service pack so far.

ACS, SQL Server, and Windows authentication

In the course of installing and configuring Audit Collection Services (ACS) I had to troubleshoot a permissions issue connecting to the database.

Specifically, the ADTServer application service running on the collector was attempting to connect to the ACS database on a separate system.  The service runs as the local network service on the collector which means that it uses the computer account to perform network authentication: DOMAIN\SYSTEM$.  This account was added to SQL Server and granted dbo privileges by the ACS installation.  However, every time the service attempted to connect to the database, authentication for the account was rejected with the below events.

I went in a lot of different directions but ended up looking at the local system policies and finally found one: Access this computer from the network.  By default, this setting is set to Administrators, Backup Operators, Power Users, Users, and Everyone.  On the database server, it was set to Administrators and Backup Operators only.  Thus in order to connect to a database on this system, a user must be an administrator (or Backup Operator) on this system.  By seemingly reducing the access to this system it is actually less secure because it requires database users to be administrators. 

Leaving this setting at its default setting does not pose a security risk, it simply allows users to be authenticated and access resources that they are authorized to use; in this case, SQL Server is controlling authorization.  Note that authentication and authorization are two distinctly different things.  The setting identified above is limiting authentication not authorization and this ultimately decreases the effective security of this system.

Event Type:      Error
Event Source:   AdtServer
Event Category:            None
Event ID:          4618
Date:                1/8/2008
Time:                9:25:10 AM
User:                N/A
Computer:         Collector
Description:
Error occured on database connection:
 Status: 0×02200000
 ODBC Error:    18452
 ODBC State:    28000
 Message:         [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user ”. The user is not associated with a trusted SQL Server connection.
 Database:        Register
 Connection:     ComplianceTest
 Statement:       -
 

Event Type:      Error
Event Source:   MSSQL$ACS
Event Category:            (4)
Event ID:          17806
Date:                1/8/2008
Time:                9:25:10 AM
User:                N/A
Computer:         DATABASE
Description:
SSPI handshake failed with error code 0×8009030c while establishing a connection with integrated security; the connection has been closed. [CLIENT: 10.32.9.69]
 

Event Type:      Failure Audit
Event Source:   MSSQL$ACS
Event Category:            (4)
Event ID:          18452
Date:                1/8/2008
Time:                9:25:10 AM
User:                N/A
Computer:         DATABASE
Description:
Login failed for user ”. The user is not associated with a trusted SQL Server connection. [CLIENT: 10.32.9.69]

Ops Manager 07 DNS Management Pack

I finally found a fix for the problems that I’ve experienced with the DNS Management Pack in Operations Manager 2007; actually it’s more of a work-around: add the Data Reader account to the Operations Manager Administrators group and all is well.  Note that this should be the group designated during installation as the Operations Manager Administrators; the setup program adds this group to the Operations Manager Administrators role.

The problem occurs when the DNS Management Pack is added to Ops Manager 07 and generates the following alert: Data Warehouse failed to deploy reports for a management pack to SQL Reporting Services Server.  Additionally, an error is logged in the Operations Manager event log (a sample of which is copied below).

I’ve experienced this problem both in the lab and in multiple production environments.  This problem does not affect any other management packs and I have no idea what it is uniquely doing to cause this issue.  I do think it’s an installation only issue and if you remove the account from that group after the MP has been successfully deployed to all agents, everything will work fine – I haven’t actually tested taking it back out though.

             Event Type:        Error

Event Source:    OpsMgr SDK Service

Event Category:  None

Event ID:              26319

Date:                     12/20/2007

Time:                     2:50:34 PM

User:                     N/A

Computer:          MOM

Description:

An exception was thrown while processing GetRelationshipTypesByCriteria for session id uuid:2856e71b-a9a4-4518-ba41-151504c5e7d1;id=19.

 Exception Message: The creator of this fault did not specify a Reason.

 Full Exception: System.ServiceModel.FaultException`1[Microsoft.EnterpriseManagement.Common.UnauthorizedAccessMonitoringException]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Microsoft.EnterpriseManagement.Common.UnauthorizedAccessMonitoringException: The user DOMAIN\scomreader does not have sufficient permission to perform the operation.).

PowerShell

As Microsoft begins incorporating PowerShell into more and more products, most admins are wondering how to get their feet wet. The way that I’ve been recommending is to stop using cmd: install PowerShell on your desktop and don’t ever launch cmd.exe again. PowerShell has aliases built in so that most of the commands available via cmd work without any changes. Thus, instantly, you’re using PowerShell without actually having to learn anything new. Then, slowly you can start investigating some of the awesome power available in the aptly named PowerShell.

Exchange and SMTP Relay Misinformation

By default SMTP relay is disabled in Exchange 200x.  This of course is a good thing because it prevents Exchange systems exposed to the Internet from being unwittingly used by spammers to “relay” their e-mail to other systems.

Often, organizations have an application that sends alerts or notifications via e-mail so they assume that they have to add an exception to allow that application to send e-mail via the SMTP service on the Exchange system.  This is only partially correct: the exception only needs to be added if the e-mail is being sent to externally hosted addresses.  If the mail is to be delivered only to mailboxes hosted by the organization that the Exchange server is part of, then this e-mail is for local delivery and not relayed.  Thus it is not subject to the relay restrictions in the SMTP service and no exception is needed.

This makes sense, because ultimately, the internal application server is no different than an external SMTP server trying to deliver e-mail.  If the address is valid and local, you would never want Exchange to reject it regardless of the source.

AD Empty Sites

Active Directory sites are considered empty if they do not contain a domain controller. This immediately begs the question: why would anyone want an empty site? The answer is that there are more applications than just Active Directory that are site aware. This, albeit short list, includes DFS, SMS 2003, SCCM 2007, and Exchange 2007. Thus, just because a site does not have a domain controller, does not mean that the boundary defined by the site is not important for another application.

So just how do clients determine which site that they are in? Sites are of course are defined by IP subnets and at the time a client is joined to the domain, it performs a lookup in Active Directory to match its IP address to a site which is then stored in the registry. This information is updated every time a client logins into the network.

Why do some consider empty sites bad? This stems from the mis-perception that clients in an empty site will not be able to choose an optimal domain controller. Clients find domain controllers by looking them up in DNS based upon the site that they are contained in. Active Directory creates DNS records in empty sites for domain controllers according to the topology generated by the KDC. This can also be manually controlled via Group Policy or the registry on specific domain controllers: How to optimize the location of a domain controller or global catalog that resides outside of a client’s site.

Empty sites do need a little extra thought, but they do have a purpose and should not be discarded. For detailed information about how clients find domain controllers, see the Locating Active Directory Servers from the Windows 2000 Resource Kit.

Intra-Organization SMTP Send Connector

This week, I learned some more about Exchange 07 senders, specifically the Intra-Organization SMTP Send Connector.  I’m working at a client that has two Exchange 07 boxes, one is supposed to be a hot spare (XYZExchange02) with no live mailboxes on it.  In the process of testing mail flow, I discovered that no mail is flowing from the hot spare to the live system and the following error appears in the Application Log:

Event Type:        Error
Event Source:    MSExchangeTransport
Event Category:                SmtpSend
Event ID:              2017
Date:                     11/27/2007
Time:                     5:01:22 PM
User:                     N/A
Computer:          XYZEXCHANGE02
Description:
Outbound authentication failed with error TargetUnknown for Send connector Intra-Organization SMTP Send Connector. The authentication mechanism is ExchangeAuth. The target is SMTPSVC/email.xyz.com.

Web searches turned up absolutely nothing except that the Intra-Organization SMTP Send Connector is created by default to handle all internal routing between hub transport servers based upon the AD topology, that it is hidden and can’t be modified in any way, and that there is no logging for it (until SP1).

After staring at the above error message way to long, I realized that it was looking for a Service Principle Name (SPN) and I found a KB article detailing similar problems in Exchange 03. I ran the following command on a DC to set the SPN and lo and behold, everything started working.

setspn -a SMTPSVC/email.xyz.com XYZExchange01

The big question to me still is why is it trying to use email.xyz.com instead of XYZExchange01 for the SPN? My only answer is that the receive connector on XYZExchange01 is configured to identify itself as email.xyz.com.

Looks like I have some research to do on SPNs.

To Partition or Not to Partition…

I visited a customer once who felt that creating partitions ended up creating problems; he felt that limiting the C drive to a specific size would eventually lead to the C drive running out of space.  Thus, most of his servers had 200 GB+ C drives and no other partitions.  To me, this is definitely the wrong answer.  His problem wasn’t that his C drive wasn’t big enough; it was that he had no control over his servers, what was on them, and where it was placed. 

Creating separate partitions requires some planning, but that’s a good thing — too many folks just stick the CD in and go.  Separating data types on different drives adds a layer of fault tolerance and allows you to easily manage those different types of data differently via permissions, quotas, backup policies, etc.  It also allows you to distribute the workload between different disk spindles which will result in performance gains.This leads to the obvious question, what is the best practice for drive partitioning?  As usual, best practices are subjective and are meant to be flexible depending on the exact situation but here are my general rules:

1.  Create partitions at the hardware level.  Use the built-in array tools, like HP’s ACU, to create RAID sets and then carve those into containers that will be seen as physical drive partitions by Windows.  This eliminates any ambiguity as to what partition is on what disks.

2.  Create a minimal C drive (boot partition).  Not much should actually go on this drive, just the OS and any minor support applications like the Windows Support tools.  The partition should be large enough though to handle Windows Update uninstall files and of course Windows itself.  Right now with Windows 2003 Server, I like 16GB as a C drive size.  This gives plenty of overhead while also being relatively small.

3.  Create a separate partition for the paging file.  Paging files today are often 2-4GB, a significant chunk of 16GB.  Creating a separate paging drive tidies things up and potentially increases performance.  The maximum paging file size is 4096kb, so I generally create a partition of 8192kb (8GB).  This prevents the partition from showing up on any low drive space reports.  I then also use this as temp space (instead of the popular C:\temp) going as far as changing the system environment variables (%TMP% and %TEMP%) to use this partition.

4.  Additional partitions for applications and data.  Using mount points, these can actually be accessed via the C drive letter or they can be assigned a new drive letter.

To some, this may seem a bit of overkill, but if done up front with planning and consistency, is easy to accomplish and makes working with your systems that much easier.