jeremYprieS.com

It’s been over a year since my last post. I’ve been super busy.

Last time I posted, I was a virtualization architect out in the professional services field implementing mostly (but not exclusively) VMware infrastructure products in large datacenters. I did that for Xcedex for several years dating back to ESX 2.0. From the early days until now, I have identified lots of gaps or opportunities to automate and make life better for myself. When appropriate, I did just that and developed solutions to problems that others are also having.

So now I still work for Xcedex, but as VP of Engineering. We have been very hard at work kicking out code. Making internal tools fit for public consumption and making new software tools. You can see what we have announced here. http://www.xcedex.com/xf. There are also more unannounced apps to come.

So here is my disclaimer. Anything on this site is my personal opinion and not endorsed by Xcedex. Xcedex knows nothing about what I publish here.

jeremy Uncategorized

VirtualCenter has all sorts of permission options.  It is nice and flexible, but can take some guess and check work.  Maybe this doc will save you some time.  It was written with VC 2.5 in mind.

A common scenario is to delegate the creation of new virtual machines to others.  This often times means deploying an existing template and customizing using an existing spec.  This means giving up some permissions on four different objects.

1. Template to deploy.
2. Customization specs
3. Resource pool (or cluster) where the new VM will live
4. Folder in the DC where the new VM will live.

So you may not want to cover all of this with one role.  It depends on how complicated your VirtualCenter layout is.  For an example, I will setup permissions for a test user using the following VC layout.

• The template will be hosted in the LAB1 datacenter, in a folder named Templates.
• The template will be deployed to the Test2 cluster in the LAB2 datacenter.
• The destination VM will be placed in the MyFolder folder in the LAB2 datacenter.

For this scenario, I will use 4 different roles.  I am trying to make it as complicated as possible to show the different areas that need permission modifications.  As a disclaimer, there are probably some other ways to get this done.

ReadCustomizationSpecs Role

The first role gives the user/group access to read the existing customization specifications.  This role should be applied at the top of the VC tree.

DeployFromTemplate Role

This role will give the user access to read the template.  This one will be assigned to the folder named Templates inside the LAB1 datacenter.

AddToResourcePool Role

This role will give the user access to add the virtual machine to the cluster/resource pool.  It will be applied to the Test2 cluster in LAB2 datacenter.

VirtualMachineDeployment Role

This last role is the general purpose role to allow the user to stick the VM in a folder and do things like power it on.  This one would be right in between the “Virtual Machine User” and “Virtual Machine Power User” default roles.

Just a couple other bullet point notes.

• When testing permission changes, I recommend logging off the VIC and back on to make sure it takes effect.
• When you apply permission for a user/group that propagates, then apply another permission lower in the tree; you will not know the first one is there by looking at the lower object.  VC will only show you this if you look higher in the tree.  So by looking at an object, you cannot say definitively what the permissions are on it.  VMware should get on fixing this one.  chop chop.

And finally, the doc on VC permissions is located here.  http://www.vmware.com/pdf/vi3_vc_roles.pdf

jeremy esx3

So Bob @ lonesysadmin.net and Scott Lowe are not so happy with the quality of VC2.5. They are unhappy with performance, reliability (unpredictable errors), and incomplete features.

I’m not sure when anyone decided that VMware’s management tools were all that great. In fact they have always been pretty crappy. Think how many times you have typed “service mgmt-vmware restart”. But I know that both these guys have worked with VMware stuff long enough to remember how far we have come, so I think this is more venting than anything.

If I have to vent anything personally, I’m not a big fan of shipping beta experimental code. Either you ship and support it, or don’t ship it at all. This isn’t a free web 2.0 service, this stuff costs a premium. I’m not here to test it for you.

Also, I think that VMware should get the point. While reliability for the virtual machines is first and foremost, expections are higher now. Mass adoption in the datacenter requires reliable easy to use management tools. Engineers need to hand off to operations. Your typical customer is the engineer who can deal with the problems. You also need to think of the operations departments.

jeremy esx3

A long time ago, I had modified a script written by Scott Herald @ vmguru.com that sends email from the service console. A while back Scott reworked his website and all of the downloads are gone. I intended to put it up here and forgot. Today I saw a post from Duncan Epping @ http://www.yellow-bricks.com/ that reminded me.

Duncan is hosting the script which is cool. I will also stick it here for when he decides to redo his site.

This is Duncan’s modified version which changes the MIME type from TEXT to TEXT/HTML. I would guess that is fine for everyone (plain text or html).

smtp_send.pl sends mail from ESX 2.x or ESX 3.x. It does require Lite.pm to be placed in
/usr/lib/perl5/5.8.0/MIME/, or whatever your version of perl is (ESX 2.x is 5.6.1). Also, don’t forget to open outbound 25/tcp in your service console firewall.

You can upload the file to your ESX host, /usr/local/bin is a good spot and run smtp_send.pl with no switches to see the command line options.

Download smtp_send.pl (628) here.

jeremy Uncategorized

I was just reading Scott’s posting regarding ESX Server and the Native VLAN.  It got me started on a post to describe my logic regarding vSwitch0 design for ESX host building. 

I can tell you that I suggest native VLAN use for many ESX installations.  I suggest this for ease of building the server.  It is not required and many network guys hate this.  I understand, but it is a convenience during the building process.  Depending on the environment, it may be a requirement (lights out data center for example).

A sample vSwitch0 network looks something like this:

The matching Cisco configuration would be something like this:

interface GigabitEthernet4/17
switchport trunk allowed vlan 15,254
switchport trunk encapsulation dot1q
switchport mode trunk
switchport trunk native vlan 15
no ip address
spanning-tree portfast trunk

It is important to note that for all port groups that need VLAN 15, you must use VLAN 0 (native/un-tagged).  A VLAN either tagged or not tagged, but not both.  I am not sure if it is a Cisco thing or a VMware thing, but if you were to add a port group to this vSwitch and tag it VLAN 15, the traffic will not work.

Build Problems

I have been unsuccessful building an ESX host using kickstart on a trunk without the use of the native vlan.  The problem is that there is no vlanid option on the boot prompt.  After we read the ks file in, we might be able to immediately start tagging packets using the –vlanid= parameter on the network line in your ks file.  It would look something like this:

network --device eth4 --bootproto static --ip 192.168.15.107 --netmask 255.255.255.0 --gateway 192.168.15.1 --nameserver 192.168.18.5 --hostname esx01.whatever.com --addvmportgroup=0 --vlanid=15

As I understand it, immediately after reading in the ks file, the network stack will be reloaded with the settings configured here.  But you have to get to the file first. 

To deal with this, I see three options.

• Put the kickstart cfg file on local media (floppy, cdrom, usb, …).
• Use a “build port” temporarily.  Usually a port on the patch panel in the server rack that is an “access” port only used to build the host.  You will need to apply the tagging to the service console post build.
• Use the native VLAN option.

Of those options I typically prefer to use the native VLAN.  So far, I have not found it to be a problem.

Building is still frustrating

I have found that even with the native VLAN, I still have too many problems with network.  It’s more so that the build environment does not have many troubleshooting tools.  You have to rely on the messages on tty 3 and 4; if they don’t fly off your screen.  Then to rerun to build, you have to wait for post and such.  Also, usually waiting for a network person to check or change settings in between each attempt.

In an effort to minimize frustration and increase productivity, I now recommend using the Ultimate Deployment Appliance in VMware player or workstation on a laptop.   For me this virtual appliance ties some pieces together, such as a web server, dhcp and tftp for PXE.  It has limited ESX support, which is fine because I use the exact same %post scripts as I did in the past. 

Here is the high level build process:

• Take your laptop to the data center and plug a cat5 cable between your laptop and the back of the server.
• Hard code the IP on your laptop and UDA.  Put it on the same subnet as where the service console will be.  Be sure to undo this before plugging into the real network.  Also, be sure to turn off the dhcp server whenever you are not using it to be sure that you are not breaking anything on the real network.
• Create a “template” for your server in UDA
• PXE boot the new ESX server and select the template name.

Here is the best part.  After the build is complete, unplug your laptop and plug in the real network.  Now you can use the full suite of tools to troubleshoot the network.  You get fancy newfangled fancy tools like ping and such.

So, to go full circle, if you are using UDA with a laptop, you don’t really need to use the native VLAN for the service console.  If you want to tag, then post-build, you will need to add the VLAN tag for the service console.  I personally dislike manual tasks and prefer to use the native VLAN all the time.

jeremy esx3

There is a problem running Xesxupdate.pl version 1.3 when your updates.list includes the ESX 3.0.2 patch ESX-1001729.  This is a problem with the VMware’s patch itself and has nothing to do with my script.  You can see the exact problem by running:

<code>
[root@mnhqesx02 scripts]# /usr/sbin/esxupdate -r http://10.10.24.250/kickstart/ks-oses/esx-302/patches/ESX-1001729 info
 
Traceback (most recent call last):
  File "/usr/sbin/esxupdate", line 481, in ?
    Main()
  File "/usr/sbin/esxupdate", line 330, in Main
    PatchInfo(db, patchlevel, repourls, listrpms)
  File "/usr/sbin/esxupdate", line 185, in PatchInfo
    print insthelper.FormatHashVerbose(h, listRpms)
UnicodeError: ASCII encoding error: ordinal not in range(128)
</code>

There is something wrong with the file descriptor.xml inside ESX-1001729.  You may fix this by editing the file and removing all text between <description> and </description>.

So your descriptor.xml should look like:

<descriptor version="1.0">
  <vendor>VMware, Inc.</vendor>
  <product>VMware ESX Server</product>
  <release>ESX-1001729</release>
  <releasedate>Fri Aug 31 18:13:57 PDT 2007</releasedate>
  <summary>Security update for gcc packages.</summary>
  <description></description>
  <contact>support@vmware.com</contact>
  <type>Partial</type>
  <info>Obsolete rpm: VMware-esx-debug-tools</info>
  <upgradepaths>
    <path release="3.0.2-52542">
      <pre>MModeOptional</pre>
      <post>RebootOptional</post>
    </path>
  </upgradepaths>
  <rpmlist>
    <rpm arch="i386" rel="59" ver="3.2.3">cpp</rpm>
    <rpm arch="i386" rel="59" ver="3.2.3">gcc</rpm>
    <rpm arch="i386" rel="59" ver="3.2.3">libgcc</rpm>
    <rpm arch="i386" rel="59" ver="3.2.3">libstdc++</rpm>
  </rpmlist>
</descriptor>

jeremy esx3

I found it is not so obvious to change your license server when you accidentally point ESX at the wrong server.  I think it has to do with the fact that this wrong server does not exist.  Here is a how to.

Software used:
* ESX 3.0.2 – no patches – build 52542
* VIC build 50618

Forgot to change the licensing lines in my kickstart file, so my config looked like this.

If you try to edit the license server field in the GUI, like this:

It will fail with the descriptive message “A specified parameter was not correct”.  There was some other message when I was connected to VC 2.0.2 with the VIC.

My guess is that it was trying to talk to the old license server before changing it.

I ended up having to edit /etc/vmware/license.cfg.  Change the line that starts with SERVER=, leaving the rest of the file alone.  I changed server name to be the correct one.  I used vi, but I bet you can use any editor you want.  Then restart the management agent.  /etc/init.d/mgmt-vmware restart.

After that I could license the different features and carry on with life.

jeremy esx3

The following procedure should work on ESX3 with a Windows 2003 virtual machine.  Assumes LSI SCSI, basic disks and NTFS.  This procedure is considered unsupported but should work just fine.  Follow along at your own risk.

Also, the can be done a number of different ways.  I think the main alternative would be to use diskpart.exe from a helper virtual machine.  Diskpart will not allow you to expand your system volume.

Steps

Power off the virtual machine

SSH into the ESX host and become root (su -)

cd to the directory where the VMDK lives.

cd /vmfs/volumes/vmfs3-os01/myvm/

Now you may expand the disk file using vmkfstools.

vmkfstools -X 20G myvm.vmdk.

The new disk size should be indicated after the -X. In this case I am specifying 20GB. The X is capital. This process will take a minute or two.

Now boot your VM from the gparted live ISO.  http://gparted.sourceforge.net/livecd.php.  Note: I used version 0.3.4-7 because version 0.3.4-8 was missing the LSI SCSI driver.

Gparted will automatically open after booting.  Select your partition, right-click, select Resize/Move. 

Drag the right side all the way to the right to expand.    Hit the Resize/Move button

Now hit apply.

You may reboot back into Windows.

CHKDSK should run at first boot.  

Windows will probably detect new hardware.  I would reboot when prompted. 

Now you are done.  Your C:\ drive should be the new size. 

jeremy esx3

UPDATE: David’s blog posting and images are now gone. My Spanish is not very good, but I don’t see an explanation on his blog.

It’s been 2 months since I last posted about WSV. Still not much happening. The first public beta of WSV has been pushed back to the 2nd half of 2007, but that has been widely publicized.

Today i ran into this translated article by David Cervigón, a Microsoft evangelist. It reveals a little bit of information that I had not known before, along with some screenshots. The original article is in Spanish.

Here is a list things that I thought were interesting.
* Remote console connection via RDP
* VLAN tagging support and vSwitches.

Still no talk of vMotion like support. This is a feature that I consider a requirement to compete with VMware. I have heard rumors of the plans here, but nothing that MS has announced so I will leave it alone for now.

jeremy WSV

So I can’t say for sure what happened, but I think I know. This is my story of how installing Vista lost my data partition. Read on if you care.

Over lunch I was going to start the Vista migration thing on my laptop. I have 2 partitions for operating systems, and one big one for data. I usually run part linux (suse or ubuntu) and part windows.

So, here is what I did.

I remove a 30GB ext3 and a 2GB swap partition in windows xp disk manager. Now I have – in order:
* 30GB NTFS
* 32GB Free Space
* 90GB or so FAT32.

I think they are all primary partitions (Can’t say for sure anymore).

Now I install Vista. Tell it to install to the free space. This was the middle clicky click option, saying something about 32GB and free space. I double checked it. I was never asked anything about partitions or filesystems, just this generic mix of where to install.

Vista finishes, rock on….where is my 90GB FAT32 partition? Gone, lost, now showing free space.

I mucked around in fdisk & qtparted in knoppix. I didn’t get the partition back (at least with my files on it). Probably because I never documented the partition table and I was only guessing at starting cylinder numbers. Vista was showing 32.01GB partition, so it may or may not be slightly bigger than 32GB. I gave up and just copied my data back from elsewhere.

After Vista installed, I now had an extended partition with only the Vista 32GB logical partition in it. This is a 130GB extended partion starting where the XP partition ends until the end of the disk. The first XP partition is still good. I am assuming the Vista installer clobbered the rest of the disk just to add the extended. I REALLY don’t remember it asking accept or deny.

So, backup your stuff and use knoppix and qtparted for partioning.

jeremy vista