mr mann's mishmash

it's just a game of give and take…

This should hopefully be, the final part of what has turned out to be a three part series documenting my thoughts on managing my laptop’s disk resources. It all started when I swapped my old laptop hard disk for a superb 500GB Seagate Momentus XT Solid State Hybrid Drive to help alleviate my disk space woes last year and then started experimenting with virtual hard disks (VHDs). Initially everything was fine but it’s taken me a while to fine-tune and accommodate a couple of new considerations posed by VHDs. This post formalises what I’ve (finally?) settled upon, so that I can manage my various laptop OS/working configurations to give me the best flexibility for the future.

Part one of this (unintentional) series, was about moving my working environment to a new disk with minimum of fuss [You can read that post here:How to turn a physical disk into a bootable VHD”]. This proved a highly successful experiment, enabling me to copy my existing laptop configuration to a VHD and then be able to dual boot into the rebuilt configuration (native partition – native OS) or run my old configuration through a contained VHD directly on the laptop hardware (no need for VMWare or Virtual PC). This was achieved with little downtime and also provided an unexpected backup strategy too. However, even though I had transferred and converted my old configuration into a single VHD file (see diagram below) it had not actually solved any long term space issues – the original partitions still remained practically the same size and were pretty much bursting (forcing me into strange backup procedures to ease the discomfort – see my post “Windows Phone 7 Backups – Release more space”].

 Blog post 1 got me to here . A bootable VHD residing on the physical disk.

Part two talked about how to extend or shrink a VHD using Windows 7 [You can read that post here:Resizing Virtual Hard Disks (VHDs)”].  By making good use of the various actions available in Windows 7 Disk Management, I managed to reconfigure my VHD layout — to a point where I got stuck at a dead end. In my eagerness I thought I could just reduce one partition and expand the other and forgot the most basic rule of partitioning…

“a partition needs to be one contiguous block,
regardless of the disk drive being physical or virtual”

Damn. I had remembered the hard way, that although I had made space by reducing the excess of my Data partition, the free space was in the wrong position to extend my nearly full OS partition (see diagram below).

 Blog post got as far as here. Wasted unallocated space on the bootable VHD.

So now I’ve had a bit of time to take stock of the situation, experiment a bit further and unravel the sequence of getting my partitions to the desired sizes. While this notion of cloning to VHD is a feasible option it does come with a couple of restrictions that may alter your course of action:

  1. Mounting a cloned VHD within the originating VHD is not possible – it’s a bit like a Star Trek time paradox. The cloned VHD will have the same logical disk identifier (which is embedded within the disk identification metadata) as the parent it came from. So even trying to mount one inside the other will cause a clash of IDs for which, Windows will prevent. This harks back to the time when physical disks ruled and were copied rather than cloned.

  2. VHDs are actually limited to 128GB in VMWare or Virtual PC. I did not expect this one, but it’s true. I’ve survived with a 155GB VHD only because I use it as a bootable VHD direct from a native Windows instance as explained in my previous post “How to turn a physical disk into a bootable VHD”.

  3. Running the cloned VHD within an instance of VMWare or Virtual PC with the originating drive running as host. Paradox time again, you’ll bump into the issue of logical disk identifiers again (as explained above) and doubly unlucky if over the 128GB limit (as explained above).

  4. VHDs are only guaranteed to be mounted and ready if you boot from them. Unfortunately, Windows does not remember what extra VHDs you attached and does not remount them after the Windows start-up sequence has completed. Impracticable - splitting to separate VHDs means one will not remount automatically on reboot.So, in the adjacent diagram, splitting my one large VHD file into two separate VHDs (so that I can expand or shrink the partitions independently) the first VHD file will always be available because it’s what I boot from. The second VHD will have to be remounted each time (hence the big red cross). You can script this, so that Windows remounts this automatically, but I moved my profile and other various files to the second partition, so scripting is not good enough since Windows might need files from this VHD before it is mounted.

  5. Reverse the procedure and go “VHD to disk” ??? – unfortunately this is not possible. Moving from either a fixed size or dynamic VHD back to a physical disk’s partition set is not possible. There are no native tools available (Disk2VHD does not go backwards!). There are a number of other backup tools that perform this action, but this is not actually a backup scenario in itself so I won’t count them as an option.

 

Disk management can be like a slider puzzle.In fact it feels like I’m playing one of those tile sliding puzzles, juggling disk resources about, trying to mount this VHD, that VHD, rebooting into a new native operating system and so on. So, having stepped back for a bit more thought I reverted back to first principles of disk management as if I was swapping physical drives rather than virtualised ones. So, to recap the problem, I need to reduce my cloned VHD to a more sensible 120GB (under the 128GB limit) and expand the C:\ logical partition upon that VHD. The following steps are how I overcame this bearing in mind the above restrictions….

 

Firstly, defrag, defrag, defrag! Remember i mentioned contiguous space for partitions? Well the same can be said within partitions; if there is any disk/partition resizing to be done, it’s best done with all the files pushed into one contiguous block. There are a number of disk tools that can do this but not all work well or at all with VHDs. So I would recommend using Auslogics Disk Defrag (free for home use) and running the tool with the “Defrag & Optimize” option which help reorder the files on disk.

 Auslogics Defrag & Optimise option

Once satisfied with the system being as optimally ordered as possible, the next step is to offload the “blocking” partition. The second partition was emptied over to another disk, preferably a local physical disk – not another VHD or network location because that would take too long. I can reply on a basic file copy for this partition because its only a data; there is no boot sector here. Also, when copying the files away from the source, forget about using Windows Explorer’s drag and drop since it seems a lot slower and only copies what you see (unless you have reconfigured the settings for hidden and system files). I would recommend using the command prompt’s XCOPY command with the options to copy all hidden/system files/folders as required. The larger the amount of data the longer it will take, so probably left best to cook overnight!

 Making room. Backup the second partition to local disk elsewhere!

Now the data partition is safely backed up, I want to delete my second partition so that it it will present the minimum amount of hassle with resizing my VHD in the next step. If I did not, I would have to reduce the last partition to a suitable size that allows the overall disk to shrink further. When shrinking a partition, its more complicated because the shrinking process is limited to working with unallocated disk – hence my suggestions to defrag and optimise the disk beforehand. Since my OS partition is well below my intended 100GB VHD target, removing the Data partition enables me to resize with impunity. Windows 7 comes with a decent set of inbuilt disk tools (as I mentioned in “Resizing Virtual Hard Disks (VHDs)”) to allow you to extend, shrink and delete partitions (Volumes). Just type “Disk Man” in the search box from the Windows Start menu and select “Create and format hard disk partitions” – the Disk Management Console will look a bit like the next diagram.

   The Windows Disk Management console.

Having deleted the second partition, shrinking the VHD will be much quicker. Reboot into a Windows OS instance that is not run from the VHD and ensure that the VHD is un-mounted – if not already. Run VHD Resizer (its’ an old tool with a very basic UI but it does the trick) selecting the VHD to shrink and the destination for the new VHD (it does not alter the original). You’ll note that when you select the source VHD, VHD Resizer will think for a while – basically it is analysing the source VHD to determine what is the maximum amount of shrinkage that can be allowed. Type the new size desired (100GB here) and ensure that the “Type” dropdown is “Fixed” because in this case my VHD is a Hyper-V/bootable VHD and they only work with fixed sized VHDs! If you have space, use a local destination (another partition, drive or USB hard disk) because the tool will want to create a 100GB file and that’s quite slow over a network. Be prepared, this process will take some time to complete.

Removing unwanted partitions makes VHD shrinking a lot easier.

The VHD should now be a neat 100GB and still with lots of free space because the second partition was previously deleted to allow a a contiguous extension. I need to bump the size of up my OS partition now, so by using the in built Windows 7 disk tools (see previously) it’s simply a case of right-clicking my partition and selecting “Extend Volume” (I won’t take you through the wizard steps because its really easy to figure out). I’m being selective about what I put back onto my second partition, so I’ve planned a 60GB/40GB partition split.

Expanding the primary partition (finally!).

It does not take long to expand a partition (given some free contiguous space), so once that is completed use the Disk Management tools to “Create Volume” in the remaining unallocated space. Windows will automatically format and assign a drive letter; if the drive letter is changed a reboot will be required. Since the two partitions reside on the same VHD, they will both be available if you boot from that VHD.

Now copy whatever data is required from the second partition backup back to the new partition. As before, use the command line XCOPY to transfer the files across since it’s faster. When this all finished, I swapped the new 100GB VHD for the old 155GB VHD being careful in my renaming. The new VHD is essentially a smaller clone, so the boot sequence is none the wiser that the VHD size has dropped.

Nearly there, restoring the data files back to the VHD. 


In Conclusion: Neat and tidy. Feels like I've done the Rubik's Cube 1000 times over.
Well gosh, this has been a road of rediscovery but I’m glad that I’ve done it. The fundamentals of disk management are not that much different between physical disks and VHDs but there are some gotchas to plan around. These have not curbed my enthusiasm for bootable VHDs since it’s still an excellent to maintain separate bootable environments from one PC without having to resort to dedicated dual boot allocations and locking down to that PC. I use my native OS for general everyday computing and a bootable VHD for whatever client project/environment I need to at that time. As the environment gets old or the project finishes, the VHD can be archived and a fresh one created in it’s place. Neat.

It’s not every day, month or even 6 months that I need to go through all this juggling – only because this VHD was a mega workhorse of an environment, namely my old laptop operating system configuration! Keeping more than one bootable VHD on my PC has taught me other tip though; keep your local data to a minimum. If you don’t look at a folder or file in over a month, then archive it and don’t make it part of the project VHD, keep it native to disk and let the VHDs dip into it. This is how I ended up with a smaller second partition on the VHD because my ratio flipped from 30/70 to 60/40 operating system to data.

I think that as I finish writing this blog post (of three) this is much less about following a set of steps to manage your system but more of a brain dump of the obstacles that I faced that you may not have appreciated till now. Believe me, hitting a dead end after waiting 14hrs or so for a series of partition operations to complete is not fun! Hopefully I have spared you (some of) that frustration.

 

This is a cross post from my EMC blog, mainly for backup duplicity and to aggregate some of my past postings. My EMC blog used to be under the Conchango brand but was acquired by EMC so I’ve also retrospectively refreshed some of the old links and maybe a tweak a bit of content too.
permalink to the original post here

We were packed out at Notcutt House for May’s Silverlight UK User Group. Excellent to see that there is a lot of interest and passion still buzzing about for Silverlight. Mike Taulty was our sole guest speaker of the night and relished the extra time to go into the depths of the new features of Silverlight 5 (beta). Unfortunately Chris Arnold was called away at last moment and could not give us an update on the imminent release of PivotViewer v2 – however I’ve got him in my sights to do a full slot in the future, rather than just an update.

Here is another reminder that the big week long event is happening next week: Tech.Days.2011. If you have not already taken a look, there are plenty of great sessions (both speakers and content) at this mini-conference. There may be some seats still available for various sessions but much of it will be recorded too.

Tech.Days.2011

I normally give a shout out to the other user groups that are running events in the area, so here is an excerpt of those I put into my introductory slides at the start of the evening.

Windows Phone 7 May 24-25th wpug.net
London .NET May 31st dnug.org.uk
Canary Wharf .NET June 1st meetup.com/cwdnug
SharePoint UK June 1st suguk.org
UX Sketch Club June 8th meetup.com/ux-sketch-club
The Fantastic Tavern tba thefantastictavern.co.uk
Bing Maps tba bingmapsuk.ning.com

One of the best resources to look at or subscribe to is MSDN Flash.

Slides: Silverlight UK User Group Introduction – by Mark Mann.

Install Microsoft Silverlight

Slides: Silverlight 5: the beta and beyond – by Mike Taulty.

Install Microsoft Silverlight

VIDEO PLACEHOLDER – 30th May

The event was recorded, however our cameraman is temporarily overseas which has made post-production editing and downloading a bit tricky. Hopefully we’ll have the video ready on his return and I’ll update this page.

With thanks to Ian Smith (irascian) for recording the event and EMC Consulting for providing venue and beers.

irascian EMC Consulting


Want to join the discussion?
The event is geared to please/interest/inform both developers and designers alike, so if you are interested in coming along then please contact either myself or Michelle Flynn (here) and we will be glad to add you to our events mailing list.

Want to present or showcase?
We are always on the lookout for presenters for future sessions – whether it be a high or low level coding walk through, a workflow perspective or showcase demo. If you think that you have a topic/presentation that ought to be shared with the community then please contact me (here) and I’ll see if I can get you scheduled in!

This is a cross post from my EMC blog, mainly for backup duplicity and to aggregate some of my past postings. My EMC blog used to be under the Conchango brand but was acquired by EMC so I’ve also retrospectively refreshed some of the old links and maybe a tweak a bit of content too.
permalink to the original post here

This is part two of my journey into managing my laptop’s disk resources. Part one, was about moving my working environment to a new disk with minimum of fuss [You can read that post here:How to turn a physical disk into a bootable VHD”].

Disk management is one of those things that is never on interest until it’s really too late – when you are running low on space on your computer. With a bit of planning you can figure out a way to make best use of existing hardware or buy some more, but there is always something you never think of which makes you want to tinker more. I upgraded a 160GB hard drive for a superb 500GB Seagate Momentus XT Solid State Hybrid Drive to help alleviate my disk space woes and on discovering the excellent level of support that Windows 7 provides for virtual hard disks (VHDs), it made good sense to use this to minimise the downtime of moving disks.

Running low on space (again) So despite increasing my disk storage, moving to a system of running and backing up to VHDs I’m still desperately low of space on my Windows 7 system partition. I guess that when I finished juggling all my disks and files about, I naturally thought “job done” and that I would not have to revisit this particular issue for a while later. Windows of course, knew the real truth and decided to remind me how dangerously close to the edge I was getting by continuously popping up “low disk space” notifications.

Only 59MB free after Disk CleanupNot only do my running applications continuously compete for resources but it’s even pervading where application data is stored – for example; I recently wrote about trying to work around where Windows Phone 7 backups are stored, because my system drive could not cope. I could only manage a week of constantly running the  “Disk Cleanup” tool and scanning my system drive for any obvious candidates of deletion, but it’s pretty dire when despite these efforts you only end up with 59MB! Clearly, I need another strategy.

Now, all this turmoil is occurring on the system partition of the bootable VHD that I created in my previous post [You can read that post here:How to turn a physical disk into a bootable VHD”]. The techniques I’m about to show you has origins with physical disks too, however having baked my own VHD file it’s more relevant to VHD management.

When Windows 7 is running from a bootable VHD it has mounted the guts of a disk image file that is sitting on a physical partition on the machine. My VHD is big enough at 156GB and it holds two virtual partitions (one is the system, the other data). Normally, it’s just a case of resizing partitions, but how does that work with VHDs? Thankfully Windows 7 does not treat VHDs any differently to physical disks and therefore this can be managed using the inbuilt disk management tools.

So, from the “Start” taskbar, search for “Disk Management” or “Create and format hard disk partitions”. This will pull up the following control panel with details of your current system. Be careful with any of these options!

Disk Management view of the VHD

Partition Options in Disk ManagementSo, with my booted system partition languishing with only 169MB, I thought that I could shrink the larger DATA partition and reallocate this to the WIN7 partition. Shrinking can be performed on any partition, basically so long as it has some free space (there are a couple more rules but the computer checks these for you later).

The Disk Management console is pretty neat, in that if any of the options are not available, it will just disable the context menu options. Here, “Shrink Volume” is available on the DATA partition of my VHD, thus by clicking it, the console will spend a number of minutes analysing the partition.

Shrinking options on my VHD partitionThe pre-shrink checker reported to me that my drive is sufficiently bare to reduce my partition by 29GB. Note that this can depend upon where your data is stored on the disk itself, that is, shrinking considers data in roughly one contiguous group, so if there are system files scattered throughout your partition, this reduces the potential for the shrink operation hence why it’s not recommending all 34GB of free space.

I just want to free up some space, not all, so I’m liberating 12GB from my DATA partition that I’ll use on my system partition. Kicking off the shrink process involved leaving he laptop uninterrupted for 3 hours, best let it do it’s shrinking business alone.

On re-examining the Disk Management console, I’ve now got a 11.72GB unallocated partition on my VHD. It’s at this point I’ve realised a fundamental flaw in my plan. Although I’ve released this space, my VHD represents one piece of disk media and the unallocated space is in the wrong position! Quite simply, when you have split partitions on a physical disk they sit back-to-back and the same applies to virtual hard disks (VHDs) too. The console view demonstrates this very simply that my system partition (WIN7) can’t be extended because the DATA partition is in the way (shown by the red arrow stopping abruptly whereas the green arrow can reach across). This is why the “Extend Volume” option is only available to my DATA partition and not my WIN7 partition.

Post shrink - created unallocated space

With good intentions, this has not really helped solve anything though, except trim some dead space off my partition but the VHD remains unchanged as a whole. Remember that a VHD is a virtual representation of a physical hard disk; it still takes up disk space on the physical medium, albeit as one massive file rather than natively scattered over the drive. My VHD has not decreased in size unless I actually want to trim it, from which I have a couple of options…

Normally I’ll use GParted Live to manage my native disks/partitions since this boots from CD and because it is not running the native OS directly, it can shift and reorder any files that it sees fit (well, within reason!). I would thoroughly recommend GParted Live as it’s never let me down (famous last words) but I don’t think that I’m able to do the same here though because GParted Live uses a Linux Live Kernel which does not know how to mount VHDs as a live partition (wail).

Unsurprisingly, there is a tool called VHD Resizer, which is despite it’s age has established itself as a simple but robust tool to change a VHD file. There is the option to resize the VHD or change it’s type to/from fixed or dynamic sizing. It performs a sector by sector copy operation from one VHD to a new VHD and although I have not used this tool in this instance, it would appear to release or grow the tail end of the VHD – best in this case to resize my VHD just a bit bigger than 40.05GB + 103.44GB so that most of the unallocated space is chopped off. The one disadvantage of VHD Resizer is that it cannot be run from a Windows instance that is running – I would have to log onto my native Windows installation and run VHD Resizer against the inactive VHD file.

The last option I considered and for which I am the most gutted about since it does not work would be to do what I did before when creating the VHD in the first place and use the Disk2VHD tool! Hoping that Disk2VHD could reliably clone the selected partitions to separate and expanded VHDs, the clue was in the name, it’s not VHD2VHD. Unfortunately, the tool detects only native drives and converts those into new VHDs rather being duped into thinking that the host VHD is a native drive.

[You can read more about how I originally used Disk2VHD in the post here:How to turn a physical disk into a bootable VHD”].

Time for a rethink of how I’m going to get more space for my system partition – which is the real threat for grinding to a complete halt. I have a couple of ideas and I’ll report back once I’ve seen if they’ll do the job. At least you’ve seen how easy it is to shrink or expand those VHD partitions using the inbuilt Disk Management tool.

This is a cross post from my EMC blog, mainly for backup duplicity and to aggregate some of my past postings. My EMC blog used to be under the Conchango brand but was acquired by EMC so I’ve also retrospectively refreshed some of the old links and maybe a tweak a bit of content too.
permalink to the original post here