VMware vs. Vista - Hooking the Kernel

VMware announced its VMsafe API availability at VMWorld Europe this week, complete with a handful of security vendors who support it.This is a timely announcement that should serve its purpose of allowing some "authorized" access to kernel operations of the hypervisor.

I say "authorized" because this approach stands in stark contrast to the challenges Microsoft had when it implemented Kernel Patch Protection, which had an API to allow security products access to kernel operations, also in an "authorized" manner. (I would enjoy hearing about specific functional differences between Vista's KPP API and VMsafe).

Of course, the big difference is that it was essentially a time-honored custom to hook Microsoft's kernel in all sorts of unauthorized ways, and KPP also comes with an inline integrity checker that is intended to negate these approaches (for good reason, since these are the same approaches used by malware). To my knowledge, VMware does not use an integrity checker on its own kernel code.

So VMware is doing what is widely seen as "the right thing" out of the gate. I think it is the right thing as well. But I can't help but wonder when the first rootkit will arise that leverages these same APIs. We have the same problem as code signing - there is no single authority in the sky, and solutions like these may need to discriminate against small companies in the future. (It is already extremely common for malware to masquerade as anti-malware, and I can't help but wonder if this extremely interesting "rootkit" detector that leverages hardware virtualization a la Blue Pill and Vitriol is in that same boat - I have requested more information on what might be a useful solution.)

All this being said, VMsafe is still the right way to go because there is nothing keeping anyone from hooking  VMware's  linux-heritage kernel for these purposes anyway.

In a much broader sense, the tradeoffs between flexible architectures - not only with VSafe but also OVF and any other protocols and formats - and security in terms of attack surface... I can feel the bloat coming... and I like it (as an IT professional) but I don't like it (as a security professional).

Comments

[John Troyer]

Pete, nice article, but one correction. The ESX Server "vmkernel" is not derived from Linux. The Service Console is derived from Red Hat, but even that has gone away in ESX Server 3i.

[Stacey]

HAPPY BIRTHDAY! How the heck are you??

[dust]

D0Z9og gps34JsDl901Wp4kV