debugging on xen

Xen greatly facilitates debugging of OS kernels and modules by allowing
source level debug. Allow me to introduce debuggers I wrote up late 2007,
mostly from scratch, for xen. The first is gdbsx which was just recently
merged into xen unstable branch. The second is kdb, whose primary goal
is to allow debug of the hypervisor and also dom0.  kdb is somewhat
a misnomer. I started with porting kdb from linux but later found it
easier and quicker to just write one up from scratch. However, I credit
linux kdb with allowing me to learn how kernel debuggers work.
gdbsx is intended for debugging non privileged guests. It runs on
dom0 and communicates to the hypervisor directly. It is fairly light
weight. By adding little support in xen for external debuggers, I was
able to keep it fairly simple. Once started on dom0, it talks to remote
gdb and allows for standard debugging of any guest, PV or HVM, 32 or 64
bits. It resides in tools/debugger/gdbsx directory, and the README gives
details on usage. It’s as simple as building and installing a hypervisor
with gdbsx support, and then just running gdbsx on dom0.  When attached to
guest, it pauses the guest, does initial setup and is ready for debug. It
accepts commands from remote gdb, and liaisons between the remote gdb and
xen.
kdb is an assembly level debugger intended for debug of the hypervisor
and dom0, but can also be used to debug non-priviliged guests also. It finds
the basic hypervisor symbols like function names in it, and shows
meaningful stacks etc. By providing some hints on basic symbol table
for guests, it can serve to be an assembly level debugger for guests
also. kdb provides fairly large variety of commands, from standard
commands to being able to examine domain/vcpu structs, get info on
pages, looking up into m2p/p2m tables, etc…  It current resides in an
external tree at http://xenbits.xensource.com/ext/debuggers.hg. The
README in the xen/kdb sub directory provides details on usage.
I hope this helps engineers with their development and support of all of non-user level software on xen. Feedback always welcome, please post to xen-devel list.
Mukesh Rathor