I’ve done some work with DTrace and Perl, using my Devel::DTrace::Provider and Andy Armstrong’s Devel::DTrace, but there’s never been a ustack helper: because of the structure of the Perl interpreter, it’s impossible to write one – at least, in the intended manner…
Perl and ustack helpers - the big problem
What we can try though, is to annotate a single C frame with as much of the Perl stack as we can find by chasing pointers from that frame.
This post explains my experiments with that idea. To give the game away early, it doesn’t entirely work, but there are some techniques here that might be useful elsewhere.
A lot of what I’ve done here is based directly on the existing ustack helpers for other languages, specifically V8 and Python.
Dynamically loading the ustack helper
If at all possible, I want the helper to work on an unmodified Perl binary - if the thing works at all, I want to be able to use it like a CPAN-style module rather than having to patch Perl. The first problem is to get the the helper loaded.
Given the way libusdt works, it
seems likely we can load a helper just like a provider, by
ioctl()ing the DOF down to the kernel from a Perl XS
extension. Of course, that’s all DTrace’s DOF-loading
routine does anyway, just we’ll be doing it slightly later on in the
Unfortunately this facility isn’t part of libusdt’s public API yet, but it’s really not that much code, especially if we’re only supporting Illumos-based systems.
Actually building the helper DOF is trivial: compile the script with
dtrace_program_fcompile(), and wrap it in DOF with
Loading the DOF containing the helper program works, and means we can initialise the helper from an extension module, rather needing to patch it into Perl’s build process.
Finding - or rather creating - a frame to annotate
Ideally we need a stack frame which is always found in the interpreter’s C stack, for which we can easily find the address of the function, and where the stack is passed as one of the arguments. There’s no such frame in the standard Perl interpreter, but we can manufacture one. Perl lets us replace the “runops loop” with one of our own, and we can use this to meet all of our requirements.
The runops loop function is the core of the interpreter, responsible for invoking the current “op”, which returns the new current op.
The usual, non-debug, runops loop looks like this (in Perl 5.16.2):
1 2 3 4 5 6 7 8 9 10 11
The top-level loop is always visible during execution, and we can replace the usual function with one of our own, fulfilling our first two requirements.
If we make this loop execute ops through another function, and pass that function a pointer to the Perl stack, we fulfill the final requirement. These functions are dtrace_runops and dtrace_call_op:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
We’ll target the annotation at dtrace_call_op(), and attempt to walk the stack starting from the PERL_CONTEXT pointer we’re given.
Actually installing the alternative runops loop is a standard Perl extension technique, and we just need to make sure it happens early enough that the top-level loop is ours rather than the standard one.
The primary purpose of the ustack helper is to provide a descriptive
string for a frame where there’s no corresponding C symbol - for
JITted code, say. If there is such a symbol, the ustack helper’s
string will be ignored - and in this case, there is,
Fortunately there’s a mechanism for adding annotations to these
frames, and that’s what we’ll use here: a string beginning with an
@ will be used as an annotation. In the Python
it looks like this:
Targetting a specific frame
In a helper action,
arg0 is the program counter in that stack
frame. If we make the address of our inserted
function available to the helper, and of the preceding function, we
can compare the pc to these two addresses to determine when we’re
annotating that function.
this->end have been initialised to the
dtrace_call_op and the preceding function:
1 2 3 4 5 6 7 8 9 10 11
For a reason I’m not entirely sure of, combining these predicates into one doesn’t work.
Passing values into the helper
With the extra control over the helper initialisation we get from loading it “by hand”, it turns out that macros work fine! We can use this to pass values into the helper: symbol addresses and structure member offsets.
It doesn’t seem to be possible to simply
#include <perl.h> - the D
compiler barfs badly on Perl’s headers, which are
.. involved. Fortunately we can do the necessary sizeof and offsetof
work in C and pass the results into D with macros. This should buy at
least some ability to cope with changes to Perl’s data structures,
though more sweeping changes will still break things entirely.
Macros are strings, so all the values passed need to be formatted with
sprintf; at least this is just setup code.
Copying a C string
Unless I’ve missed something, this is awkward. Our final stacktrace
string that the helper will return as the frame’s annotation is
allocated out of D scratch space, so we need to copy C strings from
userspace into it. If we have the string’s length available this is
easily done with
copyinto(), but if we’ve just got a
Ideally we could take the string’s length with
strlen() and do a
copy – but
strlen isn’t available to helpers.
It doesn’t seem to be possible to use
strchr() either, since it
string and not
char *, so we can’t find the length
I’m not sure if the lack of
strlen is an oversight, or if there’s
some reason that it’s unsafe in arbitrary context: it seems that if
strchr is safe,
strlen also ought to be.
We can’t just copy a fixed length of data, so a character-by-character “strncpy” is needed:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Walking the stack
After all that, actually walking the stack from the pointer we’ve been passed is relatively simple. Using the information in Perlguts Illustrated, we walk the context stack, appending frame annotations to our string buffer.
Obviously it’s only possible to walk a limited number of frames, and with the default size limit on helper size and the ops required for string copies, quite a limited number of frames.
Here’s an incredibly simple example of the output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
This shows file:lineno pairs for each stack frame representing a subroutine call that was found walking the context stack.
Here’s a (slightly) less trivial example, taken during a run of the CPAN shell program:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
The code, and its limitations
The code is available on Github. I don’t plan to release this module to CPAN any time soon!
For anything but the most trivial examples this code probably won’t provide useful Perl stacktraces, and it’s only been tried on Perl 5.14.2 built with threads, on an Illumos-derived system.
It certainly won’t work on the Mac, since ustack helpers are disabled there, and won’t work without threads enabled in Perl because of an implementation detail of Perl OPs we’re exploiting that’s different without threads.
Hopefully though, this post sheds a bit of light on ustack helpers, and maybe there are some interesting techniques here for other situations.