Tracing vmkernel and the TraceViz GUI
 
TraceViz allows a developer to annotate source code with trace points
carrying an ID, a class, and some data, then later view the sequence
of events in a friendly java gui.
 
Basic outline of how to enable tracing and get data:
 
+ Enable the event classes you're interested in by doing "echo enable
  [class name list] > /proc/vmware/trace/trace-control"
  For example, if you want scheduler per-world and per-pcpu events,
  do "echo enable SCHED_PCPU SCHED_WORLD >/proc/vmware/trace/trace-control"
 
+ Activate tracing, specifying a buffer size and whether or not you
  want a circular buffer by echoing the command "start bufsize
  [circular]" into the trace-control proc node. For instance, "echo
  start 100000 > /proc/vmware/trace/trace-control" begins tracing with
  a 100000 entry NON-circular buffer, which will stop tracing when it
  fills up. The command "start 100000 circular" would have made the
  same size, but with a circular buffer.
 
+ Wait a little while for your relevant trace data to build up. You
  can cat the trace-control proc node to see how many events have
  been seen so far.
 
+ Optionally stop tracing when you're done with interesting work:
  "echo stop > /proc/vmware/trace/trace-control". This will
  preserve existing data.
 
+ Dump the data to a file:
  "bora/build/<BLDTYPE>/apps/vmktracedump/vmktracedump outfilename"
 
+ Run the GUI on the generated file:
  "java -jar bora/support/tools/java/traceviz.jar outfilename"
 
+ Configure the colors/heights/patterns of the diplay in the floating
  window that pops up. Use the drop-down menus on the left side of the
  main graph to configure what type of events you see in each timeline.
  Use the slider in the floating window to scroll through time.
 
 
Events:
 
 An event is the basic primitive of TraceViz. Events have the following
 properties:
 
  - class
  - id
  - pcpu
  - currently running world
  - custom data
  - timestamp
     + implicitly, duration (see discussion of widths in the section
       below about defining timelines)
 
 These properties can be used to customize their display in the
 TraceViz GUI. For instance, the height of a bar for an event may be
 made conditional on its custom data and it may be colored differently
 depending on the currently-running world.
 
 
Defining timelines in the GUI:
 
 Each timeline (horizontal graph in the GUI) is defined by a
 constraint.  Only events matching this constraint are shown in that
 line and the width of an event in a timeline is determined by the
 time distance between the start of that event and the start of the
 next event that ALSO matches this constraint.
 
 An example constraint is "class == SCHED_PCPU && pcpu == 1", which
 would show per-pcpu scheduler stats for pcpu 1. There's nothing
 special about the class/pcpu combination, however. You could have
 specified a constraint "class == SCHED_WORLD && pcpu == 1", but
 you're likely to get nonsensical data (all per-world state
 transitions that happen when any world is on pcpu 1), or at least
 nonsensical widths. These constraints are configured by the drop-down
 boxes to the left of the timeline.
 
 
Customizing display in the GUI:
 
 TraceViz's usefulness stems largely from its customizability. However,
 it often takes a while to get a feel for how to best customize the
 display to investigate a particular problem. The floating controller
 window contains one tabbed pane for each event class. Within that,
 its table contains one line per event ID within that class. On a per-
 event basis, you can configure the following properties:
 
- Height depends on (wid|pcpu|data|eventID|none):
         This property makes the height of bars for this event proportional
         to the specified event property. This is particularly useful
         for events whose "data" properties represent a magnitude of sorts,
         such as vmkperf events, whose data property represents the number
         of cpu events that that took place during the previous millisecond.
 
 
- Label type (wid|pcpu|data|eventName|none):
        Controls the text that appears on top of the bar for this event.
        If the bar is too narrow to fit the text, the text won't be shown.
       
- Height multiplier:
         Simply multiplies the height of the bar by this factor. This is
         a handy way to visualize certain kinds of events. For instance,
         with per-pcpu scheduler events, you might make RUN have a multiplier
         of 1.2, BWAIT have a height multiplier of 0.6, and IDLE have a
         multiplier of 0.4, while they all have height depends on "none."
         This makes it easy to see at a glance "how active" the pcpu is. A
         similar trick works well with per-world data (setting RUN to a
         greater height)
 
- Color/Patter:
        These allow you to select the color and pattern for the event type
        from a color picker or drop-down pattern selection list.
 
 
 
Remote connection:
 
 It's easy to connect to a remote host and get live data over the network.
 
 On the destination host:
    + run "vmktracedump -p portnum"
      This starts a server listening on "portnum" for connections. It doesn't
      need to be restarted even if you restart the trace.
 
 On the gui workstation:
    + run "java -jar traceviz.jar -h hostname portnum"
 
 This will connect remotely and obtain all available data. You can click the
 "refresh" button at the top to load new data. This is particularly useful
 with a circular buffer that keeps filing with all new data.