It's hard to come by any sort of "official"-looking standard or such, but the
_MOTIF_WM_HINTS property does seem to come from the... Motif toolkit (ok, you
guessed that :-) ).
See the MotifZone site.
Warning: what follows is incomplete, but should shed some light I hope.
Documentation for the XmNmwmDecorations, XmNmwmFunctions and XmNmwmInputMode
functions of the toolkit indicates that that property is a bitmask of various
values used to control the appearance, functions (resize, move, ...) and input
mode that the window manager should provide/give to the window.
See man vendorshell, or Oreilly Motif reference books, Vol6a chapter 16.
Properties are a part of the whole X11 thing. A window can have any number of
properties defined on it. Properties have a name, but setting/getting
properties is done through an "atom" (identifier of sorts) to avoid sending
the whole string on the wire for every get/set call.
See Properties and Atoms
The currently window manager (if any) can react to window property
changes by setting the appropriate event filter and acting on PropertyNotify
events, or simply inspecting the properties the window has when it gets mapped (or moved, or whatever). If the window manager knows of the _MOTIF_WM_HINT property, it'll
interpret those and (hopefully) do what you want.
(Note: I'm not entierly sure if that privilege is devolved to the window
manager, or if other windows can listen to those "PropertyNotify" events. Not
sure that's actually relevant to your question.)
So the code you have works just fine as long as your window manager knows
about the _MOTIF_WM_HINTS property.
You start by getting the atom
(identifier/shortcut) for it with XInternAtom, and setting its value via XChangeProperty()
before the window is actually drawn via MapWindow() (not sure if that would
work if you do it after the MapWindow(), that could depend on your window
manager).
[Edit: setting the .decorations part to zero clears all the decoration bits, so this requests that the window manager leave your borders the hell alone, essentially.]
I can't come up with somewhere "official" with the definition of that struct. It's defined in lib/X11/MwmUtils.h of the openmotif-2.3.3 distribution. Not sure how the .flags entry is used (can't find the code for the life of me :-/ ) but I suspect it's used to indicate which of the {decoration, function, inputMode} "sub-properties" you're acting on. (Don't take my word for that.)
As a side note, using _MOTIF_WM_HINTS might not be your best option right
now. Have you looked at the Extended Window Manager hints
specification and other information/standards/drafts over at freedesktop? I'll wager most "modern" window managers/desktop environments/kitchen sinks will tend to adhere to that rather than keeping backwards compatibility with Motif. All depends on what you're coding for I guess.
Thanks for reading :-)