c++ - What's the meaning of "Internal" in "!heap -h" output in windbg?

Question

I am following this stackoverflow post What do the different columns in the "!heap -flt -s xxxx" windbg command represent

I am trying to understand the information printed out for one of the heaps that using up a lot of memory.

I can understand most of the columns but on my windbg, I see an additional column. Most of my entries are marked as Internal. I wonder what that means. I have done !gflags +ust. So, I can see the call stack for making the memory allocation. I can do it on most of the entries except the ones marked as Internal.

What does Internal mean? Is it something related to the implementation of LFH? If this is the internal implementation of LFH, how and when will these Internal heap entries return to the free list? It's holding up my memory for no reasons now.

Here is the output of !heap -h 0000000002330000 for your reference.

Index   Address  Name      Debugging options enabled
  8:   02330000 
    Segment at 0000000002330000 to 0000000002340000 (00010000 bytes committed)
    Segment at 00000000032b0000 to 00000000033b0000 (00100000 bytes committed)
    Segment at 00000000065a0000 to 00000000067a0000 (00200000 bytes committed)
    Segment at 00000000067a0000 to 0000000006ba0000 (00400000 bytes committed)
    Segment at 0000000006d80000 to 0000000007580000 (006f2000 bytes committed)
    Flags:                08001002
    ForceFlags:           00000000
    Granularity:          16 bytes
    Segment Reserve:      01000000
    Segment Commit:       00002000
    DeCommit Block Thres: 00000400
    DeCommit Total Thres: 00001000
    Total Free Size:      0000274d
    Max. Allocation Size: 000007fffffdefff
    Lock Variable at:     00000000023301f8
    Next TagIndex:        0000
    Maximum TagIndex:     0000
    Tag Entries:          00000000
    PsuedoTag Entries:    00000000
    Virtual Alloc List:   02330118
    Uncommitted ranges:   023300f8
    FreeList[ 00 ] at 0000000002330158: 0000000007454600 . 00000000032e3de0   (24 blocks)

Heap entries for Segment00 in Heap 0000000002330000
    0000000002330000: 00000 . 00a70 [101] - busy (a6f)
    0000000002330a70: 00a70 . 00860 [101] - busy (85f)
    00000000023312d0: 00860 . 038b0 [101] - busy (38af)
    0000000002334b80: 038b0 . 00330 [100]
    0000000002334eb0: 00330 . 00b60 [101] - busy (b34)
    0000000002335a10: 00b60 . 00160 [101] - busy (134)
    0000000002335b70: 00160 . 00090 [101] - busy (5c)
    0000000002335c00: 00090 . 00090 [101] - busy (5c)
    0000000002335c90: 00090 . 00040 [100]
    0000000002335cd0: 00040 . 00090 [101] - busy (5c)
    0000000002335d60: 00090 . 00020 [100]
    0000000002335d80: 00020 . 00130 [101] - busy (104)
    0000000002335eb0: 00130 . 00080 [101] - busy (53)
    0000000002335f30: 00080 . 00090 [101] - busy (65)
    0000000002335fc0: 00090 . 01060 [101] - busy (1034)
    0000000002337020: 01060 . 01020 [101] - busy (ff0) Internal 
    0000000002338040: 01020 . 00420 [101] - busy (3f0) Internal 
    0000000002338460: 00420 . 00090 [101] - busy (64)
    00000000023384f0: 00090 . 00260 [101] - busy (234)
    0000000002338750: 00260 . 00090 [101] - busy (5c)
    00000000023387e0: 00090 . 00080 [101] - busy (54)
    0000000002338860: 00080 . 00080 [101] - busy (4c)
    00000000023388e0: 00080 . 00030 [100]
    0000000002338910: 00030 . 00090 [101] - busy (5c)
    00000000023389a0: 00090 . 00090 [101] - busy (64)
    0000000002338a30: 00090 . 00260 [101] - busy (234)
    0000000002338c90: 00260 . 00060 [101] - busy (35)
    0000000002338cf0: 00060 . 00160 [101] - busy (134)
    0000000002338e50: 00160 . 00260 [101] - busy (234)
    00000000023390b0: 00260 . 00160 [101] - busy (134)
    0000000002339210: 00160 . 000c0 [101] - busy (94)
    00000000023392d0: 000c0 . 00080 [101] - busy (4c)
    0000000002339350: 00080 . 000c0 [101] - busy (84)
    0000000002339410: 000c0 . 000c0 [101] - busy (84)
    00000000023394d0: 000c0 . 000c0 [101] - busy (94)
    0000000002339590: 000c0 . 000c0 [101] - busy (94)
    0000000002339650: 000c0 . 000a0 [101] - busy (6c)
    00000000023396f0: 000a0 . 000c0 [101] - busy (94)
    00000000023397b0: 000c0 . 000a0 [101] - busy (6c)
    0000000002339850: 000a0 . 000a0 [101] - busy (6c)
    00000000023398f0: 000a0 . 02020 [101] - busy (1ff0) Internal 
    000000000233b910: 02020 . 000a0 [101] - busy (74)
    000000000233b9b0: 000a0 . 00060 [101] - busy (35)
    000000000233ba10: 00060 . 02020 [101] - busy (1ff0) Internal 
    000000000233da30: 02020 . 000a0 [101] - busy (6c)
    000000000233dad0: 000a0 . 000c0 [101] - busy (94)
    000000000233db90: 000c0 . 000a0 [101] - busy (6c)
    000000000233dc30: 000a0 . 00060 [100]
    000000000233dc90: 00060 . 001c0 [101] - busy (194)
    000000000233de50: 001c0 . 00260 [101] - busy (234)
    000000000233e0b0: 00260 . 000b0 [101] - busy (80)
    000000000233e160: 000b0 . 00020 [100]
    000000000233e180: 00020 . 000c0 [101] - busy (94)
    000000000233e240: 000c0 . 000a0 [101] - busy (6c)
    000000000233e2e0: 000a0 . 000a0 [101] - busy (74)
    000000000233e380: 000a0 . 001c0 [101] - busy (194)
    000000000233e540: 001c0 . 00020 [100]
    000000000233e560: 00020 . 000c0 [101] - busy (84)
    000000000233e620: 000c0 . 000c0 [101] - busy (84)
    000000000233e6e0: 000c0 . 000c0 [101] - busy (94)
    000000000233e7a0: 000c0 . 000c0 [101] - busy (94)
    000000000233e860: 000c0 . 00260 [101] - busy (234)
    000000000233eac0: 00260 . 000b0 [101] - busy (82)
    000000000233eb70: 000b0 . 00350 [100]
    000000000233eec0: 00350 . 00330 [101] - busy (2fc)
    000000000233f1f0: 00330 . 00440 [101] - busy (40c)
    000000000233f630: 00440 . 00420 [101] - busy (3f0) Internal 
    000000000233fa50: 00420 . 00460 [100]
    000000000233feb0: 00460 . 000b0 [101] - busy (80)
    000000000233ff60: 000b0 . 00060 [100]
    000000000233ffc0: 00060 . 00040 [111] - busy (3d)
    0000000002340000:      00000000      - uncommitted bytes.
Heap entries for Segment01 in Heap 0000000002330000
    00000000032b0000: 00000 . 00070 [101] - busy (6f)
    00000000032b0070: 00070 . 0c470 [101] - busy (c440) Internal 
    00000000032bc4e0: 0c470 . 00280 [101] - busy (254)
    00000000032bc760: 00280 . 000a0 [101] - busy (70)
    00000000032bc800: 000a0 . 00080 [101] - busy (4c)
    00000000032bc880: 00080 . 00080 [101] - busy (58)
    00000000032bc900: 00080 . 00070 [101] - busy (48)
    00000000032bc970: 00070 . 00080 [101] - busy (4b)
    00000000032bc9f0: 00080 . 00070 [101] - busy (42)
    00000000032bca60: 00070 . 00080 [101] - busy (4d)
    00000000032bcae0: 00080 . 000a0 [101] - busy (72)
    00000000032bcb80: 000a0 . 00080 [101] - busy (51)
    00000000032bcc00: 00080 . 000b0 [101] - busy (7c)
    00000000032bccb0: 000b0 . 00070 [101] - busy (46)
    00000000032bcd20: 00070 . 00080 [101] - busy (4c)
    00000000032bcda0: 00080 . 00080 [101] - busy (4f)
    00000000032bce20: 00080 . 00080 [101] - busy (52)
    00000000032bcea0: 00080 . 00090 [101] - busy (5d)
    00000000032bcf30: 00090 . 00080 [101] - busy (4b)
    00000000032bcfb0: 00080 . 00070 [101] - busy (43)
    00000000032bd020: 00070 . 00080 [101] - busy (4a)
    00000000032bd0a0: 00080 . 00080 [101] - busy (49)
    00000000032bd120: 00080 . 00070 [101] - busy (48)
    00000000032bd190: 00070 . 00070 [101] - busy (44)
    00000000032bd200: 00070 . 000a0 [101] - busy (69)
    00000000032bd2a0: 000a0 . 00070 [101] - busy (46)
    00000000032bd310: 00070 . 00070 [101] - busy (3c)
    00000000032bd380: 00070 . 000c0 [101] - busy (8c)
    00000000032bd440: 000c0 . 00070 [101] - busy (3c)
    00000000032bd4b0: 00070 . 00090 [101] - busy (5c)
    00000000032bd540: 00090 . 00090 [101] - busy (5c)
    00000000032bd5d0: 00090 . 00090 [101] - busy (5c)
    00000000032bd660: 00090 . 000a0 [101] - busy (5c)
    00000000032bd700: 000a0 . 00070 [101] - busy (44)
    00000000032bd770: 00070 . 00090 [101] - busy (5c)
    00000000032bd800: 00090 . 00070 [101] - busy (3c)
    00000000032bd870: 00070 . 00050 [100]
    00000000032bd8c0: 00050 . 00260 [101] - busy (234)
    00000000032bdb20: 00260 . 00070 [101] - busy (3c)
    00000000032bdb90: 00070 . 00090 [101] - busy (5c)
    00000000032bdc20: 00090 . 00070 [101] - busy (3c)
    00000000032bdc90: 00070 . 00070 [101] - busy (3c)
    00000000032bdd00: 00070 . 00090 [101] - busy (5c)
    00000000032bdd90: 00090 . 00070 [101] - busy (3c)
    00000000032bde00: 00070 . 00070 [101] - busy (3c)
    00000000032bde70: 00070 . 00090 [101] - busy (5c)
    00000000032bdf00: 00090 . 00070 [101] - busy (3c)
    00000000032bdf70: 00070 . 00cc0 [100]
    00000000032bec30: 00cc0 . 00330 [101] - busy (2fc)
    00000000032bef60: 00330 . 00440 [101] - busy (40a)
    00000000032bf3a0: 00440 . 00220 [100]
    00000000032bf5c0: 00220 . 00330 [101] - busy (2fc)
    00000000032bf8f0: 00330 . 04020 [101] - busy (3ff0) Internal 
    00000000032c3910: 04020 . 02020 [101] - busy (1ff0) Internal 
    00000000032c5930: 02020 . 00210 [100]
    00000000032c5b40: 00210 . 01020 [101] - busy (ff0) Internal 
    00000000032c6b60: 01020 . 01020 [101] - busy (ff0) Internal 
    00000000032c7b80: 01020 . 00440 [101] - busy (40c)
    00000000032c7fc0: 00440 . 00440 [101] - busy (40a)
    00000000032c8400: 00440 . 00430 [101] - busy (3f0) Internal 
    00000000032c8830: 00430 . 02020 [101] - busy (1ff0) Internal 
    00000000032ca850: 02020 . 02020 [101] - busy (1ff0) Internal 
    00000000032cc870: 02020 . 01020 [101] - busy (ff0) Internal 
    00000000032cd890: 01020 . 00420 [101] - busy (3f0) Internal 
    00000000032cdcb0: 00420 . 00420 [101] - busy (3f0) Internal 
    00000000032ce0d0: 00420 . 00420 [101] - busy (3f0) Internal 
    00000000032ce4f0: 00420 . 003a0 [100]
    00000000032ce890: 003a0 . 02020 [101] - busy (1ff0) Internal 
    00000000032d08b0: 02020 . 02020 [101] - busy (1ff0) Internal 
    00000000032d28d0: 02020 . 01020 [101] - busy (ff0) Internal 
    00000000032d38f0: 01020 . 00420 [101] - busy (3f0) Internal 
    00000000032d3d10: 00420 . 00420 [101] - busy (3f0) Internal 
    00000000032d4130: 00420 . 003a0 [100]
    00000000032d44d0: 003a0 . 00420 [101] - busy (3f0) Internal 
    00000000032d48f0: 00420 . 01020 [101] - busy (ff0) Internal 
    00000000032d5910: 01020 . 04020 [101] - busy (3ff0) Internal 
    00000000032d9930: 04020 . 01020 [101] - busy (ff0) Internal 
    00000000032da950: 01020 . 04020 [101] - busy (3ff0) Internal 
    00000000032de970: 04020 . 01020 [101] - busy (ff0) Internal 
    00000000032df990: 01020 . 04020 [101] - busy (3ff0) Internal 
    00000000032e39b0: 04020 . 00420 [101] - busy (3f0) Internal 
    00000000032e3dd0: 00420 . 00020 [100]
    00000000032e3df0: 00020 . 04020 [101] - busy (3ff0) Internal 
    00000000032e7e10: 04020 . 02020 [101] - busy (1ff0) Internal 
    00000000032e9e30: 02020 . 01020 [101] - busy (ff0) Internal 
    00000000032eae50: 01020 . 02020 [101] - busy (1ff0) Internal 
    00000000032ece70: 02020 . 01020 [101] - busy (ff0) Internal 
    00000000032ede90: 01020 . 000f0 [100]
    00000000032edf80: 000f0 . 01020 [101] - busy (ff0) Internal 
    00000000032eefa0: 01020 . 01020 [101] - busy (ff0) Internal 
    00000000032effc0: 01020 . 02020 [101] - busy (1ff0) Internal 
    00000000032f1fe0: 02020 . 02020 [101] - busy (1ff0) Internal 
    00000000032f4000: 02020 . 00420 [101] - busy (3f0) Internal 
    00000000032f4420: 00420 . 00160 [100]
    00000000032f4580: 00160 . 02020 [101] - busy (1ff0) Internal 
    00000000032f65a0: 02020 . 02020 [101] - busy (1ff0) Internal 
    00000000032f85c0: 02020 . 02020 [10

Answer 1

TL;DR: Heap blocks marked as "internal" have a special flag in _HEAP_ENTRY.Flags

[edit] revised my previous answer with a proper answer.

---

Here's my guess attempt to your question.

According to the windbg help, the "!heap" command code is located in exts.dll (i.e. winxpexts.dll).

Put this DLL on IDA and downloaded symbols for it. There’s only one occurrence of "Internal" in the DLL, inside the DumpHeapEntry() function :

.text:0192463D                 movzx   eax, byte_1963152
.text:01924644                 test    eax, eax
.text:01924646                 jz      short loc_1924656
.text:01924648                 push    offset aInternal ; " Internal "
.text:0192464D                 call    _ExtensionApis.lpOutputRoutine ; some sort of printf routine

The output of "Internal" is therefore conditioned by the value of byte_1963152 : if byte_1963152 is not 0, then "Internal" is printed. Only once occurrence of write value with anything else than 0 happens (in ReadHeapEntry() which is called at the start of DumpHeapEntry() ):

.text:0191F025                 movzx   eax, [ebp+var_B]
.text:0191F029                 and     eax, 8
.text:0191F02C                 jz      short loc_191F035
.text:0191F02E                 mov     byte_1963152, 1

This translates to:

if((UINT)var_B & 8)
    byte_1963152 = 1;

var_B is set here :

text:0191EFF7                 mov     eax, [ebp+var_18]
.text:0191EFFA                 mov     edx, [ebp+var_14]
.text:0191EFFD                 mov     cl, 10h          ; shift right by 0x10 bits
.text:0191EFFF                 call    __aullshr
.text:0191F004                 mov     [ebp+var_B], al

__aullshr stands for "Arithmetic Unsigned Long Long Shift Right". In the above code eax is the low 32-bit part of a 64-bit unsigned long long, while edx is the high 32-bit part. Notice that var_B is a 8-bit quantity ('al' register is used).

Hence:

// where var_14_18 is a combination (64-bit) of var_14 and var_18
var_B = (char)(var_14_18 >> 0x10 );

var_14 and var_18 are set here :

.text:0191EF01                 push    0
.text:0191EF03                 push    offset aAgregatecode ; "AgregateCode"
.text:0191EF08                 push    0
.text:0191EF0A                 push    0
.text:0191EF0C                 call    _GetShortField@16 ; GetShortField(x,x,x,x)
.text:0191EF11                 mov     [ebp+var_18], eax  ; high part
.text:0191EF14                 mov     [ebp+var_14], edx  ; low part
; cut
.text:0191EF28                 mov     ecx, [ebp+var_18]
.text:0191EF2B                 and     ecx, _EncodeFlagMask ; from HEAP.EncodeFlagMask
.text:0191EF31                 jz      short loc_191EF75
.text:0191EF33                 mov     edx, [ebp+var_18]
.text:0191EF36                 xor     edx, _CrtHeapCode ; from HEAP.Encoding.Code1
.text:0191EF3C                 mov     eax, [ebp+var_14]
.text:0191EF3F                 xor     eax, dword_1963194 ; from HEAP.Encoding.Code2
.text:0191EF45                 mov     [ebp+var_18], edx
.text:0191EF48                 mov     [ebp+var_14], eax    

So, windbg use the GetShortField() function on "AgregateCode" and sets both of the aforementioned variable (which is also a single unsigned long long value). Note that it also uses the HEAP.Encoding.Code1 and HEAP.Encoding.Code2 to XOR both of the value (HEAP is the the current heap from which the heap entry is a part).

"AgregateCode" is a field of both HEAP_ENTRY and HEAP_FREE_ENTRY structures (from Win 8.1 x86):

0:000> dt _heap_entry -r2
ntdll!_HEAP_ENTRY
   +0x000 Size             : Uint2B
   +0x002 Flags            : UChar
   +0x003 SmallTagIndex    : UChar
   +0x000 SubSegmentCode   : Uint4B
   +0x004 PreviousSize     : Uint2B
   +0x006 SegmentOffset    : UChar
   +0x006 LFHFlags         : UChar
   +0x007 UnusedBytes      : UChar
   +0x000 FunctionIndex    : Uint2B
   +0x002 ContextValue     : Uint2B
   +0x000 InterceptorValue : Uint4B
   +0x004 UnusedBytesLength : Uint2B
   +0x006 EntryOffset      : UChar
   +0x007 ExtendedBlockSignature : UChar
   +0x000 Code1            : Uint4B
   +0x004 Code2            : Uint2B
   +0x006 Code3            : UChar
   +0x007 Code4            : UChar
   +0x004 Code234          : Uint4B
   +0x000 AgregateCode     : Uint8B

This translated to C, gives:

          typedef struct _HEAP_ENTRY                       // 20 elements, 0x8 bytes (sizeof) 
          {                                                                                   
              union                                        // 6 elements, 0x8 bytes (sizeof)  
              {                                                                               
                  struct                                   // 3 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             UINT16       Size;                                                      
/*0x002*/             UINT8        Flags;                                                     
/*0x003*/             UINT8        SmallTagIndex;                                             
/*0x004*/             UINT8        _PADDING0_[0x4];                                           
                  };                                                                          
                  struct                                   // 4 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             ULONG32      SubSegmentCode;                                            
/*0x004*/             UINT16       PreviousSize;                                              
                      union                                // 2 elements, 0x1 bytes (sizeof)  
                      {                                                                       
/*0x006*/                 UINT8        SegmentOffset;                                         
/*0x006*/                 UINT8        LFHFlags;                                              
                      };                                                                      
/*0x007*/             UINT8        UnusedBytes;                                               
                  };                                                                          
                  struct                                   // 2 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             UINT16       FunctionIndex;                                             
/*0x002*/             UINT16       ContextValue;                                              
/*0x004*/             UINT8        _PADDING1_[0x4];                                           
                  };                                                                          
                  struct                                   // 4 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             ULONG32      InterceptorValue;                                          
/*0x004*/             UINT16       UnusedBytesLength;                                         
/*0x006*/             UINT8        EntryOffset;                                               
/*0x007*/             UINT8        ExtendedBlockSignature;                                    
                  };                                                                          
                  struct                                   // 2 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             ULONG32      Code1;                                                     
                      union                                // 2 elements, 0x4 bytes (sizeof)  
                      {                                                                       
                          struct                           // 3 elements, 0x4 bytes (sizeof)  
                          {                                                                   
/*0x004*/                     UINT16       Code2;                                             
/*0x006*/                     UINT8        Code3;                                             
/*0x007*/                     UINT8        Code4;                                             
                          };                                                                  
/*0x004*/                 ULONG32      Code234;                                               
                      };                                                                      
                  };                                                                          
/*0x000*/         UINT64       AgregateCode;                                                  
              };                                                                              
          }HEAP_ENTRY, *PHEAP_ENTRY;

Thus we have the following pseudo-code (minus some other checks):

high_part, low_part = GetShortField(0,0,"AgregateCode", 0);
high_part ^= HEAP.Encoding.Code1;
low_part ^= HEAP.Encoding.Code2;
AgregateCode = Make64BitFromTwo32Bit(high_part, low_part);    
char var_B = (char)(AgregateCode >> 0x10);
if(var_B & 8)
    printf("Internal");

Given that "AgregateCode" is ... well, an aggregate of Code1 to Code 4 :

                  struct                                   // 2 elements, 0x8 bytes (sizeof)  
                  {                                                                           
/*0x000*/             ULONG32      Code1;                                                     
                      union                                // 2 elements, 0x4 bytes (sizeof)  
                      {                                                                       
                          struct                           // 3 elements, 0x4 bytes (sizeof)  
                          {                                                                   
/*0x004*/                     UINT16       Code2;                                             
/*0x006*/                     UINT8        Code3;                                             
/*0x007*/                     UINT8        Code4;                                             
                          };                                                                  
/*0x004*/                 ULONG32      Code234;                                               
                      };                                                                      
                  };                                                                          
/*0x000*/         UINT64       AgregateCode; 

If you shift 0x10 and AND 8 the AgregateCode field you end up finally testing the 11th bit (start counting at 0) of Code1.

As the structure is a big union, you finally end up testing: _HEAP_ENTRY.Flags

It happens that a heap flag has already the value 8, its name is: HEAP_ENTRY_VIRTUAL_ALLOC

http://doxygen.reactos.org/da/ddb/heap_8h_source.html#l000