InfoPurge Programming Tips

Setting up a Symbol Server

Wed, 21 Sep 2011 09:51:00 +0100

Setting up a Symbol Server

Overview.
Set your projects up to generate symbol files (PDB - Program DataBase).
Set your release build or daily build to add PDBs and product binaries to a symbol store.
Change debugger configuration to point to use a symbol server to access the symbol store (and the Microsoft online symbol store).
How the debugger locates symbol files.

Overview

In order to effectively debug problems reported by customers in release versions of your products, you will need to generate and store the symbol files (PDB files - program database) for all DLLs and EXEs that comprise the product. This applies to debugging both live applications and crash dumps.
[Note that you must create the PDBs at the same time as the product binaries because the debugger uses timestamp and GUID information to match PDB files to binaries during debugging and these will change when you rebuild - even if the source code is the same.]
Native code symbol files contain full annotations of the binary including: source filenames and line numbers, functions (static, private and public), types, variable names, Frame Point Omission (FPO) data etc. Managed symbol files contain less debug information because most of the symbol information is available in the assembly file metadata.
You can either store the symbols with the rest of the product in your “release directory” or you can store the symbols (and binaries) in a symbol store located on a PC on your intranet for easy access by all members of the programming team. The latter case is described here.
You can configure your daily build or automated release procedure to automatically add symbols to the symbol store after a successful build.

When debugging, you configure Visual Studio or WinDbg to access the Symbol Store using a symbol server (symserv.dll).

In addition to product specific symbols, you will also need matching symbols for the system DLLs that were loaded on the customer’s machine at the time a dump file was generated. Microsoft provides an online symbol server containing symbols for these system DLLs.
You may also need copies of the various .NET version DLLs in order to debug managed applications.

Generating Symbol Files

For C# and VB.NET: View the project properties. Select Build tab - then scroll down and click the Advanced button. In the Advanced Build Settings dialog make sure that Debug Info is set to “pdb-only” for release builds and “full” for debug builds. Full allows the debugger to attach to the process when it is running. From the command line use /debug:full or /debug:pdbonly.

For C++: View the project properties. Select C/C++. Select the General property page. For the Debug Information Format option select Program Database. From the command line use /Zi.

The PDB file will be called project.pdb and will be placed in the project output folder along with the binary (EXE or DLL).

Setting up a Symbol Store

To ensure that you always have the correct symbol files matching the binaries you are debugging you should set up a company Symbol Store.
You then add your binaries and symbols to the symbol store during your daily builds or at least during your release builds.
Your build server machine is a good candidate for the location of your symbol server.
The symbol server tools are part of the Debugging Tools For Windows. Make sure you use the latest version of the debugging tools and check for updates regularly.

To create a Symbol Store.

Download the Debugging Tools for Windows from http://www.microsoft.com/whdc/Devtools/Debugging/default.mspx and install on the server.
Create a folder to store the symbols (e.g. c:\SymStore)
Share this folder on the network so other users can reach it.
Add the symstore command to your daily build or release batch files. e.g. to add files recursively (/r) from the product root folder (/f r:\rootproductfolder) to the symbol store folder (/s c:\symstore) with the product name (/t “MyApp”) with a specified version (/v “%1”) and comment (/c “Daily Build”) use…

symstore add /r /f r:\rootproductfolder /s c:\SymStore /t “MyApp” /v “%1” /c “Daily Build”

This example uses /v “%1” where the symstore command is in a DOS batch file. %1 is just parameter 1 passed into the batch file. If you are running inside an MSBUILD file then use “$(Version)” or something similar. The main point here is that you use a different version (e.g. the build number of the product) each time you add files to the symstore. This way you can easily identify versions that can be deleted from the symbol store should the symbol store become bloated over time.

For detailed options to the symstore command see http://msdn.microsoft.com/en-us/library/ms681378(VS.85).aspx

Debugger access to the symbol store

Install Debugging Tools for Windows if you are using WinDbg. Visual Studio should come with a version of symsrv.dll.

Visual Studio - Select Tools \ Options \ Debugging \ Symbols. Add “http://msdl.microsoft.com/download/symbols” and “\MyServer\SymStore”. Include a local cache for speed (e.g. c:\localstore). Symbol files and binaries downloaded from the symbol servers are cached in c:\localstore for future use. If c:\localstore gets too large you can either delete the whole folder or use AgeStore.exe (part of Debugging Tools for Windows) to remove files older than a particular date.

WinDbg - Select File \ Symbol File Path and include: SRV*c:/localstore*http://msdl.microsoft.com/download/symbols;SRV*c:\localstore*\MyServer\SymStore

You can append additional paths on the end delimited by a semicolon.
This command says: SRV* (shorthand for symsrv*symsrv.dll) - indicates that this part of the path is a symbol server reference and the debugger should use symsrv.dll to access the remote symbol store. The*c:\localstore*” indicates that downloaded symbols and binaries should be cached in the specified folder for future use.
http://msdl.microsoft.com/download/symbols is the location of the Microsoft public symbol store and \MyServer\SymStore is the location of your local store.
Various cascading cache options are available. For more information see the Symbol Store help files contained in Debugging Tools for Windows.

Also, in WinDbg you should add the same paths to the File \ Image File Path dialog.

How the debugger finds the right symbol file

The linker (compiler for managed code) embeds the path to the PDB file into the exe or DLL along with a GUID and timestamp. You can see this header using dumpbin.exe (part of the windows SDK). From a command prompt type dumpbin /headers myapp.exe >headers.txt. Then open headers.txt in notepad and you should see something like…

Debug Directories
Time Type Size RVA Pointer
———— ——— ———— ———— ————
4B73E828 cv 48 00002BC8 1BC8 Format: RSDS, {2AB1D3E2-6F7F-43EB-903B-CA57510C08D5}, 2, c:\dev\myapp.pdb

The debugger will search the following folders.

The folder where the assembly was loaded (exe or dll).
The folder specified in the Debug Directories section of the assembly (c:\dev\myapp.pdb in the example above).
Any folders that you add to the symbol search path (see next section).

Tools.

Dumpbin.exe - Dumps (native code) information contained in a binary file (DLL, EXE, COM).
SymChk.exe - part of Debugging Tools For Windows - you can use to determine if a PDB has public or private symbols present.
SymStore.exe - part of Debugging Tools For Windows - used to create a symbol store and add/remove files.
AgeStore.exe - part of Debugging Tools For Windows - can be used to delete files in a directory heirarchy (e.g. a local symbol cache) older than a particular age.
SymSrv.dll - must be installed on the same machine as the debugger.

More Information.
http://msdn.microsoft.com/en-us/library/b8ttk8zy.aspx
http://www.microsoft.com/whdc/Devtools/Debugging/default.mspx - See the help file description of symbol servers.
http://msdn.microsoft.com/en-us/library/ms681378(VS.85).aspx

This blog was transferred from the StackHash product web site. StackHash is now an OpenSource code project at codeplex.

Integrate creation of WinQual mapping files with your build

Wed, 21 Sep 2011 09:41:00 +0100

If you use WinQual you’ll be familiar with running the Microsoft Product Feedback Mapping Tool each time you ship your product. You might not have discovered that the tool includes command line support, which means you can create mapping files as part of your build.

Run appmap.exe /? for a list of command line options:

-s - Folder containing files to be mapped
-d - Output product map xml file
-n - Product name
-v - Product version
-l - Log file
-e - Skip scanning subdirectories
-f - Allow tool to overwrite output file

The snippet below shows an MSBuild target that invokes appmap.exe for StackHash (each time our release installer is built):


<ItemGroup>
<FilesToMap Include="R:\Path\StackHash.exe" />
<FilesToMap Include="R:\Elsewhere\StackHashUtilities.dll" />
<FilesToMap Include="..." />
</ItemGroup >

<Target Name="CreateWinQualMap"
DependsOnTargets="BuildInstaller"
Condition=" '$(Configuration)' != 'Debug' " >
<Message Text="Creating WinQual Product Mapping File" />
<Exec Command="DEL /q R:\StackHash\AppMap\*.*" />
<Exec Command="COPY %(FilesToMap.Identity) R:\StackHash\AppMap" />
<Exec Command="R:\3rdparty\Microsoft\appmap.exe 
 -s R:\StackHash\AppMap 
 -d R:\StackHash\WinQualMapFiles\$(VerText).xml 
 -n StackHash 
 -v $(VerText) 
 -e 
 -f" />
</Target>

The FilesToMap item group lists all of the product EXE and DLL files that should be included in the mapping. The first two commands in the target clear out a scratch folder and then fill it with these files. This is because our files are located in various folders so we need to pull them together for mapping. The third command invokes appmap.exe — note that $(VerText) is set to the full product version and that the line is broken up for clarity (it should be a single line of XML).

Every release build of StackHash now has a matching product mapping file.

You probably don’t want to upload the mapping files for each build, but it’s good to have the archive so you can quickly map a beta or release candidate if you want to start collecting crash reports. If you want to automate uploading the mapping file vote for adding this feature to StackHash, and check out this tool on Google Code.

[Blog post by Rob Ellison - taken from StackHash]

The format of a minidump (mdmp) file.

Tue, 20 Sep 2011 17:53:00 +0100

Introduction

This blog describes the basic structure and contents of a minidump file for those that want a brief overview without having to trawl the header files. A mdmp file is generated when an application crash (known as an exception) occurs in a Windows application. It contains varying amounts of information that can help the application developer determine the reason for the crash. Minidumps are normally analyzed using a debugger (windbg or Visual Studio) using symbol data that the programmer has access to.

A minidump created following an application crash is optionally sent to the Microsoft Windows Error Reporting (WinQual) site by the user where it is added to a database ready for the application developer to download and debug, either manually or with a productivity tool such as StackHash. A minidump can be generated manually for a live application using a debugger or by loading Task Manager, right clicking the application and selecting Create Dump File.

You can analyze the structure of a dump file using the dumpchk.exe utility that comes with Debugging Tools for Windows by simply opening an MSDOS prompt in the folder containing the minidump and typing:

dumpchk -v some.mdmp >log.txt
notepad log.txt

Much of the detailed structure of the minidump file can be found in the header files ImageHlp.h and dbghelp.h which are installed as part of the Windows SDK. A full description of all fields can be found in the DbgHelp API documentation. Other resources include the toolhelp API and the ApplicationVerifier help.

I’ll update this blog over time as I discover anything non-obvious about the stream data.

Minidump header file

The file starts with a header section (_MINIDUMP_HEADER) identifying the contents of the file. A dump file contains a number of Streams of data. Each Stream contains a particular type of information as follows.

ThreadListStream(3) - Identifies the threads that were alive in the application at the time of the crash.
ModuleListStream(4) - List of modules (DLLs and EXEs) that were loaded in the address space of the application.
MemoryListStream(5) - Dump of various parts of memory that existed at the time of the crash.
ExceptionStream(6) - Contains information about the exception that occurred in the application (if any).
SystemInfoStream(7) - Contains information about the machine on which the application was running.
ThreadExListStream(8) - Similar to the ThreadListStream but contains BackingStore information (Intel Itanium only).
Memory64ListStream(9) - Memory blocks from a 64 bit machine.
CommentStreamA(10) - ANSI string describing the dump.
CommentStreamW(11) - UNICODE string describing the dump.
HandleDataStream(12) - Contains additional information about system level handles for processes, threads and mutants.
FunctionTableStream(13) - Contatins a number of function tables each describing 1 or more functions.
UnloadedModuleListStream(14) - Includes a list of modules that were recently unloaded, if this information is maintained by the operating system.
MiscInfoStream(15) - Contains general system wide information such as process up time, processor information and timezone.
MemoryInfoListStream(16) - Contains memory region description information. It corresponds to the information that would be returned for the process from the VirtualQuery function.
ThreadInfoListStream(17) - Contains thread state information.
HandleOperationListStream(18) - Contains a list of operations performed on selected OS handles.

An example header and stream directory is shown below.

Minidump Header
===========
Signature: MDMP
Version: A793(5128)
Number of Streams: 9
Stream RVA: 20
Checksum: 0
TimeDateStamp/Reserved: 4B7CD4F6
Flags: 21
(00000001) MiniDumpWithDataSegs
(00000020) MiniDumpWithUnloadedModules

Stream Directory…
Stream 0: type ThreadListStream (3), size 00000394, RVA 00000184
Stream 1: type ModuleListStream (4), size 00002818, RVA 00000518
Stream 2: type UnloadedModuleListStream (14), size 000000CC, RVA 00002D30
Stream 3: type MemoryListStream (5), size 00000154, RVA 00009B22
Stream 4: type ExceptionStream (6), size 000000A8, RVA 000000DC
Stream 5: type SystemInfoStream (7), size 00000038, RVA 0000008C
Stream 6: type MiscInfoStream (15), size 00000018, RVA 000000C4
Stream 7: type UnusedStream (0), size 00000000, RVA 00000000
Stream 8: type UnusedStream (0), size 00000000, RVA 00000000

The following sections describe some of the main stream types in more detail.

ThreadListStream(3) and ThreadExListStream(8)

The ThreadList stream identifies the threads that were alive in the application at the time of the crash. A thread is the basic unit that is scheduled by the operating system (OS). An application may have 1 or more threads. The OS allocates processor time for each thread to run in turn depending on its priority. Each thread has:

ThreadId - Uniquely identifes the thread.
SuspendCount - Indicates whether the thread is suspended (a number greater than 0 indicates it is).
PriorityClass - Determines the system wide scheduling priority class of the thread (IDLE, BELOW_NORMAL, NORMAL, ABOVE_NORMAL, HIGH, REALTIME) and is specified as a thread behaviour requirement when the thread is created.
Priority - Within a process, threads can be given different priorities. e.g. a user input thread might be given a higher priority than a computationally intensive thread so that the latter doesn’t make user input unresponsive for long periods. (IDLE, LOWEST, BELOW_NORMAL, NORMAL, ABOVE_NORMAL, HIGHEST, TIME_CRITICAL).
Thread Environment Block (TEB) - Contains per-thread context information used by the Windows components running in user mode (e.g. Windows subsystem and the Application Image Loader). This field contains the address of the TEB (as opposed to its contents). A typical value is 0x7FFDF000.
Stack - Start address on the stack as well as the location of the stack information in the file.
ThreadContext - Locates the architecture specific thread context data including the thread’s volatile register contents, stacks and private storage area.
BackingStore - Only valid for ThreadExListStream. Contains backing store information for the thread on Itanium machines.

The following is an example of the first 2 threads of a thread stream.

StreamType: ThreadListStream
Location: Length: 394, Rva: 184
NumberOfThreads: 19
RVA: 0188 ThreadId:000003C4 SuspendCount:1 PriorityClass:0 Priority:0 TEB:000000007FFDF000 Stack: (Address: 12F2A8 Length: D58, Rva: E776) ThreadContext: (Length: 2CC, Rva: 5708)
RVA: 01B8 ThreadId:000004EC SuspendCount:1 PriorityClass:0 Priority:0 TEB:000000007FFDE000 Stack: (Address: ABFE38 Length: 1C8, Rva: F4CE) ThreadContext: (Length: 2CC, Rva: 59D4)

ThreadInfoListStream(17)

The ThreadInfoList stream contains dynamic information about threads including information about how long the thread has been running and how much time the thread spent in user and kernel mode.

ThreadId - Uniquely identifies the thread.
DumpFlags - Indicates if the thread had exited at the time of the dump. Also defines the progress during the attempt to dump the thread information.
DumpError - Status of the dump.
CreateTime - The time when the thread was created, in 100-nanosecond intervals since January 1, 1601 (UTC).
ExitTime - The time when the thread exited, in 100-nanosecond intervals since January 1, 1601 (UTC).
KernelTime - The time executed in kernel mode, in 100-nanosecond intervals.
UserTime - The time executed in user mode, in 100-nanosecond intervals.
StartAddress - The starting address of the thread.
Affinity - The processor affinity mask. This is a bit mask indicating the processors that the thread is allowed to run on. This will normally be the same as the owning process and is usually allocated by the OS.

HandleOperationListStream(18)

Some operating systems can track the last operations performed on a handle. Application Verifier can enable this for some versions of Windows. Application Verifier is a tool that can be downloaded from Microsoft to detect unusual behaviour in native Win32 applications. The recorded information for each handle includes the ProcessId and ThreadId of the thread accessing the handle; the OperationType (Open, Close or BadRef); and the back trace identifying the code addresses that made the access. An example of an open and close operation on a handle is shown below.

SizeOfHeader: 10
SizeOfEntry: 120
NumberOfEntries: 1000
Reserved: 0

HeapHandle: B24
ProcessId: 1A5C
ThreadId: 1B94
OperationType: 2-OperationDbCLOSE
Spare0: 0
BackTraceInfo: Depth: 10
Index: 0
ReturnAddresses: FFFFF80002FEB254 FFFFF80002CD6993 7715F7AA 7FEF57D6237 7FEF57D6368 7FEF57DB26C 772086C8 7716ED9A 7714B80D 771444CE 7FEEF1F7048 7FEFD12A4CC 7FEF3325E91 7FEF3326035 7FEF3323596 7FEF706A317 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

HeapHandle: B24
ProcessId: 1A5C
ThreadId: 1B94
OperationType: 1-OperationDbOPEN
Spare0: 0
BackTraceInfo: Depth: 10
Index: 0
ReturnAddresses: FFFFF80002FDA617 FFFFF80002FE4360 FFFFF80002CD6993 7715FC0A 7FEF57D60DF 7FEF57D6368 7FEF57DB26C 772086C8 7716ED9A 7714B80D 771444CE 7FEEF1F7048 7FEFD12A4CC 7FEF3325E91 7FEF3326035 7FEF3323596 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ModuleListStream(4)

The ModuleList stream contains a list of modules (DLLs and EXEs) that were loaded in the address space of the application. The presence of unrecognised modules, such as viruses or virus checkers that have been injected into the application address space may indicate a problem. The absence of some DLLs may help identify how far the application got in its processing before it failed if delayed loading of DLLs was enabled. The list of modules is used when identifying the instruction addresses contained in other streams. An address lying within the range ImageBase to (ImageBase + ImageSize) means the instruction refers to code in that module. For each module the following information is listed.

ImageBase - Base address in memory of the image.
ImageSize - Size of the image.
Name - Full path and filename of the DLL or EXE file.
VersionInfo - Full version information for the file including: FileVersion; ProductVersion; OS the file was designed for (e.g. Win32); FileType - e.g. DLL, EXE.
CheckSum - Hash of bytes in the file.
TimeStamp - Date and Time of file creation.
CodeViewRecord - This record may contain a symbol filename (PDB or DBG) and GUID that a debugger can use to locate the correct symbol file matching the binary file. The CodeView data is part of the original DLL or EXE file and is extracted from the Portable Executable - (PE = COFF) format.
MiscRecord - This contains miscellaneous symbol information.

An example of module data extracted from the module stream is shown below.

Base: 7E410000 Size: 91000 Rva: 3098 C:\WINDOWS\system32\user32.dll CheckSum: 8FC76 TimeDateStamp: 4802A11B
FileVersionInfo:
Signature: FEEF04BD
StrucVersion: 1.0
FileVersion: 5.1.2600.5512
ProductVersion: 5.1.2600.5512
FileFlagsMask: 3F
FileFlags: 00
FileOS: 40004 VOS_NT_WINDOWS32
FileType: 2 VFT_DLL
FileSubtype: 0
FileDateMS: 0
FileDateLS: 0

CodeView record: Length: 23, Rva: 8D91
52 53 44 53 B7 41 8A D1 7F 4E 8C 45 AA AC 18 47 RSDS?A???N?E???G
E2 D8 BF 02 02 00 00 00 75 73 65 72 33 32 2E 70 ?????????user32.p
64 62 00 db

CodeView Signature: RSDS
CodeView GUID: B7 41 8A D1 7F 4E 8C 45 AA AC 18 47 E2 D8 BF 02
CodeView Age: 2
CodeView PDB FileName: user32.pdb

Misc record: Length: 0, Rva: 0

UnloadedModuleListStream(14)

Includes a list of modules that were recently unloaded, if this information is maintained by the operating system. For Windows Server 2003 and Windows XP: The operating system does not maintain information for unloaded modules until Windows Server 2003 with SP1 and Windows XP with SP2. An example of the data stored in the UnloadedModuleList is shown below.

StreamType: UnloadedModuleListStream
Location: Length: CC, Rva: 2D30
NumberOfModules: 8
Base: 78080000-78091000 Size: 11000 NameRva: 5354 MSVCRT40.dll CheckSum: F7C0 TimeDateStamp: 48025155
Base: 747B0000-747F7000 Size: 47000 NameRva: 5372 msnsspc.dll CheckSum: 4D6A9 TimeDateStamp: 4802A16A
…

MemoryListStream(5) and Memory64ListStream(9)

The memory stream contains blocks of memory whose contents might be useful for debugging a crash. They include portions of the stack.

21 memory ranges
range# RVA Address Size
0 00009C76 7c90e494 00000100
1 00009D76 7c97e000 00004a00
2 0000E776 0012f2a8 00000d58
3 0000F4CE 00abfe38 000001c8
4 0000F696 00c8fcd0 00000330
…

StreamType: MemoryListStream
Location: Length: 154, Rva: 9B22
NumberOfMemoryRanges: 21
Address: 7C90E494 Length: 100, Rva: 9C76
C6 EB FF FF EB 0B 5B 59 6A 00 51 53 E8 09 F5 FF ??????[Yj?QS????
FF 83 C4 EC 89 04 24 C7 44 24 04 01 00 00 00 89 ??????$?D$?????
5C 24 08 C7 44 24 10 00 00 00 00 54 E8 63 00 00 \$?D???T?c
00 C2 08 00 55 8B EC 83 EC 50 89 44 24 0C 64 A1 ???U????P?D$?d?

Address: 7C97E000 Length: 4A00, Rva: 9D76
63 6F 73 00 00 00 00 00 00 00 00 00 00 00 00 00 cos????????????
6C 6F 67 00 00 00 00 00 00 00 00 00 00 00 E0 3F log????????????
70 6F 77 00 00 00 00 00 00 00 00 00 00 00 00 00 pow???????????
73 69 6E 00 00 00 00 00 00 00 00 00 00 00 00 00 sin????????????

MemoryInfoListStream(16)

This stream contains memory region description information. It corresponds to the information that would be returned for the process from the VirtualQuery function. VirtualQuery retrieves information about a range of pages in the virtual address space of the calling process.

SizeOfHeader: 10
SizeOfEntry: 30
NumberOfEntries: 8C67

BaseAddress: 1981000 AllocationBase: 1980000 AllocationProtect: 1(PAGE_NOACCESS) RegionSize: 69000 State: 1000(MEM_COMMIT) Protect: 4(PAGE_READWRITE) Type: 20000(MEM_PRIVATE)
BaseAddress: 10000 AllocationBase: 10000 AllocationProtect: 4(PAGE_READWRITE) RegionSize: 2000 State: 1000(MEM_COMMIT) Protect: 4(PAGE_READWRITE) Type: 20000(MEM_PRIVATE)
BaseAddress: 12000 AllocationBase: 0 AllocationProtect: 0(0) RegionSize: E000 State: 10000(MEM_FREE) Protect: 1(PAGE_NOACCESS) Type: 0(0)
BaseAddress: 40000 AllocationBase: 40000 AllocationProtect: 2(PAGE_READONLY) RegionSize: 2000 State: 1000(MEM_COMMIT) Protect: 2(PAGE_READONLY) Type: 40000(MEM_MAPPED)
BaseAddress: 19EA000 AllocationBase: 1980000 AllocationProtect: 1(PAGE_NOACCESS) RegionSize: 1000 State: 2000(MEM_RESERVE) Protect: 0(0) Type: 20000(MEM_PRIVATE)
…

SystemInfoStream(7)

The SystemInfo stream contains details of the hardware and operating system that the application was running on.

StreamType: SystemInfoStream
Location: Length: 38, Rva: 8C
ProcessorArchitecture: 0 - PROCESSOR_ARCHITECTURE_INTEL
ProcessorLevel: 15
ProcessorRevision: 20226
NumberOfProcessors: 1
ProductType: 1 - VER_NT_WORKSTATION
Version: 5.1.2600
PlatformId: VER_PLATFORM_WIN32_NT
ServicePack: Service Pack 3
SuiteMask: 300 - VER_SUITE_SINGLEUSERTS, VER_SUITE_PERSONAL

ExceptionStream(6)

The exception stream contains information about any exception that may have occurred in the application. An exception is a condition that gave rise to an application crash. Inner exceptions may also be present by following the ExceptionRecord field. The ExceptionAddress probably points into one of the modules in the ModuleStream so the exact position of the crash inside the binary file should be determinable.

StreamType: ExceptionStream
Location: Length: A8, Rva: DC
ThreadId: 3C4
ExceptionCode: CFFFFFFF-Application Hang
ExceptionFlags: 0-EXCEPTION_CONTINUABLE
ExceptionAddress: 7C90E514
ExceptionRecord (InnerException): 0
NumberOfExceptionParams: 0

Location: Length: 0, Rva: 0

Common ExceptionCode values include

EXCEPTION_ACCESS_VIOLATION = 0x0C0000005,
EXCEPTION_ARRAY_BOUNDS_EXCEEDED = 0x0C000008C,
EXCEPTION_BAD_COMPRESSION_BUFFER = 0x0C0000242,
EXCEPTION_BREAKPOINT = 0x080000003,
EXCEPTION_CALLBACK_POP_STACK = 0x0C0000423,
EXCEPTION_DATATYPE_MISALIGNMENT = 0x080000002,
EXCEPTION_FLOAT_DENORMAL_OPERAND = 0x0C000008D,
EXCEPTION_FLOAT_DIVIDE_BY_ZERO = 0x0C000008E,
EXCEPTION_FLOAT_INEXACT_RESULT = 0x0C000008F,
EXCEPTION_FLOAT_INVALID_OPERATION = 0x0C0000090,
EXCEPTION_FLOAT_OVERFLOW = 0x0C0000091,
EXCEPTION_FLOAT_STACK_CHECK = 0x0C0000092,
EXCEPTION_FLOAT_UNDERFLOW = 0x0C0000093,
EXCEPTION_FLOAT_MULTIPLE_FAULTS = 0x0C00002B4,
EXCEPTION_FLOAT_MULTIPLE_TRAPS = 0x0C00002B5,
EXCEPTION_GUARD_PAGE_VIOLATION = 0x080000001,
EXCEPTION_ILLEGAL_FLOAT_CONTEXT = 0x0C000014A,
EXCEPTION_ILLEGAL_INSTRUCTION = 0x0C000001D,
EXCEPTION_INSTRUCTION_MISALIGNMENT = 0x0C00000AA,
EXCEPTION_INVALID_HANDLE = 0x0C0000008,
EXCEPTION_INVALID_LOCK_SEQENCE = 0x0C000001E,
EXCEPTION_INVALID_OWNER = 0x0C000005A,
EXCEPTION_INVALID_PARAMETER = 0x0C000000D,
EXCEPTION_INVALID_PARAMETER_1 = 0x0C00000EF,
EXCEPTION_INVALID_SYSTEM_SERVICE = 0x0C000001C,
EXCEPTION_INVALID_THREAD = 0x0C000071C,
EXCEPTION_INTEGER_DIVIDE_BY_ZERO = 0x0C0000094,
EXCEPTION_INTEGER_OVERFLOW = 0x0C0000095,
EXCEPTION_IN_PAGE_ERROR = 0x0C0000006,
EXCEPTION_KERNEL_APC = 0x00100,
EXCEPTION_LONGJUMP = 0x080000026,
EXCEPTION_NO_CALLBACK_ACTIVE = 0x0C0000258,
EXCEPTION_NO_EVENT_PAIR = 0x0C000014E,
EXCEPTION_PRIVILEGED_INSTRUCTION = 0x0C0000096,
EXCEPTION_SINGLE_STEP = 0x080000004,
EXCEPTION_STACK_BUFFER_OVERRUN = 0xC0000409,
EXCEPTION_STACK_OVERFLOW = 0xC00000FD,
EXCEPTION_SUCCESS = 0x00000,
EXCEPTION_THREAD_IS_TERMINATING = 0x0C000004B,
EXCEPTION_TIMEOUT = 0x00102,
EXCEPTION_UNWIND = 0x0C0000027,
EXCEPTION_UNWIND_CONSOLIDATE = 0x080000029,
EXCEPTION_USER_APC = 0x000C0,
EXCEPTION_WAKE_SYSTEM_DEBUGGER = 0x080000007,
EXCEPTION_APPLICATION_HANG = 0xcfffffff,

CommentStreamA(10) and CommentStreamW(11)

These stream contains a string used for documentation purposes. For CommentStreamA, this is an ANSI string and for CommentStreamW this is a Unicode (double byte) string.

HandleDataStream(12)

A handle is a pointer to a data structure containing information about a consumed system resource such as a process, thread, registry key, event and mutant. The Handle stream, if present, contains data associated with these high level system handles. The HandleId, TypeName, ObjectName, Attributes, GrantedAccess, HandleCount and PointerCount are stored for each handle. Most of these fields are specific to the type of handle that is being stored.

Extended object information may also exist for a handle. The extended information may be present for threads, mutants and process handles. This data varies depending on the architecture of the machine. More than 1 extended block may exist for each handle as shown in the example below.

StreamType: HandleDataStream
Location: Length: 7B8, Rva: B44AAC
SizeOfHeader: 10
SizeOfDescriptor: 28
NumberOfDescriptors: 31
Reserved: 0

Handle: 4
TypeNameRva: B44000
TypeName: Key
ObjectNameRva: B4400C
ObjectName: \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
Attributes: 0
GrantedAccess: 9
HandleCount: 2
PointerCount: 3

Handle: 40
TypeNameRva: B443EE
TypeName: Mutant
ObjectNameRva: 0
ObjectName:
Attributes: 0
GrantedAccess: 1F0001
HandleCount: 2
PointerCount: 3
Next: B44400 InfoType: 3-MiniMutantInformation2 SizeOfInfo: 8
00 00 00 00 00 00 00 00 ????????

Next: 0 InfoType: 2-MiniMutantInformation1 SizeOfInfo: 8
01 00 00 00 00 00 47 00 G???????

FunctionTableStream(13)

The function table stream contains a number of function tables, one for each module. Each function table contains one or more function entry.

MiscInfoStream(15)

The MiscInfo contains miscellaneous information pertaining mainly to the process and processor. The amount of information stored has changed over time so different OS’s may contain different information. Information includes…

ProcessId - ID of the crashing process.
ProcessCreationTime - Time the process was started.
ProcessUserTime - Time the process (all of its threads) has spent running user mode code.
ProcessKernelTime - Time the process (all of its threads) has spent running kernel mode code.
ProcessorMaxMhz - The maximum specified clock frequency of the system processor.
ProcessorCurrentMhz - The processor clock frequency, in MHz. This number is the maximum specified processor clock frequency multiplied by the current processor throttle.
ProcessorMhzLimit - The limit on the processor clock frequency, in MHz. This number is the maximum specified processor clock frequency multiplied by the current processor thermal throttle limit.
ProcessorMaxIdleState - The maximum idle state of the processor.
ProcessorCurrentIdleState - The current idle state of the processor.
ProcessIntegrityLevel - (low,medium,high,system). Lower integrity programs have more limited access to the file system and other system resources.
ProcessExecuteFlags -
ProtectedProcess - The process is to be run as a protected process. The system restricts access to protected processes and the threads of protected processes.
TimeZoneId and TimeZoneInformationTimeZone - under which the machine is running.

An example Misc information sections follows…

StreamType: MiscInfoStream
Location: Length: 18, Rva: C4
SizeOfInfo: 18
ValidFieldFlags: 3-MINIDUMP_MISC1_PROCESS_ID, MINIDUMP_MISC1_PROCESS_TIMES
ProcessId: 394
ProcessCreateTime: 4B7CD46A
ProcessUserTime: 0
ProcessKernelTime: 1

Setting up a Source Server

Tue, 20 Sep 2011 14:18:00 +0100

Introduction

In order to effectively debug dump files from end users, it is necessary to have access to product symbols. Product symbols are generated during the product build and are normally contained in PDB (or DBG) files. When using a debugger to analyze a dump file, a PDB matching the exact version of the binary file (DLL or EXE) is required.

The PDB must contain additional information in order to access the correct version of source files for source level debugging. This extra source code version information is contained in a data Stream called “srcsrv” within the PDB and must be added as an additional step after the main build is complete. This additional step is called Source Indexing.

A full description of source indexing can be found in the srcsrv.doc in the srcsrv subfolder of Debugging Tools for Windows. This blog summarises the main steps required to set up source debugging.

Configuring the Build Environment

The following components must be installed on the build machines.

Debugging Tools for Windows. Install the latest version that is available (http://msdn.microsoft.com/en-us/windows/hardware/gg463009). This contains the actual source indexing tools in a \srcsrv subfolder of the installation folder.
Perl 5.6 or above. Some of the source indexing tools are written in Perl and thus require a version of Perl to be installed. See http://www.perl.org.
Set the PATH environment variable to include the \srcsrv subfolder of the Debugging Tools for Windows installation folder e.g. C:\Program Files\Debugging Tools for Windows (x64)\srcsrv.
Set the PATH environment variable to include the Perl installation folder (if not already done by the installer).
Copy the file srcsrv.ini from the \srcsrv subfolder to a common location (company network share or folder containing the project build files) where it can be accessed by the build scripts and edit it to include a line in the [variables] section for your source control system. e.g.

[variables]







MYSERVER=sourcecontrolmachine:1666.

Build Changes

The following diagram shows the main components required during a build.

Running msbuild on an automated build file (e.g. mybuild.proj) will at some stage invoke the linker to produce the product binary and symbol files (e.g. myapp.exe and myapp.pdb) in one or more output folders, usually based on build configuration. e.g. c:\myproduct\debug and c:\myproduct\release.

A new target needs to be added to mybuild.proj to call the Source Indexing utility (ssindex.cmd) with parameters particular to your source control system (e.g. Perforce, SourceSafe etc…). e.g. for Perforce the new target might look something like this.


<Target Name="SourceIndexing” DependsOnTargets="ProductBuild">
<Message Text="Source indexing pdb files"/> 
<Exec Command = "ssindex.cmd /ini=r:\srcsrv.ini /source=r:\ /symbols=
 r:\ProductOut\$(Configuration) /debug /system=p4 /dieonerror"/> 
</Target>

The target must depend on the main product build so it doesn’t get executed until the main build is complete.
If the product contains a number of output folders then the ssindex.cmd command can be called multiple times.
The /ini option points to the network share containing the srcsrv.ini file containing the master list of source control systems and their location.
The /symbols option should point to the folder containing the build symbol files (PDB).
The /system option specifies the source control system in use. In this case P4 for Perforce.
The /source option should point to the top of the source code tree on the local machine.
Type ssindex.cmd –?? for help with other options.
The /Debug switch should always be used as ssindex.cmd can otherwise fail to index PDBs without providing any error or warning messages (for example if your Perforce workspace includes forward slashes indexing will fail and with debug output enabled you’ll see a “zero source files found” message).
Source code indexing varies by source control provider and so you will need to review the documentation and the command line help from ssindex.cmd to tune your settings.
By default, the source indexing tool adds an entry to the PDB for each source file. This entry contains the version number of the file used to build the component. This is taken from the source control system as the latest version of that file that has been checked in. The PDB can be marked with a LABEL rather than a specific file version by using the /label switch. Note that the label must exist for this to work. See the section below on labelling for more information.
Important Note: Source Indexing with Source Safe only works with labels, so you must label the source before source indexing. See labelling section below.

Ssindex.cmd produces output as follows…

ssindex.cmd [STATUS] : Server ini file: r:\srcsrv.ini 



ssindex.cmd [STATUS] : Source root    : r:\ 



ssindex.cmd [STATUS] : Symbols root   : r:\ProductOut\Debug 



ssindex.cmd [STATUS] : Control system : P4 



ssindex.cmd [STATUS] : P4 program name: p4.exe 



ssindex.cmd [STATUS] : P4 Label       : <N/A> 



ssindex.cmd [STATUS] : Old path root  : <N/A> 



ssindex.cmd [STATUS] : New path root  : <N/A> 



ssindex.cmd [STATUS] : Partial match  : Not enabled 



-------------------------------------------------------------------------------- 



ssindex.cmd [STATUS] : Running... this will take some time...   



ssindex.cmd [INFO  ] : ... indexing r:\ProductOut\Debug\MyApp.pdb 



ssindex.cmd [INFO  ] : ... wrote C:\Users\Me\AppData\Local\Temp\



index16A62.stream to r:\ProductOut\Debug\MyApp.pdb ...

SrcTool for testing.

You can use srctool.exe (from Debugging Tools for Windows) to test that the PDBs are in fact correctly set up with source file version information.

Type srctool –d:c:\src <pdbfilename> to see a list of source files and their version numbers. A line similar to the one that follows will be displayed for each indexed source file. The output is in 2 parts: the original filename (r:\MyApp\main.cs); and the command that is to be run on the source control system to get the specified version (in this case Perforce). The –o switch specifies the output path where the source file is to be copied. You can override this on the command line (as I have) with the –d option. The –q switch requests a particular version of the file from source control. Note that this is the command for Perforce and that for other source control systems the command will be different, but essentially it will contain an output location and required version.

[r:\MyApp\main.cs] cmd: p4.exe -p myserver:1666 print -o “c:\src\P4SERVER\depot\MyApp\main.cs\2\main.cs” -q “//depot/MyApp/main.cs#2”

To test the source indexing is correct you can extract the files with the –x option. e.g. srctool –x –d:c:\src <pdbfilename> will extract the source code to subfolders of c:\src. This will produce output like this (note that c:\src must exist) where version 4 of main.cs has been copied from source control.

c:\src\P4SERVER\depot\MyApp\main.cs\4\main.cs







MyApp.pdb: 1 source files were extracted

Using Labels

The ssindex.cmd defaults to updating the PDB with the latest version of the file that has been checked in to source control. i.e. if the PDB contains a filename r:\MyApp\main.cs as a listed source file, then //depot/MyApp/main.cs#2 might be added to the PDB indicating that version 2 of the file was used to build the product.

However, source control systems often allow labelling of the source tree. If the build process labels the source tree then you can use the /label switch to add the label to the PDB for that file instead of the file version number. Note that the label must exist. In this case SrcTool outputs something like…

[r:\MyProduct\main.cs] cmd: p4.exe -p myserver:1666 print -o “c:\src\P4SERVER\depot\MyProduct\main.cs\MyLabel1.2.3.4\AtomCab.cs” -q //depot/MyProduct/main.cs@MyLabel1.2.3.4]

If the same build file is used on developer machines and the release build machine, the project files should be set up to use version numbers on developer machines and labelling on the main build machine. This assumes that the labelling of source code only takes place during a build of a release candidate.

Important Note: Source Indexing with Source Safe only works with labels, so you must label the source prior to source indexing if you are using Source Safe.

Source Level Debugging

When performing debugging on a dump file using Visual Studio, Windbg or cdb, the debugger will locate the correct binary and pdb files from your local symbol server or network share based on the symbol path and binary paths that have been configured.

To tell the debugger to retrieve the correct source files, add “SRV*c:\source” to the source path (.srcpath or environment variable _NT_SOURCE_PATH).

For Visual Studio, click Tools -> Options -> Debugging -> General and tick “Enable Source Server Support”.

The debugger will then call into the source server DLL (srcsrv.dll) to extract the correct source files from your source control system to a local cache folder (c:\source) based on the source index information contained in the PDB. i.e. the PDB will be downloaded first; the PDB srcsrv stream data is then extracted and passed to srcsrv.dll by the debugger; and then srcsrv.dll is called again for each PDB to get the correct source files from the appropriate source control system (remember the PDB contains the command required to extract the appropriate version from source control).

For automated processing of WinQual errors see the StackHash Open Source project.

Debugging a managed dump file with sos and psscor.

Tue, 20 Sep 2011 13:50:00 +0100

Introduction

The debuggers that come for free with Debugging Tools for Windows are native code debuggers. Out of the box, they don’t understand managed code or the Common Language Runtime (CLR) internal data structures.

In the early days of .NET, Microsoft developed a debugger extension DLL they called Strike to aid debugging the .NET framework and CLR. This DLL was originally intended for internal use within Microsoft. However, they soon realized that it would be invaluable as a debugging aid for the wider .NET programming community, so they changed its name to Son Of Strike (SOS.DLL) and released it as part of the CLR binary distribution.

SOS.DLL is closely tied to another DLL (mscordacwks.dll) that provides the data access (DAC) layer to the CLR. Mscordacwks.dll abstracts the internal workings of the CLR and its data structures into a more consistent interface.

In the following sections it is assumed that you have the debugger symbol and binary paths set up to include the Microsoft public symbol server.
e.g. SRV*c:\localstore*http://msdl.microsoft.com/download/symbols

.NET and CLR Versions

There is some confusion surrounding the difference between the CLR and .NET Framework versions. The CLR is the basic core engine that provides the virtual environment in which .NET framework applications are compiled and executed. The .NET framework is a collection of higher level object libraries that are used to simplify the programming task: e.g. LINQ, WCF, WPF, etc…

A new release of the .NET framework adds new layers of objects but is not always accompanied by a new release of the CLR, although a patch of the existing CLR version may be necessary to support a new framework release. The following table identifies the main .NET releases to date and the corresponding version of the CLR that they are built upon.

Framework — Date — CLR Version — CLR DLL

1.0 2002-02-13 1.0 mscorwks.dll

1.1 2003-04-24 1.0 mscorwks.dll

2.0 2005-11-07 2.0 mscorwks.dll

3.0 2006-11-06 2.0 mscorwks.dll

3.5 2007-11-19 2.0 mscorwks.dll

4.0 2010-04-12 4.0 clr.dll

As you can see, Framework versions 2.0, 3.0 and 3.5 all utilize the 2.0 CLR contained primarily in mscorwks.dll. Framework version 4.0 uses the version 4.0 CLR contained principally in clr.dll.

You can therefore identify the version(s) of the CLR loaded in a dump file using the list modules command lm.
e.g.

lm v m mscorwks
start end module name
000007fe`f0170000 000007fe`f0b1e000 mscorwks (deferred)
Image path: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
Image name: mscorwks.dll
Timestamp: Sat Feb 05 02:55:00 2011 (4D4CBC04)
CheckSum: 009A0D7A
ImageSize: 009AE000
File version: 2.0.50727.4959
Product version: 2.0.50727.4959
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: mscorwks.dll
OriginalFilename: mscorwks.dll
ProductVersion: 2.0.50727.4959
FileVersion: 2.0.50727.4959 (win7RTMGDR.050727-4900)
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail

or for CLR version 4.0

lm v m clr
start end module name
000007fe`e8760000 000007fe`e90c6000 clr (deferred)
Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Image name: clr.dll
Timestamp: Thu Feb 10 04:21:10 2011 (4D5367B6)
CheckSum: 0095F083
ImageSize: 00966000
File version: 4.0.30319.225
Product version: 4.0.30319.225
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: clr.dll
OriginalFilename: clr.dll
ProductVersion: 4.0.30319.225
FileVersion: 4.0.30319.225 (RTMGDR.030319-2200)
PrivateBuild: DDBLD252
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail

Loading SOS.DLL

We have seen in the previous section that the CLR will already be listed as a loaded module in a managed dump. To perform managed debugging, SOS.DLL must now be loaded manually using the .load command. However, SOS.dll is tightly bound to a particular version of mscordacwks.dll and mscorwk.dll. This means that the version of SOS.DLL matching the exact version of MSCORWKS.DLL (or CLR.DLL for version 4.0), that is already loaded in the dump file, is required.

If the source of the dump file can be contacted, then the required .NET CLR DLLs (SOS.DLL and MSCORDACWKS.DLL), can be copied from that machine. If not, then you are pretty much out of luck unless you happen to have a machine with the same version of .NET installed already. Alternatively, you can build up a library of .NET versions for debugging by installing the earliest version of .NET you are interested in and then applying patch after patch from the Microsoft download site.

When you attempt to load a version of SOS.DLL into the debugger by using the .load c:\dotnetversions...\sos.dll command, the debugger will load sos.dll which will then attempt to find the correct version of mscordacwks.dll and consequently mscorwks.dll. If this does not match the version of mscorwks.dll already loaded in the dump file, then the debugger will output an error such as that shown below.

Failed to load data access DLL, 0x80004005
Verify that

you have a recent build of the debugger (6.2.14 or newer)
the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

This tight link between SOS and the CLR makes debugging dump files using SOS quite frustrating… step in psscor.

Using Psscor

Psscor contains a superset of commands from sos.dll. However, its biggest benefit is that the matching up of precise versions with the CLR, as for SOS.DLL, is no longer necessary. Psscor comes in 2 versions matching the CLR MAJOR version number.

psscor2.dll - used to debug all CLR 2.0 versions (i.e. framework 2, 3 and 3.5) for x86, AMD64 and IA64.
psscor4.dll - used to debug all CLR 4.0 versions (i.e. framework 4.0) for x86 and AMD64.

AMD64 is used for Intel x64 debugging.

As stated above, the version of the CLR loaded in the dump file can be determined using the lm command. To determine the architecture of the target machine you can use the vertarget command. The example below shows a x64 dump.

vertarget
Windows 7 Version 7600 MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.1.7600.16385 (win7_rtm.090713-1255)

Now, all that is necessary is to load either psscor2.dll or psscor4.dll from the appropriate architecture install folder e.g.

.load c:\psscor\psscor4\x64\psscor4.dll

and then managed debugging can start.

StackHash and Psscor

StackHash is an open source project that allows you to create debugger scripts that are run automatically (or not) on dumps downloaded from WinQual. As of version 1.20, StackHash now automatically locates the correct version of psscor*.dll based on the architecture and CLR version of the dump. Psscor2 and psscor4 are both installed in the StackHash install folder.

To load psscor.dll to do managed debugging, the StackHash service must be running on a 64 bit machine with both x86 and x64 bit versions of Debugging Tools for Windows installed and configured in the service profile settings.

Within a debugger script you can just use the following command (note the absence of a 2 or 4 in the name of the DLL).

.load psscor.dll

and StackHash will take care of loading the correct variant of psscor. The debugger script output will identify the version of psscor loaded. e.g.

.load C:\Program Files\StackHash\psscor4\amd64\psscor4.dll

References

sos - http://msdn.microsoft.com/en-us/library/bb190764.aspx
sos for dumps - http://msdn.microsoft.com/en-us/library/yy6d2sxs.aspx
psscor2 - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=5c068e9f-ebfe-48a5-8b2f-0ad6ab454ad4
psscor4 - http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a06a0fea-a4d4-434e-a527-d6afa2e552dd

How does cdb access the Microsoft Symbol Server from within a Windows Service.

Tue, 20 Sep 2011 11:55:00 +0100

This blog addresses some of the issues related to running cdb from within a Windows Service.

What is cdb?

Cdb is the user mode command line debugger that comes as part of the Debugging Tools for Windows toolset which can be downloaded from Microsoft for free. Also contained in the Debugging Tools for Windows are the other kernel level debugger (kd) and windbg, which can be used to debug user applications and the Windows kernel. This blog refers to cdb.

Symbol Files.

Windows applications consist of executable files (.exe and .dll). Symbol files are built by the developer at the same time as the executable files and normally have the same name with a .pdb (program database) or .dbg (debug) extension. e.g. somefile.pdb might be the symbol file for somefile.dll. Symbol files contain additional information useful for debugging applications and it is imperative that developers create and store them for released versions of their products.

Normally a developer would set up a symbol store to contain product specific symbol files which can then be accessed through a symbol server. For more information see http://msdn.microsoft.com/en-us/library/ff558840(v=VS.85).aspx. This blog deals with the public Microsoft symbol server.

Symbol Path.

In order for cdb to display symbol information for stack traces etc… it needs to be told the path from where to retrieve the symbol files.

Symbols can be specified to the cdb command by setting up the _NT_SYMBOL_PATH environment variable, e.g.

SET _NT_SYMBOL_PATH = SRV*c:\localcache*http://msdl.microsoft.com/download/symbols

or can be specified as a command line argument with the -z switch

cdb -z c:\test\dump.mdmp -y “SRV*c:\localcache*http://msdl.microsoft.com/download/symbols”

The symbol path can contain multiple local folders separated by a semicolon. In the example above it represents a path to a remote symbol server thus:

SRV* is short for symsrv*symsrv.dll and says use the DLL symsrv.dll (supplied in Debugging Tools for Windows) to access the Microsoft symbol server. Developers could create their own symbol server interface for connecting in a bespoke way to their own symbol server.

The path following SRV*, i.e. c:\localcache, is the location where all downloaded symbols are stored locally and http://msdl.microsoft.com/download/symbols is the web address that symsrv.dll uses to access the public Microsoft symbol server.

In summary, symsrv.dll downloads symbols (e.g. kernel32.dll) from http://msdl.microsoft.com/download/symbols and stores them in c:\localcache.

Compressed symbol file.

The Microsoft symbol server stores many symbol file in compressed format to save space and download time. Symsrv.dll knows how to access and decompress these files to the local cache.

Compressed versions of files are stored with the last extension character replaced with an underscore. e.g.

kernel32.dll (not compressed)
kernel32.dl_ (compressed version)

Symbol Server Requests.

The following is an example URL request to the symbol server for the compressed version of the file kernel32.dll. The 49E037DDdc000 is taken from various version and timestamp information that uniquely identifies the version of the file required.

http://msdl.microsoft.com/download/symbols/kernel32.dll/49E037DDdc000/kernel32.dl_

When downloaded and decompressed, the resultant kernel32.dll file will be stored as:
c:\localcache\kernel32\49E037DDdc000\kernel32.dll

WinINet versus WinHTTP.

When running from a Command Prompt, cdb uses WinINet to access internet resources.
When running from a Windows service, cdb uses WinHTTP to access internet resources.

Microsoft says… “Microsoft Windows HTTP Services (WinHTTP) is targeted at middle-tier and back-end server applications that require access to an HTTP client stack. Microsoft Windows Internet (WinINet) provides an HTTP client stack for client applications, as well as access to the File Transfer Protocol (FTP), SOCKSv4, and Gopher protocols.”

If cdb is running within the Windows service environment, as it would be when lauched from StackHash, then it should (and does) use WinHttp.

Forcing cdb to use WinHTTP.

You can force cdb to use WinHTTP from a Command Prompt by setting the following environment variable.

SET DBGHELP_WINHTTP=AnythingOtherThanEmpty.

You can turn this off again with

SET DBGHELP_WINHTTP=

Note you will have to retype these commands if you open a new command prompt.
Setting cdb to operate in this way is useful if you are having problems accessing symbols when launching cdb from within a Windows Service environment.

Cdb with WinHTTP and Proxy Servers.

When running cdb using WinHttp, as described in the previous section, you will still be unable to access symbols on the Microsoft symbol server.
Enabling HTTP trace in netsh winhttp (see appendix to this blog) uncovers the problem.

09:40:06.299 ::»» WinHttp Version 6.0 Build 6.1.7600 »»Process cdb.exe [11000 (0x2af8)] started at 09:40:06.299 03/09/2011
09:40:06.299 ::WinHttpOpen(“Microsoft-Symbol-Server/6.12.0002.633”, WINHTTP_ACCESS_TYPE_NAMED_PROXY (3), “SymSrvBogusProxy”, “<local>”, 0x0)

As the trace exposes, WinHttp is trying to use a proxy SymSrvBogusProxy to access the internet.

Microsoft have documented a related issue with SymProxy http://msdn.microsoft.com/en-us/library/ff539229(VS.85).aspx.

“The default behavior of SymProxy is to use whatever HTTP proxy is designated by either ProxyCfg or Netsh. If no HTTP proxy is configured, SymProxy uses a dummy proxy to allow access to secure HTTP sites within your intranet. As a side effect, this technique prevents SymProxy from working with direct connections to the external Internet. If you wish to permit SymProxy to operate with a direct connection to the Internet, create a REG_DWORD value named NoInternetProxy in the Symbol Server Proxy key of your registry. Set the value of NoInternetProxy to 1 and verify that there is no HTTP proxy indicated by ProxyCfg.”

To disable the HTTP proxy for cdb and symsrv you need to set the following keys in the registry.

For x32 version of cdb running on a x32 bit machine from the Windows Service environment.
HKLM\Software\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

For x32 version of cdb running on a x32 bit machine from a Command Prompt.
HKEY_CURRENT_USER\Software\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

For x32 version of cdb running on a x64 bit machine from the Windows Service environment.
HKLM\Software\Wow6432Node\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

For x32 version of cdb running on a x64 bit machine from a Command Prompt.
HKEY_CURRENT_USER\Software\Wow6432Node\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

For x64 version of cdb running on a x64 bit machine from the Windows Service environment.
HKLM\Software\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

For x64 version of cdb running on a x64 bit machine from a Command Prompt.
HKEY_CURRENT_USER\Software\Microsoft\Symbol Server\NoInternetProxy DWORD 1.

Http Proxy configuration.

For the above settings to work, you also need to ensure that there is no HTTP proxy configured. To do this…

Open an administrator command prompt (in elevated mode).
Type netsh winhttp show proxy. The following should be displayed.

Current WinHTTP proxy settings:

Direct access (no proxy server).

On older XP systems use ProxyCfg to configure the proxy.

The result of setting the registry and ensuring that no HTTP proxy has been configured should be that no proxy is used by WinHTTP and symbols are successfully downloaded.

13:00:24.967 ::»» WinHttp Version 6.0 Build 6.1.7600 »»Process cdb.exe [7540 (0x1d74)] started at 13:00:24.967 03/02/2011
13:00:24.968 ::WinHttpOpen(“Microsoft-Symbol-Server/6.12.0002.633”, WINHTTP_ACCESS_TYPE_NO_PROXY (1), “”, “”, 0x0)

Still problems accessing the Microsoft Symbol server?

If you are using Version 6.11.1.404 of cdb or below, then the above solution may not work. The reason for this is that cdb attempts to retrieve the full version of the symbol file (kernel32.dll) which is unavailable (returns a 404 error File not found) but does not attempt then to retrieve the compressed version of the file (kernel32.dl_).
To fix this issue, upgrade to the latest version of the Debugging Tools for Windows - 6.12.0002.633 at the time of writing.

The symbol server needs to be able to access the local symbol store c:\localstore in the examples above. Make sure that the User Account that is running the cdb.exe has Full Access to the local store folder.

This blog was transferred from the StackHash product web site. StackHash is now an OpenSource code project at codeplex.

What is StackHash?

Tue, 20 Sep 2011 11:41:00 +0100

Windows Vista “Problem Reports and Solutions” in Control Panel (click “Classic View”) allows you to define how application crashes are handled by Windows.
When you double click the item and select “Change Settings” you can configure Windows to “Check for solutions automatically” and in Advanced Settings, clicking “Automatically send more information if it is needed to help solve problems” will allow the transmission of more information to help the developer of the application fix issues.

When an application crashes, a Problem Report is sent to Microsoft. The developer of the application can log on to the Microsoft WinQual site and download problem report information associated with the applications they have developed. This information is used to help identify the cause of the error and hopefully fix it. If the error is identified and fixed, the developer posts a “response” on the Microsoft web site. This response may be that a new version or patch to the application is available to fix the issue, or it might be a description of a workaround. Either way, your PC will periodically check for fixes to previous problems on the Microsoft site and allert you if solutions are available.

But how is the specific problem identified?

Each application installed on your computer is probably made up of a number of modules.
e.g. Application.exe might use some code inside Application.dll and Utilities.dll
and these modules make use of general system modules e.g. user32.dll and kernel32.dll installed as part of the Windows Operating System.
When an application runs, it loads into memory the modules that it uses (in this case, Application.exe, Application.dll, Utilities.dll, user32.dll and kernel32.dll) and the program begins to execute instructions (code) in those modules.
To keep a track of which instruction is to be executed next, there is an Instruction Pointer (IP) which contains the address of the instruction currently being executed.

There are a number of reasons why an application may crash when executing one of these instructions. These crashes are known as exceptions. If the program has jumped into an area of memory that doesn’t contain any instructions for example, you may see an Invalid Operation exception. Or perhaps an instruction attempts to divide 10 by zero - this will cause a Divide By Zero exception.

In our example, if Utilies.dll tried to divide a value by zero then the problem report data may look something like this.

Problem Event Name: APPCRASH
Application Name: application.exe
Application Version: 2.0.02510.12346
Application Timestamp: 42e4438e
Fault Module Name: utilities.dll
Fault Module Version: 1.0.02310.12345
Fault Module Timestamp: 4549bdf8
Exception Code: c0000094
Exception Offset: 000a102b
OS Version: 6.0.6000.2.0.0.256.1
Locale ID: 1033

In this case the module is nicely identified as utilities.dll and the offset to the faulty instruction within the module is given as 000a102b and the reason for the crash is given as C0000094 (Divide By Zero).
This, along with the version of the module, goes a long way to assist the developer of the application identify and fix the fault. It also allows the particular fault to be uniquely identified so that if another user has the same fault then their information is added to the pool of information associated with the fault - thereby increasing the chances of the developer finding the reason for the fault. e.g. if the only users in the world reporting the fault are in Greece (identified by the locale id) then perhaps there is a problem displaying Greek characters.

But this information only uniquely identifies a fault if the module (utilities.dll) can be identified. Consider this next example.

If our application erroneously jumps to an area of memory that does not contain any instructions and attempts to start executing code at that address, then the Instruction Pointer will now be pointing outside any identifiable module.
i.e. it isn’t in utilities.dll or any other modules that come as part of Windows. How then can the crash that results be uniquely identified? The Problem Reports and Solutions system in Windows attempts to “invent” a unique module name as follows.

A program is made up of smaller units called functions. Functions call each other to do work (e.g. display an icon or work out some calculation). A called function may call other functions which themselves call other functions. The information about who has called whom is stored in what is known as a Stack. Every application thread has its own stack and, when a crash occurs, the stack contains useful information that identifies who was calling whom at the time of the crash. This stack information is combined into what is known as a Hash to create a “semi” unique number for the crash. The prefix StackHash_ is added to the number (which is in hexadecimal (base 16) form) giving a module name that appears in the crash information as shown below.

Problem Event Name: APPCRASH
Application Name: iexplore.exe
Application Version: 7.0.6000.16757
Application Timestamp: 48e4238e
Fault Module Name: StackHash_1ea1
Fault Module Version: 6.0.6000.16386
Fault Module Timestamp: 4549bdf8
Exception Code: c0000374
Exception Offset: 000aa0fb
OS Version: 6.0.6000.2.0.0.256.1
Locale ID: 1033

Therefore, StackHash is not a real module. It is a constructed name because the Instruction Pointer was pointing to a known module at the time of the crash. The number after the StackHash_ is a semi-unique number calculated at the time of the crash such that if the same crash occurred on multiple PCs then they have a reasonable chance to be correlated.

For a programmer to successfully track down the cause of the problem, he needs either to be able to reproduce the problem or he needs a copy of the Stack at the time of the crash. The stack will be included in a minidump which is a file automatically generated and stored at the time of the crash and sent to Microsoft as part of the crash information if the PC is configured to do so.

Some of the reasons for StackHash crashes include…

The application itself has a fault causing it to call a function in a piece of memory that does not contain any code as in the example above.
A virus checker, virtual drive S/W, security systems (or other application) has injected a piece of code into the address space of the application that crashes for some reason. In this case there is no identifiable module so a StackHash_XXXX will be created.
A virus has infected the machine and the virus code is executing in the context of the faulting application.
There is a bug in the operating system or other shared component on the system.
The application has become corrupted either by a disk error or virus.

What to do if you get a StackHash?

First configure Problems Reports and Solutions in Control panel to Check for Solutions automatically and Automatically send more information. Then check if there are fixes available for your application. If there are then download and install them.
Make sure your Window Operating system has all the latest service packs, security updates and other fixes installed.
Most applications have a Check for Updates feature. Use it to make sure you have the latest version of the S/W. Alternatively visit the company website and check for new versions and patches.
Run an up-to-date virus checker on the machine and fix any issues. If a virus is detected in the application then you may need to reinstall the application.
Re-install the application. Through Control Panel \ Programs \ Uninstall a program. Right click the application and click Uninstall / Change. This should launch the installer which may give the option to Repair the installation. If not then you will need to uninstall and reinstall - make sure you have the original installation discs available before uninstalling.
Disable any Virtual Drive software that may be hooking in to the application. If the problem goes away, contact the developer of the Virtual Drive software.
Disable any Virus checkers on your machine (temporarily) and run the application again. If the problem goes away, contact the developer of the Virus Checking software.
Check if the problem is reproducible (after a reboot). If it is, then attempt to narrow down the exact steps that cause the fault, write them down and contact the support department for the company that developed the application providing as much information about the crash and what you have done so far to attempt to resolve the issue.

This blog was transferred from the StackHash product web site. StackHash is now an OpenSource code project at codeplex.