Setting up a Symbol Server

1. Overview.
2. Set your projects up to generate symbol files (PDB - Program DataBase).
3. Set your release build or daily build to add PDBs and product binaries to a symbol store.
4. Change debugger configuration to point to use a symbol server to access the symbol store (and the Microsoft online symbol store).
5. How the debugger locates symbol files.

Overview

In order to effectively debug problems reported by customers in release versions of your products, you will need to generate and store the symbol files (PDB files - program database) for all DLLs and EXEs that comprise the product. This applies to debugging both live applications and crash dumps.
[Note that you must create the PDBs at the same time as the product binaries because the debugger uses timestamp and GUID information to match PDB files to binaries during debugging and these will change when you rebuild - even if the source code is the same.]
Native code symbol files contain full annotations of the binary including: source filenames and line numbers, functions (static, private and public), types, variable names, Frame Point Omission (FPO) data etc. Managed symbol files contain less debug information because most of the symbol information is available in the assembly file metadata.
You can either store the symbols with the rest of the product in your “release directory” or you can store the symbols (and binaries) in a symbol store located on a PC on your intranet for easy access by all members of the programming team. The latter case is described here.
You can configure your daily build or automated release procedure to automatically add symbols to the symbol store after a successful build.

When debugging, you configure Visual Studio or WinDbg to access the Symbol Store using a symbol server (symserv.dll).

In addition to product specific symbols, you will also need matching symbols for the system DLLs that were loaded on the customer’s machine at the time a dump file was generated. Microsoft provides an online symbol server containing symbols for these system DLLs.
You may also need copies of the various .NET version DLLs in order to debug managed applications.

Generating Symbol Files

For C# and VB.NET: View the project properties. Select Build tab - then scroll down and click the Advanced button. In the Advanced Build Settings dialog make sure that Debug Info is set to “pdb-only” for release builds and “full” for debug builds. Full allows the debugger to attach to the process when it is running. From the command line use /debug:full or /debug:pdbonly.

For C++: View the project properties. Select C/C++. Select the General property page. For the Debug Information Format option select Program Database. From the command line use /Zi.

The PDB file will be called project.pdb and will be placed in the project output folder along with the binary (EXE or DLL).

Setting up a Symbol Store

To ensure that you always have the correct symbol files matching the binaries you are debugging you should set up a company Symbol Store.
You then add your binaries and symbols to the symbol store during your daily builds or at least during your release builds.
Your build server machine is a good candidate for the location of your symbol server.
The symbol server tools are part of the Debugging Tools For Windows. Make sure you use the latest version of the debugging tools and check for updates regularly.

To create a Symbol Store.

1. Download the Debugging Tools for Windows from http://www.microsoft.com/whdc/Devtools/Debugging/default.mspx and install on the server.
2. Create a folder to store the symbols (e.g. c:\SymStore)
3. Share this folder on the network so other users can reach it.
4. Add the symstore command to your daily build or release batch files. e.g. to add files recursively (/r) from the product root folder (/f r:\rootproductfolder) to the symbol store folder (/s c:\symstore) with the product name (/t “MyApp”) with a specified version (/v “%1”) and comment (/c “Daily Build”) use…

symstore add /r /f r:\rootproductfolder /s c:\SymStore /t “MyApp” /v “%1” /c “Daily Build”

This example uses /v “%1” where the symstore command is in a DOS batch file. %1 is just parameter 1 passed into the batch file. If you are running inside an MSBUILD file then use “$(Version)” or something similar. The main point here is that you use a different version (e.g. the build number of the product) each time you add files to the symstore. This way you can easily identify versions that can be deleted from the symbol store should the symbol store become bloated over time.

For detailed options to the symstore command see http://msdn.microsoft.com/en-us/library/ms681378(VS.85).aspx

Debugger access to the symbol store

Install Debugging Tools for Windows if you are using WinDbg. Visual Studio should come with a version of symsrv.dll.

Visual Studio - Select Tools \ Options \ Debugging \ Symbols. Add “http://msdl.microsoft.com/download/symbols” and “\MyServer\SymStore”. Include a local cache for speed (e.g. c:\localstore). Symbol files and binaries downloaded from the symbol servers are cached in c:\localstore for future use. If c:\localstore gets too large you can either delete the whole folder or use AgeStore.exe (part of Debugging Tools for Windows) to remove files older than a particular date.

WinDbg - Select File \ Symbol File Path and include: SRV*c:/localstore*http://msdl.microsoft.com/download/symbols;SRV*c:\localstore*\MyServer\SymStore

You can append additional paths on the end delimited by a semicolon.
This command says: SRV* (shorthand for symsrv*symsrv.dll) - indicates that this part of the path is a symbol server reference and the debugger should use symsrv.dll to access the remote symbol store. The*c:\localstore*” indicates that downloaded symbols and binaries should be cached in the specified folder for future use.
http://msdl.microsoft.com/download/symbols is the location of the Microsoft public symbol store and \MyServer\SymStore is the location of your local store.
Various cascading cache options are available. For more information see the Symbol Store help files contained in Debugging Tools for Windows.

Also, in WinDbg you should add the same paths to the File \ Image File Path dialog.

How the debugger finds the right symbol file

The linker (compiler for managed code) embeds the path to the PDB file into the exe or DLL along with a GUID and timestamp. You can see this header using dumpbin.exe (part of the windows SDK). From a command prompt type dumpbin /headers myapp.exe >headers.txt. Then open headers.txt in notepad and you should see something like…

Debug Directories
Time Type Size RVA Pointer
———— ——— ———— ———— ————
4B73E828 cv 48 00002BC8 1BC8 Format: RSDS, {2AB1D3E2-6F7F-43EB-903B-CA57510C08D5}, 2, c:\dev\myapp.pdb

The debugger will search the following folders.

1. The folder where the assembly was loaded (exe or dll).
2. The folder specified in the Debug Directories section of the assembly (c:\dev\myapp.pdb in the example above).
3. Any folders that you add to the symbol search path (see next section).

Tools.

• Dumpbin.exe - Dumps (native code) information contained in a binary file (DLL, EXE, COM).
• SymChk.exe - part of Debugging Tools For Windows - you can use to determine if a PDB has public or private symbols present.
• SymStore.exe - part of Debugging Tools For Windows - used to create a symbol store and add/remove files.
• AgeStore.exe - part of Debugging Tools For Windows - can be used to delete files in a directory heirarchy (e.g. a local symbol cache) older than a particular age.
• SymSrv.dll - must be installed on the same machine as the debugger.

More Information.
http://msdn.microsoft.com/en-us/library/b8ttk8zy.aspx
http://www.microsoft.com/whdc/Devtools/Debugging/default.mspx - See the help file description of symbol servers.
http://msdn.microsoft.com/en-us/library/ms681378(VS.85).aspx

This blog was transferred from the StackHash product web site. StackHash is now an OpenSource code project at codeplex.