svr333 advanced windows troubleshooting with sysinternals filemon and regmon mark russinovich chief...

61

Upload: cody-lucas

Post on 17-Dec-2015

241 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©
Page 2: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

SVR333SVR333 Advanced Windows Advanced Windows Troubleshooting with Troubleshooting with Sysinternals Filemon and Sysinternals Filemon and RegmonRegmon

SVR333SVR333 Advanced Windows Advanced Windows Troubleshooting with Troubleshooting with Sysinternals Filemon and Sysinternals Filemon and RegmonRegmon

Mark RussinovichMark RussinovichChief Software ArchitectChief Software ArchitectWinternals SoftwareWinternals SoftwareCopyright © 2006 Mark RussinovichCopyright © 2006 Mark Russinovich

Page 3: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

3

Mark RussinovichMark Russinovich

Co-founder and chief software architect Co-founder and chief software architect of Winternals Software of Winternals Software ((www.winternals.comwww.winternals.com))

Co-author of Windows Internals, 4th Co-author of Windows Internals, 4th edition and Inside Windows 2000, 3rd edition and Inside Windows 2000, 3rd Edition with David SolomonEdition with David Solomon

Author of tools on Author of tools on www.sysinternals.comwww.sysinternals.com Home of Mark’s blog and forumsHome of Mark’s blog and forums

Microsoft Most Valuable Professional Microsoft Most Valuable Professional (MVP)(MVP)

Senior Contributing Editor to Windows IT Senior Contributing Editor to Windows IT Pro MagazinePro Magazine

Ph.D. in Computer EngineeringPh.D. in Computer Engineering

Page 4: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

4

David SolomonDavid Solomon

President of David Solomon Expert President of David Solomon Expert Seminars (Seminars (www.solsem.comwww.solsem.com) )

Founded in 1992Founded in 19921982-1992: VMS operating systems 1982-1992: VMS operating systems developmentdevelopment

Teach public and private live classes on Teach public and private live classes on Windows Internals and Advanced Windows Internals and Advanced TroubleshootingTroubleshootingMicrosoft Most Valuable Professional (MVP)Microsoft Most Valuable Professional (MVP)BooksBooks

Windows Internals, 4th editionWindows Internals, 4th editionInside Windows 2000, 3rd editionInside Windows 2000, 3rd editionInside Windows NT, 2nd editionInside Windows NT, 2nd editionWindows NT for OpenVMS ProfessionalsWindows NT for OpenVMS Professionals

VideosVideosWindows Internals COMPLETEWindows Internals COMPLETENew! Sysinternals Video Library (see DVD in New! Sysinternals Video Library (see DVD in bag)bag)

Page 5: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

5

OutlineOutline

IntroductionIntroduction

Troubleshooting with FilemonTroubleshooting with Filemon

Troubleshooting with RegmonTroubleshooting with Regmon

Using Filemon and Regmon TogetherUsing Filemon and Regmon Together

Page 6: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

6

Troubleshooting Application Troubleshooting Application FailuresFailures

Most applications do a poor job of reporting Most applications do a poor job of reporting file-related or registry-related errorsfile-related or registry-related errors

Permissions problemsPermissions problems

Missing filesMissing files

Missing or corrupt registry dataMissing or corrupt registry data

Errors manifest in several different waysErrors manifest in several different waysMisleading error messagesMisleading error messages

CrashesCrashes

Silently exitingSilently exiting

HangsHangs

Page 7: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

7

Troubleshooting Application Troubleshooting Application FailuresFailures

When in doubt, run Filemon and Regmon!When in doubt, run Filemon and Regmon!Filemon monitors file I/OFilemon monitors file I/ORegmon monitors registry activityRegmon monitors registry activity

Ideal for troubleshooting a wide variety of Ideal for troubleshooting a wide variety of application failuresapplication failuresAlso useful for to understand and tune file Also useful for to understand and tune file system and Registry accesssystem and Registry access

Understand hard drive activityUnderstand hard drive activityOptimize application installation and Optimize application installation and configurationconfiguration

Filemon and Regmon run on Microsoft Filemon and Regmon run on Microsoft Windows 95, Windows 98, Windows Me, Windows 95, Windows 98, Windows Me, Windows 2000, Windows XP, Windows Windows 2000, Windows XP, Windows Server 2003, x64 64-bit Editions, Windows Server 2003, x64 64-bit Editions, Windows VistaVista

Page 8: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

8

Using Regmon/FilemonUsing Regmon/Filemon

Two basic techniques:Two basic techniques:Go to end of log and look backwards to where Go to end of log and look backwards to where problem occurred or is evident and focused on problem occurred or is evident and focused on the last the last things donethings doneCompare a good log with a bad logCompare a good log with a bad log

Often comparing the I/O and Registry Often comparing the I/O and Registry activity of a failing process with one that activity of a failing process with one that works may point to the problemworks may point to the problem

Have to first massage log file to remove data Have to first massage log file to remove data that differs run to runthat differs run to run

Delete first 3 columns (they are always different: line Delete first 3 columns (they are always different: line number, time, process id)number, time, process id)

Easy to do with Microsoft Office Excel by deleting Easy to do with Microsoft Office Excel by deleting columnscolumns

Then compare with FC (built in tool) or Windiff Then compare with FC (built in tool) or Windiff (Resource Kit)(Resource Kit)

Page 9: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

9

OutlineOutline

IntroductionIntroduction

Troubleshooting with FilemonTroubleshooting with Filemon

Troubleshooting with RegmonTroubleshooting with Regmon

Using Filemon and Regmon TogetherUsing Filemon and Regmon Together

Page 10: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

10

How Filemon WorksHow Filemon Works

Filemon is based on a file system “filter driver” Filemon is based on a file system “filter driver” Extracts the driver to \Windows\System32\DriversExtracts the driver to \Windows\System32\Drivers

Installs the driverInstalls the driver

Deletes the driver fileDeletes the driver file

Requires “Debug Programs” user rightRequires “Debug Programs” user rightFirst run requires the “Load Driver” user rightFirst run requires the “Load Driver” user right

ApplicationApplication

Filemon Filemon DriverDriver

FilemonFilemonGUIGUI

File SystemFile SystemDriverDriver

User ModeUser ModeKernel ModeKernel Mode

Page 11: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

11

Using FilemonUsing Filemon

# - operation number# - operation number

Process: image name + process idProcess: image name + process id

Request: internal I/O request codeRequest: internal I/O request code

Result: return code from I/O operationResult: return code from I/O operation

Other: flags passed on I/O requestOther: flags passed on I/O request

Page 12: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

12

Controlling FilemonControlling Filemon

Start/stop logging (Control/E)Start/stop logging (Control/E)

Clear display (Control/X)Clear display (Control/X)

Open Microsoft Internet Explorer window to Open Microsoft Internet Explorer window to folder containing file:folder containing file:

Double click on a line does thisDouble click on a line does this

Find – finds text within windowFind – finds text within window

Save to log fileSave to log file

Advanced modeAdvanced mode

Network optionNetwork option

Page 13: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

13

What Filemon MonitorsWhat Filemon Monitors

By default Filemon traces all file I/O to:By default Filemon traces all file I/O to:Local non-removable mediaLocal non-removable media

Network sharesNetwork shares

It saves all output for displayIt saves all output for displayCan exhaust virtual memory in Can exhaust virtual memory in long runslong runs

You can limit captured data with You can limit captured data with history depthhistory depth

You can limit what is monitored:You can limit what is monitored:What volumes to watch in Volumes menuWhat volumes to watch in Volumes menu

What paths and processes to watch in Filter dialogWhat paths and processes to watch in Filter dialog

What operations to watch in Filter dialog What operations to watch in Filter dialog (reads, writes, successes and errors)(reads, writes, successes and errors)

Page 14: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

14

Filemon Filtering and Filemon Filtering and HighlightingHighlighting

Include and exclude filters are substring matches Include and exclude filters are substring matches against the process and path columnsagainst the process and path columns

Exclude overrides include filterExclude overrides include filter

Be careful that you don’t exclude potentially Be careful that you don’t exclude potentially useful datauseful data

Capture everything and save the logCapture everything and save the log

Then apply filters (you can always reload the log)Then apply filters (you can always reload the log)

Highlight matches all columnsHighlight matches all columns

Page 15: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

15

Understanding Disk ActivityUnderstanding Disk Activity

Use Filemon to see why you’re hard disk is Use Filemon to see why you’re hard disk is crunchingcrunching

Process performance counters show I/O activity, but not Process performance counters show I/O activity, but not to whereto where

System performance counters show which disks are System performance counters show which disks are being hit, but not which files or which processbeing hit, but not which files or which process

Filemon pinpoints which file(s) are being accessed, by Filemon pinpoints which file(s) are being accessed, by whom, and how frequentlywhom, and how frequently

You can also use Filemon on a server to determine You can also use Filemon on a server to determine which file(s) were being accessed most frequentlywhich file(s) were being accessed most frequently

Import into Excel and make a pie chart by file name or Import into Excel and make a pie chart by file name or operation typeoperation type

Move heavy-access files to a different disk on a Move heavy-access files to a different disk on a different controllerdifferent controller

Page 16: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

16

Polling and File Change Polling and File Change NotificationNotification

Many applications respond to file and directory Many applications respond to file and directory changeschanges

A poorly written application will “poll” for changesA poorly written application will “poll” for changesA well-written application will request notification by the A well-written application will request notification by the system of changessystem of changes

Polling for changes causes performance Polling for changes causes performance degradationdegradation

Context switches including TLB flushContext switches including TLB flushCache invalidationCache invalidationPhysical memory usagePhysical memory usageCPU usageCPU usage

Alternative: file change notification Alternative: file change notification When you run Filemon on an idle system you When you run Filemon on an idle system you should should only see bursty system background activityonly see bursty system background activity

Polling is visible as periodic accesses to the same files Polling is visible as periodic accesses to the same files and directoriesand directoriesFile change notification is visible as directory queries that File change notification is visible as directory queries that

have no resulthave no result

Page 17: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

17

Demo: Change NotifyDemo: Change Notify

Explorer posts change notify to know when Explorer posts change notify to know when directory contents change for open directory contents change for open Internet Explorer windowsInternet Explorer windows

Open Internet Explorer window to a folderOpen Internet Explorer window to a folder

Create or delete a fileCreate or delete a file

Page 18: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

18

Demo: Demo: Understanding Notepad’s File Understanding Notepad’s File SaveSave

1.1. Run FilemonRun Filemon

2.2. Set filter to only include Notepad.exeSet filter to only include Notepad.exe

3.3. Run NotepadRun Notepad

4.4. Type some textType some text

5.5. Save file as “test.txt”Save file as “test.txt”

6.6. Go back to FilemonGo back to Filemon

7.7. Stop loggingStop logging

8.8. Set highlight to “test.txt”Set highlight to “test.txt”

9.9. Find line representing creation of new fileFind line representing creation of new fileHint: look for create operationHint: look for create operation

Page 19: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

19

Basic vs. Advanced ModeBasic vs. Advanced Mode

Basic mode massages output to be Basic mode massages output to be sysadmin-friendly and target common sysadmin-friendly and target common troubleshootingtroubleshooting

Things you don’t see in Basic mode:Things you don’t see in Basic mode:Raw I/O request namesRaw I/O request names

Various internal file system operationsVarious internal file system operations

Activity in the System processActivity in the System process

Page file I/OPage file I/O

Filemon file system activityFilemon file system activity

Page 20: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

20

Example: Word CrashExample: Word Crash

While typing in the document Microsoft While typing in the document Microsoft Office Word XP would intermittently close Office Word XP would intermittently close without any error messagewithout any error message

To troubleshoot ran Filemon on user’s To troubleshoot ran Filemon on user’s systemsystem

Set the history depth to 10,000Set the history depth to 10,000

Asked user to send Filemon log when Asked user to send Filemon log when Word exitedWord exited

Page 21: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

21

Solution: Word CrashSolution: Word Crash

Working backwards, the first “strange” or Working backwards, the first “strange” or unexplainable behavior are the constant unexplainable behavior are the constant reads past end of file to MSSP3ES.LEXreads past end of file to MSSP3ES.LEX

User looked up what .LEX file wasUser looked up what .LEX file wasRelated to Word proofing toolsRelated to Word proofing tools

Uninstalled and reinstalled proofing tools and Uninstalled and reinstalled proofing tools and problem went awayproblem went away

Page 22: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

22

Example: Build FailsExample: Build Fails

While building a program using nmake on While building a program using nmake on a command line link reported an error:a command line link reported an error:

““error writing to program database, check for error writing to program database, check for insufficient disk space, invalid path, or insufficient disk space, invalid path, or insufficient privileges”insufficient privileges”

Page 23: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

23

Solution: Build FailsSolution: Build Fails

Saw sharing violation in Filemon:Saw sharing violation in Filemon:

Performed a handle search for the file in Performed a handle search for the file in Process ExplorerProcess Explorer

Saw Windbg had it opened from an earlier Saw Windbg had it opened from an earlier debug session even though debug session was debug session even though debug session was closedclosed

Closed WindbgClosed Windbg

Page 24: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

24

Example: Example: Useless Excel Error Useless Excel Error MessageMessage

Excel reports an error “Unable to read file" Excel reports an error “Unable to read file" when startingwhen starting

Page 25: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

25

Solution: Excel Error MessageSolution: Excel Error Message

Filemon trace shows Excel reading file in Filemon trace shows Excel reading file in XLStart folder XLStart folder

All Microsoft Office apps autoload files in their All Microsoft Office apps autoload files in their start foldersstart folders

Should have reported:Should have reported:Name and location of fileName and location of file

Reason why it didn’t like itReason why it didn’t like it

Page 26: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

26

DLL ProblemsDLL Problems

Process Explorer may solve a DLL Process Explorer may solve a DLL versioning issue, but may not if:versioning issue, but may not if:

A DLL is missingA DLL is missing

The order of DLL loads is relevantThe order of DLL loads is relevant

So, use Filemon!So, use Filemon!Look at the last DLL opened before the Look at the last DLL opened before the application diedapplication died

Compare the startup of a working with a Compare the startup of a working with a failing applicationfailing application

Missing or inaccessible DLLs often not reported Missing or inaccessible DLLs often not reported correctlycorrectly

Look for “NOTFOUND” or “ACCESS DENIED”Look for “NOTFOUND” or “ACCESS DENIED”

May be opening wrong versions due to wrong versions May be opening wrong versions due to wrong versions being in folders in PATHbeing in folders in PATH

Page 27: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

27

Example: Word DiesExample: Word Dies

Word starts and a few seconds later gets a Word starts and a few seconds later gets a Dr. Watson (access violation)Dr. Watson (access violation)

Customer tried re-installing Microsoft Office Customer tried re-installing Microsoft Office – still failed– still failed

Solution:Solution:Ran Filemon, looked at last DLL loaded before Ran Filemon, looked at last DLL loaded before Dr. WatsonDr. Watson

It was a printer DLLIt was a printer DLL

Uninstalled printer – problem went awayUninstalled printer – problem went away

Page 28: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

28

Problem: Perfmon HangsProblem: Perfmon Hangs

Perfmon hung when startingPerfmon hung when starting

IIS performance counter DLL was last thing IIS performance counter DLL was last thing Perfmon referencedPerfmon referenced

Page 29: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

29

Solution: Perfmon HangsSolution: Perfmon Hangs

Services snapin showed IIS was hung Services snapin showed IIS was hung starting starting

Investigation revealed an IIS Investigation revealed an IIS configuration problemconfiguration problem

Page 30: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

30

Example: Help FailsExample: Help Fails

The Help command in an application failed The Help command in an application failed on Windows 95, but worked fine on on Windows 95, but worked fine on Windows 98/Windows Me/Windows Windows 98/Windows Me/Windows NT4/Windows 2000/Windows XPNT4/Windows 2000/Windows XP

Failed with meaningless error messageFailed with meaningless error message

Ran Filemon on failing system and Ran Filemon on failing system and working systemworking system

Reduced log to file opensReduced log to file opens

Compared logsCompared logs

Page 31: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

31

Solution: Help FailsSolution: Help Fails

At the point logs diverged, looked At the point logs diverged, looked backwards to last common thing donebackwards to last common thing done

An OLE system DLL was loadedAn OLE system DLL was loaded

Noticed this OLE DLL was loaded from a Noticed this OLE DLL was loaded from a directory in the user’s PATH on Windows 95, but directory in the user’s PATH on Windows 95, but from \Windows\System on other versionsfrom \Windows\System on other versions

Conclusion:Conclusion:DLL loaded on Windows 95 system was not for DLL loaded on Windows 95 system was not for Windows 95Windows 95

Got proper version for Windows 95, problem Got proper version for Windows 95, problem went awaywent away

Page 32: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

32

Example: Access HangsExample: Access Hangs

Problem: Access would hang when trying Problem: Access would hang when trying to import an Excel fileto import an Excel file

Worked fine on other users’ workstationsWorked fine on other users’ workstations

Traced startup of Access on failing and Traced startup of Access on failing and working systemsworking systems

Page 33: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

33

Solution: Access HangsSolution: Access Hangs

Compared logs and looked for first Compared logs and looked for first unexplainable differenceunexplainable difference

First unexplainable difference was that First unexplainable difference was that Accwiz.dll was being loaded from two different Accwiz.dll was being loaded from two different directoriesdirectories

Failing system was loading an old Access DLL Failing system was loading an old Access DLL from \windows\system32 due to having from \windows\system32 due to having installed older Access previouslyinstalled older Access previously

Solution: Removed DLL in \windows\Solution: Removed DLL in \windows\system32 and problem went awaysystem32 and problem went away

Page 34: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

34

Example: Pinnacle Studio Example: Pinnacle Studio HangsHangs

User had a hang when launching Pinnacle User had a hang when launching Pinnacle Studio Studio

Filemon showed accesses to CyberPatrol's Filemon showed accesses to CyberPatrol's DLL, an Internet filtering toolDLL, an Internet filtering tool

Cyberpatrol monitors processes by loading Cyberpatrol monitors processes by loading a DLL into thema DLL into them

Uninstalling CyberPatrol fixed the problemUninstalling CyberPatrol fixed the problem

Page 35: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

35

Example: Misleading AOL ErrorExample: Misleading AOL Error

AOL worked in one user’s account, but AOL worked in one user’s account, but failed with this on another account on the failed with this on another account on the same system:same system:

User reinstalled AOL, but problem User reinstalled AOL, but problem persistedpersisted

Page 36: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

36

Example: Misleading AOL ErrorExample: Misleading AOL Error

User did not have admin rights to AOL User did not have admin rights to AOL directorydirectory

This version of AOL was not limited-user This version of AOL was not limited-user account friendlyaccount friendly

Page 37: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

37

Example: Microsoft Office Example: Microsoft Office Outlook Application ErrorOutlook Application Error

For example, an Outlook application failed with For example, an Outlook application failed with this error:this error:

Ran Filemon and found it was getting Access Ran Filemon and found it was getting Access DeniedDenied

Someone had misread a request to remove EDIT Someone had misread a request to remove EDIT rights and removed all rightsrights and removed all rights

Page 38: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

38

Example: Microsoft Software Example: Microsoft Software Installer Misleading ErrorInstaller Misleading Error

User received this message trying to install User received this message trying to install something:something:

Filemon showed the real reason:Filemon showed the real reason:

Page 39: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

39

Demo: Permission Problems Demo: Permission Problems and Misleading Error Messagesand Misleading Error Messages

1.1. In Explorer, create a folder c:\noaccessIn Explorer, create a folder c:\noaccess

2.2. Remove all rights to the folderRemove all rights to the folder

3.3. Run Notepad & type some textRun Notepad & type some text

4.4. Run Filemon – set filter to Notepad.exeRun Filemon – set filter to Notepad.exe

5.5. In Notepad, File->Save As to c:\noaccess\In Notepad, File->Save As to c:\noaccess\test.txttest.txt

Note error reportedNote error reported

6.6. Look at Filemon trace and find Access Look at Filemon trace and find Access DeniedDenied

Page 40: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

40

OutlineOutline

IntroductionIntroduction

Troubleshooting with FilemonTroubleshooting with Filemon

Troubleshooting with RegmonTroubleshooting with Regmon

Using Filemon and Regmon TogetherUsing Filemon and Regmon Together

Page 41: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

41

Configuration ProblemsConfiguration Problems

Missing, corrupted or overly-secure Missing, corrupted or overly-secure Registry settings often lead to application Registry settings often lead to application crashes crashes and errorsand errors

Some applications don’t completely Some applications don’t completely remove registry data at uninstallremove registry data at uninstall

Regmon may yield the answerRegmon may yield the answer

Page 42: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

42

How Regmon WorksHow Regmon Works

Regmon uses a driver to intercept Regmon uses a driver to intercept Registry operationsRegistry operations

Up until now Regmon has relied Up until now Regmon has relied on system call “hooking” to on system call “hooking” to intercept Registry accessesintercept Registry accesses

Hooking isn’t supported by the Hooking isn’t supported by the kernelkernel

As of Windows XP the system call As of Windows XP the system call table is write-protected by default table is write-protected by default if a system has <256 MB, requiring if a system has <256 MB, requiring

a tricka trick

Windows Server 2003 introduces Windows Server 2003 introduces a Registry callback mechanisma Registry callback mechanism

Driver can see and modify Driver can see and modify Registry behaviorRegistry behavior

Latest version of Regmon comes Latest version of Regmon comes with two drivers: one for Windows with two drivers: one for Windows Server 2003 and one for Server 2003 and one for previous versionsprevious versions

Defined in a DDK header file and Defined in a DDK header file and used by antivirus productsused by antivirus products

ApplicationApplication

Regmon Regmon DriverDriver

RegmonRegmonGUIGUI

Registry Registry SubsystemSubsystem

Page 43: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

43

RegmonRegmon

UI is similar to FilemonUI is similar to Filemon

Request: OpenKey, CreateKey, SetValue, Request: OpenKey, CreateKey, SetValue, QueryValue, CloseKeyQueryValue, CloseKey

PathPathHKCU=HKEY_CURRENT_USER (per-user settings)HKCU=HKEY_CURRENT_USER (per-user settings)

HKLM=HKEY_LOCAL_MACHINE (system wide settings)HKLM=HKEY_LOCAL_MACHINE (system wide settings)

Result – return code from Registry operationResult – return code from Registry operation

Other – extended information or resultsOther – extended information or results

Page 44: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

44

Polling and Registry Change Polling and Registry Change Notification Notification

Many applications want to respond to Many applications want to respond to

Registry changesRegistry changesPolling the Registry is just as bad for Polling the Registry is just as bad for performanceperformance

Applications can request to be notified of Applications can request to be notified of changeschanges

Like with Filemon, Regmon should be idle Like with Filemon, Regmon should be idle on an idle systemon an idle system

Page 45: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

45

Registry TroubleshootingRegistry Troubleshooting

If you suspect registry data is causing If you suspect registry data is causing problems, rename the key and rerun the problems, rename the key and rerun the applicationapplication

Most applications recreate user settings when Most applications recreate user settings when runrun

In this way, the data won’t be seen by the In this way, the data won’t be seen by the applicationapplication

Can always rename the key backCan always rename the key back

Use Regmon to discover application Use Regmon to discover application settings locationsettings location

Page 46: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

46

Demo: Finding Notepad’s Demo: Finding Notepad’s SettingsSettings1.1. Run NotepadRun Notepad

2.2. Change FontChange Font

3.3. Run Regmon and filter to Notepad.exeRun Regmon and filter to Notepad.exe

4.4. Exit NotepadExit Notepad

5.5. In Regmon log, find location of user-In Regmon log, find location of user-specific Notepad settingsspecific Notepad settings

6.6. Double click on a line to jump to RegeditDouble click on a line to jump to Regedit

7.7. Delete top level Notepad user settings keyDelete top level Notepad user settings key

8.8. Re-run Notepad and confirm font resets to Re-run Notepad and confirm font resets to default settingdefault setting

Page 47: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

47

Example: Missing Word ToolbarExample: Missing Word Toolbar

Problem:Problem:User somehow disabled all toolbars and menus User somehow disabled all toolbars and menus in Word in Word

No way to open files, change settings, etc.No way to open files, change settings, etc.

Solution:Solution:With Regmon, captured startup of WordWith Regmon, captured startup of Word

Found location of user-specific settings for WordFound location of user-specific settings for Word

Deleted this Registry keyDeleted this Registry key

Re-ran Word, which recreated user settings Re-ran Word, which recreated user settings from scratchfrom scratch

Page 48: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

48

Example: Misleading Internet Example: Misleading Internet Explorer Error MessageExplorer Error Message

Internet Explorer failed to start with Internet Explorer failed to start with this error:this error:

First, looked on system for ICFGNT.DLLFirst, looked on system for ICFGNT.DLLNot thereNot there

Not on other systems in the network, eitherNot on other systems in the network, either

Page 49: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

49

Solution: Misleading Internet Solution: Misleading Internet Explorer Error MessageExplorer Error Message

Captured Regmon trace and looked Captured Regmon trace and looked backwards from end of Regmon logbackwards from end of Regmon log

Saw query of Completed value in Internet Saw query of Completed value in Internet Connection Wizard keyConnection Wizard key

Value read was 0Value read was 0

Value was 1 on other systemsValue was 1 on other systems

Solution: Set value to 1 and problem went Solution: Set value to 1 and problem went awayaway

Page 50: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

50

Example: Internet Explorer Example: Internet Explorer HangsHangs

Internet Explorer hung when started unless Internet Explorer hung when started unless user manually dialed ISPuser manually dialed ISP

Captured a Regmon trace and looked Captured a Regmon trace and looked backwards from point Internet Explorer backwards from point Internet Explorer was hungwas hung

Found references to ATT under a RAS Found references to ATT under a RAS PhoneBook keyPhoneBook key

Solution: renamed ATT key and problem Solution: renamed ATT key and problem went awaywent away

Conclusion: previous ISP’s dialer had left Conclusion: previous ISP’s dialer had left junk behindjunk behind

Page 51: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

51

Example: Example: Misleading Microsoft Misleading Microsoft Visual Basic for Applications (VBA) Visual Basic for Applications (VBA) ErrorErrorUser got this error installing an application:User got this error installing an application:

Regmon showed permissions problemRegmon showed permissions problem

Solution: Edited permissionsSolution: Edited permissions

Page 52: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

52

Missing SettingsMissing Settings

Sometimes queries to what is Sometimes queries to what is notnot there is there is more interesting than what more interesting than what isis there there

Identify missing Registry keysIdentify missing Registry keysSearch for status “NOTFOUND”Search for status “NOTFOUND”

May reveal hidden capabilitiesMay reveal hidden capabilities

Page 53: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

53

Example: Hidden CapabilitiesExample: Hidden Capabilities

User tried installing Compaq utility on non-User tried installing Compaq utility on non-Compaq hardwareCompaq hardware

Regmon showed hidden key to override:Regmon showed hidden key to override:

Solution: created key and install succeededSolution: created key and install succeeded

Page 54: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

54

OutlineOutline

Troubleshooting with FilemonTroubleshooting with Filemon

Troubleshooting with RegmonTroubleshooting with Regmon

Using Filemon and Regmon TogetherUsing Filemon and Regmon Together

Page 55: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

55

Filemon and RegmonFilemon and Regmon

Many times its not clear whether a problem Many times its not clear whether a problem is Registry or file relatedis Registry or file related

And sometimes problems involve both Registry And sometimes problems involve both Registry and file configuration problemsand file configuration problems

Always run Filemon and Regmon Always run Filemon and Regmon when troubleshootingwhen troubleshooting

Page 56: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

56

Example: Internet Explorer Example: Internet Explorer HangsHangs

Internet Explorer started hanging on Internet Explorer started hanging on certain folders certain folders

Hangs were up to a minuteHangs were up to a minute

Internet Explorer would work normally for a Internet Explorer would work normally for a minute and then hang againminute and then hang again

Page 57: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

57

Solution: Internet Explorer Solution: Internet Explorer HangsHangs

Ran Filemon and saw network path errorRan Filemon and saw network path errorContained references to decommissioned Contained references to decommissioned computercomputer

Regmon showed icon lookup configured for Regmon showed icon lookup configured for missing computermissing computer

Fix: Delete Paint Shop Pro (PSP) browse Fix: Delete Paint Shop Pro (PSP) browse files and all PSP file associationsfiles and all PSP file associations

Page 58: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

58

Running Filemon/Regmon Running Filemon/Regmon Before LogonBefore Logon

Sometimes need to capture I/O or registry Sometimes need to capture I/O or registry activity during boot, the logon or logoff activity during boot, the logon or logoff process process

Problem: when you logoff all your processes Problem: when you logoff all your processes are terminatedare terminated

Solutions:Solutions:Run Filemon/Regmon in a different logon Run Filemon/Regmon in a different logon sessionsession

psexec –s –i –dpsexec –s –i –d

Run Filemon/Regmon from a serviceRun Filemon/Regmon from a serviceUse Srvany (Resource Kit)Use Srvany (Resource Kit)

Use Regmon’s log boot optionUse Regmon’s log boot option

Page 59: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

59

ResourcesResourcesTechnical Chats and Webcastshttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp

Microsoft Learning and Certificationhttp://www.microsoft.com/learning/default.mspx

MSDN & TechNet http://microsoft.com/msdnhttp://microsoft.com/technet

Virtual Labshttp://www.microsoft.com/technet/traincert/virtuallab/rms.mspx

Newsgroupshttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspx

Technical Community Siteshttp://www.microsoft.com/communities/default.mspx

User Groupshttp://www.microsoft.com/communities/usergroups/default.mspx

Page 60: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

60

Fill out a session Fill out a session evaluation on evaluation on

CommNet andCommNet and Win an XBOX Win an XBOX

360!360!

Page 61: SVR333 Advanced Windows Troubleshooting with Sysinternals Filemon and Regmon Mark Russinovich Chief Software Architect Winternals Software Copyright ©

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.