white paper

Outlook personal folder files, better known simply as

PST files, have been around for a very long time: they

were first introduced with the Exchange client that

shipped with Exchange 4.0. PST files allow users to

manage their own mail storage and archiving, and for

that reason they are widely used within organizations

of all sizes. Users love having a convenient, portable,

personal message store. On the other hand, companies

want to be able to manage, monitor, and control infor-

mation on their networks, especially as compliance and

electronic discovery have grown in importance since

the time when Exchange 4.0 shipped. Striking the right

balance between these two opposing sets of demands

is tricky, although Microsoft has tried. This paper will

discuss the broad history and functionality of PST files,

point out some specific dangers and benefits of their

use, and describe methods you can use to control (or

prevent) the use of PST files in your organization.

The PST File RodeoBy Paul Robichaux

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 2

Understanding PST Files One of the key concepts email users have had to absorb is that their mail can be stored on a central server, not on a local com-puter. In the days when for-pay services like CompuServe and AOL dominated the email landscape, and company-provided email was relatively rare, it was natural to think of email as something you downloaded to your computer and stored locally. Most us-ers work with email on a single computer, and as a result many people still have the idea that email is stored and managed locally. (Ask around your office or dinner table if you do not believe that this is true!)

PST files were originally introduced to give users a means of locally storing messages that could tie in with Exchange Server-based storage. As originally envisioned, PST files were meant to:

• allow users to retain access to their email even while disconnected

(an especially important feature given that dial-up access was the

norm when this feature was designed!)

• provide a simple way to move mail content from place to place

• give users a way to manage their own message storage, freeing

up resources on the server

Don’t be too hard on Microsoft for the faults of PST files. The constraints that Microsoft had to meet with the initial design for PST files were pretty daunting—an email server might have a few hundred megabytes of space shared among all mailboxes on the server, while individual clients might have more. Multi-gigabyte mailboxes, or mailboxes with hundreds of thousands of items, were pretty much beyond imagining at the time, which is why the original PST file format couldn’t handle them.

Understanding PST Files

page 2

A Brief History of PST Files

page 3

PST vs. OST Filespage 3

PSTs: Pros and Cons

page 3

PSTs and Exchange 2010’s

Archivingpage 4

Managing PSTspage 5

contents

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 3

A Brief History of PST FilesThe objects we think of as “PST files” have undergone a number of major changes since their original introduction. The Outlook PST format we know and love was first introduced with Outlook 97, where it was officially known as the “Microsoft Outlook Personal Folders file.” It allowed users to store up to 2GB of mail data in a single file, with a limit of 65,000 items per folder. This format suffered from a number of limitations, though. First was the 2GB limit; that seemed like a very high limit back in 1997, but as disk capacities steadily grew, users started to run into the limit. A bigger problem was that the PST structure itself was brittle and easily corrupted. Transient problems with disk hardware, flaws in Outlook and Windows, or plain old random bad luck could quickly render an entire PST file unreadable—but so could adding data that pushed the file past its 2GB limit! These problems were com-pounded by the fact that most people did not bother to back up their local PST files, so once a PST file was damaged its contents might be unrecoverable. In addition, the original PST file format couldn’t handle content using the Unicode system of content encoding.

To address these problems, Microsoft introduced a new file format with Outlook 2003. To help keep things straight, the new format got a new name: the Microsoft Office Outlook Personal Folders file1. This new version removed the 2GB size limitation, as well as the 65,000-item cap, and it supported the Unicode character en-coding system. By default, Outlook 2003/2007/2010 Unicode PST files can grow up to 20GB, although the actual limit can be config-ured by setting a registry key. As part of the new format, Microsoft made internal format changes to help make PST files more robust. These changes are invisible to users, except that some errors that would have rendered old-format PSTs unreadable are now fixable.

Outlook 2010 did not make any changes in the PST file format (at least not that Microsoft is admitting). Some of the much-heralded performance improvements in Outlook 2007 SP2 were actually fixes from Outlook 2010 that were backported to the earlier ver-sion, but there do not seem to be any major changes to the PST file format itself.

1 If you didn’t notice much difference between the two names, you’re not alone. Most Outlook experts call the Outlook 2003-and-later versions “Unicode PSTs” and the older versions “ANSI PSTs.”

PST vs. OST FilesOne point of confusion that still lingers is the difference between PST files and the similar-sounding offline folder file (OST) format. While the PST format is designed to serve as a more or less static container for mail, the OST file is best thought of as a cache that holds a copy of all the data in a user’s mailbox. When Outlook is running in cached Exchange mode, updates from the server are applied to the OST file, as are changes the user makes locally.

When Outlook is not connected to the Exchange server, the user’s changes are still applied to the OST file; when Outlook reconnects, any changes made to the Exchange mailbox or the OST file are synchronized automatically.

Contrast this with a PST, which does not have a server-side counterpart. If your Exchange profile is set to deliver mail to a PST, when Outlook connects to the server, the mail is automatically downloaded and stored in the PST file, then removed from the server. Whatever changes you make to the local PST file, remain in the file; they are never replicated back to the server. Depend-ing on why you are using PST files, this may or may not be a good thing.

There is an additional wrinkle, too. Each OST file is linked to a spe-cific Outlook profile on an individual computer by the presence of an encrypted token. If you delete the profile, or move the OST to another computer, you can no longer open it. By contrast, PST files are designed to be freely moved between machines.There are a number of commercial tools that can convert OST files into PST files. However, the simplest way to do this is to open the OST file, create a new PST, and manually move content to it. That does, however, require you to be able to open the OST file; so if you really need to recover content from an OST file, you may end up using a commercial tool.

PSTs: Pros and ConsBecause PST files have been around for so long, many Exchange administrators think of them as a natural part of the Exchange landscape. Like other flora and fauna though, some PSTs are noxious, unwanted, and even dangerous (like, say, black widow spiders or rattlesnakes), while others see them as useful, like mosquito-eating bats. A thorough examination of the pros and

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 4

cons of PST files will help you better understand how they might fit into your landscape.

The Dangers of PST FilesOne of the first things I learned as a neophyte Exchange adminis-trator was a simple rule: PST = BAD. There are ample justifications for that rule, including these:

• Because PSTs are stored on individual workstations, they frequent-

ly do not get backed up. That means critical mail data may exist

in only one place. As an additional insult, users who move their

PSTs from place to place (or who keep multiple copies) can become

confused at the lack of integration between their PST file(s) and

their online mailbox.

• Neither Windows nor Exchange provides any tools to inventory or

track the contents of PST files; so there is no built-in way to find out

what PST files exist on your network or what is in them.

• Because users control the contents of PSTs, they may freely violate

your organization’s message retention and management poli-

cies—and they can probably do so with impunity, given the lack of

built-in PST management tools.

• PSTs are normally tied to a single computer, and they cannot be

viewed within Outlook Web Access or on mobile devices. Users who

routinely switch between OWA, mobile devices, and Outlook will

only have partial access to their mail.

• Microsoft has improved the robustness of PST files since their

introduction, but they are still relatively easy to damage or corrupt

(compared to, say, a Word document or a TIFF image).

• PST files can be password protected, but the algorithm used is

relatively weak. This can sometimes mislead users into believing

that their data is more secure than it really is.

• PST files are difficult, even impossible, to use on other platforms.

There are few reliable tools for opening and working with PST files

on Mac OS X or Linux computers.

• Versions of Outlook prior to Outlook 2007 SP2 are slow to read

and write PST files, particularly if the PST files are stored on network

shares. In fact, Microsoft’s official stance is that they do not support

the use of PST files stored on network shares because doing so can

cause the file server to exhaust its non-paged pool resources.

The Benefits of Using PST FilesAfter that long list of disadvantages you may wonder if PST files

have any redeeming features. (In fact, for a good laugh, plug “PST advantages” into your favorite search engine and examine the results—none of them actually talk about PST advantages!) The truth is that the biggest benefit of PST files is their familiarity to users; once users become accustomed to storing their own mail data, using their own preferred filing system that is totally under their control, it can be difficult to win them back. PST files do offer some other advantages though:

• They are intentionally designed to be portable. You can put a PST

file on a thumb drive or another storage device and open it from

any machine that has Outlook installed, which is often easier than

creating a new Outlook profile or otherwise connecting to your

actual Exchange server.

• They allow users to store mail in a way that does not count

against the user’s storage quota. This can be useful for users who

have a genuine business need to keep more mail around than will

fit under their actual quota.

• The PST file format is a standardized way to import and export

mail content. Microsoft’s own tools (including the Exchange

2007/2010 import-mailbox and export-mailbox cmdlets and

the recovery storage group tools included in Exchange 2003) can

produce and consume PST files, as can many third-party archiving

and management tools.

• They give users a location to store personal or sensitive content

that they do not want read by nosy administrators.

• They provide a storage mechanism for user mail that Outlook

retrieves from external IMAP or POP accounts.

• They provide an inexpensive, albeit limited, way to archive mail.

Although it is certainly not advisable to rely on PSTs as a full

archiving solution.

PSTs and Exchange 2010’s ArchivingSpeaking of archiving, it is worth mentioning the new archiving features in Exchange 2010. They are designed to provide reten-tion and control features (including litigation holds and a much-improved cross-mailbox search experience) that were not part of Exchange 2007, as well as the ability for users to create and manage their own personal archives without using PST files. The archiving functions in Exchange 2010 are well-described in the product documentation, but there are a few highlights you should know about:

• The archive mailbox for a user has to remain in the same mailbox

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 5

database as the primary mailbox. This means that you cannot

separate archive mailboxes into their own databases for disaster

recovery, backup, or high availability reasons.

• The archive mailbox is only available in Outlook 2010 and OWA

2010. There is no current way to access it from other clients.

• The archive mailbox is only available when the user is online;

there is no way to work with an offline copy.

• Users can drag and drop items between their primary mailbox,

their archive mailbox, and individual PST files, but there is no exist-

ing way to import PST files directly to users’ archive mailboxes.

Managing PSTs The bad news: you may not be able to eliminate PST files entirely. The good news: there are some things you can do to help control them. In addition, there are a few management tasks (like fixing corrupted PST files) that you will probably need to know how to do until you can eliminate them altogether.

Repairing PST FilesMicrosoft has made available a repair tool that can sometimes rescue the contents of a corrupt PST. According to its documenta-tion, the Inbox Repair Tool (better known as “scanpst.exe”) does four things:

• First, the Inbox Repair Tool makes a backup copy of whatever file

you provide it.

• The Inbox Repair Tool will try to turn any file you give it into a PST

file. For example, if you rename an executable file to junk.pst, the

tool changes the file to a mountable .pst file. The results of this

conversion may range from laughable to useful to dangerous.

• The Inbox Repair Tool analyzes the PST’s directory structure and

item headers to try to recover all folders and items.

• If the Inbox Repair Tool reports that it recovered the PST file, it

means that the repair tool found problems, and repaired what it

could. In this case, you may still have lost data.

Running the tool is simple. Here is what you need to do:

1. Quit Outlook.

2. Launch the ScanPST.Exe tool. Depending on which version of 3.

Windows you are running, it can be located in a number of differ-

ent locations, so your best bet is to search for it.

3. Click the Browse button, then navigate to the PST file you want

to repair.

4. Click the Start button. The Inbox Repair Tool will start its work.

When it is done, you will see a summary dialog like the one shown

in Figure 1. You can click the Details button to see a summary of er-

rors found during the scan, although normally the summary will be

something meaningless like “Internal errors were found in this file.”

Figure 1: After running ScanPST you can repair the file and optionally see a list of errors

5. Click the Repair button.

6. When ScanPST has finished its repair, it will display a confirma-

tion dialog telling you so. Click OK.

7. Launch Outlook and create a new PST file. This is where you will

copy information recovered from the repaired PST file.

8. Open the repaired PST file. You will notice that it contains a folder

named “Lost and Found.” This is where all your recovered items now

live. Be aware that depending on how successful the Inbox Repair

Tool was, this folder may not contain all of the items that you had

hoped to recover.

9. Drag the recovered items from the Lost and Found folder to your

new PST file.

If there are particular items that you were hoping to recover that the tool did not catch, you can open the backup file that the

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 6

Inbox Repair Tool creates; it will have the same path and name as the original file, but with a “.bak” extension. Copy the file, give the copy a “.pst” extension, and then use Outlook’s Import and Export Wizard to try to import it.

Setting a Maximum PST File SizeYou can restrict the maximum size that a PST can grow to, by add-ing registry keys to the appropriate version-specific path (HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\PST for Outlook 2007 and HKEY_CURRENT_USER\Software\Poli-cies\Microsoft\Office\11.0\Outlook\PST for Outlook 2003). There are four separate keys, as shown in the table below:

As you can see, there are separate keys for ANSI and Unicode PSTs, with different size limits for each. The Unicode format can theo-retically handle up to 33TB, but the maximum size you can specify is 20GB for Unicode PSTs and 2GB for ANSI PSTs.

Interestingly, the behavior of these keys may be different from what you expect. The WarnLargeFileSize and WarnFileSize keys do not actually generate a warning. Instead, they prevent adding items to the PST file once it reaches the specified limit. However, Outlook itself can cause the file to grow by reindexing or changing items—so the PST file’s physical size on disk can still exceed the limit. When you use the MaxFileSize and MaxLargeFileSize keys, they set a hard limit on the file size, which Outlook will not exceed.

Controlling Where PST Files GoBy default, Outlook stores users’ PST files inside their profile’s local

application data folder, the name of which varies between differ-ent versions of Windows. However, you can force Outlook to place PSTs at a path you specify.

The first route is to bundle the desired setting as part of the Of-fice installation itself. For Office 2003, you would use the Custom Installation Wizard (CIW); for Office 2007, you will need to use the Office Customization Tool (OCT). In either case, you can specify the path where PST files will be stored. This will have no effect on existing PST files, but users will only be able to create new PST files at the location you specify. Unfortunately, this approach works best when you use it before installing Office, because the configu-ration you specify is applied as part of the installation.

The second method allows you to enforce a specific path at any time, and as a bonus it can be applied to individual users. In fact, it must be applied to individual users, so keep that in mind if you want to apply a restriction globally. To use this method, you will need to add a new string value to the HKEY_CURRENT_USER\Soft-ware\Policies\Microsoft\Office\12.0\Outlook (for Outlook 2007) or HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\11.0\Outlook (for Outlook 2003) keys. The new value should be named ForcePSTPath, and it should contain the path you want to require users to use. Remember that when you put a string value in the registry that you must use “\\” to encode a single backslash, so the correct value for a desired path of z:\email would be “z:\\email.”

Preventing the Use of PST FilesYou can actually prevent Outlook users from accessing PSTs in several different ways.First, you can prevent users from creating PST files at all with the DisablePST registry key (HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\12.0\Outlook\DisablePST), a DWORD value that forces Outlook to remove the File | New | Outlook Data File and File | Archive commands from the Outlook menu bar. This keeps users from making new PSTs, as well as blocking their ability to import and export items with existing PST files.

You can also remove users’ ability to use Outlook’s AutoArchive features. Even though you block PST files, users can still use AutoArchive unless you proactively disable it. To do so, you will need to ensure that the ArchiveDelete, ArchiveMount, ArchiveOld, DeleteExpired, DoAging, and PromptForAging registry values (all

Name Applies to Valid range Default

MaxLargeFile-Size

Unicode PSTs Up to 20GB (0x00005000)

20 GB

WarnLarge-FileSize

Unicode PSTs Up to 20GB (0x00005000)

19GB

MaxFileSize ANSI PSTs 0x001F4400 – 0x7C004400 (2 GB)

1.933 GB

WarnFileSize ANSI PSTs 0x00042400 – 0x7C004400 (2GB)

1.816 GB

The PST File Rodeowhite paper

sponsored by The PST File Rodeo 7

DWORDs) are set to 0. These values live under the HKEY_CUR-RENT_USER\Software\Policies\Microsoft\Office\12\Outlook\Prefer-ences\ key. Add them manually or using your favorite policy distri-bution tool and users will no longer be able to use AutoArchive.

Auditing and Inventorying PST FilesOne of the most important tasks in PST management is finding out where PSTs exist on the network. Sadly, Microsoft still does not offer even a rudimentary tool for doing this. Third-party vendors such as Sherpa Software offer inventory and management tools that can scan network shares and individual workstations for PST files; you will need to be able to do such scans in order to bring

your PST files under control.

PST files certainly have their drawbacks, and my long-standing recommendation is to avoid their use whenever possible. For many organizations though, the flexibility and usefulness of PST files for users outweigh their faults; so knowing how to effectively manage them can greatly ease your life as an Exchange administrator.

Paul Robichaux is a founding partner at 3Sharp LLC, an MCSE, and an Exchange MVP. He is the author of several books, including The Exchange Server Cookbook (Published by O’Reilly and Associates); he is also an active blogger at http://www.robichaux.net/blog.

Managing PST FilesAccess to Outlook Personal Folders (PST files) is necessary for legal discovery, regulatory compliance, and implementation of corporate policy. The decentralized nature of PST files combined with their proliferation throughout the enterprise on desktops, file servers, and laptop machines often make the task of finding, accessing and managing these information stores extremely difficult for administrators. To alleviate the headaches often caused by PST files, Sherpa Software offers a variety of PST management tools specifically designed to locate, search, manage, migrate, and delete content in desktop and network-based PST files.

Software Highlights:Locate, Manage, and Search PST Files—PST content is not easily accessible. When data is stored locally on desktop or laptop computers, or scattered around your network, finding PST files and collecting information can be a difficult and time-consuming task. In addition, enforcing archiving, re-tention and deletion policies are much more difficult when data stores are not centrally accessible to IT staff. Sherpa’s products provide content discovery, migration and PST management for local and network PST files by allowing administrators to create flexible rules that monitor the email infrastructure and automatically archive and enforce corporate policies.

Archive and Delete PST File Content—Transferring PST content into an archive, between e-mail stores and giving users of those PST files access to the archived messages through Outlook, Sherpa’s solutions allow you to delete unwanted PST files. Since messages from the PST files are now available to be searched and restored from the archives, the use of PST files and the creation of new PST files are no longer necessary.

Backup and Synchronize PST Files—The practice of storing PST files on the network is not recommended or supported by Microsoft. However, data stored locally on desktop or laptop computers is difficult to manage and makes archiving, backup, and deletion policies challenging to enforce. PST Backup Attender is a lightweight, PST file backup utility used to locate, manage, and back up PST files on user computers, eliminating the need for storing PST files on the network.

Capture Statistics and Generate Reports—The ability to report on the content of all PST files across the entire organization is an important key step in understanding the information assets an organization possesses. With the use of Sherpa’s tools, gather comprehensive statistics and gain a total perspective over your entire email environment, including all PST files, with information such as: total number of PSTs, total message/attachment counts, size, and age.

Mail AttenderMailbox, PST File and Public Folder ManagementLearn More

Archive AttenderMailbox, PST File and Public Folder ArchivingLearn More

Discovery AttenderElectronic Search and Discovery Learn More

PST Backup AttenderPST File Backup & SynchronizationLearn More