Skip to Content
Skip to Table of Contents

← Previous Article Next Article →

ATPM 8.11
November 2002

Columns

Segments

How To

Extras

Reviews

Download ATPM 8.11

Choose a format:

The User Strikes Back

by Ken Gruberman, kgruberman@atpm.com

Put Your E-Mail on a Diet

We hear it on the news every day: after 30 years of dieting and eating healthier foods, Americans are fatter than ever. Our pets are overweight as well. And now, it’s time to face the awful truth…even our e-mail has gotten too big for its britches.

Most of us don’t realize the volume of e-mail each of us deals with each day. Within a few months to a year, it’s not uncommon to have sent and received thousands—or even tens of thousands—of messages. The problem is that most e-mail programs were never designed to hold such voluminous amounts of mail. Just as hard drives get fragmented and, as a result, slow down over time, so too does e-mail software.

For this article, I’m going to focus on Microsoft’s popular e-mail clients, Outlook Express and Entourage, and how to get them back into fighting shape. (OE is Microsoft’s free e-mail client for Mac OS 8 and 9, while Entourage is part of Microsoft Office 2001 and also Office for OS X.) Of course, there are many other e-mail programs out there, such as Eudora, Mailsmith and even Netscape mail, but most people I know use one or the other of the Microsoft products. If possible, I’ll try to cover slim-down techniques for these other programs in a future article.

Let me say something about Entourage before I continue: if you’re using Outlook Express and you like it, but either wish it could do more or are moving to OS X (where Outlook Express no longer exists), you should check out Entourage. It has schedules, a calendar, notes and to-do’s that are all linked to your e-mail, a beefy address book that’s printable, and Entourage can even HotSync with a Palm organizer. Unfortunately, Microsoft doesn’t sell Entourage by itself (although that may change in the near future), but on occasion there are bundle deals that come along. Last year, you could purchase just Word and Entourage for $149; that deal is over but there is another promotion currently underway that gets you Office X for only $199 instead of $499. The catch? You have to buy a new Mac at the same time! This promotion will last until January 7th, the date of Steve Jobs’ next keynote at Macworld San Francisco.

Bob the Rebuilder

OE and Entourage, like many other e-mail programs, keep all your sent, received and filed messages in a single file, or related files. These are typically referred to as the “mail database.” The files in question are called “Database,” “Database Cache,” and “Messages,” and they can be found in the following places in Mac OS 9.x:

1. If you do not have a folder called “Documents” at the root level of your hard drive and use OE, you will find them in a folder called “Identities” inside your Outlook Express folder itself.

2. If you do have a Documents folder at the root level of your hard drive, or if you’re using Entourage, you will find a folder in the Documents folder called “Microsoft User Data.” (If you didn’t have a Documents folder before, Entourage will create it during first run.) Inside the Microsoft User Data folder you’ll find the aforementioned Identities folder for OE and an Office 2001 Identities folder for Entourage. Inside each of these folders you will find a sub-folder called “Main Identity” (unless you renamed the Main Identity to something else) and inside of that folder you’ll find the mail database files.

In Mac OS X, OE is out of the picture; the Microsoft User Data folder will be found in the Documents folder that is found inside the Home folder (the one with the “Home” icon on it that is created for each user).

The other files in these “Identities” folders deal with other functions of the e-mail software such as rules, mailing lists, schedules, and the like and are not changed in any way during a typical rebuild: only the mail database files are involved.

OE and Entourage use mail database files because they were inherited from one of the best e-mail clients of all time: Claris Emailer. That’s right—after its demise, the same people who created and refined Claris Emailer were hired by the Macintosh Business Unit of Microsoft several years ago to work the same magic for Outlook Express, then Entourage. This also explains why both products play so well with AppleScript: David Cortright, a guiding force behind all three products, is an AppleScript maven of the highest order.

Exercising Your Option (key)

Claris Emailer had a technique for compacting and rebuilding its mail database files, and OE and Entourage use the very same method. The funny thing is, few people seemed to know about this when Emailer was at its peak of popularity, and even fewer know about the feature today in OE and Entourage. So listen up, because here’s something everyone should know and hardly anyone does: to rebuild the mail database in all three programs (for those of you who still are using Emailer), hold down the Option key while launching the program.

The easy way to do this is to launch the program first, then immediately hold down the option key until you see a dialog come up (see Figure 1) that asks if you want to do a “typical” rebuild or an “advanced” rebuild. I am happy to tell you that in all the years I’ve been using these products, I’ve never had to pick the Advanced option; the only reason to do so is if the Typical rebuild fails to solve whatever problems you’re having.

strikes-01-rebuild-dialog

Figure 1: Holding down the Option key while launching Outlook Express or Entourage reveals an important, but mostly unknown, feature.

The truth is, I rebuild my mail database every few months to prevent problems from happening. The concept is a simple one: over time, you create and send messages, read and store messages, and deal with attachments. Some you save, while others you leave with the messages they came in on. The database files inflate a bit, just like a balloon, every time you get and send e-mail.

And of course, you’re going to (hopefully) delete a lot of messages after you read them, such as spam, stupid jokes you’ve already received a dozen times, and the like. But the database just keeps expanding from all the “hot air” from the daily torrent of e-mail. And after awhile, those files get awfully big. You can tell if this is happening to you by how long it take for your e-mail program to open and then display the main browser window. What once may have taken a few seconds could now be taking 30 seconds or more. When this happens it’s time for a rebuild, which not only compacts the message database, but also your contacts as well (see Figure 2).

strikes-02-rebuild-prog

Figure 2: If you have a lot of e-mail to process, consider giving your e-mail program lots of RAM before starting the rebuild.

When the rebuild is finished, you’ll be told that it was successful and that it is okay to throw away the previous files which have now have the word “old” in front of them. (See Figure 3). Rebuilds can yield results that range from the subtle to the dramatic. For this article I did a rebuild and, as you can see in Figure 4, the results are pretty impressive: an almost two-thirds drop in file size. But I’ve seen even better results, such as mail database files that were 400, 500 or even 800 MB in size dropping down to under 50 MB. In any case, after the rebuild is complete the main e-mail browser window will appear, and the next time you launch your e-mail program it should appear once again with the same pep and vigor it did when you first started using it.

strikes-03-rebuild-done

Figure 3: Success at last!

strikes-04-rebuild-results

Figure 4: Here we see the mail database files drastically reduced in size after the rebuild. Your mileage may vary.

Trouble in E-Mail Land

If you’re getting error messages upon launch, or if your e-mail program is quitting suddenly and unexpectedly, this may also be a sign that either a typical or advanced rebuild is in order. In OS 9, you should also throw out all the Microsoft libraries so they can be replaced; in OS X, you don’t need such a complicated routine but rather can simply toss the Microsoft “plist” files. Here’s what I do when I am asked to deal with a recalcitrant Microsoft e-mail program in OS 9:

  • Go to the System Folder’s “Extensions” folder and throw away any extension that starts with the word “Microsoft” or the letters “MS”
  • Throw away the “MS Preference Panels” folder in the System Folder
  • If a control panel is present called “Configuration Manager,” I’ll toss that one too.
  • Remove either the Outlook Express preference from the System Folder’s “Preferences” folder, or the Entourage preferences found in the “Microsoft” folder inside the Preferences folder.
  • Finally, I’ll restart, launch the program in question, and hold down the Option key until the rebuild dialog comes up, then do a typical rebuild.

I have found that this technique solves 95% of all OE and Entourage problems. If the problems remain, you should try the Advanced rebuild. And if that doesn’t work, consider reinstalling the software itself and also restoring the mail database files from a backup. (You do backup, don’t you?) Also, you could be having a font problem. Historically, Microsoft software programs, including Office components and Internet Explorer, are sensitive to either too many fonts, or things like duplicate fonts or font corruption. The good news is that you can fix all these things (and much more) by simply using Morrison SoftDesign’s invaluable FontDoctor 5.1.2 for both OS 9 and X.

Oh yes, one more simple but not-so-obvious thing to try if you’re having problems: in OS 9 give the program in question more RAM. Click once on the icon of the actual program itself and hit Command-I; try doubling or even tripling the preferred amount of RAM. In OS X, of course, this is no longer necessary as the OS itself figures out how much RAM things need. This is one of the reasons I like OS X!

Tons of Text

Even with rebuilding, the time may come when you have too many messages for your e-mail program to handle in an effective way. If you’ve got hundreds of folders containing thousands of messages in each one (trust me, I’ve seen it), the time has come for you to consider archiving them and getting them out of your e-mail program for good.

While it’s true that many e-mail programs use database files to keep track of everything, a real database program is far more efficient at handling this kind of thing. And the one people seem to use the most is FileMaker Pro. If you have a copy of FileMaker on your hard disk, that means you’ve also got a very cool way to archive your e-mail.

What bridges the gap between your e-mail program and FileMaker is a nifty shareware program called Entourage Email Archive by Softhing; it’s available for both OS 9 and OS X. (They also make a version for Outlook Express that is virtually identical.) According to Softhing’s site, here’s what you can do with it:

Entourage e-mail Archive (EEA) is a simple and fast utility for archiving e-mails and attachments you have received or sent using Microsoft Entourage. Entourage e-mail Archive can archive your e-mail in three different ways:

  • Archive e-mail and/or attachments in the Finder
    1—Archive e-mails + attachments
    2—Only e-mails
    3—Only attachments
  • Export or append e-mail in a text file
    4—Export new text file
    5—Append to text file
  • Export or append e-mail in tab-text format
    6—Export new tab-text file
    7—Append to tab-text file (for this function a freeware FileMaker Pro template is enclosed in OEA folder)

Settings 1,2,3 produce a Finder-structured-folders archive where e-mails and/or attachments are grouped by day. Settings 4 and 5 produce a long ‘paper trail’ file that can be viewed with a robust text editor like BBEdit, Text-Edit Plus or Microsoft Word. Settings 6 and 7 produce a tab-text file that can be imported into computer database programs like FileMaker Pro.”

It’s those last two options that are my favorites. If you don’t own a database program such as FileMaker (remember there’s a smaller version of it inside AppleWorks), archiving each message as a text file, or in one long text file, may be enough for you. If you choose this option, you can use Sherlock to either find messages by title, or, if you have indexing turned on, by keywords or phrases inside of messages. You can even use Sherlock to find messages you might want to restore to either OE or Entourage: to do so, just drag and drop them into the main browser window of either program. Neat!

EEA features a dizzying array of options for naming and saving archived text files and the folders they’ll reside in. For more details on this, check out the concise and easy-to-understand HTML-based manual on Softhing’s site, or download a copy for yourself.

Text files are nice, but the database option is a better idea, because then you can really put the power of FileMaker to work. Previous versions of utilities such as EEA required the installation and use of obscure AppleScript add-ons, but no longer. Archiving to FileMaker now involves the exporting of selected messages or folders to a tab-delimited text file, which is then properly imported by the EEA-supplied FileMaker Pro template, which requires FileMaker Pro 5 or higher.

Using EEA is easy and enjoyable. You first launch either OE or Entourage depending on which you’re using; for best results, make sure either program has enough RAM assigned to it under OS 9. When first launching either EEA or OEA on top of your e-mail program, you’ll be greeted with the opening screen (see Figure 5.) From here on out, we’ll stick with EEA and Entourage, but rest assured the same also applies for OE and OEA.

strikes-05-eea-opening

Figure 5: Make sure to launch OE or Entourage first before running the EEA e-mail archiving application.

If this is the first time you’re using EEA, you’ll want to go to the File menu and click on Preferences. Decide first what kind of archiving you’d like to do, then click from tab to tab and make the appropriate selections (see Figure 6). Here, I’ve told EEA I want a new, tab-delimited export that will be imported into FileMaker. I could also say I want the new data appended to a pre-existing text file. EEA conveniently lets me pick the creator of the resulting text file; in this case, I’ve picked Tex-Edit, my favorite shareware text editor. This is important because, in OS 9, SimpleText has a 32K file size limit. Bets are, I’m going to exceed that even with one folder full of imported messages.

After setting Preferences and hitting OK, you’ll be returned to the main screen. For most archives, simply click on the Include Subfolders checkbox, then click on the icon on the left of the pop-up menu to have EEA gather all the folder information from Entourage. You’ll then see all of your Entourage folder names displayed; select the one that has messages you want to export (see Figure 7). You may only pick one folder at a time to export, but if you would rather export every message in every folder, click on the All Structure checkbox instead.

strikes-06-eea-preferences

Figure 6: Before starting your first archive session in EEA, make sure to visit the Preferences menu and tell EEA exactly how you want your archive done.

strikes-07-eea-folders

Figure 7: Like magic, here’s our selection of Entourage folders now appearing inside of EEA.

Clicking on the “Destination” icon below the Entourage icon lets you pick where you want your text file saved, and how you want it named. If you’ve decided on a straight Finder-based individual message archive, this is where you’ll decide where the files and folders should be saved. Finally, hit the “Process” button to get underway (see Figure 8).

strikes-08-eea-processing

Figure 8: Once underway, EEA shows you its progress as it exports every message in a given folder to a master text file.

After the export is complete, it’s time to open the FileMaker Pro template provided by EEA and import the data you just exported from Entourage. The first time you import a tab-delimited text file into EEA, you’ll be asked to match up the fields from the Export file with the fields inside the EEA template. They should already match up (see Figure 9), but if they don’t you can simply drag the fields on the right side (the template side) so they match up with the fields on the left side (the text file side). This also gives you some latitude in what you decide to import: if you want header information, go ahead and click on “Headers” so that the arrow appears.

strikes-09-fm-import

Figure 9: The last step in the Archiving process involves importing the tab-delimited text file you just created into FileMaker. Make sure the fields on the right match the fields on the left; you’ll only need to do this once.

strikes-10-finished-import

Figure 10: Here’s what the EEA FileMaker template looks like. The information in every field is editable.

If you don’t care about headers, leave them turned off. It’s up to you exactly how much information on each message you want imported. When you’re happy with the field names, click on “Import,” and quick as a wink, your e-mails will now reside within FileMaker. Here, you can very quickly search and sort to your heart’s content; copy, paste, or export messages as you see fit; and otherwise play with the data (see Figure 10 for a sample page).

When you’re satisfied that the e-mails in question have been imported properly, you may return to Entourage and delete the original e-mail messages from the folder in question and then go on to other folders, or quit both Entourage, EEA, and FileMaker. Keep in mind that e-mail attachments are not archived to FileMaker…only the reference to them is. If you delete a message out of Entourage that has an attachment, that attachment will also be deleted. Therefore, if you decide you need the attachment, you’d better save it to disk first; you could, if you wanted to, create a folder and call it “Archived Attachments.” Then when you see a reference to one inside the EEA FileMaker template, you’ll know where to find it. Of course, not every attachment is worth keeping, so perhaps having the reference to it is enough for you. Also keep in mind that the EEA FileMaker database file is completely editable, so you can add or remove information from selected entries at will.

Now It’s Your Turn

Compacting your e-mail databases will make your e-mail program snappy and responsive again, while archiving will give you greater power over your old e-mail, as well as peace of mind. That’s important, because should the worst happen and your entire e-mail database is somehow destroyed or corrupted and there is no backup, at least you’ll have whatever you’ve saved in a virtually indestructible FileMaker file.

Now it’s time to hit the gym, and to put your e-mail program on a diet!

Also in This Series

Reader Comments (7)

Ken Gruberman (ATPM Staff) · November 4, 2002 - 23:44 EST #1
I have a feeling something is coming, although it's not here yet. Mail.app still has its own share of quirks to work out -- especially when it comes to file attachments -- and I wouldn't be surprised if a maintenance update is issued soon.

As to an archiving solution, I promise to keep an ear to the ground.
Ken Gruberman (ATPM Staff) · November 5, 2002 - 00:11 EST #2
Well, that didn't take long! Here's an entry in the www.macintouch.com "Jaguar" series of Reader Reports:
-----------------------------------------------------
Date: Wed, 30 Oct 2002 19:11:00 -0800
From: Charles McQuary 
Subject: Archiving "Mail"

I don't know if anyone has mentioned this yet. 

There was a lot of talk in reader reports re: the
inability to properly back up/save the mail that one
received using Jaguar's Mail.app. 

I just opened the updated Casady & Greene's
iDataPro and was able to import every mailbox from
Mail into it's own freeform database! It couldn't have
been easier. Now I can move those files to my backup
disk and not worry. 

Another cool thing: I bought Info Genie lord knows how
many years ago. 

All I needed was the serial number, and I was able to
buy iDataPro for the upgrade price. C&G knows
how to keep customers.
-----------------------------------------------------
I'm also an iData Pro user, and recommend it highly. I used it when it was called InfoGenie, and before THAT when it was called QuickDEX. You can download a demo, or buy it outright, by visiting casadyg.com. It's very reasonably priced.
Charlie · November 9, 2002 - 01:07 EST #3
Hi Ken. Excellent article. Thanks!

Here is what I want to be able to do, and I am guessing it should be "simple."

  1. I have a file for contacts in Filemaker. I want to be able to export new contacts into Entourage. Can you tell me a simple way to do this?

  2. I want to be able to send e-mails from Filemaker. I want to be able to send to select groups that I sort for in Filemaker. Can you be of help?


Thanks for your time and consideration.

Regards,

Charlie
Ken Gruberman (ATPM Staff) · November 9, 2002 - 11:17 EST #4
Charlie,

Answering question 1: the only way I know of is to export out of FileMaker as a text file and then import into Entourage using the same "field matching" technique as described in my article, even though it was for IMPORTING into FileMaker. Entourage uses the same field mapping function that FileMaker does.

Answering question 2: a shareware item called .mail.it 2 ought to fit the bill. It promises "full e-mail functionality inside FileMaker!"
anonymous · November 21, 2002 - 03:23 EST #5
The new OS X 10.2 Mail.app did a superior job of importing my Windows e-mail. Entourage has failed...so far.

Can you assist me?
Lee Bennett (ATPM Staff) · November 21, 2002 - 08:51 EST #6
J. Schneider - we can try, but you must provide more information. Try dropping an e-mail to help@atpm.com and describe your problem and the steps you took. Things to include would be what e-mail software you were trying to import from, how you were trying to retrieve the messages from your PC, and describe what Entourage did more completely than that it just "failed."
B.G. Meyer · August 10, 2006 - 22:04 EST #7
Hi Ken, I see this article may be a few years old but it was right near the top of a google search!

I'm in a real pickle and wondered if you have any ideas...

My trusty blueberry's ibook ethernet went "ka-put" so I'm temporarily back on dial-up until I can get the new white ibook set up and everything transferred over (via 100mb zip drive/disks since I can't use the blueberry ethernet port and it doesn't have firewire, of course).

My problem is: I have compacted my Biggest "Identity" (of 8 email addy's I have with earthlink) in my OE 5.0.6 on the blueberry ibook. Yet the main "database" is STILL at 111.1 mb. Since formatted 100mb zip disks really only hold about 94mb, how how how can I get my email contents for this particular email addy/Identity over to the new white ibook?

The "Messages" for that email addy, after compacting and "rebuilding" are only at 29.8 mb, which is just about what they were BEFORE compacting/rebuilding, so that's not a problem, they WILL fit on a zip disk. But it's that main DATABASE that is over the size of a zip disk.

I do not have that FilemakerPro or any archiving software.

I'm wondering if I slide all my folders/emails in that one email addy over to a hard-drive folder, will it make the DATABASE shrink in size? Other than that, I don't have a clue what I can do... :-/

Thanks for any tips/ideas!
Best regards,
Gail

Add A Comment





 E-mail me new comments on this article