Here's a stab at it. First keep in mind that MODAL dialogs come in a couple of flavors: (1) The type that blocks interaction with the entire system; and (2) The type that blocks interaction with some subset of the system (e.g. it blocks just one application on a multitasking system, often referred to as "App Modal" dialogs).
Guidelines:
1. Favor modeless dialogs over modal where possible, and "app" modal over "system" modal. This allows the user to interact with the system in the order which makes most sense to them.
2. Consider using modal dialogs in circumstances like these:
(A) When you need additional information in order to carry out a user request. (B) When allowing the user to set properties or preferences. (C) Progress dialogs. (D) Confirmation dialogs. (E) Informational dialogs (i.e. when you want to be 100% certain the user sees a message before doing anything else).
There's nothing about (A), (B),(C), or (D) that says these MUST be modal. The reason to consider making them modal is to reduce implementation complexity. In cases (A), (B), and (D) you throw up a dialog and you come out of it either with the info you need, or a "Cancel" message. In case (C), you limit user interaction with the system while it's busy doing something else and is in a potentially vulnerable/inconsistent state. These strategies reduce the potential number of paths through the code and the number of simultaneous states the software developer (and user) must manage. This results in fewer bugs.
In some cases the usability effect of modal dialogs is negligible or even positive: Many users won't mind being prompted for print preferences for example. Most users would prefer the dialog "Shut down the system now? [Yes][No]" be modal.
In terms of a practical rule of thumb, tell your developers to consider the code and UI complexity which results if a dialog is NOT modal. In some cases there is little added cost: A dialog which allows sorting or one which sets properties of an object can often just as easily be modeless. In other cases significant complexity is introduced. Developers will have to keep track of potentially several pending operations at once. For example, consider if in response to a "Print" command you put up a modeless "Print Preferences" dialog. You now have the possibility that the user will go back and say "Print" again, resulting in a 2nd "Print Preferences"dialog. The user could then go off and delete the object which was to have been printed. Is the added complexity of managing these contingencies worth it? That's the key to deciding modal versus non-modal.
I think this rule of thumb is way too "developer-centric" - design the best UI you can by understanding the user's task requirements, and don't think about programming complexity until you understand what you want to do. You will get a much better UI this way. My rule of thumb is: don't use modal dialogs unless you can really justify it through a proper task analysis, e.g., loss of data will occur with a probability of X, some percentage of users will be unable to continue the task, etc.
Yeah, I think it came out sounding more developer-centric than I meant.
My point was that modal dialogs usually don't benefit the user directly. There probably aren't many people who, after using a new software package, say to themselves: "You know that was a neat product... I only wish there were more MODAL DIALOGS in it!". So don't fool yourself. You're not usually doing it for the user's sake.
Basically you're tying the user's hands every time you use a modal dialog (by its very definition). In an ideal world you would never or seldom do this. So why do it?
The answer is usually that this keeps the user out of trouble at some delicate point. Beneath this superficial explanation (i.e. if you really own up to it) is usually something like "I'm putting up a modal dialog now because it's the only way I can think of to prevent the user from inadvertently and totally hosing himself in this situation...", or "I'm using a modal dialog here because I don't happen to have the two person-years of work it would take to identify and trap all possible situations which would arise if I DON'T make it modal..."
In practice modal dialogs are often benign because they are transitory enough, or the user takes them at face value, or occasionally appreciates them because they provide an unambiguous cue about what's required next. They're not the main point of the application. They're more like the plumbing.
If the user DOES mind, i.e. if it constrains the user in some important or undesirable way, then it's probably a bad idea, and you probably DO want to consider doing something else.