Most domains list more than one incoming Mail Exchangers (a.k.a. “MX hosts”). If you do so, then bear in mind that in order to have any effect, any SMTP time filtering you incorporate on the primary MX has to be incorporated on all the others as well. Otherwise, the sending host would simply sidestep filtering by retrying the mail delivery through your backup server(s).
If the backup server(s) are not under your control, ask yourself whether you need multiple MXs in the first place. In this situation, chances are that they serve only as redundant mail servers, and that they in turn forward the mail to your primary MX. If so, you probably don't need them. If your host happens to be down for a little while, that's OK -- well-behaved sender hosts will retry deliveries for several days before giving up [9].
A situation where you may need multiple MXs is to perform load balancing between several servers - i.e. if you receive so much mail that one machine alone could not handle it. In this case, see if you could offload some tasks (such as virus and spam scanners) to other machines, in order to reduce or eliminate this need.
Again, if you do decide to keep using several MXs, your backup servers need to be (at least) as restrictive as the primary server, lest filtering in the primary MX is useless.
See also the section on Greylisting for additional concerns related to multiple MX hosts.