The standard allocation of interrupts is:

IRQ(interrupt)4 for COM1 & COM3
IRQ3 for COM2 & COM4

Assuming the driver is the correct one for your modem, check which port the modem is on, probably COM4 or COM3. Then, assuming you are using WIN9x, in Device Manager, look at Properties, Resources for the COM ports, and make sure the IRQ used for the port the modem is on is appropriate. If it seems to be wrong, the COM (serial) port settings may be wrong in your CMOS setup.
If the modem is installed on COM2, you will need to disable your motherboard COM2 setting in the CMOS setup, because they will fight for the resources. This may be your problem. Also, if you have a serial mouse on COM1, the modem should be on COM2 or COM4 to avoid a clash with the mouse.

Hope this helps.