Well, unfortunately they do not have to have access to your computer, or your email account. It's called "Spoofing." They could have harvested your email by many, many various means. The easiest way is by the way people forward emails with everyone's email address intact. The address can be harvested in many ways and by hacking into someone's computer that has your email address. You can have the most sophisticated security software in the world, and your email address can be spoofed. If you look at the headers in an email (NOT just the from, to, cc line, but the internal header) the only thing that points to you is your name and email address, but the reply address will be to the person that is spoofing your email.
Please do a search on "spoofing email" without the quotes for information.
From CERT:
"It is easy to spoof email because SMTP (Simple Mail Transfer Protocol) lacks authentication. If a site has configured the mail server to allow connections to the SMTP port, anyone can connect to the SMTP port of a site and (in accordance with that protocol) issue commands that will send email that appears to be from the address of the individual's choice; this can be a valid email address or a fictitious address that is correctly formatted."
Here is some information that explains
"Spoofing"