Ok, while the "Right to Privacy" is usually a code word for pro-abortion, lets take people at their word for this. Jay Bookman (Atlanta Constitution Journal) who I respect a lot wrote a piece supporting the finding of a right to privacy.
Amendment IV
"The enumeration in the Constitution of certain rights shall not be construed to deny or disparage others retained by the people."
Ok, I'm not a lawyer but I read that to say that you can't use the constitution to say that you cannot pick your nose because there is no specific right in the constitution to allow nose picking.
But I don't see that it says that therefore I have any other rights that I come up with under this umbrella. Let's take one that is not much of a reach - if the "right to privacy" means birth control must be legal (the original case that found this), then by the same argument, shouldn't it mean suicide is legal? In both cases it is an action by a person that primarily impacts that person, but does have an impact on others.
And by the same token, doesn't this mean that using any drug is a right that cannot be "deny-ed or disparaged"? Using Amendment IV as the basis, I don't see how it allows a right to abortion, but not to drug use. (Selling drugs is different - I'm talking about using.)