I do not believe that the numbers are appropriate to consider that the effort was not worth it. First the majority of the visitors will be developers. Most of us would be accessing your site using our desktop or supped up laptops since most likely we would be developing, testing, or debugging something.
Second, responsive web design is focused on providing content in the context of what the user needs. So if I am in the situation that I need to reference your web site and I only have a phone with me, having the site utilize responsive design makes my life easier to read and digest. I hate going to a web site in which I can not read because I need to expand it.