I solved the problem!
You suggested the following:
If you use a PNG of that SVG, at say 100x100 resolution, since you tend to draw it much smaller, does that have the same problem? If that fixes it, is it worth switching to a PNG for you?
This did not help, however, I tried to take this approach further and created an SVG with width and height=16. It seems very odd, but this solved the problem. I suspected that those SVG will look ugly again if I make them very big now, but the actually look good, no matter how big I set them.
Conclusion: Using SVGs with high “width” and “height” values and scaling them down very much makes them look very ugly (not in firefox though). Maybe this information proves usefull to you and further development