One of the things I like about photographing the details of a subject is that you can separate it from the reality of what and where it is. This can give you more freedom with your composition, as you are not tied to the exact appearance of something, such as its orientation. When photographing a leaf or a flower I may adjust the angle of the camera for a better flow of the lines in the composition. While this changes the angle of a stem or leaf edge, if it's a tight enough shot there's no reference to the angle of the surrounding leaves, stems and flowers. Therefore there is no visual reference to indicate something is askew. Of course it all depends on the subject. If there are flowers hanging off a stem and you adjust the angle of the image too severely then the viewer will know that something is "off" with the angle of the flowers. In these cases adjustments should be minimal. On the other hand if there are no such revealing visual details, creative freedom abounds.
When I have an image that displays as fully abstract I will rotate it as part of my editing process. I check out all four orientations and see how they strike me. The great thing about this approach is it's not limited to a particular type of subject. The image above is an arrangement of glass pieces in the wall of a church. With this tight of a shot we lose connection to which way is up or down.
Now let's explore the variations. Take a look at the other three orientations of the image. Notice how the eye moves differently through each image. Compare the flow, structure and balance of the shapes and colors. Notice that in all images the red rectangle in the corner draws the most attention. Do you have a favorite orientation?