I used to fell that way until I came across a book called Art and Physics, by Leeonard Shlain.
In it Shlain traces the history of art in relation to physics and the world view of the artists of the time.
Beginning with classical Greece he relates art forms to the current understanding of physics.
The Greek artists positioned their figures in a linear manner which relied on the horizon. Transcending the world of myth they began to incorporate space and proportion as a real thing and this is shown in their art. The world was made up of real measurable things and logic and literacy prevailed.
With the fall of civilization and the rise of the dark ages the view of real things began to change and the world became a flat Earth with a real Heaven above. Classical art was destroyed as only art related to Christ was seen to be valuable. Linear art disappeared as time and space became irrelevant to the views of salvation. Figures in paintings became simpler as the written word almost disappeared and they were unrelated in space and time, a single figure could be repeated many times in the same painting showing different divine aspects. Logic in art was abandoned.
During the late middle ages literacy returned with the printing press and with it the rules of logic again reestablished themselves in art. Perspective was developed and with it art began to represent one instant of time. Like a photograph art began to represent the real world as it appears in this instance. This form of art prevailed until the modern era.
With the invention of the camera art again began to change as it provided an exact copy of the world without being influenced by the artist's perception. What the camera saw and the artist painted were not the same. The camera allowed for a precise measuring of space and also provided an exact moment of time. Artists panicked as they saw themselves being put out of work by a machine that could offer a perfect copy of the world.
At the same time non-Euclid geometry was born and curved space became a reality. Space, Time, and Light were about to change again.
Monet, Manet, and Cezanne began to use light and perspective in new ways. Objects were arranged without regard to space, multiple point perspective was introduced, and light was allowed to enter a picture from different angles. All following the views of the new physics.
With the revelations of Einstein the art world blew apart. Classical reality was shattered and artist began exploring multiple time frames, and point of view as in Picasso's works that show the same figure from many different angles and times all fused into one body. Relativity entered art. It wasn't good enough to capture a single moment of time, art had to express all the different aspects of relativity.
Objects became distorted as they do at different relative speeds, colors took on importance as they were seen not a aspects of objects but as objects themselves. Cubism presented the front, back, sides, top and bottom of objects without requiring the viewed to move through time and space.
Futurist destroyed time depicting what was to come along with what was here and now reflecting the distortion and relativity of time found by Eisenstein.
Freud entered the picture and introduced the concept of the unconscious. Together with Einstein's destruction of time and space Freud's ideas entered the art world in the form of Surrealism.
In Dali's Persistence of Memory we see time melting in a subconscious view of the world.
And so it goes this book really helped me understand what's going on in modern art. There's much more but this is getting long.