While I'm taking a break from "do list" duties, here's my thought on Google and Apple phones caching GPS locations and then sending it back to Google and Apple, respectively.
You gave them permission to do it and you take advantage of it, so what's the big deal?
Many people use the applications that allow them to "check in" to locations where they can then get store specific discounts and/or use applications that let others know where they are, or use applications that let friends know their specific location when they are going to meet someplace and/or use GPS direction applications.
When I got the Droid Incredible, the first thing I did was turn of GPS location applications and disabled the picture geo-tagging ability. But, months later, I turned GPS back on to use Google Maps for driving directions. I matched the directions given by Google Maps to the directions given by my car's built in GPS capability. Google Maps was better AND provided visual indicators of backed up traffic along the route. The only way the near real-time traffic conditions update can be given is if your telephone is giving out GPS locations along the way and then on the back end system, the system is calculating average speed of your telephone and others in the area.
So, I don't see it as a big deal. If you don't like it, turn it off. It's always a good idea to turn off geo-tagging of pictures so no one can obtain the location of your activities. That reduces the risk of crimes against you or your family based on "Internet snooping".