What happens now that phones can see?

Photos give us a window to the world -- our own experiences can only show us so much. Photos give us a richness of experience and perspective that we would otherwise be devoid of. The sudden existence of billions of smart phones, which take trillions of photos and videos per year, has given us the biggest increase we’ve seen yet in rich user data -- and now we have the technology to start to make sense of that data.

With this new technology, it’s now possible to analyze photos for myriad applications. For example, at Jetpac we use 100s of millions of travel photos shared on Instagram to develop the Jetpac City Guides app, and do object recognition on the photos to be able to recommend places such as restaurants with patios (from the blue skies in the photos), hipster bars (by the unusually high number of mustaches we spot in photos), and the best coffee shops (from the highest proportion of latte art in photos). We've also now developed deep learning AI technology to broaden our ability to be able to recognize all objects in photos. Putting deep learning on the iPhone (Spotter by Jetpac) means we can now recognize many objects from the iPhone's real-time video feed, locally on the phone. Having minimized the technology significantly, we also have room to allow you to train the phone to recognize a specific object (Deep Belief by Jetpac).

It's worth taking a moment to marvel that computer vision algorithms have come a long way, and what was previously the province of super computers, is now being done on your smart phone, in milliseconds, in the palm of your hand.

Now, it's no human intellect, but for the first time, the phone can "see" -- and it can also handle the massive amount of data it consumes. It can not only place a phone call, send a text or like your grandma’s status update, but phones can also make sense of what they are looking at by recognizing objects and scenes. Check out the amazing capabilities this gives us:

Photo/video tagging and search

Have you ever scrolled through your photos on your phone, looking for a particular one? Every wanted to be able to search for "the one with me and my sister next to the tree by the beach"? Instead of scrolling through endless selfies and photos of cats, you can now use tags for all your photos, describing what’s in them, and enable a targeted search. Film and TV companies will now be able to search through their movie archives for the frame with the shark tornado in it. Private investigators will be able to remember whether it was the ballroom they saw the candlestick in.

Automatic camera filters/settings

Alongside the existing smartphone camera sensors and settings, imagine having a particular set of default settings if the phone recognizes that the scene is a ski slope, a beach or a nightclub? With real-time recognition of the video stream coming through the camera before the shutter button is pressed, the phone would be ready for the conditions before you.

Automatic home security alerts

Cameras on phones can certainly take killer photos, but they can also do much more than that. They can take your pulse, give you an impromptu flashlight and they can even protect your home. Want to lock your cat flap against raccoons, or even cats other than yours? With Spotter and Deep Belief on even lower-end devices like the Raspberry Pi, you can avoid having to duct tape your iPhone to the back door, and throw a cheap device with a battery at the problem of recognizing which animal this way comes. It wouldn't even need to be networked.

Mobile Search and Rescue

Phones don’t have a corner on the market of device visibility. Internet of Things devices, robots and drones also share similar vision-wielding capacities. People who have trouble moving around their houses could send a robot to look for their car keys. Drones could look for that missing cow in a farmer's field. As some media outlets have shown ad nauseam, satellites could figure out with parts of the ocean had possible matches for missing airplane debris.

Hard-to-identify objects

Ever looked at food in a foreign market and wondered what it is? Ever wanted to tell your kids what kind of bird is chirping from the tree? Want to know the breed of dog that is looking for a home at the dog shelter? Ever wondered if a plant that your child just rolled over was poison oak? Well with the power of object recognition, you could figure out if your child needs some calamine lotion.

We live in a visual world, and the amount of image data we have access to is exploding. Now that the phone can see, and technology like Jetpac's can help us make sense of image data, we're going to be able to make a lot more sense of our world.

Let me know if you have any suggestions for other applications of the technology.

Photo Credits: Maksim Kabakou/Shutterstock

Julian Green is the co-founder and CEO of Jetpac, the company behind the Webby Award-winning Jetpac City Guides, Spotter by Jetpac and Deep Belief by Jetpac apps. He is a serial entrepreneur who also co-founded Houzz and is a leading authority on mobile, social and ecommerce. Follow him on Twitter: @juliangreensf.