ChatGPT can See Images: 10 Best Ways to use the Multimodal Feature




chatgpt robots stock trading

Welcome to the future, where not only can ChatGPT talk, but it can also see! Mind-blowing, isn’t it?

Here are 10 super useful ways you can use the multimodal image recognition feature inside the ChatGPT chat window.

1. Language Translation

Imagine you’re traveling, and you come across a signboard that looks important, but alas! It’s in a language that seems as decipherable as the enigma code to you. We’ve all been there, haven’t we?

Strolling down the cobbled streets of an old European town, you stumble upon an ancient-looking signboard. It’s intriguing, it’s mysterious, it’s…in Italian. And your Italian, let’s face it, is as rusty as the signboard itself. What do you do? Take a snap and toss it to ChatGPT!

“Hey ChatGPT, I’ve got this signboard here, and I’m as lost as a cat in a dog show. Can you translate it for me and give me some context?”

sign board

This isn’t just about deciphering foreign menus so we don’t accidentally order snails (unless you’re into that sort of thing, no judgment here!). It’s about breaking barriers, making connections, and ensuring that language is no longer a roadblock, but a gateway.

Also, Imagine if you are a student or researcher and need to translate an entire page of text.

2. Healthcare Guidance

We all have those “uh-oh” moments when a new spot, bump, or rash decides to set up camp on our bodies. Rather than jumping down the WebMD rabbit hole – you know, the one that convinces you you’ve got a rare, unpronounceable disease from the 18th century – let’s hit up ChatGPT.

“Hey ChatGPT, woke up feeling like a human dot-to-dot. Can you take a look at this image of my skin and give me some preliminary advice?”

ChatGPT will give you the usual lecture that it’s not a doctor blah blah, but it can then certainly point you in the right direction, giving you a nudge before you consult the real doctors. You’ve got to love the future!

It’s all about that initial step, the “hey, should I be worried about this?” moment. Because who wants to spend hours in a waiting room only to be told it’s a bug bite? Not me!

IMPORTANT: just make sure you don’t include any faces in the image as ChatGPT has been restricted from analyzing images with human faces.

3. Educational Tutoring

We’ve all been there, staring at a math problem that seems to be written in hieroglyphics, or a historical map that might as well be a treasure map, because, where’s the gold? Aha! Snap a pic, shoot it over to ChatGPT, and watch the magic happen!

Note: Just make sure you have the Wolfram plugin enabled if you’re using it for math.

“ChatGPT, this equation has more letters and symbols than my grandma’s crosswords! Can you help me solve it? And what’s the deal with this map? Is X marking the spot?”

Isn’t learning supposed to be fun?

Useful for trading chart analysis too – as Twitter user @youraimarketer demonstrates below:

4. Fashion Recommendations

Ready to step out but your outfit’s screaming “Help!” louder than a horror movie victim?

We’ve all faced the closet conundrum. Stacks of clothes but nothing to wear. It’s like being hungry with a fridge full of food and still ordering takeout.

“Hey ChatGPT, about to hit the town and my outfit’s giving bland vibes. Can you take a peek and spice up my style?”

Who knew your grandma’s brooch and those forgotten boots could look so chic together?

5. Fitness Technique Correction

Flexing in front of the mirror but something feels off? Maybe your plank looks more like a saggy bridge than a sturdy board. Don’t sweat it (well, keep sweating, but you know what I mean).

You snap a pic of your form – no faces, we’re keeping it anonymous! ChatGPT doesn’t need to see your game face.

Check out my warrior pose. What tweaks do I need?

warrior pose yoga

In no time, ChatGPT is analyzing your form, making sure you’re not heading to Snap City. We get it, everyone’s fitness journey is unique. Whether you’re lifting, running, or getting down with downward dog, proper form is key!

6. Cooking Assistance

ChatGPT has already been incredible with recipes and cooking guidance, especially when you give it a list of ingredients. Now you don’t have stand in front of your fridge, eyes darting between a lonely carrot and a half-eaten block of cheese, wondering if there’s a culinary masterpiece lurking in there?

Snap a picture and ChatGPT can get to work.

“ChatGPT, here’s what I’ve got in my fridge. What are my meal options.”

fridge of food

ChatGPT turns those ingredients into a symphony of flavors.

7. Home Improvement Advice

Staring at your living room, feeling like it’s more “meh” than “yeah”? Dreaming of a space that screams “you” but stuck in a décor dilemma?

Snap a pic of your space and let ChatGPT be your interior design guru! Whether you’re in a cozy corner apartment or a sprawling suburban dwelling, it’s about to get a whole lot homier in there!

“Hey ChatGPT, here’s my living room. It feels a bit dull. Can you help me spice it up? I’m thinking maybe a Boho vibe, but I’m open to ideas. Oh, and can we find a spot for my vintage lamp?”

Before you know it, ChatGPT is dishing out advice like an HGTV star. Color palettes? Check! Furniture arrangement that flows? Double-check! Lighting that makes everything glow? Triple-check! It’s like having a home makeover show right in your living room?

How to use ChatGPT to 10x your Instagram account in 7 steps

Chatgptguide Instagram ebook

We teach you how to build your own super-useful prompts

View our Ebook, with a step-by-step
walkthrough to conquer Instagram using ChatGPT even if you are not tech-savvy

8. Wildlife and Pet Behavior Understanding

Ever watched your cat perform Cirque du Soleil stunts at 3 AM and wondered, “What’s up, Whiskers?” Or stumbled upon a creature during a hike and thought, “Who’s this little fella?” Snap a pic and ask away, because ChatGPT is about to become your wildlife whisperer and pet psychologist!

“Hey, ChatGPT! Spotted this bird in my backyard – never seen one like it before. Can you tell me what species it is and what it likes to eat? Oh, and here’s a pic of my dog, Max. He’s been digging around the garden lately – any idea why?”

ChatGPT is dropping knowledge bombs about species traits, dishing out fun facts, and serving up tailored tips to understand and care for your furry, feathered, or scaly friends!

9. Identifying Plant Species

Stumbling upon a mysterious leafy fellow in your garden, eyes squinting, scratching your head? Or, maybe you’ve got this potted buddy at home looking a bit, well, under the weather?

Take a picture and….

““Hey ChatGPT, just found this plant in my backyard, any idea what it is?”

Before you can say “chlorophyll”, ChatGPT dives into its botanical brain, spitting out species names and care instructions like a horticultural hip-hop artist. Whether it’s a Sunflower or a Sneezewort, you’re getting the 411 on suitable environments, watering rhythms, and maybe, just maybe, how to finally keep that basil alive!

10. OCR Feature

Imagine this – you’ve got a pile of old receipts, documents, and, oh, maybe a cryptic note or two. Instead of diving into deciphering that jumbled jungle, snap a pic, upload, and let ChatGPT do the heavy lifting!

“Can you read this receipt for me and put the input into a table. I will be sending more receipts. Add them to the table”

And here’s the kicker – got a test question that looks like it’s written in ancient runes? Snap it, send it, and watch ChatGPT turn into your personal tutor, breaking down the question, serving up the answers, all while making you feel like the genius! I pity teachers though.

FAQs about ChatGPT’s New Multimodal Image Feature

You’re traveling and snap a photo of a landmark. Now, you can have a live conversation with ChatGPT about it! Or, you’re at home pondering dinner; a quick pic of your fridge and pantry, and voila, dinner suggestions coming your way!

Plus and Enterprise users. Voice will be available on iOS and Android (just opt-in in your settings), and image capabilities will be available on all platforms.

Easy peasy! Tap the photo button to capture or choose an image. If you’re on iOS or Android, hit the plus button first. You can discuss multiple images or use our drawing tool to guide the conversation.

The image understanding is fueled by multimodal GPT-3.5 and GPT-4. These models apply their language reasoning skills to a wide range of images, including photographs, screenshots, and documents containing both text and images.

Vision-based models can sometimes misinterpret images, especially in high-stakes domains

ChatGPT has been working closely with Be My Eyes, an app for blind and low-vision people, to understand the uses and limitations of the new features. This collaboration has helped it make the feature both “useful and safe”.

No OpenAI has significantly limited ChatGPT’s ability to analyze and make direct statements about people in images to respect individuals’ privacy.

Yes. Users find it valuable to have general conversations about images, even if they contain people in the background. But remember, detailed analysis of individuals is a no-go!

Share this content

Latest posts