{"id":950,"date":"2020-11-07T14:00:04","date_gmt":"2020-11-07T14:00:04","guid":{"rendered":"https:\/\/thenextweb.com\/?p=1326883"},"modified":"2020-11-07T14:00:04","modified_gmt":"2020-11-07T14:00:04","slug":"how-to-build-an-ai-stylist-inspired-by-outfits-on-instagram","status":"publish","type":"post","link":"https:\/\/www.londonchiropracter.com\/?p=950","title":{"rendered":"How to build an AI stylist inspired by outfits on Instagram"},"content":{"rendered":"\n

Last year, in a universe where it still made sense to owns pants, I decided to hire a personal stylist.<\/p>\n

In our first appointment, the stylist came to my apartment and took pictures of every clothing item I owned.<\/p>\n

In our second appointment, she met me at Nordstrom\u2019s, where she asked me to try on a $400 casual dress, a $700 blazer, and $300 sneakers. (I never thought to ask if she worked on commission.)<\/p>\n

But only after our third and final appointment, when she finally sent me a folder full of curated \u201clooks\u201d made from my new and old clothing items, did it finally click: I\u2019d just blown a <\/span>lot<\/em> of money.<\/p>\n

I had a suspicion we were on different pages when, as we walked through the shoes section at Nordstrom, the stylist said, \u201cThe problem with you people in tech is that you\u2019re always looking for some kind of theory or strategy or formula for fashion.<\/em> <\/span>But there is no formula\u2013it\u2019s about <\/span>taste<\/em>.\u201d<\/em><\/p>\n

Pfffft. We\u2019ll see about <\/span>that<\/em>!<\/p>\n

I returned the pricey clothing and decided to build my own (cheaper!) AI-powered stylist. In this post, I\u2019ll show you how you can, too.<\/p>\n

Want to see a video version of this post? Check out it here<\/a>.<\/em><\/p>\n

My AI Stylist was half based on this smart closet from the movie <\/span>Clueless<\/em>:<\/p>\n

and half based on the idea that one way to dress fashionably is to copy fashionable people. Particularly, fashionable people on Instagram.<\/p>\n

The app pulls in the Instagram feeds of a bunch of fashion \u201cinfluencers\u201d on Instagram and combines them with pictures of clothing you already own to recommend you outfits. Here\u2019s what it looks like:<\/p>\n

\"Screenshot<\/p>\n

(You can also check out the live app here<\/a>.)<\/p>\n

On the left pane\u2013the closet screen\u2013you can see all the clothing items I already own. On the right pane, you\u2019ll see a list of Instagram accounts I follow for inspiration. In the middle pane (the main screen), you can see the actual outfit recommendations the AI made for me. The Instagram inspiration picture is at the top, and items for my closet are shown below:<\/p>\n

\"Screenshot<\/p>\n

Here my style muse is Laura Medalia, an inspiring software developer who\u2019s <\/span>@codergirl_<\/a> <\/span>on Instagram (make sure to follow her for fashion and tips for working in tech!).<\/p>\n

The whole app took me about a month to build and cost ~$7.00 in Google Cloud credits (more on pricing later). Let\u2019s dive in.<\/p>\n

The architecture<\/h2>\n

I built this app using a combination of <\/span>Google Cloud Storage<\/a>, <\/span>Firebase<\/a>, and Cloud Functions for the backend, <\/span>React<\/a> <\/span>for the frontend, and the <\/span>Google Cloud Vision API<\/a> <\/span>for the ML bits. I divided the architecture into two bits.<\/p>\n

First, there\u2019s the <\/span>batch process<\/em>, which runs every hour (or however frequently you like) in the Cloud:<\/p>\n

\"Diagram<\/p>\n

\u201cBatch process\u201d is just a fancy way of saying that I wrote a Python script which runs on a scheduled interval (more on that later). The process:<\/p>\n

    \n
  1. Pulls photos from social media<\/li>\n
  2. Uses the Vision API\u2019s <\/span>Product Search<\/a> feature<\/span> to find similar items in my closet<\/li>\n
  3. Scores the matches (i.e. of all the social media pictures, which can I most accurately recreate given clothing in my closet?)<\/li>\n
  4. Writes the matches to <\/span>Firestore<\/a>\n<\/li>\n<\/ol>\n

    This is really the beefy part of the app, where all the machine learning magic happens. The process makes outfit recommendations and writes them to Firestore, which is my favorite ever lightweight database for app development (I use it in almost all my projects).<\/p>\n

    The actual <\/span>app<\/em> (in this case, just a responsive web app) is simple: it just reads the outfit recommendations from Firestore and displays them in a pretty interface:<\/p>\n

    \"Architecture<\/p>\n

    Let\u2019s take a look!<\/p>\n

    Ideally, I wanted my app to pull pictures from Instagram automatically, based on which accounts I told it to follow. Unfortunately, Instagram doesn\u2019t have an API (and using a <\/span>scraper<\/a> <\/span>would violate their TOS). So I specifically asked Laura for permission to use her photos. I downloaded them to my computer and then uploaded them to a <\/span>Google Cloud Storage bucket<\/a>:<\/p>\n

    \n
    \n
    # Create a cloud storage bucket<\/span> gsutil mb gs:\/\/inspo-pics-bucket # Upload inspiration pics <\/span> gsutil cp <\/span>path\/to\/inspo\/pics\/*.jpg<\/span> gs:\/\/inspo-pics-bucket <\/code><\/pre>\n<\/div>\n<\/div>\n
    Filtering for fashion pictures<\/h2>\n
    I like Laura\u2019s account for inspiration because she usually posts pictures of herself in head-to-toe outfits (shoes included). But some pics on her account are more like this:<\/p>\n
    \"Dog<\/p>\n
    Adorable, yes, but I can\u2019t personally pull off the dressed-in-only-a-collar look. So I needed some way of knowing which pictures contained outfits (worn by people) and which didn\u2019t.<\/p>\n
    For that, I turned to my trusty steed, the Google Cloud Vision API<\/a> (which I use in many different ways for this project). First, I used its classification feature, which assigns labels to an image. Here\u2019s the labels it gives me for a picture of myself, trying to pose as an influencer:<\/p>\n
    \"Screenshot<\/p>\n
    The labels are ranked by how confident the model is that they\u2019re relevant to the picture. Notice there\u2019s one label called \u201cFashion\u201d (confidence 90%). To filter Laura\u2019s pictures, I labeled them all with the Vision API and removed any image that didn\u2019t get a \u201cFashion\u201d label. Here\u2019s the code:<\/p>\n
    <\/figure>\n<\/p>\n
    If you want the full code, check it out <\/span>here<\/a>.<\/p>\n
    Digitizing my closet<\/h2>\n
    Now the goal is to have my app look at Laura\u2019s fashion photos and recommend me items in my closet I can use to recreate them. For that, I had to take a pictures of item of clothing I owned, which would have been a pain except I happen to have a very lean closet.<\/p>\n
    I hung each item up on my mannequin and snapped a pic.<\/p>\n
    \"gif<\/p>\n
    Using the vision product search API<\/h2>\n
    Once I had all of my fashion inspiration pictures and my closet pictures, I was ready to start making outfit recommendations using the <\/span>Google Vision Product Search API<\/a>.<\/p>\n
    This API is designed to power features like \u201csimilar product search.\u201d Here\u2019s an example from the Pinterest app:<\/p>\n
    \"Screenshot<\/p>\n
    IKEA also built a nice demo that allows customers to search their products via images with this kind of tech:<\/p>\n
    \"Video<\/p>\n
    I\u2019m going to use the Product Search API in a similar way, but instead of connecting a product catalog, I\u2019ll use my own wardrobe, and instead of recommend similar individual <\/span>items<\/em>, I\u2019ll recommend entire outfits.<\/p>\n
    To use this API, you\u2019ll want to:<\/p>\n
      \n
    1. Uploading your closet photos to Cloud Storage<\/li>\n
    2. Create a new Product Set using the Product Search API<\/li>\n
    3. Create a new product for each item in your closet<\/li>\n
    4. Upload (multiple) pictures of those products<\/li>\n<\/ol>\n
      At first I attempted this using the <\/span>official Google Python client library<\/a>, but it was a bit clunky, so I ended up writing my own Python Product Search wrapper library, which you can find <\/span>here<\/a> <\/span>(on PyPi). Here\u2019s what it looks like in code:<\/p>\n
      <\/figure>\n<\/p>\n
      Note this wrapper library handles uploading photos to a Cloud Storage bucket automatically<\/strong>, so you can upload a new clothing item to your product set from a local image file:<\/p>\n
      <\/figure>\n<\/p>\n
      \n
      If you, dear reader, want to make your own product set from your own closet pics, I wrote a <\/span>Python script<\/a> <\/span>to help you make a product set from a folder on your desktop. Just:<\/p>\n
        \n
      1. Download the code from GitHub and navigate to the instafashion\/scripts folder:<\/li>\n<\/ol>\n
        <\/figure>\n<\/p>\n
          \n
        1. Create a new folder on your computer to store all your clothing items (mine\u2019s called <\/span>my_closet<\/code>):<\/li>\n<\/ol>\n
          <\/figure>\n<\/p>\n
            \n
          1. Create a new folder for each clothing item and put all of your pictures of that item in the folder:<\/li>\n<\/ol>\n
            \n
            \n
            <\/figure>\n<\/div>\n
            \n
            So in the gif above, all my black bomber pics are in a folder named <\/span>black_bomber_jacket<\/code>.<\/p>\n
            To use my script, you\u2019ll have to name your product folders using the following convention: <\/span>name_of_your_item_shoe<\/code> <\/span>where <\/span>shoe<\/code> <\/span>can be any of <\/span>[skirt, dress, jacket, top, shoe, shorts, scarf, pants]<\/code>.<\/p>\n
            <\/p>\n
              \n
            1. After creating your directory of product photos, you\u2019ll need to set up some config by editing the `.env_template` file:<\/li>\n<\/ol>\n
              <\/figure>\n<\/p>\n
              (Oh, by the way: you need to have a Google Cloud account to use this API! Once you do, you can create a new project and download a credentials file.)<\/p>\n
                \n
              1. Then install the relevant Python libraries and run the script <\/span>product_set_from_dir.py<\/code>:<\/li>\n<\/ol>\n
                <\/figure>\n<\/p>\n
                Phew, that was more steps than I thought!<\/p>\n
                When you run that Python script, <\/span>product_set_from_dir.py<\/code>, your clothing photos get uploaded to the cloud and then processed or \u201cindexed\u201d by the Product Search API. The indexing process can take up to 30 minutes, so go fly a kite or something.<\/p>\n
                Searching for similar items<\/h2>\n
                Once your product set is done indexing, you can start using it to search for similar items. Woohoo! \ud83c\udf8a<\/p>\n
                In code, just run:<\/p>\n
                <\/figure>\n<\/p>\n
                <\/figure>\n<\/p>\n
                The response contains lots of data, including which items were recognized in a the source photo (i.e. \u201cskirt\u201d, \u201ctop\u201d) and what items in your project set matched with them. The API also returns a \u201cscore\u201d field for each match which tells you how confident the the API is that an item in your product set matched the picture.<\/p>\n
                From matching items to marching outfits<\/h2>\n
                The Product Search API looks at an inspiration picture (in this case, Laura\u2019s fashion pics) and finds similar items in my wardrobe. But what I really want to do is put together whole <\/span>outfits<\/em>, which consist of a single top, a single pair of pants, a single set of shoes, etc. Sometimes the Product Search API doesn\u2019t return a logical outfit. For example, if Laura is wearing a long shirt that looks like it could <\/span>almost<\/em> <\/span>be a dress, the API might return both a similar shirt and dress in my wardrobe. To get around that, I had to write my own outfit logic algorithm to build an outfit from the Search API results:<\/p>\n
                \n
                <\/figure>\n<\/div>\n
                \n
                Scoring outfits<\/h2>\n
                Naturally, I couldn\u2019t recreate every one of Laura\u2019s outfits using only items in my limited wardrobe. So I decided my approach would be to look at the outfits I could most accurately recreate (using the confidence scores returned by the Product Search API) and create a \u201cscore\u201d to sort the recommended outfits.<\/p>\n
                Figuring out how to \u201cscore\u201d an outfit is a creative problem that has no single answer! Here are a couple of score functions I wrote. They give outfits containing items that have high confidence matches more gravitas, and give a bonus to outfits that matched more items in my closet:<\/p>\n
                \n
                <\/figure>\n<\/div>\n
                \n
                If you want to see all this code together working in action, check out <\/span>this Jupyter notebook<\/a>.<\/p>\n
                Putting it all together<\/h2>\n
                Once I had written all the logic for making outfits in a Python script, I ran the script and wrote all the results to Firestore. Firestore is a serverless database that\u2019s designed to be used easily in apps, so once I had all my outfit matches written there, it was easy to write a frontend around it that made everything look pretty. I decided to build a React web app, but you could just easily display this data in a Flutter or iOS or Android app!<\/p>\n
                And that\u2019s pretty much it! Take that, expensive stylist.<\/p>\n
                \n
                This article<\/a> was written by <\/span><\/i>Dale Markowitz<\/span><\/i><\/a>, an Applied AI Engineer at Google based in Austin, Texas, where she works on applying machine learning to new fields and industries. She also likes solving her own life problems with AI, and talks about it on YouTube.<\/span><\/i><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
                 Published November 7, 2020 \u2014 14:00 UTC <\/p>\n
                 Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
                Last year, in a universe where it still made sense to owns pants, I decided to hire a personal stylist. In our first appointment, the stylist came to my apartment and took…<\/p>\n","protected":false},"author":1,"featured_media":951,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts\/950"}],"collection":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=950"}],"version-history":[{"count":0,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts\/950\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/media\/951"}],"wp:attachment":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=950"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=950"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=950"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}