Nick Abouzeid🕴
@nickabouzeid · Social & Growth Intern @ProductHunt
Can someone speak to the technical aspects of how this store works? I don't quite understand how "machine learning" would be used to figure out exactly which bottle of soda I grab.
Are there cameras all over the store and/or are they tracking my phone through the store?
15sharetweet・
Michael
@bikebodenberger · Project Lead, A&G Labs
@nickabouzeid From looking at the video and website, it seems like they're using a combination of cameras and sensors to understand who you are and which product you're selecting, respectively. They don't mention bluetooth or other wireless tracking tech for the user, so my understanding is that cameras "watch" you as you navigate around the store. Simultaneously, computer vision places you at precise points in the store. The "sensor fusion" is probably integrated into each shelf, which identifies the product you select. The name suggests that it might not be a single type of sensor, but perhaps a series of sensors. I'd like to think they could be motion and RFID sensors like those used by shipping/logistics companies (which is essentially Amazon's core business). Machine learning ties it all together (where you are, what products you've selected, and what account to charge). These are all just assumptions based on the website and video. Would love to hear what other folks think.
12sharetweet・
Noah Kim
@wuss · Startup in Progress ▓▓▓▓░░░░░░░░ 30%
@bikebodenberger @nickabouzeid That's my assumption as well, but since it uses cameras, how would it handle a really busy store with a bunch of people all exactly 5' 10" tall? I'm assuming it could potentially fall back to it's other inputs, but the complexity of doing what it's doing would seem like it needs every piece of the triangulation to make an accurate charge.
Maybe the cameras are embedded in the shelves themselves (vs maybe in the ceilings) and it uses some sort of pattern/face recognition to identify you. There has to be some fringe cases where it could break though. Ex. when you're sitting there thinking about what to get, and you're in someone's way so they kind of duck underneath you and reach in front of you to get what they need. Or what if you get cold while you're shopping, and put a jacket and hat on?
My gut says this could be hacked by people pretty easily and they seem to be trusting the tech a lot, but then again, we have cars driving around by themselves, so maybe it all works?
4sharetweet・
Michael
@bikebodenberger · Project Lead, A&G Labs
@wuss @nickabouzeid Maybe I'm way off, but I'm thinkin' the cameras use facial recognition as an identifier. If they aren't using something like bluetooth, then there needs to be a unique identifier to tie you to your app. The baseline photos could be uploaded during app onboarding or maybe there's even a facial scan feature. And the more you use the store, the more data (images of you) Amazon feeds back into their neural net. Which is where the machine learning comes in. The store is learning more about you each time you use it, painting a clearer picture of you and your buying behavior.
upvotesharetweet・
Ethan Kravitz
@ethosventures · CTO, Overnight
@bikebodenberger @wuss @nickabouzeid They are absolutely using bluetooth, as well as wifi, and other RFID-based technologies. They just called it "Sensor Fusion" to keep it simple or maybe it sounds better with their focus groups. They are going to use as much data as possible to match your location in the store with the sensor of the the item that was removed from the shelf (just like the mini-bar in a hotel). So this is your phone's location combined with computer vision to recognize your face and body. The machine learning is just to get smarter about handling different behaviors and corner cases as they get more training data.
1sharetweet・
Kyle Dumovic
@kdumovic · UX Designer at Stella.ai
@nickabouzeid I feel like they're embellishing the 'computer vision' aspect of this. I wouldn't be surprised if all they're doing is scanning RFID tags on all your items with the turnstile looking things at the store's entrance when you walk out.
That said, such a system wouldn't allow you to peruse your cart or view your subtotal while still shopping -- so perhaps that's not how they're doing it and they are in fact using cameras and facial recognition -- but doesn't that seem over-the-top for something that RFID scanners (or equivalent tech) should be able to do easily?
1sharetweet・
SyCo4Sports
@saccoallen · me
@bikebodenberger @nickabouzeid Somehow each product has to be identifiable. If someone picks something up and puts it down elsewhere it would still need to be recognized. My guess is of course RFID which is already usable and the cost is lowering. The Shelves could not be use to track product for the same reason.
upvotesharetweet・