{"id":8507,"date":"2021-10-21T11:28:49","date_gmt":"2021-10-21T11:28:49","guid":{"rendered":"http:\/\/TheNextWeb=1370630"},"modified":"2021-10-21T11:28:49","modified_gmt":"2021-10-21T11:28:49","slug":"terrifyingly-facebook-wants-its-ai-to-be-your-eyes-and-ears","status":"publish","type":"post","link":"https:\/\/www.londonchiropracter.com\/?p=8507","title":{"rendered":"Terrifyingly, Facebook wants its AI to be your eyes and ears"},"content":{"rendered":"\n<p>Facebook <a href=\"https:\/\/www.cnbc.com\/2021\/10\/14\/facebook-announces-ego4d-first-person-video-data-set-for-training-ai.html\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">has announced<\/a> a research project that aims to push the \u201cfrontier of first-person perception\u201d, and in the process help you remember where you left your keys.<\/p>\n<p>The <a href=\"https:\/\/ai.facebook.com\/research\/publications\/ego4d-unscripted-first-person-video-from-around-the-world-and-a-benchmark-suite-for-egocentric-perception\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Ego4D project<\/a> provides a huge collection of first-person video and related data, plus a set of challenges for researchers to teach computers to understand the data and gather useful information from it.<\/p>\n<p>In September, the social media giant launched a line of \u201csmart glasses\u201d called <a href=\"https:\/\/about.fb.com\/news\/2021\/09\/introducing-ray-ban-stories-smart-glasses\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Ray-Ban Stories<\/a>, which carry a digital camera and other features. Much like the <a href=\"https:\/\/www.google.com\/glass\/start\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google Glass<\/a> project, which met <a href=\"https:\/\/theconversation.com\/see-change-is-google-glass-all-its-cracked-up-to-be-13268\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">mixed reviews<\/a> in 2013, this one has prompted complaints of <a href=\"https:\/\/www.gizmodo.com.au\/2021\/09\/ray-bans-stories-investigation-facebook-privacy-watchdog\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">privacy invasion<\/a>.<\/p>\n<p>The Ego4D project aims to develop software that will make smart glasses far more useful, but may in the process enable far greater breaches of privacy.<\/p>\n<h2>What is Ego4D?<\/h2>\n<p>Facebook describes the heart of the project as a massive-scale, egocentric dataset and benchmark suite collected across 74 worldwide locations and nine countries, with over 3,025 hours of daily-life activity video.<\/p>\n<p class=\"p1\">\n<figure>\n<p><iframe loading=\"lazy\" src=\"\/\/www.youtube.com\/embed\/taC2ZKl9IsE\" height=\"240\" width=\"320\" allowfullscreen frameborder=\"0\">[embedded content]<\/iframe><\/p>\n<\/figure>\n<p><!--resp-video-container--><\/p>\n<p class=\"p1\">The \u201cEgo\u201d in Ego4D means egocentric (or \u201cfirst-person\u201d video), while \u201c4D\u201d stands for the three dimensions of space plus one more: time. In essence, Ego4D seeks to combine photos, video, geographical information and other data to build a model of the user\u2019s world.<\/p>\n<p>There are two components: a large dataset of first-person photos and videos, and a \u201cbenchmark suite\u201d consisting of five challenging tasks that can be used to compare different AI models or algorithms with each other. These benchmarks involve analyzing first-person videos to remember past events, create diary entries, understand interactions with objects and people, and forecast future events.<\/p>\n<p>The dataset includes more than 3,000 hours of first-person video from 855 participants going about everyday tasks, captured with a variety of devices including GoPro cameras and <a href=\"https:\/\/www.softwaretestinghelp.com\/best-augmented-reality-glasses\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">augmented reality (AR) glasses<\/a>. The videos cover activities at home, in the workplace, and hundreds of social settings.<\/p>\n<h2>What is in the data set?<\/h2>\n<p>Although this is not the first such video dataset to be introduced to the research community, it is 20 times larger than publicly available datasets. It includes video, audio, 3D mesh scans of the environment, eye gaze, stereo, and synchronized multi-camera views of the same event.<\/p>\n<blockquote class=\"twitter-tweet\" data-width=\"500\" data-dnt=\"true\" readability=\"10.606232294618\">\n<p lang=\"en\" dir=\"ltr\">Ego4D is a massive-scale egocentric video dataset and benchmark suite. <\/p>\n<p>It offers 3,025 hours of daily life activity video spanning hundreds of scenarios captured by 855 unique camera wearers from 74 worldwide locations and 9 different countries.<a href=\"https:\/\/t.co\/oJHBTdQp3b\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">https:\/\/t.co\/oJHBTdQp3b<\/a><a href=\"https:\/\/t.co\/K90k9MQHyQ\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">pic.twitter.com\/K90k9MQHyQ<\/a><\/p>\n<p>\u2014 Papers with Datasets (@paperswithdata) <a href=\"https:\/\/twitter.com\/paperswithdata\/status\/1448658370961948688?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">October 14, 2021<\/a><\/p>\n<\/blockquote>\n<p>Most of the recorded footage is unscripted or \u201cin the wild\u201d. The data is also quite diverse as it was collected from 74 locations across nine countries, and those capturing the data have various backgrounds, ages and genders.<\/p>\n<h2>What can we do with it?<\/h2>\n<p>Commonly, computer vision models are trained and tested on annotated images and videos for a specific task. Facebook argues that current AI datasets and models represent a third-person or a \u201cspectator\u201d view, resulting in limited visual perception. Understanding first-person video will help design robots that better engage with their surroundings.<\/p>\n<figure class=\"align-center \"><\/figure>\n<figure class=\"align-center \" readability=\"2\">\n<p><figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip\" sizes=\"(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px\" alt=\"Future robotic agents will benefit from a better understanding of their environment\" width=\"600\" height=\"850\" class=\"js-lazy\" data-srcset=\"https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=1 600w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=30&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=2 1200w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=15&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=3 1800w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=1 754w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=30&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=2 1508w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=15&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=3 2262w\"><figcaption><a href=\"https:\/\/thenextweb.com\/news\/facebook-ai-watch-listen-your-surroundings-syndication#\" data-url=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Feditorial.thenextweb.com%2Fneural%2F2021%2F10%2F21%2Ffacebook-ai-watch-listen-your-surroundings-syndication%2F&amp;via=thenextweb&amp;related=thenextweb&amp;text=Check out this picture on: Future robotic agents will benefit from a better understanding of their environment. Wikimedia\" data-title=\"Share Future robotic agents will benefit from a better understanding of their environment. Wikimedia on Twitter\" data-width=\"685\" data-height=\"500\" class=\"post-image-share popitup\" title=\"Share Future robotic agents will benefit from a better understanding of their environment. Wikimedia on Twitter\"><i class=\"icon icon--inline icon--twitter--dark\"><\/i><\/a>Future robotic agents will benefit from a better understanding of their environment. Wikimedia<\/figcaption><noscript><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip\" alt=\"Future robotic agents will benefit from a better understanding of their environment\" width=\"600\" height=\"850\" class srcset=\"https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=1 600w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=30&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=2 1200w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=15&amp;auto=format&amp;w=600&amp;h=850&amp;fit=crop&amp;dpr=3 1800w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=45&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=1 754w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=30&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=2 1508w, https:\/\/images.theconversation.com\/files\/427172\/original\/file-20211019-13-8ljn5r.jpg?ixlib=rb-1.1.0&amp;q=15&amp;auto=format&amp;w=754&amp;h=1068&amp;fit=crop&amp;dpr=3 2262w\"><\/noscript><\/figure><figcaption><span class=\"caption\"><\/span><\/figcaption><\/p>\n<\/figure>\n<p>Furthermore, Facebook argues egocentric vision can potentially transform how we use virtual and augmented reality devices such as glasses and headsets. If we can develop AI models that understand the world from a first-person viewpoint, just like humans do, VR and AR devices may become as valuable as our smartphones.<\/p>\n<h2>Can AI make our lives better?<\/h2>\n<p>Facebook has also developed five benchmark challenges as part of the Ego4D project. The challenges aim to build a better understanding of video materials to develop useful AI assistants. The benchmarks focus on understanding first-person perception. The benchmarks are described as follows:<\/p>\n<ol>\n<li>Episodic memory (what happened when?): for example, figuring out from a first-person video where you left your keys<\/li>\n<li>Hand-object manipulation (what am I doing and how?): this aims to better understand and teach human actions, such as giving instructions on how to play the drums<\/li>\n<li>Audio-visual conversation (who said what and when?): this includes keeping track of and summarising conversations, meetings, or classes<\/li>\n<li>Social interactions (who is interacting with whom?): this is about identifying people and their actions, with a goal of doing things like helping you hear a person better if they\u2019re talking to you<\/li>\n<li>Forecasting activities (what am I likely to do next?): this aims to anticipate your intentions and offer advice, like pointing out you\u2019ve already added salt to a recipe if you look like you\u2019re about to add some more.<\/li>\n<\/ol>\n<h2>What about privacy?<\/h2>\n<p>Obviously, there are significant privacy concerns. If this technology is paired with smart glasses constantly recording and analyzing the environment, the result could be constant tracking and logging (via facial recognition) of people moving around in public.<\/p>\n<p>While the above may sound dramatic, similar technology has <a href=\"https:\/\/www.ibtimes.co.uk\/chinese-police-deploy-black-mirror-facial-recognition-scans-using-smart-glasses-1659508\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">already been trialed<\/a> in China, and the potential dangers have <a href=\"https:\/\/www.theguardian.com\/commentisfree\/2021\/feb\/27\/facebook-facial-recognition-glasses-would-endanger-women\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">been explored by journalists<\/a>.<\/p>\n<p>Facebook says it will maintain high ethical and privacy standards for the data gathered for the project, including consent of participants, independent reviews, and <a href=\"https:\/\/www.oaic.gov.au\/privacy\/guidance-and-advice\/what-is-personal-information\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">de-identifying data<\/a> where possible.<\/p>\n<p>As such, Facebook says the data was captured in a \u201ccontrolled environment with informed consent\u201d, and in public spaces \u201cfaces and other PII [personally identifying information] are blurred\u201d.<\/p>\n<p>But despite these reassurances (and noting this is only a trial), there are concerns over the future of smart-glasses technology coupled with the power of a social media giant whose intentions have <a href=\"https:\/\/www.cnbc.com\/2021\/09\/14\/facebook-documents-show-how-toxic-instagram-is-for-teens-wsj.html\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">not always been aligned to their users<\/a>.<\/p>\n<h2>The future?<\/h2>\n<p>The <a href=\"https:\/\/www.image-net.org\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">ImageNet<\/a> dataset, a huge collection of tagged images, has helped computers learn to analyze and describe images over the past decade or more. Will Ego4D do the same for first-person video?<\/p>\n<p>We may get an idea next year. Facebook has invited the research community to participate in the Ego4D competition in June 2022, and pit their algorithms against the benchmark challenges to see if we can find those keys at last.<!-- Below is The Conversation's page counter tag. Please DO NOT REMOVE. --><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/counter.theconversation.com\/content\/170092\/count.gif?distributor=republish-lightbox-basic\" alt=\"The Conversation\" width=\"1\" height=\"1\" class=\"js-lazy\"><!-- End of code. If you don't see any code above, please get new code from the Advanced tab after you click the republish button. The page counter does not collect any personal data. More info: https:\/\/theconversation.com\/republishing-guidelines --><\/p>\n<p><noscript><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/counter.theconversation.com\/content\/170092\/count.gif?distributor=republish-lightbox-basic\" alt=\"The Conversation\" width=\"1\" height=\"1\" class><\/noscript><\/p>\n<p><em>Article by <a href=\"https:\/\/theconversation.com\/profiles\/jumana-abu-khalaf-1206676\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Jumana Abu-Khalaf<\/a>, Research Fellow in Computing and Security, <a href=\"https:\/\/theconversation.com\/institutions\/edith-cowan-university-720\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Edith Cowan University<\/a> and <a href=\"https:\/\/theconversation.com\/profiles\/paul-haskell-dowland-382903\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Paul Haskell-Dowland<\/a>, Associate Dean (Computing and Security), <a href=\"https:\/\/theconversation.com\/institutions\/edith-cowan-university-720\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Edith Cowan University<\/a><\/em><\/p>\n<p><em>This article is republished from <a href=\"https:\/\/theconversation.com\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">The Conversation<\/a> under a Creative Commons license. Read the <a href=\"https:\/\/theconversation.com\/facebook-wants-ai-to-find-your-keys-and-understand-your-conversations-170092\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">original article<\/a>.<\/em><\/p>\n<p> <a href=\"https:\/\/thenextweb.com\/news\/facebook-ai-watch-listen-your-surroundings-syndication\">Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Facebook has announced a research project that aims to push the \u201cfrontier of first-person perception\u201d, and in the process help you remember where you left your keys. The Ego4D project provides a&#8230;<\/p>\n","protected":false},"author":1,"featured_media":8508,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts\/8507"}],"collection":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8507"}],"version-history":[{"count":0,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/posts\/8507\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=\/wp\/v2\/media\/8508"}],"wp:attachment":[{"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.londonchiropracter.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}