In this episode, Rene is joined by Dedipya Laidlaw, as they talk about synthetic data and generative AI for the metaverse.

Transcript

Rene – Hi! Welcome to Meta Minutes, your bite-sized pieces of the metaverse. My name is Rene from Valorem Reply and today we're going to talk about synthetic data and generative AI for the metaverse. And for this, I'm honored to have a special expert guest today, Dedipya Laidlaw. Hi Dedipya and welcome to the show. How are you today?

Dedipya - Hi Rene! Thank you! I'm really good. How are you doing?

Rene - Fine and I'm really excited to talk with you, but first of all let's introduce you to the audience here. Tell us a little bit about yourself and your background as it relates to the metaverse, 3D and any related topics.

Dedipya – So, I started my career as a 3D animator and then since then I've had a chance to like work on different parts of the 3D pipeline. Currently, I'm a technical space so that means I not only make art but I also build tools that help me make art. So I started my career in film and then after filming games, and then after games, I had a really good opportunity to start working on the Microsoft HoloLens and so since then I've been able to use immersive tech in a lot of enterprise situations context and being able to help a lot of customers’ challenges. So currently with the Valorem team, which is an awesome team, we get to do a lot of the same work and we use a lot of like cutting-edge technology with our latest foray which I'm super proud about machine learning.

Rene - Awesome stuff and yeah the Dedipya, you're a fantastic 3D animator but so much more as you said. So yeah, really impressive. Let's dive into our today's topic. You recently actually told me about a great project you have been working on that was leveraging synthetic data for AI model training right and which is super exciting because it will allow us to scale but anyhow you will tell us I'm sure about it, like what is it about?

Dedipya – So, this project kind of… so it came out of a conversation that we had with the customer and the customer basically said they thought they had everything working, so the biggest challenge they were facing was that the model that they were using, the images that were used for training the model itself for actual real-world images but the issue with that was that they didn't have a proper cadence. There was no process of like how often the images would come in, how long it would take for the model to train the images and the images would come in at random times. So that meant that they had to keep pushing off model training or reporting accurately or by the time the images came in they had to get new updated images with those images weren't relevant anymore. So then when they talked to us, we suggested using synthetic data because with synthetic data you can just build a whole huge database of images but a huge variety and additionally it cuts down the amount of time you're waiting to train the model and you have varieties that you otherwise wouldn't be able to get in a real-world data center. So, it just opened up this huge possibility for them to be able to get a much more reliable model but also in times in a way that would keep their model updated in a way how often it was changed.

Rene - This is pretty impressive and especially like the part you mentioned that you can basically synthesize or render variations that you would not be able to otherwise see in the real-world data right so you could augment your data, right?

Dedipya – Definitely. I mean some of the images you will see with a lot of synthetic data are not meant for the human eye. Like, if you give like an example that anybody could see online, if you go to any media and you look at their synthetic data example, you look at their forklift truck being tumbled on the floor in like different colors and different textures and it's just a way for us to tell the model that hey, no distractions, this is what you need to learn but that's the part of synthetic data that I think it's a little bit jarring. The first time you see it, you, like, this doesn't make any sense to me but it's not mentioned like it's meant to make sense to the machine. So, it's really interesting sometimes, you can have fun with synthetic data.

Rene – Yeah. Oh. this is great and so do you think for the metaverse like the whole kind of you know vision of the metaverse and do you think this is relevant for the metaverse and it's a building block for like large scale experiences?

Dedipya – Definitely. I mean I think if you look at the way3D and one of the examples was Google Nerf, like the way it's been able to generate an image even from something that was just shot in darkness with a lot of noise was able to take that and make a 3D rendering of that, that amazing and to be able to take that kind of learning and put it into building a whole new world, in fact in metaverse that opens up so many possibilities, that opens up like things where you might not have imagined as a person but the machine suddenly has a new idea and that can be experienced inside metaverse.

Rene - Yeah very good. I'm pretty sure it's super relevant. I'm pretty sure you're right there because like if we think about like just a huge amount of… I mean, let's face it, like the metaverse, the true vision is the next iteration of the internet which means it's the 3D immersive internet which means we need a lot of 3D data, we need a lot of data and we need to also recognize a lot of things automatically and I guess there's no way actually that you could have all, you know, human workers or whatever you want to call it, you know, provide all of this data. I don't think that is even possible and it might actually be an essential building block, right? Without it, it might not even work.

Dedipya – Exactly, and I mean with synthetic data, with the tools that are available today, the commercial possibility for somebody who can just come in, pick up those tools and start building that synthetic data like it's not something that only a person with a very specific skill set will be able to build. Like obviously with my background and TV, there's a lot more understanding of like photorealism and all of that but anybody can actually go and pick up a few tools, learn how to build this and then you have this data that can then be used train. Obviously, like, the biggest part about synthetic data is that it takes away any bias that people might have on generating the data. So, then when you have data coming in from all these different sources it kind of makes sure to cover all the different facets and different kinds of technology.

Rene - That makes a lot of sense and I think like also another part like not so much synthetic data but the generative AI where you have these AI systems that can generate new content out of a text prompt for example right, I also think this will be gaining quite some momentum already but I think it will be even more important like with you know these models that are out there like DALL-E or Mid-Journey and Stable diffusion just to name the three most popular at the moment, they can create like I said these novel images by an AI that learned basically all the internet images and you can just give it a text prompt and then it hopefully has a desired result and it's really impressive what we see all around these days. I've been playing around quite extensively with those but like you said also what you see is certain buyers right when you ask certain questions like, “hey give me a computer scientist”, you will always get a white male in the mid-40s or age or something right and so that's an issue for sure because like that's not the real world anymore and so fortunately, there's a change, this is amazing. But it's still the data so right, maybe we can use synthetic data actually to generate enough you know other permutations and then we can get better results with less bias. But I actually had another question regarding this generative AI stuff for you because you know you're also 3D artist by trade or if you will and so the whole kind of idea that basically these generative AI models like mid-journey, stable diffusion, DALL-E, they use all the existing original of artists basically, because they are trained on this right and so it's kind of like some artists what you read online they see it as a threat right because like it's kind of using the original artwork and generating new stuff though but it's still based on what they have done and so what do you think is it an opportunity or do you rather see it as a threat for the audience in their original work.

Dedipya - I mean I think that if somebody wanted to steal somebody's work that's already happening today right? Like this as humans we're pretty creative, we'll always find a way to go around the road. I think with generative AI is still a tool, so it depends on whether you can either use it for good or you can use it for bad. So, it depends on the person using it. I think with generative AI, the really exciting part is that you can train it on these limitless possibilities and then suddenly you have a version of an image or a word that you didn't even imagine. So like one of the most recent examples I was looking at earlier last week was somebody who used mid journey to make Metallica's entertainment video and so it basically used the song lyrics to then generate the visuals and what was really interesting to me is that like every time a lyric repeated the visual, that it showed kind of fit in the world of that original image that it showed. I think one of the lyrics was enter neverland or something and like the image the first time it showed or exit light, enter night and like the image that showed was like the exit door and the light coming out of the person walking to the other side but the next time it shows that image for that same lyric it was slightly different. It was something had changed. So I think, it's like generative AI has this… because you're not limited by the human mind it gives you all these like new ideas that you otherwise might not have thought. One of the images the houses were just floating on top of this cloud, so I think it's important to have the opportunity to have those ideas because when you're building the metaverse if you're building this like digital world and you can't possibly have all the ideas manually like you have to have all these different inputs and different sources. I don't think generative AI will ever replace an original artist's work but I think it will take us to a point where we give us a lot of new ideas, it'll give us a lot of it's kind of like, you're as an artist, you're kind of collaborating with the machine to think of something new.

Rene - I like this a lot. I mean like if you think about when computers came out and then you know the first computer graphics happened even then already like, in the 70s or 80s like artists use this, I, unfortunately, I forgot the name of an artist, that's one of the famous artists that first used like digital images and he recently died a few months ago. But he was one of these pioneers and they all also were looking at him, “oh he's using computers to generate images like what is he doing or even like I don't remember who said it to me but it was also a good analogy I think it's similar like when you know of you know photography it came along and people could you know finally take photos of the real world and you know painters were saying oh they're cheating. Kind of like and now we have AI helping us and cheating but it's I think it's like you're saying. I fully agree it's rather a tool in the tool belt of an artist but you're still the artist and you have to use the tool in the right way and in a unique way because in the end odd is also about having a unique idea and knowing how to use the tools correctly right.

Dedipya - I mean like even if you look at the history of animation when Disney made Beauty and the Beast and like they were animated in the dancing, they actually had its own dancers doing those exact steps and then they drew on top of it and people are like oh well that's cheating, like that's not animation and then motion capture came along and everybody was like, ‘oh well now this is even more cheating’ and you're like no it's just a way to improve what you're already working on so I think like each time technology makes a leap forward there's always like naysayers, like, ‘oh well this is just gonna kill innovation but I think we just need time to kind of get used to this new iteration of technology.

Rene - And that's actually a fantastic final word here. We just have to get used to it, same applies also to the metaverse I guess. We'll see, you know, but anyhow we're already at the end of the show. Thank you so much to Dedipya for joining us today and sharing your insights, that was very much appreciated.

Dedipya - Thank you so much Rene. Thank you so much for having me. I enjoyed it.

Rene – Awesome. Well and thanks everyone for joining us for Meta Minutes, your bite-sized pieces of the metaverse. Watch our blog, follow our social media channels, subscribe to our YouTube channel to hear all about the next episodes, and of course, you can visit us at valoremreply.com to recap all previous episodes at any time. Take care, and see you soon in the metaverse.