Meta Minutes 2.6 - Developing Metaverse Apps

In the sixth episode of Meta Minutes, Rene and our expert guest Jesse, McCulloch from Microsoft, discuss getting started with metaverse app development.

Transcript

Rene – Hi! Welcome to Meta Minutes, your bite-sized pieces of the metaverse. My name is Rene from Valorem Reply and today we're going to talk about developing metaverse apps and for this I'm very honored to have a special expert and friend today, Jesse McCullouch. Hi Jesse and welcome to the show. How are you today?

Jesse – Hi Rene, glad to be here. I'm doing good and I'm excited to talk about the metaverse.

Rene – Awesome! First of all can you tell us a little bit about yourself and your background as it relates to metaverse, AR, mixed reality, all of these things?

Jesse – Yeah. I started off in in this area, I guess, coming up on five or six years ago now when the HoloLens one first announced and I got super excited, went out and bought my own HoloLens one and decided I was going to learn how to build AR  and then I started building a community around it because it was so new, there wasn't really one out there to get involved with and then actually four years ago this week, I joined Microsoft to start doing our developer community relations activities regarding you know AR and VR and mixed reality and so I am continuing to do that today and now I'm working on the mesh team which is one of our tools at Microsoft that we're building to help people build on the metaverse.

Rene - Well that sounds great and you remind me how time flies. Already been four years since you joined Microsoft.

Jesse – Yeah, it’s crazy.

Rene – Well, but like so many good things have happened since then, and yeah you're really great in your role and what you're doing but let's talk about some of our topics. First of all, typical simple but also complex question, what is the metaverse for you and where do you see the potential?

Jesse – Yeah, the metaverse is an interesting topic for me because there's so much hype around it and I'm not one of those people who really jump in on the hype generally. My girlfriend will tell you it took me like almost 20 years before I read the first Harry Potter book, so I'm really slow and jumping behind the bandwagon on some of this stuff but for me like metaverse. I think is that a time, a point in time, you know, title that we're giving to this broader thing of just bringing more interactivity to our real world at some point but to Virtual worlds at this point just because the virtual reality stuff is so much more prevalent and easy for people to access at the moment but when I think back to, you know, the internet, which is really where all this gets kind of based in. If you went back to the 80s when it was like BBS bulletin board systems, you know, we didn't call it the internet then, I didn't call it anything then because I was like five or six years old but you know it’s evolved into the internet and even the internet of 15years ago is not the internet of today. So I think the metaverse is just an extension of the internet and as we grow into these new ways of interacting and building things that are connected to our world into each other, we grow into this you know super exciting area where you know you'll be able to have digital and real content interacting with each other and being able to pull collaboration and you know social stuff in from people around the world in a much easier and more natural way than we do it today.

Rene - Yep and I think it's great what you're saying also the reference to the beginning of the internet right because that's what you hear a lot. It's like this is the new immersive internet basically that we're seeing well we're actually helping to develop kind of, which brings us to the next question about app development and how to get started. So if I would be a person that is trying to get started with building applications for the metaverse, where would you send me?

Jesse - You know, now is like the most exciting time for this in multiple ways like for the first part there's this is such a new area for us, you know, building these 3D immersive worlds and applications is kind of a brand new paradigm over the last I'd say six years maybe a little bit longer than that for some people but there's no experts like nobody's been doing this for 40 years and as the tried and true expert. So there's still plenty of room for anybody who wants to get involved to go out and do so and the amazing thing is with the internet the way it is today there's so many resources out there you know you can go to YouTube and start learning something like Unity or Unreal or any of these real-time game engines which is where a lot of the base of the metaverse is and will be you know if you go to universities or starting to have courses on building 3Dexperiences or mixed reality, you know, community colleges have a lot of game development and 3D art classes these are all important parts of it. I think it's really easy for us to focus on developers and bringing developers along to start building these things but you know really there's 3D artists who have a lot of skills in building the assets. I know that I can develop that application super easy but it's going to look like stick figures and cubes and triangles just because I don't have the capabilities of doing the art part of it. You know, producers are generally something we think about in movies and game development somebody who kind of goes through and figures out what the experience is going to be like and builds towards that. I think that's going to be an important role as we start thinking more about how we tell stories versus how we just go interact with stuff. So I think there's just a ton of tools out there and a lot of resources right now and it's super-exciting because even things like WebXR, super low barrier entry, anybody with a browser on a phone can do WebXR. So you don't have to have all these expensive tools and stuff that that are part of the world today but I think as we transition in, it becomes a lot easier.

Rene - Yeah and especially what you mentioned about the WebXR and the big progress we're seeing there I think, this is really amazing because like you said it first of all like from the from development side or like you said it's interesting developers right, it's, let's say, creators yeah like you know whatever that it is and so like if I'm a Creator, I can quickly build applications in fact with certain WebXR or engines or platforms and this is fantastic but if what is even better is like we're saying is it will run on a mobile phone it will run on pretty much any platform and so you have the super wide reach and the capabilities are increasing quite dramatically, right?

Jesse – Yeah, I mean web development is the largest kind of development out there today and you know being able to bring that audience of creators and people in who can start building, based on the technologies, they know is super important and another interesting thing is, of course, like when we look into low code no code development efforts for even metaverse things like one example is Roblox right like if you look at ROBLOX, a lot of folks like see it just as a game, like an online game for kids mainly but you know, my kids, for example like the older one they play a little bit of it but yeah it's actually local platform right you can build custom experiences, on top of it or actually in it and a great example, with Nike land you know what they did there, reaching seven million people in like two months or something like this, this is incredible. I mean just the reach and the reach of the audience is incredible but also what is really amazing is how easy it is to build and you know kids can get started with this and then they can move on to more you know more. I'm saying these are not good platforms but or models right you can build amazing things but at some point you might either hit a roadblock or something and then you will move on but similar to, like you know… but Microsoft is doing with the Power Platform and all of this right, low code no code efforts.

Jesse - And you know PowerApps and Power Platforms an interesting one because we're starting to see you know a few more pieces of mixed reality start entering there where you can build experiences where you can bring like 3Dmodels into your real space using microphone or a tablet and that's just kind of part of the platform and super easy to do. So it's exciting that you know people are able to build stuff again, low code no code just because they've got the tools super easy and super accessible.

Rene - Yeah but also the other platforms like Snap for example or you know some other like you have these cradle tools, really amazing, you can, without writing any line of code, you can get some pretty fancy AI effects done and then you might be able to write a little bit of like Swift Code or so if you want but yeah, I guess this is where a lot of new experiences we will see, actually be born right like in these tools.

Jesse – Yeah and it's nice because it starts to become a continuum, right? So there's some people who design a snap filter maybe as far as they ever want to go or need to go but then somebody who starts there and wants to ramp up a little bit and can start to see where they want to go, has these tools to go in and like you said maybe do a little bit of scripting, a little bit of Java script and then kind of work your way up to some of these bigger platforms that require more time spent learning them. But you see this progression, you know, available to people with and they have the option to stop anywhere along there and still be successful and contribute.

Rene - Fully agree and that actually also is a very important part of the whole app development and creator economy. Of course, a community is like you know, we're all part of a certain community, you know, it depends on you, where you identify and so on but anyhow in general I mean there's a large creator community or developer community whatever you want to call it, right? Like let's be inclusive, let's say creator economy and to have been working for many years with the creative economy, especially as part of the mixed reality effort and HoloLens and so on but also in a broader sense like you know you did these hackathons and a lot of things like, you're super active in the community, of course, right like playing in the intro and so my question is, is there anything particular you have seen in the community that has changed since you started like five years ago, as you mentioned or maybe also in relation to the metaverse hype since last year like what has changed since then and the last sub-question, are there any certain things you would try to change or make difference?

Jesse - I mean I think I think the hardest part for anybody who's in this space or been in the spaces that we have hopes for where it will end up and how fast we'll get there and it's just you know, there's still a lot of hard computer science and hard physical science that we have to work around, you know, to get to where I think this is, where we all are looking forward to it being. I know when I first got my HoloLens, I was super-excited, I'm like this is going to be the next big thing next year and that's just not as hopeful as I was and maybe as has dreamy as that was, it's not it wasn't based in reality as much as I wanted to, you know, to buy into the hype and I think that's one of the great things about you know my position at Microsoft has given me a new perspective of, you know, just how much work it is that goes into all of this and to kind of innovating in this space and building on it. So I'd say for people who are in the community again it's a great time to get started just to have, you have realistic expectations that we're not going to be in this Ready Player One world next year, you know, maybe in a few years from now and we may decide that that’s not really where we want to be. So, I think, you know, when we look at things like Open Standards and being able to have open metaverses that you can traverse between, so it's not just one metaverse to rule them all but you have a lot of choice in how you view the world and how you interact with it is important. So I'd say on the community front, you know, I think we're seeing an influx of people who are joining and from all sides of the world, you know, in the last year with the metaverse hype and the way you know web3 or blockchain or crypto or however you want to refer to it is bringing a lot of new people and a new a lot of new perspectives into the space, which is great it brings you know that diversity of thought, I'm not sure where I exactly stand with all of it but you know, I think the more voices we hear and the more people we have in the community the better it's going to be overall, you know, to learn and grow in.

Rene - And I think this is an is an important point is like a saying, like there's a lot of different technologies in building blocks coming together for the metaverse, like we are coming like more from the kind of like AR VR spatial computing kind of a world mixed reality and so on and then you have also of course like folks that come more from a collaboration background right or folks that like I said more from a crypto like decentralized platform background web3 and or NFTs and so on right and it's all coming together somehow because these are all the building blocks, right, we have like you know, spatial computing, computer vision, AI for these advanced capabilities with these devices and then we have the whole, you know, kind of decentralization with blockchain and you know lateral technology there and 3D rendering and in general of course all of this coming together and of course also massive computational power is required, right, also cloud folks are joining the bandwagon, if you will, like you know, Azure services and so on that are required because like, just be honest, I mean, if we want to have these large online worlds to a realization where it can scale up this requires a lot of computational power.

Jesse – Oh yeah, for sure and when you if you think through it and I'm not even pretending that I have all the answers but you know if we want devices to be, you know, similar to like the pair of glasses that you're wearing, you know, having a GPU and a CPU and all the stuff that's required to to render you know if we're gonna have all that stuff in a pair of glasses like that we have along long way to go of, how we build, you know, that kind of technology you know. The other option is we think about pervasive high-speed internet through 5G or  future Technologies and then you start thinking about you know offering or offloading all of that compute and render to the cloud and just streaming it down you know, that's another whole area where we're seeing a lot of advances from multiple companies from our side, you know, I think the precursor to this is like the Xbox game streaming where you can you know we cert, we spin up your game on an Azure server or somewhere and do all the rendering and compute there and just stream it down with hopefully low latency and you can play your Xbox games on a tablet or a phone or where you know even your computer browser but thinking forward to how that brings into you know the offerings and being able to have devices that are smaller and much more socially acceptable you know, it's all coming together slowly but surely and we'll get there.

Rene -Absolutely and you make a good point I think really the key enabler is like having smaller form factors and yeah the challenge is just like how small can you make it with still having the performance and I think what is what will be a big enabler is the advancements we see with AI deep learning models in particular with neural radiance fields and this kind of you know neural rendering which is a whole different approach to 3D rendering in general instead of like scan line based rendering or so on, right, here you can interpolate in in essence that's like what I'm sometimes saying is like, you know, the AI models are basically a really good compressors and you can compress a lot of data on it and it can interpolate and you know just like we as humans interpolate certain things which sometimes you might say these are assumptions right, but they are really good and I think they will be it will be key enablers and where you can just take a few signals like recent paper I've seen just using the head pose and two controller poses to basically fully get a body pose into interfere with interference from that AI model and just with these little signals you can actually create very realistic things and probably this will be also key enabler to actually make it happen at scale.

Jesse – Yeah, I think the hard part for you and me is like we like to be generalists and have some knowledge of all this stuff and that's just becoming bigger and bigger and harder and harder to be able to keep track of it all and keep on top of it all because it's just moving so fast.

Rene- Yeah and another interesting thing while we're talking AI for the moment like the whole diffusion models approaches right like this stable diffusion, DALL E, Mid-journey, and well you name it like all of these in general generative AI approaches. I think this is also a key enabler for making a lot of these also the creator economy, right, like you were saying that I'm the same right? It's very hard for me to make a 3D model that is more than a cube or sphere I mean but maybe capsule and Taurus you know, but it's tough, but you know we're getting there where we have generative AI with basically text to image where you give a text prompt and it generates an image out of this and there's already some early versions we're seeing with 3D like, I want a 3D model of a donut or well doing that it's easy, I want a 3D model of a chicken and then you get you model of a chicken.

Jesse - Yeah for my birthday somebody took one of the MVPs actually took my profile picture off Twitter and did it as a mutant Simpsons character and they based off my picture and they sent it to me and it was actually pretty funny how well it did it in in a really humorous way you know and I think all that generative technology is super awesome and super fun to play with I think. The harder part is thinking through the way we think about the ethics and ownership and contribution right because all these models are built off of somebody's work previously that we've trained all these models on and so how do we use it responsibly so that we're making sure that we get permissions to use all those images and you know kind of give attribution where it's due and also how do we keep from, you know, making those skills so commoditized that people can't make a living doing them, you know. It's super easy for me to go out and do a generative AI and get a3D model and maybe that's where I'll decide to stop because that's my, the low bar that I have, whereas if I went and hired a 3D artist to go do it you know I'm contributing to the larger overall economy of building in the space.

Rene – Yeah, it's a very good point, right? This is the whole controversy around generative AI and I talked with a bunch of folks like also artists and designers and so on and you know some are split between but like I got it I got a really nice kind of analogy from someone that basically said it's similar like when photography came along right like painters were basically saying, oh they are cheating, they're just taking photos. And so now it's also an art form, of course, photography, and no one's saying like they're cheating anymore or things like this. So what I'm trying to get to is like what I heard from a couple of folks of because like you're fully right like what was the original artwork then but the thing is like these models are not like stealing someone else's work, they make new stuff and the point is like what they were saying it's basically another tool in a tool belt and it will make certain mundane tasks easier because like you're saying, I'm pretty sure like although we have these big advancements and you can quickly like for prototyping you can quickly get some 3D models like having really good topology and, you know, all these texturing and all of this stuff requires very skilled people that have a very good training behind them for many many years and I don't think we will get to this kind of high quality. So I don't think it will cause a lot of job loss in that particular sector but I think it's rather kind of another tool in the tool belt to be used by real artists right that need creativity in the idea.

Jesse – Yeah. I think it's like anything with technology, there's the utopian view and the dystopian view and we could go either way with it and hopefully there's more of us who lean towards the utopian versus the dystopian and we end up in the right place.

Rene - I'm all in for that. Well, thank you so much, Jesse, we are already at the end of the show, but we could talk for many more hours but anyhow thank you so much for joining us today and sharing your insights very much appreciated.

Jesse - Yeah great to be here and thanks for inviting me.

Rene - Well and thanks everyone for joining us today for yet another episode of Meta Minutes, your bite-sized pieces of the metaverse. Watch our blog, follow our social media channels, subscribe to the YouTube channel, and of course visit our website to view all the previous episodes. Until then take care and see you soon in the metaverse!