I am the founder of the virtual civilization Second Life, populated by one million active users, and am now CEO and co-founder of High Fidelity — which has just released a real-time spatial audio API for apps, games, and websites. If you want to check it out, I’d love to hear what you think: highfidelity.com/api
High Fidelity’s Spatial Audio was initially built for our VR platform — we have been obsessive about audio quality from day one, spending our resources lowering latency and nailing spatialization.
Ask me about immersive spatial audio, VR, virtual worlds and spaces, avatars, and … anything.
Would you like to return to investing your time with virtual reality and virtual worlds again? What would you do differently?
I'll be working on virtual worlds in one form or another until I die.
I'd like to see a physics/simulation model that creates some sort of low-level digital 'atoms' that are used to build everything, using a decentralized compute model.
Whats going to be the next big step in vr?
Typing at normal speed (probably by using see-through camera to show you a real keyboard) is the biggest change that would make VR devices usable for general computing.
Next would be reduced weight and greater comfort for longer sessions.
The 'big step' is simply to get to something that everyone is comfortable using.
What are your thoughts on the dapp economy right now? From your experience in Second Life, do you see a world in Earth's future where people sustain themselves by making income as worker-consumers in a virtual world fulltime?
Yes: A few thousand people make their living today in Second Life, and that is a place that typically has around 50,000 people online. So absolutely people will sustain themselves increasingly as time goes by from work in virtual worlds.
Always respected HiFi's user-focused stance on privacy vs. data-mining. Wonder where you think the future in that area is heading, and do you ever see working together with other cool companies like Mozilla, DDDGo, Proton, etc. to create any alternative Social Media platforms?
BTW: High Fidelity sounds GREAT using Audeze Mobius headphones w/head-tracking!
Thank you. Yes I bet a number of companies will need to keep working together on standards and approaches for privacy as well as new protocols that might create new kinds of social networks. Mozilla has done lots of great work.
Thx re Audeze - will try to find a pair!
Hi Philip - we met many years ago on SL, I was a fairly well-known content creator and we chatted on a few occasions. Whatever happened to Starax? And what is the future of SL? (I left many years ago, but notice its still going).
I don't know what happened to Starax, but I can certainly say that his Wand was one of the coolest things I had ever seen, and it gave me real hope that Second Life was going to make it. His wand was something like $50USD (if I remember), and it was the gift everyone was dreaming of getting in SL one Holiday season. Who wouldn't want a magic wand that makes a 3D locomotive appear and drive through you and your friend if you happened to type "train".
Do you have any interest in augmented reality projects or purely virtual?
I think AR and VR are very different. Mostly I think about VR.
AR has some tremendous technical and privacy challenges. For example, I can't tell exactly (better than GPS) where I am standing unless my always-on-AR-camera is uploading my 3D surroundings to the cloud to match up the surfaces. Privacy implications there are chilling - we are not going to want to live in that world unless things are designed very safely.
AR is also further out than VR because of mobile device battery life and CPU/GPU.
Second Life seemed to follow a definite hype cycle in 2006-2007. After a lot of it died down, "influencers" and brands seemed to pull out and bad press replaced the hype. What lessons have you and Linden Lab learned about that experience?
Certain ideas (like the chance that people might leave earth for a digital world) seem to evoke a degree of FOMO that greatly exceeds the baseline utility of the idea. I think that was the case with Second Life. We didn't spend anything on marketing before 2008 or so, and never have spent that much.
I see the same thing happening right now with Clubhouse - everyone is afraid of missing out on something without knowing yet what the something is.
I don't think there is any action item here, just an observation.
Do you think a shift in consciousness (social or spiritual) could be triggered through VR? Did you see any signs of something like that yet?
The internet (and VR specifically) has the potential to deeply connect people. But it can also disconnect and separate people. The difference is in the choices we all make about the products we build. But sometimes maximizing profit does not maximize public good. We are going to have to make the decision to use it for good, at a cost.
Philip, I have tried your High Fidelity 3D sound many times, and it's stunning. I wonder, however, if you still have dreams/plans for another metaverse? Thanks. DrFran
I love virtual worlds and plan to keep working on them in one form or another until I die!
Hey Philip! I'm an entrepreneur since 18 (33 now), and most of my successful business ventures reside within Second Life (Fennux, Fawns, Kreatures.)
Do you ever have a vision for something that no one else seems to really grasp? Maybe you find it hard locating others that share that same vision?
If so, how do you deal with that? (Clearly you still make things happen.)
If that's not an issue, what's your secret to attracting those that share your vision? Thanks!
Don't let it get you down. Sometimes you can see something (or an opportunity) that others can't see. When I was younger I would blame myself for being unable to communicate it, or get mad at people for not hearing me, or funding me, or whatever.
Now I relax and realize that we all have gifts to offer, we're all different, my gift is (sometimes) to have these strange ideas, and it's OK if those gifts are not always accepted.
Hey Philip. Cool to see you doing an AMA. If you were rebuilding a virtual world today, would you be including cryptocurrencies in there? What's your take on the timeline to making virtual worlds so realistic that people start being unable to distinguish them from reality?
Yes I would use cryptocurrencies, but NOT using proof-of-work as the consensus mechanism, since this puts the environment at risk. The consensus mechanism cannot trade units of the currency for electrical power - that is a recipe for global disaster. Fortunately there are many other ways of maintaining a distributed record-keeping system.
If we haven't already done so, will we ever reach a point where some humans spend so much time in VR - including working and socialising - that they will come to perceive the virtual world as their seat of reality, and only grudgingly interact with the real world as little as they can get away with?
I mean, like the human/VR equivalent of a whale, that needs to surface now and then to breathe, but which doesn't consider the realm above the water to be its home.
My answer when I was younger would have been unreservedly "Yes, of course!". And there are so many ways that Virtual Worlds can extend our lives - new jobs, new friends, new kids of artistic expression. The list goes on and on, and VR devices (once they become mainstream) only make it bigger.
As I've gotten older and wiser, I'm struck by how deeply connected we are to the physical world we exist in, as well as to our precious planet. Our brains and bodies have evolved in a way that connects us perfectly to the world around us - things like touch and proprioception are incredibly important to most of us and may always lie beyond the reach of VR. Also - as we are waking up to most recently - the game called 'Planet Earth' is a special game we all have to play together, with a finite set of resources. I think there is something very powerful and affirming about that idea. If we are to build worlds (perhaps the vaunted 'metaverse') beyond this one, I bet that the idea of sharing a finite set of things (as opposed to just building into infinity with infinite resources) is part of what will make them 'real'.
Yep, we are working right now on Unity and iOS. Coming soon...
Hi Philip! Thanks for doing this AMA.
In terms of future game development, where do you see VR heading in the next few years? At what point will AAA games become mainstream on VR?
VR HMD's have to get mainstream first, which I think means comfortable for long-term use and in use by a diverse range of age/sex/race/global people - and they aren't there yet. I think it is going to take another 5 years or so.
Taking into account that current audio production is done thinking about descrete channels, what are the steps do you believe the industry has to take to make spatial audio a standard in audio production? New formats like MPEG-H across the whole chain? Better tools?
Sources of sound should be mono, unprocessed. The spatialization needs to happen on the server, so that everyone can essentially 'sample' the experience at a given point. Existing low-latency compression formats like Opus are adequate for compression.
What surprised you about how people acted in second life?
Lots! But I was deeply moved by how people came to know each other through their avatars, despite being unable to hear or see each other. I built Second Life with a focus on the 'lego kit' / Minecraft dream of building a simulated world and seeing what people would make, but I came to regard the connections between people it enabled as the most important thing.
Why not do some of this AMA in High Fidelity? It would be a great way to demo the tool for others.
Great idea - We will do that toward the end so it's not distraction to typing fast!
Years ago I saw you do an interview where you created an avatar in AOL and jumped into the Sims I think. But your goal at the time was to make it so people could have avatars to jump from site to site. In second life, people jumped from unique created world like the sims to sim. And in high fidelity it was more jumping specifically to a URL. Are your creations getting you closer to that end goal?
Also, in High fidelity with spatial audio, how do you make a grid with entities horizontal and vertical where depending on where an avatar stands, they would sound different in reference to another avatar. Or the sound of a fountain as you get closer to it gets louder. These are all perceptions to make things seem more real as we know it. What will be the new reality we will experience in a virtual world. i.e. what is next?
The hard question, I think, is why we will want to jump between these worlds or share a larger world. Most games are intentionally holistic - it doesn't really make sense to jump between them. You don't want to drive a car from GTA into Among Us. So the big question about "what is the metaverse" is what sort of space(s) we want to share, and why? I don't think anyone has very good answers to this, myself included!
I'm excited by their work! There are so many experiments needing to happen in shared 3D spaces with live building, and it is great to see the original High Fidelity code being useful for some of those experiments.
I own a vr arcade in Tahiti. The majority of people here can't afford / don't have room for a vr setup at home. How can we create a world for them to jump into for a while, leave and come back a few weeks later?
I mean shooting zombies and crap is cool for a first timer but if I want them to keep coming back, they should have a virtual home to revisit over and over again.
VRChat or RecRoom or Tivoli or Vircadia (latter two are based on High Fidelity) are some possibilities for creating stable spaces like you suggest. I agree it seems like a great direction for an arcade.
Hi Phillip, Perhaps this is not the place for politics, but would you vote for The Boy Mayor of Second Life in a Second Term? Why or why not?
"Don't hate the game, hate the player." :)
Hi Philip (and company),
I've been sporadically keeping up with High Fidelity for a few months, and attended a short chat with Philip. I deeply appreciate your work with foregrounding an audio experience!
I am working with colleagues on creating, rehearsing and presenting music/soundart works through separate direct url connected via audio vst plugins into DAWs, simultaneously connecting on Skype or Zoom for communication. (We turn off the video when playing - much better for sonic intimacy.)
Our next step is using spatialization software with this configuration. I wonder if it is possible to experiment in High Fidelity - that is, could an ensemble spread themselves in a HF location, each broadcasting our sound, and have the listeners move in and around us? It would be interesting to create things specifically for the technology, rather than present existing things, to be honest. (I also work in Second Life, and gave you some links via a Twitter convo about the Avatar Orchestra.)
Thanks for your work.
Yes - an ensemble can play into a High Fidelity space and let people move around among the performers. It is a fascinating experiment and I'd encourage you to try it (and invite me please!).
Thanks for taking questions, Philip. I’m curious what you think about the potential for using VR spaces in education, particularly in support of students with exceptional needs (e.g. non verbal, students with autism)? If a teacher and student(s) can have avatars interacting in a virtual world, I presume we could eliminate a number of limitations we have in the real world. Do you have any insight into who might be already doing this well or where the technology is going in terms of serving students with special needs?
There were remarkable studies done in SL with adults with autism that you might want to look up. A guy named John Lester, aka Pathfinder Linden did some of the early work, but you will find numerous academic papers on the subject.
Hey Philip. I had the chance to try out an HF demo last summer and think it is a very interesting concept for developers to build immersive, interactive spaces. I have two questions, you can decide if they are related.
- You oversaw the creation of one of the most immersive interactive spaces to date. Why do you think someone else will come up with something better than you as opposed to just building it yourself using your technology?
- We are seeing a rapid rise of virtual humans along with easier methods for people to create their own avatars and an increased interest in spending time in virtual spaces. When should we expect a new space where humans via avatars and virtual humans congregate and interact on a regular basis and when do you expect that space to reach some significant scale?
I think these are different questions:
There are 100+ apps/sites right now that are trying to create social spaces of one kind or another with a ton of highly specific features for different verticals, like customized buildable objects or shared whiteboards, or avatar pickers. That's a ton of people working on new worlds - hard to compete with them all. But none of them have audio that is even close to doing what we do with spatial audio. So enabling all of them to move forward faster (with our API) seems like the right move overall if we want to see more spaces out there.
As to virtual humans (powered by AI) - I think that AI is emerging as the most important and potentially dangerous area of human progress. We've done work on virtual humans and there are a ton of big problems still - we are very early. Visual representations can still be very uncanny. Another huge problem is bias, racism, and polarization coming through in their behavior and communication. As much as with AI as with ourselves, this is something we must address before moving forward. So I don't think we want to congregate with our digital children, yet.
I love virtual worlds, and I've been an SL resident since the early years. My work study in college was in SL.
This probably sounds like a dumb question, but... one thing I don't get is why HMD adoption is such a big deal.
I've used HMDs, I think they are cool, but I can't think of any content I really want to use an HMD for. Maybe a flight simulator, where my head can be another axis of control for guns as my hands control the space ship, but other than that... I just don't think it's necessary.
Why should I make the jump from developing content in SecondLife to develop for HMD platforms?
You should wait until there is a large diverse set of people using HMDs. There isn't yet. HMDs need to be accessible and comfortable for everyone: across age, gender, and race. They aren't yet.
Hey there! Big fan of your work.
There's one thing I wondered for a while: how on earth did you manage to make High Fidelity (the original, 3D one) happen? I mean, it was a platform that was fully open both on the client and server side, heavily distributed, with a cryptocurrency and made in such a way that made central control difficult. I have a very hard time imagining how one makes a business pitch for that. How do you convince people to invest in something you're allowing people to just take and do whatever they please with?
But however it happened though I'm very happy it did -- there's some excellent work in there, and we hope to keep it going.
Thanks! For a while there (2013-2019) it seemed like we would have hundreds of millions of people with immersive VR devices by 2020. That was a sound thesis for investing in a project like High Fidelity - especially given the success of Second Life in delivering many of those things to the desktop.
The failure of the VR HMD to reach mass-market (I think it is going to take about another 5 years) made it too long a project to keep pushing forward. And this kind of open / social experience would require mass adoption of the headsets.
I was involved in academic research on SL as an undergrad anthropology student. Have you read any of the (surprisingly prolific) academic work which has been done on your virtual world like "Coming of Age in Second Life"?
That's a great one - Tom Boellstorff is amazing - I try to collect/read all the books, yes. So many great ones. Many amazing collections of SL art too.
I remember having this "now we've really arrived" moment when I was flying into NYC and heard some super cool art gallery on prince street (I think) was having a gallery opening of these huge landscape images from inside Second Life. Not even avatars - just avant garde landscapes.
But yes, still lots more anthropology and sociology and economic to do in virtual worlds. For example - we should be using them to prove the value of UBI!
I give to local charities to help people near San Francisco get by during and beyond the pandemic.
I am also devoted to using both money (funding) and my time and skills to make virtual spaces that are of a benefit to people, particularly in addressing racism and polarization.
I think that it is vital that those who have any degree of influence or power use them now and use them carefully to help steer the world toward greater compassion and connection.
What is the physical limit of latency for avatar communication? Can we get to a point that is genuinely imperceptible from two people talking in real life, where no one overlaps the other talking due to latency, the way many people do in today's video calls?
For people speaking to each other, the one-way latency needs to be below about 180 milliseconds to speak comfortably. There is a famous ITU paper from 2003 with a graph, if I remember... don't have the URL handy.
Old-school land-lines have a latency of 60 milliseconds (great!), cell phones typically are about 350 milliseconds (why we hate them).
A good Zoom / WebRTC call can be at 150msecs or even lower, which is why our company can exist! :)
Have you ever watched Sword Art Online?
Yep. The idea of being 'trapped in the simulation' is resonant for me. I think the 'boundary value problem' of how to make a consistent virtual world that nevertheless is acted upon by outside agents may turn out to be very problematic.
Look for example at things like Stephenson's 'Fall' - where a lot of how the world works is contingent on there being no way to influence the world. Important to think about.
What do you think of Decentraland?
It's an amazing experiment to watch! For simulation, I definitely think that long-term there will be a distributed compute model that makes the physics of the virtual world some sort of inviolable consensus. I'd like to see Decentraland add our spatial audio API!
Why do you think Facebook is keeping Horizon so closed in testing?
Could they be worried about anti-trust action if/when social VR starts dominating the VR software market?
(I've been bullish on social VR engines "swallowing" games since at least 2012)
I don't know.
Social VR needs to be larger than one app or company or we are all at great risk.
What's a good entry point for VR gear that will work with High Fidelity?
The current version of High Fidelity is audio-only, so you only need a browser and (hopefully) some good wired headphones. We work on mobile and desktop, too.
Godot Engine, a fully open source 3D game engine, has really matured in the last several years and even now supports VR. Do you see any viability in an engine like this for developing a social VR platform?
I haven't looked at Godot. What I think matters most in the 'engine' for Social VR is the low-level simulation rules, the communication affordances, the support for great scale, and the ability to build live. Graphical detail seems much less important.
When will we have VR tech that I can wear without being teased by my wife?
I guess maybe that depends on you. :)
But seriously - the fact that different people have such different responses to VR devices (both wearing them themselves as well as their perspective on other's using them) is evidence of the failure thus-far of VR to be a general-purpose computing device. No one felt this way about the iPhone.
Do you think a new virtual world will ever come along with a large focus on creating in the same way SL does? Why hasn't anything come along yet that overtakes SL in that way?
Creating programmable objects that you can edit live in a big world where things can both move and teleport around at will is a really hard problem. We wrote a lot more code to do it than people think - and as SL people will tell you there are still lots of bugs and edge cases. I think this is why we haven't seen more things like it - we were really ahead of our time. I bet we'll see some amazing new projects soon (but I guess I've said that before and been wrong).
You have been working on sound, curious what other senses might be added in future to make a fuller experience? Many times I wish I could smell a candle, or a campfire or rain on pavement. Do you think some day we will be able to smell things, because as someone create a candle they could add the smell of watermelon to it in creation?
I love the idea, but I think the physics of simulating smell may never be possible.
But an even bigger problem is touch: We can't use our body's largest sense in virtual worlds. We can't 'feel' our bodies there. Our body is basically a part of our brain - pretty inseparable - when it comes to feeling things and moving around. I think there are some ideas yet to try (we built one called 'the rig' that was actually the first project at Linden Lab, before Second Life), but this is what we really need to make work - touch.
What’s your view on the future of VR? And which company/companies will be leading this industry? Thank you.
VR devices need to get much more comfortable and enable typing at a normal speed for them to become mainstream. I think that will be about 5 years from now.
I don't think VR and AR will merge anytime soon.
I hope that many companies rather than a few compete to drive the future of VR.
What is High Fidelity? And what is SL doing to compete with VRChat?
VRChat is amazing to watch develop! But we are still early - there isn't yet a mass market for VR headsets.
I recently discovered data sonifications of astronomical images, specifically by SYSTEM Sounds and since then hoped to hear these images in 3D audio. Wouldn't they be a perfect match?
I'd LOVE to hear those sounds in High Fidelity. Put up a space? I'll come.
I led my school's VR club in 360 Film and Animation. This was a particular topic of interest to me, so tons of questions incoming:
Binaural recording (with a rubber head with ears and two mics) is the other interesting way to capture a space from the point of one observer. As you say, some sort of spherical array of mics is the other.
Yes, an HRTF applies a frequency convolution to a source as a function of direction to the source. It simulates (roughly) the passage of sound through the Pinna and head.
This transformation must be applied independently to each source. This is computationally expensive, and is where a lot of our R&D went - doing spatialization for many (hundreds) of sources for hundreds of receivers on a server.
It turns out that most people are pretty insensitive to the pitch angle. You have to be trained to have a good sense of it.
I've gone to most of the Oculus Connect events. Sure - connect me to your club.
Greetings Philip! There is a ton of 'buzz' lately about NFTs and Virtual Worlds. As someone who has created virtual economies from scratch and understands the desire of creators to control and derive value from their work, do you see a future where there is a standard, secure, privacy-focused #Metaverse currency? If yes, how can we avoid the negative ecological effects of current Crypto offerings?
Love the idea of NTFs as a durable way for art to move around and create more income for artists. Have seen similar things already in Second Life, which had transferrable/resellable secure digital assets from day one and a really big GDP in part because of that.
But... we can't use cryptocurrencies with Proof-of-work move them around because of the ecological impact. But of course there are alternatives to POW so that's great. Strictly speaking, a provable NFT doesn't strictly require a blockchain. Web of trust + signed proofs would suffice. But I love the overall progress so far and idea.
My question is: 1. Apple is developing AR technologies. Do you think VR will eventually replace AR or do you think they can coexist in harmony.
Thanks in advance
I think AR and VR are very different things. AR requires approaches that blend information with the real world and balance attention. VR is the opposite - total immersion. Both the hardware, software, and economic/social/moderation implications are very different for the two.
I don't think VR (if by that you mean HMDs) are essential to virtual worlds. We interact with virtual worlds with whatever interfaces we care to. We will probably never be 'natives' to those worlds through our interfaces, however good, btw.
What are three core features / capabilities that you would like to see built out in Second Lifw over the next few years, and why?
I'm not running Second Life, so I don't have specific answers there. Some good recent changes have made it faster at the simulation layer (things like faster region crossing) - I know that is something people value and that I'd like to see get faster and faster.
Where’s my stipend?
Just kidding! What’s the plan for SL in the future? Are you going to go full VR or will it remain essentially as-is?
I'm not running SL. :)
And, yes, Stipends in SL are a great example of the potential for UBI.
What effect do you feel 5G and lower lag times will have in these areas? Would you see a Surrogates type of existence?
5G can drop one-way latency by about 40 milliseconds, but 5G coverage isn't very broad yet. Once it gets there it will be a big help with smoother communication, yes.
Have you seen any interesting peripherals to encode spatial audio for hearing impaired or deaf audiences (perhaps in a spatial vibration apparatus)?
I haven't, but I would love to see more. Really great text captioning (which has gotten so much better with latest AI) seems to be one big tool for the hearing impaired - we are passionate about accessibility and are working on that.
I haven't been back in hifi recently, have you gone to an open world yet? that's the one thing I LOVE LOVE about SecondLife that there is a whole world to explore, not just individual rooms. Part of why I love Bellisseria so much inside SL.
Come and use our new API to build an open world using High Fidelity!
Does it work with the new airpods max?
Answering for both Airpods Max and regular Airpods: Not quite, but it's fixable by Apple and other hardware vendors. Airpods and other bluetooth devices shift to mono/16Khz when the microphone is turned ON. So this means you can use spatial audio only if you are NOT using the microphone. This is fixable! We're bugging Apple and others about it.
What is the theoretical and realistic minimum audio latency for High Fidelity? Could it be used to organize a virtual choir?
Within a large city you can use special tools like JackTrip to get latency lower than 20 msecs to enable musicians to perform together live. But browser-based WebRTC isn't fast enough yet to make this work for everyone. Typical 'real-world' latency is more like 100-200 msecs.
I see you answered a question about cryptocurrencies, have you had chance to check out Upland? I think it's got potential to be a big game like second life, what do you think about it?
I have heard but have not checked it out yet. Will do, thanks for mentioning.
More broadly: Is the future of the real world to be sold parcel-by-parcel to the highest bidder? We should be very careful what we wish for. I, for one, don't think that would move us forward.