Unpack the Stack w/ Harshil from Contentful

okay I don't know if it's going to record in my side as well but let's do that well we are live and we were just discussing if we can do local recording or not and I'm seeing a recording icon on the right so I'm assuming we're good so um I love it when a live stream goes live while we are still discussing it's kind of part of the game I guess um so welcome everybody and this is yet another unpack the stack and today's stack is going to be really fun it has to do with gaming and gamepads and you know what maybe not so my guest is harshil did I say that correctly yeah that's perfect okay so um I'm gonna be super bad with your voice with your name anyway so why don't you say who you are your proper name and then also what you do sure first of all thank you for having me my name is Hershel and I work as a developer Advocate at contentful for the folks who don't know about contentful contentful is a composable Content platform with an API first architecture but you're not gonna talk about contentful on this streams I'm just gonna spare you for that uh I do a lot of crazy things I love experimenting a lot with different kind of Technologies and today that's something that we are going to talk about like one of my crazy experiments uh with game controllers and web applications yeah that's the thing um I kind of like you said what what contentful is and every a lot of people know what contentful is right and I somehow tripped over your how you explained it like it's a composable not like all the words I I have opinions and I want to go into it but that's not today yeah thank you for saying it like that because that's that sounds like a bit of a Rebrand from recent times so I like that anyways no matter what so you're doing crazy projects I like crazy projects I'm doing a few on my own as well where I control my guitar sound with midi through a browser like all this stuff and so what is your crazy project that you want to talk about so I recently came across a web uh a browser API called Web hid and this API basically allows you to connect any hid that is a human interface device with your web applications so it's like my I have a PS5 laying around I have a controller which I'm hardly using so why not you know try to have fun with that and add it to the web applications and try to create a web game right so that's a crazy leader project who has a PlayStation 5 laying around what What's Happening Here what does that exactly mean it means that you buy a PS5 at a flash sale when it's almost too difficult to buy one and then you don't play games on that rather yeah you use it to watch Netflix yeah so it's it's a case of real life happen sometimes and then you just go along it's awesome all right so you so you're using this controller to to kind of do something on the web and so is there like maybe a demo where you can show us where you want to describe it what's the thing how should we do it sure uh so I have like two kind of games some one is still a work in progress the other is a fork uh that I just cloned it and I implemented the controller over here so I'm gonna try to share my screen yeah sure let's have a look I'm really excited now like these kind of things are always so fun I'm gonna have a few questions on how it was built soon we're going to look into that as well yeah great like generally on this unpack the stack streams I just want to go into the nitty-gritty and really like look at these code lines and see what that does um so let me know when you're able to share your screen I'll just put it up all right I will have to unfortunately restart my browser oh it's one of those MacBooks yeah that's all good we will be here I'll just take a few minutes yeah of course remember the URL and you'll be back soon and then um we'll um I'll just monologue it somehow we'll be fine I'll be back sure sure so I think this always happens right when you have a new Mac or something or you have never shared your screen with chrome it's quite um it's good for security but then you have to restart the Browser turn on some things um so I'm actually quite excited to see what harshall has built because generally making a game in a browser it's it's not super easy I imagine because you have this game Loop that has to run 60 frames per second and then in that Loop things need to happen and if you don't really pay attention to how complicated that could be performance could be horrific and so I haven't actually seen what he's made so we have kept that away so it's nice for the Stream and so next to that I kind of want to see if it's like maybe Commander Keen style like I'm dating myself here or like whatever right I'm really excited and he is back hello okay now this should work let's see if I do the all right thank you we're closing in present screen okay now this is the snake game which I'm not gonna show right now because it's kind of fully complete so it's good to show that we're not at the end okay so let's see a little bit of like your process and what you're doing here and maybe a bit of a demo in a browser to see what's what's up with this all right so this uh is a simple Mario game that I have created now as you can see whenever you work with web hid because of security reasons you always need to ask the user for the permission to connect the controller so here is a simple connect button yeah it gives me an option to connect it to a dual change wireless controller I'm gonna connect it and now the game loads it's pretty simple nothing fancy I can just move Mario back and forth and up and down with my PS5 controller and so is this over like Bluetooth or is it connected with a cable so right now I have just implemented the USB connection so I'm working with the USB but you can do that via Bluetooth as well sure sure and so is this already a game or is this kind of is this is kind of what what you have right now code wise at least this is what I have like code wisers right now it's still not a full game I'm trying to implement some fun stuff like this in here it is nice I'm pretty happy no but the thing is your scope is relatively small which means if we look at the code we can actually really start to understand it because there isn't that much yet right we can actually fully understand in the scope of this stream what that looks like I'm pretty excited by that so let's say you build something like this what do you start with what is the main thing that you need to get going uh I would say that like two things the first is now because it's a web hid API that we are using you need to have a device that uses hid under the hood so it does not necessarily have to be a game controller it can you can even remark your whole Mac keyboard because it uses hid under the hood so it just has to be an HID enabled device for that you just need that the other thing is a chromium based browser this is still an experimental API so not all the browsers supported I think it's the Chrome and Opera these are the only browsers the strictest people on the planet are now actually supporting in one of those apis I like that that's cool and so um in this system like most games if like I just when you were offline I was actually talking about this for a sec like there's a game Loop right in this because this is how your animation works because it refresh is 60 frames per second or whatever your browser can deal with I guess so is that what you've implemented here as well not really uh so what you're talking about uh you need to do that with the GamePad API because with the GamePad API what happens is you will have to set up a loop which would check if uh the state of the controller changed or not but with uh the web hid API what happens is it has an even trigger so if whenever you interact with that it triggers a function it runs a trigger and then based on you know the inputs you can then do whatever you want ah so that sounds like it's an easier approach to things it feels more like a defense based system rather than something that's always running and you kind of have to hook into that Loop and hope it works right that's a different approach so um would you like to show some of the code just to see how you got to this point sure yeah uh just gonna try to zoom in yeah we need a little bit of Zoom here I think there we go just good enough yeah yeah perfect I'm gonna skip this CSS part quickly go to the HTML so this is not a camera or a webgl or anything it's literally just dumb yeah nice at least when I see this I know how it works that helps I wanted to keep this demo very simple I'm also working on kind of a npm package for this so that people can then use it in react view whatever framework they want exactly so this is so this is kind of the base of your getting started system at one point that you will release exactly right all right so this was the sorry go ahead to connect no no go for it like I'm just looking at code and starting playing things but it's better if you present because you actually build it so again this was the first button that we saw when the page loaded and this allows us the user to give the permission to connect or not to the application and then we have the game where we're simply rendering the sky the grass Road and then this I am assuming is Mario over here yeah this looks like the mario.png here yeah exactly and then I wanted to add sound to this so I just added a sound over here as well now cool all the fun stuff is happening in here in the script part so basic JavaScript still over here so we just getting all the elements from the term and I've created some couple of states and I know what uh keys I want to use for example the left tick or the left X is the right access and now yeah the interesting thing is these X's they move both in the x-axis and the y-axis as well it makes it interesting and I'll try to also show how the data looks like later on in that game because it's it's a bit crazy down there as well and then yeah I can imagine when you console lock whatever when you moved it it's pretty crazy right it's it's crazy it's crazy and I'll come to that I'll talk a bit more about that about how the data looks like later on sure all right uh over here I have just added like four buttons but I'm right now only using one button which is the circle button which basically shoots that leisure and now we are actually uh started implementing the web hid API so the first thing we need to make sure is check if that API is available in the browser or not and if it's available that's when we use that API to request a device now this particular request function it actually takes a filter and in this filter you specify which devices you want the users to connect right so if I remove this yeah it's gonna return me an empty array and I don't get to select a device so the first challenge was to like get the vendor ID and the product ID for the device uh fortunately there is a website uh I should have got the link over here but there is a website I'll share it later on with you so that you can edit uh in the description where folks can find out the vendor ID and the product ID for almost all the hid devices out there so this can be let's just say you if you have a PS4 controller exactly because this looks like a pretty crazy code is there also maybe a list you can filter from not not really you have to manually specify which all devices you wanted to connect that's where it becomes a bit crazy because you don't know what devices your users have so you're basically limiting your users for this yeah the other crazy part about this is the data that this device is sent are completely different so let's just say I have a PS5 controller and a PS4 controller I'll have to map that data uh based on how they interact with the applications so there is no wow but there's like no standard way how data is communicated that's crazy why wouldn't that like so imagine you're like a PlayStation developer and you're releasing Call of Duty for four and four five you have a different implementation of the input device isn't that crazy it's that's that's what I've been thinking that's what I've been trying to understand like why interesting thing I uh because I already have implemented this over USB I wanted to try it with Bluetooth I was able to connect the device but then the data changes when you are using Bluetooth so the same code does not work for Bluetooth I need to map that data so that it works for Bluetooth as well holy moly okay well there's some lovely um content or data mapping exercise in your future there true and that's why I want to have that Library which handles everything for me and I don't have to worry about these things all right uh so yeah with the request device we pass on the filter and this filter basically tells like which all devices are allowed to be connected to the application once the user says like hey okay they allow the device we get a device array and then basically I am just hiding out the button and rendering the game the other thing is whenever a device gets connected you cannot start interacting with the application immediately you have to open up the connection with the device and that's what we are doing over here so even though it's connected the user cannot use it until unless you open up the connection and when that is there that's when we can have the on input report function there's another way to write this function which this can be something like uh window dot event listener and then have input report and then you can have the same thing going on in there as well so what this function does basically it's going to just take in the input coming in from the controller and handle all that foreign this is kind of just listening for whatever change is coming and then you get a well looking at the next line it's a u and H so it's a pretty big number I guess or a lot of stuff coming in a lot of stuff coming you're normalizing that somehow yeah okay yeah awesome uh for this one over here yeah I'm normalizing that I'm also checking the report ID because oftentimes the input uh might have a report ID and if there is no report ID you can use like zero as the report ID but this controller gives me a report ID so it's always good to you know have a report ID to check because there are again a lot of security issues that are still not fixed with this API it's always good to have this text in place okay but uh let's see I normalize it so now I get the values between minus one to one for both my uh joysticks and then I do the same for the buttons and for the buttons I'm just simply setting the value of true or false oh yeah and so basically you made your own little State machine on on the top where you say the buttons Falls are true and whatever the X and Y is of the sticks exactly yeah so for the people watching this you don't need react or few to do this you can actually just have vanilla JavaScript because why not it works fine uh I uh because you mentioned this I initially started trying this out with uh like different kind of Frameworks and I was like this is more difficult in like those Frameworks I'm just gonna simply do vanilla JavaScript and HTML and just do this exactly because I'm just now thinking like I'm more most proficient with few GS I would want like some sort of State machine to do this and then everything becomes reactive and then it actually gets quite slow when you add proxies to this whole mess right and it's it's a lot less easy and it seems like what you're doing now is you have the async away do you have all the modern stuff you need I'm kind of excited to see later on how you actually change Mario and what you do for that a disclaimer I am not too good with CSS I have like a couple of hacks in there and we're happy to see the hacks that's good uh all right so let's let's jump into that because that's the next thing so in the update uh position I'm simply passing on the access and this is what the code looks like let's just go there step by step so I'm first of all getting the value uh from Mario oh no I'm basically getting the whole CSS style because when I was working with this I was trying out on different screen sizes and I realized that is I haven't created uh or I haven't kept responsiveness in mind so at least when I'm demoing it it should look decent dinner which should not look like you know the screen is only this much while Mario just goes out of the screen as well so just take care of those things I have added a couple of uh rules in here as you can see the max weight over here and I'm checking the value for left tick over here so if the left stick X such like the horizontal horizontal layer yeah so if the left stick uh moves in the horizontal Direction the value changes to one and then I am uh updating the value over here so I'm gonna give you a minute to go through this code and try to understand it because I also yeah yeah so initially when you this update position when is that fired so it's fired every time I interact with the joystick so sorry the update position yeah it's fired uh every time I interact with the joystick but then it checks you know if the values are of the values fit over here or not exactly so initially you just stay you use use keep the state of your previous state left and top and then based on certain behavior of the sticks you just recalculate it based on the previous one yeah and so how are you you're replacing PX and all that stuff there's a few things you can probably do a little easier with the CSS API in JavaScript but I like the hacking approach but it might be a bit slow Global replace with the regex every time you move the stick I I would love to know that I've been trying to figure that out could not find it I'm like I'm just gonna yeah hack stuff around over here and make this [ __ ] you know what I wouldn't know by heart right now I I think I know what you're talking about and I did try out a bit of that I think but I'm might not have fully implemented that yeah it's because if you look at style dot left the one online 146. you can probably that can also probably be a getter and you probably get it without the PX for example okay but I'm not entirely sure but this might not be the moment to change your whole game and figure that out but I do like your approach because you're basically just you're literally changing CSS properties based on how much you move that stick around which is really cool and so how do you calculate how much the difference is between the previous and the next what uh what exactly do you mean by that well for example you have previous acts so you literally get the left value but then at one point you say left stick x equals or if it equals one do something but um after that this the left style that you're giving is again the same previous X but just two pixels more so is that two pixels more is that actually your acceleration yeah so this is like moving it like by two pixels right okay that makes sense okay cool so if you wanted to maybe make a variable to see how fast Mario is going that could also be a variable rather than just two pixels true true because a fun feature would be and now I'm destroying your whole app but if you go all the way to the right with your um you know with your thumb stick you might want to increase that performance value of how fast it can go I love that idea I love that idea oh my God I'm going to note it down and I'm going to try to do something nice because remember back in the day when we'd have like these car games where you can just go a little bit of gas or a lot and that's why you have the thumbstick right you can do these things and then you can do some crazy things like figuring out the acceleration Delta from zero to the end because you can go slow but you can also go fast you might want to do some craziness there um just a little bit I'm going to try it out it seems like a really here we go maybe just put this on GitHub and somebody from this from our audience will actually give you a pull request it's already on GitHub I'm just gonna quickly try to ignite the link but it's already on GitHub so if someone wants to try it out they can just go there forget or just run it as well I'm sharing that with you maybe you can later on early to the notes as well yeah so I'm just looking back at this drawer I have here and I know there's a PS4 controller in there so who knows okay I'm gonna put your link um here I'll put it in the Stream all right while you do that yeah it's gonna go ahead and oh thank you yes of course right so I'm doing the same right so we've figured out the left under drop right yeah so that's what is happening with the top as well and the shoot part is like more more ugly than the the move position part because this is like I spent a lot of time to figure this out it never worked out oh boy but I learned a lot of CSS I learned about request animation frame which was fun oh yeah because in my opinion or at least how I thought this would work is you have the whole thing in a request animation frame it is recursively loading something and then the position the left position of Mario would be basically just a number that is changing every time regress animation frame runs because it could go to 100 frames per second based on your screen and so that's that's a different approach to it and so whatever you you don't actually then listen to the thumbstick you just listen to a number that comes in 60 times per second so that would be very different than actually waiting for input and then changing something so I'm I'm excited that you did took it this direction because this is actually a bit simpler but now you have to put request animation frame inside of this thing which then makes it a bit harder because that's supposed to always run or this one is just whenever there is a moment the process processor has space it will fire this function right right so whenever like all right so let's click through shooting okay yeah it works for shooting right because I just want to show it once that's it yeah so that's what is happening with shooting I know that there's a lot of things that can be improved in over here because every time I'm creating a new element creating like giving it the class doing the styling and then animating it which I feel like can be improved but I've never got a chance to do that well if you think about the the CSS and the Dom aspect of this looking at this with my old school hat on how I would have done it I would probably make a shot somewhere already in the Dom that just exists and then I would give it the bomb Clause already right and then use CFS variables for the left and the top calculation stuff and the only thing you'd have to do when you hit you know if Circle and something's going to happen then the only thing is to do is you just change two or three cacs as variables that are on the root of wherever your game is and that that CSS class will already listen to these and CSS in JavaScript is a lot more optimized than for that because right now you're creating it let's say you're like bashing the button to shoot all the time you're creating a new Dom node for the div you add the class it has to apply the CSS but it cannot really apply all the CSS because you're then calculating after applying the CSS and then you append the child so you're you're adding the CSS in memory and then adding the child but then again you set the opacity after and then you do a translate so there's lots of repainting actions going on in a game like this this is fine but if you start to do more I think you're let's let's say you do it on your phone right it's going to heat up for sure I kind of like it [Laughter] I mean so many directions you could do this right this worked fine for a demo this works fine and so oh perfect and so what is circle here on line 169. so circle is again like a button that we are getting from the state oh of course of course so we check if it's fresh or not and if it does we do the shooting thing exactly makes sense and so how do you calculate the speed of the of the um of the button or the bomb there's no calculation of the speed for that right now it just no so so I'm looking at you just look at the original left position and then you add 85 to it and that's the speed essentially but you add it as a Translate yeah so uh what over here I'm doing right now is because Mario can be at any position right so I want to make sure that yeah the laser actually comes on from where Mario is right that's why I have the styling over here I get my used position then I give this position to the I don't know let's call it laser for now yeah yeah and then once it starts from that and then I'm translating its position okay so this is actually more performant than I thought it would be because I thought you were calculating the left position for the shooting but you're actually using a translate for that which is a lot better yeah the translate is basically doing the moving part this is just calculating where Mario is and setting uh the laser from that position that's it yeah and on that Line 183 you could even add like an easing I think so it starts fast and ends slow or some fun stuff true yeah this is cool man this is like I love to see that you actually went really relatively basic because we are now 30 minutes in and I guess if you know HTML and CSS and a bit of JavaScript you understand this thing so tell me why do you need to request animation frame here okay so I created this a month back I need to refresh my memory uh but basically what is happening is because I want to add animation to uh this particular element and I'm doing it uh within JavaScript I think this is that's why I need to do request animation uh frame because again if you see over here it's using the transform Style um and we transform here I need to refresh my whole GSS skills now oh I'm thinking the fact that you use a transform in CSS should be quite optimized what you can do in CSS is say this thing will have a transform on it soon then you can you can look at that there's an extra property and then I don't think you need to request animation frame I think it should probably work but to be safe this works well because this kind of request animation frame just looks for the first moment that the browser has or the processor has space to just fire it off so this is the safest you can be good fun I didn't realize I had so many opinions on this suddenly I have it's kind of fun I love that uh like we got roughly 10 minutes I want to show the other part of this because right yeah exactly we're just seeing how you know we can receive input but on the other part I'm gonna try to show how to send input to the controller as well oh nice I'm gonna stop this one I'm gonna come to another one now this game I haven't created myself I just found a repository on GitHub it is a snake came uh spelled on one knife server is it on come on okay yeah it's you it uses JavaScript so I was like why not add a game controller to this so it works it's kind of in the same way perfect I'm gonna do start game over here and for this I already just before the live stream I tried out so right now I'm just using my controller to play the game and oh wow people can choose but now the LED uh changes the color of the LED changes over here nice when the game ends and it also Rumbles so I'm kind of giving the user feedback that is super fun let's see if I can find the code for this so again it's kind of doing the same things I have dxx is set up over here now over here I'm just using them so I did not configure the buttons um start game end game show let's try to find it okay area so whenever we do the start game we connect the device like we saw earlier uh we passing on the filter with the request device function it's not connected we throw an error to the user and then over here we are simply reusing the functions as the states that are available just checking except for the values and updating those values and so while the snake moves can you feel it Rumble is that what you build no I haven't enabled uh Rumble when the snake moves I've enabled rumbling when the game ends okay cool and this is the this is the place where the output happens on the controller so over here I have another function and as I mentioned the data mapping in this is like very tricky so I had to try out a lot of different things to actually find the correct configuration but the way this works is I need to send out a report ID and then set out some values for different things now because I want to enable Rumble and the change the light bar don't ask me how I found it it was a lot of trial and experiments with these values I know the values lie between 0 to FF that's 0 to 255. I just want to find out I just need to find out the correct uh data point where this actually very nice wow okay I forgot this I'm also setting up the mute oh cool sorry go ahead hey you might want to create an npm Library just to map these things so nobody else has to go through that's crazy like finding something out of like 400 values or something yep that's that's as I mentioned still uh work in progress because I've almost done with the USB part but the Bluetooth part is a different story so I need to do the mapping again with the Bluetooth thing yeah that's still a bit tricky now and then uh for the last three bits sorry oh no I'm just I was gonna ask what report is but I can see in line 403 that you're actually just sending a report to the device and that has like this fun un8 number in it yeah yeah so uh when we talk about data in the web hid API space we actually use the term report instead of data and this report is basically like binary data in itself and in web hid there are three different kind of reports the first is the input report that we saw with the Mario game so whenever you would interact with your controller it would send out an input report and you can do stuff over there the next thing that we are looking out right now over here is the output report so whenever we want to send out some uh feedback to the user we use the output report and we use the function send report for that so that's what we are doing over here and then there is a third one that is called feature report which allows you to like both send and receive input and output which is again a bit complicated to implement but web hid already gives you this functions for that but then the data mapping is a whole different story and there's a controller like this support like rotation movement and stuff like a switch controller would or do these not have that that's a good question see this is it does have that I know I I remember playing a car racing game where I used to control the car with the controller so it does have that yes uh and it does also like the hid also supports that the EPA supports that but I haven't got a chance to look into that yet it seems like there's a whole world of stuff that we can still figure out here so maybe as a last thing can you show um like the on input of this to like just console log just for people to see whatever comes out of this controller sure uh let me do that with the Mario one because over here I know where exactly the code is yeah sure it's not oh yeah you might have to restart your server also server is running okay yeah all right here we go connecting it over here and now you can just throwing me like streams of output coming in or the input coming in well that's a lot okay so what is in this data view exactly it's an edit buffer and if you want to look more closely this is what it kind of looks like I have never seen this I love it and I have no idea how to read it it's just crazy that's that to me that's me working with the API oh no there must be some sort of abstraction on top like they did with webgl and 3GS right it has to have something because this is just craziness so they do it in this array buffer because there's just too much data for JavaScript to otherwise parse I guess uh kind of a scary part of this because I was going through the security issues with this particular API and because uh there is no defined structure for the data you can send like any kind of data in the binary format which can harm both your application as well as the device as well so that's why a lot of browsers like for example Mozilla is not in favor of supporting this right now they're like we need to have proper security implementations for this before we roll it out exactly well dude I want to thank you for showing all this off because I've never seen this and somehow I thought it would be very easy but it's clearly not but I do love the fact that you that your code is actually just super down to earth very simple and that in the the time of this stream I have not understood how to do this and I will probably be able to do that myself exciting times love it I'm I am excited to see what you build with this oh I just need to find some time and do it so um is there anything you might want to add finally to um our lovely our lovely viewers I don't have anything in mind but yeah folks uh if you are hacking with the web hid API and if you build something with it I would love to see because I'm also still learning about the API so feel free to you know share it with me this is me on the internet yeah me too but yeah you can find it on the internet over there uh and I would love to see what you have been building with web hid awesome thank you so much for joining and everybody look at contentful as well because that's herschel's CTA here well not really but I'll just hand it to you I know it's fine um thank you very much everybody and I'll see you in the next stream cheers all right thank you have a nice one