Programmatically import data into the Hygraph Headless CMS

yes we are live yeah apparently the timer went down and then I did not click the button I've had it once with like a famous guest on we were thought we were live for five minutes until my friend sent me a message where is your stream and then it's just one button make it one thing though I have YouTube opened on the stream and nothing is happening no but there's a 30 second delay there's there's a delay ah all right we just went right in live while we were chatting about things this is how we do it at this stream this is Vibes chill we'll learn something together so today we are here with the HRA stream with Nikki and he will tell you his real name in a sack because I just cannot pronounce and turns out I can also not spell if you look at the social post about this it's wrong my bad true anyways um we will be talking about how you can get data from other systems into high graph because we get tons of questions about this in our community like hey I have this older system now I want it in the new one and then do I have to do everything in the UI or my code what can I do so we'll discuss that today and so Nikki take it away first what's your proper name how do you pronounce it where you're from what do you do all the things yeah well to be fair you never had problems pronouncing it it's just how you type it but that's fine so hi my name is Nikolai Nikki Nik whatever you want to use it's absolutely fine I I'm used to any variation of it so feel free to improvise um yeah I'm a I'm a Solutions engineer at HRA my job is basically to show customers our product and make them see the value in it meaning I work in pre-sales mostly uh I've been a developer for the majority of my career I honestly haven't really really written production ready code in a while and you're going to see that today so don't be too too judgmental it's you know just to make things work and see how it actually can happen but yeah are you like Nicholas like Alex is saying that's like the French version of of our Nikki isn't it that's actually a very fun internal joke because this is how our head of product is calling me for some reason but again I don't judge I'm just gonna let a slide because on the stream you can say whatever you want man we're good true true we're live and it's recorded so exactly it's on the internet it will never be deleted the magic exactly true anyways cool so we're here to talk about how you can migrate migrate to hgraph right and as Tim said this is something that we always hear when we start talking to a potential customer like they're coming from a monolithic system or another head system like how do I move our our data into hgraph how does that work do I have to start everything from scratch do I have to type it in manually can I build a scheme of via code you know stuff like that and it's Al always a very very heavy topic to discuss because it will vary right you come from a system you have your own things built there if you've been there for a while even if you're using a monolithic system that usually have you know predefined structures down the road you add customizations you you add new content types and so on and so on and all of those can be you know different across different customers so it means you still have the challenge ahead of you to customize everything that you're going to be importing yeah to fit your new narrative so this is what we're going to go before we go in what is your worst experience with looking at like an older Legacy system that you had to import data from o that's a very tough question to answer but let me think I'm gonna say and this actually is not related to hgraph at all but it was back in the day back in my developer days we needed to build a system for you know it was a scoring system for gymnastics and we needed to import the data from an older data base and it was a postr SQL and we needed to import that in the Microsoft SQL server and I have to say even with the tooling that Microsoft provides which is excellent by the way it still took a tremendous amount of time to actually do it and it was mind-blowingly bad oh boy but yeah actually mine was I have similar ones in the past but the most recent one was when you and I were on a prospect call and they were showing us their CMS we don't talk about that come on I didn't say which Prospect I didn't say which CMS but it was mindboggling okay I'll leave it at that and I want to just quickly say hi to Jamie who is here he is old school high graph or graph CMS I don't know graph CMS hey Jamie I actually removed your name from the thing that we're going to import today I should have left it oh no is that code that he also worked on that we're going to show today well he was a part of the initial demo when I initially oh come on we have him come on the stream and then happy to have him okay he he he'll post something in the comments and if he does I'll send him a DM perfect if not we're good so you kick it off and I'll see um what Jamie has to say on all this perfect you want some you want your stuff on the screen perhaps probably there we go there we go cool so let me switch to my presentation so we can actually kick this off right so as I said we're going to talk about how you can migrate content to hgraph and there are several distinct steps that we found out that you need to follow and we're going to go through them today for some of them we actually have a live demo so that's going to be interesting the first thing you need to do is kind of know where you're coming from you need to examine your current system and get an idea of what the data format U you're keeping your information in and how you can migrate that to hgraph which is where creating the schema in hgraph comes in this is where you're going to examine our content types how they relate one another and so on and so on we're going to we're going to see that in a bit then is how you actually plan to execute that this includes writing the scripts for the migration having the the priorities set straight in terms of what goes first and so on and then finally and this is probably the most important part is how do you you execute the whole thing so we're going to see that today as well so you say we have to think before we do normally I would suggest yes um but speaking from experience it's not always the case even I don't do it yeah I mean just saying I built this today how about that yeah you you can you can guess how many hours I spent thinking about it yes you I remember I saying I saw told you hey do you have this thing still she yeah but it's broken yeah we're live streaming tomorrow no fix it but that's we're living on the edge right we're developers this is this is the way ah there was a let's just present you know act like we do no act like we say not as we do is is that the one exactly do as I say not as I do there we go okay let's do this perfect so where do you start easy you start with examining what currently have you don't even have to think where you're going to you need to have an idea of what you currently have so where are you migrating from there are pretty much three distinct options that we see and the first one is the classic CMS you can think of that as the mons out there that can be WordPress umrao sitecore stuff like that even though sitecore already has headless offering but we don't talk about that let's call it hybrid let's call it hybrid that that's fine so all of the challenges when you are m migrating from a classic CMS there are several several things that you have to keep in mind and it's basically your content content scheme is not necessarily optimal it's optimized for you know that particular CMS style so there might be some things that you might need to reconsider there are some hidden dependencies nine out of 10 cases that you don't even know about so it will be something that you need to examine the export formats might be weird so you might need to translate that somehow and then the API standards might be even non-existent like Jamie wrote about the XML thing if you have XML you're a lucky guy I've seen stuff that then you can do xslt and then you can add CSS so that classic CMS title is in the center because my front end eyes are are crying right now I'm sorry dude I probably should I don't know why it isn't but yeah let's not touch that now we're fine again this today I did this today come on all right second option you're migrating from a different headless CMS this is our favorite use case because it's the easiest one why is that easy well you have a very high chance that the schema will actually be very compatible to us because pretty much all the Headless systems out there currently have relatively the same way of defining the content structure there are some caveat here and there but yeah for the most part you're pretty much all set with minor tweaks one of the things that you should be careful about in that case is whether you using some proprietary features that might be native extensions to that particular platform it might be um a particular integration that you've built and you're depending on that you need to check how that will translate and think about the future as well again same considerations can you export the data and what are the API standards that's always something that you need to think about and then the final case is where things get funky this is where you have something very custom built that has evoled through time and it's basically homegrown and it doesn't conform to any standards now every developer did that once right exactly exactly I have three of those behind my back I certainly did even recently oh yeah yeah I mean to be fair in some cases you don't really need a run-of-the-mill CMS right you can go get away with something custom but when we're talking about Enterprise features you know bigger projects this as much as it's comfortable in the beginning once it grows you get in a world of hurt when you want to move away from it yes yeah so we're not GNA even mention that today because it's a very custom use case and we can spend an entire other session talking about that but yeah that's the third and final option that we see cool so where do you go from here once you know what your content structure is you need need to start thinking about how is that's going to translate into high graph what are the structural elements how do you need to move that there so it's still works can you potentially optimize something like if you have a oneway relation some somewhere which is by the way foreshadowing that's exactly what we're going to do today if you have a one-way relation somewhere does it make sense to make it a two-way relation maybe it does um and also how are you going to move the assets are you still going to keep your current asset manager or you going to move to to another one those are all things that are very important even though they don't really require uh action at this point of this of the of the process and then the final and this is a very important thing is how you can get that information out of the current system is it going to be via an API can you export it in a Json or CSV format how will that work this will basically dictate how you're going to build the scripts to move to yeah and we're not probably not going to focus on that today we're just going to start with something we have we are essentially going to assume that this part has already been done uh and we're going to work with the end product from that yeah exactly so we will have some CSV file but exactly if someone else might do this they might in in their code first query their other CMS maybe contentful or something like that sorry I didn't want to name drop that but here you go and then you get all that data through SDK have it in memory and then push it into high graph right you can do that in one file unless it's like something super complex and you do it in way more it's like depends on the size yeah you you're right I honestly don't like the idea of doing that in the in the same file because it's always PR to errors and at least for me like the developer in me tells me that I need to split my work in chunks so I know where if something fails I know that this chunk is the problem so I need to you know go and fix that where if you I put everything in the same place you get in a kind of a tricky situation where you need to determine where exactly it breaks and it's ah it's not great so that's probably that means I'm more of a hacker than a developer nowadays but we're all hackers at this point that's true though all right now here's what we're going to do today we're going to move a blog post website from an unnamed system that Tim might have dropped in just now to HRA and the structure is basically we have a blog post that has a title content and an author and we have the author model that has a name and a portfolio link and they're connected by a reference from the blog post to the author right this is our current situation you have a question no I'm assuming that so this is more of an observation I this is a simple approach this fine but what I would love to do is click on the author's name and see the block posts and right now I don't think we can do that unless it's some magic do we do some magic en code not easily and this is exactly what I mentioned a second ago like if when you see what you currently have in your system you might start thinking about optimizations yeah because even with the current setup you can potentially still build that somehow on the front end so you when you click on the author you can get all their blog posts oh yeah justar all the blog post and all the authors and filter by the author yeah it's a little silly but we could do it exactly it's not optimal it's much easier if you actually build the other way around and you have a relation from the author to the blockbox well yeah sure right cool so this is where we go to creating a new schema and I'm gonna speed through this and then we can jump into the actual thing yeah let's let's let's show some code on the screen soon yeah so first thing is you need to design the structure and there are two ways well several ways to think about it the first one is whether you're going to use the UI to build the schema or you're going to go with code first what's the difference the difference is how you want to involve evolve that schema down the road whether you want to replicate that project down uh down the line whether you want to have traceability of the changes um that that had happened incrementally and you can think of this as the classic code first approach where you have different migration that you can then execute against the blank project so you have you know traceability of everything this is sorry to to interrupt here um you said the word code first and I didn't expect you to say that but I come from the side core world where code first is like a thing and you just gave me shutters down my spine when you said code first based on the experiences I have because we all want to do code first but then when you start it's it's so hard to do sorry this is one of those things like that that you hear after like six years you're like oh crap I have so many horror stories SP coming from Cod first approaches that are they're mind-blowingly horrific I'd say I can imagine doing a data import like this code first totally cool maintaining your schemas and models in code cool but doing everything code first including for Content editors that then have to make new models and stuff in code yeah no the the the the the cool thing about quote First is that it's great to start with but then nobody wants to deal with the actual end product that comes from that first appro the cool thing about hgraph is that even if you go with the UI essentially what you're going to be doing there will translate to the exact same thing if you do it code first every single thing that you do from the UI you are doing from the from the code first approach so we're actually going to show both of those today so you you can see what the difference is so with that since we already know what we're building essentially we have that current structure and we essentially want to kind of change it to have to we reference and we're going to add the blog post collection in the author let's go and see that in action so switching to the project that we're going to use essentially this is a blank project that we're going to use for data import the schema is empty right we have just the system models here and we need to create that author and that blog post so right first I'm going to do that in the UI so we can see how that looks like and then I'm going to show you what it means if we do it by a code so we're going to start with adding an althor add that model we're going to add their name use that as a title field and we're going to make it unique and we do that so we can query the where in graphql I learned that recently from our PM guys forgot yeah I'm gonna add their portfolio link and we're pretty much done with the author okay so this is this is not the two-way binding yet right not yet we need the other model first we're GNA build yeah we're going to build a blog post going to add that title for it again uses title field now the content of the blog post I'm just going to keep us marked down for Simplicity reasons you can have that as Rich Text um which will give you a lot more customization options but for this particular demo let's keep it simple let's go with markdown we're going to name this content at that and this is where we actually going to make the connection and this is where hgraph is extremely powerful powerful so we're just going to go down here and we're going to add a reference and maybe I should zoom in a bit we're going to add a reference from the blog post to the author we're going to make that a two-way reference and we're going to allow multiple blog post per single author right this is the end just a quick Interruption Brian was almost ready to start bitching on the fact that we didn't or the rich text that we used markdown oh so whole stream on it exactly we'll make another stream specifically for the Rich Text Editor and then Brian can come and talk about why that is important all right yes yes cool so we build a connection from the Block post to the author and the cool thing that just happened because this is the original setup right we have the block post with those fields but since it's a two way reference if we go through the author we're going to have the collection of blog posts so we don't have to go through that mental gymnastic of you know querying the block Bol filtering by author and so on and so on cool so this is how easy it is to make it from the UI right building the schema I did it in what two minutes give or take that was yeah relatively fast pretty much the same thing you can run via code uh and it's done via our management SDK and now we have the basis for that we have the creation of the author already done in this cool project and I hope it's visible well on the screen yeah it's it looks all right there Post in the in the chat if if anyone likes this more zoomed in but is we I can see it well and my screen is like smaller so we're good I think okay perfect cool so the current state of things is we're basically using our management uh management API SDK sorry in here which maybe I should scroll in a bit there this is the one that you're going to use when you start building the schema bya code and we pretty much support the whole creation process from here like we're creating our author model why are those popups there we go we're creating our author model we're creating the name field we're creating the portfolio Link Field this is the pretty much the basis of everything we're going to do yeah now for this to be a bit more interactive we're going to build it run time so I'm just going to open my cheat sheet and we're going to start putting stuff here so the first thing we're seeing in here is that there are some stuff that we don't really need like the name is made multi line which we don't need so we're going to that we go we need to make the name unique so we're going to say is unique and so is this type will it just give you all that stuff it is yes most of it at least it's typed and we need to make thistle oh the title field yeah exactly cool yes exactly cool the portfolio link seems to be fine and then this is pretty much it for the author what we need to do now is create the blog post and this is where I'm going to start copy and pasting so we can speed through this so so just a quick question before we go into this so right now this client. creat simple field is basically just a global function it's not in context so how does it know which schema is adding this to this is a very good question and it's basically your configuration for the client right when you're creating the client that comes from the SDK you need to supply the authentication token for that particular project and the endpoint that you're using and in this case I'm getting them from mym file I have exctly you just put them in there sure exactly but then you make the author right how does the client when you add a simple field know it has to add it to the author model that's a very good point so once you specify the author everything basically works on the API IDs uh ah you link it together with IDs correct so once we have how can I stop those problem so once we have the AP ID for the author come on you basically reference that same thing here and it knows that that name is going to go there cool that's easier than chaining this stuff together yeah initial version yes the initial version of this was actually the chain way of doing things and it was kind of a nightmare if you had something a bit more complex yeah exactly this gives you a lot more flexibility exactly exactly cool okay clear thanks let's build the blog post model so well it's already there so I'm excited to see if it's throws an error or not it will we need to clean the scheme of first but I was keeping that as oh sorry let let let it see an error please let let's see we'll see an error let's see the error cool so this pretty much creates the model for the blog post we need to make the title for that which is essentially the same thing we did for the author I'm just going to copy and paste that again is title yes you know is is almost like copy this and change the model API ID right precisely which is pretty much what I did I'm literally learning the SDK while he's showing this stuff Nikki is showing this stuff so hands me asking these questions so you can do the the next life alone oh sure it's a lot less fun but yes sure true true so here is um something that's interesting and we recently stumbled upon this when you're creating a string field we have several different options for right if we take a look at the schema the string type has a single line multi-line markdown or a slug type now these essentially down down below are all strings right the difference is how they are visualized in the content editing section so if you go to blog post and create a new one this is a string this is also a string but since this is rendered as markdown you have that cool you know kind of which text editor yeah exactly Brian please don't kill me uh but yeah so in the configuration of the scheme of cod you need to somehow configure this in the code so the way we do that is by providing two separate renderers one is for the form meaning inside here and let me just type in something so we have an entry and the other way the other renderer is actually in the content table which is here oh all right yeah now good to know this is where my gripe with d SDK comes into play um these things should be an eneration but unfortunately they're not so you need my next question can I not just import these things from the SDK and have them typed and everything I I'm currently pretty much pushing the the de team we have Alexi in the Stream he's here oh let see I think he's here I'm so happy you're here so this to be an En please make it all right let's let's can we run it yet um one more field and we will so we pretty much currently have the block post with the title and the content field yeah this is where the most important field comes into play and this is the relation field and I'm actually going to type this out because it's interesting we have Alex saying that he thought it was an enum already well there you go it's not it's not apparently or maybe just not exported um through the the export of the SDK itself maybe they exist internally they might actually exist yeah that's a good point maybe it's just the export it could be cool let's create that relation so gonna make oh yeah exactly this I've not seen before cool there we go so now we're going to do that we need to create a relational field which is different than the simple field thingy so we're gonna start with that and haven't called it for real in a long time please bear with me you need to specify the API ID for this so we're going to go with the relation From the Block post to the author okay we're gonna add uh what was it display name wait this is not graph Nikki we going to add display name which is going to be author capital A there we go and then we need to specify what type of a relationship this will will be so we need to add type this is where this fortunately is an enom so hey there we go there we go so it's going to be a relation type field there we go and now this is where it gets interesting on how you create to we relations I can leave it as as as is basically and it will create a relation to the author but if we wanted to make it uh to a relation we can configure a reverse field right and in here we can specify the AP ID of the other model which is going to be sorry AP ID of the field in the other model which is going to be blst okay yep and this is not graph I need the commas are those popup still there yeah this just autocomplete I guess in uh play name we need the posts there we go comma that and what else where is this going to go model API ID this is going to go in the author specify that and this is where you control the cardinality of the of the connection meaning if this is going to be a one to many many to one and so on and you can say is list and if you say true that means that there are going to be multiple references in the reverse field meaning the blog posts okay so this is important um spicy question outside of the typescript of this where is this documented Alexi are you here so I'm going to be sending people with a question to this live stream recording then no um that we actually have a document it's it is in the it is in the docs and there might be stuff missing here and there which we try to to complete every single day uh but I think everything that I'm showing is currently documented quite well okay I'm going to look for that while you continue and I post it in the chat please do please do and just so everyone knows I I asked this question because I knew documentation is hard but we have some amazing people working on that and lots to come like Brian is saying how about this lots of docs to come and new things because this is stuff that we see so often true now one thing we forgot to do is to specify for which model we're creating that relational field so we need to add the model ID here which is post with that I think we're ready to run that whole migration so for the folks who wants to go to the API reference I just put it on the screen you do perfect that's it and you can post it in the chat so it's yeah I just did there we go cool so let's run this and see what happens and your prediction Tim is it's gonna fail because you just made one exactly exactly question though that run function where did you get that from I copy pasted that yeah but is it like an npm package or something um it's coming from the client essentially that is nice Yeah so basically you can run in this case it's a migration when we go to the actual data uh the data insert data moving part is going to be imitations yeah okay so this is just oh so the Run comes from our client that's awesome yes okay cool thanks cool so we're just going to run this file and see what happens prediction is it's going to fail right shoot exactly and there we go we have an error it's already existing both exactly both of those models are already there so what are we going to do we're going to delete those and by the way you you get that is super cool that you just saw that yeah you'll get an error in the UI as well that is super cool right so let's delete that let's delete the blog posts there we go let me clear those now something that I didn't mention when we started building uh the schema through the UI is that it's always a good idea to have a backup environment before you start this whole process so you would go to Project settings go to environments and you would clone this as a backup especially if you already have some data structures there that you don't don't want to compromise I'm just going to name this backup going to clone it with content obviously and while that's happening actually let's wait for it because yeah I have to be safe here yes yeah yeah Brian we shouldn't live on the Ed I'll just put this on the screen for a second yeah we do live on the edge but we are also praying to the demo Gods right now here we go there we go perfect cool so let's try that again let's run that same migration so when you run this and there's a result can you zoom in a little bit on the the the console here we go so we can actually see what happened so we finished and it worked that's it and if we go back to whoops here go to the schema Tada there we go have the author with the single line unique title the same way we wanted to have that the block po posts multiple Val that's a two-way reference how about that exactly quick question this multiple reference thing that that you see there in your previous in the author one that is because you use block posts with an s in the end no right no that's because we did this oh there is list you just told this okay cool yeah yeah cool right so now we have the schema right we have the structure in high graph we we pretty much ready to start shaping nice the actual migration back so as we said in the beginning we're going to treat this as we did our homework we exported all the information from our other system and we have that in some kind of a format right now and the way we did it is we have a CSV file that's basically containing all of our posts first it has the name of the author then it has their portfolio link then then it has the title of the specific blog post then the content of that blog post so this file was generated by someone who coded something and took some data out of a system stored in this so we're not doing that this stream no okay we're going to assume that we've already done that now one very important thing when you start about thinking moving the data is where you're going to start and our suggestion is always to start with with the most reusable models first so in this case if we look at the whole schema that we had we're going to have one author authoring multiple blog posts right which means that the author is going to be reused across multiple different blog posts and it's reflected in this export right we have me authoring several posts dim authoring several posts so if we go with just inserting those lines directly that means we're going to have duplicates over the author model or we're actually not going to have duplicates because we marked the the name as unique we're going to get an error get an eror right away exactly exactly so normally the way you would go about this is you would build a mutation and the very cool thing about the platform is that you can play around with this from the API itself so we can just go ahead and build uh a utation here with create block post so we're just going to create one see what happens if we try to recreate it again so basically we're going to build a blog post with the my second block post title we're going to run this cool we got the result Let's see we have P teal as an author we have the blog post here perfect that's fine so what happens if I try to run this with my third blog posts third content and if I try to run this with the same author yeah not no we're going to get an error because the name of the author is yeah so now the system tried to make another author with the same name which is not possible because it's a unique field correct all right so this is where we would switch things around and we're not going to be creating blog posts but we're actually going to be creating authors and for each author we're going to create their block post yeah that makes sense so this is the thinking part that people need to do when they have a data model and how do we put this in yeah exactly but then again we're going to run into a very similar issue here so if we go with create block posts and we're going to get that data field here let start doing that the title is going to be it's first post oh you might want to do a back slash or double quotes quot go title and content how far along do you know Laura mson uh to about here let's go with this fine now you do and now I'm just doing the same thing though like come on Nikki we we should have started with the author right we exactly that's the point content come on all right let's do that again data so when we film videos together this happens about 16 times you're much better life dude you're like way am I we should just do this live then all right exactly name yep return the favor here we can do it like this is an mistake I don't mind because it feels like I'm part of Asterix and Obelix as the the comment how many ways can you sp can you spell Google wrong yeah exactly all right this is where this is where it gets interesting so I can now go ahead and basically add the block posts right I can do that and it will basically be either create or connect since we're importing it's going to be a create mutation and it's going to be pretty much the same thing uh so we need to add a new blog post which again formatting kind of lud me here so this is going to be Tim's first block post content is going to be first post right now if we run this I'm missing something you might want to click on prettify prettify maybe that works do I have PR oh no so if that doesn't work it means there's something wrong yeah I have an error somewhere what is not corresponding there oh I need to close another curly bracket there we go so we have a question in the chat let me just put that on so after the benefits there are there benefits to creating this inline versus creating all authors and then creating all blog posts there are and it's pretty much in the simp simplification of the of the script now there are actually benefits to both approaches depending on what you're building if you have something extremely complex I would do them separately hands down I would first implo uh import all the authors uh and then import all the blog posts but then you run into another issue that you have to S and that's extracting and mapping the old IDs for the authors versus the new ones and mapping those to the yeah then you have to do a query in between to get the new Authors link these to the old authors and then add that to your new query for the block post holy crap my bra just made a leap but yeah there we go pretty either that or you can save the IDS as you import them on the client and basically have a lookup table locally either way you you're still going to have to do some kind of a lookup yeah that being said one very important step we're skipping today is importing assets and this is where you would do them separately because they use an entirely different API to to upload right so you would have to do that mapping anyways so to answer Brian's question um there are benefits to doing it in line sometimes but you can also do them separately depending on what your use cases right the prettyy thing didn't do a very good job to be honest did the worst job it actually uglified it so let me re yeah Brian the answer is as always it depends should we maybe make that the name of this stream series please do can you right I'll make I'll make a podcast myself anyways let's run this let's see if this works let's run this so first okay we created that author perfect which means if we go to content we should have Tim as the author there we go yep and we have your Tims first BL now let's go back and create a new blog post for you so it's going to be Tims second right and here we're going to run into the exact same problem if I run this exactly yeah is cool so this is where the magic of high graphs API and SDK comes into play you can use a different mutation called absur so so this is not something other cms's offer often right this kind of stuff not often no it's normally the way you would do that is with manual logic and you would execute the let's say the background thing that this absur does manually but essentially what it boils down to is it has two options oh three well two if you two if you have the model that you're inserting it will update it you're going to use this section if you don't have it then it will create it so what we can do with this it's going to be pretty magical is we can say if the author doesn't exist we're going to add the name important to spell it correctly because we're going to use that for the filter boy po link I love that autoc completion and this is where we can add the block posts as well and again here we don't worry about connecting existing ones so we're going to create a new one say title why is this formatting so bad Lexi you're here right need better formatting tools title second content BM go cool I'm actually just going to copy and paste that for for the update because we don't really have different project there okay cool so if it exists somehow and it figures that out based on some of these fields it will update and otherwise it will create but it it's it's it's all depending on the where Clause I imagine exactly and here we need to specify the name that we're going to be checking against so in this case we need Tim and we need to spell it correctly so we're going to go with benex there we go and now if we run this we should get there it is so if we go to contents what's our our expectation we should have Tim benck but he should have two block posts nice second actually let's spify this because I'm kind of getting lost yeah so everybody always does that that's why in our new um interface be done by default G to be a lot more smooth yes cool did I just catch a bug don't say oh no if you want to do this it this should be Sav in your local storage until you come back next time or you have to really make your own view but I think we found a bug online like on the stream interesting let's add a now you make a custom View and then it will always sit it should have set in the default view as well interesting all right doesn't matter n it's fine let's not spend too much time here authors let's just have more user friendly there we go cool all right we now know the basics right we're going to use an upsert thingy here create benic at all the different things that's perfect how do we translate that to an automation way of doing things let's go back to our Visual Studio code here and instead of checking the Great Migration thing and the import data we're going to go to our author's server and since this is kind of a demo slash actual Spike of the is this Express JS website it is I have not used that in a long time punish me please so let's actually start that I actually believe this is one of Jam's first spikes uh if Jamie is still here you should be proud we still use thism yeah we still use this it's the basis of this whole thing still runs cool it's running on Port 3000 let's go and check it out you just started a node server essentially that does http expressjs okay cool yeah and what it does it basically lists our authors in oh it actually queries HRA also yeah it's actually live so if I go and create a random author which will essentially create a random name for them I can go to our system check the authors and we have there it is that here that is so fancy Prett cool cool but what's interesting here is that we need to see how we can add our export from our old system which might or might not be contentful we don't know Tim kind of dropped it but it might be quum stack who knows it might be who knows cool so we need to examine how this thing works before I click it so I'm not going to click it just yet by the way maybe it's content AI could it be why am I wearing this hoodie at this moment nobody needs to know come on we're begging to get fired right now I'm living on the on the edge of work all right so let's run this thing anyways let's oh wait let's have a look first yeah so we need to examine what this thing is doing before I click it and if I can't really highlight this but if you take a look at the bottommost there on the screen you're going to see that we are calling at CSV authors in our that's the URL yes of course yes exactly so if we go here and look for that at CSV there we go it pretty much calls an importer and if we go and examine what that does is it calls import CSV which pretty much reads that CSV file it makes it the Json and then for every role in that Json it calls an import we job thing that's going to execute this is a q system right this is mainly there because if you're on the free version and you're going to just Spam a thousand things at the same time is GNA crap out that's not how it's meant to be used that's why we exactly you can pretty much make them syn and execute the whole thing as a batch but yeah let's let's roll with this so in here o I probably should have updated this to say high graph yeah I saw some graph CMS references before from your environment variables but that's fine if there's anyone new on this uh live stream graph CMS is the old name of KRA clarify things cool so what are we doing here well essentially we are creating a new graphql client which now is not exactly that graph CL that management a CL that we saw in the migration thing this I think is this is just an npm package everybody can download yeah it's an npn Pro I think called graphql request isn't it if you scroll up uh graphql client actually yeah no yeah gra your request yeah there we go so here W how you would do a post with fany gra queries yeah exactly exactly how are we with time okay we should wrap it up yeah we have 10 minutes we should be fine yeah so in here we have the basic create author right so this is pretty much creating the author with the name with the portfolio link and so now what we need to do here is change this create mutation to the upsert thing that we just played around with oh yeah exactly exactly so this stuff yes I can just copy and paste that and start changing the variables with uh the ones that we have there but through the magic of being prepared and actually the magic of being prepared through the magic that's our sales engineer for you folks oh well you always need to be prepared I can just copy and paste that so that's what I'm gonna do just gonna copy the call query let see where it ends here there we go and I'm just going to replace this whole thing which again through the magic of being prepared voila we have there we go upsert author with all the necessary name for the author portfolio link title for the blog post and content for the blog post so now actually before executing that I'm going to go back here and I'm just going to clean everything like I'm just going to delete all the authors and all the blog posts we have a clean system there we go nice and if we come back here we are expecting to see two authors after this is done Tim and me each with five block posts yeah so let's come here click the magic button author important is kind of f because they oh we have an eror people perfect what let's Zoom this in value is not unique for the field name oh maybe if I saved the file first oh that would have helped maybe all right let's try that again here let's start the server again and before we do that let's see if we actually managed to import something with did no nothing happened oh yeah some stuff we imported the same name of course yeah yeah yeah yeah cool so it's running let's refresh this saved happens to me at least once a day when I'm coding so this is actually a very good example of why you need that backup environment right because if a lot more things got imported and they got messed up with different already existing things that you have in the system imagine having a 100,000 items of something that all need to connect somewhere exactly so in that case you could just roll back to the previous backup environment that you had and just start from there once you fix the script so now that we fixed it let's hit that again check Visual Studio it should be trying to import them it's a que and this is your Q essentially exactly exactly wait a second before the next one yeah so once that's done our expectation is to see Tim and me here we are if we go to the system and we go to content we have two authors as expected it and if we go to blogs we're going to see five for me nice and five for Tim so they're all in draft mode is there a way in go to also publish them there is and let me check if we could do that from here no but if we add a random author I think I had it from here yeah I have the I hit that yeah uh and if we come to that particular author you can see that can you show that code I can and let's go to the author server it should be publish author it's basically a simple mutation oh that's very simple okay um cool that's pretty much the whole scenario several gotas that I need to mention first you need the right permissions for the token that you're going to be using in here right that token that is configured in the platform and if we go to the project settings and the authentication tokens you pretty much will need the correct permissions depending on what you're doing so for this one particularly because we are doing mutations because we we are publishing for the content I pretty much enabled everything yeah but since we're doing doing pretty much the uh the same no since since we're executing the schema creation from that same token you need to make sure that the management API permissions for that are enabled so basically it would be create new Fields create new models and stuff like that right all of that this stuff I would kind of suggest for people to make a specific token just for importing and then just trash that after I I agree yes yeah like how I did it my site for sure yeah exactly all right folks more or less I think that is more or less it well we got really far actually these these are all the basics I think the the hard bit from now on is looking at your current system and seeing how many connections you have to things and how big that system is and if you also have Legacy and if you just how to get it out so there's like a whiteboard session need of what do we do when and if it's an upser or an update or an insert once you have that done backup your environment and try and then probably try a few times inste put back the backup all the time and then call me yeah sure that we could do that you mean you want to sell like a license or something now the the interesting thing is like you could also do it in a few sessions right you could do the first one you like it okay now I write new code to connect things up to the next one split it up that's there's nothing against that right doesn't have to that's actually very important and it's one of the official suggestions that we have like if you have a big system do it in chunks you don't have to do everything at once yeah this is where Brian's uh question actually comes into play quite quite a lot because yeah planning over coding and you know what's interesting here I've done huge migration projects before while we were building a new system and we were like oh boy we actually have to migrate eight years of data from 300 websites into this new system that might be be multi1 into 10 sites or something and then that would take us like six months and it was five months of planning and one month of trying the code and then I was fine but it's it's quite planning heavy because I can probably you can count on the fact that there's a dark corner of data somewhere that is not the same everywhere the structure is not clear so you almost have more work in the conversion to something clean than actually putting it in HRA we just give you all the tools that you need essentially yeah so if if anything anyone here from the from the community has any questions you can ask them here what you can also do is actually join us at the community if you're not there yet or if you are um feel free to just put all these questions in and if you feel like hey the general channel has like 6,000 people I'm not going to ask my question there you can just send us a DM or you can actually go and into our getting started Channel where we kind of created a f that's a bit smaller and you can just ask anything without having 6,000 people looking over your shoulders um and we're all in there right so we have done this stream and we will know this stuff so we'll be in there if you ask a question we'll answer hopefully and so y Nikki one question Tim sure how are we going to send the actual code repository that we just created and the the easiest way is if you give me some links now and I put them as banners and then later on in the description of this live stream they'll be here I'll also post them on our um Community because we have let's do that later because the links yeah we'll do it after the fact it's easiest because we have a link to clone this repository in hgraph we have a repository in GitHub you can clone yeah and we also have our slides because Nikki also has an extended version of his slides that has like per slide there's like all the steps right we make that a PDF and you can just get that because if you have to tell your boss we're going to need to do this migration send them this deck and they'll be like oh sure let's go that's what I'm hoping this will do um we are exactly at five or at least C my time um is there anything left Nikki that you wanted to give our audience think on your feets and you're good to go I was thinking he's gonna say happy coding or something in that case I'll say happy coding and I'll see you all next week in next stream and thank you Nikki for all your hard work in the prep for this in the two hours you spent but there you go you can do a lot in these Hours come on true no that's a compliment anyways cheers everyone take care cheers guys bye bye