Episode 192: Hackbot Proof-of-Concept Skill Creation

[00:00:01.11] - Justin Gardner
Your AI should be iterating, uh, on, on this POC as it's building, right? So as it's writing its report, it should build it, run it, modify it, that sort of thing. And it should always be running it in the exact same way the triager will be running it.  So you guys know how I've said on the pod a couple of times, taking your spiky bug bounty income, right? You get a big bounty and then maybe no bounty for a while and converting it into something that gives you more consistent passive income is a great idea, right? For me, that is real estate and I've done that a couple of times. I have a couple of rental properties. It's a great situation. However, I will say I live in the US, and if you're looking at the screen right now, you can see ThreatLocker's write-up of a blog talking about how Romania's land registry, the whole country of Romania, was brought to its knees this past July by a threat actor going in with valid credentials that they got from phishing and end-of-day vulnerabilities and pivoting around and completely destroying the entire system. This whole attack could have been completely avoided had they been using ThreatLocker's technologies like ThreatLocker Patch Management, which uses hash-based detection to find unpatched software, including portable applications that are often missed by traditional tools, and PAM, Privileged Access Management, which prevents them from pivoting around using their local admin rights on different machines. Okay, you can read the whole write-up. It's a great blog post. It's in the description. And if you're interested in using these technologies to protect your company, please book a call with ThreatLocker via the link in the description. Also, just want to give ThreatLocker a shout out. You guys have been such a great supporter of the pod. I really appreciate you guys and you keep us being able to do what we do. All right, that's it. Back to the show. All right. Sup hackers. This week we got 2 items on the This Week in Bug Bounty segment before we hop into the episode. First one up is Yes We Hack is doing a live hacking event at Ekoparty this year. So if you're going to be there October 7th through 9th, then definitely be on the lookout for this. I think the program's name is Banco Galaxia. I probably butchered that pronunciation, but the life hacking event is always an amazing experience and it is open to the participants at Ekoparty. So that should be a really cool experience. We'll put the link in the description on how to get involved, but essentially you just need to be on site and have a Yes We Hack account. Okay. All right. The second item today is very exciting, you guys. This is going to be a short episode of CTBB this week. Okay, so here's what I want you to do with the rest of your hour and a half that you give me most every week. Okay. I want you to go to YouTube and look up Stokke's account and check out the new Builders and Breakers podcast, Building Hackbots Models Harnesses Human Expertise with Hamid from Expo. Okay. This is an amazing episode. The production quality is always so high with Stokke, and it's great to see, you know, bug bounty content creator legend Stokke come back. and create some new material out there. I've already listened to it. It's only been out an hour, and I did that right before recording this episode. And I'll give you a little teaser. One of the questions they answer on this episode is something that we've discussed in the last Full-Time Hunters Guild meeting, which is should you provide your AI with a specific set of methodology and tooling to accomplish a set of goals, or should you let it use its creativity and intelligence to cater its methodology specifically to the target you're working with. So Hamid from Expo in this episode answers that with an answer that is backed by the benchmarks that they've done at Expo. So that's— that alone is high value. So go watch that episode. We'll link it in the description. And shout out to Stalk for getting back in the content creation game with Builders and Breakers. All right. That's it. Let's go to the episode. Peace. Alrighty, hackers, it's your boy Justin Roland solo this week, and I figured we'd keep it direct and to the point with a masterclass on how to build a POC creation skill, which is something that I've talked about on the pod a couple times and is very actively discussed on the Discord. And we recently did a masterclass in the Discord where I share my POC creation skill and talk in more depth than I will even on this episode about it. So if you're interested in checking that out, you can go to ctvb.show/discord. sign up for the Critical Thinkers tier and we have the full recording as well as the skill available on there. But today I'm going to give you a quick rundown of what tips and tricks I've pulled out of my experiences with my hackbot creating POCs that make it really easy for the triager to reproduce and to— and for the teams actually to continue to test their remediation with these, with these POCs. All right. So let's jump right into it. So first, this kind of assumes that you have a hack bot, although it's not necessarily strictly necessary, right? You could be finding bugs yourself and then using AI to create highly, you know, easily reproduced POCs. And I would suggest that you do that as well if you are a manual hacker. This can be, in fact, a good transition for you into using AI in your hacking because I know a couple of you guys are still holding out there a little bit. I met a hacker this past week at a live hacking event who performed extremely well at the live hacking event and is not using AI at all. But I do think he's probably putting a lot more time into it than some of the rest of us are. So yeah, if you're holding out on that front, then this could be a good opportunity for you to pivot into it a little bit with high-quality POCs. There's a couple reasons to do this. One, it increases your chance of getting your submission processed quickly in this era of clogged queues in bug bounty. If you have a POC that is just straight fire, right? That just like, I mean, as soon as you click it, it just demonstrates the full, the full vulnerability very cleanly and easily for the triager. So that's, that's a part of the goal. So let's, let's talk about the goals really quick of this POC creation skill so that we can be aligned on what we're trying to create here. Okay. Number 1 goal, provide the triager with the easiest and seamless triage experience. Okay. Queues are clogged, triagers are stressed. I think as a community, we need to be coming together right now rather than being jaded against the other side. Right. I know that It's a pain in the butt when your, your, you know, bug sits in queue for months or whatever. But the triagers are overwhelmed. The, the programs are overwhelmed, the platforms are overwhelmed, the hackers are stressed. Right. So let's try to do our part. Right. As critical thinking listeners, as a part of this community to, to push the community forward. Okay. And we need to do that with high quality POCs. That's one of the best ways we can do that. Okay. So goal number 1, provide the triagers with the easiest triage experience. Goal number 2. Provide the team with easy triage and remediation retest experience. Okay, so this is, you know, they've also got to triage the bug and understand where it's coming from. They've got to give the bug to devs, and those devs have to be able to reproduce it, right? Sometimes without a proxy, right? Because they're devs. And, and then we need to help them be able to test that their remediation works by being able to run the same skill again now or run the same POC again. Now, some of you guys might say, well, this is getting us, you know, burning us out of retest, you know, payment here. And perhaps that's true. But I still think that at this point, the more throw— this is now I'm getting— I got in trouble last week for saying this. The more throughput that we have in bug bounty right now, the better. Okay. So I think it's a higher value for us to provide a really good POC to allow them to do remediation correctly than for us to grab at that $50 for a retest. Okay, to each their own. But that's my, that's my take. Okay. So goal number 2, provide the team with easy triage and remediation retest experience. Last goal, provide— and maybe this one should have been first chronologically, I don't know, but provide yourself with an easy way to visualize and validate the report that comes from your hack bot. So this is largely a goal for those of you that are using hack bots and for this with this POC creation skill. But, you know, I'm not gonna lie, my life feels a lot like a triager right now because I'm going into my queue and I am looking through all the bugs that my hackbot has reported and I'm reading them, I'm trying to understand them, I am running the POCs that it creates and I am trying to optimize my own experience as a, you know, hacker-side triager to, to be able to quickly understand and validate that this is an actual vulnerability. Okay. So obviously these two go hand in hand, the triager experience and us, the hacker side triager experience. But I will say this has been one of the most powerful, the most powerful impetus for me to improve my own POC skill because I'm like, dang, this POC sucks, you know, and I don't, I don't understand this bug. And I can't easily see that this is a vulnerability. And that provided me with a lot of drive to get this to a better spot. And now I can, I can say 90% of the time when I click into my HackBot and I look at a report, I grab the PoC script, I drop it right into my terminal, I watch the results that comes out of it, and then I smile and I click report. You know, like it's that simple nowadays. Okay, so how do we get there? Let's, let's jump into the nuts and bolts. Now, like I said, I'm not going to give you guys the skill that is going to be left for the critical thinkers. And I'm actually held back 3 or 4 kind of pivotal pieces of this, this puzzle. So, but this will get you 90-something percent of the way there. OK, for sure. So let's break it. Let's break it down. First, your POC should be hosted. I think that this is a no-brainer because for the triagers, you know, they've, they've got their, you know, they've got the report up or whatever and they've got their VM where they've got to run everything and then they've got to like take the file and transfer it into the VM and then run it. And no, no. Okay. Like, it's just, it would be much easier if you could just curl pipe to Python. Right. And so I've got 2 major formats for my POCs that I run. I give the AI the option of creating a curl or a Python curl to Python curl -s to Python script or an HTML file for client-side exploits. Okay. Those are its 2 options. It's got to fit. It's got to work within those parameters. Sometimes there are like more complex server-side chains that, you know, require both components or even some client-side stuff that requires a little bit more dynamic All right, now let me hit this, you know, cached thing, pull the value back down from the server side, which I can't do from the client side because of CORS or whatever. Um, but that's that in those situations, you know, you, you just going to have to go in and make it work yourself. But the, the primary, uh, you know, the primary flow should be curl -s| python3 or whatever, uh, or an HTML file. Okay. So those are the options I give it. Um, Your AI should be iterating on this POC as it's building, right? So as it's writing its report, it should build it, run it, modify it, that sort of thing. And it should always be running it in the exact same way the triager will be running it. So it should be running curl, you know, with the URL and piping into Python so that it knows, you know, exactly that it is validated exactly how the triager is going to do it. And then also I tell my, my, you know, report writing skill and also sort of integrate into this POC skill that this POC URL, automated POC URL needs to go at the top of the steps to reproduce section. Because so many times I've like gone to the steps to reproduce and it's like, do this, do this, do this, do this. And then at the very end it's like, or run this automated script. And if I were a triager, I would be ripping my hair out because I'm like, just spent 10 minutes trying to, you know, 15 minutes trying to reproduce this and you codified it for me at the bottom of the— at the bottom of the report. So make sure that gets popped up to the top. Speaking of codifying, I kind of like to think about this whole POC creation thing as, as sort of a combination of 2 things. One, codifying your report. So taking the whole report and creating a sort of holistic experience that allows— that walks them through, okay, you know, what is the current environment that we're in? This is an app that does XYZ. You know, who are we? Okay, I'm, I'm user 1. I am a low privilege user, right? And then, or actually, we shouldn't even use user 1. That's one of the things down here. We should be using attacker and victim so we can be very clear, right? So attacker, you know, is a low privilege user and, you know, victim is a high privilege user, right? And we show that with HTTP requests as well, right? And then we walk through the whole flow. Okay. you know, attacker can't access the data via this route, attacker can access the data via this route. Right. And then it's very, very clear and straightforward right within the POC. Right. So you barely even have to look at the report. You can look at the POC and the POC will walk you through why this is a vulnerability. Okay. And then the second piece that I find helpful when thinking about this is actually thinking about, thinking about your POC as a unit test for this specific vulnerability. Okay. So for those of you that have done software engineering, a unit test is when, or for those of you that haven't done software engineering, for those of you that have done software engineering, you're like, Justin, why are you talking to me? You've done like 2 years of software engineering. What are you saying? But a unit test is when you have sort of a piece of code that you want to make sure is functional over multiple changes to the codebase so that this piece of code Doesn't, doesn't get warped or the inputs that are coming into it don't cause it to break with other changes that occur. So you're validating that your, your code is, is running correctly over the course of time. And that gives you peace of mind that, you know, the application is behaving in a certain set of ways that will keep your application uptime high. Okay. That's kind of a shitty explanation, but you guys get the point. Yeah. So the other way that I like to think about this is unit tests for your report, for your specific vulnerability, where you give, at the end of these tests, you give the company sort of a not vulnerable, vulnerable verdict, right? Like, okay, so I've logged in, I've run all this, and then as a result of accessing this data, the state is vulnerable, right? And then if they do some remediation and I get a 401 somewhere, the error catching Catches in and says state not vulnerable, right? And that allows them to know very easily whether they've remediated the vulnerability. Okay, so vulnerable, not vulnerable output from the script. All right. So codify your reports, provide a unit test for the vulnerability. All right. So now let's get into some of the nuts and bolts for the .py file and the HTML files that will actually be the POC. So first things first, one of the things that came out of the recent discussion with the Full-Time Hunters Guild and with the Critical Thinkers about the POC scripts is that we should probably be providing Unix and Windows-based reproduction commands. So prior to this conversation that I had with them, I was just providing curl -s whatever.py| python3, right? But, you know, maybe we're using a stock Windows box and I don't know, PowerShell doesn't have an alias for curl or something like, I think most of them do nowadays, but sometimes they don't. So maybe we provide a Windows and a Unix-based reproduction step here. I think we can safely assume that everybody's going to have Python though. That's, I think that's just one of the ones that we're just going to have to live with. If you don't have Python, you need to go install Python. Inside of that, I think also there's this component of we don't want them to have to install additional dependencies for this to work. Okay. We, and there was some back and forth with the community on this, but I firmly believe that even requests, right? The requests library in Python, which should be in the standard package, in my opinion. Is just adding an additional dependency that's unneeded, an additional point of failure. So we want to try to keep our Python code, we want to tell the AI that we want to keep the Python code completely self-contained, right? No dependencies, no references to local paths, that sort of thing. And it should be able to run on any machine that has Python installed, right? We want to try to make it as portable as possible. Additionally, we want to have cert validation off when you're doing this because, and this is a very important piece of this, you must provide the triager with a way to proxy the request coming through the script. Okay. And obviously, you know, a lot of people know that you can just run HTTP proxy, you know, environment variable or whatever, but let's just make it easy. Okay. So they, they, they run it. If they don't have that variable set, I prompt them. to input where their proxy's at so that they can see the requests that come through. Some triagers have represented to me that some of you tricky little, um, out there will just print things in your Python scripts rather than actually sending requests. Okay. So because of that, now the triagers have to go back and review the code, or they can look at the HTTP request, which is something they're going to be a lot more familiar with. Okay. So we need to be providing them a way to, you know, pipe their requests through Kaido, through Burp, whatever, so that they can see. Additionally, we should have an X-POC-Actor header if at all possible. Okay. If it's not possible, it is what it is. But if possible, we should have that to represent what user this session is on behalf of here. Right. So if you've got attacker, then we can say X-POC-Actor: attacker, right? So that they know when, you know, without having to like inspect the JWT and see the email that's associated with it, what exactly is going on here. And the victim likewise, you know, should have a victim value for that header. And that will allow them to know which requests are being made as the victim. And there's no session confusion going on there. Okay, that just makes it extra clear and allows them to see a lot easier. And you should denote that in your POCs as well. Actually, I don't know that I have that. Let me write that down. Denote in POCs. Yeah, so that you can see, you know, like, okay, hey, by the way, you know, if you're confused about which is which, just look at the POC actor header. Yeah, I need to, I need to put that in my skill. All right, so let me run through each one of these. Oh, Another, another piece, or actually additional 2 pieces here that I think add a lot of value is one, if at all possible, if you are creating a POC for an authenticated vulnerability, you should include credentials in your report. You should include a session, your credentials, or tell the AI also if you know a way to just register an account on the fly. and create this whole state that's required for this bug, then that's totally fine as well. One of the things that I have heard from triagers when I was getting feedback on this was sometimes they have to use their own accounts because who knows what kind of janky stuff we did to our accounts. And that's fine. Just put a constant up at the top that has, you know, account username, account password, whatever is necessary to do that. And this just makes everything so simple. I love it when I'm testing, you know, triaging a a bug from my hack bot and it's authenticated and I just go in there. I don't have to put any credentials, anything. It's just embedded right in there. And if you're worried about, you know, embedding your credentials in a report, you know, you have delusional problems because nobody cares. So you should just do it. And yeah, the other piece is if you can get a registration flow, that's great. And just go ahead and register yourself some guest accounts or whatever. That's not CAPTCHA'd. And then prove the concept that way. Also, you should be discovering the IDs that you need for this exploit or the objects or creating the objects on the fly in the, in the script. Okay. So let's say you need to create an organization and that organization needs to have, I don't know, a team underneath it. Right. So your script should spin up a user, create the organization, Create the team, you know, print out the team IDs, name the team, you know, this is the vulnerable team, right? And then the attackers should access that team, right? We just got to make it very clean so that they don't have to like go and get any IDs or anything. And you narrate how you're getting the IDs in the codified report. Easy peasy. AI makes really quick work of this, guys. And this makes it much easier for the triagers to reproduce. Okay, um, I think that's all I had for the Python. Yes. Okay, let's jump to the HTML. The HTML stuff is normally pretty simple. Um, the only big tip that I have for this one is once again, we're codifying the report, so the page should be explaining what is happening, uh, as the exploit proceeds. Um, so make sure you have your AI do that. Additionally, I always like to provide sensible defaults in my input fields. And also like to provide credentials for them to authenticate so that, you know, the IDs match and there's no problems, right? Because oftentimes you can't go and fetch the IDs you need in that sort of scenario. So have them log into your account, do the exploit, and you're good to go if there's like prerequisite IDs that are needed. Additionally, I always add an autorun=1 parameter to my POCs because I want to make sure I demonstrate to the, the program how few clicks are required for this exploit to actually work. If it is a one-click, like come to the attacker's page sort of scenario and I can do the exploit, that is different than I come to the attacker's page and click a button barely because of CAPTCHAs and accept cookie requests and stuff like that that we've been programmed to just auto-click without using our brains. But nonetheless, we want to be clear about what the requirements of the exploit are. So I always provide an autorun=1 and the ability to specify any IDs that are required for the exploit in the query parameters so that they can just click a link and the exploit can occur. All right, so that, that was it for HTML. I'm going to wrap it up here. We've just got a couple more tips for both the Python files and HTMLs. We already covered that each of these should be a narrative of the report. It should set up all the required things. It should provide a control request. It should show the exploit and like actually dump the data that proves the result. And then it should clean up after itself to leave the account in the same state that it was when it started. And then the last piece here that I really want to emphasize is you should also host on your website, on your POC environment, artifacts showing a successful run. So now whenever I run a Python file, whenever my AI is building it, it runs a Python file. it must take that resulting text that comes from that run and dump it into a TXT file and host it on our, on the PoC server and put that right below the, you know, command so that they can see a successful, what a successful output looks like. This proves that the vulnerability did work, at least in some capacity, and shows them what they should be looking at here. Likewise, for client-side exploits, have it pop open a headless browser. Take a picture here, take a picture there to prove that the exploit actually worked and host those on your server as well for easy, easy access. All right, y'all, that is my framework for creating a nice POC skill. I'm sure I'm missing a ton of things, but I'm getting pretty good results with this right now. I'm not going to lie. So if you want more in-depth conversation about this, you can see the Discord because people that are better than me at this are dropping their tips in the Discord. Um, and yeah, if you guys have any tips on how to improve this, let me know, because every time I talk about it, I write down notes like I just did in the middle of the episode, and it gets better each time. Um, so yeah, shoot me a DM, drop something in the Discord. We'd love to hear, uh, what stuff has worked well for y'all in your, uh, POC creation skills. All right, that's it. That's the pod. Peace, y'all. And that's a wrap on this episode of Critical Thinking. Thanks so much for Thanks for watching to the end, y'all. If you want more critical thinking content, uh, or if you wanna support the show, head over to ctbv.show/discord. You can hop in the community. There's lots of great high-level hacking discussion happening there on top of masterclasses, hackalongs, exclusive content, and a full-time Hunter's Guild if you're a full-time hunter. It's a great time, trust me. All right, I'll see you there.