Episode 187: Are Live Hacking Events even worth it?

[00:00:01.02] - Justin Gardner
My  Codex subscription tapped out and I was like, oh, I gotta put some more money on this. So I went and put $40 on it.

[00:00:08.43] - Joseph Thacker
Instantly gone.

[00:00:09.82] - Justin Gardner
It's like it just ate it for breakfast. 

[00:00:35.71] - Justin Gardner
Sup hackers, hope you guys are amped up and ready to hack after all the festivities at DEF CON this past week and all the crazy research as well that dropped. Um, always very inspiring, I think. Uh, so we have an exciting announcement to give you as well. Um, Adobe is going to be shifting their program over to Intigriti. Okay, the cutover date is going to be September 1st. So if you're hacking on them, they're going to pause their program, shift over to Intigriti. So what we want you to do right now is go register for Integritee if you haven't already. Okay. We're going to put the link in the description to do that. And in order to celebrate this shift with them, Critical Thinking is going to give away a lifetime Critical Thinkers tier sub. Okay. Normally this just goes to the guests of the show. We don't normally give away lifetime subscriptions, but we're going to make an exception this time. So go to the Discord, go to the giveaways channel and say, ready to hack on Adobe. and then paste in your link to your Intigriti profile and you'll be entered in the raffle to win a lifetime Critical Thinkers tier subscription. This is like several thousand dollars worth of value. Okay. So definitely get on it. Um, all right. That's it. Congrats to Adobe on the shift and let's get back to the show. Sup hackers. Got a quick This Week in Bug Bounty segment for you. The article that I wanted to highlight this week is Exploiting Web Cache Poisoning Vulnerabilities by the Intigriti research team here. And this article is particularly interesting because it was co-authored by none other than Zero, Z-H-E-R-O, the guy that's been doing a bunch of web cache vulnerabilities in Next.js and all sorts of different frameworks, right? This is a really cool article because throughout there's these little, you know, green sections that you can see for those of you on YouTube that include the commentary by Zero on various sections. And each one of these gets you a layer deeper and deeper and deeper into web cache poisoning as a vulnerability type. So definitely worth reading if you want to get up to date on this vulnerability type. The next item on the list is also from Intigriti. We just wanted to do a quick shout out, a congratulations to the hackers Lucky Luck, Fatman, and MDR01 particularly for their induction into the new Integrity Elite group. Specifically, MDR01 has some friends here on the Critical Thinking team that requested they get a shout out. So congrats to all of y'all and keep up the good work. All right, that's it. Let's get to the show. Dude, you see this? I got my— I got this. I got the test phone. I got the old Linux laptop pulled out. I got a router strewn across my desk here. I got a router motherboard on the hardware hacking station back here. And we just crossed the finish line on a hardware event, dude.

[00:03:18.63] - Joseph Thacker
Detoxing from the hardware event.

[00:03:21.21] - Justin Gardner
Yeah.

[00:03:21.68] - Joseph Thacker
Now you have to clean up.

[00:03:23.36] - Justin Gardner
It's, it's rough, dude. I mean, like I've, hmm, I've got so much like, especially when it, when you're with the routers and you've got to like emulate network configs. One of the bugs that I found like required a double NAT and like, it was just getting it set up itself would have been the worst if it wasn't for AI, dude. Yeah. Yeah. And now it's—

[00:03:47.06] - Joseph Thacker
how'd you do?

[00:03:49.28] - Justin Gardner
Um, you know, all right. I, to be honest, I only, uh, because of various other things that popped up in life, I only got to hack for a couple of days, but I had like 6, 6 bugs. Um, 1 crit, 2 or 3 highs.

[00:04:06.28] - Joseph Thacker
Nice.

[00:04:07.31] - Justin Gardner
And the rest lows or mediums.

[00:04:08.84] - Joseph Thacker
Um, since you submitted them later, I assume they haven't paid out yet.

[00:04:12.05] - Justin Gardner
Well, the other thing is I, I, they haven't paid out, but I found 2 more crits, but they were dupes, you know? And I was like, ah, hahaha. And I knew they were dupes before I submitted them. Cause I was talking to one of my friends who was working on the same target. He's like, yeah, sorry. Submitted that 4 hours ago. And of course, like, you know, there was a massive window for this event. And, um, and of course he submits it 4 hours before I find it, you know? And I'm like, oh, RCE too, dude. Oh my gosh.

[00:04:40.68] - Joseph Thacker
Yeah. Was it a big event?

[00:04:43.99] - Justin Gardner
No, wasn't, wasn't too big. It was a fun one though. I really love, I love hardware hacking stuff, man. And a little bit of FOMO of not being there in person because it is more fun in person, especially at a hardware hacking event. Like, yeah, you've got like, you're sitting there with the device and like, I don't know, it's special.

[00:05:01.42] - Joseph Thacker
Yeah, I do think that it feels way more like cool slash cyberpunk-y to actually have like hardware stuff you're messing with.

[00:05:09.70] - Justin Gardner
Yeah. Yeah. I love the hardware hacking side of it too. Like, like getting, you know, tearing it apart, like ripping a chip off. Like, you know, I say ripping a chip off, gently, gently lifting a chip off. Like, it's so fun, dude. Um, so yeah, great, great event overall. I got to catch up with some of the guys after DEF CON and, and see. how their results were because it— this event butted right up to DEF CON. So everybody's at DEF CON today. But, um, yeah, really fun. I— the FOMO is real, man. I want to be at DEF CON so bad right now.

[00:05:47.00] - Joseph Thacker
Yeah, I feel like I feel a lot better that you're not there and, and that a couple other people I know aren't there. But, uh, but yeah, dude, I'm glad.

[00:05:56.58] - Justin Gardner
I'm glad I'm not there because I just had a family thing come up that I was like, this could come up while I'm away. And it did, and I'm here for it. So the right choice was made.

[00:06:06.43] - Joseph Thacker
That's good. I was gonna ask you, do you think that, um, smaller counts of people are better kind of in this like post-AI cyber world?

[00:06:16.94] - Justin Gardner
Okay, let's, let's talk about this a little bit. Uh, this— I don't know, man, because we— you and I were chatting the other day and it's like, I don't even know that LAGs are I don't want to be hyperbolic about it. LHEs are definitely worth my time, and I'm going to continue doing LHEs no matter what because they're so freaking fun and they give a lot of motivation. For, for, for, if for no other reason besides that, if they give a lot of motivation, then that's great, right?

[00:06:43.79] - Joseph Thacker
Yeah.

[00:06:44.33] - Justin Gardner
Um, but I'm, I'm certainly feeling the difference now, and I think the reason for that is that one, Claude is just— and Claude and Codex, you know, they're just very you know, they can cover a lot of ground.

[00:06:58.95] - Joseph Thacker
Mm-hmm.

[00:06:59.66] - Justin Gardner
You know, and, and I feel like prior to them being there, I would go hard and I would get deep and then people would not go as deep as I would go.

[00:07:08.43] - Joseph Thacker
Mm-hmm.

[00:07:08.87] - Justin Gardner
You know, and then I would find a bug and I'd be like, yes, this is— no one's gonna get this. Like, this is a Justin bug, you know?

[00:07:14.98] - Joseph Thacker
Yeah.

[00:07:15.87] - Justin Gardner
And then now I'm, I'm, you know, and it's— I'm not even sure that it's that that, um, other people would find the bugs that I found before, but I'm not doing that now. I'm using, you know, Codex and Claude to go deep.

[00:07:32.18] - Joseph Thacker
Mm-hmm.

[00:07:32.56] - Justin Gardner
Right. So I don't, I like, and then, and thus other people are also using Codex, uh, you know, Codex and Claude and finding similar bugs. So the dupes are much higher. Right. And so I don't, I don't know, man. I think either I need to change the way that I hack with Claude or Codex. Let's just call it— I'll just say AI for now because, because Codex, you're right, Codex is freaking killing it right now. Um, or I need to like and be more guided, or I, I need to figure out some something else because too many dupes are happening and it's not fun.

[00:08:06.63] - Joseph Thacker
Yeah, it's kind of interesting because like I feel like, um, as things have changed, they've also kind of stayed the same. Like I was talking to someone else who just recently did a different event And they said that they found more bugs than other people because they were running basically AI on a small little niche part that no one else was looking at. And it's like, oh, that's exactly how it used to be. But now instead of doing it yourself, you're kind of like pointing AI at those parts. And then I was also just thinking about how having kind of program expertise would lead to the same thing, because whenever you're writing your prompts, you're going to be like, hey, this is how this works. And here's— and maybe you're seeding it with your own reports, or maybe you're seeding it with your own notes or whatever. And then it ends up uh, you still, you end up finding 10 versus everyone else's 5 or 6 or whatever because you just have more program-specific expertise. So which just feels like more of the same, the way it always was. Um, but yeah, that is interesting the way that like your depth, um, allowed you to in the past kind of get down there and no one else was down there. But now everyone's able to use, you know, those, these AI harnesses to kind of like go similarly deep, maybe not exactly as deep or something.

[00:09:11.95] - Justin Gardner
Yeah.

[00:09:12.28] - Joseph Thacker
But you're also not going in that same depth like you said because You're able to be much more time efficient and, you know, bug finding efficient whenever you're using these harnesses.

[00:09:20.75] - Justin Gardner
Yeah. Yeah. I think that that's the pinnacle of it. I mean, that's the crux of it, man. Like, is that if, if I want to, you know, I'm thinking, okay, I can put less, you know, pointed effort into understanding and more effort into like processing and triage, right? Whereas I really don't think that that's my best move. I think my, I think my best move, like you said, is like investigating and finding niche areas where the bugs will live and then directly prompting the AI to go down those rabbit holes specifically. But I mean, as I'm saying this, your voice is ringing true in my head, which is that this world of AI-powered hacking is like built for ADHD people, dude.

[00:10:05.67] - Joseph Thacker
That's true.

[00:10:05.98] - Justin Gardner
I cannot do that shit. I cannot. Like, like, this is, this is my, my little, you know, gripe as a neurotypical here is Like, you know, if I have, you know, 8 panes of Claude open on my, on my computer here, your brain goes— and I, and I run it for 4 hours and I'm jumping between all of them, I am dead. I am like dead dead. Whereas I can hack, you know, I can see—

[00:10:31.75] - Joseph Thacker
You could have just locked in for 12 hours.

[00:10:33.98] - Justin Gardner
I, I could hack for 12 hours straight without feeling it, right? But if I'm trying to like that My brain is gone. It's completely mush.

[00:10:43.46] - Joseph Thacker
Yeah, that's really interesting. It makes me wonder if you could like somehow have SOL like look over your shoulder and then you went deep and then it was like, you know, keying off of the things you were doing to know what to like test and try, just kind of like automatically. Like just let's just imagine some like perfect new cool app that just like watched your screen and like it didn't— like you didn't start it until you were like an hour in or something, or maybe like it wasn't allowed to hack for like an hour. So you're like reading documentation, you're going deep, you're starting to test things, and then it starts watching and it's like Oh, now we're in this little small thing. And of course you could steer it or whatever, but I kind of think that'd be an interesting app for someone like you.

[00:11:19.01] - Justin Gardner
Yeah, dude. You know, the, the other thing is that like, I think I just need, I think I need to run Codex and Claude or whatever in like, maybe I should limit it to like a 4-pane window rather than 8. Um, and then also just have Cursor open. And just have like a cursor agent in there as well and, and just have the code open and just get it like, and then just follow my own intuition.

[00:11:46.98] - Joseph Thacker
Yeah.

[00:11:47.13] - Justin Gardner
You know, like what would I do? Okay. Well, I would say, where are all the sources? And then it says, the sources are here, here, here, and here. And I go to those and I start reading, you know? And then I say, all right, well, you know, this seems like an interesting source. Like walk me through this. And then it goes, okay, here, here, here, here. You know?

[00:12:02.54] - Joseph Thacker
Play out your chain of thought.

[00:12:04.23] - Justin Gardner
Yeah, exactly. And then, yeah, that's where I kind of think like the, the IDE experience is way better, right? In cursor, right? Like certainly Codex and Claude or whatever are gonna be the daily driver, but I think having something inside of your IDE is pretty sick for like actually human assisted. Like what I'm not, what I'm doing right now is not human assisted. What I'm doing is like.

[00:12:28.67] - Joseph Thacker
Fire and forget.

[00:12:29.34] - Justin Gardner
Here's an idea, figure it out. You know, it's not human-assisted, you know, it's, it's ridiculous.

[00:12:37.10] - Joseph Thacker
It's tempting though. It's so tempting to do that, right?

[00:12:39.77] - Justin Gardner
Yeah, totally. Yeah. And then the other thing that I'm doing is you keep saying shit to me like, oh, well, you know, I could take this other opportunity and I'll just throw Claude at it. And I'm like, oh yeah, I could do that too. And then I realize, no, that's just going to give you the same results as Reza. And I'm like, okay, well, you know, okay, great. Well, now I can't do that on this engagement that we're both on because I know Rezo is just yeeting codecs at it the instant the, the program opens. And And

[00:13:09.50] - Joseph Thacker
And And I'm just— It, it, it, it's found 3 full bypasses while we've been on the, on the recording in the last 15 minutes. Oh, minutes. Oh,

[00:13:14.89] - Justin Gardner
minutes. Oh, minutes. Oh, shut up, dude. Screw you, man. How many have you submitted?

[00:13:18.10] - Joseph Thacker
Do you want me to show you?

[00:13:19.53] - Justin Gardner
Yeah, just, just Richard, cut this for now. Like just let, let, cut back in after this. Gosh, I'm such a noob, man. To be honest, I really feel like such a noob. Like, like, I don't know. I mean, surely I'm doing something that other people are not doing with AI because I'm finding a bunch of bugs with it. But like, also, I just, I'm not like, I feel like I need to have more harness in place so that we have not what just happened, right?

[00:13:51.01] - Joseph Thacker
Yeah.

[00:13:51.20] - Justin Gardner
Which is you, you know, like us finding the exact same bugs completely independently because we just told Claude, go do the thing.

[00:13:58.46] - Joseph Thacker
Yeah.

[00:13:58.48] - Justin Gardner
You know?

[00:13:59.62] - Joseph Thacker
Uh-huh. Well, I mean, I do think like I, uh, have spent a lot— I was thinking a lot about this over the last day or two just as I was doing this thing because like it kept taking my time. Like we were on a date like 2 nights ago and it found some stuff, right? And so the next morning I'm like excited to get back to it and like go through it. And I'm going through it and I'm just like, I don't understand half these. And the reports are written so autistically that like I don't even really understand them. And so I'm like telling it to like rewrite all the reports, remake all of the zip files for upload, and then like Then I'm like, I still need to verify these myself. So I'm like, set up workspaces. So then I'm like, you know, attaching the workspace and I'm like, give me the prompt to run. And I'll like run the prompt and I'm like, okay, yes, it did work. That is safe to report. And then, and then at the end of it, you know, like 3 hours later, I'm like, my workday is over. And I was like, okay, Codex did not do all of that. I actually did a lot of work on that.

[00:14:45.69] - Justin Gardner
Right.

[00:14:46.37] - Joseph Thacker
You know, and it made me feel a little bit better in that moment. It was like, okay, not everyone could do this. One, just even grokking what the goal is, is kind of like, pretty technically challenging for the thing that we're in. And then also understanding like what's actually security relevant versus not is like, uh, and like, you know, who the attacker and the victim are. Like, so anyways, that did make me feel a little bit better at the end of that. But you're right, you know, we are going to find the same things. And I was actually thinking about prompting in, in context of, um, Shubh's blog post and HashKitten's write-up for WP2Shell. Um, because I was talking to our friend, uh, Demo/Turbo, uh, Nick about Like he doesn't love that WP to shell.

[00:15:25.45] - Justin Gardner
It's just Turbo now, bro. It's just Turbo.

[00:15:27.84] - Joseph Thacker
I can't do it. Can't do it. I'm either gonna call him Nick or Demo. And anyways, so yeah, he like has some qualms to pick with that WP to shell prompt. And I kind of did too. And, but it clearly gets like really good results. It did. It did in that case. And so there's still, I think, like some meat on the bone for that. And I think I sent you like a mega prompt I was using. Um, to try to like find some like epic, you know, internet melting bugs in the words of, uh, Shubs there. And so I do think that there's still like meat on the bone slash slack in the rope for finding kind of like crazy bugs and for, you know, modifying our harnesses to be better.

[00:16:04.79] - Justin Gardner
Yeah. Yeah. And, and I think one of the things that really shines through all of this is that if you had the experience before, then you will wait, do way better now too. You know, like, like, I would really hate to be a beginner right now in some, in some capacity.

[00:16:22.40] - Joseph Thacker
Like, well, I know what it looks like. I'm sure you do too. We get— I get DMs that are just like, hey, my agent said this is a critical, can you help me validate it? It's like, yeah, I guess maybe some. Yeah, usually, usually no. But if it looks legit, I quickly look at it, but it's almost always like pretty difficult. Like, it's like, oh, I'm gonna need to set up an account or I'm gonna need to test this myself. And so I just don't have time to like be your expertise. And at that point, what use are you? You know?

[00:16:44.25] - Justin Gardner
Like, yeah, yeah. And, and I think the other thing is Like, for example, I'll, I'll say this as well. One of the bugs that I found, I found that Codex found, you know, recently is, is like a, like an actual binary exploitation bug, right? Which is something that I have wanted to do for a very long time.

[00:17:03.34] - Joseph Thacker
Yeah.

[00:17:03.50] - Justin Gardner
Right. And so I took the opportunity to kind of deep dive it and really understand what's happening here. And I learned a bunch about like, you know, bypassing ASLR and like, you know, all of the things that, that come along with all of this. But man, there's so much gap still, you know, like, like I, I'm, I'm through this bug. I have RCE and I probably understand 40% of the bug.

[00:17:27.05] - Joseph Thacker
Yeah. Yeah.

[00:17:27.52] - Justin Gardner
Yeah. You know, maybe 60% if I'm lucky. Right. You know, and, and I'm like, and, and I, and like, I can give you a full high-level explanation of it, but like, I cannot tell you exactly why each one of these like ASLR oracles that I'm, you know, that I'm, I'm using, what, how that works, you know? Like, yeah. And I'm like, and I just, it, hmm. I don't know, man. Like, I don't, because I get the result already, because the result is in my hand, I feel a lot less motivated to like go and, and understand exactly.

[00:18:04.03] - Joseph Thacker
Yes.

[00:18:04.42] - Justin Gardner
Even though this is something that I've wanted to learn forever and I've got like a perfect use case in front of me and I've got a teacher that will endlessly answer my questions. Right?

[00:18:11.41] - Joseph Thacker
Yes.

[00:18:11.80] - Justin Gardner
I just feel less pulled to it. So I think really learning and actual improvement from a hacker perspective now is only for the ravenous, really, like to learn, right? You have to really be ravenous and disciplined.

[00:18:25.74] - Joseph Thacker
Because it takes extreme discipline. Yeah.

[00:18:27.91] - Justin Gardner
Yeah. Yeah, it really does. So yeah, anyway, be ravenous, folks. I am really trying to either that or be disciplined, right? Be strict with yourself, right? And, and, and say like, no, you may not report this until you understand everything, you know? And then, and you might lose your bug, you know, because dupes, dupe, that's the other thing is like dupe probability is way higher. So it's like, okay, I'm, I am like gonna lose this bug because I can't understand it fast enough.

[00:18:57.99] - Joseph Thacker
Yep.

[00:18:58.30] - Justin Gardner
As a 15-year expert in this, you know, like, like what is going on here, man? It's, it's crazy.

[00:19:04.73] - Joseph Thacker
It really is crazy. And I think that like, I don't know, man. This is just on my mind a lot because I, I want to make sure that, you know, I make the optimal place In the next couple months. And obviously we want to tell our listeners how to do that too. And it feels like the optimal play is just get more Codex subs and give it really intense prompts to find crazy, like, internet melting bugs and then hope it pops them out. Actually, so I, I have a really big question for you. And actually, I think we should post this as a poll in the Critical Thinkers or maybe just in general for the Discord. Um, of how many times you push back, because our friend JD, is willing to do that over and over and over in ways that I'm not. I, like, I would stop at like, you know, maybe one pushback or 2 pushbacks. I think some of the bugs that he got in that event you're talking about, he like had to tell it, no, there's definitely a bug here. Even though Sol, 5.6 Sol, which is like smarter than all of us, was like, no, there's definitely not. I've tried literally everything. He's like, no, I know there is. And like the 3rd or 4th time ended up getting a bug and like a critical like RCE or something. And I just don't understand I feel like that's just a heuristic that we're all going to have to learn is how much to gaslight the model. Yeah, I don't know. What do you think?

[00:20:16.75] - Justin Gardner
Yeah, dude, I have no idea the solution to that. I think that hacker intuition is really valuable here, right? And I also did push back very hard during this event and ended up getting a critical from it. But it's because I had preexisting subject matter expertise.

[00:20:37.68] - Joseph Thacker
Yeah.

[00:20:37.81] - Justin Gardner
You know, like I, I, I knew, I knew where the holes were in this product cuz I've spent much time, you know, working on this product in the past. Um, and I was like, I know that there's something here, you know?

[00:20:51.48] - Joseph Thacker
Mm-hmm.

[00:20:51.84] - Justin Gardner
And I was very confident about it. And then there was, despite the, the, you know, denials. But I, I don't know that that's, maybe it's cross-applicable. I don't know. You know, if JD's just, if JD's just saying, no, there is, there is, there is on everything, then Maybe it doesn't even matter anymore. Screw it.

[00:21:09.08] - Joseph Thacker
Well, I mean, I mean, we all— none of us have like infinite tokens. Um, yeah, I would like to have one eventually, except for people who are working at the labs. And so like, we do have to like allocate our resources. And so I think when you're saying that over and over and over, you're giving up pointing at other scope, right?

[00:21:23.60] - Justin Gardner
Yeah, yeah, that's true. Well, but you know, that's only because you don't have 40 subs, you know, like, you know, like And when you think about it, 40 subs is what, 80 or 8 grand, right? This thing's making me more than 8 grand a month.

[00:21:39.16] - Joseph Thacker
Oh yeah.

[00:21:39.55] - Justin Gardner
I should just keep pressing the button. This is what you were telling me to do the other day as well.

[00:21:43.59] - Joseph Thacker
Yeah, I told you that.

[00:21:44.85] - Justin Gardner
That's one of the things that I, that I think I'm going to do after this live hacking event is like, I am, which ended yesterday. So right now is what I'm going to do, uh, is, is, um, just my number one goal now is to have too many subs and be paying too much money, you know, out of the subs, you know.

[00:22:07.00] - Joseph Thacker
And like, I think you should— like, you should basically be able to— you should be maxing it all the time. But— and that's gonna— what that's gonna require—

[00:22:12.29] - Justin Gardner
I am maxing. I am maxing my subs. I have, I have a Codex and a Claude, and I am maxing them consistently. But I need to have some— I need to not be the person maxing them. I need to have them maxing themselves, you know what I'm saying?

[00:22:24.88] - Joseph Thacker
Yeah, yeah. Yeah, so I have, you know, me and JD kind of have that harness for our hackbot and it's like, it's always maxing the Claude maxes, but it's like slowly finding less and less and less. And I think it's because the bar is raising. And I think, and I think 5.6 SOL is still high enough to find good stuff. And so you need to be maxing that and it can actually go crazy deep on like, you know, internet melting bugs. Um, the thing is you can't scale it Unless you have old accounts. So this is something everyone should think about. Like, you might as well go ahead and go make some ChatGPT accounts so that you can scale in a month or 2, because if you try to get trusted access for cyber on a new account, they'll reject you. So if you actually do want to scale, you have to like get some lead time. But I think, Justin, what I was going to tell you is, you know, you have to verify multiple accounts. Yeah. With the same ID.

[00:23:11.92] - Justin Gardner
Yeah. Yeah.

[00:23:12.72] - Joseph Thacker
Okay.

[00:23:13.68] - Justin Gardner
Um, they don't take that away.

[00:23:15.27] - Joseph Thacker
Yeah. I I

[00:23:15.72] - Justin Gardner
I I would be like stealing my wife's like identity to That's funny.

[00:23:21.75] - Joseph Thacker
Just to get more accounts. Yeah. I mean, so I think, I think we're going to go to ID verification for a lot of stuff. I mean, obviously HackerOne just implemented that. We could talk about that. But anyways, um, in my opinion, like the way that you can scale up with different codec subs is like, um, also going to force you to fail fast and when you're failing fast, the things you're going to learn is like, hey, how then the bottleneck all of a sudden, instead of becomes tokens, it becomes you as triage, right? So then you're going to have to shore up your automated triage service, like your validation pipeline, your report writing pipeline. And then it's like, oh, I have to clean up all these reports for 40 minutes. I'm just going to improve my report writer. And then maybe the bottleneck eventually is responding to NMIs. I recently found out this is such a clutch thing. Sorry, Douglas, if you don't want me to tell the community, but our buddy Archangel Douglas has um, a like /nmi command where he gives it the report and it goes, reads the NMI, goes and finds the evidence and the, and the, then like gives him like a draft response that he can then clean up to respond to the NMIs.

[00:24:29.28] - Justin Gardner
Yeah. Um, so anyway, I do that too, except I just go to HackerOne and I click export on this and copy the markdown and then say, respond to this. Also, that's such a dumb thing, man. Like I feel so ducky using AI to respond to everything, but it's like, Why would I not do this? The responses are better and it's like, it's so much faster.

[00:24:47.91] - Joseph Thacker
It can go find the real evidence and the real request and like set up the— yeah, exactly.

[00:24:52.34] - Justin Gardner
Ugh. Crazy world, man. Crazy world. I don't, I don't know, man. Like, yeah, consider this my, my commitment and I want you to hold this, hold me to this, Rezo. I will not get distracted by another event. or another program until I have implemented a, a harness for my harnesses that will be maxing out multiple subscriptions constantly and, and moving forward like that. Because I think leaving it like the way that it is right now, where I am having to manually direct it to max out the subscriptions is not efficient. And I should be at least at 10 subscriptions by now, you know, really.

[00:25:37.14] - Joseph Thacker
Yeah, really.

[00:25:38.27] - Justin Gardner
Yeah, so, um, I want you to hold me to that. Okay.

[00:25:41.65] - Joseph Thacker
Okay, I'll hold you to it. You need, you need to at least upgrade your subscriptions and get a harness of harnesses.

[00:25:47.10] - Justin Gardner
Yeah, exactly. Yeah, that's what— that's where I'm at.

[00:25:49.81] - Joseph Thacker
Um, hey, speaking of this, I was gonna mention something. So I recently got a your request has been blocked from 5.6 Soul, and, um, I like had to go outside and touch grass, and, um, and I also went and cried, and I didn't know what to do. And then I just started copying and pasting everything to Opus and just hoping that it would not block me. But it turns out all that was useless. And I wanted to tell the listeners because it's like a super good practical tip. That is apparently just a compaction— is sometimes if it just says like your request has been blocked, it's very often just a compaction error. I think that the model they use for compaction has like stronger blocks whenever you're doing it in 5.6.

[00:26:30.29] - Justin Gardner
Really?

[00:26:30.58] - Joseph Thacker
So if you just downgrade to 5.5, Let it compact. You can then switch back up to 5.6 SOL and just keep going. Um, dude, it was such a lifesaver.

[00:26:39.32] - Justin Gardner
You're a, you're a beast. Thank you for telling me that.

[00:26:41.32] - Joseph Thacker
Did you get that?

[00:26:41.80] - Justin Gardner
That is really helpful. I haven't, but like, I know that I would be so mad. Also, the other day I did, you know, in the middle of this crazy exploit that I'm working on, right? The binary exploitation, my Codex subscription tapped out and I was like, oh, I gotta put some more money on this. So I went and put $40 on it.

[00:27:00.48] - Joseph Thacker
Instantly gone.

[00:27:01.44] - Justin Gardner
It's like, it just ate it for breakfast. It was like gone in less than an hour. And I was like, ah, shoot. So then I had to switch over to, um, I had to switch over to Claude.

[00:27:11.64] - Joseph Thacker
Yeah.

[00:27:11.88] - Justin Gardner
Uh, you know, and Claude, Claude picked it up fine, by the way. You know, you mentioned putting the effort on ultra and stuff like that. I think that does help.

[00:27:19.26] - Joseph Thacker
Uh, oh, it does. Yeah.

[00:27:20.47] - Justin Gardner
Yeah. I, I definitely got some better results from that. I, I'd been kind of, I had had some bad experiences with effort max in Claude in the past.

[00:27:29.22] - Joseph Thacker
Mm-hmm.

[00:27:29.65] - Justin Gardner
But Ultra seems to be doing quite well.

[00:27:32.14] - Joseph Thacker
In Claude or in Codex?

[00:27:34.15] - Justin Gardner
Both.

[00:27:34.89] - Joseph Thacker
Okay, yeah, because I haven't tried Ultra in Claude yet.

[00:27:38.18] - Justin Gardner
Yeah, no, no, it seems to be good.

[00:27:40.00] - Joseph Thacker
Okay, I will say this though. So I did this and you can do this yourself or maybe we'll put the prompt in the, or yeah, the prompt adjustments in the show notes. But I had Codex analyze a Codex Ultra, a 5.6 Sol Ultra session that I ran for a few hours when hacking and was like, hey, I love Ultra. It finds way more bugs, but it just burns through tokens. Can you find any efficiencies? And it came up with 11 things, 11 different ways that it could be more efficient. And I read through them and I only implemented 4 of them. But implementing them, I just told it to add it to its agent's MD, which is like, hey, when you're running in Ultra mode, Yeah. make sure you do these things. So basically it can batch the things it's trying into like less tool calls. And it can also decide to like reuse subagents instead of like spinning up new ones with like large context. And then that lets it use the cache more effectively.

[00:28:37.77] - Justin Gardner
That's huge.

[00:28:38.13] - Joseph Thacker
And since we're, since we're all like token starved, especially for SOL, like there's so many people who are going to be not able to like get a second subscription or whatever, but they still want to use the power of SOL. So I feel like it's like a really practical tip. Like basically ask Codex to analyze some of your Ultra Logs and then implement a way for it to be more token efficient but lose zero quality. And I think it is possible.

[00:28:59.57] - Justin Gardner
Yeah. Yeah, dude. One, one other thing related to that, cuz you mentioned the agents. I do not like the way that agents work in Codex, dude. Like it, it's so hard to navigate and I can't, like, why do, why is it when I type /agents, I see a, every single agent that is running across all of my Codex things on my machine.

[00:29:19.72] - Joseph Thacker
I've never used /agent.

[00:29:22.03] - Justin Gardner
Well, like sometimes you wanna drop into like subagents and stuff like that and see what's going on.

[00:29:26.05] - Joseph Thacker
Mm-hmm.

[00:29:26.53] - Justin Gardner
And then, you know, it like provides you with a list of every agent running on the whole machine. And I'm like, no, no, no, no, just this session. Like, you know, and I, and then I can't find it. Like there's literally like 60, you know, running, you know, and, and the completed ones are not hidden, you know? And it's like, oh, this is yucky. Have you tried the app? No, I haven't.

[00:29:46.27] - Joseph Thacker
Yeah.

[00:29:46.35] - Justin Gardner
Cause I'm on WSL.

[00:29:47.38] - Joseph Thacker
I only use the app. Yeah.

[00:29:49.38] - Justin Gardner
Maybe I need to just, maybe if I'm going to be using Codex, I need to, to, but then, you know, if you've got the app, you can't pop multiple subs, you know, it's like, yep.

[00:29:58.05] - Joseph Thacker
That's the problem unless you have a second computer and that's what I'm using my second computer for. And I'm thinking about using a third old computer.

[00:30:03.75] - Justin Gardner
That was the other thing I was going to say though, which not a lot of people are talking about, I don't think, which is that even if you're using these subs, like these still, these things still require a shit ton of CPU and memory and stuff like that, dude.

[00:30:18.39] - Joseph Thacker
They do.

[00:30:18.97] - Justin Gardner
So each, each, each one of these things needs to be running on like a minimum 16 gig RAM, you know, box. And like my tank of a server of a desktop, 128 gigabytes of RAM, you know, I don't even know how many, uh, you know, core CPU. Uh, is getting tapped out on a regular basis.

[00:30:40.36] - Joseph Thacker
Yeah, it'll like, uh, well, I think there's some memory leaks. So I was going to mention this as well. Very often when I'm running like codecs at work, and it's happened to me with Cloud Code a lot too, on my, on my machine, all of a sudden, like, have you ever had it in macOS where like it like starts pausing programs and you can like kill them? It like tells you how much RAM. I've never I've never

[00:30:57.34] - Justin Gardner
I've never I've never had anything in macOS because I don't use macOS.

[00:31:00.39] - Joseph Thacker
Anyways, imagine, imagine like you're in Windows and like the activity monitor pops up and it's like, it like pauses a bunch of these Spoken like a guy that's never used Windows. The The

[00:31:09.48] - Justin Gardner
The The Activity Monitor pops up.

[00:31:11.03] - Joseph Thacker
Okay, Task Manager.

[00:31:11.44] - Justin Gardner
That doesn't happen.

[00:31:11.75] - Joseph Thacker
It's called Task Manager.

[00:31:13.16] - Justin Gardner
It doesn't happen. You have to pull up the Task Manager yourself.

[00:31:16.16] - Joseph Thacker
Okay, well, in Mac, it just pops up if, if like things are like using way too much memory, and it tells you how much memory they're using, and then it like also like it pauses them so that your machine can keep running. And very often I'll come back to my computer when it's been hacking and like half the stuff will be paused. And the 2 things are either codecs will be using like 50 gigs of RAM, no idea why. Or like iTerm2, like my terminal, which is running Claude code, will have like 50 gigs of memory and be paused. And I'm just like, I don't know how you did that. Maybe you wrote, like, maybe there's memory leaks in the actual harnesses, or maybe they're writing really bad code that then is like spiraling out because it's writing some Python script that's then like, you know, allocating a bunch of memory. I don't know, but you're right. Yeah, I think you can run, if you're using a HackBot, I think you can use like an 8 gigabyte droplet or, you know, VPS. I don't think you need 16.

[00:32:03.39] - Justin Gardner
I haven't had— I mean, I've had it max out my 16 a lot. I have like watchdog stuff in place now to like—

[00:32:10.53] - Joseph Thacker
I bet it would have maxed out a 32 or 64 though also. Like, I think that it's just doing something dumb and so it's gonna crash no matter what size machine you have. So you might as well just use a smaller one.

[00:32:18.49] - Justin Gardner
Oh, that's interesting. That's an interesting point. You see, you guys are running yours on 8 gig servers?

[00:32:23.95] - Joseph Thacker
I think so. I can double-check.

[00:32:25.88] - Justin Gardner
Yeah, double-check. That'd be good to know. I I, I struggle to believe that, but yeah, I, I, I think whatever's happening, it's, it's crazy and it is knocking out tanks of machines as well. I have, I've got this laptop right here. Um, I pulled out my, an old laptop that, you know, battery's completely dead on it, but it's got like 64 gigs of RAM. Yeah. I'm like, all right, this guy's going to be my, my Claude boy.

[00:32:51.09] - Joseph Thacker
That's amazing. Yeah. I was about to do that on one of my old Windows machines too. Also, just because a lot of times when I have, um, when I'm using AI to hack a bunch of different programs and a lot of them were desktop apps, testing Windows desktop apps is like a really great way to find some bugs.

[00:33:04.00] - Justin Gardner
Yeah. Well, this guy, this guy is, um, Linux native, you know, just like straight up installed on it. And, uh, that is one of the other things that I think I've really needed is like just a straight Linux machine because WSL does great before, but especially when you're doing hardware hacking and you need like raw interface access and stuff like that. WSL is just a nightmare.

[00:33:25.67] - Joseph Thacker
We're actually using 4 gigabytes of RAM. We're using—

[00:33:28.35] - Justin Gardner
Are you really?

[00:33:29.15] - Joseph Thacker
Yeah, 2 vCPUs and then 4 gigabytes of RAM. No, it's definitely crashing and getting slow and stuff. I'm just saying I think that would've happened on almost any size.

[00:33:40.10] - Justin Gardner
Yeah, you might want to have an agent take a peek at that and see what kind of operations are getting limited. For a little while I had, um, something that would kill runaway greps essentially, right? Where, where it was just consuming everything and, um, it was silently killing them. So the agent thought that it got the results back from the grep and it was not, you know? And so it was just missing massive swaths of scope, you know?

[00:34:09.21] - Joseph Thacker
Right.

[00:34:10.17] - Justin Gardner
And so, uh, yeah, I, I, patched that and things got better, which was nice. But yeah, just keep an eye on that.

[00:34:18.34] - Joseph Thacker
That could explain a lot. Also, JD's JS analyzer stuff does take a lot of RAM. So I say there's no way 4 is doing it.

[00:34:25.46] - Justin Gardner
So anyway, I think that there's more opportunity for scale than there ever has been. But there's also the costs of servers and stuff like that are not— I say they're not nominal. They are totally nominal. But it's like— It adds up. We got to put these things in perspective. One bug can give you $6,000 easy. And it's like, okay, well, that's like 6 months of compute and subscriptions in one app.

[00:34:52.38] - Joseph Thacker
That's true, yeah. But you did put in time and effort into it. I've thought a lot about that. Our spend is definitely, as bug hunters, used to be less than 1% of our— people don't even talk about revenue and profit in the bug bounty world. It's all profit. But now it's like, it's probably not gonna look like that this year. It's gonna be more like 90% or 95%.

[00:35:14.92] - Justin Gardner
That's a great point. That's a great point. And you know, my tax person is like, so where are your expenses? And I'm like, yeah, what expenses?

[00:35:21.65] - Joseph Thacker
I bought a monitor.

[00:35:22.63] - Justin Gardner
Yeah, exactly.

[00:35:23.51] - Joseph Thacker
Yeah.

[00:35:23.69] - Justin Gardner
Like I bought this $3,000 laptop and they're like, that's it.

[00:35:27.36] - Joseph Thacker
Are you sure?

[00:35:27.96] - Justin Gardner
And I'm like, yeah. So I, yeah, I think maybe this year I'll, I'll make them, uh, you know, actually be like, okay.

[00:35:35.78] - Joseph Thacker
You know, okay, he's spending money on servers.

[00:35:38.23] - Justin Gardner
80% margin. Okay, that's pretty good. You You

[00:35:40.07] - Joseph Thacker
You You know, like, yeah. That should be your goal. Get your margins down to something like 80 or 90%.

[00:35:45.13] - Justin Gardner
Yeah, seriously. Maybe, maybe it should be. Yeah. But you know, if it really does scale like that, then I really should, you know, if I'm, if I'm, you know, making X at my current margin and I could be making 10X, you know, uh, with, you know, 80% rather than 99%, you know, then I should definitely be doing that.

[00:36:04.59] - Joseph Thacker
Here's another crazy idea. What about hiring a virtual assistant overseas, or just somebody from the CT community, to keep Auth alive for the Harness of Harnesses? So I've like legitimately thought about this because like I did the thing you did and basically got an Auth Keeper and gave it like 10 programs, and like it does stay alive for a while, but then they just die, and then getting it set up again is annoying and logging is annoying. I've legitimately thought about like, what if I just hire somebody That just, it's their job is just to keep everything alive. Like, sorry, to keep auth alive across like 100 programs.

[00:36:35.82] - Justin Gardner
Yeah, that's a good, that's a good idea. Yeah. Maybe I'll, I'll just, cause I've got a couple of friends that want to get into the bounty space and just be like, hey guys, I'm going to ping you.

[00:36:46.26] - Joseph Thacker
Yeah.

[00:36:46.28] - Justin Gardner
And then you just log in, generate traffic. That's a good idea. Yeah. Yeah. I bet, I bet I would make money on that, you know, if I paid them $50 a site, you know.

[00:36:56.38] - Joseph Thacker
Mm-hmm.

[00:36:59.03] - Justin Gardner
Good thought.

[00:37:00.01] - Joseph Thacker
All right, dude.

[00:37:00.71] - Justin Gardner
All right. Hey, do we want to cover this in more detail, this post from, from Chubbs?

[00:37:07.69] - Joseph Thacker
Yeah, let's break it down.

[00:37:09.21] - Justin Gardner
Yeah. So guys, one of the core tenets of critical thinking is when Chubbs says something, you shut up and listen. And often, often I, I skim through articles and stuff like that. Anytime Shubs posts something, or really any of the asset node SL Cyber team, I literally sit down and read it word for word, you know?

[00:37:34.36] - Joseph Thacker
Yeah.

[00:37:35.23] - Justin Gardner
And so Shubs, you know, just posted this post, Frontier Class Vulnerabilities: It Gets Worse Before It Gets— It Maybe Gets Better. And he was just kind of talking about how You know, they've been in the industry for a while. They've consistently found top-tier zero days for, for years and years and years. And the world is changing just like we, we, we just talked about, um, you know, with Adam, one of the, the people from the team finding, uh, WP-Disshell using Sol in, you know, 5 or 6 hours or 6 to 10 hours.

[00:38:13.73] - Joseph Thacker
Uh, crazy.

[00:38:15.03] - Justin Gardner
And now it takes more time for us to understand the bug, even as like extremely S-tier researchers like Adam and Chubbs. It takes more time for us to understand the bug sometimes than it does for, you know, Sol to find it.

[00:38:28.90] - Joseph Thacker
Mm-hmm.

[00:38:29.69] - Justin Gardner
Um, so the, the premise that the walk away from this, um, was that he has directed his team to find internet-melting bugs.

[00:38:40.76] - Joseph Thacker
I love that quote. I think that's the quote of the, of the whole article. He says, I directed them to find bold internet melting bugs.

[00:38:48.07] - Justin Gardner
Yeah. And I think we probably do need to think bigger, you know, as far as what our attack surface looks like, uh, because, you know, going for these big shots, you know, are, is a lot more achievable now. And if we don't find them, then other people will find them first. And that one means one loss and bounty or two. catastrophic event for that company.

[00:39:13.40] - Joseph Thacker
Did JD tell you he, he like went to bed and was like, hey, find me an XSS on www.fortune5.com, and it— he woke up to an alert box?

[00:39:23.19] - Justin Gardner
Are you kidding me?

[00:39:24.23] - Joseph Thacker
No, I'll show you.

[00:39:28.88] - Justin Gardner
Oh my gosh, dude, why? Why?

[00:39:32.01] - Joseph Thacker
Yeah, but anyway, sorry, I don't know if you, uh, had more to say about the article, but—

[00:39:36.19] - Justin Gardner
That's all, is that we need to think bigger and we need to use these tools. And I, I, I'll close with this, this, this thought. I think right now, especially, you know, I, I mean, you guys have heard me talk about it, uh, on this podcast, uh, you know, Rezo as well. We are all feeling like we need to be leveraging AI to the best we can to make money and to secure the internet right now. One of the things that I'm so grateful for in bug bounty right now is that we have a way, a very material way to convert tokens to dollars, you know? Yep. Yep. And not a lot of other people have that, to be honest.

[00:40:16.65] - Joseph Thacker
That's right.

[00:40:17.46] - Justin Gardner
Like, like, you know, everybody else in this AI world is sitting there like, oh wow, how can I use AI to make, you know, this app? And then I need to get users for the app.

[00:40:27.23] - Joseph Thacker
Right.

[00:40:27.42] - Justin Gardner
And then I need to monetize the app.

[00:40:28.76] - Joseph Thacker
Yeah.

[00:40:29.03] - Justin Gardner
You know, that's what they're thinking. And for us, we, we have a system We have an avenue to turn these, these tokens into money.

[00:40:36.40] - Joseph Thacker
Infinite demand.

[00:40:37.92] - Justin Gardner
Yeah. Infinite demand. We just need to ram it down, you know, and just go as hard as we can. Yeah. I've even toyed around with the idea of like not doing bug bounty right now and like doing other AI-related stuff because the leverage is so high. Hell no. Hell no.

[00:40:54.42] - Joseph Thacker
Yeah. So this is, you might find this really funny. I was, um, I'm in like a ship or die group with a few other top hackers. Um, like, uh, Palm and Monkey and some other guys. And obviously you all have seen me launch like aisafetyforparents.com and rezzesrascals.com.

[00:41:10.17] - Justin Gardner
Yeah.

[00:41:10.32] - Joseph Thacker
And, uh, the AI Safety for Parents, I charged $10 for it, and so I got a payout. And so after tax and everything, it was like $919. And so I was looking at my bank account, and right below that was the $35,000 bounty that I recently got. And I took, I took a screenshot of it and sent it to the Ship or Die group, and I was like, I think God's trying to tell me something, like where I should spend my time right now. Uh, and I mean, that's a really astute observation, right? It's like, wow. Yeah, this is, uh, we, like you said, we have a system that turns tokens into cash and we should just make that system better and better.

[00:41:42.13] - Justin Gardner
Ram it down, dude. Just keep, keep turning them out.

[00:41:44.86] - Joseph Thacker
Do you see that screenshot?

[00:41:46.55] - Justin Gardner
Yeah, dude. So dumb. Screw that. Freaking JD, man. What a guy.

[00:41:52.00] - Joseph Thacker
JD literally said, guys, to his agent, find me an XSS that I can report tomorrow and woke up to an alert screen on a Fortune 500.

[00:42:00.90] - Justin Gardner
Ridiculous. All right, let's, let's, let's wrap it. That's it.

[00:42:04.63] - Joseph Thacker
See you guys.

[00:42:06.84] - Justin Gardner
And that's a wrap on this episode of Critical Thinking. Thanks so much for watching to the end, y'all. If you want more Critical Thinking content, uh, or if you want to support the show, head over to ctbv.show/discord. You can hop in the community. There's lots of great high-level hacking discussion happening there on top of masterclasses, hackalongs, exclusive content. And a full-time Hunter's Guild if you're a full-time hunter. It's a great time, trust me. All right, I'll see you there.