Episode 180: State of Bug Bounty Maturity Posture Report

[00:00:00.92] - Steve Hernandez
Hackers are the one and only ICP, right? To use a marketing term, they are the ideal prospect, the only prospect of a bug bounty program. Without the hacker, you don't have anything at the end of the day. Similarly—

[00:00:14.89] - Justin Gardner
Amen to that!  Amen to that, dude! 

[00:00:40.70] - Justin Gardner
all right, guys, quick disclaimer before we jump into this episode. I got my boy Steve here. We're going to talk about Bug Bounty Maturity Framework. It's an awesome tool for bug bounty programs to understand where they are in their maturity. This is not a technical episode, not very in-depth from that perspective, but I think that researchers would still gain value from it in getting insight into what the programs are dealing with right now. In what areas they can improve. And we will put some stuff down in the description of particularly researcher-relevant sections that you can jump to. Okay, so check those out. If you are a program manager listening to this, this is your episode, man. Hopefully there'll be a lot of actionable elements here. Check out bugbountymaturity.com for the self-assessment that helps you understand where you are in your maturity and gives you actionable steps on how to do it. And if you are one of the, uh, programs that have, um, already filled out this assessment, it's probably been about 90 days since we, uh, mentioned it on the pod. So I know that they've updated it so you can see your DELTAs, uh, on the website. So log back in, take another assessment. Um, all right, with that, let's jump into the main content. All right, Steve, dude, I'm so glad to have you on the pod. Um, I guess just given a little bit of backstory, Steve worked with the podcast a while back, running our sales and our partnerships, made a massive difference in us being able to continue doing this podcast. And I'm having him back on today to talk about Bug Bounty Maturity Framework, which is his recent framework that he's releasing out of love for the bug bounty community. Right, Steve?

[00:02:18.31] - Steve Hernandez
That's it. Pure passion, baby.

[00:02:21.21] - Justin Gardner
No, it's great to have you back involved and definitely appreciate all the work you did here. Typically at the beginning of our guest interviews, we have the guests, you know, drop a bug or something like that. And while you have done some hacking, I've seen you find bugs before.

[00:02:36.24] - Steve Hernandez
Yes.

[00:02:37.46] - Justin Gardner
I think the more interesting research that you've done is actually on the state of bug bounty maturity posture. Yes. Which is this annual report you released, 2026, first one, talking about where bug bounty programs are. As far as their maturity. So could you give us a little overview of that research and also maybe include why this is particularly relevant to the researchers?

[00:03:01.09] - Steve Hernandez
Yeah, no, well, first off, thank you so much for having me, Justin. You know, like you mentioned, I had the pleasure and the privilege of working with you, you know, for about a year, year and a half or so, and helping you build out the partnerships program. And that was a ton of fun. Of course, you and I, you know, we hit off a great friendship a few years back at HackerOne, and, you know, haven't looked back since. You know, so thankful to have you, your friendship in my life, And, you know, thanks for inviting me to the podcast today to talk a little bit about this, this later project that you and I have been, you know, thinking about and, you know, putting some time and, and love into, like you said. While in my day job, I've actually, you know, moved into cloud runtime security. My passion and my love in my heart definitely still lies in part with bug bounty and the ecosystem. At the end of the day, I really just wanted to find a way to contribute beyond just finding vulnerabilities and submitting them. And, you know, I wanted to do something that would benefit both programs and hackers. So a bit about the why behind that for me is kind of my journey started at HackerOne, and I started there by working with programs. I would help them set up their, you know, their policy pages, you know, basically their storefront for you as a hacker to come in and participate, right? Hopefully find some great vulnerabilities and have a wonderful experience and come back and dig deeper and provide additional research that, you know, they that could provide great security signal ultimately to these organizations that they could use and, you know, action at the end of the day. And so out of that was kind of, you know, this idea was kind of born. And I didn't have the time, and I'm sure many other people have talked about this, right, as a concept, this Bug Bounty Maturity Framework. But, you know, it's not enough to have an idea. You've got to carve out the time. And so I was able to do that in the last year or so and really ideating on this idea of a framework.. And in particular, I wanted to tackle it from the researcher perspective. Now, you might ask yourself, well, why? Why from the researcher's perspective, right? There's operational aspects, there's teams aspect of this, there's, there's, it's a very complex, you know, thing to set up a successful bug bounty program. You need funding, you need buy-in, engineering, security folks. I mean, you know, just about everybody in, in the house, right? And so, for me to answer that succinctly, real quickly is, Hackers are the one and only ICP, right? To use a marketing term, they are the ideal prospect, the only prospect of a bug bounty program. Without the hacker, you don't have anything at the end of the day. Similarly—

[00:05:38.12] - Justin Gardner
Amen to that. Amen to that, dude. The platforms are, and some of the programs are forgetting this nowadays, dude. It is true.

[00:05:46.50] - Steve Hernandez
And you know, you make a funny point here. You know, you said nowadays. I'd actually say it's been going on, unfortunately, among some, some groups from the very beginning, you know, for many years now. One of the reasons, or one of the impetus behind me setting this up was it was really disappointing and disheartening to see so many potentially good programs decide to turn away from bug bounty because the leadership was unable to realize a return on their investment. At the end of the day, right, this is like anything else in business. These organizations aren't running a bug bounty program to have a financial loss. They're running them because ideally they would get great security signal, the ability to identify themes, patterns, you know, find out if there's, you know, patternistic issues that, you know, are pervasive within their architecture, their technology stack. But the only way that they can surface that is by running again a, you know, when it comes to bug bounty, a program that actually attracts the vector that, you know, that is intended to utilize these programs. Again, the hacker, to your point. And so, I got really tired of seeing, again, good programs go by the wayside because they didn't understand where they were falling short. And so, that's where the Bug Bounty Maturity Framework was born. I wanted to create something that would help programs to self-identify without the shame, without the public callouts, because God knows, right, where we are good about being loud as hackers and saying, you guys really stink. You guys are dropping the ball on this. Like, we are very vocal, but I really wanted to create something that removed all of that bias, removed the shame potentially from a program going, hey, I really want to know where I'm doing great, but also where can I do better? And so this framework really allows, um, program owners who care about maturity to use it as a mirror and a map, if you will, right? That's, that's kind of a really cool analogy because you're able to self-assess if you're honest, and hopefully you are if you're taking the time to take this assessment in the first place, and go, okay, where am I on my journey of the bug bounty when it pertains specifically to how we're providing this interaction for the hackers, the ICP, right, going back to that term, and where can we do better? And just, What I wanted to make sure that I did was to be prescriptive enough that it really gave you some actionable ideas and thoughts that you can action plan within 90-day span to really start to see some improvements. And so, again, that's to help the programs, to help them to avoid, you know, running into the frustration of, "We tried Bug Bounty, the platform that we were partnered with, or if we're running it ourselves, we don't understand why hackers aren't submitting reports or why they're not submitting valuable, helpful reports." This really allows you to look at the three pillars as I learned them from working directly with you guys and with programs over the years at HackerOne and at the podcast, and quickly identify, okay, these are the likely culprits. Let me apply some of these fixes. Let me work towards remediating some of these potential issues. And hopefully, we see some of that, you know, start to change.

[00:08:57.15] - Justin Gardner
Okay, so, so what we've got here is a self-assessment framework, right, that people can go they can take a quiz, they can understand what metrics they need to improve on as a platform or as a, as a program, excuse me. And, you know, understand what actionably the next steps are for their organization. And then you released this, we did talk about it on the pod a bit already. And what this annual report for 2026 is, is, I guess it looks like n equals 33 here. So 33 programs worth of results for this and it shows where they lie on your, your spectrum of maturity.

[00:09:36.38] - Steve Hernandez
Yes.

[00:09:36.75] - Justin Gardner
So, so what was the, what were the main takeaways from this State of Bug Bounty Maturity Posture report that was released?

[00:09:44.59] - Steve Hernandez
Yeah. Yeah. And I got long-winded. Sorry, you asked me on the opener, but there I go getting long-winded.

[00:09:50.19] - Justin Gardner
You're good, man. You're good. It's good. We did need the back, you know, we did need the backstory a little bit there to understand what this product was in the first place. So I think that's good. I know you have 3 major pillars: researcher interface, operational signal, asset hygiene, right? And then a bunch of subcategories under that. And yeah, so if you could speak to that and to where the programs fell on that maturity framework and what that means for us as an industry.

[00:10:18.35] - Steve Hernandez
Yeah, no, absolutely. So I'll dive right in. So first off, speaking to the N33, it was— I wanted to be really arduous and really self-critical when I you know, came to the data, I wanted it to be very defensible when I put it out there. Right. And so not only did I want to gather the information, but I wanted to also kind of scrutinize and make sure that it was good data points that were valid and that were truly coming from at least at a higher, very high likeliness from programs self-assessing. And so you and I talked about this before the pod, but, you know, we did have a little over 70 submissions that we captured. By, you know, about mid-March. So, you know, about 2 or 3 weeks before I actually published the research. And so in looking through them, outside of the 33, there were a substantial amount of what I think were tests, not just by people just, you know, from the platforms kind of checking things out, of course, naturally curiosity, right? But also hackers who were very clearly running some self-assessments themselves, right? Just, and I could tell that by—

[00:11:26.52] - Justin Gardner
Interesting.

[00:11:27.25] - Steve Hernandez
Interesting. Even though they were, you know, using, I could tell that by, you know, some Burp-related workflows that were in there. Hackers gonna hack, right?

[00:11:37.19] - Justin Gardner
Hackers gonna hack.

[00:11:39.22] - Steve Hernandez
And so, of course, I wanted to remove those before going through and really understanding where the patterns, you know, were. And so, again, went through that and narrowed that down to 33 that I verified were actually programs. So for when it comes to data, right, one of the things I learned from my Gartner Research days was that, you know, depending on the size of the industry or the niche, you know, can really dictate how valid and, you know, how many data points you might need to start to make a case for something. Right. And so given the fact that, you know, we have 33 real programs that were verified that took the self-assessment, this gives me a really strong indicator of patterns and themes. Now, it's still, you know, a small subset at the end of the day, you know, of, of where it could be. And I'd love to see this get up to, you know, 70 or 100+ inputs over time to really say, hey, I feel very confident there's a, you know, 90% likeliness that this is, you know, pervasive across the ecosystem. But for now, I want to be clear, this is a smaller sum, but these are verified programs and within a niche You can usually get away with a smaller number and make some pretty strong assertions. So yeah, I just wanted to kind of call that out.

[00:12:52.64] - Justin Gardner
I think 33 is great, man. I, I think 33, you know, for, cuz this is not a super short, you know, assessment. I mean, it's not super long, but it's like, you know, it requires you to sit down and do some thinking. Right. Um, and so that 33 programs or, you know, more than 33 maybe, uh, but at least 33 went in there, did a, you know, sat down, put their thinking cap on. You know, went through the assessment, I think that's really quality data and probably pretty representative of the, of the ecosystem. So yeah, dude, I'm really impressed. I was just, you know, reading through this. I'm impressed with the quality of this research, especially like knowing that you're not like an academic, like a PhD or anything like that, right? Like, I think this is the research procedure that you went through here was, is really clean. And I think we can pull some really good insights out of it because of that.

[00:13:45.67] - Steve Hernandez
Oh, thank you. No, I really appreciate that, Justin. And again, completely agree. You know, I know that a niche like bug bounty and other niches that are smaller like that, you can usually get away with, you know, even 20 as your N and make some pretty strong assertions. But again, I just, I want to be clear, I want to be defensible. And so I'd rather err on the side of caution, you know, when it comes to these things. But that being said, let me share some, some really interesting numbers that came out of that. I think the most interesting to me was, and this was kind of my thesis from the get go, right, based on all the background I shared with you today and what you and I both know from being on kind of both sides of bug bounty. And that is that a lot of times, and this was a catchy little phrase that I did in the research, but operations mature before relationships do and confidence in that, right? And what that means is that a lot of times bug bounty programs are great about investing in some of the workflows and getting, you know, setting up their policy, some of their design and procedure. But the thing that a lot of them have a hard time with is actually investing in building the relationship with the community, the very community that they're trying to attract. And so not surprisingly, a little over 61%, um, uh, of the people that submitted these, you know, these, uh, self-assessments were, had a weakness in their research interface pillar, right? So there were 3 pillars, research interface, operational signal, and asset hygiene. And for 61% of these programs, that was the weakest pillar.

[00:15:14.41] - Justin Gardner
Wow. That is, that is very telling. Yeah. I mean, I guess that is telling to me. I know what the three pillars represent, you know, researcher interface, operational signal, asset hygiene. Give me a quick, like, one-liner on what each one of those represent.

[00:15:31.11] - Steve Hernandez
Yeah, absolutely. So, so basically the researcher interface is really— it comes down to the relational aspect, right? So anything from, you know, communication of expectations responses, escalations, you know, communication of severity, anything when it comes to building the relationship with the hacker. On the operational signal, it, you know, kind of within the title itself, right? It's more of the operations, right? Do we have a policy in place for these different things? Do we abide by those policies? Do we abide by them consistently? Which is the other piece, right? Consistency is an important aspect. And then asset hygiene. That was a really interesting one for me to land on, but really important as well. And that really comes down to, have we prepared our house for these hackers to come in and do their testing?

[00:16:18.91] - Justin Gardner
Right.

[00:16:19.62] - Steve Hernandez
I'll give you a quick little analogy there that really resonated with me when I was thinking about this framework and debating, right, which way I should go with that third pillar. I went with asset hygiene because when you think about it, right, your level of preparedness when it comes to your house and inviting guests into your house really is indicative on how much you prepare your house, right? Yeah. Well, the more you care about the guests that are coming in, the more you're thoughtful that you're going to do the dusting, you're going to do all the detailed work, right? You're going to make sure it is spick and span, beautiful, clean, vacuum. You're going to light a candle, the whole nine yards, right?

[00:16:54.35] - Justin Gardner
So I'm going to come visit you, man. That sounds good. I've extended that to you and I. I'll be looking for that candle next time I stop by your place.

[00:17:01.88] - Steve Hernandez
Absolutely. You'll be like running your finger through the windowsills and everything. But it's, it's, acid hygiene is that idea. How much, you know, effort, how much time, how much love are you putting into preparing your house to invite these researchers in to actually run their testing? Because from what I experienced in working with programs, unfortunately, a lot of them really struggled with that piece. They would go, here's the scope, but not once test to make sure that it was actually accessible or useful and reachable for the hackers.

[00:17:33.56] - Justin Gardner
Well, that's interesting, man. You know, looking at the three of them. I mean, it is pretty telling that researcher interface is, is, uh, the lowest. Um, I, I definitely think that is something that I've experienced. I also think that asset hygiene has probably a massive area where it can improve. And I'm curious because you actually did drop a graph in here somewhere. Um, yeah, I think up operational signal was actually— so the middle one was actually the one that Uh, so I guess it depends on the tier, cuz I'm looking at these tiers here. I'll put the graph up on the screen, y'all. Um, depending on what band you are, emerging, developing, established, advanced, uh, it depends on where that those other two are gonna be. Because they, they flip-flop between one and the other here, right? So if we look, for those of you that are following along on YouTube right here, uh, you know, OS, operational signal. Is the next one above researcher interface in emerging, but then in developing it flips a little, or it flips a little bit here. Operational signal is the highest and then asset hygiene is right next to researcher interface. So we don't see a consistent second place here across the various tiers, but we do see a consistent first place, which is always that, that researcher interface.

[00:18:56.42] - Steve Hernandez
Yeah, yeah, that's a really astute observation. And yeah, it was really interesting to me about how the investment shifts, you know, seemingly for these folks that are running programs as things progress and mature, right? Like you said, it's not linear for operational signal and asset hygiene, but it's, it's pretty stark and pretty telling to see how the research interface is consistently the lowest across all maturity levels. Hmm. Again, just further painting the, the point that, you know, that's kind of the last space we're investing in, unfortunately. And again, I, I saw it, you know, firsthand, and I know you've experienced it firsthand.

[00:19:35.19] - Justin Gardner
Yeah, yeah, for sure. It's very, it's very interesting data. Asset hygiene, I definitely think I would love to see a big increase in, in the community. You know, just giving us data in our test accounts, you know, and making sure we have access to credentials to get, you know, into certain parts of the scope, that will also really improve. I think operational signal, probably one of the reasons why that is a little bit more worked on is because I think it's probably the more systemifiable, you know, like you can make this into a system, right? Like, you know, okay, this is after the report comes in, it goes into Jira and and, you know, if it's in this range, then it gets this amount, you know, I believe it's probably pretty system, system heavy, whereas asset hygiene, you probably have to coordinate with the, with the dev team to like get these credentials and get, you know, a staging environment or an SSRF share or whatever.

[00:20:30.88] - Steve Hernandez
Yeah.

[00:20:32.32] - Justin Gardner
And then at the end of the day, these engineers are probably thinking like, oh, researcher interface, that is like probably not the thing that I am going to excel in, you know, as the engineer, as the AppSec engineer here, like, you know, going in there being like, hey, wow, great report, you know? But that makes all the difference, man. As a researcher, it really does. When you get a message from the team member, you know, that's not coming back from H1 Triage, it's the actual team member, you know, within 6 hours of you submitting your report, you're— and you're like, oh shit, this is an amazing program. I cannot wait to like work on them, you know?

[00:21:06.75] - Steve Hernandez
Exactly. And that's what I experienced too, on both when I was in, you know, both roles, whether I was running programs and as a CSM, right, customer success manager, or whether, you know, it was when I, Chad and I started off the hacker success function organization and, you know, dealing directly with you as hackers. It was the same story. You know, the programs that saw consistent, you know, play from the hacker community and consistently great security signal from you as a community, It was those programs that you just described, the ones that were engaging with you directly, the ones that actually took operational ownership, didn't just offset everything to, to the platform, right? They were involved. They weren't scared or apprehensive or worried about, you know, paying you a compliment or saying, hey, thanks so much. This was a really interesting find. You know, tell me more about this. Some really cool examples that I saw from, from some teams were like things like Hey, you're— this is really interesting. Here's, here's some other guidance for you. You poke them around. And I just, I just remember, you know, the hackers that I got to deal with, they would eat that stuff up. And I mean, they became loyal.

[00:22:12.27] - Justin Gardner
I freaking love it, dude.

[00:22:13.78] - Steve Hernandez
Right?

[00:22:14.51] - Justin Gardner
Oh my gosh.

[00:22:15.66] - Steve Hernandez
It doesn't take a lot when it comes to relationships, but it's, it's harder to measure, I think, you know, for, for a lot of these folks that are, you know, looking at metrics. You know, but, but unfortunately and fortunately, it's one of the most important things in most aspects of business. And it certainly hasn't been different for the bug bounty program side of things, you know, and I think that's why, you know, Bug Bounty Maturity Framework is so helpful and so pivotal at this time, especially in the rise of AI. I think relationship has become even more important because there is— it's so easy to discover, it's so easy to submit. You have so much volume. The difference maker I'm banking on is relationship and trust.

[00:22:57.22] - Justin Gardner
That's interesting.

[00:22:58.10] - Steve Hernandez
Trust is the currency.

[00:22:59.29] - Justin Gardner
That's an interesting thing. Okay, let me, let me think about that for a second. So you're saying, you know, even with AI, the volume is through the roof, you know, that, that researcher interface piece is, is even more key. And that rings true because most programs are gonna kind of be treating us like crap right now. And that's what everybody's sort of experiencing. Because they're overworked and they're overwhelmed, you know, which is fair. You know, there's lots of grace for the season. Everybody's trying to figure out the AI, you know, twist on everything right now. But one thing that does pop into my head as well is I wonder if from the researcher side, we will see a decrease in the importance of researcher interaction because there is less investment in the researcher in their own report because of the effects of AI. For example, right before this call, I submitted 3 bugs to a program that I have a great relationship with. I did not find any of these bugs. These were completely found by my HackBot, right? And I went in, I went in there, the POC was there. I just like literally curled it and piped it to Python. And I saw the data come out and I was like, great, this is, this is awesome. And I submitted the report. And surely if the team comes back and pushes back on this, you know, it will, it will hurt my heart, but it's not going to hurt my heart as much as, you know, that bug where I like put my blood, sweat, and tears into it. And I like banged my head up against the keyboard until the bug fell out. Right. And then I wrote the report with all of the passion that, you know, that had been welling up inside me for the past weeks as I've been writing, you know, fighting against this target and just to have them be like, sorry, you can't reproduce, or this doesn't seem like a real bug, you know? Yeah. So I don't know. Do you think I'll, I'll pose that same question back to you. Do you think that that is the same trajectory that we're going to see that researcher involvement is going to continue to be super important? Or do you think that that will actually decrease as researchers are less invested in their own bugs?

[00:24:58.27] - Steve Hernandez
That's a, that's a great, uh, great question and great framing. Uh, really interesting framing. I think we could talk a little more about everything you just shared. Um, I found it interesting that you qualified the start of your sentence though with submitted to a program that I trust.

[00:25:12.32] - Justin Gardner
I don't know if it's—

[00:25:14.58] - Steve Hernandez
I think therein lies the rub, as they say, right? I think that, I think in this new economy, a few different things, but one thing that I'm thinking about is the shift. And I was talking to Shubs about this as well. He wrote, he did a write-up recently on his experience on one, a program on HackerOne. I think that because of AI, I think that relationship and trust will continue to be important and even more so than it was. I think it's even going to surpass the emphasis on bounties for the very reasons you just talked about, right? The cost of discovery, you just, you just said it, has, has been dropped significantly. What hasn't changed is the importance of knowing that the programs you submit to are going to treat you respectfully, consistently, and that you can have a useful, productive disagreement and conversation with them, knowing that it's going to— that when you land on a, you know, on an end result, it's going to be one that you feel good about and not one that you're feeling mistreated or that was mishandled. I don't think that's going to go away anytime soon. What do you think?

[00:26:29.15] - Justin Gardner
Yeah, yeah, I, I tend to agree with you as I think on it a little bit more, you know. I think that, I think that will probably be a little like, I don't know, like almost like touch-starved, you know, like, like, like to put it in like the romance terms, right? Like, like you get so little of this nowadays, right? Yeah, you know, that it, when you do get it, it's like, oh wow, that's amazing. You know, so I think it will still have a very strong impact, whether it will— man, my green hat is, is, you know, shaking a little bit on my head here. But I do think— I don't think it will surpass bounties. I think bounties will be king. If there's anything that's consistent across the bug bounty ecosystem, it's that hackers, they like their— they like their bounties, man. But, but I do think that, you know, I think that, that those are probably the two strongest components. Is the bounty there and is the team talking to me, right? Yes.

[00:27:30.39] - Steve Hernandez
Um, let me ask you this then, right? Because I don't disagree that bug bounties are important, right? Or the rewards table will say, right?

[00:27:38.14] - Justin Gardner
Yeah.

[00:27:38.25] - Steve Hernandez
Are important. Agreed. But let's say you're in a scenario where you're dealing with a program, right? It's a new program. They have, they're, they're paying at the very highest of crits and highs for you that you've seen. But, and then you have, you know, program B that's maybe a mid-level, mid-tier program as far as crits and highs, right? So not the lowest, not the highest. However, A, you've submitted a couple of crits and you've heard nothing. It's like submitting into a dark void. On the other hand, program B. You've submitted and they're responsive. You get your first response within a couple days. Within 2, you know, you've been triaged. Within a month, you've been paid and you've been treated well. Which one are you going to continue to spend time on?

[00:28:25.88] - Justin Gardner
Okay, well, hold on. There's a nuanced answer to that question though. It depends. Like, the thing is, new program, I, do I know that their threat model is validated, right? Like, okay, look, if they're just dropping PII everywhere and it's like clearly a crit or like I have a shell or something, then, and they're not responding for forever. One, that's going to piss me off. Cause I'm like, this is a critical vulnerability. Like, please help a brother out here. Um, but I'm not going to be as worried about the bounty because it's like, okay, how can they, especially if there's a platform involved, right? Like how there's no way that, that they're not going to pay this, you know? Um, however, one of the, one of the beautiful things about quick response times and good researcher interface, to use your, your pillars here, is that you get to validate your threat model so much faster, right? Like, I'm trying to understand, you know, this company's threat model, what data they think is valuable, what assets are important to them, what is their core business flow, right? And I— the way that they validate that for me is by triaging my reports and paying bounties, right? So I say, I know I'm on the right track here with, you know, this IDOR or whatever, right? And so I think while I would be tempted to hack on that program B much more, and I, yeah, I do feel what you're saying. Um, I think that it's always nuanced at the end of the day, right?

[00:29:47.29] - Steve Hernandez
It is always every, every question you ask me or ask you, you could always answer, well, well, it depends.

[00:29:53.58] - Justin Gardner
Yeah.

[00:29:54.57] - Steve Hernandez
You know what I mean? Like there's always so many variables, but if we want to be a little bit binary, then to get some in the conversation. I, at the end of the day, I don't think that trust will be less important. I think it'll be more important as we progress into this world of AI, which we are already way into, right? And things only exponentially growing. I think bounties will continue to matter, of course, and I think it'll continue to serve as a priority matrix more than just like, hey, this is where I am or I'm not going to spend, you know, my time because To your point, if it's easier and cheaper to discover, then perhaps you can afford to get wider with your net and still get a similar result from programs that are actually responsive and, you know, working with you in a way that you feel is fair.

[00:30:41.51] - Justin Gardner
Yeah. Yeah. No, that, that's fair. That makes sense. I, I, I would, I would be inclined to agree with that. And, and I think to answer your question though, as I, as I'm thinking about it more, yeah, I think I would hack on program A. Until I found some bugs. And then I would hack on program B. And I would continue to hack on program B until program A paid out. And then as soon as program A paid out, and I know that my threat models are valid, and that their rewards table isn't just like a, you know, a thing that I'm never gonna get, you know, they're like, oh no, this is all low, you know, like, then I would, you know, as soon as I knew that it wasn't that, I would go back to A and I would gridded out in the pipeline, right? And then, you know, hack for a month so that the month ago's bugs are getting paid, you know, today. And then we have that the next week, the next week, and I'm still getting, you know, bugs coming through in the pipeline. And then I'd be harvesting those bigger bounties. I will say the other piece of that is sort of technically intriguing scope for me. Like if, if program A here was just like, I need to sit here and bang my head up against the wall with IDORs. Like the whole time, uh, you know, and program B had like a sandbox and like a, you know, crazy thing that I could, you know, play around with, I would definitely be more inclined to go over to program B. Um, but so where does that fall in your, in your, um, your pillars here? Like how interesting the scope is?

[00:32:07.50] - Steve Hernandez
That's a great question. Um, so I basically, I, I didn't look at it from a Um, I guess that doesn't really relate to maturity very much.

[00:32:16.48] - Justin Gardner
Yeah.

[00:32:17.45] - Steve Hernandez
So, so here's the thing, right? You know, particularly in my HSM role, what I learned was that even among the top tier of hackers in the world, there are slightly different avatars. Again, going back to marketing, right? Yeah. Not everyone has the same goals. Not everyone has the same interests. And you are all on the top, let's say, 200 hackers in the world, right? Even the top 50. Everybody's goals were slightly different. And so I didn't want to build something that was specific to like, well, this hacker wants only 20,000 crits and he only wants technically, you know, interesting scope and it has to be public cloud. Like, you know, I think, I think we're getting, you know, outside of the scope of a maturity model from the researcher experience perspective, which is what, you know, what this is intended for. So, yeah, I think you nailed it. I think that's a little bit outside of the parameters for, for what we're doing. Um, but it is an interesting topic to talk about because I think when it came to my times when I was on the program side, so before the HSM side, one of the biggest things that I had to help programs with was identifying their likely candidates. And a lot of programs come in and they're like, oh, we just want these, you know, top 50 guys that we know, guys and gals that we know about. Like we, we want these people, but I would have to coach them and consult with them and go, well, hold on, let's talk about your scope. Well, these 2 or 3 researchers, they won't even touch that scope. That's not something they're even interested in. So, you can invite them, but they're, they're never going to spend time hacking on the program.

[00:33:45.65] - Justin Gardner
Yeah. Inviting, inviting Today Is New to a fully, you know, locked down auth-based scope is not going to work.

[00:33:52.30] - Steve Hernandez
That's a great example. Exactly. And so, I think it's really, it's an important topic to cover anyways, right? For programs, because, um, a lot of the newer ones into the world of bug bounty, they don't understand that. They're just like, I just want, you know, I'm coming to the platform. Crowdsource, you know, hacking, great, love it. I want these, you know, top people that I know about that are famous in the ethical hacking world to hack on my program. And a few might be great for that, but not everyone is interested in, you know, that scope. And so you just have to understand that and, you know, consult with your people that you're working with.

[00:34:25.96] - Justin Gardner
Let's take a quick deviation away from the State of Bug Bounty Posture, Maturity Posture Report here, and we'll swing back around to it just in a second. We've talked a little bit about verification of this for the researchers, and let's, let's go down that rabbit hole really quick because as I'm looking through this, it would be great for me to, to be able to have, you know, look at a program and in addition to all of the badges and whatever that HackerOne and Bugcrowd put on their programs, also know, hey, this program has been vetted by a third party and comes out at this specific, you know, tier of maturity. So talk about your vision a little bit for that.

[00:35:07.34] - Steve Hernandez
Yeah, no, I'd love to. Thank you. That's, that's a great question. So, um, again, going back to, you know, we started with the self-assessment, right? That's meant just for the program leaders to use it as a mirror and a map, right? You can't, it's hard to chart a path forward when you don't know where you are on the map, right? You might end up completely lost and confused. And so that's the starting point, but that doesn't signal to the hacker community where they should spend their time, where they're going to get the best ROI on any investments they make, you know, where should they go and set up their stack and figure out how deep they want to go, right? And so what came out of that, you know, expression from, from you and from other hackers that I, you know, kind of validated that self-assessment with was, what about us? What, what can we do for us? And so what's been kind of born out of that that I've been developing is, you know, something we're calling verification. And the idea is to create a standards body, if you will, right? Or at least begin that process that's very transparent and that basically matches the self-assessment, but audited by third independent, unbiased, objective body, right? Like Bug Bounty Maturity Framework, where you're actually able to stress test whether, you know, what they're claiming is actually happening, right? Operationally, they're really having these types of conversations. Their reports show it, you know, the research community is saying, yes, I'm having that type of experience with them. And so what that will do, I think it'll do a couple of things on the researcher side. To your point, I'd love to eventually have like a directory, right? One for public programs that have gone through this verification process to use as a signal for them to go, hey, where do I— I want to go hack. I need to find a couple of new programs and they can sort through the public directory and see, okay, these have actually been verified through BBMF. We know they're at least an established mature program or at least an advanced mature program, I want to go spend my time there because I know I will be treated well. I know they have this level of maturity, right? Which is, I think, the, the ultimate vision. On the other hand, on the other side, it's going to be a great acquisition tool for programs, right? Like we were talking about just a few minutes ago, I think trust is the most, you know, expensive currency right now. And that'll immediately— the idea is that it would elicit trust in the hacker community. And they're like, awesome, because of the transparency, because how it's been built out and the fact that I'm not attached to any particular program or platform, they can trust that it's being treated fairly and correctly.

[00:37:33.05] - Justin Gardner
That's a, that's a good, that's an excellent point. You know, you said program or platform because like the program, obviously they want to get a good rating, but the platform also wants them to get a good rating so that they keep their program open and that they are, you know, can say that they're providing good value. To the researcher. There needs to be a third party that does this. And I'm thinking like, man, it would be really great if the third party also wasn't aligned with the customer. Like, like if the customer didn't have to pay the, the, you know, uh, Bug Bounty Maturity Framework to get that assessment. I don't think there's any way around it. The only other way I could think of is if the researchers themselves funded, you know, these verifications across the board, right? Because then the incentive would be completely aligned, right? Like nobody at Bug Bounty Maturity Framework has any relationship, you know, with the, with the company. But then how do you get the access? How do you get them to cooperate? You know, so I don't know. I feel like, I feel like it will have to be something that's done between Bug Bounty Maturity Framework and, and the actual company, you know, the actual program themselves. And we should just have a set of controls in place to ensure that these outcomes are being served well. And, you know, it is aligned with your global best interest as well, you know?

[00:38:53.73] - Steve Hernandez
Exactly.

[00:38:54.51] - Justin Gardner
So I think that that's a satisfactory solution.

[00:38:57.32] - Steve Hernandez
Yeah, no, those are great points. And it's funny you mentioned the work controls because that's exactly what I've been working on and developing, right? These controls that will align with, you know, the pillars and the dimensions from the self-assessment. There will be a, you know, mapping, if you will, right? So that if you know if you're being completely honest and transparent within the self-assessment, there should be some pretty close, uh, you know, um, variation when it comes to the audit of— unless, you know, you completely withheld in some areas, or maybe not completely aware of all of the inner workings. That could happen too, right?

[00:39:28.21] - Justin Gardner
Maybe, maybe they're using faulty metrics. They pull from their, their data. Okay, you know, how fast did we get back to a researcher on our vanity? They pull, they pull the first reply, right? But then, you know, the first reply is like Welcome.

[00:39:40.34] - Steve Hernandez
Exactly.

[00:39:40.96] - Justin Gardner
You know, that sort of thing.

[00:39:42.42] - Steve Hernandez
It's AI answering AI, which exactly is not— it's not always a very good metric. Yeah, more like vanity metrics. But, but yeah, back to what you were saying. Yeah. So developing these controls and I'm going to, I'm going to publish them once they're, once they're ready. I want to be completely transparent about what someone might be signing up to get verified and audited on. And to your point, these, these programs, they're going to have to be willing and inviting you a little bit into their world, right? And going, hey, here's what— here's what's going on. In order to audit what's really taking place, I'm going to have to see some report samples. I'm going to have to really go through and check everything out. And not just— it can't be that they're just telling me that they're doing these things. I have to validate that they're truly doing these things in practice. If I can't do that, then this doesn't have any trust and it won't— it won't stand the test of time.

[00:40:31.90] - Justin Gardner
Doesn't work at all. Yeah, no, that makes sense. Well, I'm glad, I'm glad that, uh, yeah, and surely that verification process will take some time, but I'm glad that that's where we're moving. And I think that that will enhance trust. Um, and you know, community members that are listening now, as soon as this comes out, I think we ought to crush the first program that, that does this verification, and we ought to just go submit them like a million reports because You know, they took that step, they went through that process, and we know that they are invested in building their maturity as a bug bounty program. And I think that's half the battle, to be honest, because half the time people are not, you know? So just showing up on the list, you know, no matter where you are at on the distribution, is a good positive signal for sure.

[00:41:20.05] - Steve Hernandez
Yeah, no, agreed. And on that note, a huge shout out to the 33 verified, you know, which programs that took the self-assessment honestly. I was really impressed by that. That. And, and I'm, you know, so, so thankful that they were willing to trust this guy, you know, that created this application and went through the process of submitting and entrusting their, their data to us. And so, you know, thank you again. I'm excited for the next, the next stage of this process, and I'm truly hopeful. And I believe that being able to signal, signal to the hacker community that they can trust and that they can, you know, that they will be treated well by your program, I think that'll be really important as we continue moving through this age of AI.

[00:41:58.65] - Justin Gardner
All right. So actually, as we're reading through this, as we're doing this episode, I'm loving this. We need to get more programs taking this assessment. So here's what we're going to do, y'all. Pick your favorite. Well, I shouldn't say your favorite program. Pick your least favorite program. Pick a program that you think would benefit from doing a Bug Bounty Maturity Framework assessment. Okay. Send them to bugbountymaturity.com. We're going to put a little snippet down below. And if you put that on a valid report, to those, those programs, um, and then submit a screenshot of it, we will enter you in a raffle to win 3 months of, of, uh, Critical Thinkers, uh, tier, uh, in the CTBP Discord. Okay. So we've got, I guess I should make that clear. We've got 3 1-month subscriptions, uh, that we're going to give out, uh, and your name will be entered into a raffle for that. Okay. Um, so check out the, the link in the description for what you should send to the programs and how to submit your proof that you've done on it. All right, let's go back to the show. Okay, jumping back to the State of Bug Bounty Maturity Posture Report. That is a little bit of a, uh, of a— it's a long title, man. We gotta, we gotta come up with something, uh, shorter than that.

[00:43:08.42] - Steve Hernandez
I'm all ears, I'm open to it.

[00:43:10.09] - Justin Gardner
Yeah, yeah, I don't have anything helpful to say, but, um, no, I, I just wanted to jump through some of this data. So the overall score was a 3.06 out of 5.

[00:43:19.71] - Steve Hernandez
Yeah.

[00:43:20.82] - Justin Gardner
Meaning that, you know, out of, out of these 33 programs, that's where most people were sitting across the 3 pillars. Is that, is that kind of what we're looking at there?

[00:43:30.69] - Steve Hernandez
Yeah. Yeah. That was the mean score across all 3 pillars. So barely into established, which again to me feels pretty honest and is another great indication that these folks were really honest and transparent with themselves. And I really appreciate that.

[00:43:44.36] - Justin Gardner
So, so emerging. You know, that's the lowest tier. The fundamentals are established, but execution is largely reactive.

[00:43:52.17] - Steve Hernandez
Yes.

[00:43:53.05] - Justin Gardner
Developing, core processes are defined, but consistency has yet to be stabilized. So they've got some procedures in place, they've got some systemization.

[00:44:01.50] - Steve Hernandez
Yes.

[00:44:02.46] - Justin Gardner
But it's not, they're not really hitting full stride. And established is where they've kind of got clear expectations and generally consistent execution.

[00:44:10.73] - Steve Hernandez
Yes.

[00:44:11.46] - Justin Gardner
But They're really, they're not like super nailing it. And then we go to advanced and leading, which are like sort of above and beyond.

[00:44:18.69] - Steve Hernandez
Yes. Yeah. Okay.

[00:44:19.94] - Justin Gardner
Yeah.

[00:44:20.05] - Steve Hernandez
So I think, I think the biggest differentiator between established and, and moving into advanced, you know, which I think only about 18% of the submitters were able to realize, um, the, the big difference there is, can you execute at the established level when it comes to seeing spikes in your load? And that's become even more important. In today's age because it went from even a year ago, right? Like you would see spikes in activity after certain events, right? Certain talks or, you know, certain inciting incidents, if you will, in the ecosystem. And so it would drive up volume. They would try to get through that and then it was back to stable. Well, now it's under load often, if not always, right? Any particularly any public programs like they're under a huge strain of volume and we've seen You know, we've seen the various leaders at the different platforms talk about this and some that are trying to figure out how to, you know, address that in various different creative ways. And so I think the big separator between an established program and an advanced one is can you still function consistently under heavy load? If you can do that, then you're likely in that advanced category.

[00:45:28.38] - Justin Gardner
Yeah, I mean, it's going to happen so often, too. You know, you mentioned inciting incidents or whatever, but just say a you know, hacker gets interested in your program and they just like, you know, blow it up with their friends, right? Or you point a hackbot at it and this hackbot is particularly, you know, doing well, right? Yeah. Definitely going to happen. We mentioned before that RI was the weakest pillar for 61%. What do you think if you had to give a tip to the program managers listening here? What do you think is going to be the strongest signal thing that they can do to increase their reacher— researcher interface, um, success?

[00:46:10.28] - Steve Hernandez
That's a great question. Um, so first, go to, if you haven't, go to bugboundingmaturity.com, take the self-assessment, see where you are. It'll give you some prescriptive ideas. But at a high level, I think the most important thing that I've learned from working with the hackers directly for a few years now is just be honest, be transparent. You know, if, if you're truly going through some changes, turnover, budget issues, whatever it is, be appropriately transparent about what you need to be and make sure that what you have on print in your policy page, right, that entry or doorway to your assets actually lines up with true expectations, right? Because if you, if you send out a note, per se, to the research community, hey, we've just had a transition or a changeover in staff. We are no longer able to meet these expectations for now. We strive to get there again in the next 30, 60, 90 days, but please have some grace with us for the next, you know, couple of months as we ramp back up. Delay— we're going to be delayed in our first response. We're going to be delayed in triaging, and we're going to be delayed in, you know, time to bounty, but we will get to you. I think just being honest and transparent about those things with the hacker community is going to go a huge long ways. And then really actually putting the effort into making those changes and not allowing, you know, things like staff changes in the long term actually impact you, right? That's what they call, I think in the military, a single point of failure, right? If everything lives in your head as a program manager, you've got to get it out of there. You've got to document it and you've got to train others around you so that if and when you transition away for any number of reasons, Things don't fall apart when you leave.

[00:47:50.55] - Justin Gardner
Absolutely. Yeah, that is, that is a great point. I think that I respect the programs that keep me up to date on what's going on. Hey, we've had, we've had a major influx. We're running behind. We've got a plan to fix it. You know, just, just keep them in the loop. Um, exactly. Several programs. I've, I respect Adobe, you know, uh, this one that I'm hacking on now that it, oh, this is Instacart is actually a public program. Shout out to Instacart.

[00:48:15.46] - Steve Hernandez
Out.

[00:48:15.94] - Justin Gardner
Uh, they went public now. Uh, yeah, so they've all done that in the past, and I've appreciated that. Um, so stepping away from RI, uh, and going back to just the way that— or some of the data that's come out of this report, we see that the program distribution, um, relative to duration, uh, is pretty interesting here in this. For those of you following along on YouTube, you can see, but for those of you not Um, if a program has been alive for 1 to 2 years, the mean score is actually 3.7 rather than the average 3.06, um, which is very interesting to me, and a much higher RI score at 3.7. Um, why do you think that newer programs, 1 to 2 years, uh, have this higher score both in the overall score and in RI?

[00:49:12.15] - Steve Hernandez
Yeah, that's, that's a great follow-up. So I think, I think there's probably a few variables that go into that. I made some, you know, I came up with some theories in the research paper that I, you know, I'd love for people to dive into, but just kind of thinking about it here again, I think that there's a couple potential things, right? One of those is, you know, when you're so new to something that you don't know what you don't know. And so as you're taking the self-assessment, maybe you, not on purpose, of course, but you're kind of incidentally grading yourself a little higher than you might be in reality. I think there's some of that, again, totally just, you know, a newness aspect to it.

[00:49:49.36] - Justin Gardner
True.

[00:49:50.11] - Steve Hernandez
Um, I also think there's, there's a, another thing at play here, which is, um, you know, you're not in that first 6-month, you know, horizon where you're just kind of cutting your teeth, trying to figure things out. You've like, you've found your pattern, your stride, uh, you're excited because it's still relatively new, but not so new that you have no foundation at all. And so you're making some genuine stride. You are very actively engaged in, you know, kind of the governance feedback loop of the program. You're trying to figure things out. You're excited. You have energy, you have resources funding this thing. So I think there's, I think there's a little bit of both aspects to it. I think what was really interesting is you see things level out in the 3+ years, and I feel really confident about this one more in particular than the others, because if you look at N, which this is important, right? There's really only 4 data points for 2 to 3 years and only 5 for 1 to 2. But for 3 years, there's 13. And even that is a little bit higher for the mean score than the overall mean. And I think that goes back to the— you're starting to know a little bit more about what you didn't know earlier on, right? You're having some realizations. Um, and so that's, that's kind of my prevailing theory currently, right? With this data set, which again has its limitations, but I feel pretty good about.

[00:51:01.82] - Justin Gardner
Yeah. Yeah. No, that makes sense. I think that that could be the case. I also think that one thing that could contribute to this is turnover, where if you've churned for a year, you know, to get your bug bounty program launched and you launch it and then you just head down, you know, for a year or two, you probably knew everything from before the program even started to the program starting to a year or two in, right? They probably haven't hit that churn mark yet. Maybe like or that burnout even as a program manager. And then that first program manager leaves and then the processes go in the garbage, you know, with them. Right. Like you mentioned. So you see that dip again at 2 to 3 years and then 3+ years, obviously, you know, representing a massive swath of the Bhagwan programs, you know. Yeah. From 3 years old to 15 years old, right?

[00:51:52.11] - Steve Hernandez
Yeah.

[00:51:52.36] - Justin Gardner
Again, you see a little bit more balance, uh, in that space.

[00:51:56.61] - Steve Hernandez
Um, yeah, those are, those are two really great and valid points. I, I agree with you.

[00:52:00.63] - Justin Gardner
Very interesting. The operation— operational model distribution as well is— it was fascinating to me. Even though this is the smallest portion, uh, self-managed, N of 6, um, we see that the mean score and, uh, the mean score for self-managed is quite a bit higher than managed service platform, which is coming in at the lowest across all metrics.

[00:52:23.73] - Steve Hernandez
Yes.

[00:52:23.98] - Justin Gardner
Um, what do you think that says about managed service platforms, which I assume is like, you know, programs that are working with, uh, almost exclusively with HackerOne, Bugcrowd, and Secret Yes We Hack versus the hybrid, uh, and then the fully self-managed, which you would see something like Google or, or, um, the ones that are working completely outside of a program.

[00:52:43.98] - Steve Hernandez
That's, that's a really great, um, question again. So I, I really took a lot of time on this particular, you know, uh, piece because of, you know, the nature of what it is. And, um, I wanted to be careful not to have it be an indictment on platforms because if you carefully look at this, right, you see that self-managed is significantly higher, a narrower, you know, data point. But then if you look at hybrid, I think this is what's really telling here. There's a pretty ample number of programs that fell under hybrid. And hybrid means that you're using a platform for intake and technology,, but you still maintain operational ownership of the program. Means they're not shy about responding to you in the reports and about getting involved. And so you can see that that's significantly higher, not as high as self-managed, but that's significantly higher also than just managed self-service platform where you offload everything to the platform and you just let them do their thing. And I saw this in some cases, right, where they're a little nervous about bug bounty. They don't understand it. Working with hackers, what? That's scary. You know, there's that kind of wall, if you will, between the program and the hacker community. And so I think this is more of a reflection of that operational ownership and where it lies. I think that you can clearly see that when the program is involved, whether they're using a platform or not, that's where you really see a significant difference and increase in maturity, particularly around the researcher engagement, above just if you offload everything and you're not involved at all. And the operations.

[00:54:15.38] - Justin Gardner
Yeah, I imagine that this is actually what the program or what the platforms actually want as well for the program. So it's like they want that involvement. They're trying to get them to be involved.

[00:54:24.61] - Steve Hernandez
Yes.

[00:54:24.84] - Justin Gardner
You know, and this sort of MSP or I guess managed service platform, we won't, we won't say, we won't mix up our, our acronyms here. But, you know, the managed service platform portion of it is where the platforms are really just picking up the slack. Which is coming from the program side operationally.

[00:54:44.69] - Steve Hernandez
Yeah.

[00:54:44.88] - Justin Gardner
So yeah, no, very, very telling data.

[00:54:47.48] - Steve Hernandez
I'm glad you say that because that was one of the biggest things for me when I first started at HackerOne was constantly and consistently saying, hey, you know, tell me about your experience, tell me about your apprehension, why are you apprehensive? And trying to break those walls down and those barriers down to help them understand the importance of having a relationship. Right. Bug bounty relationship, I think it's just like any other healthy, good relationship. If you want a good, healthy relationship with your spouse or with a friend or whatever, right? You've got to have communication. You can't not talk to them ever or else there is no relationship.

[00:55:22.71] - Justin Gardner
Right, right.

[00:55:23.32] - Steve Hernandez
So, I think that's very true even here in bug bounty. And so, yes, HackerOne, I'm sure other platforms as well, we tried, we tried to preach that, we tried to get, you know, programs as involved as possible, but Sometimes it, you know, it, it doesn't work out for any number of reasons, not necessarily malicious ones. But yeah, the ones that did get involved and were excited to work with the ecosystem, they had the best results and continue to.

[00:55:47.34] - Justin Gardner
Yeah. Yeah. I think, I think that makes sense. And, and looking at this 5 lowest scores dimension graph here as well and thinking about like why we will say, see, you know, re researcher interface making its its appearance on this table so many times. I wonder if that response SLAs and engagement, if the bottleneck is outside of security, you know, like it could very much be security. But I think one of the things that is probably smart when architecting your program from the very beginning is requiring as little input as possible, as you can possibly get away with, from your actual engineers, your software engineers, the people that are writing the code or the business owner or anything. Having people on your security team that are familiar enough with your product and the, and the threat model to say, to be able to look at a bug and say, yes, this is a valid bug for sure, 100%. I'm going to have confidence to pay the hacker without even sending this to the developers. Right. I imagine that that would have a big piece because I, you know, security is just a little, little, you know, box for most of the developers and the business people out there.

[00:56:56.80] - Steve Hernandez
Right.

[00:56:57.13] - Justin Gardner
Whereas it's our world. So I imagine that does contribute largely to why the response SLA engagement here is sitting at a mean 2.42 or 4.2, 2.42 across all the 33 assessed programs.

[00:57:13.32] - Steve Hernandez
Yeah, yeah. No, I can't disagree with you on that. And I think that, you know, even I think the graph right above that, you can see that, you know, 36% of all the assessed, you know, programs scored at a floor, or meaning the very first, where there's basically a structural absence of any commitment to response clarity and reliability. So, I think that's, I think that's very telling, and to your point, you know, and I, and I want to make this broader statement, and because I've talked to a few other people on your team as well, and just again over the years, running a bug bounty program is not easy. It's, it's, it's not easy. And so I don't want this to be an indictment or, you know, just an unfair critique. This is in the heart of self-assessment and holding up a mirror to help you improve. And, you know, they're, they're honest about where they're at. And I think that, you know, the nature of many large, especially organizations, there is a lot of teams that they have to interact with, like you said, and they can't always have the knowledge of every, you know, potential vulnerability, all the product, you know, that they might be you know, having as far as offerings go. And so, they do require consultations with these various engineering teams and, you know, product teams, and it does slow things down a little bit. But ideally, what you mentioned would be fantastic. If anybody on the security team working with bug bounty programs had at least enough knowledge to go, "Yes, that is definitely a problem. Let's go ahead and pay it, and we'll figure everything else on the backend." That would be ideal.

[00:58:46.36] - Justin Gardner
Yeah, for sure. 100%. All right. Walk me through these other 5 lowest scoring pieces across the bug bounty programs. We've got 2 from Operational Signal and 3 from Researcher Interface, starting with continuous improvement coming in at number 2 from Operational Signal.

[00:59:02.63] - Steve Hernandez
Yeah. Yeah. So continuous improvement, it's basically, you know, do you have a governance loop in place? Do you have a feedback loop where you're able to collect and actually action the feedback provided to you, right? Let's say, you know, you and 10 of your friends submit a report, and all of you kind of experience the same breakdown patternistically, you communicate that, what's done with it? Is it collected? Is it even looked at? That's kind of, you know, unfortunately, this is, this is saying that a lot of organizations, they're not able to, or they're not at the maturity yet, where they're able to do something with that and actually put it into their governance loop and make some changes.

[00:59:42.01] - Justin Gardner
Getting, getting feedback from the hacker.

[00:59:44.40] - Steve Hernandez
Exactly. Hmm.

[00:59:45.78] - Justin Gardner
Yeah, that, that should be, I mean, that's another plea I'm just trying to think about as well for, uh, BBMF, you know, what, what kind of ways you can reduce friction for, um, and provide tooling for organizations to increase their, their ratings across the, the score, you know, and, and increase their maturity. I think this is another great area where it's like, okay, we can provide infrastructure, you know, that is fully encrypted, you know, like you can just tack on to the end of your, um, report or responses or whatever, uh, that will invite the researcher to provide feedback. And then, you know, once a month we, we come in here, we provide you that feedback, you know, and, and we strategize, right? I think that could be a super valuable thing, um, for companies. Yeah.

[01:00:33.48] - Steve Hernandez
And actually action it, right? Not just collect it, but do something with it.

[01:00:37.21] - Justin Gardner
Yeah.

[01:00:37.38] - Steve Hernandez
That's, that's the important piece as well. Yeah. A lot of them collect it, but we don't always see that come full circle.

[01:00:42.38] - Justin Gardner
That's interesting. Uh, I, I just, I haven't seen a lot of like well-defined program feedback mechanisms.

[01:00:51.90] - Steve Hernandez
Mm.

[01:00:52.69] - Justin Gardner
Sometimes like in HackerOne or whatever, after you submit a report, you'll get a little popup that's like, did you like this? You know, like, yeah. yeah.

[01:01:00.30] - Steve Hernandez
yeah. yeah. Yeah.

[01:01:00.51] - Justin Gardner
And I just click it away, you know, real quick. But if I actually like had an actual person, I guess this goes back to that, that researcher interface, uh, you know, piece, right? Where an actual person from the, you know, not a bot account, an actual, um, analyst from the company says, hey, if you got any feedback, freeform box here.

[01:01:21.76] - Steve Hernandez
Yeah. Yeah.

[01:01:22.75] - Justin Gardner
You know, um, I think I would take that.

[01:01:25.69] - Steve Hernandez
Oh man. Well, I'm, let me just say I'm, I'm sad to hear you haven't seen that.

[01:01:29.63] - Justin Gardner
Yeah.

[01:01:29.71] - Steve Hernandez
Or maybe, or maybe not enough for you to remember.

[01:01:31.88] - Justin Gardner
Yeah.

[01:01:32.13] - Steve Hernandez
Um, because when I was, you know, on the program side, I, One of the, one of the tools and toolbox that I use often was Google Surveys.

[01:01:40.46] - Justin Gardner
Yeah.

[01:01:40.96] - Steve Hernandez
Hey, send this out once a month to your top hackers, ask them for feedback, honest feedback, and then do something with it. Look for themes and patterns. Um, so I'm disappointed to hear that you haven't, you know, initially experienced that.

[01:01:52.07] - Justin Gardner
Yeah. Well, I, I'll say don't send stuff to our @WeAreHackerOne or our @BugCrowdNinja addresses. We never read that shit. There's like, 20,000 emails in there.

[01:02:04.53] - Steve Hernandez
Oh no.

[01:02:05.44] - Justin Gardner
Uh, like, like, because we use that to sign up for all of these things and all of these things automatically put you in their mailing list. Right. So my @wearehackerone and @bugcrowdninja, I've got them just going straight to junk in my, in my, uh, my email. So I know I miss stuff there, but it's a survival technique because like you literally can't, you know, go through all of those or else you just die.

[01:02:27.98] - Steve Hernandez
Um, well, I'm curious, have you put any thought into what the ideal mechanism looks like for that for you? For, for, for you to receive solicitation of, hey, please give me feedback on your experience with us.

[01:02:40.32] - Justin Gardner
Yeah. I mean, the, the programs have, or the platforms, you know, have access to our actual emails. Um, like shoot us an email, uh, you know, like use, use the email that I used to log in to HackerOne and Bugcrowd and it will get to me. If you use my alias, my hacker You know, alias, then it's gonna, it's gonna get totally, you know, gone.

[01:03:02.51] - Steve Hernandez
No, that's fair. Well, there you go. Platforms.

[01:03:04.53] - Justin Gardner
Yeah. Yeah. Yeah. So maybe I've just been missing it all, but, um, yeah, I mean, I can think of like 3, uh, programs off the top of my head that I've filled out, you know, surveys for to help them improve, but—

[01:03:16.01] - Steve Hernandez
Wow. Yeah. That's not many for how many years you've been doing this. No.

[01:03:19.21] - Justin Gardner
Uh, no, not indeed. Um, program ownership clarity. What is that? That's a part of, uh, Researcher Interface.

[01:03:26.90] - Steve Hernandez
Yeah, so I think a lot of times, this came down to kind of responsibility and accountability. There's not always clarity internally on who ultimately owns this decision and who, like, who this, where does the buck stop? And when that's lacking, that creates all those downstream problems we were just talking about, right? Those inconsistencies in severity classifications, inconsistencies in payouts and the amount of payouts, inconsistencies in response times. So not having clear program ownership causes all of those downstream effects. And so that's what this is in, in light of. Hmm.

[01:04:10.03] - Justin Gardner
Okay. That makes sense. Yeah. And once again, a score of sub-3, uh, across the mean 2.97.

[01:04:16.25] - Steve Hernandez
Yeah.

[01:04:16.48] - Justin Gardner
As we move into the lower 4 and 5, uh, we're seeing the mean go to 3.06. And 3.09, so we are into established here. Yes. Talk to me about reward alignment and payment predictability.

[01:04:32.59] - Steve Hernandez
Yeah, those are, those are interesting. So reward alignment has more to do with, from the program owner perspective, do you feel like your bounty table is aligned with the amount of work and effort that goes into submitting and getting validation for this bug? And so, from their mouths, directly, you can see that maybe half of the time, right, they feel like the reward aligns with the effort required to submit those vulnerabilities. So, I think that's good. I think that's really great self-awareness. And I think that also shows that a lot of programs aren't necessarily not on your side when you kind of dispute, like, hey, this, you know, I feel like this should be worth more. I think that, I think that a lot of programs might feel that way and just maybe can't say that.

[01:05:24.34] - Justin Gardner
You know, it could be the other way around too though, Steve. It could be them saying, yeah, we're paying them way too much for this crap. You know, like, you know, like you don't know.

[01:05:32.78] - Steve Hernandez
Maybe I'm being too positive. Yeah.

[01:05:34.98] - Justin Gardner
You gotta look at the data, man. How did you word that question? Did you say, are you feeling like you pay the researchers too little? You know? So that would be an interesting insight to see if you go back and crunch the data again.

[01:05:49.61] - Steve Hernandez
Yeah, for sure. For sure. Yeah, no, I—

[01:05:51.26] - Justin Gardner
payment predictability here.

[01:05:52.88] - Steve Hernandez
Yeah, payment predictability. Yeah. So that's going back to, are they consistent about their payment timelines, right? Very simple, very direct. Are you paying when you said you were going to pay? Yeah.

[01:06:06.40] - Justin Gardner
Yeah. Okay. That makes sense. Those are definitely areas that are important. Like we said, talking to the researcher and bounties, those are the things that we like, you know? Yeah. I mean, at the end of the day, we really like technical intrigue and like curiosity and that sort of thing too. But we can manufacture that for ourselves in a lot of scenarios. The things that we don't control are, are you, do we trust you? And are you going to give us money?

[01:06:32.00] - Steve Hernandez
You know?

[01:06:32.84] - Justin Gardner
Like, so yeah, very interesting. Amazing. Let's, as we get ready to close here, let's hop over to another piece of research you did, AI is Compressing the Bug Bounty Maturity Curve. Some really interesting data in here about how AI is sort of exacerbating the situation, right? You mentioned advanced and leading being, you know, one of the main factors of that is how you perform under volume. Volume.

[01:07:02.40] - Steve Hernandez
Yeah.

[01:07:02.88] - Justin Gardner
And we've seen a massive increase in volume and a massive decrease in, uh, you know, program efficacy. So how is AI affecting this bug bounty maturity posture overall?

[01:07:15.30] - Steve Hernandez
Yeah, so I, I think, you know, we briefly touched on it earlier in the pod, but I think that it's made discovery a lot cheaper. And because of that, um, you know, hackers, to your point, right, you're able to, with a hackbot, quickly submit several CRITs, several high vulnerabilities while you're having a podcast interview with me.

[01:07:37.44] - Justin Gardner
Exactly. Hey, don't call me out like that, Steve.

[01:07:41.69] - Steve Hernandez
You know, I love that. And I think so. I think, I think the cost of, you know, of, of, of the AI is that vulnerability discovery is cheaper. It's faster. It's expedited. And so because of that, volume has increased tremendously. But I think programs and platforms, I think, and I've seen them admit this as much on LinkedIn and other social media platforms, have struggled to figure out how to keep up with that volume. Because I think it was Daniel Stenberg of the CURL Project who, who made a couple of statements, right, just a few months apart, 2 or 3 months apart, right? Once it was like, hey, we're getting a ton of just AI slop. Then we have to get into defining slop. But nonetheless, in things that were not valuable, right, from a report perspective, to the CURL project to within, again, a few months, him saying, oh crap, I'm actually getting a ton of valuable signal from these AI-generated reports. And so, you know, I think, I think we got left a little behind on the program and operations perspective when it came to being prepared for, for this AI age. But I'm starting, you know, I have some hope. I am starting to see some changes, some intentionality behind design and kind of making some changes to triaging. So I do have some hope that things will continue to get better. But there's all, you know, there's always a learning curve. And I think going back to what you and I were talking about in the beginning, I think there has to be some grace that is extended and understanding that, hey, we're all trying to get adjusted to this new world of AI. At the end of the day, we're all in it together. And I think we need to be able to continue to be vocal about the issues, And then on the program side and platform side, they need to be open to receiving that and working through those things and trying to find solutions that work for everyone across the board.

[01:09:31.71] - Justin Gardner
Nice. Yeah, I think so, man. It's, it's, it's confusing. I'm, I, you know, you asked me right before the pod when we were talking beforehand, like, wow, is this really the world we're in? As I was talking about the bugs that my HackBot is finding right before this. Yeah. And I, I feel disconnected from it, dude.

[01:09:47.07] - Steve Hernandez
Yeah.

[01:09:47.21] - Justin Gardner
I, I had, I hate to admit this, I actually reported the same bug twice. Twice. Like I literally, like I had it in my queue and then my bot, like I, I marked it as like reported and then my bot found it again. The dupe checker didn't work or something. And then I've reported it again and I was like, oh, I'm such an idiot. Yeah. You know, like, like, but that's how disconnected I am with it right now. You know, I'm like, wow, uh, great bug. Let me submit it. And then I submit it.

[01:10:10.48] - Steve Hernandez
Yeah.

[01:10:10.89] - Justin Gardner
And then 2 days later, wow, great bug. Let me see. You know, like it's, and, and I, I try to be pretty intentional about this stuff. Yeah. Yeah, I don't know, man. It's— it is definitely confusing. It will definitely exacerbate, you know, the problem if there's a throughput issue. But yeah, it'll be exciting to see how, how the communities evolve. Yeah, I think we'll close there, man. Did you have any, any final thoughts you wanted to give on Bug Bounty Maturity Framework or the industry in general right now?

[01:10:41.77] - Steve Hernandez
Well, first off, thank you so much for having me again, Justin. Always an absolute joy and a pleasure to get to catch up with you. You know, I really value our friendship. And so thank you for that. I think, you know, when it comes to, you know, the Bug Bounty Maturity Framework, just a quick shout of encouragement to everybody. You know, if you're a hacker, encourage your favorite programs to go through, take the assessment. It's free. It takes 15 minutes and it's very intentionally written and very prescriptive and helpful. You know, if you're a program owner and you're trying to figure out why aren't you getting the the level of researcher, the pedigree of researcher that you want on your program, go to bugboundingmaturity.com, take the self-assessment. I promise you'll walk away with some great insights. And, you know, just be honest, be sincere about it. And again, I think you'll get some great, you know, actionable 90-day plans out of that. And then be on the lookout for verification. I'm working, you know, tired looking on that with, you know, with a couple of people, including, you know, Justin's team and a few others that I know from my time at HackerOne. The program and hacker side to make sure that they're transparent, that they're legible, understandable, but also very fair in assessing the actuality of a program so that it can provide both a good signal for the program and, you know, from an acquisition standpoint and also the researcher trust standpoint. So that's what I'll leave you with.

[01:12:03.68] - Justin Gardner
Thank you for that, Steve. Yeah, I will definitely be on the lookout for that. I'm excited for verification. Um, we'll, we'll see how it goes. It's going to be an exciting, uh, rollercoaster ride, uh, in the future. All right. As As

[01:12:15.23] - Steve Hernandez
As As always, like, awesome. Yeah. Thank Thank

[01:12:17.57] - Justin Gardner
Thank Thank you. And that's a wrap on this episode of Critical Thinking. Thanks so much for watching to the end, y'all. If you want more critical thinking content, uh, or if you want to support the show, head over to ctbb.show/discord. You can hop in the community. There's lots of great high-level hacking discussion happening there. There on top of masterclasses, hackalongs, exclusive content, and a full-time Hunter's Guild if you're a full-time Hunter. It's a great time, trust me. I'll see you there.