Software Security Gurus Episode #21: Brian Levine

Matias Madou:
Welcome to today's Software Security Gurus webcast. With me today, Brian Levine. Welcome, Brian.

Brian Levine:
Thank you, Matias. Glad to be here.

Matias Madou:
Thank you. No, thanks for accepting. Hey, Brian, do you mind sharing a few words about yourself?

Brian Levine:
Sure, be happy to. Yeah, thanks for inviting me on. Just a few words about myself. Currently I'm leading product security and cloud security for practices at Axway. My responsibilities cover the full breadth of secure software, development life cycle, SDLC. Also, I have responsibility for cloud security operations and cloud security architecture. For those that may have not ever heard of Axway, we're an enterprise software provider, and I manage the security. But we're an enterprise software provider to government agencies, large enterprises, banks, financial institutions. We aim at the Fortune 500. We've been recognized by Gartner and other outlets for API gateway cloud services. That's just kind of a little bit aside about Axway. My background, how I got into security, so now in my younger days, I kind of approached things from a hacker mindset.

Brian Levine:
I'd say that's kind of how I got into security, from that hacker's perspective. So at a younger age, you know, I was playing with a lot of different emerging technologies. You know, that my in my youth don't want to date myself too much, but you know, when modems first started coming on the market and we had a lot of fun there doing things with war dialing and finding Telnet systems and logging in and seeing what we could do and, and had a lot of fun with that. And the early days of the internet and PCR, really just kind of seeing what happens if I crash this game or circumvent the, the copy protection controls, privacy controls limits on this cool game. I want to flare. And yeah. Then on the professional side I've worn a lot of hats, you know, the customer facing roles on the technical sales engineering that led me to a security product management role. And that led me into the more full time post SAC AppSec or court roles that I'm in right now. And yeah, just love to have the opportunity to connect and learn from other people, thrown by the challenges and learning something new every single day. And just love to meet up with others and share experiences, compare notes, and seeing what we're seeing that it works and seeing what doesn't work.

Matias Madou:
Absolutely thank you very much for a lovely introduction. And I think we all have similar backgrounds where we started off, you know, breaking things, hacking things, experimenting with things. And then ultimately we decide like, gee, can we make it better? And that's how we roll into what we're doing today. I think to make the world a better place. And to actually what sparked this conversation is a talk that you gave at OWASP Global AppSec, 2020, and the title of that talk was a Warrior's journey, building a global AppSec program. I was very fascinated by, by that, but our presentation, because the thinking is very much aligned with how we would approach security and how we would roll out security. So I would love to have a chat around the three topics that you described in that particular talk, which is around culture process and governance. So I would say, let's start, let's start with culture. Security is everybody's job and referred to in BSIMM as the software security group. You refer to openSAMM, the software security center of excellence. You refer to the product security group, but how do you get started with something?

Brian Levine:
Yeah, that's great. I'm glad you enjoyed the talk. So we've all heard that phrase public security is everyone's job and you know, those of us who, it's our role and we put that slogan maybe in front of the organization say, Hey, security is everyone's really, how do you make that reality? Because not everyone can do security every day, another job. So how do we get a security culture going so that people know what their security role is and that they are executing their responsibility around security, as well as what their primary jobs may be a developer, operations manager, product manager, engineering manager. So, there's a lot of interest out there and you know, when you starting off, it's like, okay, where do I start? You know, there's openSAMM there's BSIMM and there's Microsoft, PCL there's the CSF. So it's really just kind of so many different frameworks, but what's interesting.

Brian Levine:
And what I found was kind of taking the commonalities of these frameworks and also just kind of, you know, what's kind of worked out for us and our personal experience. So I think, you know, the first thing is you have to build that core software security group and, you know, without this central organization, to at least set some ground rules and steps, some objectives, and kind of set the charter, you're not going to get very far. So defining that team and defining your software security group, your central team, and how do you do that? Well, the first thing you got to get executive sponsorship, so you really want to make work. And this depends on what level of the organization you're working in. Obviously you need your direct managers approval and sponsorship, but it's highest level of the organization and the highest level of the business units that you're going to be working with.

Brian Levine:
So if I'm working with operations and I need the operations executive responsible, the operations manager, responsible, responsible, agreeing with this program and agreeing to our charter as well, you can get the CEO sponsorship and the CEO awareness and the CEO to write a nice letter or email about your program. That's what you need. And so how do you do that? Well, you know, this could start with a semiformal presentation, a proposal, some socialization, we sought some one-on-ones with a different line of business owners. And then once you have the kind of socialized alignment, then you're ready to launch and kind of formally announce the program. And so they really want to make it formal. We're going to have a charter that's published. It could be in a policy. It could be, you know, part of your broader, we might have an ISO governance policy.

Brian Levine:
You can include it into your ISO information, security policies. But the important thing is that it's published by and everyone, all the stakeholders agree to it. This is what we're going to do. And then part of that, that the breakdown, the responsibility. So everyone's kind of favorite word, the RACI matrix, right? What are the responsibilities and roles within the policy or within the charter, right? Who's, who's going to do what, what's the software security group responsible for, what are the engineers responsible for? What are your lines of business responsible for? And because it's a shared responsibility model. So having that RACI as part of it is key to getting that sponsorship that buy-in and the establishment of your charter.

Brian Levine:
One other interesting data point when you're starting out, I would say is the size and scale and the scope. And so you have to take a look at your organization and say, okay, well, how many developers do we have in our company? If you look at their base and they have some really interesting data, and some other groups studies that looked at the data and said about 1% to 3% of your engineering or your operations should be security, full time security engineers. So, you know, if you do the math, it's about 3% range, you want at least one full-time application security engineer to about 20 to 30 developers or 20 to 30 people that you're serving.

Matias Madou:
That's a good ratio. If you can get that, that's a good ratio. You're doing pretty good. There's not a lot of people in that position.

Brian Levine:
It depends on the scale of the organization. If you are a hundred developers, okay, maybe you can get by with a lower end of the ratio, but if you got 3000 developers in your organization, you're probably going to be on the higher end of that ratio. So yeah, definitely it depends kind of statistics.

Matias Madou:
Absolutely. So, first of all, I couldn't agree more. I think it's top to bottom. If you have executive sponsorship, that's the only way to get started with the program like that and build a, a really good core and then break it out and make sure you have some champions in, in these groups. How do you find these champions? Like how do you bridge from going to a central piece to something that the entire organization rallies around it? Because if it stays central, it's not going to work either.

Brian Levine:
Yeah, absolutely. And again, that's about the scale of the team and the scale of the organization, because at some point, you can't have your core security team, that three application security experts and 50 different R and D teams. I'm not going to be in every single scrum. I'm not going to be able to be in every single meeting. And they might, my team, your core security team won't know the intimate, intricate details of each engineering project in each environment. And so that's really where the security champions fit in and why you need the security champions program.

Brian Levine:
So when you're selecting security champions, it's best to have your team members that are volunteering versus "voluntold," that you want to identify the individuals who either they're coming to you directly and say, Hey, I want to join your team, how do I do that? Or you can say, well, you don't have to join my team to do security. Here's how, and here's what make you a security champion as well. They may have already demonstrated expertise within their products and within their development project to say, maybe they brought in a security tool. Maybe they identified a number of security issues, security risks that the rest of the team hadn't considered. You know, those are your good candidates to be champions as well. If you have a training program, you know, the folks that are always, you know, first to complete the training, gaining of the highest scores in the training, you're asking what's next. Those are again, great candidates to align with and bring into the champions program.

Matias Madou:
Absolutely. And I personally, I think it takes more than only having technical skills for the security champions. It starts with having technical skills. If that's not there, you know, it's not going to work, but if they have the technical skills, they also have to have a little bit of social skills and be able to persuade other people and help other people in moving that forward.

Brian Levine:
Yeah. I could definitely agree with that point. And, and yeah, I think that may even be more important that you have to have that social skills to be able to persuade and to show up and, and stop being disruptive and collaborate with the rest of the team, because obviously everyone has competing priorities and selling security incidents. Definitely a social skill.

Matias Madou:
Yeah. Let's, let's switch topics. Let's go to process. We know we have the, the Microsoft SS DLC. We have the touch points. There's plenty of stuff out there. Every company has its own version, which is just good because every, every company is unique, but they quite often do a little bit of the same things. And, and one thing that you described is, Hey, you have to identify gates, you have to collect results, do not enforce them. Can you give a little bit more color on how you approach these gates? What are good gates? A little bit more around the SDLC and how to weave insecurity.

Brian Levine:
Yeah. And that's a great point to start with too, is that if you're just starting out or just implementing a program, you don't want to show up and say, okay, you found some security tests. So therefore the project is blocked written. You need that. The first thing you do is just start observing, start measuring, start testing, and you're the result, but you don't really enforce a hard gate. We don't enforce a block if you're just starting out, because when you start using security tools, you know, you're going to get a lot of your data that no one's seen before. You might have a lot of false positives. So there's a lot of tuning that goes on there. And so that's, that's definitely a key is that whatever the gate is, you start out observing and start out and measuring and getting again, building that awareness that we will say, Oh, okay, now I understand what this is doing and what we need to be knowing about.

Brian Levine:
But an example of a gate, and you know, when you're talking about the SDLC and the process of simple example would be you have an ISR security review, initial security review meeting with a team, and you have maybe a checklist there and you say, okay, one checklist item might be, does your product have an API? If it's yes, that's okay. Yes. Your product has an API. Okay. Now we have a gate that says, north, your API must be scanned using a bask tool, using a security scanner. And the result must show that there's no critical highs or mediums before you release your products. And so that would be just one example of a security gate.

Matias Madou:
Sure. And so, how do you do that continuously? That's another question like a lot of people say, you know what I mean, when you think DevOps and everything goes continuously and we do it quite often, how do you approach something like that? Because the models that we've just described, you know, the digital touchpoints or the, the Microsoft SDLC, they're fairly old and everything that's 10 plus years old is fairly old in our industry. So how do we do that in the new world?

Brian Levine:
Yeah, exactly. And, and kind of up to the, the starting point as well, in terms of leveling up and building that organization and building that team. So to have security at scale, to have continuous security and, you know, working within the development organization and the development culture, the number one thing is you want to make the developer's job as easy as possible, as frictionless as possible. And so when you're scaling up, really, you have to maintain that customer focus and remember who is your customer? It's, it's the developer, it's the person that is responsible to do security, but it's not their full-time job. And so that's your customer and how do you make their job easier? And so they're really the culture and the process is okay, it's I allow you to do it. Self-service it's not, I do it for you, but I'm going to give you the automation.

Brian Levine:
We can give you the tools. I'm going to give you the training to integrate it right into your pipeline, to integrate it right into your build. And so I have what we do for our CI/CD and continuous security. We have pipelines defined and automation defined this. Okay. We like the example I gave. Okay, you have to run a desk scan. Okay. Well, I have a hook that I can put into my Jenkins pipeline or whatever I'm using. As my build automation, I have a hook that will trigger that scan. And then I have an API feed the results of that scan back into their dashboard, or directly back into Jenkins to say, okay, did a password failed in the past? Then we just move forward in the building, moves forward. If it failed, then okay, now we can block the build and somebody has to do something.

Brian Levine:
But the great thing about that approach is that it doesn't require going to a different tool, logging into a different dashboard, looking at some other third party thing. It's putting the data right there in the pipeline where the developer works every day in their common set of tools with that automation. And so really, like you said, it's dev ops and DevOps, continuous security scout, where we're building multiple times a day, we're releasing multiple times a day. We don't have this time to do the classic sort of waterfall, you know, SDL, you know, if you have four weeks, five weeks or months for a software project. Great, okay. You can do that sort of linear SDL, but you know, like you said, in the DevOps continuous world, that doesn't adopt itself too well. And so, you know, when you have these, these automation and frameworks, you know, that's what enables the continuous security. And that can talk a lot more and give some more examples about that, but that's sort of a high level intro of how we do it.

Matias Madou:
No, no, absolutely. I couldn't agree more. I think we have to do it in a continuous fashion and you have to interact over there. And essentially the developer already has an environment. He has a computer with intelligence and he's working with particular systems. It's not a good idea to, to add more systems to it, you know, for security reasons, let's just go there where he is an integrator solutions, where he is and make it as frictionless as possible.

Brian Levine:
I will add that one of the critical hurdles that people have to get over when they talk about, because when people say, Oh, you're going to run the security scan in the build, they get scared because they want their builds to run in 30 seconds. You know, one minute, two minutes. If you tell them how you're going to run this security tool and it's going to take an hour, okay, that doesn't work for us because we need them. So the hurdle that people have to get over is some of your, your security tooling can be automated in line and it will run quickly. And it can be run in line with your bill, other jobs that may take hours, days, what have you, those can be run out of band, but you still can have an API that gives the results to that job every time that job runs.

Brian Levine:
So I ran the scan yesterday out of bed. I can still pull and find out, okay, did it pass or fail? And every 20 jobs I run today, as long as it passed last night, then my jobs today still pass. And so that's kind of one of the nuances that I think some people get hung up on when they're trying to do the security scanning between directly in the build and, and maintain that speed and cycle time that the developers want to have.

Matias Madou:
Yeah. I love that, that I, some people would refer to that as, as the coffee test, you know, you need to cram as many tests as possible in the time you can actually go for a coffee and come back. And if it doesn't fit into that coffee test, you have to actually externalize it or paralyze it or get it exactly right. So love that. Love that example. Last topic that you talked about was governance, KPIs, and dashboard. And I have to be honest, I've, I've seen multiple KPIs dashboards. They all kind of look the same, but again, they're all different as well. Nobody's, by the way, using one single tool for that. Everybody built its own dashboard. So we'd love to hear your take on KPIs and dashboards and especially around what do you think are good ones? What are bad ones? What are your opinions.

Brian Levine:
Yeah. I can understand too what you mean too. And especially when you look at every vendor that comes in and, you know, they have a tool and Hey, we have a dashboard that we do correlations, et cetera. So it, it's a really interesting subject. And I haven't found a good sort of neutral take from like a PC or a Alaska, Sam or others that say, okay, these are the metrics to use. I think there's a lot of thought out there from other organizations, but one that I've found that is very interesting. It's really high fidelity is the number of customer or third party reported issues. And we don't have product or in a service. And I like it because it's, it's very, it's very high fidelity, it's hard.

Brian Levine:
It's like, here's somebody that has a problem with the product. They reported it to you and you acknowledged it was an issue and you fixed it. So, you know, it kind of cuts through a lot and hearing false positive. That's not an issue. And so the tooling, you know, they can raise those issues with that because it's a single kind of KPI that we can look at, quantify, and it cuts through a lot of the noise and it's high fidelity. And so obviously we want to see that trend always going in the downward direction and we want to be minimizing and finding and fixing things before they release. And before they launch before a third-party pin tester gets a hold of it and reports something. And so that one, I found to be a useful for us internally, when we're doing those tests, really what I'm looking at is more of a risk aggregation.

Brian Levine:
So it's easy to run one to one scan and get a result from that one tool or one scan. But what is the aggregation of all of the tools that you're using and then look at, of course the classic, critical number of mediums overnight, et cetera, but also really we want to look at what's the time horizon on that. What does the real Delta look like over time? Obviously we want to see security debt going down over time and that's the metric and the KPI we'll look at it over time. Is it increasing or decreasing? And that's something we share we surface and, you know, back to the executive sponsorship and back to the security culture for putting that somewhere, that's visible that teams can see how they stack up or how's their product performing versus their peers. So those are definitely human motivators. It creates a little friendly, friendly competition as well within the groups.

Matias Madou:
Yeah. Security. One of the few departments in an organization where managers like the graph to go down. Normally they always like it to see, go up, but in terms of risk and vulnerabilities, they like it to go down. Brian, maybe a last question. If my sources are correct, you are a jujitsu champion. If my sources are correct, what belt are you? And does it rank higher or lower than your AppSec belt?

Brian Levine:
Oh, great trick question. So yeah, I practice Brazilian jujitsu. I'm a Brown belt today, not quite gotten to that black belt level yet. The nice thing about jujitsu is it's, it's definitely a journey that it takes some time to get to that black-belt level, but hopefully it's still a goal of mine to get there. And so I'm going to have to come up with a higher belt in application security, that black belt rank, I think maybe hopefully a few stripes on that belt, but I wouldn't want to face a Brown belt, my peer Brown belt in a competition. I'd rather fight in jujitsu than fight in application.

Matias Madou:
Very nice. Very nice. Keep up the good work, Brian. Thank you very, very much for accepting to be the 21st guru on the software security gurus webcast. It was a fantastic chat. Thank you very much.

Brian Levine:
Yeah. Thank you so much Matias. I appreciate it as well. Always happy to connect and share notes. If folks would like to discuss more. I'm always keen to have a chat.

Matias Madou:
Sounds good. Thank you very much.