A few weeks ago we released a security scanner for
Gemfile.lock files called Hakiri Facets. It got some traction because the Ruby subreddit and Ruby Weekly picked it up. In two weeks the counter of scanned Gemfiles hit 1,500 scans and two weeks later got to 2,000 scans.
I was always very curious about the state of security for regular Ruby projects, but there was no good source of data that could be used for statistical analysis. It seemed like the Facets experiment could fill the gap, so I decided to export data from 2,033 Gemfile submissions, crunch some numbers, and practice Excel-fu.
All that considered, let’s dive into numbers and graphs!
The first metric I was wondering about is the distribution of gems in Gemfiles. How many gems does a common Ruby developer use in their projects? The numbers are somewhat expected: the average number of gems per Gemfile is 113.08 with the standard deviation of 52.19.
The next question I had was how many of those gems contain at least one vulnerability? The numbers are staggering! 1,333 Gemfiles, or 66% of the total, are affected! I definitely didn’t expect that two thirds of all projects would contain at least one publicly known vulnerability.
What’s the average number of vulnerabilities per Gemfile? The mean is 3.84 with the standard deviation of 5.39. That’s a lot of vulnerabilities out there in the open waiting to be exploited by the attacker.
It’s known that some vulnerabilities are more severe than others, so the fact that a project has half a dozen security issues might not mean much. What’s the percentage of projects that have severe and critical security issues? Now, that’s a real question!
First of all, how can we determine if a vulnerability is severe or critical? All confirmed CVE vulnerabilities have a security score that is assigned based on CVSS Guidelines. The maximum score is 10, so I consider everything above 5 severe or critical.
Certain types of vulnerabilities are easier to exploit than others. The Open Web Security Project (OWASP) provides a list of the top 10 most serious security risks. If you haven’t seen it yet, take a look—it’s a great read!
So, I narrowed down my new search to top 3 OWASP security risks (injection, authentication, and XSS) and vulnerabilities that are severe or critical (CVSS score greater than 5). After putting the numbers together, the picture is still pretty frightening. 268 Gemfiles, or 13% of the total, had at least one serious, easily exploitable vulnerability. That’s every 10th project, guys.
Let’s look at some other statistics that are not directly related to scanned Gemfiles but are still relevant.
What gems are the most vulnerable? The winner is certainly very predictable: Rails has 53 vulnerabilities. Rack and Spree follow up with 6 and 5 vulnerabilities accordingly.
How often do vulnerabilities come out on a monthly basis? It’s a fairly common event that happens almost every month with some spikes every now and then. Developers and devops should really keep an eye on official vulnerability trackers or use third-party monitors. Here are stats for the past 2 years.
How do vulnerability disclosures for gems compare historically on a year to year basis? The general trend is that every year, more and more vulnerabilities are disclosed, with 2013 being a real outlier at 50 vulnerabilities! Here is a graph for the past 6 years—2014 is not over yet! ;)
I hope you found this analysis interesting and motivational. Vulnerabilities and, consequently, security breaches are closer than most people think. It’s hard to keep projects up to date but it’s something that engineers and business people should be aware of and factor into their development sprints and corporate strategy.
The good thing is that the community is becoming much more mature about reporting and addressing vulnerabilities, which leads to much higher quality code and general awareness.
Thank you and stay secure!
P.S. Don’t evaluate your project solely on the amount of public vulnerabilities. Truth be told, it’s very hard to evaluate security for any software project because there are many more factors affecting evaluations than simple correlations like “project A has a critical vulnerability B, hence its security is compromised” or “project C has a new vulnerability that was disclosed yesterday, so it’s screwed.” Engineers can patch vulnerabilities internally, preserving the same “vulnerable” version stamp or their application might not have any input fields for clients to mess with XSS.