Cracking NTLM Hashes on Google Cloud's Nvidia Tesla T4 GPU
For my coursework for CMP506 Computer Security at Abertay University, I had to carry out a penetration test on a small company network. Besides basic enumeration and vulnerability scanning, we also had to apply various system hacking techniques, including password cracking of NTLM hashes. While in class, we’d primarily used Cain with dictionaries and rainbow tables (alphanumeric up 7 digits) for cracking, I also wanted to recover even more passwords by brute-forcing a larger candidate space. This post will talk about my experience of cracking the hashes on the Google Cloud both using John and hashcat. I will share how to set up hashcat on the Google Cloud.
First Failed Attempt: John the Ripper 🧛♂️
At first, I’d tried to crack the hashes using John, as it’s my go-to application for cracking and brute-forcing. While it was running faster on the Google Cloud than inside my VMware environment (no surprise!), I was still a bit disappointed by its overall speed. When I checked the resource monitor nvidia-smi, it became obvious that only six per cent of my rented GPU was utilized, while the CPU was running at full load.
After a little bit of research I found out that John creates the candidate passwords by the CPU, which makes the processor the actual bottleneck. So I could either scale up my CPU performance drastically or change to a GPU optimized cracker like hashcat. In light of my tight budget for this project, I went for the latter.
Second Successful Attempt: Hashcat 🐱
As a second attempt, I’d tried to use hashcat to crack my NTLM hashes, which I’d previously dumped using Meterpreter’s hashdump. With an average performance of around 16000 MH/s the speed was really good and allowed me to brute-force all passwords up to ten digits within seven days. With an average GPU utilization of 99 percent, I was also finally getting my money’s worth!
The overall performance could possibly even be improved by selecting a more potent setup and optimizing the driver and application. However, as the password cracking was only part of larger coursework, I didn’t want to invest more time and money.
Setting up Hashcat on Google Cloud Platform
I started by configuring the instance through the web portal. I went for the NVIDIA Quadro virtual Workstation - Ubuntu 18.04 with 8 vCPUs, 30 GB of RAM and a single NVIDIA Tesla T4 for a monthly cost of slightly above $500. Please not that GPUs are billed by the minute and that you can not use them in conjunction with cheap Preemptible VM instances.
Basic Checks
Once you are connected to the instance via SSH or through the browser shell, you can run the nvidia-smi tool, which should give you a nicely formatted overview of the GPU, its driver version and the utilization. By default, the GPU should idle.
| |
Which gave me this output:
| |
Tuning the GPU
At first, we enable the persistence mode of the GPU to keep the GPU initialized even when no client connects and therefore reduce the startup time. It’s also necessary to have persistence mode enabled to be able to change clock speeds of the GPU and the memory, which we’ll do right afterwards.
| |
Next, I could optimize application clocks. At first, I’d queried the available application clocks by running this simple query:
| |
Which gave me a lengthy list, which I truncated for ease of reading. Basically, I could choose between two memory speeds in combination with various GPU clock speeds.
| |
Please note that the supported graphics clock rates are tied to a specific memory clock rate so when setting application clocks you must set both the memory clock and the graphics clock. In the above example, I could either go for the combination 5001,1590 or 405,645. Obviously, I went for the former.
| |
This should give you the following output:
| |
In case you want to reset the clock speeds to their defaults, simply run a sudo nvidia-smi -rac -i 0 command.
Installing Hashcat
We can download the latest precompiled binaries of Hashcat from their website using curl. When I’d tried this out, this was 5.1.0.
| |
Once downloaded and verified, we got a legit copy, it’s time to unpack the Hashcat binaries using 7-Zip.
| |
The output should look like this:
| |
Let’s crack some hashes!
For my coursework I had a list of 52 hashes in the following format, which were stored in a file named ntlm.txt:
| |
To start Hashcat, simply change into hashcat-5.1.0 directory and run the executable like this:
| |
Using parameters I’d set the type of the attack to brute-force attack (mode 3) and the hash mode to 1000 (NTLM). Additionally, I wanted to have my recovered passwords written to a separate text file, which I named ’ntlm_results.txt'.
After running Hashcat for a little while, I found the first passwords in my ntlm_results.txt:
| |
Optimising Hashcat
When running with the default parameters, I averaged around 8600 MH/s which is good but not great. By appending -O (optimized kernel mode) and -w 3 (reduce writes to the console), I could drastically increase the speed to around 17000 MH/s.
| |
Conclusion 🎇
This was a really fun little project, as it allowed me both to play around with Hashcat as well as use the Google Cloud Platform to solve my coursework. Even though I couldn’t recover all 52 passwords, I was at least able to find all passwords with up to 10 digits in a modest time. I hope this post is also useful for your own password cracking attempt.