Welcome to the “Python for Hackers” blog series! In this edition, we’ll create a powerful tool called “Directory Buster” using Asynchronous Programming. It’ll help us find hidden web resources swiftly and efficiently. In this blog, I will explain most of the necessary details required to build this.
🐋💣 What is directory Busting
Directory busting, also known as directory enumeration or directory brute-forcing, is a technique used in cybersecurity to discover hidden or obscure directories or files on a web server.
🐋Where Directory Busting helps?
- Discovery of hidden content: Directory busting helps reveal hidden or obscure directories and files that may not be easily accessible or linked from the main web pages.
- Identification of forgotten or backup files: Sometimes, developers or system administrators may forget to remove backup files or directories containing older versions of web pages.
- Vulnerability assessment: By systematically scanning a website’s directory structure, directory busting can help identify misconfigurations, improper access controls, or other security weaknesses.
🛡 Building The tool
🗡Before building the tool itself let me give a brief of what you are going to build.
🛠 Importing all the necessary modules in our file
import argparse
from bs4 import BeautifulSoup
from sys import exit
from termcolor import colored
from os import path
import asyncio
import aiohttp
These are the modules, and from these modules, we will use specific functions.
⚒ Taking command-line arguments
You may have seen various tools which take input from the command line itself, which provides flexibility, enabling users to tailor the tool’s functionality to their specific needs without modifying the source code. Using command-line arguments helps us in attaining various things such as
- Reproducibility: By specifying the exact options and parameters used during a particular run, the same results can be obtained consistently in subsequent executions
- Automation and scripting: Command-line tools are ideal for automation and scripting purposes. They can be easily incorporated into scripts or batch files, allowing repetitive tasks to be automated.
- Documentation and discovery: Command-line tools typically come with built-in help and usage information. By invoking the tool with specific command-line arguments like
--help or -h
Using argparse to define command-line arguments
In Python, Argparse provides a convenient way to parse command-line arguments. It simplifies the process of defining, organizing, and handling command-line options and arguments in Python scripts.
Before using it, you will have to install it using the below command
pip install argparse
Basic Example
import argparse
parser=argparse.ArgumentParser()
parser.add_argument('--name',type=str)
arguments=parser.parse_args()
print(f"HI! {arguments.name}")
Let’s understand this code:
import argparse: This line imports the argparse module
parser = argparse.ArgumentParser(): Here, an instance of the ArgumentParser class is created. This object is responsible for defining and parsing the command-line arguments.
parser.add_argument('--name', type=str): This line adds a command-line argument to the parser. The argument is specified with the --name flag, indicating that it is an optional argument. The type=str argument specifies that the argument’s value should be interpreted as a string. You can add various optional arguments like the required which tells that the argument is essential for the tool to run or not.
arguments = parser.parse_args(): This line triggers the parsing of the command-line arguments. The parse_args() method examines the command line, extracts the provided arguments, and stores them in the arguments object.
print(f"HI! {arguments.name}"): Finally, the value of the name argument is accessed using arguments.name and printed with a greeting message.
As you can see we are able to the command-line argument that we created.
Argparse, by default also creates a documentation of tool which can be accessed through -h or --help.
To get detailed info on the argparse you can check out : Link
Creating get_args() function
So now we know the basic overview of the argparse, so now let’s start creating what we will need in our tool.
def get_args():
parser = argparse.ArgumentParser()
parser.add_argument('-u', '--url', dest="target",
help="The target URL to attack (provide it with http or https)")
parser.add_argument('-x', dest='extensions', nargs="*",
help="List of extensions to check (e.g., php asp)")
parser.add_argument('-r', '--follow-redirect', dest="follow_redirect", action='store_true',
help="Follow redirects")
parser.add_argument('-H', '--headers', dest="header", nargs='*',
help="Specify HTTP headers (e.g., -H 'Header1: val1' -H 'Header2: val2')")
parser.add_argument('-a', '--useragent', metavar='string', dest="user_agent",
default="directorybuster/1.0", help="Set the User-Agent string (default 'directorybuster/1.0')")
parser.add_argument('-ht', '--hide-title', dest="hide_title", action='store_true',
help="Hide response title in output")
parser.add_argument('-mc', dest='match_codes', nargs='*',
help="Include status codes to match, separated by space (e.g., -mc 200 404)")
parser.add_argument('-ms', dest='match_size', nargs='*',
help="Match response size, separated by space")
parser.add_argument('-fc', dest="filter_codes", nargs='*', default=["404"],
help="Filter status codes, separated by space")
parser.add_argument('-fs', nargs='*', dest='filter_size',
help="Filter response size, separated by space")
parser.add_argument('-w', '--wordlist', dest='wordlist',
help="Path to the wordlist file to use", required=True)
parser.add_argument('-o', '--output', dest='output',
help="Path to the output file to save results")
try:
return parser.parse_args()
except argparse.ArgumentError:
parser.print_help()
exit(1)
Let me first explain the various arguments which are being used while defining the command-line arguments.
- dest: The
dest parameter specifies the name of the attribute where the argument value will be stored. This means if dest=“target” then you can retrieve the value using arguments.target.
- Nargs : The
nargs parameter in argparse is used to specify the number of command-line arguments that should be consumed for a particular option or argument. Nargs argument takes these values as input.
None (default): Consumes a single value for the option or argument.'?': Consumes zero or one value. If the option is not provided, a default value can be specified.'*': Consumes zero or more values. Allows providing multiple values separated by spaces. '+': Consumes one or more values. Requires at least one value to be provided.
While using nargs, we are returned with a list
- Required: Using this argument, you can specify if the argument is required or not. It takes only two values True or False.
- help: In the help argument, you can specify the message to be displayed.
Let’s explain the code then:
parser = argparse.ArgumentParser(): Creates an instance of ArgumentParser which will be used to define and parse command-line arguments.
parser.add_argument('-u', '--url', dest="target", help="The target URL to attack (provide it with http or https)"): Defines an argument -u or --url that expects a value. The value will be stored in the target attribute.
parser.add_argument('-x', dest='extensions', nargs="*", help="List of extensions to check (e.g., php asp)"): Defines an argument -x that accepts zero or more values. The values will be stored in the extensions attribute.
parser.add_argument('-r', '--follow-redirect', dest="follow_redirect", action='store_true', help="Follow redirects"): Defines an argument -r or --follow-redirect that, when provided, sets the follow_redirect attribute to True. It does not expect a value.
parser.add_argument('-H', '--headers', dest="header", nargs='*', help="Specify HTTP headers (e.g., -H 'Header1: val1' -H 'Header2: val2')"): Defines an argument -H or --headers that accepts zero or more values. The values will be stored in the header attribute.
parser.add_argument('-a', '--useragent', metavar='string', dest="user_agent", default="directorybuster/1.0", help="Set the User-Agent string (default 'directorybuster/1.0')"): Defines an argument -a or --useragent that expects a value. The value will be stored in the user_agent attribute. It has a default value of "directorybuster/1.0".
parser.add_argument('-ht', '--hide-title', dest="hide_title", action='store_true', help="Hide response title in output"): Defines an argument -ht or --hide-title that, when provided, sets the hide_title attribute to True. It does not expect a value.
parser.add_argument('-mc', dest='match_codes', nargs='*', help="Include status codes to match, separated by space (e.g., -mc 200 404)"): Defines an argument -mc that accepts zero or more values. The values will be stored in the match_codes attribute.
parser.add_argument('-ms', dest='match_size', nargs='*', help="Match response size, separated by space"): Defines an argument -ms that accepts zero or more values. The values will be stored in the match_size attribute.
parser.add_argument('-fc', dest="filter_codes", nargs='*', default=["404"], help="Filter status codes, separated by space"): Defines an argument -fc that accepts zero or more values. The values will be stored in the filter_codes attribute. It has a default value of ["404"].
parser.add_argument('-fs', nargs='*', dest='filter_size', help="Filter response size, separated by space"): Defines an argument -fs that accepts zero or more values. The values will be stored in the filter_size attribute.
parser.add_argument('-w', '--wordlist', dest='wordlist', help="Path to the wordlist file to use", required=True): Defines an argument -w or --wordlist that expects a value. The value will be stored in the wordlist attribute. It is marked as required.
parser.add_argument('-o', '--output', dest='output', help="Path to the output file to save results"): Defines an argument -o or --output that expects a value. The value will be stored in the output attribute.
return parser.parse_args(): Parses the command-line arguments and returns the resulting namespace object containing the attribute-value pairs.
except argparse.ArgumentError: parser.print_help(); exit(1): If there is an error during argument parsing, the help message is printed, and the script exits with an exit code of 1.
Getting colored outputs using termcolor
You have seen several tools which generate output in color format which enhances the tool or script by providing it enhanced readability and error identification.
In Python, you can use the termcolor module for colors in your output. The best thing about this, it is platform-independent so it works on almost all the platforms.
Installation
pip install termcolor
Basic usage
from termcolor import colored, cprint
# using print
print(colored("Hello Guys", "red"))
# using cprint
cprint("Hello Guys", "green")
There are two functions in the termcolor library that helps in colors to the output: colored and cprint.
In the first example, print(colored("Hello Guys", "red")), the colored function is called with two arguments: the text string "Hello Guys" and the color "red". This function returns the text string wrapped in special ANSI escape codes that represent the specified color. The print function then outputs the colored text to the console.
In the second example, cprint("Hello Guys", "green"), the cprint function is used directly to print colored text. It takes the same arguments as the print function, with the text string and the desired color.
To see the various types of color it supports, you can visit : Link
🔥 Asynchronous Programming
Asynchronous programming is a programming paradigm that allows tasks to be executed concurrently and independently, without waiting for each other to complete. In simpler words, it enables a program to perform multiple operations simultaneously, making it more efficient and responsive.
In traditional synchronous programming, tasks are executed one after another, and each task must complete before the next one can start. This can lead to delays and inefficiencies, especially when dealing with tasks that involve waiting, such as network requests or file operations. Asynchronous programming, on the other hand, allows the program to continue executing other tasks while waiting for certain operations to complete.
🛠 How is Asynchronous Programming different from multiprocessing, multithreading
Although multiprocessing, multithreading, and Asynchronous programming are all used to deliver concurrent execution in a program, they are different in the ways these technologies work.
Multiprocessing involves creating multiple processes. Each process has its own memory space and resources, providing true parallel execution. Multiprocessing is suitable for CPU-bound tasks that can benefit from utilizing multiple processor cores effectively. Multiprocessing is suitable for CPU-bound tasks that can benefit from utilizing multiple processor cores effectively. This process takes the most resources.
Multithreading involves creating multiple threads within a single process. Each thread represents a separate flow of execution, allowing multiple tasks to run concurrently.
Threads share the same memory space and resources of the process they belong to. This can be useful for CPU-bound tasks where multiple threads can perform computations in parallel. This takes less resources compared to multiprocessing.
Asynchronous Programming involves using non-blocking operations and event-driven mechanisms to achieve parallelism. Instead of creating separate threads or processes, asynchronous programming focuses on efficient task handling and continues executing other tasks while waiting for certain operations to complete.
🔗 Example-based learning
Imagine you are a fruit shop owner who has just set up your shop where you sell fruits. After a month you see that your business has been expanding and there are several customers buying from your shop.
Now to control that rush you can do three things :
- Hire additional staff to assist you with various stocks. Each staff member would be responsible for a specific task, such as one person attending to customers’ inquiries, another person arranging the displays, and a third person handling the cash counter. They would work in parallel, allowing you to serve customers more effectively and maintain a well-organized shop. After your work is done, you can simply fire these staff. This is the simple example of Multithreading and in this additional staff is the threads
- Opening a new fruit shop: having multiple separate fruit shops, each with its own set of employees, resources, and responsibilities. In this one shop could be dedicated to handling customer assistance, another shop could focus on restocking, and a third shop could manage the cash counter. This is the example of multiprocessing
- Efficiently handling multiple tasks : Example: While a customer is selecting fruits, you can start restocking the shelves with fresh produce and also handle the payment of another customer at the cash counter. So you don’t need separate employees and separate shops for various tasks. This is an example of Asynchronous programming
🔥 In short Asynchronous programming uses the main thread and switches over the tasks even ever a task is taking time to execute.
🔥 Using Asynchronous Programming in Python
In Python, asynchronous programming is supported through the asyncio module, which provides tools and utilities for writing asynchronous code. Here’s a simple example to illustrate the basic concepts:
import asyncio
async def task1():
print("Task 1 started")
await asyncio.sleep(1) # Simulating some time-consuming operation
print("Task 1 completed")
async def task2():
print("Task 2 started")
await asyncio.sleep(2) # Simulating another time-consuming operation
print("Task 2 completed")
async def main():
# Creating tasks and scheduling them
t1 = asyncio.create_task(task1())
t2 = asyncio.create_task(task2())
# Allowing tasks to run concurrently
await asyncio.gather(t1, t2)
# Running the main function
asyncio.run(main())
Output:
Understanding Elements from code
Corroutine: A coroutine is a special type of function or routine that can be paused and resumed at specific points during its execution. In Python, coroutines are defined using the async keyword before the function declaration.
To define a coroutine, just add async keyword before the function declaration.
async def task():
statement1
statement2
Event Loop: An event loop keeps track of all the registered tasks and decides when and in what order they should be executed. It ensures that tasks progress without blocking each other and maximizes the utilization of system resources.
Await: The await keyword is used in Python within asynchronous functions or coroutines to pause the execution of a coroutine until a particular asynchronous operation is complete. It is typically used with functions or methods that return awaitable objects, such as asyncio.sleep() or asyncio.gather().
import asyncio
async def my_async_function():
print("Before await")
await asyncio.sleep(2) # Pause the coroutine for 2 seconds
print("After await")
asyncio.run(my_async_function())
asyncio.create_task(): asyncio.create_task() is a function provided by the asyncio module in Python. It is used to create a task that represents the execution of an asynchronous function or coroutine. The task can then be scheduled and run concurrently with other tasks using the event loop.
asyncio.gather(): asyncio.gather() allows you to execute multiple awaitable objects concurrently. It schedules the awaitables for execution and waits for all of them to complete. This concurrent execution can improve the overall performance of your code by utilizing available system resources efficiently.
asyncio.run(): asyncio.run() is a function provided by the asyncio module in Python. It is used to run an asynchronous function or coroutine and manage the event loop for you. It is simply used to call the coroutine function. You can’t call the async function so you need this function.
Flow execution
The code begins by importing the asyncio module, which provides tools for asynchronous programming.
Two asynchronous functions, task1() and task2(), are defined. These functions represent individual tasks that will be executed concurrently.
The main() function is defined as an asynchronous function. It serves as the entry point for running the asynchronous code.
Inside the main() function, the asyncio.gather() function is used to concurrently execute task1() and task2() and gather their results. The await keyword is used to wait for the completion of the gather() function, indicating that the program should pause until the tasks are finished.
The results of the tasks are stored in the results variable. In this case, since there are two tasks, the results variable will contain the results of both task1() and task2().
The individual results are accessed by unpacking the results variable into separate variables, result1 and result2. This allows you to access and process the specific results of each task.
Finally, the event loop is run using asyncio.run(main()). This executes the main() function, which triggers the execution of the tasks specified in asyncio.gather().
Coding out remaining code
Let’s start with coding the code which will be outside the functions:
if __name__ == "__main__":
arguments = get_args()
target = arguments.target
extensions = arguments.extensions
hide_title = arguments.hide_title or False
redirection = arguments.follow_redirect or False
useragent = arguments.user_agent
match_codes = arguments.match_codes
match_size = arguments.match_size
filter_codes = arguments.filter_codes
filter_size = arguments.filter_size
output = arguments.output
header = arguments.header
wordlist = arguments.wordlist
if match_size and filter_size:
print(colored(
"[+] For now Using Matching and Filtering Response Length together is not available !", 'red'))
exit(1)
if match_codes and filter_codes:
print(colored(
"[+] For now Using Matching and Filtering Response Status code together is not available !", 'red'))
exit(1)
if not path.exists(wordlist):
print(colored("[-] Provide a valid wordlist file!", 'red'))
exit(1)
headers = {}
if header:
for h in header:
key, value = h.split(':', 1)
headers[key] = value.strip()
headers['User-Agent'] = useragent
bruteforcer = Dir_bruteforcer(target, wordlist, extensions, redirection, headers,
match_codes, match_size, filter_codes, filter_size, output, hide_title)
bruteforcer.print_banner()
asyncio.run(bruteforcer.main())
Ques: What is __name__==__main__ ?
Answer: The __name__ variable in Python holds the name of the current namespace or module. Each module has its own namespace, which acts as a container for the variables, functions, and classes defined within that module.
In this case the code block inside if __name__ == "__main__": will only execute if the current script is being run as the main program. It will not execute if the script is imported as a module by another script.
- Firstly, we retrieve our command-line arguments in line
arguments = get_args()
- Then we assign the values of command-line arguments to their specific named variables. But in these two lines :
hide_title = arguments.hide_title or False
redirection = arguments.follow_redirect or False
In these lines, we are using the or operator, If these command-line arguments, they will be assigned a false value.
- This block checks if both
match_size and filter_size arguments are provided together. If so, it prints an error message indicating that using matching and filtering response length together is not currently available, and exits the script with a status code of 1.
if match_size and filter_size:
print(colored("[+] For now Using Matching and Filtering Response Length together is not available !", 'red'))
exit(1)
- This block checks if both
match_codes and filter_codes arguments are provided together. If so, it prints an error message indicating that using matching and filtering response status codes together is not currently available and exits the script with a status code of 1.
if match_codes and filter_codes:
print(colored("[+] For now Using Matching and Filtering Response Status code together is not available !", 'red'))
exit(1)
- This block checks if the provided
wordlist file exists. If it doesn’t exist, it prints an error message indicating that a valid wordlist file should be provided, and exits the script with a status code of 1.
if not path.exists(wordlist):
print(colored("[-] Provide a valid wordlist file!", 'red'))
exit(1)
- Then these lines create an empty
headers dictionary and populate it with key-value pairs extracted from the header list obtained from the arguments. The header values are split at the first colon (:) occurrence and the resulting key-value pairs are stored in the headers dictionary. Additionally, the User-Agent header is set using the value of the useragent variable.
headers = {}
if header:
for h in header:
key, value = h.split(':', 1)
headers[key] = value.strip()
headers['User-Agent'] = useragent
- Below line creates an instance of the
Dir_bruteforcer class, passing the extracted values as arguments to its constructor. It initializes the bruteforcer object. We will define this class after this function.
bruteforcer = Dir_bruteforcer(target, wordlist, extensions, redirection, headers,
match_codes, match_size, filter_codes, filter_size, output, hide_title)
- Then we print the banner, in which we are executing the class print_banner method.
bruteforcer.print_banner()
- Then we run the method named
main of the bruteforcer object, as the object is a coroutine so we will use asyncio.run to run that coroutine.
asyncio.run(bruteforcer.main())
Coding the class DIR_bruteforce class and its methods
To define a class we can simply write,
class Dir_bruteforcer():
statement1
statement2
statement3
Now let’s define start by defining our init method.
The __init__() method is a special method in Python classes that is automatically called when an object of the class is created. It is also known as the class constructor. The purpose of the __init__() method is to initialize the attributes (or properties) of the object with the provided values.
class Dir_bruteforcer():
def __init__(self, target, wordlist, extensions, follow_redirect, headers, match_codes, match_size, filter_codes, filter_size, outputfile, hide_title) -> None:
self.target = target
self.wordlist = wordlist
self.extensions = extensions
self.follow_redirect = follow_redirect
self.headers = headers
self.hide_title = hide_title
self.match_codes = match_codes
self.match_size = match_size
self.filter_codes = filter_codes
self.filter_size = filter_size
self.outputfile = outputfile
Remember above, we had created an instance object named bruteforcer from the class Dir_bruteforcer(), and while creating that instance we also provided many values.
bruteforcer = Dir_bruteforcer(target, wordlist, extensions, redirection, headers,
match_codes, match_size, filter_codes, filter_size, output, hide_title)
So when we create an instance of the object the __init__ method executes on its own and in this __init__ method, we are defining variables using the self keyword.
The self keyword in Python is a conventionally used parameter name that refers to the instance of the class itself. It acts as a reference to the instance, allowing you to access its attributes and methods within the class
means if you want to access the variable within the class you can use that using the notation self.var_name.
So In this whole __init__ method, we define a variable which can be used in the whole class.
Defining the print_banner() method
The print_banner() method in the class is responsible for printing a banner or introductory message with information about the directory and file brute-forcer
def print_banner(self):
from datetime import datetime
print("-"*80)
print(colored(
f"Directory and file bruteforcer starting at {datetime.now().strftime('%d/%m/%Y %H:%M:%S')}", 'cyan', attrs=['bold']))
print("-"*80)
print(colored("[*] Target Url".ljust(20, " "),
'light_red'), ":", f"{self.target}")
print(colored("[*] Wordlist".ljust(20, " "),
'light_red'), ":", f"{self.wordlist}")
if self.headers:
print(colored("[*] Headers".ljust(20, " "),
'light_red'), ":", f"{headers}")
if self.extensions:
print(colored("[*] Extensions".ljust(20, " "),
'light_red'), ":", f"{self.extensions}")
if self.outputfile:
print(colored("[*] Output File".ljust(20, " "),
'light_red'), ":", f"{self.outputfile}")
if self.match_size:
print(colored("[*] Match Res size".ljust(20, " "), 'light_red'),
":", f"{self.match_size}")
if self.match_codes or self.filter_codes:
if self.match_codes:
print(colored("[*] Match Codes".ljust(20, " "),
'light_red'), ":", f"{self.match_codes}")
if self.filter_codes:
print(colored("[*] Filter Codes".ljust(20, " "), 'light_red'),
":", f"{self.filter_codes}")
else:
print(colored("[*] Status Codes".ljust(20, " "),
'light_red'), ":", f"All Status Codes")
if self.filter_size:
print(colored("[*] Filter Response Size".ljust(20, " "), 'light_red'),
":", f"{self.filter_size}")
print("-"*80)
print("-"*80)
The code here is self-understandable, so I will not explain this code.
Defining the main method
async def main(self):
try:
tasks = []
dirs = []
with open(self.wordlist, 'r') as f:
for word in f.readlines():
word = word.strip()
dirs.append(f'/{word}')
if self.extensions:
for ext in self.extensions:
dirs.append(f'/{word}.{ext}')
async with aiohttp.ClientSession(headers=self.headers) as session:
for dir in dirs:
tasks.append(asyncio.create_task(
self.brute_dir(word=dir, session=session)))
if len(tasks) >= 50:
await asyncio.gather(*tasks)
except aiohttp.ClientError as err:
print(f'An error occurred during the HTTP request: {str(err)}')
except KeyboardInterrupt:
print("Process interrupted by keyboard")
exit(0)
except Exception as err:
print(f"An unexpected error occurred:he {err}")
exit(1)
The aiohttp library is a popular Python library for building asynchronous HTTP-based applications.
The aiohttp library provides both client and server functionality for making HTTP requests and handling HTTP responses asynchronously. *
Then ClientSession manages a connection pool internally, allowing you to reuse connections for multiple requests to the same server. This can improve performance by reducing the overhead of establishing new connections for each request.
- After creating a connection pool, we iterate over the dirs list and create an asynchronous task using
asyncio.create_task(). The task is to call the brute_dir() method of the current Dir_bruteforcer instance, passing the directory and the session as arguments. The task is added to the tasks list.
- If the number of tasks in the
tasks list reaches or exceeds 50, it waits for all the tasks to complete using await asyncio.gather(*tasks). This allows concurrent execution of up to 50 tasks at a time.
- Then after these lines , we are handling exceptions.
- aiohttp.ClientError: Catches the error that happened while creating the HTTP request and prints the error message
- KeyboardInterrupt: When the user interrupts the process with Ctrl+C, it prints a message indicating that the process was interrupted and exits gracefully with a status code of
0.
- If any other exception apart from the mentioned above occurs, it prints an error message with the specific error and exits with a status code of
1.
Defining the brute_dir method
async def brute_dir(self, word, session):
try:
url = f"{self.target}{word}"
async with session.get(url) as response:
html = await response.text()
response_length = str(len(html))
status_code = str(response.status)
url = url.split('://')[1]
soup = BeautifulSoup(html, 'html.parser')
title = soup.title.string if soup.title else []
if self.match_codes:
if status_code in self.match_codes:
if self.filter_size:
if response_length not in self.filter_size:
await self.print_and_save_output(
status_code, response_length, url, title)
elif self.match_size:
if response_length in self.match_size:
await self.print_and_save_output(
status_code, response_length, url, title)
else:
await self.print_and_save_output(
status_code, response_length, url, title)
elif self.filter_codes:
if status_code not in self.filter_codes:
if self.filter_size:
if response_length not in self.filter_size:
await self.print_and_save_output(
status_code, response_length, url, title)
elif self.match_size:
if response_length in self.match_size:
await self.print_and_save_output(
status_code, response_length, url, title)
else:
await self.print_and_save_output(
status_code, response_length, url, title)
else:
if self.filter_size:
if response_length not in self.filter_size:
await self.print_and_save_output(
status_code, response_length, url, title)
elif self.match_size:
if response_length in self.match_size:
await self.print_and_save_output(
status_code, response_length, url, title)
else:
await self.print_and_save_output(
status_code, response_length, url, title)
except aiohttp.ClientError as err:
print(f'An error occurred during the HTTP request: {str(err)}')
except KeyboardInterrupt:
print("Process interrupted by keyboard")
exit(0)
except Exception as err:
print(f"An unexpected error occurred:he {err}")
exit(1)
Then after defining the main asynchronous function, we will then define the brute_dir() function which is mainly responsible for sending requests and getting the response back, and sending the command execution to the print_and_save_output function to print results while considering various conditions.
- Firstly, we define an async coroutine named
brute_dir inside the class Dir_Bruteforcer() , inside the brute_dir function we are taking the word to brute force and the main session created in the main coroutine.
- Then we start a try-except class to catch various exceptions which can occur while sending requests.
- Now in a try block, we will write our code.
url = f"{self.target}{word}"In this line, we are combining the target URL and the word- Then we send that request in line
async with session.get(url) as a response: , by using with, it ensures proper handling of the HTTP request and in the next line html = await response.text() we are waiting for the response and storing the data in the html variable.
response_length = str(len(html)) in this line, we are extracting the response length - s
tatus_code = str(response.status) in this line, we are retrieving the response code like 200, 404
url = url.split('://')[1] now in this line you are removing the https or http part from the URL and reassigning it to the same variable. soup = BeautifulSoup(html, 'html.parser'), now we are creating a soup named parser object from using the BeautifulSoup class which is a class present in the bs4 module.
Beautiful Soup is a Python library used for web scraping and parsing HTML and XML documents. It provides a convenient way to extract and navigate data from web pages, making it easier to work with the complex HTML structure of websites.
now in this line title = soup.title.string if soup.title else []: we are setting the title to the response title only if it exists else we are assigning it to an empty list.
Now the code after this checks the filtering and blocking conditions, for the first block - If self.match_codes is set, the code checks if the extracted status_code is in the list of matched codes. If it matches, it proceeds to further checks for filtering by size (self.filter_size) or matching by size (self.match_size). If no size filters or matches are specified, the response is printed and saved to the output file (if provided).
In the next block it checks - If self.filter_codes is set, the code checks if the extracted status_code is not in the list of filtered codes. If it is not present in the filtered codes, it proceeds to further checks for filtering or matching by size, similar to the previous case.
And in the next block, it checks - If neither self.match_codes nor self.filter_codes are set, it directly checks for filtering or matching by size, as specified in the command-line arguments.
In each of these blocks the method await self.print_and_save_output(...) is called to display and save the output if the conditions are met.
- In the except block, we are catching various exceptions like aiohttp.ClientError allows us to catch various exceptions which can happen during sending the response and after we are also catching the KeyboardInterrupt exception which helps us in instructing the program what to do in case of user press
ctrl + c.
Defining the print_and_save_output method
async def print_and_save_output(self, status_code, response_length, url, title):
status_code = int(status_code)
color = 'grey'
if status_code >= 200 and status_code < 300:
color = 'green'
elif status_code >= 300 and status_code < 400:
color = 'yellow'
elif status_code >= 400 and status_code < 500:
color = 'red'
elif status_code > 500 and status_code < 600:
color = 'magenta'
status_code_str = str(status_code).ljust(9, " ")
response_length_str = str(response_length).ljust(9)
url_str = url.ljust(30)
output = f"{colored(status_code_str, color)} {response_length_str} {url_str}"
if not self.hide_title:
output += f" [{title}]"
print(output)
if self.outputfile:
with open(self.outputfile, 'a+') as f:
f.write(
f"{status_code_str} {response_length_str} {url_str} [{title}]")
This coroutine method is responsible for printing the results in a well-formatted manner by taking the necessary parameters from the method calling in the brute_dir method and if opted for saving the output, it also saves the output in the file.
Let’s understand this code!
- In the first line
status_code = int(status_code) we are converting the status_code value to int type for the furthur comparison reasons.
color = 'grey' In the next line we are setting a default color for the response code color display.- And in the next few lines if the status code varies between 200 and 300, we are setting the color to green, and if it varies between 300 and 400, then we are setting the color to yellow and if it varies between 400 and 500 we are setting the color to red and in the end if it lies between 500 and 600, we are setting it to magenta.
- Now we are creating some variables for formatting and aligning them using the string class’s just method.
- After creating the variables we are joining them and storing them inside the output variable.
- If you have not chosen the hide title option so we will also add the title to the output.
- Now we will print the output.
- At the end if you have chosen to save the data, we open a file in append mode and save the data inside it.
So this is the end of the code, if you want to reach out the whole code you can check it out at: 🔗Link to code
❌Disclaimer
Note: The following disclaimer is intended for educational and ethical use of the Directory-Bruteforce script.
This script is for educational purposes only. Unauthorized use of any website or network is strictly prohibited. The authors are not liable for any misuse or legal consequences resulting from its use. Use responsibly and abide by applicable laws. Obtain proper authorization before security testing.
