Turbo Data Miner

Turbo Data Miner is a Burp Suite Extension, which adds a new tab Turbo Miner to Burp Suite's UI as well as a new entry Turbo Data Miner to Burp Suite's Extensions context menu entry (see Context Menu Analyzers).

Within the Turbo Miner tab, Turbo Data Miner provides a basic Python IDE to efficiently and dynamically accomplish different types of intelligence objectives. Turbo Data Miner's API is documented below.

For more information, refer to Turbo Data Miner's Wiki.

Author

Lukas Reiter (@chopicalquy) - Turbo Data Miner

License

This project is licensed under the GPLv3 License - see the license file for details.

Turbo Data Miner API

The following table provides an overview about the variables and methods that are available for Python scripts in the respective tab.
Name Type Scope Proxy History Analyzer Site Map Analyzer Context Menu Analyzer HTTP Listener Analyzer HTTP Listener Modifier Proxy Listener Modifier Custom Scanner Check Custom Message Editor Description
add_table_row
Method
-
Refer to Method Details for more information.
analyze_request
Method
-
Refer to Method Details for more information.
analyze_response
Method
-
Refer to Method Details for more information.
analyze_signatures
Method
-
Refer to Method Details for more information.
callbacks
Variable
r
This variable is of type IBurpExtenderCallbacks, which can be used to access functionalities provided by Burp Suite. For more information refer to Burp Extender API.
compress_gzip
Method
-
Refer to Method Details for more information.
decode_html
Method
-
Refer to Method Details for more information.
decode_jwt
Method
-
Refer to Method Details for more information.
decompress_gzip
Method
-
Refer to Method Details for more information.
encode_jwt
Method
-
Refer to Method Details for more information.
find_domains
Method
-
Refer to Method Details for more information.
find_error_messages
Method
-
Refer to Method Details for more information.
find_versions
Method
-
Refer to Method Details for more information.
get_content_length
Method
-
Refer to Method Details for more information.
get_content_type
Method
-
Refer to Method Details for more information.
get_cookie_attributes
Method
-
Refer to Method Details for more information.
get_cookies
Method
-
Refer to Method Details for more information.
get_extension_info
Method
-
Refer to Method Details for more information.
get_header
Method
-
Refer to Method Details for more information.
get_headers
Method
-
Refer to Method Details for more information.
get_hostname
Method
-
Refer to Method Details for more information.
get_json_attribute_by_path
Method
-
Refer to Method Details for more information.
get_json_attributes
Method
-
Refer to Method Details for more information.
get_jwt
Method
-
Refer to Method Details for more information.
get_parameter_name
Method
-
Refer to Method Details for more information.
get_parameters
Method
-
Refer to Method Details for more information.
has_header
Method
-
Refer to Method Details for more information.
hast_stopped
Method
-
Refer to Method Details for more information.
header
Variable
rw
This variable must be a one-dimensional list. Use this list to set the GUI table's header. The GUI table header is set at the beginning when variable ref is 1.
helpers
Variable
r
This variable is of type IExtensionHelpers, which can be used to access functionalities provided by Burp Suite. For more information refer to Burp Extender API.
in_scope
Variable
r
This variable is of type bool and specifies whether the current request/response item is in the project's scope. You could also use callbacks.isInScope(java.lang.URL) to determine whether the given URL is in scope.
invocation
Variable
r
This variable is of type IContextMenuInvocation and contains details about a context menu invocation. Context Menu Analyzer scripts can query this interface to obtain details of the invocation event.
is_request
Variable
r
This variable is of type bool and specifies whether the given IInterceptedProxyMessage object is a request (True) or a response (False).
message_info
Variable
r
This variable is of type IHttpRequestResponse and represents the request/response item that was processed or from which information was extracted. Clicking on a table row in one of Turbo Data Miner's Analyzers, will always display the information of this variable in tab Message Information. For more information about the IHttpRequestResponse interface, refer to Burp Extender API
message_infos
Variable
rw
This variable is of type dict and can be used by any Turbo Data Miner Analyzer to add additional IHttpRequestResponse items to the message info tabs. Note that the dict keys must be of type str and their values of type IHttpRequestResponse in order to be displayed.
plugin_id
Variable
r
This variable is of type int and specifies the Turbo Data Miner module that executes the script. Flag 0 stands for the Proxy History Analyzer, 1 for the HTTP Listener Analyzers, 3 for the HTTP Listener Modifiers, 4 for the Proxy Listener Modifiers, 5 for the Custom Message Editors, 6 for the Site Map Analyzers, 7 for the Context Menu Analyzers and 8 for the Custom Scanner Checks.
proxy_message_info
Variable
rw
This variable is of type IInterceptedProxyMessage and represents the request/response item that can be processed or from which information can be extracted. For more information refer to Burp Extender API.
ref
Variable
r
This variable is of type int and is reset to 1, if the Start button is pressed and is then auto incremented with each subsequent request/response item (IHttpRequestResponse). You can for example use this variable to reset the session dictionary at the beginning of a script execution or determine that several table rows belong to the same request/response item.
request_info
Variable
r
This variable is of type IRequestInfo and represents the current request of the current message_info variable. For more information refer to Burp Extender API.
row_count
Variable
r
For the Proxy History and Site Map Analyzer this variable holds the total number request/response items.
rows
Variable
rw
This variable is a two-dimensional list. Use this two-dimensional list to add content to the GUI table. Thereby, each row in rows represents one row in the GUI table. If the variable is a one-dimensional list, then each element is added as a single row to the GUI table.
ScanIssue
Class
-
This class is derived from IScanIssue and can be used by Custom Scanner Check plugins to store and register IScanIssue. For more information refer to Burp Extender API.
send_http_message
Method
-
Refer to Method Details for more information.
session
Variable
rw
This variable is of type dict. You can use this variable to store information across processing iterations. For example, this variable can be used to deduplicate content
show_scope_parameter_dialog
Method
-
Refer to Method Details for more information.
split_http_header
Method
-
Refer to Method Details for more information.
tool_flag
Variable
r
This variable is of type int and indicates the Burp Suite tool that issued the request. The Burp Suite tool flags are defined in the IBurpExtenderCallbacks interface. To convert the value into a human readable string, use callbacks.getToolName(tool_flag). For more information refer to Burp Extender API.
url
Variable
r
This variable is of type java.net.URL and holds the URL of the current message_info variable so that you do not have to explicitly extract it.
url_decode
Method
-
Refer to Method Details for more information.
xerceslib
Variable
r
Per default, BApps written in Python cannot use Burp Suite's build in XML library as documented in the Burp Suite User Forum thread SAXParser Dependency Delimma. This same thread describes a workaround by loading the Apache Xerces library within BApps themselves.
This variable contains a reference to a java.net.URLClassLoader object, which loaded Turbo Data Miner's own local Apache Xerces library (version 1.12.1). This variable can then be used within Python scripts to load Turbo Data Miner's own Apache Xerces library into the current execution thread and subsequently, to successfully process XML files. In order to make XML library available, use the following Python code:

from java.lang import Thread
Thread.currentThread().setContextClassLoader(xerceslib)
import xml.etree.ElementTree as ET
[...]
root = ET.fromstring(body_content.encode("utf-8"))
[...]

Method Details

This chapter provides an overview about Turbo Data Miner's API methods.
add_table_row(self, row, message_infos=None)
This method adds the given row to the GUI table as well as assigns additional IHttpRequestResponse instances to
the current row, which are displayed in the GUI table's details pane if the row is selected.

:param row (List[str]): One-dimensional string list that will be added as a new row to the GUI table.
:param row (Dict[str, IHttpRequestResponse]): Dictionary containing the IHttpRequestResponse instances that shall
be displayed in the GUI table's details pane when the given row is selected. The dictionary's key specifies the
tab title.
analyze_request(self, message_info)
This method returns an IRequestInfo object based on the given IHttpRequestResponse object.

:param message_info (IHttpRequestResponse): The IHttpRequestResponse whose request should be returned as an
IRequestInfo object.
:return (IRequestInfo): An IRequestInfo object or None, if no request was found.
analyze_response(self, message_info)
This method returns an IResponseInfo object based on the given IHttpRequestResponse object.

:param message_info (IHttpRequestResponse): The IHttpRequestResponse whose request should be returned as an
IResponseInfo object.
:return (IResponseInfo): An IResponseInfo object or None, if no response was found.
analyze_signatures(self, content, strict=False)
This method checks whether the given string matches one of the known file signatures specified in an internal
database.

:param content (str): The string that is tested for known file signatures.
:param strict (bool): Bool which specifies whether the file signatures can appear anywhere within the given
string (False) or at the expected position (True). Set this parameter to False (default) if you, for example,
want to determine, whether a request or response might contain a file somewhere. Note that this might return an
increased number of false positives.
:return (List[Dict[str, object]]: List of dictionaries. Each dictionary contains the following keys that specify
information about the matched file signature: extensions (List[str]), category (str), description (str),
offset (int), hex_signatures (str), str_signatures (list), b64_signatures (list)
compress_gzip(self, content)
This method compresses the given string using GZIP and returns the compressed byte array. Note that this
method might throw an exception.

:param content (str): The string that shall be GZIP compressed.
:return (List[bytes]): Byte array containing the GZIP compressed string.
decode_html(self, content)
This method can be used to HTML decode the given string.
:param content (str): Value that shall be HTML decoded.
:return (str): The HTML decoded version of the provided value.
decode_jwt(self, jwt)
This method decodes the given JSON Web Token (JWT) and returns a triple containing the JWT header, JWT payload,
and JWT signature.

:param jwt (str): String containing the JWT.
:return (List[str]): List with three string elements. The first element contains the header (or None), the
second element the payload (or None), and the third element the signature (or None) of the JWT.
decompress_gzip(self, content)
This method decompresses the given GZIP compressed byte array.

:param content (list[bytes]): The byte array whose content shall be decompressed.
:return (str): Decompresed string.
:raise ZipException: If a GZIP format error has occurred or the compression method used is unsupported.
:raise IOException: if an I/O error has occurred.
encode_jwt(self, header, payload, signature=None)
This method encodes the given JSON Web Token (JWT) header, JWT payload, and JWT signature into a complete JWT
string.

:param header (str): The dictionary containing the JWT header information.
:param payload (str): The dictionary containing the JWT payload information.
:param signature (str): The JWT's signature.
:return (str): The final JWT string.
find_domains(self, content)
This method searches the given text for valid host and domain names.

The search is based on a regular expression and in order to decrease the likelihood of false positives, each
identified identified domains top-level domain (TLD) is compared to a known list of TLDs.

:param content (str): The string in which the domain names are searched.
:return (List[str[): List of identified domain names.
find_error_messages(self, content)
This method checks whether the given string matches one of the known error signatures based on an internal database.

:param content (str): The string that is tested for known file signatures.
:return (List[Dict[str, object]]: List of dictionaries. Each dictionary contains the following keys that specify
information about the matched error signature: regex (str), group (int), type (str), severity(str), confidence (str).
find_versions(self, content)
This method searches the given text for known software versions based on an internal database
(source: vulners.com).

:param content: The string in which the software versions are searched.
:return (List[Dict[str, str]]): List of dictionaries containing details about the identified software versions.
each dictionary contains the following keys: software, type, version, cpe, alias, source
get_content_length(self, headers)
This method returns the first occurrence of the Content-Length from the given list of headers.

:param headers (List[str]): The list of headers that shall be searched for the first occurrence of the
Content-Length header. Usually, the list of headers is obtained via the getHeaders method implemented by
Burp Suite's IRequestInfo or IResponseInfo interfaces.
:return (int): Integer containing the content of the Content-Type header or None if it does not exist.
get_content_type(self, headers)
This method returns the first occurrence of the Content-Type from the given list of headers.

:param headers (List[str]): The list of headers that shall be searched for the first occurrence of the
Content-Type header. Usually, the list of headers is obtained via the getHeaders method implemented by
Burp Suite's IRequestInfo or IResponseInfo interfaces.
:return (str): String containing the content of the Content-Type header or None if it does not exist.
get_cookie_attributes(self)
This method returns a static list of all possible cookie attributes. This list can be used in combination with
API method get_cookies to convert all obtained cookies from a dictionary into a list/table format.
:return (List[str]): Static string list containing the following elements: name, value, expires, max-age,
domain, path, secure, httponly, samesite
get_cookies(self, item, filter=[])
This method takes an IResponseInfo or IRequestInfo object as the first argument and extracts all its cookie
information. The second optional argument acts as a filtering option that limits the cookies to be extracted.

:param item (IRequestInfo/IResponseInfo): The IRequestInfo or IResponseInfo item whose session information shall
be obtained.
:param filter (List[str]): List of cookie names whose attributes shall be extracted and returned.
:return (List[Dict[str, str]]): The method returns a list of dictionaries. Each dictionary contains the
following keys, which are also returned by API method get_cookie_attributes: "name", "value", "expires",
"max-age", "domain", "path", "secure", "httponly", "samesite"
get_extension_info(self, content)
This method analyses the file extension of the given string and returns additional information like file
category about the first extension that matches.

:param content (str): The string whose file extension should be analyzed.
:return (dict): Dictionary containing information about the string's file extension or None if no extension
was identified. The dictionary contains the following keys: extension (str), category (str), description (str)
get_header(self, headers, header_name)
This method is a simplified version of API method get_headers. It analyses a given list of headers and returns
the first occurrence of the header information that matches a given name.

:param headers (List[str]): The list of headers that shall be searched for the first occurrence of the given
header. Usually, the list of headers is obtained via the getHeaders method implemented by Burp Suite's
IRequestInfo or IResponseInfo interfaces.
:param header_name (str): The name (case insensitive) of the header whose value shall be returned.
:return (tuple): Returns the value of the first occurrence of the header as a tuple; the first element is
the header name and the second elements is its content. If the header was not found, then a tuple containing
(None, None) is returned.
get_headers(self, headers, re_headers)
This method analyses a given list of headers and returns all occurrences of the header information that
matches a given list of regular expressions.

:param headers (List[str]): The list of headers that shall be searched for the first occurrence of the given
header. Usually, the list of headers is obtained via the getHeaders method implemented by Burp Suite's
IRequestInfo or IResponseInfo interfaces.
:param re_headers (List[re.Pattern]): List of regular expressions that specify the patterns for header names
whose header values shall be returned.
:return (Dict[str, List[str]]): The keys of the returned dictionary are always the strings of the re_headers
list ({item.pattern: [] for item in re_headers}) and the corresponding dictionary values contain the
identified header values.
get_hostname(self, url)
This method removes the file and query part of the given URL so that only the protocol, hostname, and port parts
remain.

:param url (java.lang.URL): The URL from which the file and query part is removed.
:return (java.lang.URL): The new java.net.URL instance containing the protocol, host, and port information
get_json_attribute_by_path(self, body, path, default_value=None)
This method returns the JSON attribute located at position path in JSON object body.

:param body (str/dict): Contains the JSON object, which is either of type string or dictionary, that shall be searched.
:param path (str): Path (e.g. data/value/) that specifies the attribute that shall be returned.
:param default_value (object): The default value that shall be returned if the requested path does not exist.
:return (dict): The JSON attribute at location path or default_value.
:raise ValueError: This exception is thrown when the given body cannot be converted into a dictionary.
get_json_attributes(self, body, attributes, must_match=0)
Searches the string stored in the body variable for those attribute names, which are specified by the
attributes list.

This method converts the given string body into a JSON object (if not already the case) and then searches
this JSON object recursively for attributes that are specified by the attributes list.
:param body (str/dict): Contains the JSON object, which is either of type string or dictionary, that shall be
searched.
:param attributes (List[str]): List of attribute names those the values should be extracted from the given
JSON object.
:param must_match (int=0): Specifies how many attributes in the provided list must be found on the save level
in the JSON object in order to be added to the return list. If the parameter is not specified or less than or
equal to 0, then any occurrence is added to the list.
:return (List[Dict[str, str]]): The keys of each dictionary represent the values specified in the provided
attributes list and the values represent the corresponding values extracted from the JSON object.
:raise ValueError: This exception is thrown when the given body cannot be converted into a
dictionary.
get_jwt(self, headers, re_header=r'^Authorization:\s+Bearer\s+(?PeyJ\w+?\.eyJ\w+?\..+?)$')
This method searches the given array of headers for the first occurrence that matches the given authorization
header and extracts as well as decodes and returns the given JSON Web Token (JWT).

:param headers (List[str]): List of strings that contain the headers to be searched. Usually, the list of
headers is obtained via the getHeaders method implemented by Burp Suite's IRequestInfo or IResponseInfo
interfaces.
:param re_header: The regular expression string (case insensitive) that specifies how the JWT can be extracted.
Note that the regular expression must contain the named group jwt, which specifies the position of the jwt to
be extracted.
:return (List[str]): List with three string elements. The first element contains the header (or None), the
second element the payload (or None), and the third element the signature (or None) of the JWT.
get_parameter_name(self, type)
This method returns the descriptive name of the given parameter type value. This method is usually used to
convert the value returned by getType method of the IParameter class into a string (e.g., value 0 is GET, value
1 is POST, etc.).

:param type (int): The integer value that shall be returned into the string.
:return (str): The descriptive name that matches the given type parameter value or None.
get_parameters(self, request_info, re_names)
This method analyses the parameters of the given IRequestInfo object and returns all occurrences of parameters
whose names match one of the given regular expressions.

:param request_info (RequestInfo): The IRequestInfo object, whose parameters shall be analysed.
header. Usually, the list of headers is obtained via the getHeaders method implemented by Burp Suite's
IRequestInfo or IResponseInfo interfaces.
:param re_names (List[re.Pattern]): List of regular expressions that specify the patterns for parameter names,
whose parameter values shall be returned.
:return (Dict[str, List[IParameter]]): The keys of the returned dictionary are always the strings of the
re_names list ({item.pattern: [] for item in re_names}) and the corresponding dictionary values contain the
IParameter objects whose names matched the corresponding regular expression.
has_header(self, headers, name)
This method checks whether the given header name exists in the given list of headers.

:param headers (List[str]): The list of headers that shall be searched to determine if the given header name exists.
:param name (str): The header name that shall be searched.
:return (bool): True, if the given header name exists in the headers list, else False.
has_stopped(self)
This method returns true, if the user clicked the Stop button to immediately stop the executiono of
the current script. This method should be used within potentially long-lasting loops to check whether the loop
should be immediately exited.
:return (bool): True, if the user clicked the Stop, else False.
send_http_message(self, request, http_service)
This method sends the given request to the given HTTP service.

:param request (str): The request (including the request headers and body) that shall be sent.
:param http_service (IHttpService): The service to which the given request shall be sent.
:return (IHttpRequestResponse): Object containing the sent and received data.
show_scope_parameter_dialog(self, request)
This method displays a JDialog displaying the given IRequestInfo's parameters. The user can then check
these parameters as well as specify whether the selection is a black- or whitelisting.

:param request (IRequestInfo): The request object whose parameters is displayed in the JDialog and whose
parameters can then be filtered using the JDialog object's match method (see below).
:return: ParameterScopeDialog object that exposes the following methods and variables:
- canceled: This variable is of type bool and specifies whether the user closed the dialog via the Cancel
button (canceled == True) or via the Ok button (canceled == False).
- match(IParameter): This method returns true if the given IParameter parameter was checked by the user in the
JDialog.
- whitelisting: This variable specifies whether the user's parameter selection is a white- or blacklisting.

The following code provides an example of how this method can be used:

    # Display the dialog so that the user can select the parameters to be processed
    scope_object = show_scope_parameter_dialog(request_info)
    # Only if the user clicked the Ok button, the script continues
    if not scope_object.canceled:
        # We iterate over each of the request's parameters
        for parameter in request_info.getParameters():
            # We determine if the parameter was selected by the user
            if scope_object.match(parameter):
                # The user selected the current parameter for a white- or blacklisting approach
                task = "Whitelisting" if scope_object.whitelisting else "Blacklisting"
            else:
                # The user did not select the current parameter
                task = "Not selected"
            # We add the results to the UI table
            rows.append([task, get_parameter_name(parameter.getType()), parameter.getName()])
split_http_header(self, header)
This method splits the given header string into the header name and value. Usually this method is used in
combination with the getHeaders method of the IRequestInfo or IResponseInfo interface.

:param request (str): The header whose header name and value should be returned.
:return (tuple): The first element contains the header name and the second element the header value. If the
header is invalid (does not contain a colon), then the (None, None) is returned.
url_decode(self, data, recursive=True)
This method can be used to URL-decode the specified data. It is an alternative to IExtensionHelpers.urlDecode
and allows recursively decoding the given value.

:param data (str): The data to be decoded.
:param recursive (bool): If true, then the given data is recusively decoded until it does not change anymore. If
false, then the value is decoded once.
:return (str): The decoded data.