https://blog.didierstevens.com/2018/01/29/new-tool-jpegdump-py/
https://blog.didierstevens.com/2022/09/02/update-jpegdump-py-version-0-0-10/
Manual:
The JPEG file format defines a JPEG file as a sequence of segments. Each segments begins with a 2-byte marker. The first byte is always 0xFF, and the second byte specifies the type of marker. Most markers are followed by 2 bytes indicating the length of the data (payload) that follows. Some marker have no data.
Start Of Image (SOI: 0xFFD8), End Of Image (EOI: 0xFFD9) and Restart (RST: 0xFF00 - 0xFF07) markers have no data.
jpegdump.py is a tool that takes one or more files to analyze them for sequences. It does this by searching for 0xFF bytes and then starts to parse them as markers (possibly followed by data).
This tool is very versatile when it comes to handling files, later full details will be provided.
This Python script was developed with Python 2.7 and tested with Python 2.7 and 3.5.
Here is an example for image j.jpg:
jpegdump.py j.jpg
File: j.jpg
1 p=0x00000000 : m=ffd8 SOI
2 p=0x00000002 d=0: m=ffe1 APP1 l=64837 e=6.886123 a=61.043573
3 p=0x0000fd49 d=0: m=ffdb DQT l= 197 e=2.343628 a=0.618557 remark: 195/65 = 3.000000
4 p=0x0000fe10 d=0: m=ffc4 DHT l= 418 e=7.009735 a=14.149398
5 p=0x0000ffb4 d=0: m=ffc0 SOF0 l= 17 e=3.189898 a=27.357143 remark: p=8 h=3456 w=4608 c=3
6 p=0x0000ffc7 d=0: m=ffda SOS l= 12 e=2.446439 a=21.222222
entropy-coded data: l=3529048 e=7.983918 a=84.905246 #ff00=8681
7 p=0x0036d92d d=0: m=ffd9 EOI
Each line represents the details of a segment. The first number, the index, is a sequential number generated by jpegdump and used to select a segment for further analysis.
The second field (p=) is the position field: it gives the position of the marker inside the file, expressed as a hexadecimal number.
The third field (d=) is the difference field: it gives the number of bytes between this sequence and the previous sequence, expressed as a decimal number. This value should be 0 for well-formed jpeg files.
The fourth field (m=) is the marker field: 2 hexadecimal bytes (0xFF??) indicating the type of the sequence followed by an acronym of the sequency type. This acronym can be:
SOI: Start Of Image
SOF0: Start Of Frame (baseline DCT)
SOF2: Start Of Frame (progressive DCT)
DHT: Define Huffman Table
DQT: Define Quantization Table(s)
DRI: Define Restart Interval
SOS: Start Of Scan
RST?: Restart
APP?: Application
COM: Comment
EOI: End Of Image
This information is followed by statistics for the data of the sequence (SOI, EOI and RST sequences have no data):
l= is the length of the data
e= is the entropy of the data
a= is the "average of the absolute difference between 2 consecutive bytes"
Finally, this can be followed by a remark specific for the sequence.
DQT: the number of tables
SOF0: the precision in bits (p=), height in pixel (h=), width in pixels (w=) and number of color components (c=).
SOS: the number of color components (c=).
A SOS segment's data is followed by the image data. The statistics of the image data are displayed on the second line. Image data can contain 0xFF bytes, which could be mistaken for markers. To clearly indicate that 0xFF bytes inside image data are not markers, 0xFF bytes are always followed by 0x00 (byte stuffing). This is counted by value #ff00=.
If the last segment is incomplete, a *warning* line will be displayed.
If the last segment is followed by extra bytes, a *trailing* line will be displayed.
Use option -t if you want to consider all bytes after the first EOI marker as trailing (e.g. without being parsed).
Option -E (extra) can be used to add extra information to the overview of segments. Like the hash of the data inside each segment (md5, sha1 or/and sha256).
Like this:
jpegdump.py -E md5 j.jpg
File: j.jpg
1 p=0x00000000 : m=ffd8 SOI
2 p=0x00000002 d=0: m=ffe1 APP1 l=64837 e=6.886123 a=61.043573 md5=4aeb69c4b5abad75f426c9e179518ac2
3 p=0x0000fd49 d=0: m=ffdb DQT l= 197 e=2.343628 a=0.618557 md5=61d311bde22762ae0e88b768e835eced remark: 195/65 = 3.000000
4 p=0x0000fe10 d=0: m=ffc4 DHT l= 418 e=7.009735 a=14.149398 md5=fc293d732e9e8ce63b9033a754454494
5 p=0x0000ffb4 d=0: m=ffc0 SOF0 l= 17 e=3.189898 a=27.357143 md5=b60979610c08a94a925fad40b54780d4 remark: p=8 h=3456 w=4608 c=3
6 p=0x0000ffc7 d=0: m=ffda SOS l= 12 e=2.446439 a=21.222222 md5=822524de858abbd7d9ccbbe35130cc2f remark: c=3
entropy-coded data: l=3529048 e=7.983918 a=84.905246 #ff00=8681
7 p=0x0036d92d d=0: m=ffd9 EOI
To display more than one hash, use a comma to separate the desired hashes.
If an hash appears more than once, then it will be followed by the index if the segment between paranthese, like this:
15 p=0x0000fe10 d=0: m=ffc4 DHT l= 418 e=7.009735 a=14.149398 md5=fc293d732e9e8ce63b9033a754454494(4)
This means that the hash of the data of segment 15, equal to fc293d732e9e8ce63b9033a754454494, is the same as the hash of the data of segment 4.
Segments can be selected to inspect their data. This can be done with option -s and the index number.
Example:
jpegdump.py -s 3 j.jpg
0000FD4D: 00 02 01 01 01 01 01 02 01 01 01 02 02 02 02 02 ................
0000FD5D: 04 03 02 02 02 02 05 04 04 03 04 06 05 06 06 06 ................
0000FD6D: 05 06 06 06 07 09 08 06 07 09 07 06 06 08 0B 08 ................
0000FD7D: 09 0A 0A 0A 0A 0A 06 08 0B 0C 0B 0A 0C 09 0A 0A ................
0000FD8D: 0A 01 02 02 02 02 02 02 05 03 03 05 0A 07 06 07 ................
0000FD9D: 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FDAD: 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FDBD: 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FDCD: 0A 0A 02 02 02 02 02 02 02 05 03 03 05 0A 07 06 ................
0000FDDD: 07 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FDED: 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FDFD: 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ................
0000FE0D: 0A 0A 0A ...
When an SOS segment is selected, the segment's data will be displayed:
jpegdump.py -s 6 j.jpg
0000FFCB: 03 01 00 02 11 03 11 00 3F 00 ........?.
To display the image data, append the indexnumber with letter i:
jpegdump.py -s 6i j.jpg
0000FFD5: F8 71 A1 37 30 97 45 FB DD 14 8E 94 D8 A1 4B 08 .q.70.E.......K.
0000FFE5: 09 91 30 4F 3C FA 57 E6 D2 7A 1F B3 47 71 00 1B ..0O<.W..z..Gq..
0000FFF5: 03 A3 9D B9 CB 63 AD 4D 25 F3 46 A3 C8 C9 E3 81 .....c.M%.F.....
00010005: 59 B4 74 44 7F 9C 7C B5 71 1E 37 2F CD 9E C6 98 Y.tD.|.q.7/....
00010015: F3 B3 61 42 13 8E E2 92 56 2E FA 10 4E F2 DF CA ..aB....V...N...
00010025: 86 D7 74 61 78 C6 3A 9F 5A D3 B3 92 68 86 64 04 ..tax.:.Z...h.d.
00010035: 81 C7 35 5B 08 9E 58 56 E4 03 BC 2E 3A FB D3 A2 ..5[..XV....:...
00010045: B1 82 10 1E 42 1B 9F BD 50 E4 1C B7 2B 5F 40 D7 ....B...P...+_@.
00010055: 2C 4B 7A F1 C7 6A 85 6D DA D1 91 F0 09 FE EB 8E ,Kz..j.m........
00010065: A2 AE 32 26 50 08 ED 05 CD C2 8D DB 55 9B 90 2A ..2&P.......U..*
If there is data between segments (difference value d= is not 0), then you can select this data by appending d to the index number.
For example, if the difference value of segment 10 is not 0, you can select that data that precedes segment 10, by selecting 10d: jpegdump.py -s 10d j.jpg
By default, and hexadecimal/ascii dump of the data dis produced (option -a).
Use option -x for an hexadecimal dump.
Option -A does an ASCII dump (like option -a), but with duplicate lines removed.
Use option -d for a binary dump, and option -u for a binary dump with byte unstuffing (replacing 0xFF00 with 0xFF).
By default, jpegdump will start to analyze the first marker it finds, regardless of its type.
To force jpegdump to search for and start with SOI markers, use option -f. With option -f, analysis will stop when an unknown marker is detected, and the next SOI marker will be searched.
Example:
jpegdump.py -f chrome.dmp
File: chrome.dmp
Found SOI:
1 p=0x000d4ebf : m=ffd8 SOI
2 p=0x000d4ec6 d=5: m=ffff l=53252 e=3.280114 a=40.576499
Found SOI:
1 p=0x0018594b : m=ffd8 SOI
2 p=0x001859d4 d=135: m=ff03 l= 0 e=0.000000 a=0.000000
...
Option f can output a lot of markers who are actually not part of a JPEG image. To use jpegdump to "carve" memory dumps or documents like PDF files, combine option -f with option -c (compliant). Used together with option -c, jpegdump will only report sequences of markers that are "compliant".
In this context, a compliant image is a sequence of markers that starts with SOI and ends with EOI, and where the difference (d=) between markers is 0.
Example:
jpegdump.py -f -c chrome.dmp
File: chrome.dmp
Found SOI:
1 p=0x01854e2f : m=ffd8 SOI
2 p=0x01854e31 d=0: m=ffd9 EOI
Found SOI:
1 p=0x02144d5b : m=ffd8 SOI
2 p=0x02144d5d d=0: m=ffe1 APP1 l= 24 e=1.945660 a=15.857143
3 p=0x02144d77 d=0: m=ffec APP12 l= 17 e=2.596792 a=23.857143
4 p=0x02144d8a d=0: m=ffee APP14 l= 14 e=2.751629 a=49.818182
5 p=0x02144d9a d=0: m=ffdb DQT l= 132 e=3.272901 a=1.573643 remark: 130/65 = 2.000000
6 p=0x02144e20 d=0: m=ffc0 SOF0 l= 17 e=2.872906 a=47.785714 remark: p=8 h=222 w=574 c=3
7 p=0x02144e33 d=0: m=ffc4 DHT l= 160 e=4.721659 a=27.662420
8 p=0x02144ed5 d=0: m=ffda SOS l= 12 e=2.446439 a=21.222222 remark: c=3
entropy-coded data: l=8481 e=7.528572 a=77.173467 #ff00=22
9 p=0x02147004 d=0: m=ffd9 EOI
...
Option -e can be used with option -f to extract all found images to disk. Images are written to the current directory with name AAA.jpeg, where AAA is the hexadecimal address of the position of the image in the file.
As stated at the beginning of this manual, this tool is very versatile when it comes to handling files. This will be explained now.
This tool reads files in binary mode. It can read files from disk, from standard input (stdin) and from "generated" files via the command line.
It can also partially read files (this is done with the cut operator).
If no file arguments are provided to this tool, it will read data from standard input (stdin). This way, this tool can be used in a piped chain of commands, like this:
oledump.py -s 4 -d sample.doc.vir | tool.py
When one or more file arguments are provided to this tool, it will read the files and process the content.
How the files are read, depends on the type of file arguments that are provided. File arguments that start with character @ or # have special meaning, and will be explained later.
If a file argument does not start with @ or #, it is considered to be a file on disk and the content will be read from disk.
If the file is not a compressed file, the binary content of the file is read from disk for processing.
Compressed files are solely recognized based on their extension: .zip and .gz.
If a file argument with extension .gz is provided, the tool will decompress the gzip file in memory and process the decompressed content. No checks are made to ensure that the file with extension .gz is an actual gzip compressed file.
If a file argument with extension .zip is provided and it contains a single file, the tool will extract the file from the ZIP file in memory and process the decompressed content. No checks are made to ensure that the file with extension .zip is an actual ZIP compressed file.
Password protected ZIP files can be processed too. The tool uses password 'infected' (without quotes) as default password. A different password can be provided using option --password.
Example:
tool.py sample.zip
To prevent the tool from decompressing .zip or .gz files, but to process the compressed file itself, use option --noextraction.
File arguments that start with character @ ("here files"), are read as text files that contain file arguments (one per line) to be processed.
For example, we take a text file with filename list.txt and following content:
sample-1.bin
sample-5.bin
sample-7.bin
When using this file (list.txt) in the following command:
tool.py @list.txt
the tool will process the following files: sample-1.bin, sample-5.bin and sample-7.bin.
A single @ character as filename is a here file read from stdin.
Wildcards are supported too. The classic *, ? and [] wildcard characters are supported. For example, use the following command to process all .exe and .dll files in the Windows directory:
tool.py C:\Windows\*.exe C:\Windows\*.dll
To prevent the tool from processing file arguments with wildcard characters or special initial characters (@ and #) differently, but to process them as normal files, use option --literalfilenames.
File arguments that start with character # have special meaning. These are not processed as actual files on disk (except when option --literalfilenames is used), but as file arguments that specify how to "generate" the file content.
File arguments that start with #, #h#, #b# or #e# are used to "generate" the file content.
Arguments that start with #c# are not file arguments, but cut operators (explained later).
Generating the file content with a # file argument means that the file content is not read from disk, but generated in memory based on the characteristics provided via the file argument.
When a file argument starts with # (and not with #h#, #b#, #e# or #c#), all characters that follow the # character specify the content of the generated file.
For example, file argument #ABCDE specifies a file containing exactly 5 bytes: ASCII characters A, B, C, D and E.
Thus the following command:
tool.py #ABCDE
will make the tool process data with binary content ABCDE. #ABCDE is not an actual file written on disk, but it is a notational convention to provide data via the command line.
Since this notation can not be used to specify all possible byte values, hexadecimal encoding (#h#) and BASE64 encoding (#b#) notation is supported too.
For example, #h#4142434445 is an hexadecimal notation that generates data ABCDE. Hexadecimal notation allows the generation of non-printable characters for example, like NULL bytes: #h#00
File argument #b#QUJDREU= is another example, this time BASE64 notation, that generates data ABCDE.
File arguments that start with #e# are a notational convention to use expressions to generate data. An expression is a single function/string or the concatenation of several functions/strings (using character + as concatenation operator).
Strings can be characters enclosed by single quotes ('example') or hexadecimal strings prefixed by 0x (0xBEEF).
4 functions are available: random, loremipsum, repeat and chr.
Function random takes exactly one argument: an integer (with value 1 or more). Integers can be specified using decimal notation or hexadecimal notation (prefix 0x).
The random function generates a sequence of bytes with a random value (between 0 and 255), the argument specifies how many bytes need to be generated. Remark that the random number generator that is used is just the Python random number generator, not a cryptographic random number generator.
Example:
tool.py #e#random(100)
will make the tool process data consisting of a sequence of 100 random bytes.
Function loremipsum takes exactly one argument: an integer (with value 1 or more).
The loremipsum function generates "lorem ipsum" text (fake latin), the argument specifies the number of sentences to generate.
Example: #e#loremipsum(2) generates this text:
Ipsum commodo proin pulvinar hac vel nunc dignissim neque eget odio erat magna lorem urna cursus fusce facilisis porttitor congue eleifend taciti. Turpis duis suscipit facilisi tristique dictum praesent natoque sem mi egestas venenatis per dui sit sodales est condimentum habitasse ipsum phasellus non bibendum hendrerit.
Function chr takes one argument or two arguments.
chr with one argument takes an integer between 0 and 255, and generates a single byte with the value specified by the integer.
chr with two arguments takes two integers between 0 and 255, and generates a byte sequence with the values specified by the integers.
For example #e#chr(0x41,0x45) generates data ABCDE.
Function repeat takes two arguments: an integer (with value 1 or more) and a byte sequence. This byte sequence can be a quoted string of characters (single quotes), like 'ABCDE' or an hexadecimal string prefixed with 0x, like 0x4142434445.
The repeat function will create a sequence of bytes consisting of the provided byte sequence (the second argument) repeated as many times as specified by the first argument.
For example, #e#repeat(3, 'AB') generates byte sequence ABABAB.
When more than one function needs to be used, the byte sequences generated by the functions can be concatenated with the + operator.
For example, #e#repeat(10,0xFF)+random(100) will generate a byte sequence of 10 FF bytes followed by 100 random bytes.
The cut argument (or cut operator) allows for the partial selection of the content of a file. This argument starts with #c# followed by a "cut-expression". Use this expression to "cut out" part of the content.
The cut-argument must be put in front of a file argument, like in this example:
tool.py #c#0:100l data.bin
With these arguments, tool.py will only process the first 100 bytes (0:100l) of file data.bin.
A cut argument is applied to all file arguments that follow it. Example:
tool.py #c#0:100l data-1.bin data-2.bin
With these arguments, tool.py will only process the first 100 bytes (0:100l) of file data-1.bin and the first 100 bytes file data-2.bin.
More than one cut argument can be used, like in this example:
tool.py #c#0:100l data-1.bin #c#0:200l data-2.bin
With these arguments, tool.py will only process the first 100 bytes (0:100l) of file data-1.bin and the first 200 bytes (0:200l) of file data-2.bin.
A cut-expression is composed of 2 terms separated by a colon (:), like this:
termA:termB
termA and termB can be:
- nothing (an empty string)
- a positive decimal number; example: 10
- an hexadecimal number (to be preceded by 0x); example: 0x10
- a case sensitive ASCII string to search for (surrounded by square brackets and single quotes); example: ['MZ']
- a case sensitive UNICODE string to search for (surrounded by square brackets and single quotes prefixed with u); example: [u'User']
- an hexadecimal string to search for (surrounded by square brackets); example: [d0cf11e0]
If termA is nothing, then the cut section of bytes starts with the byte at position 0.
If termA is a number, then the cut section of bytes starts with the byte at the position given by the number (first byte has index 0).
If termA is a string to search for, then the cut section of bytes starts with the byte at the position where the string is first found. If the string is not found, the cut is empty (0 bytes).
If termB is nothing, then the cut section of bytes ends with the last byte.
If termB is a number, then the cut section of bytes ends with the byte at the position given by the number (first byte has index 0).
When termB is a number, it can have suffix letter l. This indicates that the number is a length (number of bytes), and not a position.
termB can also be a negative number (decimal or hexademical): in that case the position is counted from the end of the file. For example, :-5 selects the complete file except the last 5 bytes.
If termB is a string to search for, then the cut section of bytes ends with the last byte at the position where the string is first found. If the string is not found, the cut is empty (0 bytes).
No checks are made to assure that the position specified by termA is lower than the position specified by termB. This is left up to the user.
Search string expressions (ASCII, UNICODE and hexadecimal) can be followed by an instance (a number equal to 1 or greater) to indicate which instance needs to be taken. For example, ['ABC']2 will search for the second instance of string 'ABC'. If this instance is not found, then nothing is selected.
Search string expressions (ASCII, UNICODE and hexadecimal) can be followed by an offset (+ or - a number) to add (or substract) an offset to the found instance. This number can be a decimal or hexadecimal (prefix 0x) value. For example, ['ABC']+3 will search for the first instance of string 'ABC' and then select the bytes after ABC (+ 3).
Finally, search string expressions (ASCII, UNICODE and hexadecimal) can be followed by an instance and an offset.
Examples:
This cut-expression can be used to dump the first 256 bytes of a PE file located inside the file content: ['MZ']:0x100l
This cut-expression can be used to dump the OLE file located inside the file content: [d0cf11e0]:
