Somebody sent me a encrypted PDF to ask me help decrypt, I tried pdfcrypt, it got error. and when used pdfcrack, got the following warning messages,
WARNING: O-String != 32 Bytes: 33
WARNING: U-String != 32 Bytes: 33
After studied the source code of iTextSharp, I noticed it is a bug of iTextSharp in SetupGlobalEncryptionKey method of the PdfEncryption class,
misused the this.ownerKey and ownerKey and we can only find this bug when the PDF has strange length of ownerkey/userkey, for example 33, not 32.
this.documentID = documentID;
//this.ownerKey = ownerKey;
Array.Copy(ownerKey, 0, this.ownerKey, 0, 32);
this.permissions = permissions;
// use variable keylength
mkey = new byte[keyLength / 8];
//fixed by ujihara in order to follow PDF refrence
md5.Reset();
md5.BlockUpdate(userPad, 0, userPad.Length);
//md5.BlockUpdate(ownerKey, 0, ownerKey.Length);
md5.BlockUpdate(this.ownerKey, 0, this.ownerKey.Length);
Then I review iText 5.1.1 and find it has the same bug:
this.documentID = documentID;
//this.ownerKey = ownerKey;
System.arraycopy(ownerKey, 0, this.ownerKey, 0, 32);
this.permissions = permissions;
// use variable keylength
mkey = new byte[keyLength / 8];
// fixed by ujihara in order to follow PDF reference
md5.reset();
md5.update(userPad);
//md5.update(ownerKey);
md5.update(this.ownerKey);