[X] My Assistant

Thanks for these changes.(IMG:[invalid] style_emoticons/default/smile.gif)

That addon can for whatever reason not handle https at all.

It *should* work. Both methodPOST_DATA and methodPOST_URL send the request with:

CODE

var flags = Components.interfaces.nsIWebNavigation.LOAD_FLAGS_NONE;
imagesearchoptions.tabBr.loadURIWithFlags(siteURL, flags, null, null, postData);

And the tabBr in there is an FF worker instance (or whatever that's called now). It is generated as:

CODE

myTabRef = getBrowser().selectedTab;
imagesearchoptions.tabBr = getBrowser().getBrowserForTab(myTabRef);

So tabBr.loadURIWithFlags is pretty much a XUL method, and therefore HTTP vs. HTTPS should not matter. Yet, maybe this is another mixed content bug. I'd try to use LOAD_FLAGS_IS_LINK (instead of LOAD_FLAGS_NONE) which should (theoretically) not fall into mixed content problems.

I have not written an add-on in years though. Am not even sure whether you can load a signed add-on after changing its code... sorry.

This post has been edited by blue penguin: Mar 20 2017, 03:24

It *should* work.
I am not even sure whether you can load a signed add-on after changing its code... sorry.

Oh I'm not worried about running it modified, as I'm already doing that fine.

About the *should* though, I'm running a Fx version that belongs into a museum (v24) for reasons so maybe that's why it fails.
As for the error, it was something about "the server did not understand the request something something".

I'll see if changing that link thing changes anything, and while at that, write down the exact error I'm getting.


Edit: nope, didn't work, and here's the result:
Bad Request
Your browser sent a request that this server could not understand.

This post has been edited by BloodyRain2k: Mar 20 2017, 14:22

That addon can for whatever reason not handle https at all.

iqdb and saucenao works fine with https in that extension, for example.

I'm running a Fx version that belongs into a museum (v24) for reasons so maybe that's why it fails.

Now, this will be a wild guess but didn't those FF versions had their CA certs inside their own installation folders? In other words maybe the HTTPS is failing simply because the CA certs are old (and e.g. let's encrypt would not be in there).

Bad Request
Your browser sent a request that this server could not understand.

Nevertheless, I'd try capturing the request and actually see how it looks. Wireshark, or even a simple TamperData should do it. The check if the target of the request (I did think it was saucenao but see below) can deal with it (or can deal with HTTPS at all).

iqdb and saucenao works fine with https in that extension, for example.

Heh, I was checking this against saucenao. That explains a few things.

I have a reverse image search extension I made for myself that should work on Firefox (it's for Chrome but Firefox recently added support for Chrome extensions, so it should be okay), but the stable branches of Firefox (stable and beta) won't let you install custom extensions unless a) they're signed and/or b­) they've been uploaded to their store, and I have absolutely no intention to put up with their imbecility. If you ever switch to a recent-ish version of Firefox Developer Edition or Firefox Nightly (those let you install unsigned extensions) and you're interested, let me know.

I'm running a Fx version that belongs into a museum (v24) for reasons so maybe that's why it fails.

Try [www.palemoon.org] Pale Moon. It's a Firefox fork that keeps the pre-Australis UI, but get security and feature updates. A lot of Add-ons work, but not all (all that work on 24 should work, as that's when PM split off and became proper fork).

unless a) they're signed

[developer.mozilla.org] https://developer.mozilla.org/en-US/Add-ons...ce#web-ext_sign

[developer.mozilla.org] https://developer.mozilla.org/en-US/Add-ons...ce#web-ext_sign

Looks like you still need to register to their Addons API to do that, so I'll pass

Nevertheless, I'd try capturing the request and actually see how it looks.

Tried that and as far as I can "read" wireshark is there only a slight difference:
At some point kinda like 20-25% in the plugin's datastream seem to have a dent, up to that point could I put them side by side and they looked exactly the same but then suddenly the server throws in a smaller TLS v1 Application Data package, it's only 539 byte compared to all others before with 1514 and the next after is an Ancrypted Alert.
After that the mostly look the same again, I assume Fx just keeps uploading just to be told it was all rejected anyways or something.

Another difference is at the beginning, the pattern is exactly the same for both but one single exchange is different for the plugin, the 2nd "Server Hello" packet isn't "Spurious" but "Out-of-order".

Probably not much of a help given I can't make much out of what I'm looking at but it doesn't scream anything obvious at me : /


I also tried it with IQDB and SauceNao with https because I actually didn't expect that to work but surprisingly it did, so something broke only EH and TinEye, funnily enough kinda around the same time.

Looks like you still need to register to their Addons API to do that, so I'll pass

Yes. Seems like it's easier through [addons.mozilla.org] https://addons.mozilla.org/en-US/developers...upload-unlisted , only needs AMO account, which can be automatically created from firefox-sync account.

so something broke only EH and TinEye, funnily enough kinda around the same time.

TinEye works for me too. Both URL and file upload.
Never used E-H search in addon, but looks like default settings contain old subdomains that don't exist anymore. Did you updated them? Same with file-upload url for google, broken long ago.

Never used E-H search in addon, but looks like default settings contain old subdomains that don't exist anymore. Did you updated them? Same with file-upload url for google, broken long ago.

Already updated because the subdomains broke long ago waaay before the HTTPS issue.
Google weirdly enough did not need any fixing despite supposedly suffer the same problem according to the addon reviews.

Might be really just the certificates but that'd throw the question why the addon uses different ones than Fx itself.
And how one would go about fixing that... I can only fix JS and even there only moderate level, not high end like YT's BS of not having a key for toggling Annoytations : /

This post has been edited by BloodyRain2k: Mar 21 2017, 10:36

I would really like to see the request itself. As far as I'm aware the EH image search uses a GET (so it is methodPOST_URL and not methodPOST_DATA). Something of the lines:

https://e-hentai.org/?f_shash=4311531d6b669...dc3340243ab9190

And this works fine

CODE

curl 'https://e-hentai.org/?f_shash=4311531d6b669c1e0bd1787ccdc3340243ab9190' | grep 'Shijou Mako'

If you want to use the similarity search function you should upload the actual image (as in, multipart form data upload) and watch where the site redirects you. If you use the SHA1 directly you're basically performing a hash-based search (as far as I understand it, the similarity search thingamagic takes an image as input and returns a set of SHA1 hashes of images that are considered similar to the input image, if you skip the upload and go directly to the SHA1 hash you're basically skipping the similarity search)

Just a random guess, but are you maybe using the old https://e-hentai.org/upload/image_lookup.php instead of the new https://upload.e-hentai.org/image_lookup.php ?

To test this I've made a script that uses the EH image search (including similarity search!), and it works fine.

CODE

#!/bin/sh

UPLOAD_URL=https://upload.e-hentai.org/image_lookup.php

if test "x$1" != "x" -a -r "$1"
then
    base=`basename "$1"`
    curl -i -F "sfile=@\"$1\";filename=\"$base\";type=image/jpeg" \
            -F "fs_similar=1" $UPLOAD_URL
fi

Usage: script.sh file.jpg (too lazy to make it work for PNG/GIF)

The interesting part is that it works with both: https://upload.e-hentai.org/image_lookup.php (HTTPS) and http://upload.e-hentai.org/image_lookup.php (HTTP).

The POST data generated is simply a multipart/form-data, it may be a little complex to parse/build but browsers already do that. And, since HTTP works you can capture that and look at how it was constructed.

P.S. A complete script (png/gif support, command line parsing and shit) can be found on github: [github.com] https://github.com/neptunepenguin/eh_script...arch_eh_file.sh

This post has been edited by blue penguin: Mar 22 2017, 01:45

The interesting part is that it works with both: https://upload.e-hentai.org/image_lookup.php (HTTPS) and http://upload.e-hentai.org/image_lookup.php (HTTP).

The POST data generated is simply a multipart/form-data, it may be a little complex to parse/build but browsers already do that. And, since HTTP works you can capture that and look at how it was constructed.

I'm very sure it's not possible to cause a 400 by querying a 404 url and I already mentioned that I updated the url because the old stopped working long ago.
After that it worked fine until EH started enforcing HTTPS as in that I can NOT use http://upload.e-hentai.org/image_lookup.php because it instantly redirects me to https://upload.e-hentai.org/image_lookup.php so I can not compare how the browser constructs the form data vs the plugin because everything's encrypted when I look at it in Wireshark.
Atleast did I not see a single bit of readable data. Maybe it has a decoder but hell if I know.

And the GET part is just the result, after uploading an image the site redirects you to a simple GET url using the hash it came up with.

As for that script, I'd need to change my OS before I could make use of that : /

This post has been edited by BloodyRain2k: Mar 22 2017, 03:27

I can NOT use http://upload.e-hentai.org/image_lookup.php because it instantly redirects me to https://upload.e-hentai.org/image_lookup.php

You can

Just change the https:// to http:// in the script above and it works, I've tested that. The GET on that HTTP url gives you an HTTP 301 to the https:// url, but a POST to that url gives you an HTTP 302 to the search result. Both in the Location: header.

And the GET part is just the result, after uploading an image the site redirects you to a simple GET url using the hash it came up with.

But the has *is* the similarity search. It is a from of a phash (although not phash itself) so it is fine to give and image in the POST and then reuse the hash that comes out for further similarity searches.

As for that script, I'd need to change my OS before I could make use of that : /

I have absolutely no clue how to write code for MS Windows, sorry.

This post has been edited by blue penguin: Mar 22 2017, 03:36

Just change the https:// to http:// in the script above and it works, I've tested that.

Tried that by changing the form submission url via local editing and it did gave me a redirect but it did nothing about the salad result in Wireshark.

I gave looking into the traffic with HTTP Fox another go, tried that long ago and I wasn't even able to find any data at all back then but surprisingly I did this time.
And I didn't find a difference in the way the plugin constructs the form data vs Fx itself, aside of a minor difference: the plugin does not seem to add a \r\n after the binary data making the next seperator line cling to the end of the binary data stream.
That was not the case in Fx's construction where every single seperator was actually seperated.

So that might be the part where the plugin screws up, the problem is that I have a hard time finding out where it joins the binary stream with the plain text seperators in the script.


Edit: ok didn't fix it either, found the right place to add that \r\n and it's still borked :<

The only other difference I can see in HTTP Fox is that Fx's stream starts with the file

CODE

-----------------------------77882627917728
Content-Disposition: form-data; name="sfile"; filename="55b1fe39dc372.jpg"
Content-Type: image/jpeg

And the top line above the POST data field shows

CODE

Type: multipart/form-data; boundary=-----------------------------77882627917728

But the plugin's line above the data field is blank and it instead starts with the full header

CODE

Content-Type: multipart/form-data; boundary=----------31337isopost
Content-Length: 127661

------------31337isopost
Content-Disposition: form-data; name="sfile"; filename="upload.png"
Content-Type: "image/png"

Edit 2: after staring at these streams side by side for a while I'm starting to think that this difference might be the plugin messing up the stream and putting the leading content type and length into the actual data stream instead of the header of the request.
Because the proper one shows these in the headers while the broken one doesn't seem to have intact headers, either way does HttpFox not wanna let me look at what they are.

Edit 3: uhm ok wtf? Now it started working for whatever reason....
Well, since I followed my bad habits of making 3 changes between retries it was one of the following or all of them:
- cleaning up " mixed with ' strings (which I honestly just did because it irritated the fuck out of me)
- changing the boundary to one that worked with Fx (maybe they have a minimum length?...)
- converting the whole .js to UTF8 (wouldn't be the first time that this broke or unbroke a script...)

oh wait, just remembered a #4: the header portion of the data was just delimited with \n's so I changed these to \r\n's because "meh it's fucked, can't fuck it up more". Might be that this is what fixed it as this section appears to not end up as a proper header but just at the beginning of the stream and is then treated like a header and maybe the \n's just fucked everything up.

Whatever, works again and I'll take it.

This post has been edited by BloodyRain2k: Mar 22 2017, 04:35

CODE

Content-Type: multipart/form-data; boundary=----------31337isopost
Content-Length: 127661

------------31337isopost
Content-Disposition: form-data; name="sfile"; filename="upload.png"
Content-Type: "image/png"

That appears to be correct. After that there should be the image bytes and the mime for the options ticked (similarity, cover, expunged)

Here is a sample of me running my script over HTTP:

CODE

0000   50 4f 53 54 20 2f 69 6d 61 67 65 5f 6c 6f 6f 6b  POST /image_look
0010   75 70 2e 70 68 70 20 48 54 54 50 2f 31 2e 31 0d  up.php HTTP/1.1.
0020   0a 48 6f 73 74 3a 20 75 70 6c 6f 61 64 2e 65 2d  .Host: upload.e-
0030   68 65 6e 74 61 69 2e 6f 72 67 0d 0a 55 73 65 72  hentai.org..User
0040   2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 35  -Agent: curl/7.5
0050   33 2e 31 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a  3.1..Accept: */*
0060   0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68  ..Content-Length
0070   3a 20 35 37 35 34 38 0d 0a 45 78 70 65 63 74 3a  : 57548..Expect:
0080   20 31 30 30 2d 63 6f 6e 74 69 6e 75 65 0d 0a 43   100-continue..C
0090   6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 6d 75 6c  ontent-Type: mul
00a0   74 69 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61  tipart/form-data
00b0   3b 20 62 6f 75 6e 64 61 72 79 3d 2d 2d 2d 2d 2d; boundary=-----
00c0   2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
00d0   2d 2d 2d 32 31 62 61 34 35 32 33 39 65 66 33 62  ---21ba45239ef3b
00e0   64 36 65 0d 0a 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d  d6e....---------
00f0   2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ----------------
0100   2d 32 31 62 61 34 35 32 33 39 65 66 33 62 64 36  -21ba45239ef3bd6
0110   65 0d 0a 43 6f 6e 74 65 6e 74 2d 44 69 73 70 6f  e..Content-Dispo
0120   73 69 74 69 6f 6e 3a 20 66 6f 72 6d 2d 64 61 74  sition: form-dat
0130   61 3b 20 6e 61 6d 65 3d 22 73 66 69 6c 65 22 3b  a; name="sfile";
0140   20 66 69 6c 65 6e 61 6d 65 3d 22 31 34 38 35 39   filename="14859
0150   39 33 38 34 31 38 32 34 2e 6a 70 67 22 0d 0a 43  93841824.jpg"..C
0160   6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 69 6d 61  ontent-Type: ima
0170   67 65 2f 6a 70 65 67 0d 0a 0d 0a ff d8 ff e0 00  ge/jpeg.........
0180   10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff  .JFIF...........
0190   db 00 43 00 08 06 06 06 06 06 08 06 06 08 0c 08  ..C.............
01a0   07 08 0c 0e 0a 08 08 0a 0e 10 0d 0d 0e 0d 0d 10  ................
01b0   11 0c 0c 0c 0c 0c 0c 11 0c 0c 0c 0c 0c 0c 0c 0c  ................
(a lot of bytes)
e0b0   51 45 00 14 51 45 00 14 51 45 00 14 51 45 00 14  QE..QE..QE..QE..
e0c0   51 45 00 14 51 45 20 0a 28 a2 80 0a 28 a2 80 0a  QE..QE .(...(...
e0d0   e2 bc 75 88 af f4 0b 92 38 17 26 36 6f 4d c5 08  ..u.....8.&6oM..
e0e0   fe 55 da d7 19 f1 11 09 b3 d3 a6 03 22 2b b4 27  .U.........."+.'
e0f0   1e f4 17 4b e3 89 d4 1a 8c d3 c9 e2 98 78 a9 67  ...K.........x.g
e100   6a 12 8a 6e 68 cd 49 76 02 69 b9 a0 9a 69 a4 52  j..nh.Iv.i...i.R
e110   14 9a 69 34 94 1c d0 52 0a 28 a2 80 3f ff d9 0d  ..i4...R.(..?...
e120   0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  .---------------
e130   2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 32 31 62 61 34  -----------21ba4
e140   35 32 33 39 65 66 33 62 64 36 65 0d 0a 43 6f 6e  5239ef3bd6e..Con
e150   74 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e  tent-Disposition
e160   3a 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d  : form-data; nam
e170   65 3d 22 66 73 5f 73 69 6d 69 6c 61 72 22 0d 0a  e="fs_similar"..
e180   0d 0a 31 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  ..1..-----------
e190   2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 32  ---------------2
e1a0   31 62 61 34 35 32 33 39 65 66 33 62 64 36 65 2d  1ba45239ef3bd6e-
e1b0   2d 0d 0a                                         -..

0d 0a are \r\n. And note that the multipart is parsed using the defined bounary= but each part of the multipart does not have its own content length. Above are two parts, one with the file itself (quite a long stream of bytes, it starts with a JPEG magic number and then goes to JFIF and goes and goes on), and another part submitting fs_similar=1.


EDIT: Damn, I was too slow to post. But yeah, these things can be hairy. Crossing fingers that it will keep working!

EDIT2:

- changing the boundary to one that worked with Fx (maybe they have a minimum length?...)

Yes, it does.

The Content-Type field for multipart entities requires one parameter, "boundary", which is used to specify the encapsulation boundary. The encapsulation boundary is defined as a line consisting entirely of two hyphen characters ("-", decimal code 45) followed by the boundary parameter value from the Content-Type header field.

[www.w3.org] RFC 1341

SO may have been that too.

This post has been edited by blue penguin: Mar 22 2017, 04:47

oh wait, just remembered a #4: the header portion of the data was just delimited with \n's so I changed these to \r\n's because "meh it's fucked, can't fuck it up more". Might be that this is what fixed it as this section appears to not end up as a proper header but just at the beginning of the stream and is then treated like a header and maybe the \n's just fucked everything up.

HTTP headers lines are indeed required to be separated with CRLF rather than just LF.

[www.w3.org] https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html