Reverse Engineer Amazon's Whispersync(ptbrowne.github.io)

124 pointsby ptbrowne6 tahun yang lalu35 comments

ptbrowne.github.io

https://ptbrowne.github.io/posts/whispersync-reverse-engineering/

36 comments

fprog6 tahun yang lalu

Loved section 1.5.1, "Bytes in color". The author wrote code to help them understand a custom binary format by coloring the bytes based on their value, making patterns in the format visually obvious. The structure of the binary becomes much more clear when visualized this way. That's a handy approach I've mentally bookmarked for the future.

p_l6 tahun yang lalu

I was in a team that, among other things, was working on integrating 3d visualisation of binary files in a hex editor with the plan of making a reverse engineering tool (mainly for security).

It's a very powerful method.

LargoLasskhyfv6 tahun yang lalu

Like CantorDust by Christopher Domas@Batelle?

https://www.youtube.com/watch?v=4bM3Gut1hIk really blew my mind when I saw it.

Btw. about 8 years later they finally delivered, by releasing it as plug-in for Ghidra! Yay!

https://github.com/Battelle/cantordust

p_l6 tahun yang lalu

I believe CantorDust was the direct inspiration for our code (I wasn't exactly involved in the editor project though, so hard to call it "our").

The moment you mentioned "CantorDust" it was like a giant lightbulb filled with memories of discussions :)

rckoepke6 tahun yang lalu

Does that have the 3D visualizations shown in Christopher's 2013-2014 talks? Both the github readme and the associated blogpost from July make it look like it only includes 2D visualizations. There are fewer "view" buttons on the right toolbar than the demo had. I should answer my own question by simply trying it out, but I'm not quite sure when I'll have a moment to get around to it.

[0]: https://inside.battelle.org/blog-details/battelle-publishes-...

LargoLasskhyfv6 tahun yang lalu

I don't know. Just discovered it because I searched for the link to the video I had in mind.

edit: Hrrmpf. "Zu früh gefreut/rejoiced too early". Scanning the git repository it doesn't look like it would.

But can't/won't install/test it atm. Maybe over the weekend :(

rckoepke6 tahun yang lalu

This is my favorite version of it:

https://www.youtube.com/watch?v=4bM3Gut1hIk&list=PLUyyOw61zx...

Some people have taken this concept very far. Batelle's CantorDust (was?) a pretty fully developed 3D visualization software for binary packages utilizing many different views and filters with convenient UX for selecting blocks of code graphically and zooming in on the corresponding hex, or vice-versa.

Edit: I looked very hard for this software about 6-12 months ago and couldn't find anything...it appeared to be vaporware. However, 'LargoLasskhyfv posted in this thread tree at the exact same time I did and noted that it was just released about 3 months ago as a plugin for Ghidra. I'm very excited to play around with it.

syoc6 tahun yang lalu

You might be interested in https://github.com/sharkdp/hexyl

> hexyl is a simple hex viewer for the terminal. It uses a colored output to distinguish different categories of bytes (NULL bytes, printable ASCII characters, ASCII whitespace characters, other ASCII characters and non-ASCII).

sdoering6 tahun yang lalu

Wow. This was a fun read. I would have been lost - but loved for somebody to explain it in a way I was able to grasp with my level of technically understanding.

Thanks to the author - I would love for this data to be in an open format, as I would love to do some kind of personal analytics on that (same with Audible data actually).

On a different tangent, I really love the ease in the Kindle/Audible ecosystem to switch between the same content in different media. Reading on the kindle app, switching to Audible in the car and back to kindle in the evening in bed. That really makes reading/book consumption great.

kilroy1236 tahun yang lalu

I'm very interested in this. I read a lot of books that aren't from Amazon on my Kindle. I do a lot of highlighting, which I like to export to readwise.

In order to do this I have to plug in my Kindle. Mount it. Find the clippings file. Upload the file to readwise.

The best I came up with was a script that detects when I plug-in my Kindle and opens the finder window and opens the readwise page. Pretty lame.

ptbrowne6 tahun yang lalu

Unfoturnately, Amazon does not sync highlights on books that have not been purchased on Amazon. This is also why Readwise cannot automatically sync those highlights.

https://help.readwise.io/article/67-some-of-my-kindle-books-...

sushicalculus6 tahun yang lalu

Could you post this script to somewhere? Gist, codepen, etc?

I want to see if can improve it. There might be a way to have the terminal auto-copy the clippings file

farss6 tahun yang lalu

Here's a similar script I wrote. Would be glad for feedback.

https://github.com/wneuheisel/Kindle-Notes-Backup

kilroy1236 tahun yang lalu

Oh awesome. I see you spent much longer than my 30 minutes on this. I'm going to take a good look at this later.

ibatindev6 tahun yang lalu

Very well explained!

I though the text was encrypted but then I realized it was just french. Damn.

adolph6 tahun yang lalu

https://en.wikipedia.org/wiki/Code_talker

tshrjn6 tahun yang lalu

readwise kinda helps do something similar?

ptbrowne6 tahun yang lalu

Hi, thanks I did not know Readwise. I signed up for an account to check how it works. It uses a browser extension to sync the highlights instead of using the API. It can sync automatically but your browser must stay open (which is OK IMHO). Readwise can sync much more sources also.

Thanks for introducing me to Readwise, I might use it :)

bredren6 tahun yang lalu

Still a valuable exploration!

That said, this suggestion and it’s method of avoiding direct interaction, but instead using rendered content in browser is another reminder of how powerful browser extensions are.

They can also make additional requests using the user’s ip address.

Would pulling data this way also seems less likely to be a TOS issue?

Reason being that you are reading content you are manipulating or displaying in a browser “after” amazon does their own highlighting.

mmastrac6 tahun yang lalu

This could also be an interesting way to liberate my Kindle books from the Amazon ecosystem.

rektide6 tahun yang lalu

would this work if Amazon was using cert pinning? cert pinning basically seems like a way to make apps able to resist user's having any freedom or power. the author here installs a custom CA to man-in-the-middle. now more & more apps use cert pinning, which would block that, I believe, & keep the user from being able to reverse engineer & probe.

imo the forces in favor of securing things have done the users great great damage. who are we securing things for? imo the security world needs to re-orient, & begin to harken to RFC 8890 The Internet is For End Users[1], allowing users some means to see how their systems are behaving. "secure" standards like cert-pinning favor giant companies & totalistic visions of security, taking from computing the person & personal. cert-pinning is a good layer of defense, but it takes away user affordances that must be added back.

[1] https://tools.ietf.org/html/rfc8890

ptbrowne6 tahun yang lalu

You're right, this technique would not work if Amazon had used cert pinning. I haven't yet tried to bypass a cert pinning app, but it seems that frameworks like Frida could help : https://medium.com/@ved_wayal/hail-frida-the-universal-ssl-p....

Godel_unicode6 tahun yang lalu

Or just open up the apk in e.g. apk studio and replace the bundled cert with the one from your mitm proxy. Effectively, that'll repin the app to your proxy specifically. There are more exotic pinning methods that this won't solve (the forever cat-and-mouse game) but it's good for many apps.

rektide6 tahun yang lalu

It's a problem that, alas, must be tackled somewhat uniquely on every platform, where-as previously users were free to manage their own Certificate Authorities as they wished. Now users are safe & secure from themselves. :/ Admittedly there were a lot of problems with poorly managed CA stores, users being abused, but cert-pinning feels like such a drastic overreach in preventing any form of user control.

Techniques like this "rebuild your apk" are interesting & good to have, but every OS needs it's own bag of tricks. It's probably not a total show-stopper, but news like today's that Windows 10 will only install signed driver software (an admittedly niche-ish case), the closed Apple store,... there's a lot of places side-loading is not an option. Do those apps get a pass, get to be complete black-box software that we the users have zero ability to look at or understand?

[1] https://www.zdnet.com/article/windows-10-will-start-blocking...

Godel_unicode6 tahun yang lalu

You're making this sound so much harder than it is, though. None of the techniques for bypassing the security enhancements is difficult, nor are they undocumented. Who are these hobbyists who want to reverse engineer the whispersync protocol but can't follow a simple tutorial to swap out a CA?

rektide6 tahun yang lalu

you've listed one platform (of many) where it's possible to (in some cases where there is a .cert or .pem file sitting in an apk), updated that apk pkg, & then use side-loading (not available on os'es) to install that hacked package.

there's a number of caveats to your steps, it will definitely not be this easy in all cases. and i while this wasn't all that difficult, it's still a situation where the OS is actively working to prevent the user from being able to understand their system. and most platforms don't provide even this much of an affordance, of opening, rebuilding, & side-loading packages.

Godel_unicode6 tahun yang lalu

You're making this sound so much harder than it is.

rektide6 tahun yang lalu

I sincerely do not think so & I genuinely & I think appropriately fear that humankind is quickly approaching an era where they have no power to understand what their softwares are doing.

This is already the case on iphone, I think. You have only outlined the most bare-basic case on Android. if certs were embedded this would not work. Who knows about osx & windows. It's much much much much harder & in many cases already impossible. The security people seem only to want to guard the applications, continually at the expense of the user. Who will fight for the user?

I think I am on the bead here.

Godel_unicode6 tahun yang lalu

I'm aware that you think that. What I'm struggling with is, in the face of all the evidence in this thread and others, why. This stuff is easy. It's well documented. There are YouTube videos literally (like, literally literally) walking through every step of the process. The fact that you specifically do not know how to do it right now doesn't mean it can't be done, or even that it's hard. The fact that your argument is "who knows about..." and not a specific example is a big clue that you might be baselessly worried about the sky falling. You claim that users are walled off, yet you haven't produced a single example of that being true.

The fact that the cost to exploit end user devices in an irreversible, hard to detect way has been raised is a real benefit to the user. The fact that mitm of banking apps is very difficult without protracted user interaction is a real benefit to the user. There are, conservatively, hundreds of millions of users having their lives made better by security people fighting for them every day to make their devices safer to use in a hostile world.

Do you also object to TLS? What about centrally generated electricity?

rektide6 tahun yang lalu

If anyone has been citing no examples, imo, it's you. I have elaborated & elaborated & elaborated, & you have said nothing to contend with other than 'it's easy'. I disagree. I've tried to talk to that at length. You've left everything but your one happy easy path to modifying a cert untouched, not commenting on a single one of the difficulties I've raised.

I think TLS was mis-designed for an in-appropriate & indecent form of security that does not give permissions to the most important actor.

Leaving the CA store on the hard drive, free to be modified by the user & sys-ops in a judicious careful manner was respectful of systems agency, giving a wide range of respect to different ways systems might need to be operated. Now, there are very few permissible ways to modify behavior. The system has closed down, locked down, become less programmable, less configurable. This advancement of the unmalleable is, imo, notable, prominent, progressing (on a wide variety of fronts), & obviously bad.

Godel_unicode6 tahun yang lalu

You provided one specific example, I explained how trivial that example was to hack around. Since then there's been nebulous pearl-clutching about how security is bad. If you had provided specific examples of how you think things are difficult, I would have explained how wrong you are. Unpinning certs is easy. Removing embedded certs is easy. Hooking verification functions is easy. If you possess the hardware, you win.

rektide6 tahun yang lalu

Most of the tool I've seen requires jailbroken/rooted devices. So only a handful of devices are even capable of these techniques. The latest iOS 14 has not yet been broken. Increasingly few Android devices can be unlocked or rooted, have been properly exploited into this mode.

I've said numerous times I thought your example was rosy. Removing embedded certs from Windows, OSX, iOS, Android programs seems like, in many cases, it could be difficult, as I don't feel like just removing the content is going to change app code that may be asking explicitly for pinning. Hooking verification functions as some of the xposed frameworks do seems viable, but again, this all is contingent on users having a level of access that most devices actively work to prevent, that requires the device to be exploited to achieve.

The security climate is in fact quite chilling. Everything you say is possible, but it requires increasingly rare access to the device, and increasing advanced levels of reverse engineering. The days when one could add their own CA to the store & intercept- those made sense- and they are long gone. The app makers, the OS makers, are securing devices against this kind of user-level control.

[deleted]6 tahun yang lalu

yellow_lead6 tahun yang lalu

This module works well too. https://github.com/Fuzion24/JustTrustMe

rektide6 tahun yang lalu

Amazing tool, neat trick. It requires xposed framework though, which is an interception framework for Android that requires root access to install.

Less and less devices seem to have root, and almost no devices these days seem to allow unlocking the bootloader, which is even better.

It feels like users are getting kicked further & further & further out. We are less & less able to have any chance to understand what computing is, & this deeply deeply hurts us, I tend to believe.

bredren6 tahun yang lalu

Anyone know if there been any action around reverse engineering the hardware connectivity?

I’ve wanted a clicker page turner for a while. It would be amazing if the usb interface was figured out.