m0shen · HackerTrans

2 เดือนที่ผ่านมาdiscuss

Ah I went through the same thing myself, but with Windows

https://moshen.net/posts/virtual_kvm/

2 ปีที่แล้วdiscuss

I thought it depended on libc by default?

2 ปีที่แล้วdiscuss

Nyancat terminal animation in perl: https://gist.github.com/moshen/1417991

On the slightly more useful side...

gotermimg to play gifs in the terminal: https://github.com/moshen/gotermimg

m0shen

2 ปีที่แล้วdiscuss

I see what you're saying. When I was writing scripts in `expect`, I didn't really ever try to automate tui programs. So, this could absolutely be a better way to script the the terminal as a ui as you said.

I'll certainly tuck it into my toolbox. Thanks :)

m0shen

2 ปีที่แล้วdiscuss

`expect` is absolutely geared towards scripting, as it's an extension of TCL. Though as far as getting a "current terminal view" `expect` has `term_expect`: https://core.tcl-lang.org/expect/file?name=example/term_expe...

m0shen

2 ปีที่แล้วdiscuss

Paperless supports OCR + full text indexing: https://docs.paperless-ngx.com/

As far as AI goes, not sure.

m0shen

2 ปีที่แล้วdiscuss

I love seeing how many people wrote one of these. I've written 2:

One to learn golang that plays gifs - https://github.com/moshen/gotermimg

And one in perl - https://github.com/moshen/Image-Term256Color

m0shen

2 ปีที่แล้วdiscuss

Yes, you're correct. I had in mind the 2022 date of the update. It also looks like you've updated the article again today! Thanks

m0shen

2 ปีที่แล้วdiscuss

This article was written when cosmopolitan: https://justine.lol/cosmopolitan/index.html was v2. v3 has since been released: https://justine.lol/cosmo3/ .

https://cosmo.zip/ Also contains a python build https://cosmo.zip/pub/cosmos/bin/python , though I'm not sure if the problems mentioned in the article are solved.

m0shen

2 ปีที่แล้วdiscuss

Anything that converts to PDF/A will strip Javascript

m0shen

2 ปีที่แล้วdiscuss

I've updated this script with some single-file cli numbers, which are (as expected) not good. Mostly just comparing python startup time for that.

    make
    sqlite3 < analyze.sql
    file_avg              python_avg         python_x_times_slower_single_cli
    --------------------  -----------------  --------------------------------
    0.000874874856301821  0.179884610224334  205.611818568799
    file_avg            python_avg     python_x_times_slower_bulk_cli
    ------------------  -------------  ------------------------------
    0.0231715865881818  0.69613745142  30.0427184289163

m0shen

2 ปีที่แล้วdiscuss

Absolutely, and honestly in a non-interactive ingestion workflow you're probably doing multiple checks anyway. I've worked with systems that call multiple libraries and hand-coded validation for each incoming file.

Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.

m0shen

2 ปีที่แล้วdiscuss

Good to know! Thank you. I'll definitely be trying it out. Though, I might download and hardcode the model ;)

I also appreciate the use of ONNX here, as I'm already thinking about using another version of the runtime.

Do you think you'll open source your F1 benchmark?

m0shen

2 ปีที่แล้วdiscuss

My little script is trying to identify in bulk, at least by passing 165 file paths to `magika`, and `file`.

Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.

Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.

m0shen

2 ปีที่แล้วdiscuss

That doesn't appear to be a valid link. Try building `file` from source and using the provided default magic database.

m0shen

2 ปีที่แล้วdiscuss

Thank you for the release! I understand you're just getting it out the door. I just hope to see it delivered as a native library or something more reusable.

I did try the python cli, but it seems to be about 30x slower than `file` for the random bag of files I checked.

I'll probably take some time this weekend to make a couple of issues around misidentified files.

I'll definitely be adding this to my toolset!

m0shen

2 ปีที่แล้วdiscuss

At multiple points in my career I've been responsible for apis that accept PDFs. Many non-tech savvy people seeing this, will just change the extension of the file they're uploading to `.pdf`.

To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.

m0shen

2 ปีที่แล้วdiscuss

This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version.

FILE_45:

    ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk
    ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block

m0shen

2 ปีที่แล้วdiscuss

Yeah, there is this line:

    By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale.

Which implies a production-ready release for general usage, as well as usage by security researchers.

m0shen

2 ปีที่แล้วdiscuss

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9...

    hyperfine ./magika.bash ./file.bash
    Benchmark 1: ./magika.bash
      Time (mean ± σ):     706.2 ms ±  21.1 ms    [User: 10520.3 ms, System: 1604.6 ms]
      Range (min … max):   684.0 ms … 738.9 ms    10 runs
    
    Benchmark 2: ./file.bash
      Time (mean ± σ):      23.6 ms ±   1.1 ms    [User: 15.7 ms, System: 7.9 ms]
      Range (min … max):    22.4 ms …  29.0 ms    111 runs
    
    Summary
      './file.bash' ran
       29.88 ± 1.65 times faster than './magika.bash'

comments