[ / / / / / / / / / / / / / ] [ dir / agatha2 / asmr / ausneets / cafechan / doomer / loomis / strek / sw ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

8chan Cup Knockout Stage - Friday, January 18 at 08:00 p.m. GMT
Winner of the 65rd Attention-Hungry Games
/cure/ - Your obscure board for medical-tan appreciation

December 2018 - 8chan Transparency Report
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,542 items

Python 3 update just happened! Check v335 release post for update instructions!

Current big job: Client API


YouTube embed. Click thumbnail to play.

efc26a  No.11355[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v336.tar.gz

Last week, v335, was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, updating to anything newer is back to normal.

I had a good week. A bunch of py3-related bugs are fixed, and I improved some heavy-load throttling.

py3 fixes

Overall, last week's update went well. CPU and memory use and overall jank is down for all users, and afaik there weren't any permanent disasters in the update. This was a big relief for me.

Post too long. Click here to view the full text.


File: 3f8a8d22364b1de⋯.jpg (111.13 KB, 400x400, 1:1, 1394958218797.jpg)

d5679d  No.9327[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

451 posts and 91 image replies omitted. Click reply to view.

a09efa  No.11352

Can I use a third party software to delete duplicate files on hydrus, like letting it scan my database and delete the duplicates, or is it gonna corrupt my db ? Can I manually delete pics from my db without corrupting it or it is a must to use the built-in feature ?


03212e  No.11358

File: 5e9338de4702586⋯.jpeg (629.43 KB, 1200x1600, 3:4, 502F6E22-47B2-4675-9DA6-5….jpeg)

With pixiv scraping still broken I'm looking for other alternatives. Is there any way to get images off of https://booru.io ? It's got a lot of images that even danbooru/gelbooru doesnt, (pic related) but neither simple download (embed) or (linked images) worked, there's no gallery option and I didn't see any parser support in https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/Parsers/png


e47e6a  No.11359

This might be a noobie question, but when tagging stuff my parent tags aren't getting added automatically. They show up in the results but when I press enter only the current one is added. Is this intentional or am I doing it wrong?


275c37  No.11360

My client.mappings.db is ~10gb, over 3x the size of my actual image collection

I assume it is full of a ton of tag data from the public tag repository etc

Is there some way to prune it so it only contains data on the files I actually have?


e578dc  No.11361

So I downloaded a few hundred images from the same site by overriding the bandwidth thing. After a while all of my queries just stopped, despite there being more to download and the website not blocking me. Re-adding the query to the downloader causes it to resume just fine. What happened?




File: ab23431121b8563⋯.jpg (96.69 KB, 492x650, 246:325, ab23431121b8563a7bc3b7ac57….jpg)

0675b5  No.5115[Reply]

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

613 posts and 193 image replies omitted. Click reply to view.

83c135  No.11319

>>11318

I've been running from source for a couple of days now and it hasn't frozen since.


000000  No.11321

when manually fixed "bytes > str" error, anoter one appear:

happens where symbols like ☆ are in neughbour txt files' tags

UnicodeDecodeError

'charmap' codec can't decode byte 0x98 in position 35: character maps to <undefined>

File "C:\Users\alexander\.virtualenvs\Hydrus_source-nKsCVeBo\lib\site-packages\wx\core.py", line 3259, in <lambda>

lambda event: event.callable(*event.args, **event.kw) )

File "C:\Hydrus_source\include\ClientThreading.py", line 382, in wx_code

self.Work()

File "C:\Hydrus_source\include\HydrusThreading.py", line 659, in Work

self._work_callable()

File "C:\Hydrus_source\include\HydrusData.py", line 1306, in call

self._func( *self._args, **self._kwargs )

File "C:\Hydrus_source\include\ClientGUIImport.py", line 1625, in RefreshFileList

self._paths_list.UpdateDatas()

File "C:\Hydrus_source\include\ClientGUIListCtrl.py", line 1042, in UpdateDatas

( display_tuple, sort_tuple ) = self._GetDisplayAndSortTuples( data )

File "C:\Hydrus_source\include\ClientGUIListCtrl.py", line 591, in _GetDisplayAndSortTuples

( display_tuple, sort_tuple ) = self._data_to_tuples_func( data )

File "C:\Hydrus_source\include\ClientGUIImport.py", line 1579, in _ConvertDataToListCtrlTuples

tags = self._GetTags( index, path )

File "C:\Hydrus_source\include\ClientGUIImport.py", line 1596, in _GetTags

tags = filename_tagging_options.GetTags( self._service_key, path )

File "C:\Hydrus_source\include\ClientImportOptions.py", line 368, in GetTags

txt_tags_string = f.read()

File "C:\Users\alexander\.virtualenvs\Hydrus_source-nKsCVeBo\lib\encodings\cp1251.py", line 23, in decode

return codePost too long. Click here to view the full text.


35b3df  No.11333

>>11321

Thank you, this should also be fixed for v336. I missed some file i/o encoding, thinking it would default to utf-8.

If you want to fix it yourself in the source, find the various open( path, 'r' ) calls across the program and insert an encoding param, like so:

with open( path, 'r', encoding = 'utf-8' ) as f:

Only the 'r' and 'w' calls–the 'rb' and 'wb' ones are raw bytes and have no encoding to do.


04b410  No.11347

File: e3f462c8675e596⋯.png (17.14 KB, 892x175, 892:175, ClipboardImage.png)

>>11211

I wanted to see if it would improve, and general usage seemed to be better but had another huge memory explosion on Hydrus shutdown, so py3 did not help that lock up.

>100% disk usage thrashing again

>14GB RAM


04b410  No.11357

File: 012b5af6b64f259⋯.png (279.06 KB, 1309x373, 1309:373, ClipboardImage.png)

On Hydrus 336 it looks like auto updating the PTR is causing Hydrus to lock up, anyway I can disable updates for it?




YouTube embed. Click thumbnail to play.

d2b76b  No.11238[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v335.tar.gz

When I first made this release, Github’s file upload was not working right, and I used Mediafire instead. Github is now working and I have updated the links above.

I had a great four weeks updating hydrus to python 3. The update went well, and the releases today are ready for all users, but there are special update instructions just for this week.

python 3

The client and server now run completely and exclusively on python 3, updating from python 2. The new version has a variety of benefits, mostly in better unicode vs. data handling, but for hydrus it also runs a little faster, uses less idle CPU and significantly less memory,Post too long. Click here to view the full text.

67 posts and 16 image replies omitted. Click reply to view.
Post last edited at

d69441  No.11345

>>11334

>Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs?

Just letting you know I whitelisted Hydrus and it works, thanks.


d2b76b  No.11348

>>11334

>>11328

Hey, on closer inspection, these were giving me 18KB 'webm' files that were actually some kind of CloudFlare html saying 'we think you are a bot, please solve this captcha'. Once I solved that in my browser, they worked mostly ok in the current dev version (which admittedly now has some unrelated ffmpeg fixes). The /trash/ ones are now 404, but https://i.4cdn.org/gif/1547302893499.webm is giving me a resolution problem that I think I can fix today.

I think part of this problem is the result of the cloudflare-4chan setup giving the wrong http header to say the file was 'webm' when it was really an html interstitial.

The other half is that hydrus has no capability to solve these sorts of captchas. I have never encountered this CF page before on a content fetch, and if it is going to be a new 4chan thing, that's a shame. Let's hope it is a temporary thing that was a misconfiguration or an accidental statistical spike on that set of files due to some mis-aimed DDoS or something. Please let me know if you encounter more of this.


1822a9  No.11351

File: 3eabd4993c89de2⋯.png (2.48 KB, 1071x24, 357:8, client_2019-01-15_22-14-49.png)

>>11348

just solved the captcha… no fucking clue why it did it for that one alone, or why it didn't give me the captcha but yea, now im getting this error with bringing it into hydrus.

that said, there has been some with captchas, there is an I believe open source project that can solve captchas without user interaction. it feeds the audio solution into google, and google spits out a solution and its 80-90% effective.

I know with fur afinity the issue is captcha, would it be possible to implement that into the program? google apparently already knows about the issue, but the captcha team more or less said, nothing we can do.


d2b76b  No.11354

>>11351

Thanks. I'd like to add captcha support, but diving back into the downloader code after all the work I did last year is not my top priority right now. It would be a lot of work, whether I would present login/download captchas to the user in the ui or try to solve them automatically.

Current plan is to do it for the next big iteration on the downloader, whenever that will be. If 4chan or other sites start getting this CF captcha shit a lot, I'll bump it up the schedule, maybe as a hardcoded fix that recognises that CF page specifically.

It is strange that both you and I got the same captcha issue, since I hadn't looked at that content/threads on my own machine/IP previously. That suggests to me the data source was the flagged problem, not the specific client, hence my DDoS-false-positive-on-that-thread suspicion. Again, we'll see if that starts happening more often.


1822a9  No.11356

>>11354

Thing is, I saw it fail, checked it, and was able to download it outside of client, then when you told me about the captcha and I checked It I got the captcha too.

No idea what what the hell was going on with that one.




File: e18c913ed90fd59⋯.gif (145.14 KB, 512x512, 1:1, e18c913ed90fd591c617879773….gif)

2a6778  No.11349[Reply]

I had a good week. I have fixed several bugs in the new py3 release (numerical ratings, some file and network unicode i/o, and some list sorting), cleaned and improved some long-time ui jank-code, and wrote in some limiters and a new workflow to smooth out mass-multiple import jobs.

The release should be as normal tomorrow.

Post last edited at

a6f687  No.11350

is this the kosher switch?


2a6778  No.11353

Post last edited at



File: 5734ceedfcc53a4⋯.png (219.04 KB, 373x323, 373:323, Tzs0oi5.png)

211bff  No.10443[Reply]

Make my day showing me your boners, lads.

I hope i'm not a degenerate by this board's standards.

9 posts and 9 image replies omitted. Click reply to view.

9803bb  No.11040

File: 636a75d8faa273f⋯.png (190.38 KB, 375x358, 375:358, ClipboardImage.png)

>>10443

+1.55TB / 210 488 files from yiff.party (including dropbox/mega downloads) since I implemented my own yiff.party downloader before it was added to hydrus and couldn't be bothered to redownload everything.

At least 25%-30% of the inbox is tumblr lower-res dupes and shitty reblogs. Fuck tumblr, glad it's dead.


3f2b96  No.11224

File: b83f51945a80e46⋯.jpg (69.64 KB, 388x362, 194:181, How Boned is Acionna [2019….jpg)

80% is from pixiv and nijie.

most was paged from 0,1,2,3…

have to change 12821 series to the 1,2,3 numbering.

tag to submit:

….

2335941 tags, I was unsure about the page numbering since I started collecting. That and a few other siblings I should triple check, it's been 2 years and I sure my quality control is quite sharper.


8c3fc1  No.11230

>>11224

Does Hydrus have a way to download from Nijie? I haven't looked too deep, but didn't see one.


3f2b96  No.11244

File: 5a847df25e48971⋯.png (6.06 KB, 512x125, 512:125, easy-import-nijie-artist-2….png)

>>11230

Yes, cuddlebear host a few downloader on his github.

Here's a freebie.

Still not perfect though, it doesn't collect either the filenames or the nijie work number.

Or perhaps I missed something on my side.


adec98  No.11338

File: a943e5eba90ddfc⋯.png (220.13 KB, 369x329, 369:329, client_2019-01-15_00-53-36.png)




File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
617 posts and 191 image replies omitted. Click reply to view.

4aa4d2  No.11060

`<svg` for svg , `<?xml` for other types of XML, `<!DOCTYPE html>` and `<html>` for HTML


e3e4ce  No.11273

Combining Hydrus and PBooru so that people can use it in TOR/I2P/IPFS/LokiNet

https://github.com/Kycklingar/PBooru


5ca661  No.11288

Idea for a rainy day: support for user.xdg.origin.url and user.xdg.referrer.url xattrs when importing from file system. I'm not sure how widespread their use actually is, but chromium family browsers on linux store the source url for downloaded files there.


690b74  No.11320

Minor huge suggestion/bugreport: yapf your sources.

Example to quickly poke around: https://yapf.now.sh/

For your code, I'd recommend 'facebook' style in particular.

And I would also recommend reducing the amount of empty lines; but that'd be more manual rather than one-step apply-a-tool.


e3e4ce  No.11323

API requests my mobile developers https://ghostbin.com/paste/qqvo7




File: 05d32f6e446a59e⋯.png (2.88 KB, 735x65, 147:13, Screenshot_2019-01-09 hydr….png)

File: f98e60cb69a9bb0⋯.png (10.76 KB, 733x268, 733:268, Screenshot_2019-01-09 hydr….png)

b745b3  No.11234[Reply]

Github is having trouble with my file uploads, so the release is delayed. I’ll give it another go later tonight, and if that fails, I’ll figure out something else tomorrow. Sorry lads!

43feb0  No.11235

Rats! I was getting excited reading the commits.


b745b3  No.11236

File: cfb0c9c54ffb312⋯.png (5.7 KB, 750x86, 375:43, fuck this.png)

>finally got upload working stable in new install of chrome

>upload proceeds at buttery smooth 85KB/s

>upload finishes, get this

>it was a .tar.gz file

I think Github may be fucked. I will try again tomorrow. Worst case, I'll throw them up on mediafire.


b745b3  No.11237

>>11236

Changed my mind. I want to hammer this shit out tonight. Uploading to mediafire now, release in 10-20mins.




File: 4067021824486d7⋯.jpg (3.25 MB, 4000x2733, 4000:2733, 4067021824486d72f958928445….jpg)

955d78  No.11223[Reply]

I had a great four weeks updating the software to use python 3, the new version of the language I write hydrus in. Overall, it went very well. The new client does not look different, but its guts are newer and neater. Some things run faster, and some long-time py2 bugs are gone.

The release will come at the normal time tomorrow, but it will have special one-time update instructions for all existing users. It is important you read the release post.

ce738c  No.11228

HYPE!


1ec45e  No.11229

File: 717a3ccdd07ed0b⋯.gif (499.23 KB, 300x250, 6:5, a pu pu pue.gif)

>>11223

Nice.


6dd3a1  No.11233

Cool! Can't wait to see that commit's diffs




YouTube embed. Click thumbnail to play.

166250  No.11002[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v334.tar.gz

I had a good week. I mostly fixed bugs in order to make a clean 'final' python 2 release.

all misc this week

I wrote a system: predicate for the new file viewing statistics. It works great!

Also, you can suspend file viewing tracking and clear all records entirely under the new database->file viewing stats menu.

Post too long. Click here to view the full text.
32 posts and 10 image replies omitted. Click reply to view.

166250  No.11213

>>11203

Hey, I am sorry, I do not totally understand what you wrote here. Can you rephrase it or add a screenshot?

I do remember a user reporting an issue with 'import folders' having bad tracking for the files it did not import (for whatever reason)–was this you? If you check the button next to 'review currently cached import paths' on the edit import folder panel, what do the paths there suggest about your problem?


1882b6  No.11226

File: 9abdfa80ff8dd0e⋯.png (3.37 KB, 378x92, 189:46, client_2019-01-09_04-03-54.png)

>>11213

oh simple. this is from a different importer where my hand saved images goes, so its showing 0 extra files.

however before I moved it to its new location, despite the folder being empty it would display 347/(however many new images where there here)

I think that it sees failed images and adds them to the beginning but regardless if you clean the images out, it sticks around, since I have been doing some cleanup on the g drive were my torrent folder is I have been going though artist archives and importing them, now there was one that had weird as fuck formats that were not importing and it skipped them, it added quite a large number to the beginning of the count, and subsequent imports have been adding 1-2 to the number due to txt documents, despite the images being removed from the folder.

at the time I didn't realize this was likely the problem and I believe you though that it was a folder that just had all those images in it and they were never moved.


1882b6  No.11227

File: d27ef43c9d18a6f⋯.png (3.89 KB, 370x83, 370:83, 2019-01-09_04-13-15.png)

>>11226

here found a small artist I didn't import yet this screenshot was taken before any imports happened.


166250  No.11231

>>11227

>>11226

Thanks. I will make a job to check this code and have it occasionally clear out paths that no longer exist.

I'll also be looking at changing some of these popup progress gauges to show progress more for the 'current job', like 1/17, rather than always showing entire-life progress like 4001/4017, which is always just a green bar with one pixel of grey, no matter how big the most recent sub sync or whatever was.


1882b6  No.11232

>>11231

lol asked about that somewhere too because that was a bit annoying seeing 'oh there are 1000 images here, the fuck happened' only to see the next stage of the sub have 975/983 and realized it remembers everything. not worth resetting the checkers for as to me it was a minor annoyance.

I should also say, the weirdness where on saving the session where it would bloat from 12gb to 18+gb has been solved with the removing the dead watchers from active view, however with a little over 100k images in active view mode, I am unable to set it to every 5 minutes as it does start to bloat and lag a bit, no where near the extent it did before, but a bit.




File: ea31ab670c8dee7⋯.png (12.55 KB, 340x175, 68:35, mariadb-usa-inc.png)

928501  No.9068[Reply]

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK

https://www.digitalocean.com/community/tutorials/sqlite-vs-mysql-vs-postgresql-a-comparison-of-relational-database-management-systems

https://www.infoworld.com/article/2611812/mysql/mysql-face-off--mysql-or-mariadb-.html

43 posts and 2 image replies omitted. Click reply to view.

f79f60  No.11053

>>10290

>https://github.com/acoustid/acoustid-index (C++)

You're looking for https://github.com/acoustid/chromaprint (C++)

To be honest though when Hydrus starts doing audio fingerprinting it should probably just use acoustid so it can grab tags from MusicBrainz ( https://musicbrainz.org/ )


3914e8  No.11058

>>11053

Or maybe others as well? What if we are getting music from torrents instead and don't want MusicBrainz to know that I got them?

Bumping to spark conversation

>>10232

http://www.scitepress.org/Papers/2016/59263/59263.pdf (Performance Evaluation of Phonetic Matching Algorithms on English Words and Street Names)

More benchmarks for major phonetic algorithms


74b0c2  No.11133

>>9068

>PostgreSQL - An SQL with complex features with less performance

1998 wants it retard memes back.


ed0a1d  No.11204

File: 880feed1fc57634⋯.png (1.57 KB, 300x300, 1:1, 下.png)

>>11023

>implying


928501  No.11206

>>11204

How so? Too many onyomi and kunyomi? Even then if we are not using phonetic fuzzy search, string fuzzy search can still be used (see https://en.wikipedia.org/wiki/String_metric)




File: 2134d74b2b06f36⋯.gif (131.16 KB, 296x262, 148:131, 2134d74b2b06f366b05549bcb7….gif)

a2edc8  No.11171[Reply]

Happy New Year!

I had a great week. The client works well, all big tests passed, and I now have functioning builds for Windows and Linux. I will spend the last week figuring out an OS X environment and doing last tests and cleaning.

I feel great about the schedule. v335 is completely on track for a Jan 9th release.

58dc14  No.11172

File: 825fe6e1e27985a⋯.jpg (138.32 KB, 1247x810, 1247:810, 825fe6e1e27985a2e2d8cf2c98….jpg)

iToddlers btfo.


0c77ee  No.11173

>>11171

Reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee


2fbb07  No.11176

Happy New Years hydrus dev!




File: c7ad8fdc159cbe0⋯.gif (99.35 KB, 320x180, 16:9, c7ad8fdc159cbe04b9fd009081….gif)

dc4159  No.11050[Reply]

I had a good week. I was able to get the client booting and closing cleanly on a new Python 3 environment on Windows. I have changed hundreds of tiny things, and there are still still a bunch more to go, and then I have to hammer out the 'freezing' process to make the executable release, and then figure out the same environment and build for the two other platforms.

I feel good about the schedule. It involves a lot of bashing my head against the keyboard, but I am moving forward.

3 posts omitted. Click reply to view.

dc4159  No.11062

>>11061

Yeah, it worked pretty good for me. A little spammy, but fast and easy to use. It mostly:

- Put safety parentheses around all my print statements (even though they all already had them)

- Put a safety list() around every iteration over a dict's keys/values/items call

- Changed up some explicit unicode language to str and cleaned up all the u stuff to just

- Did some really helpful urlparse -> urllib.parse and cStringIO -> io.StringIO stuff

This was all stuff I had feared I would have to do myself, so I was happy it was able to catch it all. The code was valid py3 and did boot. Unfortunately, it couldn't handle the encode/decode calls I had all over the place to handle some str/bytes conversion, so I knuckled down and learned how it works in py3 and rewrote the outstanding shit until it booted right.

I am overall very happy with the unicode changes in py3. They realised what was bad in py2 and fixed it all.

I still have a load of bytes/unicode stuff to clean up and test, stuff like object serialisation and network read/write, but that seems to be mostly it. Once I was booting into something stable, video rendering somehow worked right out of the gate.

I wrestled a bit more with pyinstaller today to get a frozen build going, and it was a pretty huge pain in the balls but I think I've figured something out. There's still a CWD issue stopping it getting too far, but all the dlls are loading and the splash screen comes up.

It may require a clean install, and <Win10 users may need Visual Studio Redistributable 2015. I will do testing with several users before I do the proper release so I know this better.


dc4159  No.11063

>>11056

There are some unit tests I threw together, and I run them every week, but I have had a lot of trouble fitting the 'make a comprehensive test suite and keep it updated' into my schedule. I would really like to have something neater here, but can't figure out how to get it done.


cc7452  No.11083

>>11062

So are there any code that you feel is slow in Python 3? And if so would you consider Nim-lang (using NimPy) to speed up parts of the code? Or maybe golang (with GoPy)?

Nim is basically a Python-like language that compiles to C, and Golang is made by Google to be a faster, better Java.

NimPy: https://narimiran.github.io/2018/05/10/python-numpy-nim.html

https://github.com/jboy/nim-pymod https://github.com/yglukhov/nimpy

GoPy: https://hackernoon.com/extending-python-3-in-go-78f3a69552ac

https://github.com/go-python/gopy https://github.com/sbinet/go-python


dc4159  No.11155

>>11083

I am happy with it so far. Some python-heavy computation like the lag on tag siblings/parents cache construction on client boot appears to be running much slower. Overall it feels it is running faster than py2. There's still a bunch of jank thread code on my end jittering things up, which is a long-term project to clean up.

There's a lot of duct tape that makes hydrus run from the frozen executables, and I have no experience with non-vanilla versions of python. I expect it is possible to get them going, but I bet trying to install wxPython and OpenCV is an additional headache. For now, I'll stick to the simpler environment. If you have experience with them and try it, let me know how it goes!


dc4159  No.11189

>>11155

Whoops, I meant the siblings/parents is running much faster. No significant problems anywhere, is the experience.




File: b8de52ee9a303f4⋯.jpg (35.02 KB, 583x263, 583:263, hydrus-encryption.jpg)

a8c3ca  No.6712[Reply]

I haven't seen this discussed so I thought I'd make a thread on it.

We are all aware that Hydrus runs fine inside a Truecrypt container, but having one huge file can be a problem.

For instance if you backup your 500GB database inside a truecrypt container and then download even just 1 image you basically need to backup the whole 500GB all over again if you run an automated backup to an external HDD or a NAS. Another problem with Truecrypt/Veracrypt volumes is you have to set the container size at the time of making it, leaving it too big or too small all too easily.

Well I've been experimenting with different crypto programs that feature encrypting directories while keeping files separate. The filenames and contents are unreadable, but they are still their own files so syncing and backup programs know exactly what to copy and what is the same.

The problem is that Hydrus doesn't want to run at all within these encrypted directories, but luckily we can have our installed client separate to the db folder.

>install an encryption program (I used cppcryptfs but there are a bunch for various platforms)

>create a new encrypted folder on any drive

>mount it as a Drive letter (you will need to use the same Drive letter each time)

>Move your Hydrus db folder to that mounted drive (or you can make a new one by ignoring this step)

>create a shortcut to your client.exe adding -d="path to db folder on virtual drive" (e.g "C:\Hydrus Network\client.exe" -d="Z:\db")

>use shortcut

it should find your database and start like normal, with any new files being encrypted as their own files that can be synced individually.

I haven't tried it but theoretically you could even mount a cloud provider as a virtual drive and store your entire database that way, probably stupid but I might try it for fun.

12 posts and 3 image replies omitted. Click reply to view.

8f670a  No.11118

>>10578

That is not going to be convenient unless we can make it simple (especially when someone has TBs of data on a RAID-enabled NAS)


8f670a  No.11123


58dbc6  No.11125

>>11118

Borgbackup is your ticket to make it simple.

>>11123

> https://github.com/bup/bup (Python, slow)

Actually this one is a fine CLI tool too, but unlike borg it's not considered stable by the developers, and you likely do want some borg features like the ability to prune older database backups and staggered retention for the other files.

Slow is extremely relative. Yea, python is not the fastest language around even if you use pypy (which however definitely makes most software that crunches some data quite a bit faster over standard python). But even that may not matter much if you're mainly waiting for IO coming from your HDD, which is usually what happens if you try to get data about your 40TB of image files (or whatever you might have on your array).

> See https://github.com/gilbertchen/benchmarking for info

A very limited test without the various configurables and apparently without any attempts to isolate how much time was spent waiting for IO or whatever, done only on the weirdly scheduled OS that is OSX?

And I don't even see what interpreters they used for python (was the "python" symlink pypy or mainline python and not a link?) and such. Never mind there are actually a LOT of configurables in these tools, and for some weird reason they didn't even try to pick the same compression algorithms even when it was possible (why not run the same settings on the same zlib or lz4 or zstd compression on all the tools so they become more comparable?).

Anyhow, use Duplicacy or restic over borg if you prefer, but I wouldn't pay too much attention to that benchmark.


d96ee2  No.11147

>>11123

I'm using duplicati myself, it's still slow because OS chokes on the amount of hard drive seeks, takes around 30 minutes to back up a 100k set for me, probably going to be even slower when you're backing up from an encrypted volume. It doesn't even matter if there were any changes to the collection, just the act of checking 100k files on a hard drive is a very slow process.


0109bf  No.11149

>>11123

I haven't used any of those, so I can't comment cleverly. I expect they do the job well. I use FreeFileSync once a week on my laptop's IRL 1.2M-file hydrus db, backing up to a WD passport. It takes several minutes to compare all the millions of files and thumbs and then a few more to sync the backup. Unless I have a gonzo week, I'd estimate it is usually less than 20min total. Since I have it going in the background as I put the hydrus build together on my dev pc, it is no trouble at all.




File: 06e65cedbc0cd2d⋯.gif (58.08 KB, 309x366, 103:122, 06e65cedbc0cd2d91f1edc786c….gif)

4a0f44  No.11098[Reply]

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had another good week. The client works about 95%, and I can get it into a proper executable release that runs fine. I now need to iron out the last issues and sort out Linux and OS X environments.

I feel great about the schedule. I am still aiming for a Jan 9th release for v335.

12 posts omitted. Click reply to view.

770c9a  No.11112

>>11110

we have IDs here cowshitter


9db34c  No.11113

>>10944

Finally this happened again. Hydrus had to run a pretty big maintenance job at exit, and once it was done it's just stuck there in the task manager as client.exe. I noticed about two and a half hours after I exited the program. There are no .db-shm and .db-wal siblings in the db folder, and it's not using CPU/HDD.

However, once I started writing this, it actually did shut down. Since I only caught this just before it shut down my notes about the .db file siblings and CPU/HDD usage might not be accurate. Here's the log, notice the 2 hour gap…


2018/12/26 09:15:06: hydrus client started
2018/12/26 09:15:06: booting controller…
2018/12/26 09:15:06: booting db…
2018/12/26 09:15:06: preparing disk cache
2018/12/26 09:15:10: preparing db caches
2018/12/26 09:15:10: booting db…
2018/12/26 09:15:10: initialising managers
2018/12/26 09:15:17: booting gui…
2018/12/26 09:15:21: Import folder cosplay imported 6 files.
2018/12/26 09:18:51: shutting down gui…
2018/12/26 09:18:51: waiting for daemons to exit
2018/12/26 09:18:53: vacuuming main
2018/12/26 09:18:53: Vacuumed Y:\db\client.db in 176 milliseconds
2018/12/26 09:18:53: Could not vacuum Y:\db\client.mappings.db (probably due to limited disk space on db or system drive).
2018/12/26 09:18:53: vacuuming external_master
2018/12/26 09:20:04: Vacuumed Y:\db\client.master.db in 1 minute 10 seconds
2018/12/26 09:20:04: vacuuming external_caches
2018/12/26 09:20:06: Vacuumed Y:\db\client.caches.db in 2.2 seconds
2018/12/26 09:20:06: database maintenance - analyzing

done!
2018/12/26 09:20:06: public tag repository sync: processing updates
2018/12/26 09:20:06: analyzing specific_deleted_mappings_cache_1_3
2018/12/26 09:20:06: analyzing specific_deleted_mappings_cache_9_3
2018/12/26 09:20:06: analyzing deleted_mappings_3
2018/12/26 09:20:06: fattening service info
2018/12/26 09:21:07: processed 1172848 definitions at 21831 rows/s
2018/12/26 09:30:02: processed 7445542 content rows at 13909
Post too long. Click here to view the full text.

dfbfbd  No.11114

File: 5bca7f91c4c2c29⋯.jpg (2.44 MB, 2480x3507, 2480:3507, 5bca7f91c4c2c29f811ce34f21….jpg)

>>11098

Merry Christmas and thanks for a great year with Hydrus!


447e8e  No.11115

File: e8db7ea03a0eaf5⋯.gif (927.84 KB, 467x467, 1:1, 1414982275812.gif)

>>11098

Merry christmas.


4a0f44  No.11148

>>11113

Thank you for this follow-up. Since the db does seem fully shut down and disconnected, my best guess here is that one of the daemon threads (which do maintenance stuff in the background) isn't waking up to receive the 'program shutdown' signal properly. In this case, the process will hang on, with that one thread asleep, until it wakes according to its natural check period (which are typically on the order of hours).

I will check this code. It may also magically fix in py3 due to the different way some thread signalling works as well, so please let me know if this improves/worsens after v335.

In this case, as the db is completely closed, there is no danger in just killing the process in task manager when this happens again.




Delete Post [ ]
[]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / agatha2 / asmr / ausneets / cafechan / doomer / loomis / strek / sw ]