Chess Databases Software surverying for limits

Well, this is not about your own repertoire chess databases...

About whole world chess databases, or any subset..... Me knowing not much...

Could we order known and still in use chess software per the abilities to manage some range of databases types still in use in chess (that also might need context).

say size in number of position, or games, or computer size...
ability to strip annotations from pure chess game data (might come in handy at times, but not urgent).

if format on computer is an issue, can we alter size limits accordingly.

Also what type of consensus testing operations would be telling about each database handling software..

How far from non-chess database formats are all the still in use chess database formats... Any big database of chess nature question that a you might think worth mentioning here is welcome.

I can start with things i have heard of..
Chessbase,
Scid vs PC (and scidb look related),

a few chess GUI that are not targetting databases but do have database related features such as book making from game database, with some limite control over that process).
LucasChess, ChessX, Banksguia, others.. i don't want to assume too much.. but suggest at least.

perhaps even old clunky GUI or not updated like Arenan, might be powerful on the DB handling side.... or more flexible at some input for user control of manipulations..

I am interested newbie in the world of chess databases (not personal repertoire only). Have been for a while, and i am still lost. but I never asked like this. I am not expecting a magic bullet.. more values or estimates of limits, things like pros and cons..

@dboing Chessbase is the answer for your curiosity. :)

I fear chessbase might be the expensive thing to compare the others with, but what are its size limits?

Also, one feature of interest to me, is being able to specificy similarity function between position, as i think this is in need of research with more than one function family.. Is this sort of thing above ground, or hard-wired in?

would chessbase allow something like a position database from a game database, that would keep all the relations intact? With the option of have that position database also be an outcome statistics explorer like lichess explorer. control of maxply maximum depth for such statistics. keeping some diffing ability between database operations...and the moon?

what else would be nice.. and might not exist at all....?

I have also seen, i think with BanksguiA (i might be a murderer of names, apologies in advance), how a big tree of variations might be plotted on some restricted 2D area.... (but that is not the thread focus, not directly).

@dboing My experience: "I recently got Chessbase 17, I haven't gotten any Chessbase product for the longest time till now (Previous Chessbase was 13 and 14.) I made a database consisting of 10+ Million games and guess what? It can handle 10 GBs (Database sort + the search booster).
I do believe that Chessbase can handle database of any size. Besides, the software isn't really the issue here, it won't hold you back. It's the PC, if your computer isn't powerful enough then most likely it wouldn't be able to open big databases. It would just crash."
Chessbase can do all of the feature you've included above. It can even segregate crazy games from "boring" games.

@Shreksify said in #4:
>
I have been testing a few.. CBR 2017 (the free reader,. a crippled version of CB14) is also able to upload in timely fashion. blitzbase DB you shared with us. earlier.

I am not looking for one software, i am curious about the current state of available options. so good to know that such big DB can be interactively handled with latest CB. I have read that CB can create position tree out of a game database (which other call book or lichess opening explorer), so that would be something i would also like to survey in other software (possibly different terminology).

Of software i am aware of being able to digest game databases in such a fashion (book building, or digesting) , i am also trying a few.
1) Lucas chess, stalled on loading the big PGN, some process hanging.
2) Banksguia gui.. is slow, but not giving up yet. actually giving good peek on its progression in more than one variable.

data
converting from PGN to OCGDB:
some 800,000 games so far.. 40 minutes loading.. speed of 282 games/s. very slow. aboring.
just loading as PGN lead to BG converint in own internal format at 6000 games/s.. more reasonable.

intent to try scid. and even arena later.. any suggestion of other software free or open source (best for many reasons).

@dboing Speed of 282 games/s? Chessbase can do 20x more than that!
Jerry Gui is cross-platform (Windows or Linux).
You can adjust strength levels to match your skill.
I was able to browser my Practical Repertoire PGN database with it.
During the analysis the MultiPV shows up to 4 lines.

github.com/asdfjkl/jerry
buildingjerry.wordpress.com/

I just discovered this chessui. I like the opening usage explorer. It's not a Gui that you can download, but it's too good to not pass the word around.
chessui.com/#
database.chessui.com/explore/

I use PyChess: pychess.github.io/

But, Lucas Chess is worth a look too: lucaschess.pythonanywhere.com/

Got this from Lichess, here's the original forum lichess.org/forum/general-chess-discussion/chess-guis#3

Currently not focusing on personal chess or engine tools.. specifically database operations.

Thanks for the links.. i did not know of them but for lucas chess will search that lichess thread for DB aspects. Most of the GUIs seem understandably target to personal use, while I am currently focussing on population level data. for some chess curiosity of mine.. where database handling might become something i would need to be familiar with and have some interactive timescales.

So I would not like to enlarge the scope to other features, as I am sure they are often well advertised. I find the book building and database aspect a bit harder to find out..

I have enough of tools on lichess for my personal chess needs. so far. Although such GUI do give me ideas too. The gui is also not a necessary condition.. although preferable, best of both worlds GUI and not, is a GUI also allowing enough control and transparency into what happen at the data level.

about Jerry.
handle large (i.e. 1 GB) PGN files. I might look into what it can do with such DB, in spite of size limitation.
the building jerry page might be interesting later for how a gui comes about. it seems also to elaborate on methods of data analysis dealing with chess DB.

about chess ui.. interesting online resource..
about Pychess. for chess variants. Also an online resource.

I think that most sites and software do target individual size repertoir access. Lichess might be unique in provide some version of its live full database.. although the size of it, is not yet in my bandwrith to deal with (literally and figuratively).. I like that it is there...

So i expect a lot of software which also deal with databases to have made some decisins about expected use case.. maybe i am wrong. This thread is kind of serving that purpose.. i think..

I like the chess ui explorer link. a bit slow to react.. but seems to offer some slice into opening data possible very current...

BanksiaGui.

good news. took some time. but now I am happily surprised or woken up, by seeing its internal format whatever it is, also come with some SQL querrying facility..

a caveat to my individual use case for database, would be engine tournament user cases, which are likely to produce lots of games... and this might be what is driving this software...

not as slow loading (and at least not stalling) on the big PGN... i have a slow machine anyway.. waiting for it to melt one hot summer day, and firing a modern browser that day....

some hope for chess relational database at tip of data analysis hands..... massaging left and right and other directions in between...:) getting ahead of myself.

@dboing Glad I could help! :) Please do update me with your experiments. It's not everyday I bump unto someone who test Chess softwares :)