NNUE controversy and some suggestions (example: Igel, Nemorino...)

[Hrvoje-Horvatic]

Hello everyone...

I'm new here, registered today, but I have been toying with chess programming for 20 years (started in 1999.) The reason for this message is a small confusion that CCRL has added to, instead of being the "good guys", and doing what I think is "the right thing"...

If you check latest CCRL rating list, you will notice that some engines (I'm using for example Igel 2.7 & 2.8 and Nemorino 6) have suddenly jumped 200-300 ELO points... to those of you who follow recent events, you might know at once that this is because of Neural Networks, or more specifically stockfish-derived NNUE (efficiently-updated-neural-networks)...

There has been lots of negative energy around both Leela and NNUE technology, so It would be nice that those "in power" don't add to that confusion and negativity... and by those "in power" I also mean CCRL...

more specifically, it would be nice if you specified which engines you tested, and which are NNUE-derived engines... Nemorino got 300 ELO, and Igel 200 ELO, and they didn't program that, they plugged stockfish-derived networks (you can actually see stockfish code in their source code repositories on github and bitbucket)... I'm not picking on this engines, I have nothing against their authors, they are just used as examples... and I'm not sugesting chess engines shouldn't use ANNs (why shouldn't they), but it should be CLEARLY STATED which is which... so my suggestion is to use engine names Igel NNUE 2.8 and Nemorino NNUE 6.0... otherwise it looks like plain Igel and Nemorino are stronger than Rubichess, which they aren't (not even close)... funny as it seems, Rubichess author allowed users to load NNUE network, but it is not setup as default, and network is not embedded in .exe file, like it is in the case of Stockfish, Igel and Nemorino... and this creates a lot of confusion... which engines are which? which allow what? which ones did you test?

As I am chess engine author myself, this makes me a little bit angry, so I apologize if I am coming on all of you a bit too hard, but I think I have a point... It would be nice if we (chess community) could work out some minimal, basic set of rules that everybody follows... and it would be nice if you could help with that...

H.

[Ray]

Hello everyone...

I'm new here, registered today, but I have been toying with chess programming for 20 years (started in 1999.) The reason for this message is a small confusion that CCRL has added to, instead of being the "good guys", and doing what I think is "the right thing"...

Well, different peoples' opinion of what is "right" does differ !!

I'm sure Volodymyr will be along to explain Igel. My understanding is that it's NNUE is an "original" net and is not Stockfish.

You are assuming that if an engine has NNUE embedded you would actually know that it has, if the author did not mention it ? And they are under no obligation to do so.

It is a minefield !!

[Hrvoje-Horvatic]

I'm sure Volodymyr will be along to explain Igel. My understanding is that it's NNUE is an "original" net and is not Stockfish.

Again, I'm not picking on this engines, and I didn't say they did something wrong... this is not a trial, witch hunt or whatever... I don't see any need for Volodymyr to explain anything... My point was that it should be clearly stated that it is NNUE, and NOT regular engine that HAS BEEN TESTED... that`s all...

You are assuming that if an engine has NNUE embedded you would actually know that it has, if the author did not mention it ? And they are under no obligation to do so.

Exactly. There is an easy was to check it for yourself... if the executable is about 20Mb big, it contains embedded network file... networks are relatively big... if it is 2 or 3 MB (or less) it doesn't...

Besides, we are talking about OPEN-SOURCE engine... you can go to repository and check it for yourself, there is nothing hidden and secret here...

https://github.com/vshcherbyna/igel/com ... 11a6e3ff72

- Switch to NNUE as a main evaluation function
- Use Dietrich Kappe's Night Nurse NNUE net as a default net for Igel 2.7.0 release

If you can read C/C++ code, you can verify it for yourself that I'm correct...

It is a minefield !!

No, it's a gray area, and a little bit of light could help...

H.

[Ray]

Exactly. There is an easy was to check it for yourself... if the executable is about 20Mb big, it contains embedded network file... networks are relatively big... if it is 2 or 3 MB (or less) it doesn't...

I don't think you should make the assumption that just because the exe is large, it uses NNUE. It could be for a different reason. But it would certainly cause you to ask the question.

NNUE is just a feature of an engine in my view. Engines contain many features, e.g. LMR etc. Personally I see no need at all to put NNUE in the name and I'm not in favour of that if it is a standard feature of the engine. It could be mentioned in the engine notes though. If you test an engine twice, once with NNUE and once without, then you need to do something, but I hope we never do that.

What I am concerned about is originality of the Net, and that could be a grey area as well. If someone simply plugs in the SF net into their engine, I hope we would not test it. But this whole originality thing is a nightmare even before NNUE came along.

[Gabor Szots]

If someone simply plugs in the SF net into their engine, I hope we would not test it.

If the engine is not open source, we can't tell if its NNUE is original or Stockfish. And even if it is open source, only a programmer could tell if the NNUE is original. I'm not even sure a programmer could tell.

PS. I think there is a debate if SF networks are original or not.

[Ray]

If the engine is not open source, we can't tell if its NNUE is original or Stockfish. And even if it is open source, only a programmer could tell if the NNUE is original. I'm not even sure a programmer could tell.

PS. I think there is a debate if SF networks are original or not.

Yes, a minefield ! Which is why we tend to steer clear of the "original" discussions and arguments.

[Ray]

... Nemorino got 300 ELO, and Igel 200 ELO, and they didn't program that, they plugged stockfish-derived networks (you can actually see stockfish code in their source code repositories on github and bitbucket)... I'm not picking on this engines
H.

Is that the same way that dozens of engine authors have plugged Syzygy tablebase code into their engines and gained Elo, or prior to that plugging in the Nalimov code ? I don't see any difference fundamentally, apart from the scale of the gains.

[Hrvoje-Horvatic]

Ray, you are not listening to me...

let`s try it one more time, in slow-motion...

you have 2 engines, let's call them engine A and B... both CAN use NNUE and both can play WITHOUT NNUE... you test one with, and one without NNUE... you end up with weaker engine being IN FRONT OF stronger engine... so CCRL has become useless...

Ray, what's the point of rating list? is it maybe to show you which engine is stronger, and by how much (approximately)?

and what if CCRL is not doing that anymore? I used to rely on CCRL, but not anymore... I had to do my own testing to find it out...

H.

[Hrvoje-Horvatic]

Is that the same way that dozens of engine authors have plugged Syzygy tablebase code into their engines and gained Elo, or prior to that plugging in the Nalimov code ? I don't see any difference fundamentally, apart from the scale of the gains.

Are you for real? or are you just trolling me? Because I honestly can't tell the difference whether you really don't understand this, or you are playing with me...

If you don't understand, ask... and you willl be educated...

and if you are trolling me... I'm outta here...

[Graham Banks]

Ray, you are not listening to me...

let`s try it one more time, in slow-motion...

you have 2 engines, let's call them engine A and B... both CAN use NNUE and both can play WITHOUT NNUE... you test one with, and one without NNUE... you end up with weaker engine being IN FRONT OF stronger engine... so CCRL has become useless...

Ray, what's the point of rating list? is it maybe to show you which engine is stronger, and by how much (approximately)?

and what if CCRL is not doing that anymore? I used to rely on CCRL, but not anymore... I had to do my own testing to find it out...

H.

I do most of the 40/15 testing, and will only test an NNUE version if the engine author has not used an SF net.
So, at present, those would be Igel and Nemorino, as the Minic author has asked that his NNUE version not be tested for the rating list.

[voffka]

Hello Hrvoje-Horvatic,

If you check latest CCRL rating list, you will notice that some engines (I'm using for example Igel 2.7 & 2.8 and Nemorino 6) have suddenly jumped 200-300 ELO points... to those of you who follow recent events, you might know at once that this is because of Neural Networks, or more specifically stockfish-derived NNUE (efficiently-updated-neural-networks)...

Igel dev here.

I think there is some confusion on your side indeed, it is great that you created this thread so that the confusion can be addressed. The decision on using NNUE in Igel is explained in http://talkchess.com/forum3/viewtopic.p ... 42#p856742. Currrently Igel uses a network that is called Night Nurse and it is not related to Stockfish.

Moreover, currently there is work ongoing on making Igel use a custom architecture network trained solely based on Igel 2.6.0 (hand crafted evaluation) and it takes time.

[Hrvoje-Horvatic]

Hello Hrvoje-Horvatic,

Igel dev here.

I think there is some confusion on your side indeed, it is great that you created this thread so that the confusion can be addressed. The decision on using NNUE in Igel is explained in http://talkchess.com/forum3/viewtopic.p ... 42#p856742. Currrently Igel uses a network that is called Night Nurse and it is not related to Stockfish.

Moreover, currently there is work ongoing on making Igel use a custom architecture network trained solely based on Igel 2.6.0 (hand crafted evaluation) and it takes time.

Again, to quote myself, there is nothing wrong in using NNUE, and this is not a witch hunt... my problem is that on the CCRL list I can't see which engine is NNNUE (or tested with NNUE) and which isn't... which completely distorts results... I can recognize Lc0 clones because they use expensive graphic cards, and it is written in their name that they do, but I can't recognize whether Minic or Rubichess were tested one way or the other... the fact that Minic author asked Graham personally that he doesn`t test NNUE is useless to me... Just because Graham knows that, I don't... I mean, I know NOW but that's after I compiled all engines (including yours) and tested them personally... to be sure of results...

I have no quarrel with you.

PS. if you want to experiment with different architectures, focus on first layer (input)... changing other things will not have so great effect, most NNUE networks will still play similar chess (which is good, logical, human-understandable chess)... learning on different sets will have relatively small effect, smaller than you expect. biggest differences (in learning and play) come from different (or additional) inputs... Making your hand-crafted eval as input is good idea (in my opinion), but you need additional inputs besides your eval features, they are not enough... my 50 cents...

[Gabor Szots]

my problem is that on the CCRL list I can't see which engine is NNNUE (or tested with NNUE) and which isn't

CCRL tests engines using default settings because those are the settings the author thinks are best. It is immaterial if those settings include NNUE usage or not or if the engine is capable of using NNUE's at all.

[Graham Banks]

A solution might be to put in the engine notes if it is using an NNUE.

[Ray]

A solution might be to put in the engine notes if it is using an NNUE.

That is the plan, where we know.

[Gabor Szots]

Looking e.g. at this post I feel we are in a complete mess. Minic author would like to see two Minics on the list. I wouldn't have two.

http://talkchess.com/forum3/viewtopic.p ... &start=130

[Ray]

As I said, it is a minefield, and we've only taken one step into it !!

I would hope that we would not test any engine twice, once with and once without NNUE.

[Gabor Szots]

I would hope that we would not test any engine twice, once with and once without NNUE.

I am hereby swearing that I will never test an engine with two different settings. If it's NNUE, I'll test it so.

[Graham Banks]

Well, we do test Komodo with and without MCTS, so how is that different?

[Ray]

Well, we do test Komodo with and without MCTS, so how is that different?

We just stepped on a mine

[alex67a]

Alliestein, Leelastein, FatFritz and perhaps Stoofvlees use the same networks as Lc0 and no one has ever said anything!
Why so much controversy now ?
Maybe someone does not want certain engines to drop from the ranking of the strongest, but it is not said

[alex67a]

Rybka was excluded from competitions because he was accused of copying, Houdini idem from the last TCEC
Instead, Stockfish copies from another board game an evaluation function that makes him win one game out of 4 in present TCEC (imagine a world championship among humans where a player in 20 games would win 5-0!)
And nobody has anything to say (not even me, but I am consistent and I recognize that certain rules apply to everyone or none)

[Rubi]

Ray, you are not listening to me...

let`s try it one more time, in slow-motion...

Calm down! This is not the kind of conversation that leads to something useful.

you have 2 engines, let's call them engine A and B... both CAN use NNUE and both can play WITHOUT NNUE... you test one with, and one without NNUE... you end up with weaker engine being IN FRONT OF stronger engine... so CCRL has become useless...

CCRL tests engines with their default settings so the author decides if the engine uses a handcrafted evaluation or a NNUE evaluation by defining these defaults. CCRL "just" has to decide if the weight file that is used for default evaluation is "original". Whatever "original" means, this is indeed controversial and is already discussed in many different places and not specific to CCRL.

If Minic author wants both evaluations tested, he probably should release two engines Minic-Legacy and Minic-NNUE like Komodo does it with the two different search implementations. A perfect analogy imo except for the different motivations seeing Minic in rating list vs. selling copies of Komodo(MC).

Regards, Andreas

[Gabor Szots]

CCRL tests engines with their default settings so the author decides if the engine uses a handcrafted evaluation or a NNUE evaluation by defining these defaults. CCRL "just" has to decide if the weight file that is used for default evaluation is "original". Whatever "original" means, this is indeed controversial and is already discussed in many different places and not specific to CCRL.

If Minic author wants both evaluations tested, he probably should release two engines Minic-Legacy and Minic-NNUE like Komodo does it with the two different search implementations. A perfect analogy imo except for the different motivations seeing Minic in rating list vs. selling copies of Komodo(MC).

Regards, Andreas

This is an area full of traps. E.g. in Nemorino 6 the default value for UseNNUE is false. I still tested it with UseNNUE=true because that was clearly the author's intention.
With Komodo we may have made a mistake: we used MCTS in the name where MCTS search was switched on, as though they were two different engines when in fact they were the same.

I agree that if the author wants both evaluations to be tested he should release them under different names. But that is in fact the same engine with the different parameter values hardcoded. Do we really want that? Two engines on the list with the same or similar name? No, the author should stick with one name and set default values as he deemes they are best.

I know (and see) many have different views. Even within CCRL views may differ. What I wrote above was my personal opinion.
This is a precarious situation and CCRL is in for criticism whatever policy it chooses to follow.

[Ray]

This is a precarious situation and CCRL is in for criticism whatever policy it chooses to follow.

That is absolutely true. We can't please everyone all of the time.

It is an incredibly difficult situation, because we really don't want to be in a position of having to judge if a net is "original" or not, and however "original" is defined. Any more than we want to have to judge if an engine itself is original or not. Almost an impossible situation for us. That is a can of worms right from the start.

I don't know what we will do. If there is a policy then people will always argue if we've complied with that policy or not. How did we decide if the net was original or not ? What steps did we take to determine that etc etc

So if we simply say each engine is decided internally on a case by case basis at our own discretion, then that could be the best way.