New Engine: Beef

[drstrange767]

Hi all,

I've spent the last couple months shut in at home making my own chess engine, and I think it's finally ready for release and rating!

Name: Jonathan Tseng
Engine name: Beef
Nationality: US
Protocol: UCI
Open source: https://github.com/jtseng20/Beef/releases/tag/v0.2

I estimate from limited fast testing that Beef is around 3000.

I work on a linux machine, so the makefile should work for linux, and my windows compiles are made with visual c++, but I am not completely sure yet if everything is correct. Please tell me if there are any problems with the given binaries. Thanks!

[Ray]

A 3,000 Elo engine created in just two months, that is impressive !

[drstrange767]

Thanks for the kind words! Really, it has been more like 4 months, and for the first 2 months, I was coding 5-10 hours a day . I'm a student, so nothing to do during this summer. I'm looking forward to getting a more official rating in here at CCRL, though.

[drstrange767]

Hi all,

I know the testers are always very busy, but any word from the blitz guys when Beef might get its turn? No rush, but it'd be great if I had some idea.

Thanks!

[Gabor Szots]

Hi Jonathan,

Speaking for myself, I have still not recovered from the shock that a new 3000+ engine could be created in such a short time. Frankly, it is hard to believe that the only thing you have taken from the engines listed on your home page is simply 'inspiration'. That's why I am still reluctant to test your engine.

It might change my stance if you pointed out which important parts are your original work and include a statement which engines Beef is based upon.

[Daniel Williams]

I took some time to investigate this new engine. It is definitely around the 3000 Elo mark, able to win 88% of games against an older version of Rybka.

The README.md file says that the engine is "inspired" by
* Defenchess by Can Cetin and Dogac Eldenk
* Ethereal by Andrew Grant, Alayan and Laldon
* Rubichess by Andreas Matthies
* Slow Chess Blitz Classic by Jonathan Kreuzer
* Stockfish by Tord Romstad, Marco Costalba, Joona Kiiski and Gary Linscott

Looking through the code, I see a lot of advanced features like parallel search, tablebase probing, automated tuning, and extensive techniques for move ordering and pruning. It seems unlikely that such an advanced engine could be developed on the first try while learning C++. It shows significant resemblance to the Stockfish source code in some areas. I did not compare the code to the other above engines.

[drstrange767]

Hi all,

I understand the above concerns. These are valid concerns, and I've discussed with fellow engine creators on the topic of similarity before. Please allow me to offer a brief defense. My engine is primarily based upon an original move generation system, which unlike every top engine to my knowledge, is legal only. The benefit to legal-only move generation systems is that it is much faster (in my own testing) compared to pseudo-legal-then-check like stockfish and others. This can be seen in movegen.cpp, where my generate legal moves method is simply an extension of generate pseudo-legal, and where my generate pinned moves and generate quiet checks method is original to all other engines. The move generation code uses the same basic data structures (arrays and structured lists in similar places) as stockfish, but is very, very different in actual functionality.

Yes, there are many features in my search. Yes, a good few of these are found in stockfish. No, I did not lift these without testing, and I am prepared to share the literally hundreds of binaries that I have compiled testing each and every line of pruning code I have included. I understand Daniel's claim that I am a first time programmer in the chess community; however, I study programming and machine learning in my university program and am (I think) advanced enough to understand the concepts that I read on the well documented wiki. To address the point that I was learning C++, this is a valid concern also, however, I actually wrote Beef in two other languages before switching to C++, each time cumulatively building upon the previous version. I also know C, so that probably helps . I first wrote it in python, then in cython, which was a good bridge for starting in C++. I am happy to share any of the aforementioned code/binaries with anyone who wants to see them, and I am more than happy to personally discuss with anyone personally who doubts that I thoroughly understand what every single line of code I have written does.

Small differences can be found in the evaluation function, too, and anyone who is tireless enough to want to, I can show them the full commit history even before I made Beef public, which will show that I have been working consistently inside the evaluation to add differentiating features. I could even show chatlogs between myself and two of my personal friends, who are chess masters, where I discuss the addition of features.

In the end, sure, many of the features in Beef are seen in advanced engines like stockfish, but only on a surface level, because as far as I know, none of them are exactly the same, which means that they weren't just lifted, and furthermore, I won't point fingers, but I do not find the similarity between Beef code and stockfish code to be any stronger than that between stockfish and other engines in the vicinity. I'm more than happy to talk in finer detail about any particular code segments with anyone.

I'm a bit sad to see these accusations, but it's understandable. I put a lot of work into the engine, so I'm more than happy to discuss and I'm enthusiastic to show anyone that I truly understand and could reproduce from my own ideas the code that is written in the engine. PM me here and we can find a time and place to talk.

Thanks,

Jonathan

[Graham Banks]

I think we should be more trusting of engine authors who are prepared to share their source code and give credit to where they got inspiration from.
Just my opinion.

[Gabor Szots]

I think we should be more trusting of engine authors who are prepared to share their source code and give credit to where they got inspiration from.
Just my opinion.

I agree with you on that.

[Daniel Williams]

I definitely do not object to having the engine tested. Clearly, a lot of work has been put into this engine. Having already programmed this engine in other languages prior to C++ lends a lot of credence to it. My only concern is that, having developed my own engine for five years from scratch, it is easy to tell the difference between a totally original engine and one that has been inspired by other's source code. Again, not to say that you haven't put a lot of work into making this a strong engine.

It may be misleading to CCRL viewers when they see this engine near the top of the list. Perhaps a description of the development of this engine and a list of the engines that it was inspired from could be added to your GitHub page?

[AndrewGrant]

Gabor made a post on Talkchess in a forum found here:
http://talkchess.com/forum3/viewtopic.p ... 28#p861228

My response to his post can be found here:
http://talkchess.com/forum3/viewtopic.p ... 39#p861239

I would encourage CCRL to test and rate Beef, and to give the author some credit for being open and willing to share.
I would encourage Beef's author to look to diversify his engine where he can, and provide reasonable acknowledgment for ideas from the things which must stay.

[Ray]

Yes, the source code is there so the author has been very open about his work. I'm happy for us to test it.

[Gabor Szots]

Yes, the source code is there so the author has been very open about his work. I'm happy for us to test it.

In fact I have already started a gauntlet with it but suspended it waiting for further assurance.

[Ray]

Yes, the source code is there so the author has been very open about his work. I'm happy for us to test it.

In fact I have already started a gauntlet with it but suspended it waiting for further assurance.

Always good to be cautious.

[Gabor Szots]

Let's have a look at these two code snippets.

1.
Defenchess

Code: Select all

int timer_count = 1024,
    myremain = 10000,
    total_remaining = 10000,
    think_depth_limit = MAX_PLY;

bool is_movetime = false;



volatile bool is_timeout = false,
              quit_application = false,
              is_searching = false,
              is_pondering = false;

Move latest_pv, latest_ponder;

Move main_pv[MAX_PLY + 1];

Beef

Code: Select all

int timer_count = 1024,
    ideal_usage = 10000,
    max_usage = 10000,
    think_depth_limit = MAX_PLY;

bool is_movetime = false;
bool is_depth = false;
bool is_infinite = false;

volatile bool is_timeout = false,
              quit_application = false,
              is_searching = false,
              is_pondering = false;

timeInfo globalLimits = {0, 0, 0, 0, 0, 0, 0, 0};
volatile bool ANALYSISMODE = false;
Move main_pv[MAX_PLY + 1];
Move latest_pv, latest_ponder;

2.
Defenchess

Code: Select all

    bool is_pv = beta - alpha > 1;
    if (is_pv) {
        md->pv[0] = no_move;
    }

    if (ply >= MAX_PLY) {
        return !in_check ? evaluate(p) : 0;
    }

    SearchThread *my_thread = p->my_thread;
    if (is_draw(p)) {
        return draw_score(my_thread);
    }

    Info *info = p->info;
    md->current_move = no_move;
    (md+1)->ply = ply + 1;
    int tte_depth = in_check || depth >= 0 ? 0 : -1;

    Move tte_move = no_move;
    int tte_score = UNDEFINED;
    bool tt_hit;
    TTEntry *tte = get_tte(info->hash, tt_hit);
    uint8_t tt_flag = tte_flag(tte);
    if (tt_hit) {
        tte_move = tte->move;
        if (tte->depth >= tte_depth) {
            tte_score = tt_to_score(tte->score, ply);


            if (!is_pv &&
                (tt_flag == FLAG_EXACT ||
                (tt_flag == FLAG_BETA && tte_score >= beta) ||
                (tt_flag == FLAG_ALPHA && tte_score <= alpha))) {
                    return tte_score;
            }
        }
    }

    int best_score;
    if (!in_check) {
        bool is_null = (md-1)->current_move == null_move;
        if (tt_hit && tte->static_eval != UNDEFINED) {
            md->static_eval = best_score = tte->static_eval;
        } else if (is_null) {
            assert((md-1)->static_eval != UNDEFINED);
            md->static_eval = best_score = tempo * 2 - (md-1)->static_eval;
        } else {
            md->static_eval = best_score = evaluate(p);
        }
        if (best_score >= beta) {
            return best_score;

Beef

Code: Select all

    bool is_pv = beta - alpha > 1;

    if (is_pv)
    {
        memset((info+1)->pv, 0, sizeof((info+1)->pv));
        info->pv[0] = MOVE_NONE;
    }

    if (isDraw(pos))
        return 0;

    bool in_check = (bool)(pos->checkBB);
    if (ply >= MAX_PLY)
    {
        return !in_check ? evaluate(*pos) : 0;
    }

    Move hashMove = MOVE_NONE;
    int hashScore = UNDEFINED;
    bool ttHit;
    TTEntry *tte = probeTT(pos->key, ttHit);
    uint8_t flag = tte_flag(tte);

    info->chosenMove = MOVE_NONE;
    (info+1)->ply = ply + 1;

    if (ttHit)
    {
        hashMove = Move(tte->movecode);
        hashScore = tt_to_score(tte->value, ply);
        if (!is_pv &&
            (flag == FLAG_EXACT ||
             (flag == FLAG_BETA && hashScore >= beta) ||
             (flag == FLAG_ALPHA && hashScore <= alpha)))
             {
                 return hashScore;
             }
    }
    int bestScore;

    if (!in_check) // stand pat
    {
        bool is_null = (info-1)->chosenMove == MOVE_NULL;
        if (ttHit && tte->static_eval != UNDEFINED)
            info->staticEval = bestScore = tte->static_eval;
        else if (is_null)
            info->staticEval = bestScore = tempo * 2 - (info-1)->staticEval;
        else
            info->staticEval = bestScore = evaluate(*pos);

        if (bestScore >= beta)
        {
            return bestScore;

They are practically identical, as far as I can see. Are these parts this much standardized?

PS. I can show more snippets upon request. Total Commander's file comparison is a wonderful tool.

[AndrewGrant]

For the 2nd segment of code, yes. Everyone flags in_check and is_pv somewhere near the top of the search. The next step is to check for drawn positions or having exceeded the maximum search ply, in which case you return an evaluation. The comparison against in_check there is not something I'm used to, but I might be one of few to not do it.

After that probe the TT, take cutoffs based on the bounds. Then evaluate the board, and make use of two facts: We could have an eval saved in the hash table, and if we made a null move we can cheaply recompute the eval by flipping the Tempo bonus.

As for the first segment, that seems shotty

[drstrange767]

Pertaining to that first segment of code, all I can say is that as far as naming variables go, I've always been quite bad, so I used the same variable names. A bad choice on my part; however, the actual system of timing beneath those variable names has been changed several times.

[Daniel Williams]

I ran a verbatim text comparison between the source files of the various programs:

Search file:
Defenchess vs Beef: 176 matches, 35% match
Rubichess vs Beef: 9 matches, 1% match
Stockfish vs Beef: 14 matches, 2% match
Ethereal vs Beef: 20 matches, 3% match
Slow Chess Blitz vs Beef: 3 matches, 0% match

Evaluation file:
Stockfish vs Beef: 35 matches, 6% match
Defenchess vs Beef: 16 matches, 3% match

Position file:
Defenchess vs Beef: 11 matches, 5% match
Stockfish vs Beef: 56 matches, 7% match

As a baseline reference, I ran the same comparisons between the other engines:

Stockfish vs Rubichess (search): 32 matches, 3% match
Stockfish vs Ethereal (search) 31 matches, 3% match
Stockfish vs Defenchess (eval): 8 matches, 2% match
Stockfish vs Defenchess (position): 17 matches, 4% match

That 35% verbatim match between Defenchess and Beef is somewhat concerning.

[drstrange767]

In general, Daniel's observation that the search routine is similar to Defenchess' holds truth. Beef's development in the search function initially used a framework based on the one found in Defenchess, and has been diverging, with my own modifications added to pruning, reduction, etc... as I go. I believe that this can be demonstrated if anyone cares to investigate prior versions. (At a glance, most of my small tweaks in the recent git commit history are in the search function) I am dedicated to continuing to make the search routine my own; however, I was encouraged by my fellow devs to: a) for now, add a note in the readme, that says something like : "Beef's search routine uses frameworks implemented in other engines, especially Defenchess..." at the top, and b) rewrite the search portion by simplifying away the old methods and reimplementing them. This, I think, I will also do, seeing as right now there is a bug in PV reporting anyway . I hope the extra attribution for those methods that are closer to derivative is more in the spirit of GPL, and that a new "simple" rewrite of the search would more adequately express my original ideas. Feedback is appreciated. Thank you.

[drstrange767]

Hi all,

Since the last events of this thread, I've passed enough patches, and made enough other changes, that I thought it appropriate to set a checkpoint, so to speak, and push a new release. I appreciate the initial criticisms that Beef received, and hope that the contents of my last 20 or so commits on the repo, if anyone cares to check, show that I am clearly working to address those criticisms. In this latest release, you will find:

a) a modified README with better attribution
b) a search.cpp with reorganized utility functions, removing and reimplementing any remaining functions unnecessarily derived from Defenchess
c) weights that were retuned en-masse

and a handful of original patches of my own creation, whose SPRT tests are now logged in commit messages.

You can find it here:
https://github.com/jtseng20/Beef/releases/tag/v0.3.6