Lag spike at game start. Unable to move and game is aborted after time limit is reached.

Next example is hard to read, not nesccesarily spam. Those requests make most of the problems, as they get often stalled:

You can also see timings on message reguests themselves.

WebSocketv5?sri=2EM8hafMx4G1&v=22 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=2EM8hafMx4G1&v=22 GET 101 websocket socket1.lichess.org websocket 0 B 15.88 s upgrade nginx WebSocketv5?sri=3c7px7DBhxA0&v=23 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=3c7px7DBhxA0&v=23 GET 101 websocket socket1.lichess.org websocket 0 B 16.97 s upgrade nginx WebSocketv5?sri=6jCyQ6EY4KIp&v=23 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=6jCyQ6EY4KIp&v=23 GET 101 websocket socket1.lichess.org websocket 0 B 6.79 s upgrade nginx WebSocketv5?sri=A9Hits96plAP&v=23 /tournament/WIFzR85V/socket/v5 wss://socket5.lichess.org/tournament/WIFzR85V/socket/v5?sri=A9Hits96plAP&v=23 GET 101 websocket socket5.lichess.org websocket 0 B Pending upgrade nginx WebSocketv5?sri=aaG81qGJMxsk&v=7 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=aaG81qGJMxsk&v=7 GET 101 websocket socket1.lichess.org websocket 0 B 8.89 s upgrade nginx WebSocketv5?sri=COZFmLr8Ys0v&v=11 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=COZFmLr8Ys0v&v=11 GET 101 websocket socket1.lichess.org websocket 0 B 5.52 s upgrade nginx WebSocketv5?sri=Ff8qrmAMhix_&v=7 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=Ff8qrmAMhix_&v=7 GET 101 websocket socket1.lichess.org websocket 0 B 13.24 s upgrade nginx WebSocketv5?sri=kysHtX_zn7at&v=17 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=kysHtX_zn7at&v=17 GET 101 websocket socket1.lichess.org websocket 0 B 5.63 s upgrade nginx WebSocketv5?sri=n19sdQv0TIIX&v=10 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=n19sdQv0TIIX&v=10 GET 101 websocket socket1.lichess.org websocket 0 B 6.10 s upgrade nginx WebSocketv5?sri=P2nYXG54CrIe&v=23 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=P2nYXG54CrIe&v=23 GET 101 websocket socket1.lichess.org websocket 0 B 11.50 s upgrade nginx WebSocketv5?sri=QC5odHRfYcHc&v=11 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=QC5odHRfYcHc&v=11 GET 101 websocket socket1.lichess.org websocket 0 B 12.06 s upgrade nginx WebSocketv5?sri=u9Jc6MsUgth4&v=13 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=u9Jc6MsUgth4&v=13 GET 101 websocket socket1.lichess.org websocket 0 B 1.27 s upgrade nginx WebSocketv5?sri=W6eoM283P65J&v=23 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=W6eoM283P65J&v=23 GET 101 websocket socket1.lichess.org websocket 0 B 8.59 s upgrade nginx WebSocketv5?sri=Wjr4t3SbfG_7&v=23 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=Wjr4t3SbfG_7&v=23 GET 101 websocket socket1.lichess.org websocket 0 B 5.75 s upgrade nginx WebSocketv5?sri=xD9_HLZBXdvd&flag=tournament /socket/v5 wss://socket1.lichess.org/socket/v5?sri=xD9_HLZBXdvd&flag=tournament GET 101 websocket socket1.lichess.org websocket 0 B 1.05 s upgrade nginx WebSocketv5?sri=yyn05E1aiLLJ&v=20 /tournament/WIFzR85V/socket/v5 wss://socket1.lichess.org/tournament/WIFzR85V/socket/v5?sri=yyn05E1aiLLJ&v=20 GET 101 websocket socket1.lichess.org websocket 0 B 12.17 s upgrade nginx WebSocketv6?sri=68pkAmCEJvNf&v=0 /play/Q3Buj6WRPAyb/v6 wss://socket1.lichess.org/play/Q3Buj6WRPAyb/v6?sri=68pkAmCEJvNf&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 47.43 s upgrade nginx WebSocketv6?sri=6jtBCCIvZ_FA&v=0 /play/RDSuYehNV2Ql/v6 wss://socket1.lichess.org/play/RDSuYehNV2Ql/v6?sri=6jtBCCIvZ_FA&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 2.1 min upgrade nginx WebSocketv6?sri=9Ua6X_y8uf_n&v=0 /play/1mCYlL0zriwn/v6 wss://socket1.lichess.org/play/1mCYlL0zriwn/v6?sri=9Ua6X_y8uf_n&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 46.02 s upgrade nginx WebSocketv6?sri=_qeSficwWqsK&v=0 /play/oz9XiRFQq898/v6 wss://socket1.lichess.org/play/oz9XiRFQq898/v6?sri=_qeSficwWqsK&v=0 Finished socket1.lichess.org websocket 0 B (unknown) WebSocketv6?sri=_qeSficwWqsK&v=0 /play/oz9XiRFQq898/v6 wss://socket5.lichess.org/play/oz9XiRFQq898/v6?sri=_qeSficwWqsK&v=0 GET 101 websocket socket5.lichess.org websocket 0 B 2.2 min upgrade nginx WebSocketv6?sri=_XLeRQvaRRNI&v=0 /play/Yh0S1s5jEc1M/v6 wss://socket1.lichess.org/play/Yh0S1s5jEc1M/v6?sri=_XLeRQvaRRNI&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 2.0 min upgrade nginx WebSocketv6?sri=b7j1OzgxJ0lJ&v=0 /play/weKduVYiOIr4/v6 wss://socket1.lichess.org/play/weKduVYiOIr4/v6?sri=b7j1OzgxJ0lJ&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 2.1 min upgrade nginx WebSocketv6?sri=EN4reiGByWp_&v=0 /play/f32V2syXU0oZ/v6 wss://socket1.lichess.org/play/f32V2syXU0oZ/v6?sri=EN4reiGByWp_&v=0 GET 101 websocket socket1.lichess.org websocket 0 B 1.4 min upgrade nginx

looks like those previous messages contain mainly chat presence, public presense in chat or tournament and etc.
Next example message -got stalled for 8,5 seconds:
4
18:31:33.207
{"t":"startWatching","d":"aIfPXkp0"} 36
18:31:33.207
{"t":"reload"} 14
18:31:33.314
0 1
18:31:33.356
{"t":"fen","d":{"id":"aIfPXkp0","lm":"e7e6","fen":"r1bqkbnr/pp1p1ppp/2n1p3/2p5/4P3/1Q6/PPPP1PPP/RNB1KBNR w","wc":59,"bc":57}} 125
18:31:33.356
{"t":"fen","d":{"id":"aIfPXkp0","lm":"b3f3","fen":"r1bqkbnr/pp1p1ppp/2n1p3/2p5/4P3/5Q2/PPPP1PPP/RNB1KBNR b","wc":58,"bc":57}} 125
18:31:33.430
{"t":"crowd","d":{"nb":20,"users":["cadetblue","pirkarpenter","Shaurya1303","tom230","BuDDhInIAM","dimoss","ppermas_3_0","bozkurtfrkn","GenaBerezov","Diksha_Alluri","kudzu12","basile73","Basico2k","lestrand","Emmanuelv7","chumbichess","Manova-jeu","theindiansniper","Ishanth43","Gleb_6_let"],"anons":0}} 303
18:31:33.954
{"t":"fen","d":{"id":"aIfPXkp0","lm":"g7g6","fen":"r1bqkbnr/pp1p1p1p/2n1p1p1/2p5/4P3/5Q2/PPPP1PPP/RNB1KBNR w","wc":58,"bc":56}} 127
18:31:34.215
{"t":"reload"} 14
18:31:34.446
{"t":"fen","d":{

There are also some random errors like this:
lichess.min.js:36 WebSocket connection to 'wss://socket1.lichess.org/play/oz9XiRFQq898/v6?sri=_qeSficwWqsK&v=0' failed: WebSocket is closed before the connection is established.
I.disconnect @ lichess.min.js:36
I.destroy @ lichess.min.js:36
I.connect @ lichess.min.js:36
(anonymous) @ lichess.min.js:36
setTimeout (async)
I.scheduleConnect @ lichess.min.js:36
I.connect @ lichess.min.js:36
I @ lichess.min.js:36
q @ round.min.js:1
(anonymous) @ oz9XiRFQ:9
Promise.then (async)
(anonymous) @ oz9XiRFQ:9

And some cosmetic bug -in every page load:
Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.

22lichess.min.js:33 Uncaught TypeError: Cannot read properties of undefined (reading 'cancel')
at HTMLDivElement.mouseenter (lichess.min.js:33:3639)
at HTMLDivElement.ya (lichess.min.js:21:306)
mouseenter @ lichess.min.js:33
ya @ lichess.min.js:21
22lichess.min.js:33 Uncaught TypeError: Cannot read properties of undefined (reading 'hide')
at HTMLDivElement.mouseleave (lichess.min.js:33:3706)
at HTMLDivElement.ya (lichess.min.js:21:306)
mouseleave @ lichess.min.js:33
ya @ lichess.min.js:21
Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.

kudzu12,

They will be reverting the widescale migration to scala3 because of performance problems soon, so I'm very interested if this behavior continues for you after the next restart.

@kudzu12 said in #21:
> Next example is hard to read, not nesccesarily spam. Those requests make most of the problems, as they get often stalled:

Thanks, I don't fully understand these comments but added them anyway:
github.com/lichess-org/lila/issues/11989#issuecomment-1336481784

@kudzu12 @klelik

Just touching base because one of you mentioned this had been worse in recent weeks. Is the tremendous spike at the start of game still occurring regularly?

Depends.

Fast answer - no tremendous spike on last week or so.

Overall after server was moved from scala3 to scala2 it was much better, almost like "good old days"
After that, soon was server back to scala3. At first, it was quite terrible for some next days. (and i don't bother complaints about analyses, chat languages support or arrows on analyses boards)
Then this lag (i found term LAG inappropriate in this context, as this is not about lag at all) did get more random.

Now there looks like is not such big start trouble on tournament beginning or countdown, but overall sluggishness is very present on some days and times and disappearing on next tournaments or hours.
It feels like something is throttling, and those are not local machine resources nor internet connection itself. (still consistent ping around 50 ms and not single packet lost.)

Overall, if you look closely on feedback topics, i am pretty sure you find many many complaints like time was running out and etc, what are also not network problems.

But looks, like enthusiasm on solving this kind of troubles is not very high amongst server side support personnel.
I still am pretty sure, that monitoring of server farm and database or network devices performance is needed for sorting things out. But as you see yourself, there is no any news, that this kind of analyses is done on server-side or results about.

And as seconds of "move time" are lost while you wait for actual opponents move - to see it for response (if playing without premove in critical positions) it makes one feel like play is clued to screen. I am old Quake player, with medium reaction time like 0,12-0,13 sec on FPS games, i am not so slow to make a move on 0,3-0,5 secs while waiting opponents response. Overall, it is kinda rising question, were is move time actually lost with ping 50 ms and reaction time say 0,2 secs.
And - most interesting is, that all that may change on some next hours. You see move and make your own move, everything moves fine and fast as it should be.

So it depends.
Currently, looks like, observing server performance is too complicated and why bother with improvements, if people still play.
You see, am still here too :)

But it could be much smoother than it is now. Interestingly, playing with mobile platform - (android) is much smoother.Maybe here can be some hints too, why does mobile platform better.

Merry Christmas!

OK, news update about this one

@schlawg said in #24:
> @kudzu12 @klelik
>
> Just touching base because one of you mentioned this had been worse in recent weeks. Is the tremendous spike at the start of game still occurring regularly?

After my last answer post i started logging client-side again regularly. There is now change, different from previous days.
I hope that is not occasion...

So web-sockets still get some slow responses, but now those are shorter. Daily bullet 1 hour event had exactly 7 long socket responses. Client is waiting for server response - exactly 1,06-1,07 seconds now. that we can call stable response
I could say, something is improved (not cured yet). Before waiting time was random, much longer and it could be stalled longer or had TLS phase issues. Looks like socket does not drop connection anymore?
If there is any change made on server side- keep going.