Language:
switch to room list switch to menu My folders
Go to page: First ... 38 39 40 41 [42] 43 44 45 46 ... Last
[#] Sun Nov 08 2009 11:04:56 EST from davew @ Uncensored

Subject: select()

[Reply] [ReplyQuoted] [Headers] [Print]

I'm after a bit of clarification on how select() works.

I know we've been down this road before but....

So my man page says that the reurn value from select is the number of fd's that were ready.

If one has a multithreaded programme where each thread goes into select() with the same fd sets and then multiple fd's trigger at the same time causing one of the threads to return from select() with a value greater than one but the thread then only deals with one fd before going back into select() what happens to the other fd's that had trigered the original select()?

 



[#] Sun Nov 08 2009 18:11:05 EST from Ford II @ Uncensored

Subject: Re: select()

[Reply] [ReplyQuoted] [Headers] [Print]

value greater than one but the thread then only deals with one fd
before going back into select() what happens to the other fd's that
nothing. they continue to be in a (I assume you're selecting on read, but it doesn't really matter) ready-to-read state and if you select again they will come back as ready again.
If you read the man page it says you're supposed to handle all the results of a select after you call it.
you don't HAVE to, but they suggest it, probably because it's more efficient.
Actually you made me think of a better question, and I know I used to know the answer to this, but it was long ago.

What happens if you have a process with multiple threads blocking on select on the same set of sockets, and they all get data, all the threads wake up and start handling data. One of them is going to win, the second one will lose, but select will have said there was data to read but when you go to read, there will be nothing left because the first thread got to it first.
I guess you're supposed to handle the synchronization yourself to avoid that mess.

[#] Sun Nov 08 2009 18:14:14 EST from Ford II @ Uncensored

Subject: Re: select()

[Reply] [ReplyQuoted] [Headers] [Print]

actually I think you asked the question I said and I misread you.
The answer still is nothing: it's your problem.
You don't want multiple threads handling the same socket, so what you should do is set up a queue or some logical equivalent about who handles what so it's not possible for multiple threads to try and process the same socket.
I think that's what citadel does.

[#] Mon Nov 09 2009 11:11:45 EST from Ford II @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

[#] Mon Nov 09 2009 11:14:11 EST from fleeb @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]


Oy.

[#] Mon Nov 09 2009 13:10:12 EST from LoanShark @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]


davew, you should never have multiple threads calling select() for the same set of sockets (or overlapping sets of sockets) at the same time. They will all wake up at the same time and cause a load spike.

Possible solutions include wrapping the call to select() in a mutex, or using a worker thread model where only one thread is reponsible for calling select(), and this thread passes units of work to other threads.

The solution I use in my multithread server at work, is to have all threads call accept() at once, which works perfectly fine. One thread "wins" each accept() call and the others do not wake up, and the winner thread processes its one connection (the one that was returned from accept()) until it is time to close that connection, at which time the thread continues its outer loop and goes back to calling accept().

[#] Mon Nov 09 2009 14:44:19 EST from Ford II @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

I have a question about that, if you know the answer.
I got into a big fight with a guy at prodigy a long time ago about the merits of, oh let's see, I forget the argument already.
I had at that point become a big fan of your technique, but rather than threads, I do it pulse's way: parent process creates listen socket, fork children, they all block on accept. Same concept, I dislike threads (oh to have this conversation with my 1993 self).
Anyway, his point was something about shit I wish I could remember. I remember he was terribly against select, because it was slow, I've heard this argument before, and it seems to me the multi-thread/process accept would be roughly the same thing except for the two system calls, but the actual divining of the information about what sockets are ready to be dealt with would be about the same, no?
I mean internally I have no idea how the OS implements select or how it's scheduler picks a process to wake up, but I gotta figure it's pretty similar.

[#] Mon Nov 09 2009 14:44:28 EST from Ford II @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

does that question even make any sense?

[#] Mon Nov 09 2009 18:23:37 EST from davew @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Thanks for all the comments on select() guys.

Given what you have said would someone like to retrieve a copy of the source for Citadel and take a look at a piece of the code for me.

I'm talking about the worker_thread function in sysdep.c

I must be missing something because I can't see how the hell this can work safely given the consesus.

Which probably explains a few things.



[#] Mon Nov 09 2009 18:37:30 EST from Ford II @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

I assume you're talking about worker_thread() ?
I'm not sure I get the whole picture, but it looks like any available thread can do the select, and start dishing out work.
it calls ctdlthreadselect to do the select, then it start's a critical section and starts going through context list which I assume is a list of sessions, and hands out sockets with work to do and sets their state to running.

Way up top just after the big while it makes a critical section to gather up the list of contexts that are idle and should be put in the state of select-on-read to look for work to do.

At the moment I don't see what stops multiple threads from doing the select though.
It might be a larger picture thing than just what's going on in this function.
There might be only one thread that does this, and all other threads are little worker bee threads doing the work that the main selecting thread is dishing out.
I defer to ig or ls, as I think one of them wrote it. :-)

[#] Mon Nov 09 2009 18:55:13 EST from davew @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Ford,

Thanks that is my interpretation.

All threads go through this function. I was the last person to make changes in there but it has always been basically the same.

I thought it was a bit naughty.

I'm going to be fixing that sooner than I thought then :-)

 



[#] Mon Nov 09 2009 22:09:28 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

At the time I wrote the original server, I naively assumed that select() was level triggered rather than edge triggered, and we just jumped onto the first fd that select() returned.

Later on, it was pointed out that select() is edge triggered on some platforms and level triggered on others. The bigger problem, however, was that my naive code was significantly favoring sessions that appeared towards the beginning of the list. So I rewrote it, with more than a little advice from LS (and I seem to remember there was also a bungee developer involved at the time who wanted us to add a dependency to libevent, but thankfully we didn't do it).

The way the Citadel server works now is, a worker thread calls select() on all of the sockets (both rendezvous sockets and sockets bound to sessions) and figured out which ones are active.

Then we change the state of *all* of the sessions associated with those sockets from CON_IDLE to CON_READY. *One* of those sessions then moves to CON_EXECUTING and the worker thread performs a server transaction. Other worker threads continue to perform server transactions on CON_READY sessions, switching them back to CON_IDLE when finished, until there are no more CON_READY sessions. Then, and only then, we go back to the beginning and call select() again.

WebCit is much more simpole, because it's only processing a single protocol, and since we don't implement HTTP keepalive there are never any existing sockets to listen on. We just call accept() on the socket, perform a transaction, and close it.

[#] Mon Nov 09 2009 22:16:23 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

The solution I use in my multithread server at work, is to have all
threads call accept() at once, which works perfectly fine. One thread

"wins" each accept() call and the others do not wake up, and the winner


Really? Doesn't that go against the generally understood behavior of the "thundering herd" problem? O'
I may be going from outdated knowledge here, but I thought that if you just let all threads call accept(), then when a connection comes in, ALL threads wake up, one thread "wins" the socket, and all of the other threads go back to sleep, creating a lot of unnecessary load.

In WebCit we wrap the accept() call in a mutex so that only one thread calls it at a time. Is my understanding of this incorrect or outdated?

[#] Tue Nov 10 2009 09:51:25 EST from Ford II @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

I may be going from outdated knowledge here, but I thought that if you

just let all threads call accept(), then when a connection comes in,
ALL threads wake up, one thread "wins" the socket, and all of the other

threads go back to sleep, creating a lot of unnecessary load.

Nope. One thread/process gets woken up, all the others remain in a blocking state.

[#] Tue Nov 10 2009 12:01:44 EST from davew @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

 

Tue Nov 10 2009 09:51:25 AM EST from Ford II @ Uncensored

Nope. One thread/process gets woken up, all the others remain in a blocking state.

Aha, That is what is saving Citadel.

IG's description is a little off from what actually happens.

If there is no activity then every thread will, in turn, construct a list of ALL the client socket fd's (there is a mutex around these as they are in contexts). Then each thread will add ALL of the master socket fd's to its list and then each thread will call select().

So if there is no activity EVERY thread will eventually end up in select with the exact same fd sets. Yes I know Citadel times out its select after 1 second but it is possible that for a period every thread is in select() with the same fd sets.

So if during the period that every thread is in select some activity occurs on one or more sockets then every thread awakens and the race is on.

Every thread sets off towards accept()

Obviously we only accept on the master sockets so there is a loop that iterates through each master socket and calls accept() on it.

First thread to accept() on one particular master socket wins, the others then fail the accept() on that socket and continue itterating the loop. Winning thread does connection setup and goes back to rebuild its fd set.

Any threads that do not manage an accept() on any master socket fall through to process client sockets which are protected by a mutex again and thus one thread one client socket.

And so the actual problem is revealed.

Assume the following:

5 threads in the system.

3 clients currently connected so 3 client sockets but clients are idle at moment.

Thus all 5 threads are in select.

At the very same instant 10 new clients connect to the master sockets AND the 3 existing clients become active.

All 5 threads awaken with all of the fd's set (say 13, 10 new clients and 3 existing)

Each thread manages an accept() on one master then rebuilds its fd's and returns to select()....No existing clients preocessed.

If select() is edge triggered we wait one second, if not all 5 threads exit select() and manage 1 accept() each before returning to select()

We wait a further second (or not as the case may be) before each thread awakens again and this time they get to the bit that handles the three existing clients who probably had to wait at least 2 seconds to have a go.

I'd say there is much room for improvement there.

I REALLY hope you guys are correct because your information is going to be the basis for changing this.

 



[#] Tue Nov 10 2009 12:19:47 EST from skpacman @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

omg.. i feel like a SuperN00b asking about this, but i need help with some php...

I want to convert this:

if($vbulletin->options['online_staff_separate_onoff']){
        if (
fetch_online_status($loggedin)){

        if(
$loggedin['usergroupid'] == '5' OR $loggedin['usergroupid'] == '6' OR $loggedin['usergroupid'] == '7' OR $loggedin['usergroupid'] == '14' OR $loggedin['usergroupid'] == '9' OR $loggedin['usergroupid'] == '13' OR $loggedin['usergroupid'] == '10'){
        
                eval(
'$activeusers_staff .= ", ' fetch_template('forumhome_loggedinuser') . '";');
                
$activeusers_staff substr($activeusers_staff2);
                    
        } else {
        
                eval(
'$activeusers_members .= ", ' fetch_template('forumhome_loggedinuser') . '";');
                
$activeusers_members substr($activeusers_members2);
            
        }

    }

into a switch() function... how would i go about doing that? the switch would be  switch($loggedin['usergroupid'])

your help would be greatly appreciated..



[#] Tue Nov 10 2009 12:25:50 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

If there is no activity then every thread will, in turn, construct a
list of ALL the client socket fd's (there is a mutex around these as
they are in contexts). Then each thread will add ALL of the master
socket fd's to its list and then each thread will call select().

Not every thread. One thread at a time. If you look closely, you will see that the section of code which calls select() is wrapped in a mutex so that only one thread does it at a time.

[#] Tue Nov 10 2009 12:26:35 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Nope. One thread/process gets woken up, all the others remain in a
blocking state.

I think that's called "wake one" ? Is that the default behavior on Linux now? It hasn't always been that way, has it?

(I'm guessing LS will have the most reliable knowledge of this)

[#] Tue Nov 10 2009 12:32:25 EST from IGnatius T Foobar @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

Not every thread. One thread at a time. If you look closely, you
will see that the section of code which calls select() is wrapped in a

mutex so that only one thread does it at a time.

Or at least we used to. Someone (possibly you?) may have changed it.

[#] Tue Nov 10 2009 12:40:07 EST from davew @ Uncensored

[Reply] [ReplyQuoted] [Headers] [Print]

 

Tue Nov 10 2009 12:25:50 PM EST from IGnatius T Foobar @ Uncensored
If there is no activity then every thread will, in turn, construct a
list of ALL the client socket fd's (there is a mutex around these as
they are in contexts). Then each thread will add ALL of the master
socket fd's to its list and then each thread will call select().

Not every thread. One thread at a time. If you look closely, you will see that the section of code which calls select() is wrapped in a mutex so that only one thread does it at a time.

NO, IT IS NOT.

If it were the probable mutext would be S_SESSION_TABLE which other threads use when parsing the list of contexts which would cause deadlock or at least ver slow client processing.

 



Go to page: First ... 38 39 40 41 [42] 43 44 45 46 ... Last