identifying the cpu a go routine is running on?

Is there anyway of telling which cpu/core a goroutine is actually running on?

Possibly by scrobbling in /proc (on linux), but that is may cause your
thread to move to another CPU.

What is the problem you are trying to solve ?

Dave
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group, send email to
> golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

So if I generate 100 goroutines on a 4-core machine I'd like to know how many threads exist on each core. I can readily profile the runtimes, but I can't grok whether they're all on a single core or split across all four or some variety in between. When the goroutines hang up (and emit a value to an exit channel I'd like them to emit the id of the core they were last present on).

Like most things, it depends. On the value of GOMAXPROCS, on how many
blocking syscalls are currently executing, on the other things going
on on your host at that time, on the overhead of whatever profiling
you are using, etc.
even if syscall getAfinity was implemented, it wouldn't work
correctly, blocking syscalls spawn a new thread to do the blocking.

I think you should be very clear about what you are trying to achieve.
You appear to be trying to benchmark lock contention, is that correct
?

Yes, I could have been clearer, I am trying to do some simple benchmarking of lock contention. I appreciate all the assumptions/dependencies etc but it would be good to know if this is possible at all (without hacking the  go src in runtime or making C calls). My first grok through the packages and exposed APIs suggests no.

Thanks, but I don't think either the benchmarks or the profiling provide the information I'm after i.e. which core did the go routine execute on. Contention for a lock will clearly be different if both routines run in the same core, cpu or same socket as opposed to running distributed across cores/cpu/sockets in a shared memory system.

Thanks for the suggestions, getcpuid and the instrumented mutex is the closest to what I needed.

It's a shame not all architectures allow user level access to getting cpuid.

In response to the code you listed on github, you probably want to use a slice (or array) for the counter inside the instrumented mutex. The map introduces a 10x hit on lock/unlock of the mutex. Additionally you probably only need to update the counter on the Lock method.

Providing a function that returns which core a goroutine is running on
does not help to implement your more efficient lock, because a
goroutine may move to a different core between the Lock and Unlock
calls. I have no particular objection to your idea, but
WhichCoreAmIOn does not seem like the right primitive.

Given a limited number of worker goroutines, you can implement what
you want more awkwardly by using a []*sync.RWMutex, assigning one to
each goroutine, having that goroutine use that for read locks, and for
write locks acquiring all the mutexes.

Ian

> > ... have an RWMutex *per core*, where goroutines only call RLock()
While it is true that a goroutine *may* be moved to a different core,
the scheduler presumably still attempts to avoid moving a goroutine
unnecessarily to not completely trash the L123 caches? As long as this
is the case, this optimization would help (and not violate correctness).
I agree it might not be the right primitive, but I haven't been able to
come up with a better alternative.

Also, even if this is not the case with Go's current scheduler, it seems
likely that this will be added sooner rather than later. It is becoming
increasingly expensive to move computation between cores as the number
of CPU cores increases, since cores are necessarily further apart,
making cross-core communication more expensive. The cache hierarchy
between (especially some) cores also makes moving computations more
expensive. For example, in a NUMA system, you wouldn't want goroutines
to bounce all over the place, as this would kill performance.
I don't see how that would work?
The writer would have to lock the list to ensure that new readers aren't
added while it is doing work on shared data, and readers would have to
RLock (and thus increment) that "meta"-lock, which would have the same
problem as the original.

Jon

You might find atomic.Value useful: http://golang.org/pkg/sync/atomic/#Value

-josh

I'm not sure how atomic.Value would work as a replacement for a lock
except by doing busy-waiting with runtime.Gosched()? The lock should
*block* if someone else is writing.

Jon

You can use a separate sync.Mutex to coordinate writes (which are
posited to be rare), and do copy-on-write.

It's not right for all uses, thus the "might" in "you might find it to
be useful". It has the advantage of being available for use right now
and scaling across cores very well.

-josh

Ah, of course. Yes, you're right, I just didn't think that one step
further. For applications that update a single value, or where copies
can be made reasonably cheap, this would work fine. It does potentially
require significant code changes though.
Unfortunately, in my particular case, this approach wouldn't work. I
have a number of goroutines that are all allowed to change the
underlying data structure concurrently, but then one particular
goroutine that needs an exclusive lock. The concurrent modifiers all
take out RLocks, but the "special" writer needs an "real" WLock where
all readers are prevented from accessing the critical section.

Jon

Wouldn't Load()s from atomic.Value also come at the additional cost of
interface{} conversion, including (potentially) an extra memory
allocation per read?

Writes may alloc, but reads shouldn't. Yes, there's an interface
conversion, but they're pretty cheap, and they'll be cheaper yet in Go
1.5.

-josh

I don't understand how your suggestion works if a goroutine moves
between cores. On core 1 the goroutine locks the lock attached to
that core. Then it moves to core 2. Then it unlocks the lock
attached to that core, but that's the wrong lock.

I guess you would suggest that the goroutine should call a function to
get a pointer to the lock, where the pointer is based on the current
CPU, and should then use that pointer to acquire and release the lock.
I suppose that would work.

You could do that today on GNU/Linux using
syscall.Syscall(syscall.SYS_GETCPU, ...)
Might be interesting to try.
All true, but, because goroutines can block for arbitrary lengths of
time, we are always going to want to reserve the right to move them
between cores when necessary.
True, I'm assuming not only a limited number of worker goroutines, but
a fixed number.

Ian

I was thinking of the goroutine just keeping track of which lock it
acquired (i.e. its index in the array of locks), but using pointers for
it works too.
I don't see a syscall.SYS_GETCPU in the syscall package?

I'm also worried that doing a syscall on every lock acquisition might
introduce quite a lot of overhead in and of itself (potentially more so
than the atomic add performed in normal RWMutex), and so it might not be
a good proxy for the gains one might see with a "proper" solution.
Presumably the runtime knows what CPU it's running on already, and
wouldn't need to ask the kernel?

FWIW, after poking around a bit, it seems like this approach to RW
locking is far from new. Linux calls then "big reader locks"
(https://lwn.net/Articles/378911/). Benchmarks have shown that this kind
of locking displays much better scaling properties than for a shared RW
lock
(http://joeduffyblog.com/2009/02/20/a-more-scalable-readerwriter-lock-and-a-bit-less-harsh-consideration-of-the-idea/),
though it does also add overhead for writers.

There's some interesting newer work in this field too:
https://www.usenix.org/system/files/conference/atc14/atc14-paper-liu.pdf
though that is beyond the scope of the change I want to make.
Oh, sure, that's true. I was more referring to the fact that the
scheduler will (hopefully?) not move goroutines unless some core is
being starved for work.
Ah, I see. I suppose the application might spawn GOMAXPROCS goroutines
on startup, each with its own lock, and reap some of the benefits above.
I suspect it might be hard to program that way though, as you now loose
some of the neat benefits of lightweight threads (e.g. you can't spawn
off a bunch of readers of the lock).

That's interesting, it's there for every GNU/Linux target except
amd64. That's an odd oversight.
The Go runtime does not in fact know which CPU a goroutine is running
on. Why would it? The runtime knows which thread a goroutine is
running on, but that is not, of course, the same thing. The runtime
leaves it to the kernel to keep threads preferentially on a single
CPU.

Ian

> algorithm for this use case called "RCU" due to scale-ability issues

RCU is great if the data you are replacing can be copied cheaply, but
not so much if you are changing a small part of a large dataset
(especially if it contains pointers internally). See also the discussion
with Josh elsewhere in this thread.

RCU also doesn't work for cases where you need a "dual-mode" lock
more-so than a RWlock. For example, in my particular case, the "readers"
all modify the underlying data, but they do so in a way that is safe in
the face of concurrent operations. The "writers" perform operations that
require the underlying data not to change while they are operating,
meaning they really do need an exclusive lock.
Sure, but I don't think sync.RWMutex could be re-implemented solely
using RCU, in particular because it's not right for all use-cases for RW
locks.

Jon

What is CSP scheduler? And why does RCU require a non-preemptive
scheduler (linux kernel is obviously preemptive)?
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

I've implemented a distributed RW mutex for Go back in 2011:
https://groups.google.com/d/msg/golang-nuts/q59mZRC16KE/WeNJhVvZxRAJ
Nobody was interested enough to push it into std lib.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Ah, that's interesting.

My guess is that the fact that your solution requires more integration
into the go runtime than mine does makes it a less likely candidate for
adoption (at least in the short term); I just need to know which CPU I'm
currently on, whereas you need thread-local storage and guarantees about
what happens if a goroutine is moved to another thread.

Admittedly, your solution might also have applications beyond RWMutex;
knowing what CPU a goroutine is currently on seems like a feature with
very limited applicability.

Jon

No, they always could have been preempted for GC at some points. And
since several thread can run in parallel these points were non even
deterministic.
I know what it RCU. It works fine with preemptive schedulers.

On GC completion, the pre-empted goroutine would not get resumed?
Not questioning your knowledge, Dmitry. Ref was to the source of my info.

Most likely it will be replaced by another goroutine from runqueue.
This effectively gives preemptive scheduling (at least the worst parts
without the good parts).

I was right that the syscall approach would slow down the lock too much,
but I managed to get something working using the CPUID assembly
instruction instead. The performance improvements are quite dramatic:

https://gist.github.com/jonhoo/05774c1e47dbe4d57169

Jon

Sorry, but I didn't get it: what's the matter with []*sync.RWMutex, and each goroutine using one? This way correctness is a fact (no need to know which CPU is the goroutine is running ATM), performance I don't know.

This would require you to know *in advance* how many goroutines you are
going to have. Say you have n goroutines and k cores, you will also need
n > k locks, which means both increased time for writers and additional
storage overhead.

Why would that be so bad? You won't have more than 2*GOMAXPROCS goroutines doing computation, do you?
But you can always limit the number of locks, with a small increase of contention on some of the locks.

As I understand, having multiple locks is to slice contention, and each additional lock slices less, and more than CPU locks is waste.

My suggestion is to try this simple and correct solution, before try hard to circumvent the runtime.

If you use goroutines in the "Go way" where you spawn off a goroutine to
handle each request, you might have thousands of goroutines, but only a
handful of cores. This is also beneficial in that it allows overlapping
computation and IO, which having a single (or very few) goroutines per
core would not.

What you are proposing is keeping a worker pool, which introduces
additional complexity to request handling, and additional communication
through some shared queue that keeps incoming requests. While this is
possible to implement, it is fairly uncommon to see in Go where
lightweight threads often obviate the need for such pools.
That is true, though then you're back to the original problem of a
lock's memory location bouncing back and forth between the cores that
have readers of that core. My guess is that you would still see a
significant slowdown once goroutines on different cores (or even
different NUMA nodes) share the same lock.
Well, it's not quite to slice contention, because a RWMutex is in a
sense uncontended if you only have readers. The problem is that the way
RWMutex is implemented; since RLocks still do a memory write, the CPUs
are contending for write access to a shared memory location, causing
them to constantly invalidate each others' caches.
The solution I proposed is *always* correct, because the reader will
unlock the same lock as they locked. Even if cpu() returns the wrong
value, correctness is guaranteed. A wrong cpu() value will only impact
performance, as it will cause a CPU to invalidate another CPU's cache.

I'm not sure I'd call this circumventing the runtime. In a sense, this
has nothing to do with the runtime at all. While it would be nice if the
runtime provided a runtime.GetActiveCPU() function, this is not
necessary for a better RWMutex to be implemented; all it means is that
the RWMutex would have to implement the function to get the current CPU
itself.

It is also worth pointing out that an entirely safe fallback
implementation of runtime.GetActiveCPU() is "return 0", because any
implementation that relies on it *must* be prepared for the case where a
goroutine has been moved between the function call and when it is
executing subsequent instructions. This will reduce performance, but not
impact correctness.

Jon

Thanks for your patience, and detailed answer, Jon!

Then instead of a CurrentCPU() call, a goroutine "identifier" (a number from a sequence, given at start time, maybe hidden in the per-goroutine lock implementation) would do modulo the number of underlying RWLocks, which would be equal with the new number of cores?

No worries at all. It is good to get all these thoughts out in writing
so that people can fully understand the change.
This would still have the issue that goroutines running on different
cores access the same lock, which is what causes the performance
problems at a large number of cores.

So, we only need the cpuid for select which lock shall we acquire, but for releasing it, stick with that lock.

Now (I think) I understand why your proposal is correct, even if the CurrentCPU function returns bogus data (say: check for the CPU only for every 10th call).

Thanks for the enlightenment!