60086 3 false 2009-07-29T07:58:49-07:00 2171 46 ruby-thread-problems 123121 25719 new ruby 2009-08-30T11:50:03-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 Calls to net/http are blocking (and never returning) if they're inside a thread. In the following code, the call to open never returns. @@@ ruby require 'open-uri' puts "Creating thread" Thread.new do puts "Calling net/http in thread" puts open('http://google.com').read puts "Should get called" end @@@ If you run the code normally outside Titanium it works as expected. I have a feeling the problem may be that the main thread is being used up. irb has similar behavior to Titanium in that regard. Calls to net/http are blocking (and never returning) if they're inside a thread. In the following code, the call to open never returns. @@@ ruby require 'open-uri' puts "Creating thread" Thread.new do puts "Calling net/http in thread" puts open('http://google.com').read puts "Should get called" end @@@ If you run the code normally outside Titanium it works as expected. I have a feeling the problem may be that the main thread is being used up. irb has similar behavior to Titanium in that regard. <div><p>Calls to net/http are blocking (and never returning) if they're inside a thread.</p> <p>In the following code, the call to open never returns.</p> <pre><code class="ruby"> require 'open-uri' puts &quot;Creating thread&quot; Thread.new do puts &quot;Calling net/http in thread&quot; puts open('http://google.com').read puts &quot;Should get called&quot; end </code></pre> <p>If you run the code normally outside Titanium it works as expected.</p> <p>I have a feeling the problem may be that the main thread is being used up. irb has similar behavior to Titanium in that regard.</p></div> 47924 0 Calls to net/http are blocking (and never returning) if they're inside a thread. In the following code, the call to open never returns. @@@ ruby require 'open-uri' puts "Creating thread" Thread.new do puts "Calling net/http in thread" puts open('http://google.com').read puts "Should get called" end @@@ If you run the code normally outside Titanium it works as expected. I have a feeling the problem may be that the main thread is being used up. irb has similar behavior to Titanium in that regard. <div><p>Calls to net/http are blocking (and never returning) if they're inside a thread.</p> <p>In the following code, the call to open never returns.</p> <pre> <code class="ruby">require 'open-uri' puts "Creating thread" Thread.new do puts "Calling net/http in thread" puts open('http://google.com').read puts "Should get called" end</code> </pre> <p>If you run the code normally outside Titanium it works as expected.</p> <p>I have a feeling the problem may be that the main thread is being used up.<br> irb has similar behavior to Titanium in that regard.</p></div> false 2009-07-13T04:16:17-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-13T04:16:18-07:00 2171 Alex MacCaw Alex MacCaw Marshall Culpepper http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 0 false 2009-07-18T18:36:04-07:00 2171 --- :assigned_user: 47924 46 ruby-thread-problems 0 25719 new ruby 2009-07-18T18:36:08-07:00 60086 Martin Robinson Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 0 I've done a bit more work with this. It looks like when you interact with Ruby (i.e. call out to it) - the thread gets run a bit, then re-suspended. <div><p>I've done a bit more work with this.</p> <p>It looks like when you interact with Ruby (i.e. call out to it) - the thread gets run a bit, then re-suspended.</p></div> false 2009-07-21T03:38:17-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-21T03:38:17-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 0 There are some relevant comments on this page: http://aeditor.rubyforge.org/ruby_cplusplus/index.html#id2937596 Basically, it looks like Ruby embedded+Thread does not jive at all with C/C++. The workaround is to use: <pre><code> fork do # your code here end </code></pre> <div><p>There are some relevant comments on this page:<br> <a href= "http://aeditor.rubyforge.org/ruby_cplusplus/index.html#id2937596">http://aeditor.rubyforge.org/ruby_cplusplus/index.html#id2937596</a></p> <p>Basically, it looks like Ruby embedded+Thread does not jive at all with C/C++. The workaround is to use:</p> <pre> <code> fork do # your code here end</code> </pre></div> false 2009-07-24T15:56:46-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-24T15:57:09-07:00 47924 Marshall Culpepper Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 0 Ok, if we're going down the fork road, we also have to work out how to get the processes talking to each other. Names pipes aren't cross platform, and DRB is ugly for this kind of thing. I think we should upgrade to ruby 1.9 which uses native threads - this is much more similar to the way Python does it and should work: http://www.infoq.com/news/2007/05/ruby-threading-futures <div><p>Ok, if we're going down the fork road, we also have to work out how<br> to get the processes talking to each other. Names pipes aren't cross platform, and DRB is ugly for this kind of thing.</p> <p>I think we should upgrade to ruby 1.9 which uses native threads - this is much more similar to the way Python does it and should work:</p> <p><a href= "http://www.infoq.com/news/2007/05/ruby-threading-futures">http://www.infoq.com/news/2007/05/ruby-threading-futures</a></p></div> false 2009-07-25T01:24:16-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-25T01:24:18-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 0 Ok, so I've ported Titanium to Ruby 1.9 - there are a few issues with require, but apart from that, everything seems to worked. Here's how I build Ruby 1.9 ./configure --enable-shared --enable-pthread --prefix=/usr/local --program-suffix=19 I've attached the patch. However, the bad news is that the thread issue remains - we're going to have to look at booting up a separate processes (like BrowserPlus does) <div><p>Ok, so I've ported Titanium to Ruby 1.9 - there are a few issues with require, but apart<br> from that, everything seems to worked.</p> <p>Here's how I build Ruby 1.9</p> <p>./configure --enable-shared --enable-pthread --prefix=/usr/local --program-suffix=19</p> <p>I've attached the patch.</p> <p>However, the bad news is that the thread issue remains - we're going to have to look at booting up a separate processes (like BrowserPlus does)</p></div> false 2009-07-29T03:24:03-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T03:24:04-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Ok, so I've been doing a bit more thinking on the subject - forked processes isn't going to work. Reason being the parent process will have to listen out on the stdout of the child process, which will block. Ok, so another plan. Put Ruby in a separate C++ thread, and make sure any calls going to/from Ruby are executed in the main thread. Will that work? How does Python get round this? Has anyone tried Threads in Python? <div><p>Ok, so I've been doing a bit more thinking on the subject - forked processes isn't going to work.</p> <p>Reason being the parent process will have to listen out on the stdout of the child process, which will block.</p> <p>Ok, so another plan. Put Ruby in a separate C++ thread, and make sure any calls going to/from Ruby are executed in the main thread.<br> Will that work?</p> <p>How does Python get round this? Has anyone tried Threads in Python?</p></div> false 2009-07-29T07:34:14-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T07:34:21-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Python uses native threads underneath, so things are fine as long as you surround non-Python with a special call to Python signaling that other threads can run. When you need to call Python functions or execute Python code, you must first obtain the GIL and then run your code. Does Ruby have a similar mechanism? <div><p>Python uses native threads underneath, so things are fine as long as you surround non-Python with a special call to Python signaling that other threads can run. When you need to call Python functions or execute Python code, you must first obtain the GIL and then run your code.</p> <p>Does Ruby have a similar mechanism?</p></div> false 2009-07-29T07:58:49-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T07:59:13-07:00 60086 Martin Robinson Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Found this, does it help? http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/10252 <div><p>Found this, does it help?</p> <p><a href= "http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/10252"> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/10252</a></p></div> false 2009-07-29T08:10:42-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T08:10:43-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 This is one half of the equation. The second half is what he alludes to with this <blockquote> You can't call Ruby's method from solve_difficult_numerical_problem(). I'll prepare API to support this. </blockquote> If that's available we should have everything we need. <div><p>This is one half of the equation. The second half is what he alludes to with this<br></p> <blockquote><br> You can't call Ruby's method from solve_difficult_numerical_problem(). I'll prepare API to support this.<br></blockquote> <p>If that's available we should have everything we need.</p></div> false 2009-07-29T08:36:59-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T08:37:01-07:00 60086 Martin Robinson Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Found this which may also help: /* * rb_thread_call_with_gvl - re-enter into Ruby world while releasing GVL. * *** *** This API is EXPERIMENTAL! *** We do not guarantee that this API remains in ruby 1.9.2 or later. *** * * While releasing GVL using rb_thread_blocking_region() or * rb_thread_call_without_gvl(), you can not access Ruby values or invoke methods. * If you need to access it, you must use this function rb_thread_call_with_gvl(). * * This function rb_thread_call_with_gvl() does: * (1) acquire GVL. * (2) call passed function `func'. * (3) release GVL. * (4) return a value which is returned at (2). * * NOTE: You should not return Ruby object at (2) because such Object * will not marked. * * NOTE: If an exception is raised in `func', this function "DOES NOT" * protect (catch) the exception. If you have any resources * which should free before throwing exception, you need use * rb_protect() in `func' and return a value which represents * exception is raised. * * NOTE: This functions should not be called by a thread which * is not created as Ruby thread (created by Thread.new or so). * In other words, this function *DOES NOT* associate * NON-Ruby thread to Ruby thread. */ void * rb_thread_call_with_gvl(void *(*func)(void *), void *data1) { <div><p>Found this which may also help:</p> <p>/* * rb_thread_call_with_gvl - re-enter into Ruby world while releasing GVL. *</p> <hr> <p><strong><em>This API is EXPERIMENTAL!</em></strong> We do not guarantee that this API remains in ruby 1.9.2 or later.</p> <hr> <p>* * While releasing GVL using rb_thread_blocking_region() or * rb_thread_call_without_gvl(), you can not access Ruby values or invoke methods. * If you need to access it, you must use this function rb_thread_call_with_gvl(). * * This function rb_thread_call_with_gvl() does: * (1) acquire GVL. * (2) call passed function <code>func'. * (3) release GVL. * (4) return a value which is returned at (2). * * NOTE: You should not return Ruby object at (2) because such Object * will not marked. * * NOTE: If an exception is raised in</code>func', this function "DOES NOT" * protect (catch) the exception. If you have any resources * which should free before throwing exception, you need use * rb_protect() in <code>func' and return a value which represents * exception is raised. * * NOTE: This functions should not be called by a thread which * is not created as Ruby thread (created by Thread.new or so). * In other words, this function *DOES NOT* associate * NON-Ruby thread to Ruby thread. */ void *&lt;br/&gt; rb\_thread\_call\_with\_gvl(void *(*func)(void *), void *data1)&lt;br/&gt; {</code></p></div> false 2009-07-29T08:40:04-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T08:40:10-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Let me try the formatting again: @@@ text /* * rb_thread_call_with_gvl - re-enter into Ruby world while releasing GVL. * *** *** This API is EXPERIMENTAL! *** We do not guarantee that this API remains in ruby 1.9.2 or later. *** * * While releasing GVL using rb_thread_blocking_region() or * rb_thread_call_without_gvl(), you can not access Ruby values or invoke methods. * If you need to access it, you must use this function rb_thread_call_with_gvl(). * * This function rb_thread_call_with_gvl() does: * (1) acquire GVL. * (2) call passed function `func'. * (3) release GVL. * (4) return a value which is returned at (2). * * NOTE: You should not return Ruby object at (2) because such Object * will not marked. * * NOTE: If an exception is raised in `func', this function "DOES NOT" * protect (catch) the exception. If you have any resources * which should free before throwing exception, you need use * rb_protect() in `func' and return a value which represents * exception is raised. * * NOTE: This functions should not be called by a thread which * is not created as Ruby thread (created by Thread.new or so). * In other words, this function *DOES NOT* associate * NON-Ruby thread to Ruby thread. */ void * rb_thread_call_with_gvl(void *(*func)(void *), void *data1) { @@@ <div><p>Let me try the formatting again:</p> <pre> <code class="text"> /* * rb_thread_call_with_gvl - re-enter into Ruby world while releasing GVL. * *** *** This API is EXPERIMENTAL! *** We do not guarantee that this API remains in ruby 1.9.2 or later. *** * * While releasing GVL using rb_thread_blocking_region() or * rb_thread_call_without_gvl(), you can not access Ruby values or invoke methods. * If you need to access it, you must use this function rb_thread_call_with_gvl(). * * This function rb_thread_call_with_gvl() does: * (1) acquire GVL. * (2) call passed function `func'. * (3) release GVL. * (4) return a value which is returned at (2). * * NOTE: You should not return Ruby object at (2) because such Object * will not marked. * * NOTE: If an exception is raised in `func', this function "DOES NOT" * protect (catch) the exception. If you have any resources * which should free before throwing exception, you need use * rb_protect() in `func' and return a value which represents * exception is raised. * * NOTE: This functions should not be called by a thread which * is not created as Ruby thread (created by Thread.new or so). * In other words, this function *DOES NOT* associate * NON-Ruby thread to Ruby thread. */ void * rb_thread_call_with_gvl(void *(*func)(void *), void *data1) {</code> </pre></div> false 2009-07-29T08:41:08-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-07-29T08:41:12-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 We should also look at Ebb: http://github.com/ry/ebb/tree/master http://github.com/ry/ebb/blob/d9c9c1e030ea888ebbbefb9013c726b78eaf2df6/ext/ebb_ffi.c#L91 <div><p>We should also look at Ebb:<br> <a href= "http://github.com/ry/ebb/tree/master">http://github.com/ry/ebb/tree/master</a></p> <p><a href= "http://github.com/ry/ebb/blob/d9c9c1e030ea888ebbbefb9013c726b78eaf2df6/ext/ebb_ffi.c#L91"> http://github.com/ry/ebb/blob/d9c9c1e030ea888ebbbefb9013c726b78eaf2...</a></p></div> false 2009-08-05T04:05:12-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-05T04:05:13-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 The BrowserPlus Ruby module is on github. http://github.com/lloyd/bp-ruby I recommend taking a look at it, especially the following line, which allows threads to run properly (even in Ruby 1.8). http://github.com/lloyd/bp-ruby/blob/a799343d35edec38da3f5e215ab6d27a824a3d8d/src/RubyInterpreter.cpp#L121 <div><p>The BrowserPlus Ruby module is on github. <a href= "http://github.com/lloyd/bp-ruby">http://github.com/lloyd/bp-ruby</a></p> <p>I recommend taking a look at it, especially the following line, which allows threads to run properly (even in Ruby 1.8).</p> <p><a href= "http://github.com/lloyd/bp-ruby/blob/a799343d35edec38da3f5e215ab6d27a824a3d8d/src/RubyInterpreter.cpp#L121"> http://github.com/lloyd/bp-ruby/blob/a799343d35edec38da3f5e215ab6d2...</a></p></div> false 2009-08-11T06:14:53-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-11T06:14:57-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 1 Ok, I've solved all the issues encountered so far, here's the patch: <div><p>Ok, I've solved all the issues encountered so far, here's the patch:</p></div> false 2009-08-19T14:25:00-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-19T14:25:04-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 2 The only things left to do is call RubyModule::Lock() and RubyModule::Unlock() whenever the Ruby C API is accessed, basically in the same places you have to lock the Python GIL. <div><p>The only things left to do is call RubyModule::Lock() and RubyModule::Unlock() whenever the Ruby C API is accessed, basically in the same places you have to lock the Python GIL.</p></div> false 2009-08-19T14:28:59-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-19T14:29:03-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 2 I've updated the patch to Martin's directions and will upload it as soon as I can. There are some issues with the patch as it stands. If you lock Ruby inside another thread that thread then sleeps (only the main one continues). So I think the solution is, in ruby_utils, change the methods on the class to call them in the main thread - I think there's a function already to do this, but I'll need a little help with this (probably use a macro). I'll pop onto IRC next week (as am back then) to discuss. <div><p>I've updated the patch to Martin's directions and will upload it as soon as I can.</p> <p>There are some issues with the patch as it stands. If you lock Ruby inside another thread that thread then sleeps (only the main one continues). So I think the solution is, in ruby_utils, change the methods on the class to call them in the main thread - I think there's a function already to do this, but I'll need a little help with this (probably use a macro).</p> <p>I'll pop onto IRC next week (as am back then) to discuss.</p></div> false 2009-08-26T08:10:16-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-26T08:10:18-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 60086 2 Here's the updated patch. You can see I've created a new method 'RubyKObjectMethodMissingKroll' in ruby_utils.cpp to show how the Thread issue could be easily solved. <div><p>Here's the updated patch. You can see I've created a new method 'RubyKObjectMethodMissingKroll' in ruby_utils.cpp to show<br> how the Thread issue could be easily solved.</p></div> false 2009-08-30T02:09:59-07:00 2171 --- {} 46 ruby-thread-problems 0 25719 new ruby 2009-08-30T11:50:03-07:00 2171 Alex MacCaw Alex MacCaw Martin Robinson http://appcelerator.lighthouseapp.com/projects/25719/tickets/46 29b937aa6029367bd30184c1e4b23a28006b2c32 text/plain 2009-07-29T03:24:03-07:00 textmate_stdin_w2rQID.txt 231789 4996 2171 http://appcelerator.lighthouseapp.com/attachments/231789/textmate_stdin_w2rQID.txt 34eeda81ad18e00aa43167c30befad2dfb9313ae text/plain 2009-08-19T14:25:00-07:00 ruby_19.patch 248048 8834 2171 http://appcelerator.lighthouseapp.com/attachments/248048/ruby_19.patch 05a7ed2f8f90a2b00adba5e22e85f49a5b91b889 text/plain 2009-08-30T02:09:59-07:00 textmate_stdin_hmVnPx.txt 256467 4741 2171 http://appcelerator.lighthouseapp.com/attachments/256467/textmate_stdin_hmVnPx.txt