首页>Program>source

创建多处理/ GUI编码系统的最佳方法是什么?

我想为互联网社区创建一个场所,以找到有关如何使用 multiprocessing的示例 python中的模块.

我已经看到了 multiprocessing的几个小例子 Internet上具有简单全局功能的进程,这些进程在一个主模块中被调用,但是我发现,这很少会轻易地转化为任何人实际上对GUI所做的任何事情.我认为许多程序将具有在单独的过程中作为对象方法使用的功能(可能是其他对象的集合等),也许单个GUI元素将具有关联的对象,需要调用此函数 流程等

例如,我有一个相对复杂的程序,并且在获取响应式GUI时遇到问题,我认为这是由于我对 multiprocessing缺乏了解 和 QThread一起穿线 .但是,我确实知道,下面给出的示例至少会以我希望的方式在进程之间传递信息(由于能够执行 print 语句),但我的GUI仍处于锁定状态.有人知道这是什么原因吗?如果我仍然缺乏对多线程/多处理体系结构的了解,这是否仍然是一个问题?

这是我正在做的一小段伪代码示例:

class Worker:
    ...
    def processing(self, queue):
        # put stuff into queue in a loop
# This thread gets data from Worker
class Worker_thread(QThread):
    def __init__(self):
        ...
        # make process with Worker inside
    def start_processing(self):
        # continuously get data from Worker
        # send data to Tab object with signals/slots
class Tab(QTabWidget):
    # spawn a thread separate from main GUI thread
    # update GUI using slot
    def update_GUI()

此代码是完全可编译的示例,体现了我程序的总体结构:

from PyQt4 import QtCore, QtGui
import multiprocessing as mp
import numpy as np
import sys
import time
# This object can hold several properties which will be used for the processing
# and will be run in the background, while it updates a thread with all of it's progress
class Worker:
    def __init__(self, some_var):
        self.some_var = some_var
        self.iteration = 0
    def some_complex_processing(self, queue):
        for i in range(0,5000):
            self.iteration += 1
            queue.put(self.iteration)
        queue.put('done with processing')
# This Woker_thread is a thread which will spawn a separate process (Worker).
# This separate is needed in order to separate the data retrieval
# from the main GUI thread, which should only quickly update when needed 
class Worker_thread(QtCore.QThread):
    # signals and slots are used to communicate back to the main GUI thread
    update_signal = QtCore.pyqtSignal(int)
    done_signal = QtCore.pyqtSignal()
    def __init__(self, parent, worker):
        QtCore.QThread.__init__(self, parent)
        self.queue = mp.Queue()
        self.worker = worker
        self.parent = parent
        self.process = mp.Process(target=self.worker.some_complex_processing, args=(self.queue,))
    # When the process button is pressed, this function will start getting data from Worker
    # this data is then retrieved by the queue and pushed through a signal
    # to Tab.update_GUI
    @QtCore.pyqtSlot()
    def start_computation(self):
        self.process.start()
        while(True):
            try:
                message = self.queue.get()
                self.update_signal.emit(message)
            except EOFError:
                pass
            if message == 'done with processing':
                self.done_signal.emit()
                break
            #self.parent.update_GUI(message)
        self.process.join()
        return
# Each tab will start it's own thread, which will spawn a process
class Tab(QtGui.QTabWidget):
    start_comp = QtCore.pyqtSignal()
    def __init__(self, parent, this_worker):
        self.parent = parent
        self.this_worker = this_worker
        QtGui.QTabWidget.__init__(self, parent)
        self.treeWidget = QtGui.QTreeWidget(self)
        self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
        self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
        self.thread = Worker_thread(parent=self, worker=self.this_worker)
        self.thread.update_signal.connect(self.update_GUI)
        self.thread.done_signal.connect(self.thread.quit)
        self.start_comp.connect(self.thread.start_computation)
        self.thread.start()
    ###############################
    # Here is what should update the GUI at every iteration of Worker.some_complex_processing()
    # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated.
    @QtCore.pyqtSlot(int)
    def update_GUI(self, iteration):
        self.step.setText(0, str(iteration))
        #time.sleep(0.1)
        print iteration
    def start_signal_emit(self):
        self.start_comp.emit()
# GUI stuff
class MainWindow(QtGui.QMainWindow):
    def __init__(self, parent = None):
        QtGui.QMainWindow.__init__(self)
        self.tab_list = []
        self.setTabShape(QtGui.QTabWidget.Rounded)
        self.centralwidget = QtGui.QWidget(self)
        self.top_level_layout = QtGui.QGridLayout(self.centralwidget)
        self.tabWidget = QtGui.QTabWidget(self.centralwidget)
        self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)
        process_button = QtGui.QPushButton("Process")
        self.top_level_layout.addWidget(process_button, 0, 1)
        QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)
        self.setCentralWidget(self.centralwidget)
        self.centralwidget.setLayout(self.top_level_layout)
        # Make Tabs in loop from button
        for i in range(0,10):
            name = 'tab' + str(i)
            self.tab_list.append(Tab(self.tabWidget, Worker(name)))
            self.tabWidget.addTab(self.tab_list[-1], name)
    # Do the processing
    def process(self):
        for tab in self.tab_list:
            tab.start_signal_emit()
        return
if __name__ == "__main__":
    app = QtGui.QApplication([])
    win = MainWindow()
    win.show()
    sys.exit(app.exec_())

More Information: 我正在编写一个程序,希望从中产生多个过程,并使它们在整个处理过程中不断显示其进度.我希望对该程序进行多进程处理,以使程序获得最佳速度。

目前,我正在尝试使用线程来生成进程,并使用信号和插槽来更新GUI,同时队列不断检索数据.看来, queuessignalsslots 使用 print时工作 语句,但无法更新GUI.如果有人对我应该如何构造它以使程序更易于管理有其他建议,我想学习。

EDIT :我进行了敏琳(Min Lin)所做的调整,并添加了 Worker QObject 以便使 会起作用。
这是我目前拥有的新代码:

moveToThread()

感谢您提供所有答案,我感谢每个人在描述他们认为是解决方案的想法时所投入的详细程度,但不幸的是,我尚未能够执行在 在GUI上显示对象的属性时,它们所属的对象。
但是,我从这篇文章中学到了很多,这使我意识到,由于GUI更新功能太大且需要太多处理,因此我目前拥有的线程版本正在挂起GUI。

所以,我服用了 from PyQt4 import QtCore, QtGui import multiprocessing as mp import numpy as np import sys import time class Worker(QtCore.QObject): update_signal = QtCore.pyqtSignal(int) done_signal = QtCore.pyqtSignal() def __init__(self, some_var): QtCore.QObject.__init__(self, parent=None) self.some_var = some_var self.iteration = 0 self.queue = mp.Queue() self.process = mp.Process(target=self.some_complex_processing, args=(self.queue,)) def some_complex_processing(self, queue): for i in range(0,5000): self.iteration += 1 queue.put(self.iteration) queue.put('done with processing') @QtCore.pyqtSlot() def start_computation(self): self.process.start() while(True): try: message = self.queue.get() self.update_signal.emit(message) except EOFError: pass if message == 'done with processing': self.done_signal.emit() break self.process.join() return class Tab(QtGui.QTabWidget): start_comp = QtCore.pyqtSignal() def __init__(self, parent, this_worker): self.parent = parent self.this_worker = this_worker QtGui.QTabWidget.__init__(self, parent) self.treeWidget = QtGui.QTreeWidget(self) self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"]) self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"]) # Use QThread is enough self.thread = QtCore.QThread(); # Change the thread affinity of worker to self.thread. self.this_worker.moveToThread(self.thread); self.this_worker.update_signal.connect(self.update_GUI) self.this_worker.done_signal.connect(self.thread.quit) self.start_comp.connect(self.this_worker.start_computation) self.thread.start() ############################### # Here is what should update the GUI at every iteration of Worker.some_complex_processing() # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated. @QtCore.pyqtSlot(int) def update_GUI(self, iteration): self.step.setText(0, str(iteration)) #time.sleep(0.1) print iteration def start_signal_emit(self): self.start_comp.emit() # GUI stuff class MainWindow(QtGui.QMainWindow): def __init__(self, parent = None): QtGui.QMainWindow.__init__(self) self.tab_list = [] self.setTabShape(QtGui.QTabWidget.Rounded) self.centralwidget = QtGui.QWidget(self) self.top_level_layout = QtGui.QGridLayout(self.centralwidget) self.tabWidget = QtGui.QTabWidget(self.centralwidget) self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25) process_button = QtGui.QPushButton("Process") self.top_level_layout.addWidget(process_button, 0, 1) QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process) self.setCentralWidget(self.centralwidget) self.centralwidget.setLayout(self.top_level_layout) # Make Tabs in loop from button for i in range(0,10): name = 'tab' + str(i) self.tab_list.append(Tab(self.tabWidget, Worker(name))) self.tabWidget.addTab(self.tab_list[-1], name) # Do the processing def process(self): for tab in self.tab_list: tab.start_signal_emit() return if __name__ == "__main__": app = QtGui.QApplication([]) win = MainWindow() win.show() sys.exit(app.exec_()) 我的多线程版本的方法,它的性能要好得多! 我建议任何面临类似问题的人至少尝试类似的尝试.

我还没有意识到这种解决GUI更新问题的方法,现在它只是针对我所面临问题的伪或临时解决方案。

QTimer()
最新回答
  • 11天前
    1 #

    GUI应用程序非常适合测试内容,因为它很容易产生新任务并可视化正在发生的事情,所以我写了一个小示例应用程序(屏幕截图,下面的代码),因为我确实想为我学习它 自我。

    起初,我采用了与您相似的方法,试图实现"消费者/生产者"模式,并且我在后台进程中苦苦挣扎,不断循环以等待新工作,并自己来回进行通信.然后我发现了池接口,然后我可以用几行代码替换所有这些令人讨厌的代码.您只需要一个池和一些回调:

    #!/usr/bin/env python3
    import multiprocessing, time, random, sys
    from PySide.QtCore import * # equivalent: from PyQt4.QtCore import *
    from PySide.QtGui import *   # equivalent: from PyQt4.QtGui import *
    def compute(num):
        print("worker() started at %d" % num)
        random_number = random.randint(1, 6)
        if random_number in (2, 4, 6):
            raise Exception('Random Exception in _%d' % num)
        time.sleep(random_number)
        return num
    class MainWindow(QMainWindow):
        def __init__(self):
            QMainWindow.__init__(self)
            self.toolBar = self.addToolBar("Toolbar")
            self.toolBar.addAction(QAction('Add Task', self, triggered=self.addTask))
            self.list = QListWidget()
            self.setCentralWidget(self.list)
            # Pool of Background Processes
            self.pool = multiprocessing.Pool(processes=4)
        def addTask(self):
            num_row = self.list.count()
            self.pool.apply_async(func=compute, args=(num_row,), callback=self.receiveResult,
                                  error_callback=self.receiveException)
            item = QListWidgetItem("item %d" % num_row)
            item.setForeground(Qt.gray)
            self.list.addItem(item)
        def receiveResult(self, result):
            assert isinstance(result, int)
            print("end_work(), where result is %s" % result)
            self.list.item(result).setForeground(Qt.darkGreen)
        def receiveException(self, exception):
            error = str(exception)
            _pos = error.find('_') + 1
            num_row = int(error[_pos:])
            item = self.list.item(num_row)
            item.setForeground(Qt.darkRed)
            item.setText(item.text() + ' Retry...')
            self.pool.apply_async(func=compute, args=(num_row,), callback=self.receiveResult,
                                  error_callback=self.receiveException)
    if __name__ == '__main__':
        app = QApplication(sys.argv)
        main_window = MainWindow()
        main_window.show()
        sys.exit(app.exec_())
    

    编辑:我做了另一个使用QTimer而不是回调的示例,它定期检查队列中的条目,并更新QProgressBar:

    #!/usr/bin/env python3
    import multiprocessing, multiprocessing.pool, time, random, sys
    from PySide.QtCore import *
    from PySide.QtGui import *
    def compute(num_row):
        print("worker started at %d" % num_row)
        random_number = random.randint(1, 10)
        for second in range(random_number):
            progress = float(second) / float(random_number) * 100
            compute.queue.put((num_row, progress,))
            time.sleep(1)
        compute.queue.put((num_row, 100))
    def pool_init(queue):
        # see http://stackoverflow.com/a/3843313/852994
        compute.queue = queue
    class MainWindow(QMainWindow):
        def __init__(self):
            QMainWindow.__init__(self)
            self.toolBar = self.addToolBar("Toolbar")
            self.toolBar.addAction(QAction('Add Task', self, triggered=self.addTask))
            self.table = QTableWidget()
            self.table.verticalHeader().hide()
            self.table.setColumnCount(2)
            self.setCentralWidget(self.table)
            # Pool of Background Processes
            self.queue = multiprocessing.Queue()
            self.pool = multiprocessing.Pool(processes=4, initializer=pool_init, initargs=(self.queue,))
            # Check for progress periodically
            self.timer = QTimer()
            self.timer.timeout.connect(self.updateProgress)
            self.timer.start(2000)
        def addTask(self):
            num_row = self.table.rowCount()
            self.pool.apply_async(func=compute, args=(num_row,))
            label = QLabel("Queued")
            bar = QProgressBar()
            bar.setValue(0)
            self.table.setRowCount(num_row + 1)
            self.table.setCellWidget(num_row, 0, label)
            self.table.setCellWidget(num_row, 1, bar)
        def updateProgress(self):
            if self.queue.empty(): return
            num_row, progress = self.queue.get() # unpack
            print("received progress of %s at %s" % (progress, num_row))
            label = self.table.cellWidget(num_row, 0)
            bar = self.table.cellWidget(num_row, 1)
            bar.setValue(progress)
            if progress == 100:
                label.setText('Finished')
            elif label.text() == 'Queued':
                label.setText('Downloading')
            self.updateProgress() # recursion
    if __name__ == '__main__':
        app = QApplication(sys.argv)
        main_window = MainWindow()
        main_window.show()
        sys.exit(app.exec_())
    

  • 11天前
    2 #

    非常感谢您发布此问题,并感谢所有贡献者提供他们的意见.它为我进行PyQt和多处理实验提供了有用的支架。

    我从问题中列出的第二个代码示例开始.我的更改和评论:

      On Windows,allarguments to Process.__init__() 必须是可腌制的.您会看到@valmynd使他/她成为 compute 正因为如此,功能是顶级功能.部分原因是多处理将重新导入包含目标函数的模块.为了提醒自己,我尝试将目标函数放入其自己的模块中(并确保将任何信息作为可腌制的参数传递).我已经将复杂的处理功能移到了自己的名为 workermodule.py的模块中 .

      在复杂的处理功能中没有足够的工作,循环完成得太快,以至于任何更改都无法在GUI中看到.因此,我在复杂的处理函数中添加了一些额外的(无用的)工作.如评论中所述,您可以 time.sleep ,但稍微点亮一下所有内核会更令人满意。

      通过以下两个代码段,我得到了一个平滑的GUI,它不断更新迭代值,并且子进程全速运行。

      请注意, self.process 可以使用两个队列作为参数创建,一个用于输入,一个用于输出.然后,复杂的处理功能将不得不定期检查输入队列中的数据。

      workermodule.py:

      import time
      def some_complex_processing(queue):
          iteration = 0
          for i in range(0,5000):
              iteration += 1
              queue.put(iteration)
              #time.sleep(20e-3) # ms
              # You could time.sleep here to simulate a
              # long-running process, but just to prove
              # that we're not cheating, let's make this
              # process work hard, while the GUI process
              # should still have smooth drawing.
              for x in range(100000):
                  y = x
          queue.put('done with processing')
      

      mainfile.py:

      from PyQt4 import QtCore, QtGui
      import multiprocessing as mp
      import sys
      import workermodule
      class Worker(QtCore.QObject):
          update_signal = QtCore.pyqtSignal(int)
          done_signal = QtCore.pyqtSignal()
          def __init__(self, some_var):
              QtCore.QObject.__init__(self, parent=None)
              self.some_var = some_var
              self.queue = mp.Queue()
              self.process = mp.Process(
                  target=workermodule.some_complex_processing,
                  args=(self.queue,)
                  )
          @QtCore.pyqtSlot()
          def start_computation(self):
              self.process.start()
              while True:
                  try:
                      message = self.queue.get()
                      self.update_signal.emit(message)
                  except EOFError:
                      pass
                  if message == 'done with processing':
                      self.done_signal.emit()
                      break
              self.process.join()
              return
      class Tab(QtGui.QTabWidget):
          start_comp = QtCore.pyqtSignal()
          def __init__(self, parent, this_worker):
              self.parent = parent
              self.this_worker = this_worker
              QtGui.QTabWidget.__init__(self, parent)
              self.treeWidget = QtGui.QTreeWidget(self)
              self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
              self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
              # Use QThread is enough
              self.thread = QtCore.QThread();
              # Change the thread affinity of worker to self.thread.
              self.this_worker.moveToThread(self.thread);
              self.this_worker.update_signal.connect(self.update_GUI)
              self.this_worker.done_signal.connect(self.thread.quit)
              self.start_comp.connect(self.this_worker.start_computation)
              self.thread.start()
          ###############################
          # Here is what should update the GUI at every iteration of Worker.some_complex_processing()
          # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated.
          @QtCore.pyqtSlot(int)
          def update_GUI(self, iteration):
              self.step.setText(0, str(iteration))
              print iteration
          def start_signal_emit(self):
              self.start_comp.emit()
      # GUI stuff
      class MainWindow(QtGui.QMainWindow):
          def __init__(self, parent = None):
              QtGui.QMainWindow.__init__(self)
              self.tab_list = []
              self.setTabShape(QtGui.QTabWidget.Rounded)
              self.centralwidget = QtGui.QWidget(self)
              self.top_level_layout = QtGui.QGridLayout(self.centralwidget)
              self.tabWidget = QtGui.QTabWidget(self.centralwidget)
              self.top_level_layout.addWidget(self.tabWidget, 1, 0, 25, 25)
              process_button = QtGui.QPushButton("Process")
              self.top_level_layout.addWidget(process_button, 0, 1)
              QtCore.QObject.connect(process_button, QtCore.SIGNAL("clicked()"), self.process)
              self.setCentralWidget(self.centralwidget)
              self.centralwidget.setLayout(self.top_level_layout)
              # Make Tabs in loop from button
              for i in range(0,10):
                  name = 'tab' + str(i)
                  self.tab_list.append(Tab(self.tabWidget, Worker(name)))
                  self.tabWidget.addTab(self.tab_list[-1], name)
          # Do the processing
          def process(self):
              for tab in self.tab_list:
                  tab.start_signal_emit()
              return
      if __name__ == "__main__":
          app = QtGui.QApplication([])
          win = MainWindow()
          win.show()
          sys.exit(app.exec_())
      

  • 11天前
    3 #

    您在worker_Thread中完成的所有操作应已移至worker。 Qt根据对象的创建位置将线程亲和力分配给每个对象. worker_Thread对象是在主线程中创建的,因此具有主线程亲和力.如果来自主线程的信号连接到在主线程中创建的对象的插槽,则该插槽也将在主线程中执行. (无论是QueuedConnection还是DirectConnection).然后插槽会阻塞GUI。

    执行此操作:

    class Worker:
        update_signal = QtCore.pyqtSignal(int)
        done_signal = QtCore.pyqtSignal()
        def __init__(self, some_var):
            self.some_var = some_var
            self.iteration = 0
            self.queue = mp.Queue()
            self.process = mp.Process(target=self.some_complex_processing, args=(self.queue,))
        def some_complex_processing(self, queue):
            for i in range(0,5000):
                self.iteration += 1
                queue.put(self.iteration)
            queue.put('done with processing')
        @QtCore.pyqtSlot()
        def start_computation(self):
            self.process.start()
            while(True):
                try:
                    message = self.queue.get()
                    self.update_signal.emit(message)
                except EOFError:
                    pass
                if message == 'done with processing':
                    self.done_signal.emit()
                    break
            self.process.join()
            return
    
    class Tab(QtGui.QTabWidget):
        start_comp = QtCore.pyqtSignal()
        def __init__(self, parent, this_worker):
            self.parent = parent
            self.this_worker = this_worker
            QtGui.QTabWidget.__init__(self, parent)
            self.treeWidget = QtGui.QTreeWidget(self)
            self.properties = QtGui.QTreeWidgetItem(self.treeWidget, ["Properties"])
            self.step = QtGui.QTreeWidgetItem(self.properties, ["Iteration #"])
            # Use QThread is enough
            self.thread = QtCore.QThread();
            # Change the thread affinity of worker to self.thread.
            self.this_worker.moveToThread(self.thread);
            self.this_worker.update_signal.connect(self.update_GUI)
            self.this_worker.done_signal.connect(self.thread.quit)
            self.start_comp.connect(self.this_worker.start_computation)
            self.thread.start()
        ###############################
        # Here is what should update the GUI at every iteration of Worker.some_complex_processing()
        # The message appears to be getting sent, due to seeing the print statement in the console, but the GUI is not updated.
        @QtCore.pyqtSlot(int)
        def update_GUI(self, iteration):
            self.step.setText(0, str(iteration))
            #time.sleep(0.1)
            print iteration
        def start_signal_emit(self):
            self.start_comp.emit()
    

  • 11天前
    4 #

    好,我对Qt本身并不熟悉,但是我已经对Tkinter做过类似的工作.我相当确定您在这里遇到了Python的Global Interpreter Lock.具体来说,您是在同一线程中启动队列和GUI应用程序,因此,当队列阻塞等待输入时,GUI也阻塞.尝试启动 app = QtGui.QApplication([]) 在自己的线程或进程中.用队列编写GUI总是很复杂,我发现它通常比我最初期望的多至少一层线程。

  • 在Python中将表格作为电子邮件正文(而非附件)发送
  • git:使用pull时快进,合并分支时no-ff