Error message

Deprecated function: Function create_function() is deprecated in GeSHi->_optimize_regexp_list_tokens_to_string() (line 4736 of /home/sites/site100029201/web/rolandtapken.de/htdocs/sites/all/libraries/geshi/geshi.php).

Create screenshots of a web page using Python and QtWebKit

Wednesday, 3 December 2008

Update 2009-10-03:

For further development and improvements, contact me or have a look at this public github repository created by Adam Nelson.

Update 2010-04-12:

If you need flash support, you should have a look at the current github version of this script at http://github.com/AdamN/python-webkit2png/ mentioned above. We've extend the script a few month ago.

From time to time you may want to create a screenshot of a web page from command line, for example if you wish to create thumbnails for your web-application. So you might search for such a program and find tools like webkit2png, which is for Mac OS X only, or khtml2png, which requires a lot of KDE stuff to be installed on your server.

But since Qt Software, formerly known als Trolltech, integrated Safari's famous rendering engine WebKit (which is based on Konqueror's khtml engine) into its framework, we are now able to make use of it with the help of some Python and PyQt4.

If you are in a hurry, click here to get a full-featured version of webkit2png.py.

I assume that you have some basic knowledge of python. If you run into problems with the Qt part of this tutorial, I suggest to have a look at the class documentation, first. Please note that Qt is a C++ framework, and most of the example code in this documentation has not been ported. So it might be helpful if you have some basic knowledge of C++, too.

Requirements: Webkit and PyQt4 (packages libqt4-webkit and python-qt4 when you're using Intrepid Ibex).

So, run your favourite editor (vim, of course) and start to enter some python code. First, we will have to organize some imports:

#!/usr/bin/env python
import sys # required to exit this program
import signal # required to catch CTRL-C (I'll explain this later)
 
# Some of the PyQt libs
from PyQt4.QtCore import * 
from PyQt4.QtGui import *
from PyQt4.QtWebKit import *

Qt is highly event based (called "slots" and "signals"), so we have to prepare a "slot" which gets called when the page has been loaded completely:

def onLoadFinished(result):
    print "loadFinished(%s)" % str(result)
    sys.exit(0) # this is the moment when we have to quit normally

Even if we intend to write a CLI based application, QtWebkit requires a GUI in the background. This is why we have to use QApplication instead of QCoreApplication. And because we will not have any visible controls, we should ensure that we can still quit this application using CTRL-C (this is why we have to import signal):

app = QApplication(sys.argv)
signal.signal(signal.SIGINT, signal.SIG_DFL)

Now we can create a QWebPage-Object without any exception or segmentation fault. Connect it with our "onLoadFinished"-slot and load the url you want to make a screenshot of (here I'm using Google):

webpage = QWebPage()
webpage.connect(webpage, SIGNAL("loadFinished(bool)"), onLoadFinished)
webpage.mainFrame().load(QUrl("http://www.google.com"))

If you run this application now, you'll see... nothing. onLoadFinished might be called, but the result will be "False". This is because Qt is so extremly event-based, and there is still no main loop to handle these events. So finally you have to start your QApplication:

sys.exit(app.exec_())

If you execute this now, the output should be:

onLoadFinished(True)

Good, the page is loaded! The next step is to render this into a file by expanding "onLoadFinished" (this means: all the code from now on have to be INSIDE of "onLoadFinished"). At first, we should ensure that we do not proceed if we got an error:

<span style="color:lightgray">def onLoadFinished(result):
    <strike>print "loadFinished(%s)" % str(result)</strike></span>
    if not result:
        print "Request failed"
        sys.exit(1)

Otherwise, we should enlarge the viewport (that is our virtual browser window) to the desired size. If you want to create a picture of the whole page, you should use the "preferred" size of the contents:

<span style="color:lightgray">        print "Request failed"
        sys.exit(1)</span>
 
    # Set the size of the (virtual) browser window
    webpage.setViewportSize(webpage.mainFrame().contentsSize()

And finally, render this into an QImage-object and store this into a file:

<span style="color:lightgray">    # Set the size of the (virtual) browser window
    webpage.setViewportSize(webpage.mainFrame().contentsSize()</span>
 
    # Paint this frame into an image
    image = QImage(webpage.viewportSize(), QImage.Format_ARGB32)
    painter = QPainter(image)
    webpage.mainFrame().render(painter)
    painter.end()
    image.save("output.png")
    sys.exit(0) # quit this application

Done. Pretty easy, isn't it? Oh, wait! QWebPage depends an QtGui, and QtGui depends on a running X server (at least on Unix systems). So how can we make use of this on a headless server machine? The answer is Xvfb, a framebuffer based X server, originally designed for testing purposes. Of course, it requires some X-libs and fonts, too (how should a page be rendered without any fonts?), but it does not have so much overhead like the real XOrg-server and don't need to be running all the time. Just call the script this way:

$ xvfb-run --server-args="-screen 0, 640x480x24" python webkit2png-simple.py

The screen size doesn't matter, but the color depth of 24 bit is important. Otherwise, the resulting screenshot would be limited to 256 colors. For more options, have a look at the man-Pages of 'Xvfb' and 'xvfb-run'.

Last, but not least, I'll provide you two versions of this script. webkit2png-simple.py is exactly the result of this tutorial, while webkit2png.py is a much more improved version with command line arguments and coded in OOP style (see the github repository for the most recent version).

Update 2009-04-01 Here's another guy who had the same idea earlier than me.

Python