First have a look at the Requirements For Factories and the Terms Of Use, and maybe sign up with the mailing list.
You will need a user account and to create screenshot factories. Click here to register an account now.
Please let us know if you find problems or have any ideas for improvement, especially how to make it more intuitive and user-friendly.
Find your computer's name
The name for your screenshot factory will be your computer's name by default. Here's how you find it:
But you can also choose a different name for your ShotFactory and then run it with shotfactory -f yourname.
Install the software
We have installation instructions for the following platforms:
Add your browsers
After your screenshot factory is created, you can add browsers to it.
Screenshot factories are the machines that run a fully automatic unattended script to get jobs from the server and make screenshots of browsers.
Hardware:CPU: Pentium III 500 MHz, RAM: 256MB
Software:The reference implementation requires the following software.
If Python is not available for your platform, you could write your own screenshot factory script in your favorite programming language, according to the FactoryInterface specification.
Interface:The screenshot factories communicate with the central server through XML-RPC which is based on HTTP. The FactoryInterface protocol will also work through most firewalls and even through web proxies.
Bandwidth:One screenshot factory can process circa one screenshot per minute. Screenshots are in PNG format for lossless compression. For screens with low resolution (800x600 pixels) and few colors (e.g. 256), the average screenshot size is somewhere around 50 kilobytes. On slightly more modern platforms (1024x768 pixels and 24 bits per pixel), a single screenshot can be as large as half a megabyte. This can amount to 30 megabytes per hour or 720 megabytes in 24 hours. Therefore, the monthly upload bandwidth usage of a single screenshot factory can be in the order of 20 gigabytes.
CPU usage:This depends on the browser and website (e.g. Flash and/or Java). For a busy factory, the CPU usage would average 20% to 50% on a modern computer, but the usage would occur in bursts during the start of the VNC server, during page loading and rendering, and during PNG encoding.
Load limit:On Linux and Mac, you can specify a load limit, the default value is 1.0. If the system load average exceeds the limit, the screenshot factory program will sleep until the load goes down.
Parallel factories:If one machine runs several screenshot factories in parallel, the bandwidth and CPU usage must be multiplied by the number of parallel factories.
The machine powers Browsershots is an example of a central server. If you want, you can set up your own server and use it for private screenshots in your web design company, for example.
Hardware:For smaller loads (e.g. in-house use), a half-modern web server with 512 MB RAM and 40 GB HD should be okay.
Software:OS: Linux (Ubuntu is recommended), Web server (Apache/Nginx), Python (version 2.6 is recommended), NetPBM graphic software
Bandwidth:For the server, the bandwidth requirements must be multiplied by the number of screenshot factories. If there are 20 active factories at any time, the bandwidth for screenshot uploads alone will have to be in the order of half a terabyte per month.
Storage:Hard drive usage for screenshots is quite flexible because when the disk gets filled up, old screenshots can be deleted automatically with a cron job.
Users are those who submit their web site URLs for testing.
Interface:Users can simply use the web frontend through their browser. Use of a web proxy is ok and works just like with any other web site.
Hardware/Software:We have tried to make the service as accessible as we could. Your screen resolution should not make too big a difference. But because the purpose of Browsershots is to show you screenshots of different browsers, you will need a browser that can display images. And because we don't like JPEG artefacts, we use the PNG format which some older browsers don't understand.
Bandwidth:The bandwidth requirements are moderate because the scaled-down screenshot previews are around 70 kilobytes each.
Install prerequisites
sudo apt-get install tightvncserver xfonts-base netpbm xautomation scrot subversion
Get the screenshot factory source code
svn checkout http://browsershots.googlecode.com/svn/trunk/shotfactory browsershots-read-only
Adjust config files
Change the file ~/.vnc/xstartup to the following three lines. This will set a white background and then run nothing except the window manager. Be sure to remove any reference to xterm because it gets in the way.
#!/bin/sh xsetroot -solid "#FFFFFF" x-window-manager &
Run
cd shotfactory screen -L python shotfactory.py
See Troubleshooting for registration and troubleshooting tips.
More useful packages
You will need a lightweight window manager because it will be restarted for each screenshot request. I don't recommend gnome-session because it uses a lot of memory. If you want nice rounded title bars, you can also install ubuntu-artwork.
sudo apt-get install metacity ubuntu-artwork
The firefox package on Ubuntu suggests these font packages (as of 2007-12-01):
sudo apt-get install latex-xft-fonts ttf-kochi-gothic ttf-kochi-mincho ttf-thryomanes ttf-baekmuk ttf-arphic-gbsn00lp ttf-arphic-bsmi00lp ttf-arphic-gkai00mp ttf-arphic-bkai00mp
If you want to run Flash 9, simply install the following package:
sudo apt-get install flashplugin-nonfree
It's also fun to track the system status graphically. Install Munin somewhere and add all your screenshot factory machines to /etc/munin/munin.conf. Then do this on your screenshot factory:
sudo apt-get install lm-sensors smartmontools sudo apt-get install munin-node
Important security notice: don't run the screenshot factory on your Administrator account. The program will load many websites in your browser, and some of them may contain exploits for known security holes. One of my screenshot factories has been hijacked in this way. It's safest to create a new unprivileged user account and run the screenshot factory there.
See also StandaloneInternetExplorer if you want to run different versions of Internet Explorer on one machine.
There are two different ways to install the screenshot factory on windows:
Simple method: binaries for Windows
This method is strongly recommended. It does not require any other installations. Helper programs and the Python interpreter with all required libraries are included.
See Troubleshooting for registration and troubleshooting tips.
Advanced method: source code from the Subversion repository
Use this method if you want to fiddle with the source code.
See Troubleshooting for registration and troubleshooting tips.
Install prerequisites
#!/bin/sh xsetroot -solid "#FFFFFF" x-window-manager &
Get the source code
svn checkout http://browsershots.googlecode.com/svn/trunk/shotfactory browsershots-read-only
Also copy the cscreen program into the shotfactory directory.
Run
cd shotfactory python shotfactory.py
See Troubleshooting for registration and troubleshooting tips.
With this method you can run multiple different versions of Internet Explorer on one machine.
Download
Get the ZIP files that you want from http://browsers.evolt.org/?ie/32bit/standalone.
Install
Unzip each to a subfolder within your shotfactory folder, so for example you would have a file
C:\shotfactory\ie501sp2_nt\iexplore.exe
Register browser versions
Enter "ie501sp2_nt\iexplore.exe" in command box when register your browsers with this form.
If you don't have an account yet, please create a user account
If you get an error saying Factory not found with name=..., please register your screenshot factory.
If you get No browsers registered for factory ..., please visit this page with each of your browsers.
You may use this API to get realtime status of your factories. Based on these status, you are able to make a script to notify you when your factory are down or not working normally.
URL: http://browsershots.org/api/factories/status?factory=[FACTORY_NAME]&password=[PASSWORD]
Arguments
Example Response
{
errors_per_hour: 8,
success: true,
last_upload: "6 seconds ago",
uploads_per_hour: 372,
last_poll: "6 seconds ago",
problems_per_day: 0
}
Run shotfactory.py -h (or --help) to see a list of command line options.
Run shotfactory.py -v (or --verbose) to see more output, including error messages from helper programs.
Run shotfactory.py -vvv (very very verbose) to also see all the commands that are run by the script.
On Linux, you can run xvncviewer :1 to see the virtual screen of the VNC server while a screenshot request is processed.
You can also look at the files ????-pgdn01.ppm or ????.png (where ???? is the request number) to see if there's a problem.
The file shotfactory.log contains error messages for failed screenshot requests.
If you use screen -L, the file screenlog.0 will contain the full output of the shotfactory script.
On Windows, if you run shotfactory.exe directly with a double click, the terminal window may be closed immediately when an error occurs. Try to start the terminal first and then run the program from the command line to avoid this issue.
The factory details page shows recent errors for your factory, with links to each screenshot.
Install Django source code
Browsershots 0.4 uses the development version (trunk) of Django.
svn checkout http://code.djangoproject.com/svn/django/trunk django cd django sudo python setup.py install
Install ShotServer source code
Click here to download shotserver source code
For the quick-start development server, you don't need to install the Browsershots source code in /usr/lib/ because you run the development server from your local copy. Only for use with a real web server like Apache, you would do the following:
cd shotserver sudo python setup.py install
Install the database
You need a recent version of PostgreSQL. If version 8.3 is not available on your distribution, 8.2 will work too.
sudo apt-get install postgresql-8.2 python-psycopg
You can create the database as the same user that will be running the development server, so you don't need to worry about passwords.
createdb shotserver04 CREATE DATABASE
If necessary, create the database user first (replace www-data with your username):
sudo -u postgres createuser www-data Shall the new role be a superuser? (y/n) n Shall the new role be allowed to create databases? (y/n) y Shall the new role be allowed to create more new roles? (y/n) n CREATE ROLE
Let Django create the database tables for you:
cd shotserver/shotserver04 ./manage.py syncdb
When it asks if you would like to create an admin account, say yes.
Create content folders
If you haven't changed the directories in settings.py, you need the following folders:
sudo mkdir -p /var/www/v04.browsershots.org/png
The png folder must be writable by the user running the server.
sudo chown www-data:www-data /var/www/v04.browsershots.org/png
On a production system, you would use Apache 2 or some other serious web server. But to get started quickly, it is recommended that you use the Django development server first, and then switch to a real web server later when it works okay.
Run the development server
You can do this on your normal user account, but you must have write access to the database shotserver04 and to the directory /var/www/browsershots.org/png/.
cd shotserver/shotserver04 ./manage.py runserver
More detailed instructions are available in the Django documentation.
Admin interface
Point your browser to http://127.0.0.1:8000/admin/ and enter your username and password.
Adjust the sites configuration
Create screenshot factories
http://127.0.0.1:8000/admin/factories/factory/add
Web frontend
Point your browser to http://127.0.0.1:8000/ to see the web interface. Replace 127.0.0.1 with the hostname of your development machine if you want to access it from other computers.
Add browsers to your factories
Visit http://127.0.0.1:8000/browsers/add with each browser on your screenshot factory. The page should detect your browser settings automatically. Enter your password and click submit to register your browser in the database.
You can use http://127.0.0.1:8000/admin/browsers/browser/ to add or delete browsers manually or change their settings.
Run Django with mod_python on Apache
After you get the development server to run, you may want to switch to a real web server for performance and stability. See the Django documentation for details, and InstallServerApache for example configuration.
For a production system, you don't want to use the development server that comes with Django. Here's how you set up Apache for Browsershots.
sudo apt-get install apache2 libapache2-mod-python
Create directory for log files:
sudo mkdir -p /var/log/apache2/v04.browsershots.org/
Enable mod_rewrite:
sudo a2enmod rewrite
Then put the following in /etc/apache2/sites-available/v04.browsershots.org:
ExtendedStatus On
NameVirtualHost *:80
<VirtualHost *:80>
ServerName browsershots.example.com # FQDN
ServerAlias browsershots # Local alias
ServerAdmin [email protected]
ServerSignature On
LogLevel warn
ErrorLog /var/log/apache2/v04.browsershots.org/error.log
CustomLog /var/log/apache2/v04.browsershots.org/access.log combined
DocumentRoot /var/www/v04.browsershots.org
<Directory /var/www/v04.browsershots.org>
Options -Indexes
RewriteEngine On
RewriteBase /
# Force canonical hostname.
RewriteCond %{HTTP_HOST} !^browsershots.example.com(|:80)$
RewriteRule ^(.*)$ http://browsershots.example.com/$1 [R,L]
# Static files at server root
RewriteRule ^(favicon.ico|robots.txt)$ static/$1 [L]
# Force HTTPS for secure pages (uncomment next line to enable)
# RewriteRule ^(admin|accounts)(/.*)$ https://browsershots.example.com/$1$2 [R,L]
</Directory>
<Location />
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE shotserver04.settings
PythonDebug Off
PythonAutoReload Off
</Location>
<Location /png>
SetHandler None
</Location>
<Location /static>
SetHandler None
</Location>
<Location /media>
SetHandler None
</Location>
<Location /server-status>
SetHandler server-status
Allow from all
</Location>
</VirtualHost>
# Local variables:
# mode: apache
# end:
Make sure that file is the first symlink in /etc/apache/sites-enabled and then restart Apache:
sudo a2dissite default sudo a2ensite v04.browsershots.org sudo apache2ctl restart
Create some symlinks for static content:
cd /var/www/v04.browsershots.org/ sudo ln -s /usr/lib/python2.5/site-packages/shotserver04/static sudo ln -s /usr/lib/python2.5/site-packages/django/contrib/admin/media
System Components
The Browsershots system consists of a central server and a large number of distributed screenshot factories.
ShotServer: The central server is a powerful web server. It has a web interface that consists of a user interface and a configuration area. The server manages the screenshot requests, uploaded screenshots, and the configuration of the factories.
ShotFactory: The screenshot factories are simple desktop machines with cheap broadband internet. They make screenshots of web pages in different browsers on different platforms. Some of them are only active at night or when they have nothing else to do. The screenshot factory program is fully automatic and runs completely unattended.
Life Cycle of a Screenshot Request
Users submit their web addresses to the central server through a web-based user interface. Then they select browsers and configuration options for their screenshots. The screenshot requests are stored in a database on the central server.
FactoryInterface: The screenshot factories poll the server regularly for requests that match their configuration. If a matching screenshot request is found, the factory loads the requested page in the requested browser and makes screenshots of it. If the page is longer than one screen, several screenshots will be merged to produce a tall picture of the whole page. Finally, the screenshot is compressed as a PNG file and uploaded to the central server.
A few minutes after the screenshot requests were submitted, users can see the results in their browser on the central server's web interface.
This is the specification for the interface between the central server and the distributed screenshot factories.
The central server is a web server with a big fat pipe.
The screenshot factories are simple workstations with cheap broadband internet access (e.g. DSL or cable). They communicate with the server by means of XML-RPC (Remote Procedure Call). This works fine with firewalls and proxies because it's like regular web surfing (except for the frequency and size of file uploads).
The screenshot factory program is a simple script that runs fully unattended. It downloads job information from the central server, runs a web browser and produces screenshots, and finally uploads the resulting PNG files to the central server.
The XML-RPC endpoint is at http://api.browsershots.org/xmlrpc/. The API documentation is available on the same URL, simply go there with your browser.
Step 0: Authentication
Get a challenge from the server and encrypt your password for each API request.
http://api.browsershots.org/xmlrpc/nonces.challenge/
Step 1: Get a screenshot request
Ask the server to find a screenshot request that matches your factory configuration, and lock it (to avoid that two screenshot factories work on the same request).
The result includes a request ID, the browser name and version number, and other request details.
Step 2: Make a screenshot
Set up the screen to match the request parameters, e.g. resolution and color depth.
Construct a redirection URL:
Start your browser with the redirection URL. The central server will store the User-Agent header of your browser and then redirect the browser to the URL for the job, using the status code "302 Found".
Now you have to wait until the browser has finished rendering the page. Then take a screenshot of the whole screen, including window decorations and task bar.
Optionally scroll down, take more screenshots, and merge the screenshots into a tall picture.
Step 3: Upload PNG file
Make a PNG file from the screenshot. Encode it to base64 and send it to the server.
http://api.browsershots.org/xmlrpc/screenshots.upload/
The central server will sanity-check your submission, then make smaller images and store the PNG files on disk. Users can see their screenshots at the following URL: http://browsershots.org/screenshots/
You can now go back to step 1 and check for more jobs.
If you don't want to produce too much load on your host, you can wait until your CPU load average has dropped below some limit. On Linux, this can be done by monitoring /proc/loadavg until the first number is below 1.0 or something.
204
No matching request.
There weren't any screenshot requests that match your factory configuration. The factory should sleep for a minute and then poll again.
All active browsers on ??? are temporarily blocked because of errors.
When a screenshot upload contains errors, the respective browser will be blocked for 10 minutes to reduce the number of errors per hour, and to give other factories a chance to process this request. If all browsers on your factory have errors, request polling will fail until the first browser is unblocked again.
401
Authentication failed (different IP address)
The IP address of your screenshot factory has changed after the XML-RPC call to nonces.challenge. Maybe it is using a different proxy server. For security, the factory must use the same IP address for the challenge and the subsequent API call.
404
No active browsers registered for factory ???.
Please visit the browser registration page with each of the browsers that you want to run on your factory.
Unknown user agent: …
Your browser was probably updated, and the user agent has changed. Visit the browser registration page again with this browser to update the user agent string in the database.
406
The browser has not visited the requested website.
Actually, the browser has not visited the redirect address on the Browsershots server. Maybe another instance of this browser is running, and the uploaded screenshot contains an error message about it. Or maybe your browser didn't have enough time to start up and load the requested page, so you may try to increase the value of the --wait option (default is 30 seconds).
408
Authentication failed (nonce expired).
The delay between the XML-RPC call to nonces.challenge and the subsequent API call was longer than 10 minutes. Maybe your screenshot factory is too busy with other work (like a software upgrade), or your internet connection is too slow.
409
Requested browser version ?.? but got ?.?.
Your screenshot factory has started the wrong browser for this screenshot request. Visit the browser registration page with each browser and enter the correct filename in the Command field.
Request ??? was not locked.
Your screenshot factory has tried to upload a screenshot for a request that wasn't locked. Locking should happen during the call to requests.poll. This is probably a programming error in the shotfactory script or the server.
412
The screenshot is ??? pixels wide, not ??? as requested.
Make sure that you have reschangecon.exe (on Windows) or cscreen (on Mac OS) to adjust your screen resolution. Or if the requested screen resolution is not supported on your factory, you can remove it on the factory page (you have to log in to see the buttons).
413
The screenshot is too tall (more than 4 times the width).
Extremely tall screenshots break the page layout on browsershots.org, and they use up a lot of disk space on the server without much benefit. Please reduce your vertical screen resolution, or the value of the --max-pages option (the default is 7).
414
The screenshot is too short (less than half the width).
If your screen resolution is really wider than 2:1, please write to [email protected] so I can change the limit.
415
Could not decode uploaded PNG file (hashkey ???).
Your screenshot factory uploaded a PNG file that could not be decoded. Please write to [email protected] with details on your software installation (e.g. operating system, installed version of NetPBM).
423
Request ??? was locked by factory ???.
Your screenshot factory took longer than 5 minutes to process the screenshot request, so the lock expired and a different factory started to process this request.
503
The server is currently overloaded. Please try again in a minute.
If the server load average (from /proc/loadavg) exceeds 10, some calls to requests.poll will be randomly rejected to reduce the server load. If the load goes higher, more polls will be rejected. The screenshot factory should sleep for a minute and then try to poll for a screenshot request again.
601
Internet Explorer has encountered a problem
This problem occurs because one of the following browser helper objects (BHO) is installed on your computer: Buyersport, Morpheus, Morpheus Shopping Club, WURLD Shopping Community. Additionally, you logged on to the computer with a limited user account. The BHO tries to modify certain system files and Windows Registry entries that require administrative credentials to access. As a result, when the BHO cannot access these files and registry entries, Internet Explorer stops abruptly. To resolve this problem, remove the BHO from your computer. More information is available in the Microsoft Knowledge Base.
602
Send Error Report
The browser processing the current screenshot request crashed and Application Error Reporting is enabled in Windows system settings. To resolve this error, either disable Application Error Reporting (System Properties, Advanced), or, better, find out why the browser crashed while attempting to load the particularly requested website.
603
Low virtual memory
Your system does not have enough free RAM to continue service. This takes place when the system is overloaded or executing external programs, such as a web server, while the shotfactory is running. For first, try to restart the operating system in order to free unused memory. If the issue is not solved by this, start Task Manager (Ctrl-Shift-Esc), and keep it running minimized. After the error occured the next time, open task manager, have a look at the 'Processes' tab and thus find out which program is consuming most memory.
631
Install language pack
The requested website contains characters that cannot be shown with the language packs currently installed. Usually, this causes Internet Explorer to display a dialog box requesting the on-demand install of additional codepages. The needed packs are often Chinese, Japanese, and other Eastern language packs. Once the additional pack has been installed, make sure to restart the browser to complete the setup. Sometimes even the whole operating system must be restarted. Note: installing language packs requires Administrator privileges. Firefox does not show a dialog asking for on-demand setup, but rather only shows boxes containing the hexadecimal unicode values of the signs not available. A good website to test if Eastern signs are displayed correctly is Toshiba.
651
You have requested an encrypted page
This is not a warning but rather a message telling you that the currently requested website is using the secure SSL protocol. Please be sure to disable such messages in the preferences of your browser in order to hide them from made screenshots.
652
Secure connection
This is not a warning but rather a message telling you that the currently requested website is using the secure SSL protocol. Please be sure to disable such messages in the Internet options of Internet Explorer in order to hide them from made screenshots.
671
Waiting for http
This message indicates that the page could not be loaded in the given wait time. To resolve this problem, first try to open the URL in your normal browser on your desktop computer. Wait some 30 seconds or even longer. If the page loads, increase the wait timeout in the shotfactory (-w parameter). If it does not load, try opening any website on the shotfactory computer. If that works the web server of the requested website is not available and you cannot do anything against the error message as the administrator of the webserver is responsible. If you cannot open any page on your shotfactory computer, there is an error with your internet connection there.
681
Do not show this warning again
Several dialog messages and warnings contain a checkbox to hide such messages in the future. Check the box once to remove the dialogs. This message is also displayed by Internet Explorer if you set the accepting of cookies to 'Confirm' instead of 'Accept' or 'Block'.
692
Recycle Bin
Your screenshot contains the Recycle Bin desktop icon; the requested browser did thus not start in maximized window state. Try to maximize it once, then close it. Most browsers then save your window state preference for the next start. Note: you must close the browser manually after maximizing the window as the shotfactory will not close it softly, but kill the process.
693
Server poll latency
Your screenshot contains the open console window with the shotfactory debug output. Try minimizing the console window, and make sure all browsers can start in the given wait time. If they can't, increase the wait time, using the -w parameter.
701
The screen is blank
Automatically detected error within the screenshot, showing that most of your screen was blank. Usually refers to a browser not being maximized or not having loaded the page fully.
702
The left side of the screen is blank
Automatically detected error within the screenshot, showing that the left side of your screen was blank. Usually refers to a browser not being maximized or not having loaded the page fully.
703
The right side of the screen is blank
Automatically detected error within the screenshot, showing that the right side of your screen was blank. Usually refers to a browser not being maximized or not having loaded the page fully.
704
The bottom of the screen is blank
Automatically detected error within the screenshot, showing that the bottom of your screen was blank. Usually refers to a browser not being maximized or not having loaded the page fully.
This also happens in Safari if there's no scroll bar and the bottom of the website is blank. In that case, you can avoid this error by enabling the status bar in Safari.
811
This is not the requested browser. / This is not ???.
User-reported error telling you that the shot screenshot does not contain the browser requested for this screenshot. Usually refers either to wrong entries within the section 'Registered browsers for this shotfactory' or to wrong program paths / exe calls.
811
This is not the requested browser. / This is not ???.
User-reported error telling you that the shot screenshot does not contain the browser requested for this screenshot. Usually refers either to wrong entries within the section 'Registered browsers for this shotfactory' or to wrong program paths / exe calls.
812
This is not the requested operating system. / This is not ???.
User-reported error telling you that the shot screenshot was not recorded in the operating system requested for this screenshot. Please check your registered browsers and compare them to the ones installed in the shotfactory computer.
821
This is not the requested Javascript version. / Javascript is not ?.?.
Indicates that the screenshot requested a different Javascript version than the one used in the recorded browser. You will have to compare the settings within your Browsershots account to the real Javascript version used in the browser, and update the setting, if needed.
