http://benchmark.ini.rub.de/?section=gtsdb&subsection=dataset

http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset#Downloads

The German Traffic Sign Detection Benchmark

The German Traffic Sign Detection Benchmarkis a single-image detection assessment for researchers with interest in the field of computer vision, pattern recognition and image-based driver assistance. It is introduced on theIEEE International Joint Conference on Neural Networks . It features ...

a single-image detection problem900 images (devided in 600 training images and 300 evaluation images)division into three categories that suit the properties of various detection approaches with different propertiesan online evaluation system with immediate analysis and ranking of the submitted results

Competion Test Dataset

If you want to participate in our competition, please download thetest dataset(530 MB) and process the images by your traffic sign detector. For details on the submission procedure, please refer to section "Submission format and regulations".

Download

Feel free to download thefull data set(1.6 GB) and use it for your purposes. The download package comprises

the 900 training images (1360 x 800 pixels) in PPM formatthe image sections containing only the traffic signsa file in CSV format containing ground trutha ReadMe.txt explainig some details

You can also still download the training and test datasets that were used for the competition (training data set(1.1 GB),test data set(500 MB) ). Momentarily it is still possible to submit results on the test data set in order to compare your performance to that of other teams. For details on the submission procedure, please refer to section "Submission format and regulations".

Image format

The images contain zero to six traffic signs. However, even if there is a traffic sign located in the image it may not belong to the competition relevant categories (prohibitive, danger, mandatory).Images are stored inPPM formatThe sizes of the traffic signs in the images vary from 16x16 to 128x128Traffic signs may appear in every perspective and under every lighting condition

Annotation format

Annotations are provided in CSV files. Fields are seperated by a semicolon (;). They contain the following information:

Filename: Filename of the image the annotations apply forTraffic sign's region of interest (ROI) in the image leftmost image column of the ROIupmost image row of the ROIrightmost image column of the ROIdownmost image row of the ROIID providing the traffic sign's class

You can download source code that will help you to read and compare the annotation files with your detector's results. For an explanation of the class IDs we refer to the ReadMe.txt in the download package.

Submission format and regulations

Similar to the annotation format, the result files that can be submitted during the competition phase (schedule) are supposed to be provided in a CSV file format, seperated by a semicolon (;). The fields are

Filename with extension (without path) of the file your algorithm has detected a traffic signDetection's region of interest (ROI) in the image leftmost image column of the ROIupmost image row of the ROIrightmost image column of the ROIdownmost image row of the ROI

Hence, the result file will contain one line per detection. You can download and use source code with functions to write those result files (C++, Matlab).

In the submission section you will be asked to choose a category (prohibitive, danger, mandatory) for your detection results. Please note that any detection of a sign that is not in that category will be counted as a false detection. For submitting the results of your detector you will have to upload a zip file. The zip file should contain the result text files of several runs with different parametrizations. The results are then evaluated and a precision-recall plot is created thereof. The precision-recall plot is always computed with all files you submitted so far.

Code snippets

C++ Code packageMatlab Code package(updated 01/19/)

Dataset

Overview

Structure

Image format

Annotation format

Result format

Pre-calculated features

Code snippets

Citation

Result analysis application

Downloads

Acknowledgements

Overview

Single-image, multi-class classification problemMore than 40 classesMore than 50,000 images in totalLarge, lifelike databaseReliable ground-truth data due to semi-automatic annotationPhysical traffic sign instances are unique within the dataset

(i.e., each real-world traffic sign only occurs once)

Structure

The training set archive is structures as follows: One directory per classEach directory contains one CSV file with annotations ("GT-<ClassID>.csv") and the training imagesTraining images are grouped by tracksEach track contains 30 images of one single physical traffic sign

Image format

The images contain one traffic sign eachImages contain a border of 10 % around the actual traffic sign (at least 5 pixels) to allow for edge-based approachesImages are stored in PPM format (Portable Pixmap, P6)Image sizes vary between 15x15 to 250x250 pixelsImages are not necessarily squaredThe actual traffic sign is not necessarily centered within the image.This is true for images that were close to the image border in the full camera imageThe bounding box of the traffic sign is part of the annotatinos (see below)

Annotation format

Annotations are provided in CSV files. Fields are separated by ";" (semicolon). Annotations contain the following information:Filename: Filename of corresponding imageWidth: Width of the imageHeight: Height of the imageROI.x1: X-coordinate of top-left corner of traffic sign bounding boxROI.y1: Y-coordinate of top-left corner of traffic sign bounding boxROI.x2: X-coordinate of bottom-right corner of traffic sign bounding boxROI.y2: Y-coordinate of bottom-right corner of traffic sign bounding box

The training data annotations will additionally contain

ClassId: Assigned class labelImportant note: Before January 17, , there were some errors in the annotion data. These errors only affected the the Width and Height information of the whole image (being off 1 pixel in one or both directions), not other columns like ClassId or the ROI information. Thanks to Alberto Escalante for pointing this out. The annotations have been fixed and are included in the training image archive. For those of you who already downloaded the image date set, we provide a ZIP file which contains only the updated annotation data only.

The annotations were created withAdvanced Development & Analysis Framework (ADAF)byNisys GmbH

Result format

The results will be submitted as single CSV.

It contains two columns and no header. The separator is ";"(semicolon).

There is no quoting character for the filename.

First columns is theimage filename, second column is the assigned class id.

The file must contain exactly one entry per element of the test set.

Example:

00000.ppm; 4

00001.ppm; 22

00002.ppm; 16

00003.ppm; 7

00004.ppm; 6

00005.ppm; 2

........

Pre-calculated features

To allow scientists without a background in image processing to participate, we several provide pre-calculated feature sets. Each feature set contains the same directory structure as the training image set. For details on the parameters of the feature algorithm, please have a look at the fileFeature_description.txtwhich is part of each archive file.

HOG features

The file contains three sets of differently configured HOG features (Histograms of Oriented Gradients). The sets contain feature vectors of length 1568, 1568, and 2916 respectively. The features were calculated using the source code fromhttp://pascal.inrialpes.fr/soft/olt/. For detailed information on HOG, we refer to

N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. IEEE Conference on Computer Vision and Pattern Recognition, pages 886-893,

Haar-like features

The file contains one set of Haar-like features. For each image, 5 different types of Haar-like features were computed in different sizes for a total of 12 different features. The overall feature vector contains 11,584 features.

Hue Histograms

For each image in the training set, the file contains a 256-bin histogram of hue values (HSV color space).

Code snippets

Matlab

The Matlab example code provides functions to iterate over the datasets (both training and test) to read the images and the corresponding annotations.

Locations where you can easiliy hook in your training or classification method are marked in the code by dummy function calls.

Please have a look at the fileReadme.txtin the ZIP file for more details

C++

The C++ example code demonstrates how to to train a linear classifier (LDA) using theSharkmachine learning library.

This code uses theprecalculated features. It was used to generate the baseline results.

Please have a look at the fileReadme.txtin the ZIP file for more details

Python

The Python example code provides a function to iterate over the training set to read the images and the corresponding class id.

The code depends onmatplotlib. Please have a look at the fileReadme.txtin the ZIP file for more details

Citation

The data is free to use. However, we cordially ask you to cite the following publication if you do:

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In Proceedings of the IEEE International Joint Conference on Neural Networks , pages 1453–1460. .

Thank you.

Result Analysis Application

We provide a simple application to facilitate result analysis. It allows you to compare different approaches, analyse the confusion matices and inspect which images were classified correctly.

The software is supplied under GPLv2 . It depends on Qt 4.7 , which is available here in source code and binary form. Qt is licensed under LGPL. Qt is a trademark of Nokia Corporation.

The software is provided as source code . The files can be found in the download section . The code is platform-independent, however, it has only been tested on Microsoft Windows with Visual Studio. So there might be a couple of issues left where GCC is more strict than Visual Studio. We appreciate any comments, patches and bug reports.

The project uses CMake , an open-source, cross-platform build system which allows you to generate project files/makefiles for your preferred compiler toolchain.

Here are some screenshots to get an idea of this tool.

Main window: Performance of one or more approaches

Compare multiple approaches and on which images they erred

Confusion matrix: See which classes got confused.

Clicking cells, rows or columns in the confusion matrix shows which images were misclassified.

Here: All "Speed limit 60" images that were incorrectly classified as some other class.

Downloads

Training dataset

This is the official GTSRB training set.If you either intend to participate in the final competition session at IJCNN or you want to publish experimental results based on GTRSB data, you must use this dataset for training.

The training data set contains 39,209 training images in 43 classes.

Images and annotations:Download (263 MB)Three sets of different HOG features:Download (870 MB)

Haar-like features:Download (944 MB)

Hue histograms:Download (17 MB)

Test dataset

Thís is the official GTSRB test set. It was first published at IJCNN during the special session "Traffic Sign Recognition for Machine Learning".All experimental results that are reported on GTSRB data must use this dataset for testing (apart from the ones already published at IJCNN ). The structure of the dataset follows the test set that was published for the online competition (and is now part of the training data).

The test dataset contains 12,630 test images or the corresponding pre-calculated features in random order.

Images and annotations:Download(84 MB)Three sets of different HOG features:Download(278 MB)

Haar-like features:Download(304 MB)

Hue histograms:Download(5 MB)

Extended annotations including class ids:Download(98 kB)

Training dataset(online-competition only!)

This was the training set during the online competition stage of GTSRB. It is kept for reproducibilty reasons. If you want to report experimental results on the GTSRB data, make sure to download the official/final training dataset (see above). It comprises this dataset and the online competition test data (see below).

The training data set contains 26,640 training images in 43 classes.

Images and annotations:Download (178 MB)Fixed annotations(already included in image archive, only required, if you downloaded the image training set before January 17, ):Download (217 kB)Three sets of different HOG features:Download (527 MB)

Haar-like features:Download (595 MB)

Hue histograms:Download (14 MB)

Test dataset (online-competition only!)

This was the test set during the online competition stage of GTSRB. It is kept for reproducibilty reasons. If you want to report experimental results on the GTSRB data, make sure to download the official/final test dataset (see above) which consists of fresh data. The testset below is part of the final training dataset and may not be used to report official results on the GTSRB dataset.

The test dataset contains 12,569 test images or the corresponding pre-calculated features in random order.

Images and annotations:Download (84 MB)Three sets of different HOG features:Download (282 MB)

Haar-like features:Download (304 MB)

Hue histograms:Download (5 MB)

Extended annotations including class ids:Download (102 kB)Sortedtestsets (i.e., same directory and filename structure as the training set, elements are sorted by class and track): Images and annotations:Download (84 MB)HOG features:Download (282 MB)

Haar-like features:Download (304 MB)

Hue histograms:Download (5 MB)

Code

Example code for Matlab to read all training and test images including annotations:DownloadExample code for C++ to train a LDA classifier using theSharkmachine learning library:DownloadExample code for Python to read all training images:Download

Result analysis application

The code is platform-independent, however, it has only been tested Visual Studio. So there might be a couple of issues left where GCC is more strict than Visual Studio.

We appreciate any comments, patches and bug reports. The code usesCMake, an open-source, cross-platform build system which allows you to generate project files/makefiles for your preferred compiler toolchain.

Readme.txt

Source code:tsr-analysis-src.zip (503 kB)

Make sure to check thenews pageregularly for updates. If yousign up, you will be notified about important updates by email.

Acknowledgements

We would not have been able to provide this benchmark dataset without the extensive and valuable help of others.

Many thanks to Lukas Caup, Sebastian Houben, Lukas Kubik, Bastian Petzka, Stefan Tenbült, Marc Tschentscher for their annotation support, to Sebastian Houben for providing the Matlab code samples, Lukas Kubik and especially Bastian Petzka for creation of this web site.

Furthermore, we thankNisys GmbHfor their support and for providing the Advanced Development & Analysis Framework.